Systems and methods for intervention in artificial intelligence models

Abstract

According to several embodiments, a computer implementation method for interventions in an artificial intelligence (AI) model is provided. The method includes the step of obtaining a request from a user computer. The method includes the step of obtaining intervention information applicable to the request. The method includes the step of generating an augmentation request based on the obtained request and the obtained intervention information. The method includes the step of providing the augmentation request as input to the AI ​​model. The method includes the step of obtaining a response to the augmentation request from the AI ​​model. The method includes the step of transmitting the obtained response to the user computer.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application incorporates the entirety of U.S. Provisional Patent Applications No. 63 / 501,147, filed on 9 May 2023, No. 63 / 501,148, filed on 9 May 2023, No. 63 / 517,900, filed on 5 August 2023, and No. 63 / 517,929, filed on 6 August 2023, by claiming and referencing priority thereto.

[0002] This application claims the benefit as of the filing date of U.S. Provisional Patent Applications No. 63 / 501,147, filed on 9 May 2023, No. 63 / 501,148, filed on 9 May 2023, No. 63 / 517,900, filed on 5 August 2023, and No. 63 / 517,929, filed on 6 August 2023, in accordance with Section 35, 119(e) of the U.S. Act.

[0003] Aspects of this disclosure relate to improving artificial intelligence systems and methods, and more specifically to improved systems and methods for intervention in artificial intelligence models. [Background technology]

[0004] For over 20 years, internet search providers have relied on sponsored search as a preferred way to derive revenue from their search engines. This practice can be traced back at least to 1996, when Open Text launched its "Preferred Listing" service, which allowed companies to pay for higher rankings on search results pages. Many of the well-known features of sponsored search today, including bidding for position in keyword auctions and payments for pay-per-click advertising, were introduced in 1998-99 by GoTo.com (later known as Overture) (see Davis et al., U.S. Patent No. 6,269,361). When Google launched its own sponsored search service in 2002, it made two further innovations: refining the auction format for generalized second-price auctions and beginning to adjust the ranking order of advertisers' bids by a "quality score" related to click-through rate, ad relevance, landing page experience, and site quality (see, for example, Fain and Pedersen, 2006; Edelman, Ostrovsky and Schwarz, 2007; Varian, 2007; and Jansen and Mullen, 2008).

[0005] It should be noted that throughout the heyday of the internet, it is difficult to identify any search engine model that provides users with pure and unprocessed search results over any given period of time. Instead, it is found that, as a general rule and not as an exception, search results are subject to intervention.

[0006] In this specification, intervention in a model generally means introducing a modification to the model that alters the output produced by the model. For example, this could refer to an intervention in a search engine model or an intervention in an artificial intelligence model (more specifically, an intervention in a search engine model that incorporates artificial intelligence). The verb form of intervention may also be used; that is, when it is said that an intervention is made in a model, this would be synonymous with introducing or implementing an intervention within that model. In many cases, the interventions discussed herein will take the form of providing additional information to the model or modifying how the model utilizes that information. A clearer meaning of the term “intervention” can be obtained by examining the examples discussed throughout this specification.

[0007] The history of sponsored search provides at least two early examples of intervention in search engine models. The first is GoTo.com's insertion of sponsored (paid) links at the top of organic (free) links, representing a modification of the ordered list of links that would otherwise be generated by the search engine model, based on advertiser bids (potentially, it could be argued that this is not an intervention because the organic links themselves generated by the search engine model remain unchanged, and instead, a separate set of sponsored links is inserted as a whole. However, this argument is considered dishonest because (1) the U.S. Federal Trade Commission has pointed out that search engines do not properly label sponsored links (Hansen, 2002; Tibken, 2013; FTC, 2017), and (2) the Pew Research Center found that 38% of survey respondents did not recognize the distinction between sponsored and organic links, and less than 17% of survey respondents could not consistently distinguish between sponsored and organic links (Fallows, 2005)). The second is Google's adjustment of bids based on Quality Score, which in itself is potentially an intervention in the sponsored search auction model by altering the ordered list of sponsored links determined by the auction (again, potentially, this is not an intervention; i.e., it is actually part of the process of determining the inherent order of sponsored links, and it could be argued that, all else equal, higher-quality advertisers naturally rank higher on the search page. However, the determination of Quality Score is highly opaque, and the rewards from price differentiation are very high for advertisers with high pay bids, so it is natural to assume that Quality Score incorporates more than just the factors mentioned above, beyond what can be attributed to quality).

[0008] The insertion of sponsored links on search results pages and the adjustment of advertiser bids based on quality scores are just a few examples of interventions in search engine models currently in use. Not all interventions are greedy or difficult to justify. For example, some interventions may occur to prevent users from receiving links to pornographic content or malicious websites. Others may occur to reduce the probability of users receiving links to websites that spread false information. Still others may occur to potentially reflect the ideological objectives of the search engine model's owner. This specification strives to avoid adding any value judgments to any intervention and chooses to focus solely on the systems and methods for intervention.

[0009] Technology companies have developed a broad toolbox of highly effective interventions in search engine models, which may be a major contributor to Alphabet Inc.'s (Google's parent company) market capitalization of over $1 trillion. [Overview of the Initiative] [Means for solving the problem]

[0010] However, the existing toolbox for interventions is largely unadjusted for the new generation of artificial intelligence models, driven by ChatGPT. There are two fundamental reasons for this. First, for the past 20 years or more, the relevant output to users of search engines has been an ordered list of hyperlinks. As a result, the existing toolbox for interventions has developed around manipulating outputs that consist of ordered lists of links. However, newly emerging artificial intelligence models are not limited to generating ordered lists of links; more usefully, they can generate unordered, free-form paragraphs of prose or other data outputs. When the format of the underlying response itself is no longer an ordered list of links, it remains to be seen whether ordered lists of sponsored hyperlinks remain an effective way to monetize search requests. Second, the relevant input to conventional search engines is short combinations of search terms, giving rise to the concept of "keywords." However, newly emerging artificial intelligence models are not limited to receiving short combinations of search terms; more usefully, they can interpret increasingly complex questions and engage in relatively nuanced interactions. Keywords can be a rough tool for identifying whether stakeholders are willing to intervene in richly expressed demands and the amount they would be willing to pay for such intervention.

[0011] To elaborate on this point, let's consider today's keyword auction systems based on the application of Generalized Second Price (GSP) auctions. Each advertiser submits a bid for a keyword. In the pure form of GSP, the highest bidder secures the top position on the page and pays the second highest bid, the second highest bidder secures the second position on the page and pays the third highest bid, and so on. However, in the future, as the majority of internet searches are performed using generative artificial intelligence systems, securing the top or second position may no longer be significant, and "organic" output may no longer differ significantly from current search pages. Instead, AI-based search engines are expected to write traditional prose, and users may prefer to receive a single answer or a few recommendations rather than browsing a list of links.

[0012] Similarly, emerging technologies may be obsolete the concept of "keywords." To illustrate this, a 2020 Google video titled "Google Ads Tutorials: How the Search Ad Auctions Work" discusses a hypothetical stakeholder selling children's shoes. It assumes bidding on six possible keywords: "kids shoes," "shoes for kids," "toddler shoes," "kids sneakers," "kids sandals," and "babies first shoes." However, with artificial intelligence now available, should this process still be followed? The keywords "kids shoes" and "shoes for kids" are perfectly synonymous, and the others are also very similar; therefore, should they be bid on separately? Even at the time of writing this patent application, Google searches for "kids shoes" and "shoes for kids" yielded different sponsored hyperlinks arranged in different orders. As user requests shift away from short combinations of search terms towards more nuanced questions or repetitive chats, it can be inferred that keywords may become an increasingly dull and ineffective means for stakeholders to express interest in user requests.

[0013] The aspects of this disclosure relate to the following two aspects:

[0014] An approach to intervention output that is more consistent with the output of newly emerging artificial intelligence systems than approaches using existing technologies.

[0015] An approach to intervention input that is more consistent with the inputs of newly emerging artificial intelligence systems than approaches using existing technologies.

[0016] When limited to interventions in existing technologies, providers of artificial intelligence-based search engine models may be constrained to a combination of the following approaches for monetizing those search models:

[0017] The output pages of newly emerging AI systems may be preceded by sponsored links, which are frequently found in today's search pages, or adorned with display ads, which are frequently employed by newspapers and other websites. However, once users become accustomed to using newly emerging AI-based search models, they are likely to pay less and less attention to the surrounding sponsored links or display ads. Furthermore, ad blockers themselves are likely to evolve, incorporating more advanced artificial intelligence to make them increasingly effective against both display ads and sponsored links.

[0018] Newly emerging AI systems may charge subscriber fees. However, consumers are likely to show substantial resistance to paid services, as they have been accustomed to free search engines for many years.

[0019] Support for newly emerging AI systems will be a social burden, meaning they may be subsidized by the government. However, in a democracy, the most important thing to avoid is an omniscient AI system closely tied to the government.

[0020] All of these approaches are considered near-ideal. The continued use of sponsored links preceding "organic" output is considered the most feasible, and some embodiments will adopt this approach. However, even sponsored links have limited potential, given their inherent inconsistencies with the organic output of newly emerging artificial intelligence models.

[0021] The need for the embodiments disclosed in this specification is evident from a recent news article: "ChatGPT, while still having significant room for improvement, has been declared a 'Code Red' by Google's management. For Google, this was tantamount to sounding a fire alarm. In some parts of the industry, there are concerns that the company may be approaching the moment that large Silicon Valley companies fear the most—the arrival of a huge technological change that could transform their business.... Google has already built chatbots that can rival ChatGPT. In fact, the technology at the core of OpenAI's chatbot was developed by researchers at Google.... However, Google may be cautious about deploying this new technology as an alternative to online search because it is not suitable for delivering digital ads, which account for over 80% of the company's revenue." (Grant and Metz, 2022)

[0022] Therefore, there is a very high need for a new approach to intervention.

[0023] Aspects of the present disclosure include a first (artificial intelligence) computer system comprising at least one computer for implementing an artificial intelligence model and a database containing intervention information stored in memory or on any storage device, and a second (intermediate) computer system comprising at least one computer for mediating user requests to the artificial intelligence computer system, the second computer system receiving requests from a user, applying the intervention information queried from the database, calculating the interventions to be performed for each request, and instructing the artificial intelligence computer system to generate a response to each request according to the associated interventions and returning the response to the user, and a network configuration enabling the artificial intelligence computer system and the intermediate computer system to communicate with each other, enabling at least one of the intermediate computer systems and the artificial intelligence computer system to send a query to the database and receive a response therefrom, and enabling the intermediate computer system to receive a request from another (user) computer system and send a response thereto, providing an improved system and method for intervention in an artificial intelligence model via a computer network.

[0024] Aspects of the present disclosure also provide an improved system and method for applying interventions to user requests to an artificial intelligence (AI) model. Requests originating from a user may be expressed as free text (and interpreted by a large language model (LLM)) or in a more structured form (as in current USPTO patent search tools). Alternatively, requests originating from a user may be expressed in any other form of data. Before proceeding to the AI model, the request is associated with one or more keywords or concepts. The database is queried to obtain intervention information corresponding to these keywords or concepts, and the intervention information is applied to calculate the intervention. The request and the calculated intervention are then sent to the AI system, which is instructed to determine a response taking into account the calculated intervention. Finally, the determined response is returned to the user.

[0025] In some embodiments, an intermediate computer system accesses a database and calculates the interventions to be applied to a request. In such embodiments, the intermediate computer system receives a request from the user, applies intervention information queried from the database, calculates the interventions to be performed for each request, asks the artificial intelligence computer system to generate a response to each request, taking into account the interventions calculated for that request, receives the response generated by the artificial intelligence computer system, and returns the generated response to the user system.

[0026] In other embodiments, the current set of intervention information (applicable to many requests) is loaded in bulk into the artificial intelligence model as part of training or fine-tuning the dataset. In many embodiments, this data will change periodically (e.g., daily or hourly), so the pre-intervention parameters of the artificial intelligence model are saved before the intervention information is loaded, and each time a new set of intervention information is scheduled to be loaded, the artificial intelligence model will first revert to the pre-intervention parameters. In such embodiments, the intermediate computer system has limited functionality for communication with the user system and queuing requests, and may even be omitted entirely.

[0027] In yet another embodiment, the intervention is applied directly to the parameters of the underlying AI model. In such an embodiment, an intermediate computer system applies intervention information queried from a database and calculates modifications to one or more parameters of the AI ​​model. In this case, the database queries are instead assigned to the AI ​​computer system itself, and the intermediate computer system may be left with only limited requirements for communication with the user system and queuing requests, and the intermediate may even be omitted altogether.

[0028] Therefore, in many embodiments, the intervention is not applied to directly modify any of the lower-level parameters of the AI ​​model. Instead, the artificial intelligence computer system is simply instructed to consider the intervention in a prescribed manner, but to apply its best available "organic" information when generating its response. In the approaches described in the three paragraphs above, the intervention is provided to the AI ​​model on a case-by-case basis, while in the approaches described in the two paragraphs above, the intervention is loaded into the AI ​​model in bulk. One important advantage of each of these two approaches is that the trained and tuned AI model does not need to be modified on a request-by-request basis, allowing requests to be processed more quickly. A second advantage of each of these two approaches is that the trained and tuned AI model may contain millions or billions of parameters, and therefore can be completely opaque regarding which parameters or to what extent they may need to be modified to achieve a given intervention. It should be noted that each of these two approaches can be implemented without a precise understanding of what the AI ​​model and each individual parameter are doing. The first approach (described in the three paragraphs above) has two additional advantages over the second approach (described in the two paragraphs above): (1) it is considered feasible to change interventions in real time only under the first approach (given that loading the entire set of intervention information may be substantial in terms of time and substantial in terms of computing resources); and (2) similarly, it is considered feasible to apply different interventions to different demands that are processed in close succession.

[0029] In some other embodiments, the intervention is implemented either by modifying records in the fine-tuning dataset or by inserting hypothetical records into the fine-tuning dataset, and then fine-tuning the AI ​​model with them. In such embodiments, an intermediate computer system applies intervention information queried from a database and calculates a modified version of the fine-tuning dataset. In this case again, the same functionality can be assigned to the artificial intelligence computer system itself, and there is no distinct need for an intermediate computer system. This approach likely has the advantage of being more precise in how the intervention should work. However, its main disadvantage is that the AI ​​model would have to undergo a reasonably expensive and time-consuming fine-tuning process each time the intervention information changes (one way to mitigate this disadvantage is probably to adopt a policy of fine-tuning the AI ​​model only on a daily or hourly basis. This effectively reduces the frequency with which the intervention information may change, but this may have both advantages and disadvantages). As described above, such an approach would likely be considered to make it impossible to apply different interventions to different requests. The intervention information can be applied via prompts or in batches.

[0030] In some exemplary, non-limiting embodiments, the intervention includes a step of fine-tuning the AI ​​model with a set of third-party ratings that may be outside the AI ​​model's pre-training dataset or may be considered more accurate than many of the AI ​​model's pre-training datasets. For example, a restaurant or travel booking service could treat Michelin star ratings or Zagat ratings as the intervention. In this case, an intermediate computer system would send a request to an artificial intelligence computer system and, in generating its response, would not only apply the AI ​​model's accumulated knowledge but also instruct it to apply predetermined weights to the Michelin or Zagat scores.

[0031] In some embodiments, the intermediate computer system does not need to communicate the intervention to the artificial intelligence computer system. Instead, the intermediate computer system sends a request to the artificial intelligence computer system, instructing it to apply the accumulated knowledge of the AI ​​model and generate a response, including an AI model rating. The intermediate computer system then combines the intervention (which can itself be interpreted as a rating) and the AI ​​model rating, returning a defined convex combination or other function of the intervention and the AI ​​model rating.

[0032] In many embodiments, intervention information includes submissions from “interested parties,” which are other system users (including, but not limited to, advertisers). In some exemplary, non-limiting embodiments, the Disclosure provides an improved system and method for intervention in an artificial intelligence model over a computer network, which also includes a third “director” computer system comprising at least one computer. This has a network configuration that enables the director computer system to receive submissions of intervention information from the interested party computer system and enables the director computer system to add or replace entries in a database upon request from the interested party computer. The intervention information submitted by the interested party computer system may be numerical, non-numerical structured text, free text such as expressed in natural language, or any other form of data. If the intervention information is numerical, it may be, but not limited to, a scalar value, a numerical vector, or a numerical array.

[0033] In many embodiments, where intervention information includes submissions from stakeholders, the submissions may be numerical, and the submitted numerical values ​​may represent payment amounts ("bids"). Such embodiments would in some respects be reminiscent of current sponsored search auctions. However, as already emphasized above, the output of such embodiments is not limited to an ordered list of internet hyperlinks; instead, the output can be arbitrary. In one exemplary embodiment, a director computer system receives bids from stakeholders on keywords, which also allows stakeholders to revise their submissions in accordance with published restrictions. When a request is received from a user, the intermediate computer system determines the keywords relevant to the request and queries the database for all intervention information (i.e., sets of bids) on these keywords that are currently in the database. The intermediate computer then calculates the intervention according to a defined function of the set of bids. Finally, as described above, the intermediate computer system sends the request to an artificial intelligence computer system and instructs it to consider the intervention in a defined manner, as well as to apply its best available information when generating its response.

[0034] With regard to embodiments in which stakeholders submit bids, the system of the present invention may include, but is not limited to, various components of auction systems described in the prior art, including, namely, Ausubel No. 5,905,975, Ausubel No. 6,026,383, Ausubel No. 7,062,461, Ausubel et al. No. 7,729,975, Ausubel et al. No. 7,899,734, and Ausubel et al. No. 8,566,211 (their disclosures are incorporated herein by reference as a whole).

[0035] It should be noted that while bids are one prominent example of numerical intervention information that may be submitted by stakeholders, they are by no means the only example. In some embodiments, intervention information includes numerical ratings submitted by experts in the field. An intermediate computer system sends a request to an artificial intelligence computer system and, in generating its response, instructs it not only to apply the accumulated knowledge of the AI ​​model but also to apply predetermined weights to the expert's numerical rating.

[0036] Intervention information submitted by stakeholders may also be non-numerical. In some embodiments, the intervention information, applied by a restaurant or travel booking service, includes free-text comments submitted by customers of the booking service. The intermediate computer system sends the request to an artificial intelligence computer system and, in generating its response, not only applies the accumulated knowledge of the AI ​​model but also instructs it to apply predetermined weights to the customer's free-text comments.

[0037] As will be seen in the detailed description below, in some exemplary, non-limiting embodiments, stakeholder bids are effectively converted into independent third-party ratings and then handled in an independent third-party rating-like manner. In some of these embodiments, the AI ​​model is explicitly instructed to generate its response to a user request by applying a convex coupling of its “organic” information and a composite third-party rating derived from the stakeholder bids.

[0038] A computer system may include, but is not limited to, a general-purpose computer, a special-purpose computer, a server, a chip, a mobile device such as a smartphone, a quantum computer, or any other device that performs functions typically described as a computer. This may be a physical computer or a virtual machine located in the cloud.

[0039] The network may be a local or wide-area network such as the Internet, an intranet, or a virtual private network, or alternatively, a public or private telephone system, fax system, email system, wired data network, wireless data network, or any other network.

[0040] An artificial intelligence computer system (or AI model) includes, but is not limited to, any computer system, network, or other computerized device that exhibits characteristics typically associated with human intelligence. An AI system may also be a computer system that implements a Large-Scale Language Model (LLM), a Generative Artificial Intelligence Model, General Artificial Intelligence, or any other form of artificial intelligence (AI). An AI system may also include, but is not limited to, generative adversarial networks, generative pre-trained transformers, and other transformer-based systems. An AI system may also be, but is not limited to, ChatGPT, Bard, OpenAI's GPT, or Google's BERT. An AI system may also be, but is not limited to, a search engine supported by an AI system, such as Bing April 2023, or a recommendation (or suggestion) system supported by an AI system.

[0041] In one aspect, a computer implementation method for interventions in an artificial intelligence (AI) model is provided. The method includes the step of obtaining a request from a user computer. The method includes the step of obtaining intervention information applicable to the request. The method includes the step of generating an augmentation request based on the obtained request and the obtained intervention information. The method includes the step of providing the augmentation request as input to the AI ​​model. The method includes the step of obtaining a response to the augmentation request from the AI ​​model. The method includes the step of sending the obtained response to the user computer.

[0042] In some embodiments, the step of obtaining intervention information further includes the steps of obtaining one or more keywords or concepts associated with the obtained request, querying a database of intervention information using one or more keywords or concepts, and obtaining intervention information in response to the query step.

[0043] In some embodiments, the method further includes the steps of obtaining intervention information from one or more stakeholder computers and updating a database using the obtained intervention information. In some embodiments, the intervention information includes ratings or comments.

[0044] In some embodiments, intervention information is based on, or includes, at least one rating or comment received from an interested party.

[0045] In some embodiments, intervention information includes bidding.

[0046] In some embodiments, intervention information is based on or includes at least one bid received from an interested party.

[0047] In some embodiments, a first weight is associated with intervention information, and the first weight indicates the amount by which the intervention information should be weighted by the AI ​​model. In some embodiments, the method further includes the step of retrieving the first weight from a database. In some embodiments, the method further includes the step of incorporating the first weight into an augmentation request. In some embodiments, the method further includes the step of incorporating a second weight into the augmentation request relating to organic information contained within the AI ​​model, where organic information includes information available to the AI ​​model in response to the retrieved request, without the intervention information. In some embodiments, the first and second weights are incorporated into the augmentation request as a convex coupling of the first and second weights.

[0048] In some embodiments, the method further includes the steps of providing a request to a second AI model, the second AI model being fine-tuned with respect to intervention information, and obtaining from the second AI model intervention information applicable to the acquired request.

[0049] In some embodiments, the method further includes the steps of identifying a first part of the response, which includes options associated with intervention information, and a second part of the response, which includes options not associated with intervention information, and applying a first marker to the first part of the response and a second marker to the second part of the response before transmitting the response to a user computer. In some embodiments, the first marker includes at least one of the following: a first color different from the second color used in the second marker; a first typeface different from the second typeface used in the second marker; a first symbol different from the second symbol used in the second marker; or a first text character different from the second text character used in the second marker.

[0050] In some embodiments, the AI ​​model is a large-scale language model.

[0051] In some embodiments, intervention information corresponds to optional independent third-party assessment.

[0052] In some embodiments, the method further includes the step of masking the extension request from the user's computer.

[0053] In some embodiments, intervention information is associated with stakeholders and options.

[0054] From another perspective, a computer implementation method for interventions in an artificial intelligence (AI) model is provided. The method includes the step of obtaining intervention information from one or more stakeholder computers. The method includes the step of creating a training set based on the obtained intervention information. The method includes the step of training the AI ​​model on the created training set. The method includes the step of obtaining a request from a user computer. The method includes the step of obtaining a response to the request from the trained AI model. The method includes the step of sending the obtained response to the user computer.

[0055] In some embodiments, the method further includes the step of updating a database using the acquired intervention information.

[0056] In some embodiments, the acquired intervention information includes ratings or comments.

[0057] In some embodiments, the acquired intervention information is based on or includes at least one rating or comment.

[0058] In some embodiments, the acquired intervention information includes bids.

[0059] In some embodiments, the acquired intervention information is based on or includes at least one bid received from an interested party computer.

[0060] In some embodiments, the AI ​​model is a large-scale language model.

[0061] In some embodiments, intervention information corresponds to optional independent third-party assessment.

[0062] In some embodiments, intervention information is associated with stakeholder computers and options.

[0063] From another perspective, a computer implementation method is provided for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model. The method includes the step of obtaining submissions from a first participant among multiple participants. The method includes the step of transforming the submissions from the first participant into a first set of one or more selections for the selection mechanism using an AI model.

[0064] In some embodiments, the method further includes the steps of obtaining submissions or selections from other participants different from a first participant, and determining the outcome of the selection mechanism based on the first set of one or more selections and the submissions or selections obtained from the other participants.

[0065] In some embodiments, the method further includes the step of training an AI model and generating selections. In some embodiments, the training step includes at least one of the steps of pre-training or fine-tuning the AI ​​model on one or more exemplary or actual submissions.

[0066] In some embodiments, the method further includes the steps of obtaining a search request from a user computer, selecting one or more selections from a first set of one or more selections, and expanding the search request with intervention information based on the selected one or more selections. In some embodiments, the method further includes the step of converting each of the selected one or more selections into an independent third-party rating, where the intervention information is based on or includes an independent third-party rating.

[0067] In some embodiments, the independent third-party assessment corresponds to at least one of the following: a hotel, restaurant, venue, shop, or commercial establishment.

[0068] In some embodiments, the method further includes the step of charging the first participant the associated selection amount for each of the one or more selections that have been selected.

[0069] In some embodiments, the method further includes the steps of transmitting a first set of one or more selections to a first participant, and obtaining a response from the first participant indicating approval of the first set of one or more selections or a modification to the first set of one or more selections. In some embodiments, the method further includes the step of training an AI model using the modification in response to a decision that the response from the first stakeholder indicates a modification to the set of selections.

[0070] In some embodiments, the method further includes the steps of obtaining submissions or selections from other participants different from a first participant; implementing an auction on a selected set of one or more selections and the submissions or selections obtained from other participants; and determining an allocation based on the outcome of the auction. In some embodiments, the auction is a generalized second-price auction.

[0071] In another aspect, a computer implementation method for intervention in an artificial intelligence model is provided. The method includes the step of transmitting a request to an artificial intelligence (AI) search system comprising an AI model. The method includes the step of receiving a response from the AI ​​search system, wherein the response includes a first part that receives at least one intervention and a second part that does not receive an intervention, and the step of applying an indicator to the first part.

[0072] In some embodiments, the sign includes at least one of the following: a first color different from a second color used in the second part of the response; a first typeface different from a second typeface used in the second part of the response; a first symbol different from a second symbol used in the second part of the response; or a first text character different from a second text character used in the second part of the response.

[0073] In another aspect, a computer implementation method is provided for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model. The method includes the step of obtaining a submission from a user. The method includes the step of transmitting the submission to an AI model. The method includes the step (s3030) of obtaining a response from the AI ​​model, which includes a set of one or more selections for the selection mechanism, and the AI ​​model converts the submission into a set of one or more selections.

[0074] In some embodiments, the method further includes the steps of obtaining feedback from a user indicating approval of one or more sets of selections or modifications to one or more sets of selections, and transmitting the feedback to an AI model.

[0075] In some embodiments, the method further includes the step of receiving an allocation based on the outcome of an auction based on one or more selections. In some embodiments, the auction is a generalized second-price auction.

[0076] From another perspective, a computer implementation method is provided for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model. The method includes the step of obtaining a first set of selections from a first participant among multiple participants. The method includes the step of obtaining a search request from a user computer. The method includes the step of selecting one or more selections from the first set of one or more selections. The method includes the step of augmenting the search request with intervention information based on the selected one or more selections. The method includes the step of providing the augmented search request to the AI ​​model.

[0077] From yet another perspective, a device is provided comprising a processing network and a memory coupled to the processing network. The device is configured to carry out one of the methods described above.

[0078] From yet another perspective, a computer program containing instructions is provided, which, when executed by the device's processing circuit, causes the device to perform one of the aforementioned methods. [Brief explanation of the drawing]

[0079] The accompanying drawings, included and incorporated herein and constituting part thereof, to provide a further understanding of the present disclosure, illustrate embodiments of the present disclosure and, together with detailed descriptions, serve to illustrate the principles of the present disclosure. No attempt is made to show structural details of the present disclosure in more detail than may be necessary for a basic understanding of the present disclosure and the various ways in which it may be practiced. The drawings are as follows:

[0080] [Figure 1A] Figure 1A is a schematic diagram illustrating the overall architecture according to an exemplary, non-limiting embodiment.

[0081] [Figure 1B] Figure 1B is a schematic diagram illustrating the overall architecture according to an exemplary, non-limiting embodiment.

[0082] [Figure 2] Figure 2 is a schematic diagram illustrating the architecture of a transducer model, which is an exemplary, non-restrictive, fundamental element of an artificial intelligence computer system.

[0083] [Figure 3A] Figures 3A and 3B are schematic diagrams that illustrate the elements of Figure 2 in more detail. [Figure 3B] Figures 3A and 3B are schematic diagrams that illustrate the elements of Figure 2 in more detail.

[0084] [Figure 4] Figure 4 is a schematic diagram illustrating exemplary non-restrictive training of an artificial intelligence computer system.

[0085] [Figure 5-1]Figure 5 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment. [Figure 5-2] Figure 5 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment.

[0086] [Figure 6-1] Figure 6 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment. [Figure 6-2] Figure 6 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment.

[0087] [Figure 7] Figure 7 is a flowchart that illustrates the elements of the flowchart in Figure 5 in more detail.

[0088] [Figure 8A] Figures 8A and 8B are flowcharts that illustrate the elements of the flowchart in Figure 5 in more detail. [Figure 8B] Figures 8A and 8B are flowcharts that illustrate the elements of the flowchart in Figure 5 in more detail.

[0089] [Figure 9A] Figures 9A, 9B, and 9C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 9B-1] Figures 9A, 9B, and 9C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 9B-2] Figures 9A, 9B, and 9C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 9C-1] Figures 9A, 9B, and 9C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 9C-2]Figures 9A, 9B, and 9C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments.

[0090] [Figure 10A-1] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments. [Figure 10A-2] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments. [Figure 10B-1] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments. [Figure 10B-2] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments. [Figure 10C-1] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments. [Figure 10C-2] Figures 10A, 10B, and 10C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary non-limiting embodiments.

[0091] [Figure 11A] Figures 11A, 11B, and 11C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 11B-1] Figures 11A, 11B, and 11C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 11B-2] Figures 11A, 11B, and 11C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 11C-1]Figures 11A, 11B, and 11C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments. [Figure 11C-2] Figures 11A, 11B, and 11C provide associated responses, both with and without exemplary requests and interventions, relating to exemplary, non-limiting embodiments.

[0092] [Figure 12A] Figures 12A and 12B are flowcharts that illustrate the elements of the flowcharts in Figures 5 and 6 in more detail, respectively. [Figure 12B] Figures 12A and 12B are flowcharts that illustrate the elements of the flowcharts in Figures 5 and 6 in more detail, respectively.

[0093] [Figure 13A-1] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B. [Figure 13A-2] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B. [Figure 13B-1] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B. [Figure 13B-2] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B. [Figure 13C-1] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B. [Figure 13C-2] Figures 13A, 13B, and 13C provide exemplary outputs of Figures 12A and 12B.

[0094] [Figure 14-1] Figure 14 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment. [Figure 14-2] Figure 14 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment.

[0095] [Figure 15-1] Figure 15 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment. [Figure 15-2] Figure 15 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment.

[0096] [Figure 16A] Figures 16A and 16B are flowcharts that illustrate the elements of the flowcharts in Figures 14 and 15 in more detail, respectively. [Figure 16B] Figures 16A and 16B are flowcharts that illustrate the elements of the flowcharts in Figures 14 and 15 in more detail, respectively.

[0097] [Figure 17] Figure 17 is a schematic diagram illustrating the architecture of an exemplary artificial intelligence selection mechanism system according to one embodiment.

[0098] [Figure 18A] Figures 18A and 18B are schematic diagrams that illustrate the elements of Figure 17 in more detail. [Figure 18B] Figures 18A and 18B are schematic diagrams that illustrate the elements of Figure 17 in more detail.

[0099] [Figure 19] Figure 19 is a flowchart illustrating the overall process of an exemplary artificial intelligence selection mechanism system and method according to one embodiment.

[0100] [Figure 20] Figure 20 is a flowchart illustrating the overall process of an exemplary artificial intelligence selection mechanism system and method according to one embodiment.

[0101] [Figure 21A-1] Figures 21A and 21B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 21A-2]Figures 21A and 21B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 21B-1] Figures 21A and 21B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 21B-2] Figures 21A and 21B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments.

[0102] [Figure 22A-1] Figures 22A and 22B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 22A-2] Figures 22A and 22B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 22B-1] Figures 22A and 22B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments. [Figure 22B-2] Figures 22A and 22B provide their conversion to a set of exemplary submissions and provisional selections relating to exemplary, non-limiting embodiments.

[0103] [Figure 23A] Figures 23A and 23B are flowcharts that illustrate the elements of the flowchart in Figure 19 in more detail. [Figure 23B] Figures 23A and 23B are flowcharts that illustrate the elements of the flowchart in Figure 19 in more detail.

[0104] [Figure 24A] Figures 24A and 24B are flowcharts that illustrate the elements of the flowchart in Figure 20 in more detail. [Figure 24B] Figures 24A and 24B are flowcharts that illustrate the elements of the flowchart in Figure 20 in more detail.

[0105] [Figure 25]Figure 25 is a block diagram illustrating a device according to several embodiments.

[0106] [Figure 26] Figure 26 illustrates the method according to several embodiments.

[0107] [Figure 27] Figure 27 illustrates the method according to several embodiments.

[0108] [Figure 28] Figure 28 illustrates the method according to several embodiments.

[0109] [Figure 29] Figure 29 illustrates the method according to several embodiments.

[0110] [Figure 30] Figure 30 illustrates the method according to several embodiments. [Modes for carrying out the invention]

[0111] Detailed explanation This detailed description is merely illustrative in nature and is not intended to limit the subject matter or the embodiments of the present application or the use of such embodiments. As used herein, the word “exemplary” means “serving as an example, case, or illustration.” Any implementation described herein as exemplary is not necessarily construed as preferable or advantageous to other implementations. Furthermore, it is not intended to be limited by any express or implied theory presented herein, including but not limited to the field of the invention, the summary of the invention, or the detailed description.

[0112] Techniques and technologies can be described herein with reference to symbolic representations of operations, processing tasks, and functions that can be performed by various computing components or devices in terms of functional and / or logical block components. Such operations, tasks, and functions are sometimes also referred to as computer execution, computerization, software implementation, or computer implementation. In practice, one or more processor devices can perform the operations, tasks, and functions described by manipulating electrical signals that represent data bits in memory locations within system memory, or by other processing of signals. The memory locations where data bits are maintained are physical locations having specific electrical, magnetic, optical, or organic properties corresponding to the data bits.

[0113] It should be understood that the various block components shown in the figure may be realized by any number of hardware, software, and / or firmware components configured to perform a defined function. For example, an embodiment of the system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, or equivalents, which may perform various functions under the control of one or more processors or other control devices. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), or other processing network capable of executing software. The processor may be implemented using one or more general-purpose and / or special-purpose processors. Alternatively, or in addition, an embodiment of the system or component may be based on a quantum computer architecture or employ various quantum computing components.

[0114] Therefore, while the drawings may depict one exemplary array of elements, additional intervention elements, devices, features, or components may be present within certain embodiments of the subject being depicted. In addition, certain technical terms may also be used in the following description solely for reference purposes and are therefore not intended to be limiting.

[0115] When implemented in software or firmware, the various elements of the systems described herein are essentially code segments or instructions that perform various tasks. In some embodiments, the programs or code segments may be stored in a tangible, non-transient processor-readable medium. “Processor-readable medium” or “machine-readable medium” may include any medium capable of storing or transferring information. Examples of processor-readable mediums include electronic circuits, semiconductor memory devices, ROMs, flash memory, USB sticks, erasable ROMs (EROMs), floppy diskettes, CD-ROMs, optical discs, hard disks, or equivalents. The embodiments described herein are intended merely as examples and serve as guidelines for implementing the novel systems and methods described herein in any application. Therefore, the embodiments presented herein are intended to be non-limiting.

[0116] Prerequisites

[0117] Figure 1A illustrates an architecture according to a non-exclusive, exemplary embodiment. The artificial intelligence computer system 10 (also known as the "AI system") is a critical computer system that implements an artificial intelligence (AI) model. The AI ​​system may be any computer system, network, or other computerized device exhibiting characteristics typically associated with human intelligence. The AI ​​system may be a computer system that implements, but is not limited to, a large-scale language model (LLM), a deep learning model, a neural network, generative artificial intelligence, general artificial intelligence, or any other form of artificial intelligence (AI). The AI ​​system may include, but is not limited to, generative adversarial networks, generative pre-trained transformers, and other transformer-based systems. The AI ​​system may also include, but is not limited to, ChatGPT, Bard, OpenAI's GPT, or Google's BERT. The AI ​​system may also be, but is not limited to, a search engine supported by an AI system, such as Bing April 2023, or a recommendation (or suggestion) system supported by an AI system. The AI ​​system may operate on any other device or set of devices, including, but not limited to, servers, arrays of servers, desktop computers, CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or other processors or processing networks, or on any other computer system that may be deployed in an office, on the cloud, in any form of data center, or anywhere else. To avoid misunderstanding, the AI ​​system (or any of the other computer systems used in this disclosure) may include a quantum computer or employ quantum computing elements.

[0118] The intermediate computer system 20 (also known as the “intermediate”) is a computer system that interacts with both the AI ​​system 10 and the user computer. Embodiments of the intermediate 20 include, but are not limited to, any other device or set of devices with servers, arrays of servers, desktop computers, CPUs, GPUs, TPUs, or other processors or processing networks, or any other computer system that may be deployed in an office, on the cloud, in any form of data center, or in any other location. User computers 30a-m (each a computer system) are used by users to submit requests and receive responses. Embodiments of user computers 30a-m include, but are not limited to, desktop computers, laptop computers, smartphones, tablets, any other device with CPUs, GPUs, TPUs, or other processors or processing networks, or any other computer system. The network 40 represents a computer network through which multiple unlocalized user computer systems may connect. In many exemplary embodiments, the network 40 is the internet. In some exemplary embodiments, user requests are communicated from user computers 30a-m to the intermediate 20 via the network 40 (responses are communicated from the intermediate 20 to the user computers 30a-m), and the AI ​​system 10 itself is not directly connected to the network 40, but the AI ​​system 10 is directly connected to the intermediate 20.

[0119] Connected to the intermediate 20 is a database 50, which may be stored in memory or on any storage device, including, but not limited to, RAM, ROM, hard disk drives, solid-state drives, or any other medium capable of storing data. The intermediate 20 queries the database 50 for intervention information associated with the user's request, the database 50 returns the intervention information to the intermediate 20, and the intermediate 20 applies the intervention information and decides on an intervention. The intermediate 20 then sends the request and the decided intervention to the AI ​​system 10, which is instructed to generate a response considering the decided intervention. The AI ​​system 10 returns the response to the intermediate 20, which in turn returns the response to the user computer 30a-m that submitted the request, via the network 40.

[0120] In some embodiments, there is also a Director Computer System 60 (also known as the “Director”), which is an additional computer system involved in establishing and updating intervention information provided by stakeholders. Embodiments of the Director 60 include, but are not limited to, servers, arrays of servers, desktop computers, any other devices with CPUs, GPUs, TPUs, or other processors or processing networks, or any other computer systems that may be deployed in an office, on the cloud, in any form of data center, or in any other location. Stakeholders submit intervention information using stakeholder computers 70a-n (each a computer system), which is communicated to the Director 60 via the network 40. Embodiments of the stakeholder computers 70a-n include, but are not limited to, desktop computers, laptop computers, smartphones, tablets, any other devices with CPUs, GPUs, TPUs, or other processors or processing networks, or any other computer systems. The director 60 is also connected to the database 50 and updates the database 50 based on intervention information submitted by the stakeholder computers 70a-n. In some embodiments, the intermediate 20, the director 60, or another computer system provides feedback to the database 50 after observing how the response affects the user's behavior.

[0121] Various aspects of the architecture depicted in Figure 1A are not essential and can be easily changed. For example, Figure 1A depicts the AI ​​system 10 as being directly connected to the intermediate 20 but not directly connected to the network 40. However, the architecture could instead allow the AI ​​system 10 to communicate with the intermediate 20 via the network 40 or via another network (e.g., a private network). Similarly, Figure 1A depicts the AI ​​system 10, the intermediate 20, and the director 60 as separate computer systems, but the functionality of any two or all three of these systems could be combined into a single computer system or distributed among additional computers, and so on.

[0122] For example, Figure 1B illustrates an architecture according to a non-limiting exemplary embodiment, in which the functionality of the AI ​​system 10 and the intermediate 20 are merged. A separate intermediate 20 does not exist in Figure 1B. Instead, the AI ​​system 10 interacts directly with user computers 30a-m via the network 40 and connects directly to the database 50. As described above, in some embodiments, there is also a director 60, which is an additional computer system involved in establishing and updating intervention information provided by stakeholders. Stakeholders submit intervention information using stakeholder computers 70a-n, which is communicated to the director 60 via the network 40. The director 60 is also connected to the database 50 and updates the database 50 based on the intervention information submitted by the stakeholder computers 70a-n. In some embodiments, the AI ​​system 10, the director 60, or another computer system provides feedback to the database 50 after observing how the response affects the user's behavior.

[0123] Figure 2 illustrates an artificial intelligence computer system according to a non-limiting, exemplary embodiment. This figure depicts the architecture of a transducer neural network model (hereinafter referred to as the "transducer model" or simply "transducer"), which is a key fundamental element of many artificial intelligence computer systems used as of 2023. The transducer model utilizes an attention mechanism known in the art. The transducer model consists of a plurality of encoder blocks 100a-m and a plurality of decoder blocks 110a-n, each containing a plurality of sublayers. Each encoder block 100i has two sublayers, namely a multi-head self-attention mechanism 103 and a position-by-position feedforward network 105. First, an input embedding 101 converts the input into a representation. However, this embedding does not have an embedded concept of order. Therefore, a position encoding 102 provides an additional positional representation of the ordered input. The resulting representation from the input is then passed through a plurality of encoder blocks 100a-m, each encoder block 100i comprising a multi-head self-aware mechanism 103 and a position-by-position feedforward network 105. Residual connection and layer normalization ("Add & Norm") are applied after each sublayer, i.e., Add & Norm 104 follows the multi-head self-awareness 103, and Add & Norm 106 follows the position-by-position feedforward 105. That is, the output of each sublayer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is a function implemented by the sublayer itself. The result of the last encoder block 100m is fed into the second layer of the decoder block, the multi-head self-aware mechanism 115, which is described in the next paragraph.

[0124] Each decoder block 110j has three sublayers: a masked multi-head self-aware mechanism 113, a multi-head self-aware mechanism 115, and a position-based feedforward network 117. First, an output embedding 111 converts the output into a representation. However, this embedding does not have an internal concept of order. Therefore, a position encoding 112 provides an additional positional representation of the ordered output. The resulting representation of the output is then passed through a plurality of decoder blocks 110a-n, each decoder block 110j comprising a masked multi-head self-aware mechanism 113, a multi-head self-aware mechanism 115, and a position-based feedforward network 117. Residual connection and layer normalization are applied after each sublayer, i.e., Add & Norm 114 follows the masked multi-head self-attention mechanism 113, Add & Norm 116 follows the multi-head self-attention 115, and Add & Norm 118 follows the position-by-position feedforward 117. Finally, linear projection (Linear 119) is applied to the result of the last decoder block 110n, and the SoftMax function (SoftMax 120) is applied to make the output interpretable as a probability vector. This gives rise to the output probability. Additional details can be found in Vaswani et al. (2017).

[0125] Figures 3A and 3B provide further details of the multi-head self-attention mechanisms 103, 113, and 115 employed in both the encoder and decoder blocks of Figure 2. The mechanism comprises several parallel attention heads, each independently calculating a set of attention weights for an input sequence. For each attention head, the input sequence is dimensional d k The query and key are vectors, and each has dimension d. v The vector has values ​​and a scaled dot product, note that the scaled dot product is defined by equation (1). [ka]

[0126] Figure 3A illustrates the application of equation (1). The queries are grouped together in matrix Q, the keys in matrix K, and the values ​​in matrix V. In step 131, the product of the transposes of Q and K is calculated by matrix multiplication (MatMul). In step 132, the result QK from step 131 is obtained. T However, d k It is scaled by dividing by the square root of . In an optional step 133, a portion of the output to be predicted (e.g., future states) is masked, i.e., this occurs in the masked multi-head self-attention mechanism 113 but not in the multi-head self-attention mechanisms 103 and 115. In step 134, the SoftMax function is applied to each row of the scaled and optionally masked matrix product. The SoftMax function applied to an N-dimensional vector z is: [ka] It is defined by and has the effect of converting arbitrary weights into non-negative numbers that are normalized so that their sum is 1. Finally, in step 135, the result of the SoftMax function is multiplied by V using matrix multiplication (MatMul).

[0127] Figure 3B depicts a multi-head self-attention mechanism (or a masked multi-head self-attention mechanism). Instead of applying a single attention function to the query, key, and value, the attention function is applied h times in parallel to different linear projections of the query, key, and value. In steps 141a-c, d k d k and d v A linear projection of queries, keys, and values ​​onto the dimension is performed h times in parallel. In step 142, the scaled dot product attention shown in Figure 3A is applied h times in parallel. In step 143, the results from step 142 are concatenated. In step 144, they are projected again to produce the final output of the multi-head self-attention mechanism.

[0128] Figure 4 illustrates the training process for an exemplary artificial intelligence computer system. Elements labeled 150-154 are frequently referred to as “pre-training,” while elements labeled 155-159 are frequently referred to as “fine-tuning.” The fine-tuning elements (155-159) are shown as dashed lines in this figure because, in some embodiments, the fine-tuning portion of the process may be omitted, and the pre-trained model may be directly deployed and prepared to prompt. According to some embodiments, training an AI model can refer to either pre-training or fine-tuning an AI model, but is not limited to this. Furthermore, some of the differences between pre-training and fine-tuning may be terminological. Pre-training is often performed once with respect to the model (the same pre-trained model is then used for various different tasks or domains). Fine-tuning is often performed separately for each task or domain, using the pre-trained model as a starting point.

[0129] The pre-training dataset 150 is a massive corpus of unlabeled text data, often containing a wide range of internet text. Typical subsets of the pre-training dataset 150 include, but are not limited to, Wikipedia, a collection of millions of encyclopedia articles on different topics, Common Crawl, a collection of billions of web pages, and BooksCorpus, which contains the full text of thousands of books on various genres and topics. Pre-training is the process of training an artificial intelligence model on the pre-training dataset, which includes, but is not limited to, learning common features and patterns of natural language. In step 151, the pre-training dataset 150 is pre-processed. Pre-processing may include tokenization, normalization, filtering, and shuffling. In this step, the dataset may also be split so that parts of the dataset are retained for evaluation and testing instead of being used for training. In step 152, the parameters of the artificial intelligence model are initialized, typically randomly and / or using parameters from previous versions of the model. In step 153, a Masked Language Modeling (MLM) task is used to train the artificial intelligence model. In the MLM task, some tokens in a sequence are masked, and the model is trained to predict the masked tokens. The masked tokens are typically selected randomly, but they can also be selected based on their importance in the sequence. Training on the MLM task helps the model learn the meanings of individual words and phrases, as well as the relationships between words and phrases. In step 154, a Next Sentence Prediction (NSP) task is used to train the artificial intelligence model. In the NSP task, the model is given two sequences of tokens and is trained to predict whether the second sequence follows the first sequence. The two sequences are typically selected randomly, but they can also be selected based on their relationships to each other. Training on the NSP task helps the model learn long-range dependencies between words and phrases.

[0130] When the artificial intelligence model is transducer-based, steps 153 and 154 both involve forward and backward paths through a transducer model as depicted in Figure 2. In the forward path, pre-processed data resulting from step 151 is fed into the transducer model. At each layer of the transducer model, the input data undergoes a series of operations related to the layer's parameters, including a self-attention mechanism and a feedforward network. In the backward path, the transducer model calculates a loss, which measures the accuracy of its predictions. This loss is then used to update the transducer model's parameters using a process called backpropagation. At high frequency, the process of feeding pre-processed data into the transducer model, generating predictions, calculating the loss, and updating parameters is repeated multiple times for both MLM and NSP tasks until the model converges. In other words, steps 153 and 154 may be repeated multiple times.

[0131] The tuning dataset 155 is typically a much smaller pre-training dataset 150, closely related to a specific task to which an artificial intelligence model is usually tuned. For example, if the model is tuned to web searches, the tuning dataset 155 may include, but is not limited to, samples of web search queries and results from different languages ​​and various fields used to demonstrate correct behavior and to demonstrate ranking of different responses. If the model is tuned to receive queries and return results as normal conversations within a chat session, the tuning dataset 155 may include, but is not limited to, samples of user messages and responses from different languages ​​and various contexts. Tuning is the process of training an artificial intelligence model on a tuning dataset, which includes, but is not limited to, learning to improve its performance on a specific task and to improve its conversational interactions.

[0132] In step 156, the fine-tuning dataset 155 is preprocessed. Preprocessing may include tokenization, normalization, filtering, and shuffling. Marking may also need to be added to the data, as fine-tuning is typically performed using marked data. In this step, the dataset may also be split so that a portion of the dataset is retained for evaluation and testing instead of being used for training. In step 157, the fine-tuning parameters of the artificial intelligence model are initialized, typically using parameters from a pre-trained model, small random values, or some combination thereof. In step 158, the artificial intelligence model is trained using the preprocessed data generated in step 156. If the artificial intelligence model is transformer-based, step 158 involves forward and backward passes through the transformer model, as depicted in Figure 2. The process of feeding the preprocessed data into the transformer model, generating predictions, calculating losses, and updating parameters is iterated multiple times until the model converges. In step 159, the artificial intelligence model is evaluated on marked data with respect to the specific task to which the model has been fine-tuned. In high frequency, the evaluation is performed on a portion of the data retained for evaluation in step 157. The evaluation results are used to determine whether the artificial intelligence model is satisfactory or whether it needs to be trained for further iterations. Steps 158 and 159 may be repeated multiple times until a satisfactory model is obtained. Finally, in step 160, the artificial intelligence model is deployed in the production environment, which is now ready for the prompt (step 161).

[0133] Broadly speaking, there are two main approaches to teaching AI models new information, such as intervention information: fine-tuning and the use of contextual prompts. Fine-tuning has already been explained in Figure 4 and the associated text. The use of contextual prompts means providing additional information to the AI ​​model within the prompt itself. Each approach has its advantages and disadvantages. Fine-tuning leads to a more precise and reliable integration of new information, but it can require substantial computational resources. Providing contextual information within prompts relies on the AI ​​model's ability to correctly interpret and utilize the provided context (which can sometimes lead to less predictive results), but it requires fewer resources, offers greater flexibility, and allows for more frequent updates of new information.

[0134] Any of these approaches can be applied to the current context of intervention in an AI model. We will first describe an embodiment that utilizes contextual prompts, and then an embodiment that utilizes fine-tuning. However, these are merely illustrative and non-limiting embodiments. For those skilled in the art, methods for constructing similar embodiments using other approaches to teaching new information to the AI ​​model will also be obvious. Furthermore, it should be understood that similar results can be obtained by using embedding, as with using contextual prompts, and therefore, methods for modifying the process in Figure 5 to utilize embedding will also be obvious to those skilled in the art.

[0135] Within that, intervention information is provided through contextual prompts,

[0136] Figure 5 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment, in which intervention information is provided to the AI ​​model through contextual prompts. In other words, the system utilizes intervention information to prompt the AI ​​model (i.e., narrowly targeted intervention information is provided to the AI ​​system each time it is instructed to generate a response to an individual request). The process begins at step 200, where the database is initialized. In many embodiments, an intermediate initializes the database with the most recent intervention information being collected. In some embodiments, the intermediate initializes the database with intervention information corresponding to a variety of "keywords" or "concepts." The initialization may further include other information about the keywords or concepts. Optionally, the database may also be initialized with intervention weights, indicating the amounts that a given intervention should be weighted in determining a response to a user request involving these keywords or concepts. The process continues to step 202, where the AI ​​model is pre-trained and fine-tuned with "organic" information (i.e., using actual, available data without interventions). The process of pre-training and fine-tuning the AI ​​model has already been described in Figure 4 and the associated text. The flow then proceeds to step 204, where the parameters of the pre-trained / fine-tuned AI model are saved for later use. The process continues to step 206, which is a command flow statement that branches based on whether new intervention information or a new user request should be processed. If new intervention information should be processed, the flow proceeds to step 208, where the director computer system receives the new intervention information from the stakeholder computer. The intervention information may be associated with a given stakeholder. The intervention information may be numerical, non-numerical structured text, free text, or any other form of data. If the intervention information is numerical, it may be a scalar value, a numerical vector, or a numerical array, but is not limited to these forms.In some embodiments, intervention information is associated with a “keyword” or “concept.” Then, in step 210, the director computer system updates the database with the new intervention information, and the process returns to step 206. Alternatively, if it is determined in step 206 that a new user request should be processed, the flow proceeds to step 212.

[0137] In step 212, the intermediate receives a new request from the user computer. The request may be expressed as free text submitted by the user, or it may be in a more structured form. The request may also take the form of a short combination of search terms, as conventionally used in Google searches. The request may also consist of a complete or partial submission of voice or other audio queries. The request may also consist of a complete or partial submission of graphic images, pictures, drawings, photographs, video images, or any other form of data. Three exemplary requests will be illustrated in the first boxes of Figures 9A, 10A, and 11A, respectively. In step 214, the intermediate asks the AI ​​system to determine one or more keywords or concepts associated with the user request (unless the request is already expressed in the form of one or more keywords or concepts, and if applicable, this and the next step may be unnecessary). Then, in step 216, the AI ​​system determines one or more keywords or concepts associated with the user request and returns them to the intermediate. Chan et al. (U.S. Patent No. 11,409,812) teach a method for associating images with concepts. In some embodiments, the AI ​​system also returns relative importance weights associated with each keyword or concept, which measure the degree to which each keyword or concept is relevant to a user request, i.e., if keyword 1 has twice the relative importance of keyword 2, then the user request is twice as relevant to keyword 2 as keyword 1. The flow proceeds to step 218, where the intermediate queries the database using one or more keywords or concepts returned in step 216. Then, in step 220, the database returns intervention information associated with these keywords or concepts. In many embodiments, if the database contains multiple records with intervention information for a k-th keyword and a j-th option, the database returns the record with the most recent intervention information.In some embodiments, step 218 involves the intermediate querying the database using a single keyword or concept, and step 220 involves the database returning intervention information for all options associated with that keyword or concept, or an aggregate of intervention information for all options associated with that keyword or concept. In other embodiments, step 218 involves the intermediate querying the database using multiple keywords or concepts, and step 220 involves the database returning a combination of intervention information associated with these multiple keywords or concepts. Optionally, the database may also return intervention weights (compared to "organic information") indicating the amount by which a given intervention should be weighted when determining a response to a user request involving these keywords or concepts.

[0138] In step 222, the intermediate applies intervention information received from the database and calculates an intervention. In many embodiments, the calculated intervention is a defined function of the intervention information. One exemplary embodiment of step 222 is illustrated in detail in Figure 7. In step 224, the intermediate instructs the AI ​​system to generate a response to a request, taking into account the intervention calculated in step 222. Three exemplary requests (including interventions) will be illustrated in the second boxes of Figures 9A, 10A, and 11A, respectively. In many embodiments, the AI ​​system may be instructed to apply defined weights to the intervention. Optionally, the database may also be via intervention weights in step 220, in which case the intervention weights may be incorporated into the instruction in step 224. Two exemplary embodiments of step 224 are illustrated in detail in Figures 8A and 8B. In step 226, the AI ​​system generates a response as instructed, the AI ​​system sends the generated response to the intermediate, and the intermediate receives the generated response. Exemplary responses, associated with the exemplary requests in the second boxes of Figures 9A, 10A, and 11A, respectively, will be illustrated in Figures 9C, 10C, and 11C. In some embodiments, the "response" generated by the AI ​​system includes not only the response intended to be automatically forwarded to the requesting user computer, but also additional information that may be utilized by the intermediate without being automatically forwarded to the requesting user computer. Next, in step 228, the intermediate uses the response (and potentially additional data from a database or other source) received from the AI ​​system to generate an output that will be returned to the requesting user computer. An exemplary embodiment of step 228 is illustrated in detail in Figure 12A. Note that step 228 is optional, and in some embodiments, the intermediate uses the response received in step 226 as an output that will be returned to the requesting user computer without modification. Then, in step 230, the intermediate returns the output to the requesting user computer.

[0139] In step 232, feedback may be provided to the database. Several exemplary embodiments of the feedback occur when the intervention information from the stakeholder computer takes the form of a bid. In that case, step 232 reports the payment to be borne by one or more stakeholders. As with current practices in sponsored search, the payment may be assessed on an impression-per-input (PPI) basis, a pay-per-click (PPC) basis, a pay-per-purchase (PPP) basis, or some future basis which is more appropriate for an artificial intelligence system. Two exemplary, non-limiting embodiments of providing feedback to the database are illustrated in detail in Figures 16A and 16B. Finally, the flow proceeds to step 234, which is a command flow statement that branches based on whether the process should continue. If the process should continue, the flow returns to step 206 (or, in embodiments, to step 212, where there is no processing of new intervention information from the stakeholder computer). Otherwise, the process ends here.

[0140] The flow shown in Figure 5 and the networks shown in Figures 1A and 1B are illustrative only. This and the following paragraphs describe some possible variations (but not limited to). In some embodiments, multiple new intervention information and multiple new requests are processed simultaneously. In some other embodiments, there is no processing of new intervention information from the interested computer, in which case steps 206, 208, and 210 are removed from the flow diagram, and the process proceeds directly from step 204 to step 212. In some other embodiments, the functionality of the intermediate computer system and the director computer system is combined into a single computer system. In some other embodiments, there may be several instances of various components such as the intermediate computer system, the director computer system, the AI ​​system, or the database. In some other embodiments, different AI systems are used in steps 214-216 (to determine keywords or concepts associated with the user request) and steps 224-226 (to obtain a response to the request, taking into account the calculated intervention). Furthermore, in some other embodiments, steps 214-216 may use a conventional computer system without artificial intelligence (for speed and cost-effectiveness), while steps 224-226 may use the best available artificial intelligence system.

[0141] Alternatively, the process may operate in a manner similar to Search-Augmented Generation (RAG). A description of RAG can be found in Lewis et al. (2020). The system may use a vector search function to retrieve the most relevant information from the intervention information database. The system may then include this information in prompts that are sent directly to the AI ​​model. For those skilled in the art, methods for implementing the modifications described in the previous and current paragraphs, and for implementing other modifications relating to the process in Figure 5, will also be obvious.

[0142] Many aspects of this disclosure highlight embodiments relating to interventions in AI models that perform “search engine” functions. However, it should be emphasized that almost the same considerations also apply to AI models that perform “recommender system” (or “recommendation system”) functions. Therefore, for those skilled in the art, methods for modifying the process in Figure 5 to illustrate interventions in AI models that perform recommender system (or recommendation system) functions will also be obvious. For example, step 212 for some recommender systems may involve a user request for recommendations, but on shopping or e-commerce websites, it is more likely that the request for recommendations for the user will actually be initiated by the shopping or e-commerce website, regardless of whether the consumer (user) actually desires recommendations. Therefore, step 212 (and Figure 5) will be modified in a trivial way for recommender systems.

[0143] Within that, intervention information is provided through fine-tuning,

[0144] Figure 6 is a flowchart providing an overview of the entire process according to an exemplary, non-limiting embodiment, in which intervention information is provided to the AI ​​model through fine-tuning. In other words, the intervention information is used collectively to train the AI ​​model (i.e., a large corpus of intervention information is provided to the AI ​​system before it is instructed to generate responses to individual requests). The process begins at step 300, where the database is initialized. In many embodiments, an intermediate initializes the database with the most recent intervention information being collected. In some embodiments, the intermediate initializes the database with intervention information corresponding to a variety of "keywords" or "concepts." The initialization may further include other information about the keywords or concepts. Optionally, the database may also be initialized with intervention weights, indicating the amount by which a given intervention should be weighted when determining a response to a user request involving these keywords or concepts. The process continues to step 302, where the AI ​​model is pre-trained using "organic" information (i.e., using actual, available data without interventions). The process of pre-training the AI ​​model was described in the first part of Figure 4. The flow then proceeds to step 304, where the parameters of the pre-trained AI model are saved for later use. The process continues to step 306, which is a command flow statement that branches based on whether new intervention information should be processed. If new intervention information should be processed, the flow proceeds to step 308, where the director computer system receives the new intervention information from the stakeholder computer. The intervention information may be associated with a given stakeholder. The intervention information may be numerical, non-numerical structured text, free text, or any other form of data. If the intervention information is numerical, it may be a scalar value, a numerical vector, or a numerical array, but is not limited to this. In some embodiments, the intervention information is associated with a "keyword" or "concept".Next, in step 310, the director computer system updates the database with the new intervention information, and the process returns to step 304. Alternatively, if it is determined in step 306 that the new intervention information should not be processed, the flow proceeds directly to step 312.

[0145] Step 312 is a command flow statement that branches based on whether the AI ​​model should be fine-tuned with the current intervention information. If the AI ​​model should be fine-tuned with the current intervention information, the flow proceeds to step 314, where the director computer system loads both the parameters of the pre-trained AI model and the current intervention information. It should be noted that the parameters of the pre-trained AI model were saved in step 304 for later use. It should also be noted that the database was initialized in step 300 and updated with the new intervention information each time step 310 is reached. Then, in step 316, the AI ​​model is fine-tuned with the current intervention information. The process of fine-tuning the AI ​​model was described in the second part of Figure 4. In some embodiments, the AI ​​model is instructed in step 316 to apply a specified weight to the intervention information (compared to the "organic" information). After the AI ​​model has been fine-tuned with the intervention information, the flow proceeds to step 318. Alternatively, if it is determined in step 312 that the AI ​​model should not be updated with the current intervention information, the flow proceeds directly to step 318.

[0146] In step 318, the intermediate receives a new request from the user computer. The request may be expressed as free text submitted by the user, or in a more structured form. The request may also take the form of a short combination of search terms, as conventionally used in Google searches. The request may also consist of a complete or partial submission of voice or other audio queries. The request may also consist of a complete or partial submission of graphic images, pictures, drawings, photographs, video images, or any other form of data. Three exemplary requests will be illustrated in the first boxes of Figures 9A, 10A, and 11A, respectively. In step 320, the intermediate instructs the AI ​​system to generate a response to the request using the AI ​​model (which has been fine-tuned with intervention information). Note that the instruction in step 320 does not need to include the intervention, as the AI ​​system has already been fine-tuned with the relevant intervention information, as these are illustrated in white text with a black background in the second boxes of Figures 9A, 10A, and 11A, respectively. In step 322, the AI ​​system generates a response as instructed, the AI ​​system sends the generated response to the intermediate, and the intermediate receives the generated response. Exemplary responses, associated with the exemplary requests in the second boxes of Figures 9A, 10A, and 11A, respectively, will be illustrated in Figures 9C, 10C, and 11C, respectively. In some embodiments, the AI ​​system's generated “response” includes not only a response intended to be automatically forwarded to the requesting user computer, but also additional information that may be utilized by the intermediate without being automatically forwarded to the requesting user computer. Next, in step 324, the intermediate uses the response (and potentially additional data from a database or other source) received from the AI ​​system to generate an output that will be returned to the requesting user computer. An exemplary embodiment of step 324 is illustrated in detail in Figure 12B. Step 324 is optional, and it should be noted that in some embodiments, the intermediate uses the response received in step 322 as output, which will be returned to the requesting user computer without modification.Next, in step 326, the intermediate returns its output to the requesting user computer.

[0147] In step 328, feedback may be provided to the database. Several exemplary embodiments of the feedback occur when stakeholder computer intervention information takes the form of a bid. In that case, step 328 reports the payment to be borne by one or more stakeholders. As with current practices in sponsored search, the payment may be assessed on an impression-per-input (PPI) basis, a pay-per-click (PPC) basis, a pay-per-purchase (PPP) basis, or some future basis which is more appropriate for an artificial intelligence system. Two exemplary, non-limiting embodiments of providing feedback to the database are illustrated in detail in Figures 16A and 16B. Finally, the flow proceeds to step 330, which is a command flow statement that branches based on whether the process should continue. If the process should continue, the flow returns to step 306. Otherwise, the process ends here.

[0148] The flow shown in Figure 6 and the networks shown in Figures 1A and 1B are illustrative only. This paragraph describes some possible variations (but not limited to them). In some embodiments, updating the database with new intervention information, fine-tuning the AI ​​system with current intervention information, and processing a new user request may occur simultaneously. In some other embodiments, one or more of the steps may be removed from the flow diagram, or one or more additional steps may be added. In some other embodiments, functionality performed by a single computer system may instead be distributed across multiple computer systems, or functionality performed by multiple computer systems may instead be centralized within a single computer system. For those skilled in the art, methods for implementing these and other variations relating to the process in Figure 6 will also be obvious.

[0149] When describing FIG. 7, we assume that in step 216, the AI system returns K≧1 keywords or concepts (denoted by k = 1,...K) associated with the user request, and in step 218, the intermediary queries the database for each of these K≧1 keywords or concepts. Also, in step 220, it is assumed that the database returns intervention information in the form of a scalar positive value for each of the J≧1 options associated with the k-th keyword or concept. The intervention information associated with the k-th keyword or concept and the j-th option regarding that keyword or concept is k shown by . Further, in step 216, it is assumed that the database returns relative importance weights for each of the K≧1 keywords or concepts, and in step 220, it is assumed that the database returns intervention weights for each of the K≧1 keywords or concepts. The relative importance weight associated with the k-th keyword is shown by r [Chemical formula] and the intervention weight associated with the k-th keyword or concept is shown by w k . Both r k k and w k are assumed to be scalar positive values.

[0150] FIG. 7 details the calculation of the intervention by the intermediary using the numerical intervention information returned by the database in step 222, that is, in an exemplary non-limiting embodiment. The process proceeds from step 222-1 to step 220. In step 222-1, the intermediary initializes k = 0 and the process enters a loop process at k. Next, in step 222-2, the intermediary increments k by 1. The process continues to step 222-3, where the intermediary calculates [Chemical formula] where, in the formula, [Chemical formula] This shows intervention information associated with the k-th keyword or concept and the j-th option related to that keyword or concept. Then, in step 222-4, the intermediate is [ka] Calculate, and in the formula, [ka] This represents the intervention associated with the k-th keyword or concept and the j-th option relating to that keyword or concept. Thus, the intervention is normalized to 1 with respect to the option with the maximum intervention information associated with the k-th keyword or concept, and the intervention for any other option associated with the k-th keyword or concept is the ratio between its intervention information and the maximum intervention information. For any option associated with the k-th keyword or concept for which the database did not return any intervention information, the intervention is treated as zero (in other embodiments, the calculated intervention may be another function of the intervention information. The particular function used in Figure 7 has the useful property of taking intervention information, which can be any positive number, and converting it to an intervention that lies between 0 and 1). The flow then proceeds to step 222-5, where the intermediate checks whether k=K. If not, the process remains in the loop and returns to step 222-2. If k=K, the process exits the loop and proceeds to step 224.

[0151] Figure 8A details steps 224 and 226 of Figure 5 in an exemplary, non-limiting embodiment (this figure also details steps 426 and 428 of Figure 14 in an exemplary, non-limiting embodiment, but any differences in its application to steps 426 and 428 are relatively minor and obvious to those skilled in the art, and will therefore be omitted for brevity). For the purposes of Figure 8A and anywhere else in this specification, we define an extension request as a result of combining or concatenating data that embodies the original user request and data that embodies the intervention, and we define an extension as data that is combined or concatenated with the original user request. Optionally, extensions and extension requests may also incorporate intervention weights or other additional data. To obtain an extension request, the original user request may be expressed as text, as numbers, or as any other form of data, and the intervention may also be expressed as text, as numbers, or as any other form of data. It should be noted that an AI model, exclusively pre-trained and fine-tuned using "organic" information (i.e., using real, available data without intervention), will often be sufficient to generate a response that takes computational interventions into account when prompted by an augmentation request.

[0152] The process proceeds from step 224-1 to step 222. In step 224-1, the intermediate generates an extended request by combining or concatenating text or other data that embodies the user request received in step 212 with text or other data that embodies the intervention calculated in step 222. Optionally, the database may return intervention weights in step 220, in which case the intervention weights may also be incorporated into the extended request. The process continues to step 224-2, where the intermediate masks the extended user request from the user computer. Exemplary results of step 224-2 are depicted in the second boxes in Figures 9A, 10A, and 11A, respectively, where the user request is depicted in black type on a white background, while the extended is depicted in white type on a black background. Naturally, this depiction is provided here for illustrative purposes only, and in this embodiment, the extended would not be visible to the user computer at all. In the second box of Figure 9A, a 40% intervention weight is incorporated; in the second box of Figure 10A, a one-third intervention weight is incorporated; and in the second box of Figure 11A, a one-half intervention weight is incorporated. It should also be noted that the combination or linkage does not simply have to be an intervention following a request; i.e., the intervention may be scattered in multiple places throughout the user request, as depicted in the second boxes of Figures 9A, 10A, and 11A. Next, in step 224-3, the intermediate instructs the AI ​​system to determine a response to the extended request. After step 224-3, the process proceeds to step 226-1, where the intermediate receives the response from the AI ​​system. After the intermediate receives the response, the flow continues to step 228, where the intermediate generates an output to be returned to the requesting user computer. Step 228 will be illustrated in detail in Figure 12A, but it should be noted that, when Step 228 follows Figure 8A, many embodiments will cause the intermediate to remove any reference to the intervention in the process of generating the output. If a copy or description of the request is included as part of the output, generating the output includes a step of removing the extension.

[0153] The data embodying the original user request and the data embodying the intervention do not literally need to be combined or concatenated. Figure 8B details steps 224 and 226 of Figure 5 in another exemplary, non-limiting embodiment, in which the data embodying the original user request and the data embodying the intervention are not literally combined or concatenated, but the process has the same overall effect (this figure also details steps 426 and 428 of Figure 14 in another exemplary, non-limiting embodiment, but any differences in its application to steps 426 and 428 are relatively minor and obvious to those skilled in the art and will therefore be omitted for brevity). The process proceeds from step 224-4 to step 222. In step 224-4, the intermediate adds or appends a unique identifier to the text or other data embodying the user request received in step 212. Next, in step 224-5, the intermediate adds or appends the same unique identifier to the text or other data that embodies the intervention calculated in step 222. Optionally, the database may return the intervention weights in step 220, in which case the intermediate also adds or appends the same unique identifier to the intervention weights. The process continues to step 224-6, where the intermediate automatically transfers the user request (+identifier) ​​to the AI ​​system. Next, in step 224-7, the intermediate automatically transfers the intervention (+identifier) ​​to the AI ​​system. After this, in an optional step 224-8, the intermediate automatically transfers the intervention weights (+identifier) ​​to the AI ​​system. The second boxes in Figures 9A, 10A, and 11A can still summarize an overview of exemplary information that will be automatically transmitted to the AI ​​system in steps 224-6, 224-7, and 224-8, but unlike step 224-1 in Figure 8A, the requests and interventions (optionally, intervention weighting) are never combined and are transmitted separately to the AI ​​system. As in step 224-2, the system is programmed to ensure that the interventions are masked from the user computer.Following step 224-8, the process proceeds to step 224-9, where the intermediate instructs the AI ​​system to determine a response based on all inputs linked to a given unique identifier. Next, in step 226-2, the intermediate receives the response from the AI ​​system. Following step 226-2, the process proceeds to step 228, where the intermediate generates an output to be returned to the requesting user system. Step 228 will be illustrated in more detail in Figure 12A, but it should be noted that, when step 228 follows Figure 8B, many embodiments will cause the intermediate to remove any reference to intervention as part of the process of generating the output.

[0154] Figure 9A illustrates an exemplary request without intervention (first box) and the same exemplary request with intervention (second box). The exemplary request submitted by the user is: "The Joneses (a conservative couple in their 50s with no children) are planning a 24-hour stay in Annapolis, Maryland. Write a one-page document outlining recommendations on how the Joneses should spend their 24 hours in Annapolis, Maryland. The document should include recommendations for activities, dining locations, and accommodations." The exemplary interventions calculated in Step 222 are given in Table 1 below. [Table 1]

[0155] Note that a base rating of 7 is provided for all other tourist attractions, hotels, and restaurants so that the ratings provided have a certain standard of comparison, with the option being given. The exemplary intervention weighting is 40% for tourist attractions, hotels, and restaurants. Using this exemplary data, an embodiment according to Figure 8A would generate an extended request as depicted in the second box of Figure 9A. This exemplary extension of the request is a concise way to utilize the same free-text input format that the system uses to process requests without intervention, namely, "Suppose I have also received restaurant and hotel ratings from independent third-party reviewers. Create your recommendations, giving 40% to the independent third-party reviewers and 60% to your own information." Note that the exemplary extension incorporates a 40% intervention weighting. Also noteworthy is the sentence at the end of the extension, which ensures that the extension is masked from the user in the response: "The text must not mention that third-party reviewer ratings were entered into the recommendation, but please consider third-party reviewer ratings."

[0156] Figure 9B depicts an exemplary response without intervention, and Figure 9C depicts an exemplary response with intervention. These are actual responses generated by ChatGPT4.0 from the exemplary request in Figure 9A. Because the AI ​​system is instructed to weight both "independent third-party reviewer" ratings and its own (trained and adjusted) information, interventions are adopted for some options but not for others. For example, checking in to "Historic Inns of Annapolis" (an exemplary response without intervention in Figure 9B) is replaced with "Rodeway Inn" (an exemplary response with intervention in Figure 9C), having lunch at "Carrol's Creek Cafe" (an exemplary response without intervention) is replaced with "Dry 85" (an exemplary response with intervention), and visiting "Hammond-Harwood House" (an exemplary response without intervention) is replaced with a visit to "William Paca House and Garden" (an exemplary response with intervention). However, the AI ​​system recognizes that the Joneses have no children and therefore does not recommend "Chuck E. Cheese's," nor does it recognize that the Joneses are a conservative couple and therefore does not recommend "Pine Tree Associates Nudist Club" (or perhaps the weather is too cold).

[0157] Figure 10A illustrates a second exemplary request without intervention (first box) and the same exemplary request with intervention (second box). The exemplary request submitted by the user is: "Assume Taylor, a high school senior from Washington, D.C., has scored 720 on the SAT's Verbal section and 640 on the Math section. Taylor is interested in majoring in International Relations. Taylor wants to attend a university located within 500 miles of her home, Taylor is an atheist, Taylor does not want to attend a university with a strong Greek (fraternity) culture, and Taylor needs a sufficient amount of scholarship or financial support. Write a one-page essay recommending five universities that Taylor should consider applying to. The essay must not mention 'atheism,' but should take that into consideration when making recommendations." The exemplary interventions, calculated in Step 222, are given in Table 2 below. [Table 2]

[0158] Note that a base rating of 5 is provided for all other universities so that the ratings provided have a certain standard of comparison, with the option being given. The exemplary intervention weight is one-third. Using this exemplary data, an embodiment according to Figure 8A would generate an extended request as depicted in the second box of Figure 10A. This exemplary extension of the request is a concise way to utilize the same free-text input format that the system uses to process requests without intervention, namely, "Assume I have also received university ratings from independent third-party reviewers. Create your recommendation with one-third for the independent third-party reviewers and two-thirds for your own information." Note that the exemplary extension incorporates a one-third intervention weight. Also note the sentence at the end of the extension, namely, "Also, the sentence should not mention the existence of third-party reviewers or their ratings, but should take third-party reviewer ratings into consideration," which ensures that the extension is masked from the user in the response.

[0159] Figure 10B depicts an exemplary response without intervention, and Figure 10C depicts an exemplary response with intervention. These are actual responses generated by ChatGPT 4.0 from the exemplary request in Figure 10A. The AI ​​system is instructed to weight both the “independent third-party reviewer” ratings and its own (trained and adjusted) information, so the intervention is adopted for some options but not for others. For example, “Old Dominion University” is added to the list as number one, and “University of Virginia” is added as number two. However, the AI ​​system recognizes that Taylor is an atheist and therefore does not recommend applying to “Liberty University,” which calls itself its own “Christian academic community.”

[0160] In this embodiment, intervention information may represent the payment amount.

[0161] In some embodiments, where intervention information includes submissions from interested user users, the submissions may be numerical, and the submitted numerical values ​​may represent payment amounts. In such embodiments, the interested user submissions, which substitute for options (in the embodiments, “options” may include, but are not limited to, tourist destinations, hotels, restaurants, and universities), may be interpreted as bids. The submission of new intervention information in step 208, the updating of the database in step 210, and the processing of intervention information applicable to user requests in steps 218-222 may therefore be interpreted as auctions. Such embodiments would therefore, in some respects, be reminiscent of current sponsored search auctions. However, as already emphasized above, the output of such embodiments is not limited to an ordered list of internet hyperlinks; instead, the output may be arbitrary.

[0162] In such a context, a reinterpretation of the present invention as an auction system can be easily seen through some of our previous embodiments. For example, in the context of Figures 9A-9C and Table 1, our previous interpretation assumed that the intervention information was derived from the opinion of an independent third-party reviewer or the overall assessment of a panel of experts. An alternative interpretation is that the owner of the option in Table 1 submitted a bid. For example, Table 1 could be derived from bids listed in Table 3 below. [Table 3]

[0163] The AI ​​system is instructed that the rating scale is 1-10, and all other tourist attractions, hotels, and restaurants have received a rating of 7, therefore, the intervention is... [ka] This can be calculated from the bid. The formula is used in step 222-4, but otherwise, literally following the process detailed in Figure 7 will reproduce Table 1 as the intervention resulting from the bid in Table 3. The AI ​​system can then be instructed in exactly the same way as above, namely, instructing the AI ​​system to treat the resulting intervention as an evaluation by an independent third-party reviewer to which a predetermined weight is assigned, which is a precise way of instructing the AI ​​system on how to apply the bid.

[0164] Similarly, in the context of Figures 10A-10C and Table 2, our previous interpretation assumed that the intervention information was derived from the opinions of independent third-party reviewers or the overall assessment of an expert panel. An alternative interpretation is that the universities in Table 2 submitted bids. For example, Table 2 could be derived from bids listed in Table 4 below. [Table 4]

[0165] Here, the AI ​​system is shown a maximum rating of 9, and is informed that other universities are receiving a rating of 5, therefore the intervention is... [ka] This can be calculated from the bid. The formula is used in step 222-4, but otherwise, literally following the process detailed in Figure 7 will reproduce Table 2 as the intervention resulting from the bid in Table 4. The AI ​​system can then be instructed in exactly the same way as above, namely, to treat the resulting intervention as an evaluation by an independent third-party reviewer to which a predetermined weight is assigned, which is a precise way of instructing the AI ​​system on how to apply the bid.

[0166] In such embodiments, the feedback in step 232 is the calculation of a payment to be borne by the stakeholders. As with current practices in sponsored search, the payment may be assessed on an impression-per-pays (PPI) basis, a pay-per-click (PPC) basis, a pay-per-purchase (PPP) basis, or some future basis which is more appropriate for an artificial intelligence system. Two exemplary, non-limiting embodiments of providing feedback to a database will be illustrated in detail in Figures 16A and 16B.

[0167] In this context, the AI ​​system directly applies the bidding process.

[0168] In some of the earlier embodiments, the method in which bids are applied as intervention information may seem somewhat indirect, in that the bids submitted by stakeholders are first rephrased as comprehensive third-party evaluations and only then applied as intervention information. Here, it can be seen that bids can also be applied directly by preferably using an "intelligent" AI system.

[0169] Figure 11A illustrates an exemplary request without intervention (first box) and the same exemplary request with intervention (second box). The exemplary request submitted by the user is: "Jane and John Doe are planning a trip to Alaska in July. On day 1, they are scheduled to arrive at Fairbanks Airport around 9pm. On day 6, they are scheduled to board a cruise in Seward around 6pm. The itinerary in between is flexible. Please recommend a travel itinerary that they might be interested in. Please include three suggestions regarding hotels for each stop. Please include the distance traveled and estimated driving time for each day." The exemplary intervention in Figure 11A is summarized in Table 5 below. [Table 5]

[0170] Note that a bid of $0 is assumed for all other hotels. The exemplary intervention weight is 1 / 2. Using this exemplary data, an embodiment following Figure 8A would generate an extension, as depicted in the second box of Figure 11A, namely, "Suppose I told a couple that the cost of their travel plans would be covered by bids from hotels in Alaskan. In Denali, we have received the following bids: Denali Crow's Nest Cabins – $9, Denali Rainbow Village RV Park and Motel – $8, Denali Princess Wilderness Lodge – $4, Grande Denali Lodge – $2. In Seward, we have received the following bids: Hotel Edgewater – $10, Van Gilder Hotel – $5. If a hotel is not listed with a bid, we may assume that their bid was $0. Giving equal weight to both the bids and the inherent quality of the hotels…" Note that this exemplary extension incorporates a 50 / 50 intervention weighting, implied by “giving equal consideration to both the bid and the hotel’s inherent quality.” Also note the concluding sentence of the extension, which ensures that the extension is masked from the user in the response: “The text provided should not state that the hotel’s bid was entered into the recommended itinerary, nor should it state the bid amount, but should take the bid into consideration in the instructions.”

[0171] Figure 11B depicts an exemplary response without intervention, and Figure 11C depicts an exemplary response with intervention. These are actual responses generated by ChatGPT4.0 from the exemplary request in Figure 11A. Notably, when bid values ​​are inserted directly into the prompt, as in the second box in Figure 11A, ChatGPT4.0 is able to interpret the values ​​as bids without any special training regarding the meaning of bids. Because the AI ​​system is instructed to consider bids and inherent qualities equally, interventions are adopted for some options but not others. For example, the recommendations for "Denali Bluffs Hotel" and "McKinley Chalet Resort" (responses without intervention in Figure 11B) are replaced with a $9 bid for "Denali Crow's Nest Cabins" and a $5 bid for "Denali Princess Wilderness Lodge" (responses with intervention in Figure 11C). However, while the $8 bid for "Denali Rainbow Village RV Park and Motel" is insufficient to replace "Grande Denali Lodge" (bid of only $2), its inherent quality is clearly considered very high by the AI ​​system. Interestingly, the lack of hotel bids in Denali and Talkeetna also influences the recommended itinerary from one night in Denali and one night in Talkeetna (response without intervention in Figure 11B) to two nights in Denali and zero nights in Talkeetna (response with intervention in Figure 11C).

[0172] Output returned to the requesting user system

[0173] In conventional sponsored internet searches in this art, the typical output returned to the user is an ordered list of "organic" clickable links, often preceded (or mixed with) an ordered list of "sponsored" clickable links. However, as emphasized throughout this specification, it is considered most natural (and perhaps most effective) for computer systems to present "sponsored" content in the same format as "organic" content. Newly emerging artificial intelligence systems are not limited to generating ordered lists of links; more usefully, they can generate unordered free-form prose, visual images, or other paragraphs of novel data output. Consequently, it is presumed that any "sponsored" content should also be presented within unordered free-form prose, visual images, or other paragraphs of novel data output. If "sponsored" content is simply presented as display advertisements accompanying pages of free-form prose, it is likely to be an ineffective tool, similar to display advertisements currently used in internet publishing (which are considered far less valuable advertising tools than internet search ads). Furthermore, if "sponsored" content is simply presented as an ordered list of links preceding the AI-generated response, there is a risk that it will be completely skipped, as users will immediately turn to more useful paragraphs of unordered free-form prose, visual images, or other new data output.

[0174] This raises the question of how a search provider can resolve the following two clearly conflicting requirements.

[0175] "Sponsored" content must be placed within paragraphs of unordered free-form prose, visual images, or other new data output.

[0176] Providers must place "sponsored" content within paragraphs of unordered free-form prose, visual images, or other new data output in order to serve as effective advertising. However,

[0177] Providers have an obligation to disclose content to users when it appears in search results as a result of receiving payment or other forms of influence.

[0178] A novel method for satisfying these competitive requirements is to include links to both organic and sponsored options, but to mark them differently in a systematic way. For example (but not limited to), any of the following marking schemes may be used, alone or in combination, within a paragraph of unordered free-form prose:

[0179] Organic links may be displayed in blue, while sponsored links may be displayed in red.

[0180] Organic links may be displayed in bold, while sponsored links may be displayed in bold italics.

[0181] Organic links may have a single underline, while sponsored links may have a double underline.

[0182] When hovering over an organic link, both the URL (web address) and a hand icon (or the absence of text or warning text) may be displayed, whereas when hovering over a sponsored link, both the URL and a dollar sign or other currency symbol (or warning text) may be displayed.

[0183] Similar marking schemes can also be used within visual images or other new data outputs.

[0184] Figure 12A details step 228, which is the generation of output by the intermediate in an exemplary, non-limiting embodiment. This embodiment assumes that all “options” provided in the response can be associated with hyperlinks. For example, in Figure 9C, the phrases “Rodeway inn,” “William Paca House and Garden,” and “Dry 85” would each be associated with hyperlinks. The process proceeds from step 228-1 to step 226. In step 228-1, the intermediate copies the response received from the AI ​​system in step 226 into a new web page and marks all options with appropriate “organic” hyperlinks. According to the marking scheme described above, each of these hyperlinks is displayed in blue font, bold typeface, and single underlined, and when the user hovers the mouse over a hyperlink, both the URL (web address) and a hand icon are displayed. The process then proceeds to step 228-2, which examines one hyperlink at a time, and there it examines the first “organic” hyperlink on the webpage that has not been considered so far. The process then continues to step 228-3, which is a command flow statement that in step 224 branches based on whether an intervention has been made instead of an option associated with the hyperlink being considered. If such an intervention has been made, the flow proceeds to step 228-4, where the “organic” hyperlink associated with the option is replaced with a “sponsored” hyperlink. According to the marking scheme described above, the hyperlink is changed to red font, bold italics, and double underlined, and when the user hovers the mouse over the hyperlink, both the URL (web address) and a dollar sign will be displayed. The flow then proceeds to step 228-5. Instead, if it is found in step 228-3 that, instead of the option associated with the hyperlink under consideration, the intervention was not performed in step 224, the flow skips step 228-4 and proceeds directly to step 228-5.The process continues to step 228-5, which is a command flow statement that branches based on whether at least one “organic” hyperlink exists on a previously unconsidered webpage. If at least one “organic” hyperlink exists on a previously unconsidered webpage, the process returns to step 228-2 and continues to inspect one hyperlink at a time. If no “organic” hyperlinks exist on a previously unconsidered webpage, the generation of the output webpage is complete, and the process continues to step 230.

[0185] Figure 12B details step 324, which is the generation of output by the intermediate in an exemplary, non-limiting embodiment. This embodiment again assumes that all “options” provided in the response can be associated with hyperlinks. The process proceeds from step 322 to step 324-1. In step 324-1, the intermediate copies the response received from the AI ​​system in step 322 into a new web page and marks all options with appropriate “organic” hyperlinks. According to the marking scheme described above, each of these hyperlinks is displayed in blue font, bold typeface, and single underlined, and when the user hovers the mouse over the hyperlink, both the URL (web address) and a hand icon are displayed. The process then proceeds to step 324-2, which examines one hyperlink at a time, therefore considering a first “organic” hyperlink on the web page that has not been considered so far. The process continues to step 324-3. In embodiments such as those illustrated in Figure 6, it can be difficult to determine whether an intervention was performed instead of an option associated with the hyperlink under consideration (in step 320, the AI ​​system has already been fine-tuned with intervention information before being instructed to respond to the user request. Unlike step 224, it is not possible to simply refer to whether intervention information about an option was included in the prompt). Therefore, step 324-3 becomes a command flow statement that branches based on whether the option associated with the hyperlink under consideration benefited from the intervention. One way to assess whether an option benefited is to examine the response to the user request generated by the AI ​​model, which has been fine-tuned with intervention information, and compare it to the response to the user request generated by the same AI model, but without any fine-tuning in intervention information, as follows.If the option associated with the hyperlink under consideration benefits from the intervention, the flow proceeds to step 324-4, where the “organic” hyperlink associated with the option is replaced with a “sponsored” hyperlink. According to the marking scheme described above, these hyperlinks will each be displayed in blue font, bold typeface, and with a single underline, and when the user hovers the mouse over the hyperlink, both the URL (web address) and a hand icon will be displayed. The flow then proceeds to step 324-5. Alternatively, if it is found in step 324-3 that the option did not benefit from the intervention, the flow skips step 324-4 and proceeds directly to step 324-5. The process continues to step 324-5, which is a command flow statement that branches based on whether at least one “organic” hyperlink exists on a webpage that has not been previously considered. If at least one "organic" hyperlink exists on a webpage that has not been previously considered, the process returns to step 324-2 and continues to inspect one hyperlink at a time. If no "organic" hyperlinks exist on a webpage that has not been previously considered, the generation of the output webpage is complete, and the process continues to step 326.

[0186] It will be obvious to those skilled in the art that many other modifications relating to the embodiments illustrated in Figures 12A and 12B are also possible. For example, in some modifications, steps 324-3 in Figure 12B can be replaced by steps 228-3 in Figure 12A. Alternatively, step 228-3 in Figure 12A can be replaced by step 324-3 in Figure 12B.

[0187] Figure 13A illustrates the output generated in step 228 from the response shown in Figure 9C, with the new marking scheme. Figure 13B illustrates the output generated in step 228 from the response shown in Figure 10C, with the new marking scheme. Figure 13C illustrates the output generated in step 324 from the response shown in Figure 11C, with the new marking scheme. Since these figures are not expected to be published in color, organic links are shown in light gray (instead of blue) in Figures 13A, 13B, and 13C, and sponsored links are shown in black (instead of red).

[0188] It is worth mentioning here the detailed implementations of the approaches discussed herein and their relative advantages and disadvantages. Among them, consider any approach in which the response comprises a free-form paragraph of text, and each “option” provided within the response is marked with a clickable hyperlink. One advantage of such an approach is that it provides an inconspicuous way for the AI ​​system to disclose to the user which options may have been made in accordance with an intervention (implicitly, the intervention may have led to an overestimation of the advantages of such options); that is, hyperlinks to options made in accordance with an intervention may be displayed in one color, and hyperlinks to options made not in accordance with an intervention may be displayed in a different color. To clearly rephrase the previous sentence in the context of sponsored search, hyperlinks to options that have received bids may be displayed in one color, and hyperlinks to options that have not received bids may be displayed in a different color (or, if the use of color to distinguish hyperlinks would raise accessibility concerns for colorblind users in Section 508, other aspects of the hyperlink appearance, such as typeface, may be used instead). Therefore, there are ways for users to understand where advertisements are intervening in their responses.

[0189] Specification of stakeholder intervention information using artificial intelligence

[0190] Up to this point in the detailed description, artificial intelligence has been used primarily to generate responses to user requests. However, in the context of intervention, to realize the full potential of artificial intelligence, we should also utilize AI models to enable stakeholders to represent their intervention information more efficiently and effectively. By using the exemplary, non-limiting embodiments described herein together with those embodiments described above, it is possible to describe an improved end-to-end artificial intelligence-based sponsored search auction system. These embodiments may be used to generate conventional sponsored search results comprising an ordered list of sponsored links, as well as unconventional search results comprising unordered free-form prose or other novel output paragraphs.

[0191] In traditional sponsored search auction systems, stakeholders submit bids on keywords or concepts. Keyword bids are very suitable for search engines, where the user request itself consists of only a few search terms. However, keyword bids are hardly adjusted for search engines, where the user request is written in more nuanced and traditional prose (and is less likely to contain standard keywords).

[0192] For example, compare the following three requirements (all variations of the requirements used in Figure 9A).

[0193] (1) Annapolis Hotel

[0194] (2) The Joneses (a conservative couple in their 50s with no children) are planning to spend 24 hours in Annapolis, Maryland. Write a one-page document outlining recommendations on how the Joneses should spend 24 hours in Annapolis, Maryland. The document should include recommendations for activities, dining, and accommodation.

[0195] (3) The Joneses (a conservative couple in their 50s with no children) are planning to spend 48 hours in Annapolis, Maryland. Write a one-page document outlining recommendations on how the Joneses should spend 48 hours in Annapolis, Maryland. The document should include recommendations for activities, dining, and accommodation.

[0196] All three of these requests focus on hotels or motels located within Annapolis, and it is likely that hotels or motels located within Annapolis would want to participate in all three requests. However, there are three fundamental problems with respect to conventional keyword auctions. First, depending on the sophistication of the search engine, it is unclear whether request #2 or request #3 will necessarily trigger obvious keyword selections such as "Annapolis" and "hotel" (as the request does not contain the word "hotel"). Second, and more fundamentally, requests #2 and #3 are likely to trigger identical keyword bids (the only difference being "24" in request #2 versus "48" in request #3), but it is clear from their simple meaning that most Annapolis hotels would want to bid substantially higher on request #3 than on request #2. Thirdly, beyond any known information about the user's computer (such as its geographical location), the words included in the request may represent substantially different values ​​to different stakeholders who would normally bid for the same keywords. Considering "Historic Inns of Annapolis," this may target couples and not children. They might be willing to bid $3 for request #1, which is entirely general. However, requests #2 and #3 indicate that the search is more likely to be associated with the hotel's target customer base, and therefore more likely to result in a booking. Furthermore, if they value a two-night stay twice as much as a one-night stay, they might be willing to bid twice as much for request #3 as they did for request #2 (e.g., $6 for request #2 and $12 for request #3).

[0197] As further justification for why artificial intelligence could be useful in regulating inputs to a bidding system, recall the aforementioned example of the keywords "kids shoes" and "shoes for kids." Since these two keywords are synonyms, requiring bidders to submit separate bids for each would be considered redundant effort. By using artificial intelligence deployed to determine bids, any need for bidders to submit separate bids for these two keywords no longer exists.

[0198] In Figures 14 and 15, for the sake of brevity and clarity, we will sometimes use the term “tuned AI model” to refer to a version of the AI ​​model that has been both pre-trained on organic information and fine-tuned on bidding information. Conversely, we will sometimes use the term “pre-trained AI model” to refer to a version of the AI ​​model that is pre-trained on organic information but not fine-tuned on bidding information. The use of these terms is not intended to suggest that a “pre-trained AI model” has not been similarly fine-tuned on any organic information; it is merely intended to make it clear that a “pre-trained AI model” has not undergone any pre-training or fine-tuning on bidding data.

[0199] Figure 14 is a flowchart providing an overview of the entire process by an exemplary, non-limiting embodiment. The diagram depicts a process in which artificial intelligence is used to define bids, allowing responses to be unordered paragraphs of conventional prose or other new outputs. The process begins at step 400, where the database is initialized. In some embodiments, an intermediate initializes the database with the most recent bid information being collected. The process continues to step 402, where the AI ​​model is pre-trained with "organic information" (i.e., using actual, available data without intervention). The process of pre-training the AI ​​model is described in the first part of Figure 4, and therefore step 402 will incorporate some or all of the pre-training activities described in the first part of Figure 4. The flow then proceeds to step 404, where the parameters of the pre-trained AI model are saved for later use. The process continues to step 406, which is a command flow statement that branches based on whether new bid information should be processed. If new bid information is to be processed, the flow proceeds to step 408, where the director computer system receives the new bid information from the stakeholder computer. Continuing our ongoing implementation of the Annapolis Hotel, rows 2-4 of Table 6 illustrate exemplary new bid information that may be received from "Historic Inns of Annapolis". [Table 6]

[0200] Still continuing our ongoing example of the Annapolis Hotel, we assume that “Rodeway Inn” targets families with children. Rows 5-9 of Table 6 illustrate exemplary new bid information that may be received from “Rodeway Inn” when we reach step 408 again.

[0201] Next, in step 410, the director computer system updates the database with the new bid information received in step 408, and the process returns to step 406. Alternatively, if it is determined in step 406 that the new bid information should not be processed, the flow proceeds directly to step 412. Step 412 is a command flow statement that branches based on whether the AI ​​model should be fine-tuned with the current bid information. If the AI ​​model should be fine-tuned with the current bid information, the flow proceeds to step 414, where the director computer system loads both the parameters of the pre-trained AI model and the current bid information. Recall that the parameters of the pre-trained AI model were saved in step 404 for later use. Also recall that the database was initialized in step 400, new bid information was received each time step 408 was reached, and the database was updated with the new bid information each time step 410 was reached. Still continuing our ongoing implementation of the Annapolis Hotel, if the current bidding information consists only of bidding information received from "Historic inns of Annapolis" when step 408 is reached at one time and bidding information received from "Rodeway inn" when step 408 is reached at another time, then Table 6 will illustrate the complete set of current bidding information. Next, in step 416, the intermediate proceeds to the step of fine-tuning the AI ​​model with the complete set of current bidding information. The intermediate does this by first constructing the fine-tuning dataset from the complete set of current bidding information. In our ongoing implementation, the fine-tuning dataset is illustrated by Table 6, but naturally, the intermediate needs to convert the complete set of current bidding information into a file format required by the AI ​​model, and in doing so, the intermediate may need to add additional fields to the database.The remaining process of fine-tuning the AI ​​model is described in the second part of Figure 4, and therefore step 416 will incorporate some or all of the fine-tuning activities described in the second part of Figure 4. After the AI ​​model has been fine-tuned with the current bid information, the flow proceeds to step 418. Alternatively, if it was determined in step 412 that the AI ​​model should not be fine-tuned with the current bid information, the flow proceeds directly to step 418.

[0202] In step 418, the intermediary receives a new request from the user's computer. The request may be expressed as free text submitted by the user, or in a more structured form. The request may also take the form of a short combination of search terms, as conventionally used in Google searches. The request may also consist of a complete or partial submission of voice or other audio queries. The request may also consist of a complete or partial submission of graphic images, pictures, drawings, photographs, video images, or any other form of data. In step 420, the intermediary instructs the AI ​​system (using an AI model fine-tuned with bidding information) to generate a list of “key stakeholders” (i.e., stakeholders who would want to make the highest bid for the request received in step 418) and the amounts they would like to bid for it. In step 422, the AI ​​system generates the list of key stakeholders and bid amounts as instructed, the AI ​​system sends the list of key stakeholders and bid amounts to the intermediary, and the intermediary receives the list of key stakeholders and bid amounts. Still continuing our ongoing implementation of Annapolis hotels, if the request was "How to spend 72 hours in Annapolis for a childless couple," the fine-tuned AI model would generate a list featuring "Historic Inns of Annapolis" and "Rodeway Inn," along with associated bids such as $18 and $3, respectively (since the request here encompasses a 3-night stay), and this list would be received by the intermediary.

[0203] The flow proceeds to step 424, where the Intermediate calculates an intervention on behalf of the key stakeholders based on their bids. An embodiment of this calculation is described in detail in Figure 7 (in other embodiments, step 424 is unnecessary, i.e., the Intermediate may treat the list of key stakeholders and bids received in step 422 as the intervention itself, similar to the approach described in the text associated with Figures 11A, 11B, and 11C). Next, in step 426, the Intermediate instructs the AI ​​system (using an AI model pre-trained with organic information but not fine-tuned with bid information) to generate a response to the request received in step 418, taking into account the intervention calculated in step 424 (or the list received in step 422, if the Intermediate treats the list of key stakeholders and bids itself as the intervention). In step 428, the AI ​​system generates the response as instructed, the AI ​​system sends the generated response to the Intermediate, and the Intermediate receives the generated response. Exemplary, non-limiting embodiments of steps 426 and 428 are illustrated in detail in Figures 8A and 8B. In step 430, the intermediate generates an output. Exemplary, non-limiting embodiments of step 430 are illustrated in detail in Figures 12A and 12B. The flow continues to step 432, where the intermediate automatically transfers the output generated in step 430 to the requesting user computer. In step 434, feedback may be provided to the database. Exemplary, non-limiting embodiments of this step will be illustrated in Figure 16A below. Finally, step 436 is a command flow statement that branches based on whether the process should continue. If the process should continue, the flow returns to step 406. Otherwise, the process completes here.

[0204] Many aspects of this disclosure highlight embodiments relating to interventions in AI models that perform “search engine” functions. However, it should be emphasized that almost the same considerations also apply to AI models that perform “recommender system” (or “recommendation system”) functions. Therefore, for those skilled in the art, methods for modifying the process in Figure 14 to illustrate interventions in AI models that perform recommender system (or recommendation system) functions will also be obvious. For example, step 418 for some recommender systems may involve a user request for recommendations, but on shopping or e-commerce websites, it is more likely that the request for recommendations for the user will actually be initiated by the shopping or e-commerce website, regardless of whether the consumer (user) actually desires recommendations. Therefore, step 418 (and Figure 14) will be modified in a trivial way for recommender systems.

[0205] Figure 15 is a flowchart providing an overview of the entire process by an exemplary, non-limiting embodiment. The diagram depicts a process in which artificial intelligence is used to define bids, and in which the output returned to the user comprises a combination of an ordered list of sponsored links (typically as found in search engine output today) and a response generated by the AI ​​model. The response itself is allowed to be an ordered list of links, an unordered paragraph of conventional prose, or other novel output. The process begins at step 500, where the database is initialized. In some embodiments, an intermediate initializes the database with the most recent bid information being collected. The process continues to step 502, where the AI ​​model is pre-trained with "organic information" (i.e., using actual, available data without intervention). The process of pre-training the AI ​​model is described in the first part of Figure 4, and therefore step 502 will incorporate some or all of the pre-training activities described in the first part of Figure 4. Next, the flow proceeds to step 504, where the parameters of the pre-trained AI model are saved for later use. The process continues to step 506, which is a command flow statement that branches based on whether new bid information should be processed. If new bid information should be processed, the flow proceeds to step 508, where the director computer system receives the new bid information from the stakeholder computer (the explanations and examples provided above with respect to step 408 are equally applicable to step 508). Then, in step 510, the director computer system updates the database with the new bid information, and the process returns to step 506. If, instead, it is determined in step 506 that the new bid information should not be processed, the flow proceeds directly to step 512.

[0206] Step 512 is a command flow statement that branches based on whether the AI ​​model should be fine-tuned with the current bid information. If the AI ​​model should be fine-tuned with the current bid information, the flow proceeds to step 514, where the director computer system loads both the parameters of the pre-trained AI model and the current bid information. Recall that the parameters of the pre-trained AI model were saved in step 504 for later use. Also recall that the database was initialized in step 500 and updated with the new bid information each time step 510 is reached (the explanations and examples provided above with respect to step 414 are equally applicable to step 514). Then, in step 516, the AI ​​model is fine-tuned with the current bid information. The intermediate does this first by constructing the fine-tuning dataset from the complete set of current bid information. In our ongoing embodiment, the fine-tuning dataset is illustrated by Table 6, but naturally, the intermediate needs to convert the complete set of current bid information into the file format required by the AI ​​model, and in doing so, the intermediate may need to add additional fields to the database. The remaining process of fine-tuning the AI ​​model is described in the second part of Figure 4, and therefore step 516 will incorporate some or all of the fine-tuning activities described in the second part of Figure 4. After the AI ​​model has been fine-tuned with the bid information, the flow proceeds to step 518. Alternatively, if it is determined in step 512 that the AI ​​model should not be fine-tuned with the current bid information, the flow proceeds directly to step 518.

[0207] In step 518, the intermediary receives a new request from the user's computer. The request may be expressed as free text submitted by the user, or in a more structured form. The request may also take the form of a short combination of search terms, as conventionally used in Google searches. The request may also consist of a complete or partial submission of voice or other audio queries. The request may also consist of a complete or partial submission of graphic images, pictures, drawings, photographs, video images, or any other form of data. In step 520, the intermediary instructs the AI ​​system (using an AI model fine-tuned with bidding information) to generate a list of “key stakeholders” (i.e., stakeholders who would want to make the highest bid for the request received in step 518) and the amounts they would like to bid for it. In step 522, the AI ​​system generates the list of key stakeholders and bid amounts as instructed, the AI ​​system sends the list of key stakeholders and bid amounts to the intermediary, and the intermediary receives the list of key stakeholders and bid amounts.

[0208] The flow proceeds to step 524, where the intermediary queries the database for additional data associated with the request received in step 518 and receives the additional data from the database. Examples of additional data obtained in step 524 include, but are not limited to, sponsored links that each key stakeholder wishes to display with the response to the request, information about the components of the quality score for each key stakeholder (click-through rate, ad relevance, and landing page experience), information about the expected impact of ad extensions and other ad formats for each key stakeholder, and the minimum price that should apply to this request. The flow continues to step 526, where the intermediary instructs the AI ​​system (using an AI model that is pre-trained with organic information but not fine-tuned with bidding information) to generate a response to the request received in step 518. In step 528, the AI ​​system generates the response as instructed, the AI ​​system sends the generated response to the intermediary, and the intermediary receives the generated response. In step 530, the intermediate selects sponsored links and their order to be sent to the user's computer, using the response received from the AI ​​model in step 528, the data obtained from the database in step 524, and the list of key stakeholders and bid amounts received from the AI ​​model in step 522, to construct a web page for the requesting user. In one exemplary non-limiting embodiment ("Plain Vanilla" Generalized Second Price Auction), the intermediate selects sponsored links in descending order of bid amount and cuts off the list either at the lowest price or so that it contains the maximum number of sponsored links. In another exemplary non-limiting embodiment, the intermediate first adjusts the bid amounts by a quality score associated with each key stakeholder.In some of these embodiments, the intermediate constructs the top of the webpage by publishing selected sponsored links in a selected order, and then proceeds to the step of constructing the rest of the webpage by publishing the response generated by the AI ​​model below the selected sponsored links.

[0209] The flow continues to step 532, where the intermediate automatically transfers the web page constructed in step 530 to the requesting user computer. Next, in step 534, feedback may be provided to the database. An exemplary, non-limiting embodiment of this step will be illustrated in Figure 16B below. Finally, step 536 is a command flow statement that branches based on whether the process should continue. If the process should continue, the flow returns to step 506. Otherwise, the process completes here.

[0210] Note that in Figures 14 and 15, the terminology has been specialized from “intervention information” (as used in Figure 6, for example) to “bid information.” The purpose of this specialization of terminology is to simplify reading and understanding, based on the recognition that stakeholder bids are likely to be one of the most useful applications of the depicted embodiment. However, the exact same process as depicted in these figures would function similarly if “bid information” were replaced by “intervention information.” For example, intervention information may include free-text opinions submitted by independent third-party experts. In step 416, the AI ​​model may be fine-tuned with the opinions of the independent third-party experts. Then, in step 420, the fine-tuned AI model may be instructed to generate a list of ratings based on the opinions of the independent third-party experts, along with key outcomes, associated with the user request. In step 424, the intervention may be calculated based on the generated ratings. Finally, in step 426, the pre-trained AI model may be instructed to generate a response to the user request, taking the calculated intervention into consideration. Therefore, we can see that the embodiments depicted in Figures 14 and 15 are useful not only as a process in which artificial intelligence is used to help define monetary bids, but also as a process in which artificial intelligence is used to help define independent third-party evaluations or other non-monetary interventions.

[0211] It should also be noted that other modifications relating to the embodiments described in Figures 14 and 15 will be obvious to those skilled in the art. Recall that, broadly speaking, there are two main approaches to teaching new information, such as intervention information, to the AI ​​model: fine-tuning and the use of contextual prompts. The processes in Figures 14 and 15 are both described using fine-tuning to teach the AI ​​model the bidding data (as in Figure 6). Obviously, the process can be easily modified to teach the AI ​​model the bidding data using contextual prompts instead (as in Figure 5). Furthermore, it should be understood that by using embeddings, similar results can be obtained to using contextual prompts, and therefore, methods for modifying the processes in Figures 14 and 15 to use embeddings instead will also be obvious to those skilled in the art. Alternatively, as discussed above, the process may operate in a manner similar to Search Augmentation Generator (RAG).

[0212] It should be noted that reliability issues may exist when using the AI ​​system to determine bidder bids if the company providing the AI ​​system has a strong financial incentive to overestimate the amount stakeholders are willing to pay for the requests. One approach to mitigate reliability issues is as follows: Stakeholders begin by submitting bid information (e.g., rows 2-4 or 5-9 in Table 6). The AI ​​system responds by providing the stakeholder with an estimated amount that the stakeholder would be willing to bid for, along with a list of the most common requests related to the stakeholder's submitted bid information. The stakeholder then has the option of approving the estimates in this list or revising the estimated bid amount using the stakeholder's own figures. This process can then be repeated until the stakeholder and the AI ​​system converge on a final list of acceptable bid amounts, which the AI ​​system will apply as the stakeholder's actual bid information. Naturally, the bid amounts received in step 422 or 522 are also subject to post-mortem review and audit.

[0213] Provide feedback to the database

[0214] The method by which the intermediary provides feedback to the database after the intervention needs to be described in detail. This main part of the process may be used to record payments, for example, that are borne by stakeholders, in embodiments where stakeholders bid for the intervention to occur.

[0215] Figure 16A details step 434, which is the provision of feedback to the database by an intermediate in an exemplary, non-limiting embodiment (the figure also details steps 232, 328, and other steps in which feedback is provided in several exemplary, non-limiting embodiments. Any differences in its application to steps 232, 328, or other feedback steps are relatively minor and obvious to those skilled in the art and will therefore be omitted for brevity). This embodiment assumes that all “options” provided in the response are marked with hyperlinks. For example, in Figure 9C, the phrases “Rodeway inn,” “William Paca House and Garden,” and “Dry 85” are presented as clickable links, and it is further assumed that clicks can be counted using techniques similar to those used in conjunction with sponsored searches today. The process proceeds from step 432 to step 434-1. In step 434-1, the intermediary receives a unique identifier for the request / response, along with a report that a given hyperlink contained within the response to the user request was clicked. We will indicate the option associated with the clicked hyperlink by k. In step 434-2, the intermediary uses the unique identifier to identify the stakeholder j associated with option k and the amount that stakeholder j is willing to bid for option k. [ka] The database is queried with respect to (this could be the actual bid of stakeholder j, or, in accordance with step 420, an estimate of the amount that stakeholder j would like to bid for this request) and (this could be the actual bid of stakeholder j, or, according to step 420, an estimate of the amount that stakeholder j would like to bid for this request) and (this could be the actual bid of stakeholder j) and (this could be an estimate of the system of the present invention) and (this could be an actual bid of stakeholder j) and (this could be an estimate of the amount that stakeholder j would like to bid for this request) and (this could be an in accordance with step 420) and (this could be an actual bid of stakeholder j) and (this could be an estimate of the amount that stakeholder j would like to bid for in this request) and (this could be an actual bid of stakeholder j) and (this could be an estimate of the amount that would be the amount that stakeholder j would like to bid for in this request) and (this could be an actual bid of stakeholder j) and (this could be an estimate of the amount that would be the amount that stakeholder j would like to bid for in this request) and (this could be an actual bid of stakeholder j) and k , associated intervention weighting w kQuery the database regarding the lowest price or other aspects of the associated intervention information. Next, in step 434-3, the database returns to the intermediary the data queried in step 434-2. After this, the process proceeds to step 434-4, which is the amount of money desired by stakeholder j to bid regarding option k

Chem.

Chem.

Chem.

Chem.

Chem.

[0216] Figure 16B details step 534, which is the provision of feedback to the database by the Intermediate in another exemplary non-limiting embodiment (the figure also details steps 232, 328, and other steps in which feedback is provided in several exemplary non-limiting embodiments. Any differences in its application to steps 232, 328, or other feedback steps are relatively minor and obvious to those skilled in the art and will therefore be omitted for brevity). This embodiment assumes that the output of the system of the present invention consists of an ordered list of sponsored links (as in a conventional sponsored search auction), along with other data as AI-generated responses and possibilities to user requests. The process proceeds from step 532 to step 534-1. In step 534-1, the Intermediate receives a unique identifier for the request / response, along with a report that one of the sponsored links was clicked by the user. Let k represent the request, and j represent one of the key stakeholders associated with request k. In step 534-2, the intermediary uses a unique identifier to identify the primary stakeholder associated with request k at the time the response was generated, and the amount that the primary stakeholder was willing to bid for the request. [ka] The database is queried with respect to (these may be the actual bids of the stakeholders, or estimates of the System of the Invention of the amount that the stakeholders wish to bid for the request in accordance with step 520). In some exemplary, non-limiting embodiments, the intermediate also queries the database with respect to additional data associated with the request, which may include, for example, components of the key stakeholders' quality score (e.g., click-through rate, ad relevance, and landing page experience), and may include, for example, the key stakeholders' ad extensions and expected impact from other ad formats. Next, in step 534-3, the database returns to the intermediate the data queried in step 534-2. The process then proceeds to step 534-4, where the intermediate calculates the transaction amount that will be associated with the clicked sponsored link as a function of the data returned in step 534-3. In some exemplary, non-limiting embodiments, the system employs a generalized second-price auction, in which the system employs the data returned in step 534-3 to calculate the minimum amount that an interest party associated with the clicked sponsored link may bid while maintaining its position in the sponsored search results. Next, in step 534-5, the transaction amount calculated in step 534-4 is added to the database. After step 534-5, the process proceeds to step 536. Note that in some exemplary, non-limiting embodiments, some of the steps illustrated in Figure 16B may be performed in other sequences, or some of the steps may be performed before or after reaching step 534.

[0217] A new challenge arises when using a click-based billing approach, particularly when responses include free-form paragraphs of text. In traditional sponsored searches, where search results are simply ordered lists of hyperlinks, users will typically either click the hyperlinks immediately or not at all (few users today save those search results and click them later). In contrast, when a search engine generates free-form paragraphs of text (e.g., in response to a request along the lines of "How to spend 48 hours in Annapolis"), users are quite likely to save the output and only click the options days later. It is unclear how well technologies for monitoring click counts will work when clicks can occur so far in the future. Furthermore, reliability issues can arise when stakeholders are billed for clicks that occur days or weeks after the search.

[0218] Artificial Intelligence Selection Mechanism System

[0219] Enabling bidders to use artificial intelligence to represent or define their bids is considered a very powerful approach. This is potentially a significant improvement over simply specifying bid amounts for specific keywords, and this approach is likely to have applications where the system implements a broader selection mechanism. We first define the following: a. A selection mechanism is defined as a procedure that asks multiple participants to make a choice from several possible choices. Such choices may include, but are not limited to, identifying one or more elements of a set, ranking one or more elements of a set, identifying the quantity of one or more elements of a set, associating a price with one or more elements of a set, associating a parameter with one or more elements of a set, or associating multiple parameters with one or more elements of a set. The selection mechanism aggregates the choices induced by the participants into an outcome, which is often a decision, allocation, or result. b. In short, the selection mechanism takes the "selection" of "participants" as input and produces "outcomes" as output.

[0220] Selection mechanisms are described in Komo and Ausubel (2020). Examples of selection mechanisms include school selection mechanisms, auction mechanisms, and voting mechanisms. In a school selection mechanism, participants may be students, the selection may be a ranked list of schools, and the outcome may be the assignment of students to schools. In an auction mechanism, participants may be bidders, the selection may be bids, and the outcome may be the assignment of items to bidders and the associated payments by the bidders. In a voting mechanism, participants may be voters, the selection may be votes on candidates (or a ranked list of candidates), and the outcome may be the winning candidate. Selection mechanisms can be further described as static selection mechanisms if there is only one round or submission window for participants, and as dynamic selection mechanisms if there are (at least potentially) multiple rounds or submission windows for participants.

[0221] The selection, as represented by participants within the selection mechanism, may be referred to as ranking, listing, bidding, voting, reporting, disclosure, preference, or other names. For the sake of brevity and clarity, we will generally use the term "selection," which is intended as a technical term to encompass, but not limited to, all other possible terms for selection. Participants within the selection mechanism may be referred to as students, bidders, voters, or other names. For the sake of brevity and clarity, we will generally use the term "participant," which is intended as a technical term to encompass, but not limited to, all other possible terms for participant. The outcomes determined by the selection mechanism may be referred to as allocation, distribution, and associated payments, results, winners, or other names. For the sake of brevity and clarity, we will generally use the term "outcome," which is intended as a technical term to encompass, but not limited to, all other possible terms for outcome.

[0222] Furthermore, an artificial intelligence selection mechanism system can be defined as a computer system that utilizes an AI model for one of the following purposes:

[0223] To assist participants in creating, representing, or defining those submissions of selections to the selection mechanism.

[0224] Suggest improvements to the participant selection process, or

[0225] To provide an economic means of submitting a choice of proxy or a conditional choice.

[0226] Before describing the auction process in detail, the architecture of an exemplary computer system according to one embodiment will be described with reference to Figure 17. In the graphical depiction of Figure 17, the computer system consists of participant computers 610a-m and manager computers 620a-n that communicate with the selection mechanism computer (CMC) 630 and the artificial intelligence computer 640 via the network 650. The computers or terminals 610a-m are employed by participants within the selection mechanism, and the computers or terminals 620a-n are employed by those who manage the selection mechanism. The selection mechanism computer 630 processes submissions, and the artificial intelligence computer 640 provides artificial intelligence services to users. User computers (i.e., participant computers 610a-m and manager computers 620a-n) will be shown in detail in Figure 18A. The selection mechanism computer 630 will be shown in detail in Figure 18B. Details of the artificial intelligence computer 640 have already been illustrated in Figures 2 and 3. In some embodiments, the system architecture is a client-server system, where the selection mechanism computer 630 is the server, and the participant computers 610a-m, manager computers 620a-n, and artificial intelligence computer 640 are clients.

[0227] Figure 18A provides a further detailed illustration of the user computers (i.e., participant computers 610a-m and manager computers 620a-n) shown in Figure 17. A typical user computer 660 includes a user interface 661 for input / output, which may include conventional keyboards, mice, displays, touchscreens, speakers, microphones, cameras, and other input / output devices. The user interface 661 is coupled to a network interface 662, which communicates through the network 650. Both the user interface and the network interface are connected to a CPU 663 within each computer. Each computer includes memory 664, which may store an operating system 665, a web browser 666 (e.g., Safari or Google Chrome), other programs 667 (but not necessarily any computer programs specific to the selection mechanism), and data 668. In each user computer 660, the CPU 663 is a logic network that executes instructions from memory 664 so that processing and input / output operations via the user interface and network interface occur as in the conventional art.

[0228] FIG. 18B is a further detailed illustration of the selection mechanism computer (CMC) 630 shown in FIG. 17. The CMC 630 typically includes a CPU 631, a memory 632, a data storage device 633, a network interface 634, and a clock 635, which are typically hardware devices that are interconnected with each other. The CMC may sometimes also include a user interface 636, but this should be considered optional since the CMC is often located within a cloud data center. The operating system 637, programs 638, and data 639 are typically stored in the memory 632. Other data 639, namely, school names 639-1, the number of available slots within each school 639-2, other information about each school 639-3, other initial parameters 639-4, institution time tables 639-5, participant names 639-6, participant addresses 639-7, participant priorities 639-8, participant login IDs 639-9, and participant passwords 639-10, etc., are typically stored on the data storage device 633. However, in some embodiments, some or all of this data may alternatively be stored in the memory 632. Clearly, the types of data included within the data 639 are specific to the type of selection mechanism. The CPU 631 of the CMC is a logic network that executes instructions from the memory 632 such that processing and input / output operations via the network interface occur as is conventional in the relevant art.

[0229] We will now describe some exemplary embodiments of the artificial intelligence selection mechanism system. Throughout the descriptions of FIGS. 19, 20, 23A, 23B, 24A, and 24B, references to "the first preferred embodiment", "the second preferred embodiment", and "the third preferred embodiment" will be repeated. For clarity, these preferred embodiments relate to artificial intelligence systems and methods for assisting participants hereinafter.

[0230] First preferred embodiment: a sealed bidding type school selection mechanism (Figures 19 and 23A) and a dynamic school selection mechanism (Figures 20 and 23A).

[0231] A second preferred embodiment: a sealed-bid auction mechanism (Figures 19 and 23B) and an ascending-order clock auction mechanism (Figures 20 and 24A).

[0232] A third preferred embodiment: a single-round voting mechanism (Figures 20 and 24B) and a runoff voting selection mechanism (Figures 20 and 24B).

[0233] Figure 19 is a process flowchart according to one embodiment. It illustrates the steps of using artificial intelligence to assist participation in a single-round selection mechanism, including, but not limited to, a static school selection mechanism, a sealed-bid auction mechanism, and a standard voting mechanism. The process begins at step 700, where the memory location of the selection mechanism computer (CMC) is initialized. In a first preferred embodiment, the appropriate memory location of the CMC is initialized with mechanism information such as the names of the schools whose slots are options within the school selection mechanism, the number of slots available within each school, the criteria by which each school's priority is applied (e.g., longest distance from the school or older siblings already enrolled at the school), other information about each school, other initial parameters, a mechanism timetable, a list of participant names, a list of their associated addresses, a list of their associated sibling statuses, a list of associated IDs, and a list of associated passwords. In a second preferred embodiment, the appropriate memory location of the CMC is initialized with mechanism information such as products in the auction mechanism, the available quantity of each product in the auction mechanism ("product" is defined as the type of item), other information about the products in the auction mechanism (e.g., population in each region if regional licenses are provided for multiple regions), initial parameters (e.g., minimum price parameter), mechanism timetable, list of participant IDs, list of associated passwords, and list of associated quantity or value limits. In a third preferred embodiment, the appropriate memory location of the CMC is initialized with mechanism information such as the names of candidates in the voting mechanism, the positions they are running for, other information about each candidate (e.g., their political party) and each position (e.g., the number of positions to be filled), initial parameters, mechanism timetable, list of participant IDs, list of associated passwords, and a list of positions each participant can vote for. In some embodiments, the mechanism information used to initialize the appropriate memory location of the CMC is obtained from a manager computer.

[0234] The process then proceeds to an optional step 702, in which at least one training dataset is created from past implementations of the selection mechanism, and then one or more AI models are trained or fine-tuned using the at least one dataset. In some preferred embodiments, the training dataset is created from the participant's own submissions within the similar selection mechanism, and in other preferred embodiments, the training dataset is created from the submissions of all participants within the similar selection mechanism. The process of pre-training and fine-tuning the AI ​​models has already been described in Figure 4, and therefore step 702 will incorporate some or all of the pre-training and fine-tuning activities described in Figure 4. Next, in step 704, the CMC transmits mechanism information, including the mechanism parameters (if applicable), to the participant computers. In some embodiments, the CMC outputs the mechanism information through its network interface and transmits it over the network, and the participant computers then receive the mechanism information through their network interfaces and display the information to the participants through their user interfaces.

[0235] The process then proceeds to step 706, where the submission window is opened. In step 708, the CMC obtains submissions from participants, including selections and requests. In some embodiments, participants input their submissions through the user interface of their participant computer, which then outputs the submissions through the computer's network interface, transmitting the submissions over the network. The CMC then receives the submissions through its network interface for use in the next step. A submission includes “selections” and / or “requests,” where “selections” are data that can be directed for use as input into the selection mechanism, while “requests” are anything else (e.g., free-form text that will request interpretation by the AI ​​model, which the AI ​​model will convert into “provisional selections”). In step 710, the CMC separates the “requests” in the obtained submissions from the “selections” and transmits the requests to the AI ​​model. In some embodiments, the CMC outputs the requests through its network interface, transmitting them over the network, and the AI ​​computer then receives the requests through its network interface. In step 712, the AI ​​model is prompted to translate the request into a provisional selection, which is then transmitted to the CMC. In some embodiments, the prompted AI model is fine-tuned in step 704 using data from past mechanisms. In some embodiments, the AI ​​model is an LLM model that has not been specifically fine-tuned on data from past mechanisms. In some embodiments, the AI ​​computers output the provisional selections through their network interfaces and transmit them over the network, and the CMC then receives the provisional selections through its network interfaces.

[0236] In step 714, the CMC applies constraints to the selections (including provisional selections), if applicable, and enters only those selections (including provisional selections) that satisfy the constraints. In a first preferred embodiment, the CMC applies a constraint on the number of schools that participants in the school selection mechanism are allowed to rank in their submissions. In a second preferred embodiment, the CMC applies a first constraint that limits quantity submissions to integer values ​​and a second constraint that limits quantity submissions to values ​​that do not exceed the supply of that product in the auction mechanism. In a third preferred embodiment, the CMC applies constraints based on a list of positions that each participant can vote for and the number of votes for a given position that participants in the voting mechanism are allowed to cast in their submissions. In step 716, the CMC provides participants with feedback on the selections (including provisional selections) entered in step 714, if applicable. In some embodiments, the CMC outputs feedback through its network interface and transmits it over the network, and the participant computers then receive the feedback through their network interfaces and display the feedback to the participants through their user interfaces. In some embodiments, this step also includes notifying the participant if any of their selections were not entered in step 714 because the selection did not satisfy the constraints. In some preferred embodiments, this step also includes giving the participant an opportunity to confirm that their provisional selections reflect the participant's intentions, to modify their provisional selections to better reflect their intentions, or to correct any selections that did not satisfy the constraints.

[0237] The process then proceeds to step 718, where the submission window is closed. In some preferred embodiments, this step also includes causing the CMC to convert the “provisional selections” whose entries are pending at the time the submission window is closed into “selections” and to merge them with the entered selections obtained in step 708. Next, in step 720, the CMC processes the selections and determines the outcome of the mechanism. In most embodiments, the selections processed in step 720 will reflect only those selections entered in step 714 (i.e., selections that did not satisfy the constraints will not be processed) and will reflect any modifications or corrections that participants were allowed to make to those selections (e.g., after providing feedback in step 716, if applicable). Some preferred embodiments of the process in step 720 will be shown in detail in Figures 23A and 23B.

[0238] Finally, the process proceeds to step 722, where the CMC outputs a final message containing the results of the selection mechanism. In a first preferred embodiment, the final message includes the assignment of students to schools and, if applicable, a waiting list for one or more schools. In a second preferred embodiment, the final message includes the final price for each product, the quantity of each product allocated to each bidder, and the payment associated with each bidder, where the payment associated with a given bidder is equal to the dot product of the quantity vector of each product allocated to that bidder and the final vector of the final price for each product. In a third preferred embodiment, the final message is a list of winners for each position and the number of votes for each candidate. In some embodiments, the CMC outputs the final message through its network interface and transmits it over the network, and the participant computers and manager computers then receive the final message through their network interfaces and display it to the participants and managers through their user interfaces. In other embodiments, the final message is output only to the manager computer so that the person managing the selection mechanism can scrutinize the results before disclosing them to the participants. The process then terminates.

[0239] Figure 20 is a process flowchart according to one embodiment. This illustrates, but is not limited to, steps that use artificial intelligence to assist participation in a dynamic selection mechanism, including a dynamic school selection mechanism, a dynamic auction mechanism, and a runoff selection mechanism. The dynamic school selection mechanism is described by Abdulkadiroglu and Sonmez (2003) and Chen and Kesten (2017). The dynamic auction mechanism is described by Ausubel (2000). The runoff selection mechanism, described by Bouton (2013), is in which the candidate with the highest number of votes is immediately declared the winner if the vote exceeds a threshold percentage; otherwise, a second round of voting takes place between the two candidates with the highest number of votes in the first round. The process begins at step 750, where the memory location of the selection mechanism computer (CMC) is initialized. In a first preferred embodiment, the appropriate memory location for the CMC is initialized with institutional information such as the names of schools whose slots are options within the school selection mechanism, the number of slots available within each school, the criteria for applying priority to each school with respect to it (e.g., longest distance from the school or older siblings already registered at the school), other information about each school, the mechanism timetable, a list of participant names, a list of their associated addresses, a list of their associated sibling statuses, a list of associated IDs, and a list of associated passwords. In a second preferred embodiment, the appropriate memory location for the CMC is initialized with institutional information such as products within the auction mechanism, the available quantity of each product within the auction mechanism, other information about products within the auction mechanism (e.g., population within each region if regional licenses are provided for multiple regions), initial parameters (e.g., minimum price parameter), the mechanism timetable, a list of participant IDs, a list of associated passwords, and a list of associated quantity limits or value limits.In a third preferred embodiment, the appropriate memory location of the CMC is initialized with mechanism information such as the names of candidates in the voting mechanism, the positions they are running for, other information about each candidate (e.g., their political party) and each position (e.g., the number of positions to be filled), the mechanism timetable, a list of participant IDs, a list of associated passwords, and a list of positions each participant can vote for. In some embodiments, the mechanism information used to initialize the appropriate memory location of the CMC is obtained from a manager computer.

[0240] The process continues to an optional step 752, where at least one training dataset is created from past implementations of the selection mechanism, and then one or more AI models are trained or fine-tuned using the at least one dataset. In some preferred embodiments, the training dataset is created from the participant's own submissions within the similar selection mechanism, and in other preferred embodiments, the training dataset is created from the submissions of all participants within the similar selection mechanism. The process of pre-training and fine-tuning the AI ​​models has already been described in Figure 4, and therefore step 752 will incorporate some or all of the pre-training and fine-tuning activities described in Figure 4. Next, in step 754, the CMC establishes initial parameters (if applicable). In a first preferred embodiment, the initial parameter is the number of available slots within each school. In a second preferred embodiment, the initial parameter is the starting price per product. In a third preferred embodiment, the initial parameter is the voting threshold percentage required per position. In step 756, the CMC transmits mechanism information, including the current parameters (if applicable), to the participant computer. In some embodiments, the CMC outputs mechanism information through its network interface and transmits it over the network, and participant computers then receive the mechanism information through their network interfaces and display the information to the participants through their user interfaces.

[0241] The process then proceeds to step 758, where the submission window is opened. In step 760, the CMC obtains submissions from participants, including selections and requests. In some embodiments, participants input their submissions through the user interface of their participant computer, which then outputs the submissions through the computer's network interface, transmitting the submissions over the network. The CMC then receives the submissions through its network interface for use in the next step. A submission includes “selections” and / or “requests,” where “selections” are data that can be directed for use as input into the selection mechanism, while “requests” are anything else (e.g., free-form text that will request interpretation by the AI ​​model, which the AI ​​model will convert into “provisional selections”). Examples of “requests” will be shown in the first boxes of Figures 21A, 21B, 22A, and 22B, respectively. In step 762, the CMC separates the “requests” in the obtained submissions from the “selections” and transmits the requests to the AI ​​model. In some embodiments, the CMC outputs requests through its network interface and transmits them over the network, and the AI ​​computer then receives the requests through its network interface. In step 764, the AI ​​model is prompted to translate the requests into provisional selections, which are then transmitted to the CMC. In some embodiments, the prompting AI model is fine-tuned in step 754 using data from historical mechanisms. In some embodiments, the AI ​​model is an LLM model that has not been specifically fine-tuned on data from historical mechanisms. In some embodiments, the context associated with the selection mechanism is included in the prompt to the AI ​​model along with the requests.Examples of such contexts will be shown in the second boxes of Figures 21A, 21B, 22A, and 22B, respectively (therefore, in these examples, prompts to the AI ​​model will be based on a combination of the contents of the first and second boxes of these individual figures). Examples of provisional selections made by the AI ​​model will be shown in the third boxes of Figures 21A, 21B, 22A, and 22B, respectively. In some embodiments, the AI ​​computers output provisional selections through their network interfaces and transmit them over the network, and the CMC then receives the provisional selections through its network interfaces.

[0242] In closely related embodiments, step 764 may operate in a manner similar to Search Enhancement Generation (RAG). The system may use a vector search function to retrieve the most relevant data from a set of mechanism information, taking care to avoid accessing other participants' selections that the participant should not have direct or indirect access to. The system may then include this information in a prompt sent to the AI ​​model. Also, since it is understood that using embeddings in step 764 can yield similar results to using context prompts, other alternative embodiments exist that utilize embeddings instead of providing context to the AI ​​model in a prompt.

[0243] In step 766, the CMC applies constraints to the selections (including provisional selections), if applicable, and enters only those selections (including provisional selections) that satisfy the constraints. In a first preferred embodiment, the CMC applies a constraint on the number of schools that participants in the school selection mechanism are allowed to rank in their submissions. In a second preferred embodiment, the CMC applies a first constraint that limits quantity submissions to integer values, a second constraint that limits price submissions to values ​​less than the round's starting price and not exceeding the clock price for the associated product, and a third constraint that limits quantity submissions to values ​​not exceeding the supply of that product in the auction mechanism. In a third preferred embodiment, the CMC applies constraints based on a list of positions that each participant can vote for and the number of votes for a given position that participants in the voting mechanism are allowed to cast in their submissions. In step 768, the CMC provides participants with feedback on the selections (including provisional selections) entered in step 766, if applicable. In some embodiments, the CMC outputs feedback through its network interface and transmits it over the network, and the participant computers then receive the feedback through their network interfaces and display the feedback to the participants through their user interfaces. In some embodiments, this step also includes notifying the participant if any of their selections were not entered in step 766 because the selection did not satisfy the constraints. In some preferred embodiments, this step also includes giving the participant an opportunity to confirm that their provisional selections reflect the participant's intentions, to modify their provisional selections to better reflect their intentions, or to correct any selections that did not satisfy the constraints.

[0244] The process then proceeds to step 770, where the submission window is closed. In some preferred embodiments, this step also includes causing the CMC to convert the “provisional selections” whose entries are pending at the time the submission window is closed into “selections” and to merge them with the entered selections obtained in step 760. Next, in step 772, the CMC processes the selections and determines the outcome of the round. In most embodiments, the selections processed in step 772 will reflect only those selections entered in step 766 (i.e., selections that did not satisfy the constraints will not be processed) and will reflect any modifications or corrections that participants were allowed to make to those selections (e.g., after providing feedback in step 768, if applicable). Some preferred embodiments of the process in step 772 will be shown in detail in Figures 24A and 24B.

[0245] The process continues to step 774, which is a command flow statement that branches based on a decision on whether the mechanism should continue. In a first preferred embodiment, the decision is based on whether steps 758-772 have been performed the requested number of times. In a second preferred embodiment, the decision is based on whether the total demand for all products is less than or equal to the available supply. In a third preferred embodiment, the decision is based on whether the maximum number of votes for each position exceeds the threshold percentage of votes for that position (or whether the second round of runoff voting has already been completed).

[0246] If the organization should continue, the flow proceeds to step 776, where the CMC establishes updated parameters (if applicable). In a first preferred embodiment, the updated parameters are the number of available slots remaining in each school after subtracting the slots allocated in step 772. In a second preferred embodiment, the updated parameters are the "starting round price" per product, based on the "posted price" determined in step 772, and the "clock price" per product, based on a certain percentage added to the starting round price. In a third preferred embodiment, the updated parameters are the names of the top two candidates who received the most votes in the first round for any position, provided that the candidate with the most votes in the first round failed to exceed the threshold percentage of votes for that position. Then, in step 778, the CMC updates other organization information (if applicable), and the process returns to step 756. Alternatively, if it is determined in step 774 that the organization should not continue, the flow proceeds to step 780.

[0247] Finally, in step 780, the CMC outputs a final message containing the results of the selection mechanism. In many preferred embodiments, the results of the selection mechanism are given by the results of the round, which were determined when step 772 performed its final round. In a first preferred embodiment, the final message includes the assignment of students to schools and, if applicable, a waiting list for one or more schools. In a second preferred embodiment, the final message includes the final bid price for each product, the final processing demand for each bidder, and the payment associated with each bidder, where the payment associated with a given bidder is equal to the dot product of the final processing demand vector and the final price vector. In a third preferred embodiment, the final message is a list of winners for each position and the number of votes for each candidate. In some embodiments, the CMC outputs the final message through its network interface and transmits it over the network, and the participant computers and manager computers then receive the final message through their network interfaces and display the final message to the participants and managers through their user interfaces. In other embodiments, the final message is output only to a manager computer so that the person managing the selection mechanism can review the results before disclosing them to the participants. The process then terminates.

[0248] Figure 21A illustrates an exemplary transformation of a “request” to a “provisional selection,” as described in Figure 20. The exemplary submission in the first box of Figure 21A is an example of a submission that could be obtained by CMC, which implements a frequency auction, which itself is an embodiment of a dynamic selection mechanism. The exemplary submission reads: “I would like to bid for one block each in New York, Chicago, Baltimore-Washington, Philadelphia, and Boston, provided that bids of up to $1.15 per MHz-pop are accepted in those markets.” Note that this submission contains a “request” and not a “selection,” as this submission is uninterpretable by selection mechanism computers in the art. In contrast, today’s computer systems for selection mechanisms have a user interface that displays each item number and each item name, which provides a numeric box (i.e., a text box that accepts a number) or a dropdown for participants to indicate quantity and / or price. Therefore, a typical submission, interpretable by CMC in the art, is a set of one or more pairs of quantities and prices associated with various items.

[0249] Regarding the submissions received in the 17th round of the frequency auction, the second box in Figure 21A shows an example of the relevant associated context (this is a reduced subset of actual data from FCC auction 107, which took place from December 2020 to February 2021. The sixth column, labeled "$ per 1 MHz pop," is calculated by determining the fifth column ("Round Clock Price") and dividing it by the third column ("Population") and 20, which is the number of MHz of spectrum associated with the items in this auction. The sixth column would not actually need to be provided to the AI ​​model as context, i.e., it would suffice to provide the context that all items contain a spectrum of 20 MHz, but the sixth column is included herein for clarity). Once the context in the second box in Figure 21A is provided, the AI ​​model would be able to transform the submissions into a set of one or more provisional selections (i.e., perform step 764 or a very similar step 712). In Figure 21A, all prices are well below the $1.15 per MHz-pop threshold specified in the submission, and therefore the provisional selection shown in the third box is the appropriate round clock price for each of New York, Chicago, Baltimore-Washington, Philadelphia, and Boston.

[0250] Figure 21B shows the same exemplary submission (first box), but here in the context of the 42nd round of the same frequency auction (second box). The AI ​​model will find here that the prices for New York, Philadelphia, and Boston are still below the threshold of $1.15 per MHz pop, and therefore will select the round clock price for these three markets. It will also find that the starting price for the round for Chicago (the lowest price that can be bid in the 42nd round) is less than $1.15 per MHz pop, but the clock price for the round for Chicago is above $1.15 per MHz pop. Therefore, it will use a price of $215,434,400 (equal to $1.15 per MHz pop). Finally, it will find that the starting price for the round for Baltimore-Washington (the lowest price that can be bid in the 42nd round) is above $1.15 per MHz pop. The resulting provisional selection is shown in the third box.

[0251] It should be noted that the exemplary submissions in Figures 21A-21B serve as a highly effective “pre-bid” tool (i.e., instructions that, once entered by a participant, can then be applied in each round of the selection mechanism without modification). A participant interested in only these five markets and a price not exceeding $1.15 per MHz-pop may submit these instructions in the first round, along with a provision that they should be applied in each round. The computer system will then acquire all of these licenses desired by the participant and increase the price only as necessary. While selection mechanism computers in the art certainly exist that possess this pre-bid capability, they lack the ability to interpret plaintext requests. The significant advantages of a CMC being able to handle plaintext requests are that (1) it makes entering submissions simpler and faster for participants, (2) it provides a more flexible (and therefore more useful for participants) pre-instruction capability, and (3) it is economical in terms of software development costs because it does not require extensive custom programming to add new cases of pre-instructions.

[0252] Figure 22A demonstrates the versatility of this approach by illustrating an exemplary transformation of a richer “request” into a “provisional selection.” The exemplary submission in the first box of Figure 22A is an example of a submission that can be obtained by a CMC implementing a frequency auction, which itself is an example of a dynamic selection mechanism. The exemplary submission is: “I would like to bid on one block each for eight largest markets, where the clock price for this round does not exceed $1.15 per 1 MHz-pop.” As with Figure 21A, note that this submission contains a “request” rather than a “selection,” because this submission is uninterpretable by selection mechanism computers in the art. With respect to a submission received in the 17th round of the frequency auction, the second box of Figure 22A shows an example of the relevant associated context. This is the exact same context information as in the second box of Figure 21A.

[0253] Once given the context of the second box in Figure 22A, the AI ​​model would be able to transform the submission into a set of one or more provisional selections (i.e., perform step 764 or a very similar step 712). In Figure 22A, all prices are well below the $1.15 per 1MHz-pop threshold specified within the submission, and therefore the provisional selections shown in the third box are the appropriate round clock prices for each of the eight largest markets.

[0254] Figure 22B shows the same exemplary presentation (first box), but here in the context of the 42nd round of the same frequency auction (second box). The AI ​​model finds here that the clock prices for the rounds concerning Los Angeles, Chicago, San Francisco, and Baltimore-Washington are above the threshold of $1.15 per 1MHz-pop, and therefore would switch instead to Miami, Houston, Detroit, and Orlando (skipping the more expensive Atlanta). The resulting provisional selection is shown in the third box.

[0255] The exemplary transformations of “requests” to “provisional choices” shown in Figures 21A, 21B, 22A, and 22B are clearly not comprehensive. First, it should be noted that in the context of an auction mechanism, the process itself, as depicted in Figures 14 and 15, for bidding on interventions is an example of the transformation of requests to provisional choices. For example, if a participant's submission indicates a value for an intervention concerning “kids shoes,” the AI ​​model can certainly construct values ​​for other sets of choices, namely, “shoes for kids,” and possibly also “toddler shoes,” “kids sneakers,” “kids sandals,” and “babies first shoes.”

[0256] Furthermore, the conversion of "requests" to "provisional choices" is also useful for school choice and voting mechanisms. For example, suppose a participant in a school choice mechanism only has time to research and rank five schools out of 40 possible choices. The participant could then indicate that their first choice is school #31, their second choice is school #29, their third choice is school #5, their fourth choice is school #25, and their fifth choice is school #14 (all of which are "choices" as defined above, since the main part of the submission is directly interpretable by the CMC). In addition, participants may include the following requirement in their submission: "The schools I have ranked are my top five choices. Please rank the remaining schools as follows: Add 40% weight to each school's school system grading, 20% weight to each school's Grade 10 standardized math exam score, and 40% weight to the travel time from my home (123 Cherry Lane) to each school (closer is better)."

[0257] The latter portion of this submission is uninterpretable by selection mechanism computers in the art. Furthermore, by inserting this requirement, the participant has provided a very concise, reasonable, and customized ranking of the remaining 35 schools. If this requirement had not been provided, i.e., if only 5 schools had been ranked, the participant could have faced a serious risk of not being matched at all. Ranking all schools ensures that the participant will almost certainly be matched with one of the schools, and the exemplary requirement further increases the likelihood that the participant will be matched with a school that is both nearby and offers a comprehensive mathematics curriculum.

[0258] Here, we consider a participant in a voting mechanism who has clear preferences only for candidates for four of the most important positions. Thus, the participant's submission could be: "Vote for Robert Grey, Samantha Green, William White, and Alexandra Orange. For the remaining positions, vote for the candidates endorsed by the Washington Post." Or, the participant's submission could be: "Vote for Robert Grey, Samantha Green, William White, and Alexandra Orange. For the remaining positions, vote for the Democratic candidates." Each of these exemplary submissions consists of a first part, including the selection, and a second part, including the request. These exemplary submissions can be said to capture well the format in which many voters today cast their votes in traditional voting booths. Furthermore, in the method and system of the present invention, these voting preferences can simply be expressed in two lines of text.

[0259] Figure 23A details the process (step 720 in Figure 19) by an exemplary, non-limiting embodiment of a static school selection mechanism, by which the CMC processes selections and determines outcomes. (This figure also details the process (step 772 in Figure 20) by an exemplary, non-limiting embodiment of a dynamic school selection mechanism, by which the CMC processes selections and determines outcomes for a given round. Any differences in its application to step 772 are relatively minor and obvious to those skilled in the art, and will therefore be omitted for brevity.) In Figure 23A, the static school selection mechanism implemented is the Gale-Shapley Delayed Approval Mechanism (Gale and Shapley, 1962). The “selection” to be processed is each student’s ranking of schools. The process proceeds from step 718 to step 720a-1. In step 720a-1, for each student who has not received an "offer" held by any school, the CMC distributes the offer to the highest-ranked school that the student has not yet excluded (if any such schools remain) (in the first iteration, the CMC distributes the offer to the first-choice school of all students). The process proceeds to step 720a-2, where the CMC considers one of the schools that has not yet been considered. Continuing to step 720a-3, the CMC sorts the students who have made offers to the schools being considered in descending order of the school's official preference. In a preferred embodiment, ties are resolved using random numbers. Then, proceeding to step 720a-4, where, if a school has m slots, and n > m offers have been made to the school, the CMC "excludes" all students except the top m in the sort (i.e., students ranked lower in the school's official preference). Next, proceed to step 720a-5, where check whether all schools have been considered. If not, the process returns to step 720a-2 for another school.Otherwise, the process proceeds to step 720a-6, where it checks if there are still any unmatched students who can be offered an offer (i.e., students who have not received an offer held by any school and there remains at least one school that has not yet excluded them from its ranking). If any such students exist, the process returns to step 720a-1. Otherwise, the flow terminates at step 722. At this point, all students who have received an offer held by a given school are assigned to that school, and all students who have not received an offer held by any school are treated as unassigned.

[0260] Figure 23B details the process (step 720 in Figure 19) by an exemplary, non-limiting embodiment of a static auction mechanism, thereby enabling the CMC to process the selection and determine the outcome. In Figure 23B, the mechanism implemented is a sealed-bid multi-unit auction mechanism for homogeneous goods such as government bonds. The "selection" to be processed is one or more bids (i.e., price-quantity pairs) from each bidder. The supply of items to be allocated is indicated by S. The process proceeds from step 718 to step 720b-1. In step 720b-1, the CMC sorts the bids in descending order of price. The process continues to step 720b-2, where Q + Q(p) (defined strictly as the total demand in all bids above p) and Q(p) (defined precisely as the total demand in all bids at price p) are calculated. Next, the flow proceeds to step 720b-3, where CMC calculates Q + (P) <S≦Q +The price P is determined such that (P) + Q(P). The process then continues to step 720b-4, where the CMC determines the winning bid. In many preferred embodiments, each bid that is exactly above P is considered a winning bid, while each bid that is exactly at P is proportionally allocated such that only the proportion of [SQ*(P)] / Q(P) of that bid is considered a winning bid. The process then proceeds to step 720b-5, where the CMC determines the winning price associated with each winning bid. In some exemplary embodiments, the winning price associated with all winning bids is considered the price P determined in step 720b-3. Such embodiments are referred to as a “uniform price” auction. In other exemplary embodiments, the winning price associated with each winning bid is considered the price included in the bid. Such embodiments are referred to as a “bid-pays” auction. After step 720b-5, the flow ends in step 722.

[0261] Figure 24A details the process (step 772 in Figure 20) by an exemplary, non-limiting embodiment of a dynamic auction mechanism, thereby enabling the CMC to process the selection and determine the outcome of the round. In Figure 24A, the mechanism implemented is an ascending clock auction mechanism for one or more “products” (where “product” is defined as “type” of item). The “selection” by each bidder processed is a bid (i.e., a price-quantity pair) for one or more products. The price can be the round's opening price, the clock price, or any price in between (“in-round bid”). The quantity supplied for a given product is denoted by S. The “price point” is the percentage of distance the bid price is between the round's opening price and the clock price. Specifically, the price point, or the bid price, associated with the bid, is equal to the following ratio: [bid price - round's opening price] / [clock price - round's opening price].

[0262] The process proceeds from step 770 to step 772c-1. In step 772c-1, the CMC adds missing bids, adds a random number to each bid, and calculates a price point associated with each bid. For each product in which a bidder had positive processed demand in the previous round, if the bidder did not submit a bid for that product during the current round, the CMC will add a “missed bid” for that bidder, along with a zero amount in the round’s opening price. The random number is generated from a pseudo-random number generator on the CMC, and the price point is calculated as the above ratio. The process continues to step 772c-2, where the CMC applies all bids from all bidders and maintains the processed demand from the previous round at clock prices. Next, in step 772c-3, the CMC sorts the remaining bids (i.e., bids that have not yet been applied) in ascending order of price points and descending order of random numbers. When the CMC first reaches step 772c-5, it will start from the top by considering the first bid in the sort order, if applicable. In subsequent iterations of step 772c-5, the CMC will sequentially consider subsequent bids in the sort order. The flow proceeds to step 772c-4, which is a command flow statement that branches based on whether there are any further bids remaining to consider. If there are no further bids to consider, the process jumps to step 772c-15. Otherwise, the flow continues to step 772c-5, where the CMC considers the next bids and applies as many bids as possible to those considered, according to constraints. Typical constraints, in exemplary embodiments, include that a bid that decreases quantity is applied only if the total demand does not reduce the quantity to less than the supply quantity S, and that a bid that increases quantity is applied only if the bidder's processed activity does not exceed the bidder's eligibility for the round. The flow then proceeds to step 772c-6, which branches based on whether the bid under consideration has been fully applied. If it has been fully applied, the flow jumps to step 772c-9.Otherwise, the process proceeds to step 772c-7, where the CMC adds the portion of the bid that was not applied (which in some situations would be the entire bid) to the "exclusion queue". The flow then proceeds to step 772c-8, which branches based on whether the bid under consideration was partially applied. If partially applied, the flow proceeds to step 772c-9; if not applied at all, it returns to step 772c-4.

[0263] In step 772c-9, the CMC sorts the exclusion queue in ascending order of price points and in descending order of random numbers. When the CMC next reaches step 772c-11, it will start from the top by considering the first bid in the sorted order of the exclusion queue, if applicable. In subsequent iterations of step 772c-11, the CMC will sequentially consider subsequent bids in the sorted order. The flow proceeds to step 772c-10, which is a command flow statement that branches based on whether there are any further bids to consider in the exclusion queue. If there are no further bids to consider, the process returns to step 772c-4. Otherwise, the flow continues to step 772c-11, where the CMC considers the next bids in the exclusion queue and applies as many bids as possible that are considered, according to the constraints. Typical constraints, in exemplary embodiments, include that a bid to decrease quantity is only applicable if total demand does not reduce the quantity to less than supply quantity S, and that a bid to increase quantity is only applicable if the bidder's processed activity does not exceed the bidder's eligibility for the round. The flow then proceeds to step 772c-12, which branches based on whether the bid under consideration was fully applied. If fully applied, the bid is removed from the exclusion queue, and the flow returns to step 772c-9. Otherwise, it proceeds to step 772c-13, where the CMC leaves in the exclusion queue any portion of the bid that was not applied (which may be the entire bid in some cases). The flow then proceeds to step 772c-14, which branches based on whether the bid under consideration was partially applied. If partially applied, the flow returns to step 772c-9; if not applied at all, it returns to step 772c-10.

[0264] In step 772c-15, the CMC determines the “Posted Price” for each product based on the processed demand and bids applied in steps 772c-5 and 772c-11. The term “Processed Demand” refers to the demand of a given bidder after all iterations of steps 772c-5 and 772c-11, and the term “Total Demand” for a product refers to the processed demand, summed up across all bidders. If total demand exceeds the quantity supplied for a product, the Posted Price is equal to the clock price for the round. If total demand is equal to the quantity supplied, and at least one bid to reduce the demand for a product was applied (either completely or partially) in step 772c-5 or 772c-11, the Posted Price is equal to the highest bid price of all applied bids to reduce the demand for the product (either completely or partially). In other words, the Posted Price is the price at which the reduction occurred so that total demand equals the quantity supplied. In all other cases, the posted price is equal to the opening price for the round (i.e., the posted price for the previous round). After step 772c-15, the flow ends in step 774.

[0265] Figure 24B details the process (step 772 in Figure 20) by an exemplary, non-limiting embodiment of a dynamic voting mechanism, by which the CMC processes the selection and determines the outcome of the round (this figure also details the process (step 720 in Figure 19) by an exemplary, non-limiting embodiment of a static voting mechanism, by which the CMC processes the selection and determines the outcome. Any differences in its application to step 720 are relatively minor and obvious to those skilled in the art, and will therefore be omitted for brevity). In Figure 24B, the mechanism implemented is a runoff selection mechanism. The “selection” to be processed is the vote by each voter for one or more candidates. The process proceeds from step 770 to step 772d-1. In step 772d-1, the CMC distributes the votes to the individual candidates running for each position. The process continues to step 772d-2, which is a command flow statement that branches based on whether it is the first or second voting round. If it is the first voting round, the flow proceeds to step 772d-3, where the maximum number of votes for each position, the associated candidate, and the total number of votes for each position are identified. Next, in step 772d-4, the CMC considers positions that have not yet been considered. The flow continues to step 772d-5, which is a command flow statement that branches based on whether the maximum number of votes divided by the total number of votes exceeds the threshold percentage of votes required for that position. If the maximum number of votes divided by the total number of votes exceeds the threshold percentage of votes, the flow proceeds to step 772d-6, where the associated candidate is considered the winner for that position. Otherwise, the flow proceeds to step 772d-7, where the second highest number of votes and associated candidates for the position are identified, and the two candidates associated with the highest and second highest number of votes are deemed to advance to the second round. Following steps 772d-6 and 772d-7, the process proceeds to step 772d-8, which is a command flow statement that branches based on whether all positions have been considered.If all positions have not been considered, the process returns to step 772d-4. If all positions have been considered, the flow ends at step 774.

[0266] If it is the second voting round, the flow proceeds to step 772d-9, where the maximum number of votes for each position and the associated candidate are identified. Next, in step 772d-10, the candidate associated with the maximum number of votes for each position is deemed the winner for that position. After step 772d-10, the flow terminates in step 774.

[0267] Figure 25 is a block diagram illustrating a device according to several embodiments. As shown in Figure 25, device 2500 may comprise a data processing system (DPS) 2502, which may include one or more processors 2555 (e.g., one or more other data processing circuits such as general-purpose microprocessors and / or application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and equivalents); a network interface 2503 for use in connecting device 2500 to a network 2520; and a local storage unit (known as a “data storage system”) 2506, which may include one or more non-volatile storage devices and / or one or more volatile storage devices (e.g., random-access memory (RAM)). In embodiments in which device 2500 includes a general-purpose microprocessor, a computer program product (CPP) 2533 may be provided. The CPP 2533 includes a computer-readable medium (CRM) 2542 for storing a computer program (CP) 2543, which comprises computer-readable instructions (CRIs) 2544. CRM2542 may be a non-transient computer-readable medium such as a magnetic medium (e.g., a hard disk), an optical medium (e.g., a DVD), a memory device (e.g., a random access memory), and equivalents, but is not limited to these. In some embodiments, the CRI2544 of the computer program 2543, when executed by the data processing system 2502, is configured to cause the device 2500 to perform the steps described above (e.g., the steps described above with reference to a flowchart). In other embodiments, the device 2500 may be configured to perform the steps described herein without requiring code. That is, for example, the data processing system 2502 may simply consist of one or more ASICs. Thus, features of the embodiments described herein may be implemented in hardware and / or software.

[0268] Figure 26 illustrates a method according to several embodiments. Method 2600 is for intervention in an artificial intelligence (AI) model. Step s2610 of the method includes obtaining a request from a user computer. Step s2620 of the method includes obtaining intervention information applicable to the request. Step s2630 of the method includes generating an augmentation request based on the obtained request and the obtained intervention information. Step s2640 of the method includes providing the augmentation request as input to the AI ​​model. Step s2650 of the method includes obtaining a response to the augmentation request from the AI ​​model. Step s2660 of the method includes sending the obtained response to the user computer.

[0269] Figure 27 illustrates a method according to several embodiments. Method 2700 is for intervention in an artificial intelligence (AI) model. Step s2710 includes the step of obtaining intervention information from one or more stakeholder computers. Step s2720 includes the step of creating a training set based on the obtained intervention information. Step s2730 includes the step of training the AI ​​model on the created training set. In some embodiments, the training step may include the step of pre-training and / or fine-tuning the AI ​​model. Step s2740 of the method includes the step of obtaining a request from a user computer. Step s2750 of the method includes the step of obtaining a response to the request from the trained AI model. Step s2760 of the method includes the step of transmitting the obtained response to the user computer.

[0270] Figure 28 illustrates a method according to several embodiments. Method 2800 utilizes an artificial intelligence (AI) model to facilitate a selection mechanism among multiple participants. Step s2810 of the method includes obtaining submissions from a first participant among multiple participants. Step s2820 of the method includes transforming the submissions from the first participant into a first set of one or more selections for the selection mechanism using the AI ​​model.

[0271] Figure 29 illustrates a method according to several embodiments. Method 2900 is for intervention in an artificial intelligence model. Step s2910 of the method includes transmitting a request to an artificial intelligence (AI) search system comprising an AI model. Step s2920 of the method includes receiving a response from the AI ​​search system, the response comprising a first portion that receives at least one intervention and a second portion that does not receive an intervention, and an indicator is applied to the first portion.

[0272] Figure 30 illustrates a method according to several embodiments. Method 3000 is for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model. Step s3010 of the method includes obtaining a submission from a user. Step s3020 of the method includes transmitting the submission to an AI model. Step s3030 of the method includes obtaining a response from the AI ​​model, which includes one or more sets of selections for the selection mechanism, and the AI ​​model converts the submission into one or more sets of selections.

[0273] While at least one exemplary embodiment is presented in the preceding detailed description, it should be understood that a wide number of variations also exist. Furthermore, it should be understood that the exemplary embodiments or configurations described herein are merely examples and are not intended to limit in any way the scope, availability, or configuration of the claimed subject matter. Rather, the preceding detailed description will provide those skilled in the art with a convenient roadmap for implementing the described embodiments or configurations. It should be understood that various modifications can be made to the function and arrangement of the elements without departing from the scope of the invention as described in the appended claims, including equivalents and predictable equivalents known at the time of filing this patent application.

[0274] References

[0275] Abdulkadiroglu, A. and T. Sonmez (2003), “School Choice: A Mechanism Design Approach,” American Economic Review, Vol. 93, pp. 729-747.

[0276] Ausubel, L. (2000), “System and Method for an Efficient Dynamic Auction for Multiple Objects,” US Patent No. 6,026,383.

[0277] Bouton, L. (2013), “A Theory of Strategic Voting in Runoff Elections,” American Economic Review, Vol. 103, pp. 1248-1288.

[0278] Chan, W., S. Agarwal, R. Ranganath, L.R. Wiseman, and D. Jindal (2022), “Method and System for Mining Image Searches to Associate Images with Concepts,” US Patent No. 11,409,812.

[0279] Chen, Y. and O. Kesten (2017), “Chinese College Admissions and School Choice Reforms: A Theoretical Analysis,” Journal of Political Economy, Vol. 125, pp. 99-139.

[0280] Davis, D., M. Derer, J. Garcia, L. Greco, T. Kurt, T. Kwong, J. Lee, K. Lee, P. Pfarner, and S. Skovran (2001), “System and Method for influencing a Position on a Search Result List Generated by a Computer Network Search Engine,” U.S. Patent No. 6,269,361.

[0281] Edelman, B., M. Ostrovsky, and M. Schwarz (2007) “Internet Advertising and the Generalized Second Price Auction: Selling Billions of Dollars Worth of Keywords,” American Economic Review, Vol. 97, pp. 242-259.

[0282] Fain, D. and J. Pedersen (2006) “Sponsored Search: A Brief History,” Bulletin of the American Society for information Science and Technology, pp. 12-13.

[0283] Fallows, D. (2005), “Search Engine Users,” Pew internet & American Life Project of the Pew Research Center.

[0284] Federal Trade Commission, “Blurred Lines: An Exploration of Consumers’ Advertising Recognition in the Contexts of Search Engines and Native Advertising,” FTC Staff Report, Dec. 2017.

[0285] Gale, D. and L. Shapley (1962), “College Admissions and the Stability of Marriage,” American Mathematical Monthly, Vol. 69, pp. 9-15.

[0286] Grant, N., “Google Devising Radical Search Changes to Beat Back A.I. Rivals,” The New York Times, Apr. 16, 2023.

[0287] Grant, N. and C. Metz, “A New Chat Bot Is a ‘Code Red’ for Google’s Search Business,” The New York Times, Dec. 21, 2022.

[0288] Hansen, E. (2002), “FTC Wants Paid Search to Shape Up,” CNET, Aug. 2, 2002.

[0289] Jansen, B. and T. Mullen (2008), “Sponsored Search: An Overview of the Concept, History, and Technology,” int. J. Electronic Business, Vol. 6, No. 2.

[0290] Komo, A. and L. Ausubel (2020), “System and Method for Cryptographic Choice Mechanisms,” US Patent No. 10,872,487.

[0291]

Chem.

[0292] Tibken, S. (2013), “FTC to Search Engines: Make it Clear Which Results are Ads,” CNET, June 25, 2013.

[0293] Varian, H. (2007), “Position Auctions,” international Journal of industrial Organization, Vol. 25, pp. 1163 - 1178.

[0294] Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin (2017), “Attention is All You Need,” Proceedings of the 31 st international Conference on Neural information Processing Systems (NIPS ’17). URL https: / / arxiv.org / abs / 1706.03762.

Claims

1. A computer implementation method (2600) for intervention in an artificial intelligence (AI) model, wherein the method is Obtaining a request from the user computer (30, 660) (s2610), To obtain intervention information applicable to the above request (s2620), Based on the acquired request and the acquired intervention information, an extended request is generated (s2630), The extended request is provided as input to the AI ​​model (s2640), Obtaining a response to the aforementioned extension request from the AI ​​model (s2650), The acquired response is transmitted to the user computer (s2660) Methods that include...

2. Obtaining the aforementioned intervention information further means To obtain one or more keywords or concepts associated with the aforementioned request, Using one or more of the aforementioned keywords or concepts, query the database (50) of intervention information, In response to the aforementioned query, the intervention information is obtained. The method according to claim 1, including the method described in claim 1.

3. The intervention information is obtained from one or more stakeholder computers (70), The database (50) is updated using the intervention information obtained above. The method according to any one of claims 1 and 2, further comprising:

4. The method according to any one of claims 1-3, wherein the intervention information includes ratings or comments.

5. The method according to any one of claims 1-4, wherein the intervention information is based on or includes at least one rating or comment received from an interested party.

6. The intervention information, including a bid, according to the method of any one of claims 1-3.

7. The method according to any one of claims 1-3 or 6, wherein the intervention information is based on or includes at least one bid received from an interested party.

8. The method according to any one of claims 1 to 7, wherein the first weight is associated with the intervention information, and the first weight indicates the amount by which the intervention information should be weighted by the AI ​​model.

9. The method according to claim 8, further comprising obtaining the first weight from a database (50).

10. The method according to any one of claims 8-9, further comprising incorporating the first weight into the extension requirement.

11. The method according to claim 10, further comprising incorporating a second weighting of organic information contained within the AI ​​model into the extension request, wherein the organic information includes information available to the AI ​​model in response to the acquired request, without the intervention information.

12. The method according to claim 11, wherein the first load and the second load are incorporated into the extension requirement as a convex coupling of the first load and the second load.

13. The above request is provided to a second AI model, the second AI model being fine-tuned with respect to the intervention information, From the second AI model described above, the intervention information applicable to the acquired request is obtained. The method according to any one of claims 1 to 12, further comprising:

14. Identifying a first part of the response that includes options associated with the intervention information, and a second part of the response that includes options not associated with the intervention information, Before transmitting the response to the user computer, apply the first indicator to the first part of the response and the second indicator to the second part of the response. The method according to any one of claims 1 to 13, further comprising:

15. The first sign is, A first color different from the second color used in the second sign, A first typeface different from the second typeface used in the second sign, A first symbol different from the second symbol used in the second sign, or A first text character different from the second text character used in the second sign. The method according to claim 14, comprising at least one of the following.

16. The method according to any one of claims 1 to 15, wherein the AI ​​model is a large-scale language model.

17. The method according to any one of claims 1-16, wherein the intervention information corresponds to an optional independent third-party assessment.

18. The method according to any one of claims 1-17, further comprising masking the extension request from the user computer.

19. The method according to any one of claims 1-18, wherein the intervention information is associated with stakeholders and options.

20. A computer implementation method (2700) for intervention in an artificial intelligence (AI) model, wherein the method is The intervention information is obtained from one or more stakeholder computers (70) (s2710), Based on the intervention information obtained, a training set is created (s2720), Training the AI ​​model on the training set created (s2730), Obtaining a request from the user computer (30, 660) (s2740), Obtaining a response to the aforementioned request from the trained AI model (s2750), The acquired response is transmitted to the user computer (s2760) Methods that include...

21. The method according to claim 20, further comprising updating a database (50) using the acquired intervention information.

22. The method according to any one of claims 20-21, wherein the acquired intervention information includes ratings or comments.

23. The method according to any one of claims 20-22, wherein the acquired intervention information is based on or includes at least one rating or comment.

24. The method according to any one of claims 20-21, wherein the acquired intervention information includes a bid.

25. The method according to any one of claims 20-21 or 24, wherein the acquired intervention information is based on or includes at least one bid received from the interested party computer (70).

26. Identifying a first part of the response that includes options associated with the intervention information, and a second part of the response that includes options not associated with the intervention information, Before transmitting the response to the user computer, apply the first indicator to the first part of the response and the second indicator to the second part of the response. The method according to any one of claims 20-25, further comprising:

27. The first sign is, A first color different from the second color used in the second sign, A first typeface different from the second typeface used in the second sign, A first symbol different from the second symbol used in the second sign, or A first text character different from the second text character used in the second sign. The method according to claim 26, comprising at least one of the following.

28. The method according to any one of claims 20-27, wherein the AI ​​model is a large-scale language model.

29. The method according to any one of claims 20-28, wherein the intervention information corresponds to an optional independent third-party assessment.

30. The method according to any one of claims 20-29, wherein the intervention information is associated with an interested party computer (70) and options.

31. A computer implementation method (2800) for facilitating a selection mechanism among multiple participants (610a-m) using an artificial intelligence (AI) model, wherein the method is: Obtaining submissions from the first participant of the aforementioned group of participants (s2810), Convert the aforementioned submission from the first participant into a first set of one or more selections for the selection mechanism using an AI model (s2820) Methods that include...

32. Obtaining submissions or selections from other participants different from the aforementioned first participant, The outcome of the selection mechanism is determined based on the first set of one or more selections and the submissions or selections obtained from the other participants. The method according to claim 31, further comprising:

33. The method according to any one of claims 31-32, further comprising training the AI ​​model and generating selections.

34. The method according to any one of claims 31-33, wherein the training includes at least one of pre-training or fine-tuning the AI ​​model on one or more exemplary or actual submissions.

35. The search request is obtained from the user computer (30, 660), Selecting one or more choices from the aforementioned first set of one or more choices, Based on one or more of the selected choices, the search request is expanded using intervention information. The method according to any one of claims 31-34, further comprising:

36. The method of claim 35, further comprising converting each of the one or more selected choices into an independent third-party assessment, wherein the intervention information is based on or includes the independent third-party assessment.

37. The method according to claim 36, wherein the independent third-party evaluation corresponds to at least one of a hotel, restaurant, venue, shop, or commercial facility.

38. The method according to any one of claims 35-37, further comprising charging the first participant the associated selection amount for each of the one or more selections selected.

39. Transmitting the first set of one or more selections to the first participant, Obtaining a response from the first participant indicating approval of the first set of one or more choices or modification of the first set of one or more choices The method according to any one of claims 31-38, further comprising:

40. The method according to claim 39, further comprising training the AI ​​model using the modifications in response to a decision that the response from the first stakeholder indicates modifications to the set of selections.

41. Obtaining submissions or selections from other participants different from the aforementioned first participant, Implement the auction on the selected set of one or more selections and the submitted or selected selections from other participants, The allocation will be determined based on the results of the aforementioned auction. The method according to any one of claims 31-40, further comprising:

42. The method according to claim 41, wherein the auction is a generalized second-price auction.

43. A computer implementation method for intervention in an artificial intelligence model (2900), wherein the method is Transmitting a request to an artificial intelligence (AI) search system (10,640) equipped with an AI model (s2910), Receiving a response from the AI ​​search system (s2920), wherein the response includes a first part that receives at least one intervention and a second part that does not receive an intervention, and the marker is applied to the first part. Methods that include...

44. The aforementioned sign is, A first color different from the second color used in the second portion of the response, A first typeface different from the second typeface used in the second portion of the response, A first symbol different from the second symbol used in the second part of the response, or A first text character different from the second text character used in the second portion of the response. The method according to claim 43, comprising at least one of the following.

45. A computer implementation method (3000) for facilitating a selection mechanism among multiple participants (610a-m) using an artificial intelligence (AI) model, wherein the method is: Obtaining submissions from users (s3010), The above submission is transmitted to the AI ​​model (s3020), Obtaining a response from the AI ​​model that includes one or more sets of selections for a selection mechanism (s3030), wherein the AI ​​model converts the submission into the one or more sets of selections. Methods that include...

46. Obtaining feedback from the user indicating approval of the set of one or more selections or modification of the set of one or more selections, The feedback is transmitted to the AI ​​model. The method according to claim 45, further comprising:

47. The method according to any one of claims 45-46, further comprising receiving an allocation based on the outcome of an auction based on the set of one or more selections.

48. The method according to claim 47, wherein the auction is a generalized second-price auction.

49. A computer implementation method for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model, wherein the method is: The selection of the first set is obtained from the first participant of the aforementioned group of participants, Obtaining search requests from user computers, Selecting one or more choices from the aforementioned first set of one or more choices, Based on the one or more selections mentioned above, the search request is expanded using intervention information. To provide the aforementioned extended search request to the AI ​​model Methods that include...

50. The method according to any one of claims 31-42 or 45-49, wherein the selection mechanism is a static selection mechanism.

51. The method according to any one of claims 31-42 or 45-49, wherein the selection mechanism is a dynamic selection mechanism.

52. The method according to any one of claims 31-42 or 45-51, further comprising applying constraints to selections and inputting only those selections that satisfy the constraints.

53. The aforementioned selection mechanism is School Choice Organization auction house, or voting mechanism The method according to any one of claims 31-42 or 45-52, comprising one of the above.

54. A device (2500), Processing circuit network (2502), A memory (2506) connected to the aforementioned processing network and A device comprising, wherein the device is configured to carry out the method according to any one of claims 1-53.

55. A computer program (2543) comprising an instruction (2544), wherein the instruction (2544), when executed by a processing circuit (2502) of a device (2500), causes the device to perform the method according to any one of claims 1 to 53.

56. A computer system for intervention in an artificial intelligence (AI) model, wherein the computer system comprises at least one computer, A first acquisition means for obtaining a request from a user computer, A second acquisition means for obtaining intervention information applicable to the aforementioned request, A generation means for generating an extended request based on the acquired request and the acquired intervention information, A first providing means for providing the aforementioned extension request as input to the AI ​​model, A third acquisition means for obtaining a response to the aforementioned extension request from the AI ​​model, A transmission means for sending the acquired response to the user computer, A system equipped with these features.

57. Obtaining the aforementioned intervention information further means To obtain one or more keywords or concepts associated with the aforementioned request, Querying a database of intervention information using one or more of the aforementioned keywords or concepts, In response to the aforementioned query, the intervention information is obtained. The system according to claim 56, including the system described in claim 56.

58. A fourth means of obtaining intervention information from one or more stakeholder computers, An update means for updating the database using the aforementioned acquired intervention information and The system according to any one of claims 56-57, further comprising the above.

59. The system according to any one of claims 56-58, wherein the intervention information includes ratings or comments.

60. The system according to any one of claims 56-59, wherein the intervention information is based on or includes at least one rating or comment received from an interested party.

61. The intervention information includes the bidding process, as described in any one of claims 56-58.

62. The system according to any one of claims 56-59 or 61, wherein the intervention information is based on or includes at least one bid received from an interested party.

63. The system according to any one of claims 56-62, wherein a first weight is associated with the intervention information, and the first weight indicates the amount by which the intervention information should be weighted by the AI ​​model.

64. The system according to claim 63, further comprising a fifth acquisition means for obtaining the first weight from a database.

65. The system according to any one of claims 63-64, further comprising a first integration means for incorporating the first weight into the extension requirement.

66. The system according to claim 65, further comprising a second incorporation means for incorporating a second weighting of organic information contained within the AI ​​model into the extension request, wherein the organic information includes information available to the AI ​​model in response to the acquired request, without the intervention information.

67. The system according to claim 66, wherein the first load and the second load are incorporated into the extension requirement as a convex coupling of the first load and the second load.

68. A second providing means for providing the aforementioned request to a second AI model, wherein the second AI model is fine-tuned with respect to intervention information, and the second providing means A sixth acquisition means for obtaining the intervention information applicable to the acquired request from the second AI model, The system according to any one of claims 56-67, further comprising the above.

69. Identification means for distinguishing a first part of the response that includes options associated with the intervention information, and a second part of the response that includes options not associated with the intervention information, Before transmitting the response to the user computer, an application means for applying a first indicator to the first part of the response and a second indicator to the second part of the response. The system according to any one of claims 56-68, further comprising the above.

70. The first sign is, A first color different from the second color used in the second sign, A first typeface different from the second typeface used in the second sign, A first symbol different from the second symbol used in the second sign, or A first text character different from the second text character used in the second sign. The system according to claim 69, comprising at least one of the following.

71. The system according to any one of claims 56-70, wherein the AI ​​model is a large-scale language model.

72. The intervention information corresponds to an optional independent third-party assessment, as per any one of claims 56-71.

73. The system according to any one of claims 56-72, further comprising masking means for masking the extension request from the user computer.

74. The system according to any one of claims 56-73, wherein the intervention information is associated with stakeholders and options.

75. A computer system for intervention in an artificial intelligence (AI) model, wherein the computer system comprises at least one computer, A first acquisition means for obtaining intervention information from one or more stakeholder computers, Based on the acquired intervention information, a means for creating a training set, Training means for training the AI ​​model on the created training set, A second means for obtaining requests from the user computer, A third acquisition means for obtaining a response to the aforementioned request from the trained AI model, A transmission means for sending the acquired response to the user computer. A system equipped with these features.

76. The system according to claim 75, further comprising update means for updating a database using the acquired intervention information.

77. The system according to any one of claims 75-76, wherein the acquired intervention information includes ratings or comments.

78. The system according to any one of claims 75-77, wherein the acquired intervention information is based on or includes at least one rating or comment.

79. The system according to any one of claims 75-76, including the acquisition of the intervention information, including the bidding process.

80. The system according to any one of claims 75-76 or 79, wherein the acquired intervention information is based on, or includes, at least one bid received from an interested party computer.

81. Identification means for distinguishing a first part of the response that includes options associated with the intervention information, and a second part of the response that includes options not associated with the intervention information, Before transmitting the response to the user computer, an application means for applying a first indicator to the first part of the response and a second indicator to the second part of the response. The system according to any one of claims 75-80, further comprising the above.

82. The first sign is, A first color different from the second color used in the second sign, A first typeface different from the second typeface used in the second sign, A first symbol different from the second symbol used in the second sign, or A first text character different from the second text character used in the second sign. The system according to claim 81, comprising at least one of the following.

83. The system according to any one of claims 75-82, wherein the AI ​​model is a large-scale language model.

84. The intervention information corresponds to an optional independent third-party assessment, as described in any one of claims 75-83.

85. The system according to any one of claims 75-84, wherein the intervention information is associated with an interested party computer and options.

86. A computer system for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model, wherein the computer system comprises at least one computer, and the computer system is A first acquisition means for obtaining submissions from the first participant of the aforementioned plurality of participants, A transformation means for transforming the aforementioned submission from the first participant into a first set of one or more selections for a selection mechanism using an AI model. A system equipped with these features.

87. A second means of obtaining submissions or selections from other participants different from the first participant, A first determination means for determining the outcome of the selection mechanism based on the first set of one or more selections and the submissions or selections obtained from the other participants. The system according to claim 86, further comprising:

88. The system according to any one of claims 86-87, further comprising a first training means for training the AI ​​model and generating selections.

89. The system according to any one of claims 86-88, wherein the training includes at least one of pre-training or fine-tuning the AI ​​model on one or more exemplary or actual submissions.

90. A third means for obtaining search requests from the user's computer, A selection means for selecting one or more choices from the first set of one or more choices, An extension means for extending the search request using intervention information based on one or more of the selected choices, The system according to any one of claims 86-89, further comprising the above.

91. The system according to claim 90, further comprising conversion means for converting each of the one or more selected selections into an independent third-party rating, wherein the intervention information is based on or includes the independent third-party rating.

92. The system according to claim 91, wherein the independent third-party evaluation corresponds to at least one of a hotel, restaurant, venue, shop, or commercial facility.

93. The system according to any one of claims 90-92, further comprising a billing means for billing the first participant an amount associated with each of the one or more selected choices.

94. A transmission means for transmitting the first set of one or more selections to the first participant, A fourth acquisition means for obtaining a response from the first participant indicating approval of the first set of one or more selections or modification to the first set of one or more selections. The system according to any one of claims 86-93, further comprising the above.

95. The system according to claim 94, further comprising a second training means for training the AI ​​model using the modifications in response to a decision that the response from the first stakeholder indicates modifications to the set of selections.

96. A fifth means of obtaining submissions or selections from other participants different from the first participant, Implementation means for implementing the auction on the selected set of one or more selections and the submitted or selected items from other participants, Based on the results of the aforementioned auction, a second decision-making means for determining the allocation and The system according to any one of claims 86-95, further comprising the above.

97. The system according to claim 96, wherein the auction is a generalized second-price auction.

98. A computer system for intervention in an artificial intelligence model, wherein the computer system comprises at least one computer, A transmission means for transmitting requests to an artificial intelligence (AI) search system equipped with an AI model, Receiving means for receiving a response from the AI ​​search system, wherein the response includes a first part that receives at least one intervention and a second part that does not receive an intervention, and a mark is applied to the first part, A system equipped with these features.

99. The aforementioned sign is, A first color different from the second color used in the second portion of the response, A first typeface different from the second typeface used in the second portion of the response, A first symbol different from the second symbol used in the second part of the response, or A first text character different from the second text character used in the second portion of the response. The system according to claim 98, comprising at least one of the following.

100. A computer system for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model, wherein the computer system comprises at least one computer. A first means of obtaining submissions from users, A first transmission means for transmitting the aforementioned submission to an AI model, A second acquisition means for obtaining a response from the AI ​​model that includes one or more sets of selections for a selection mechanism, wherein the AI ​​model converts the submission into the one or more sets of selections. A system equipped with these features.

101. A third acquisition means for obtaining feedback from the user indicating approval of the set of one or more selections or modification of the set of one or more selections, A second transmission means for transmitting the feedback to the AI ​​model and The system according to claim 100, further comprising the above.

102. The system according to any one of claims 100-101, further comprising receiving means for receiving an allocation based on the outcome of an auction based on the set of one or more selections.

103. The system according to claim 102, wherein the auction is a generalized second-price auction.

104. A computer system for facilitating a selection mechanism among multiple participants using an artificial intelligence (AI) model, wherein the computer system comprises at least one computer. A first acquisition means for obtaining the selection of a first set from the first participant of the plurality of participants, A second means for obtaining search requests from the user's computer, A selection means for selecting one or more choices from the first set of one or more choices, An extension means for extending the search request using intervention information based on one or more of the selected choices, A means for providing the aforementioned extended search request to an AI model A system equipped with these features.

105. The system according to any one of claims 86-97 or 100-104, wherein the selection mechanism is a static selection mechanism.

106. The system according to any one of claims 86-97 or 100-104, wherein the selection mechanism is a dynamic selection mechanism.

107. The system according to any one of claims 86-97 or 100-106, further comprising means for applying constraints to selections and inputting only those selections that satisfy the constraints.

108. The aforementioned selection mechanism is School Choice Organization auction house, or voting mechanism A system according to any one of claims 86-97 or 100-107, comprising one of the above.

Images