Artificial intelligence interview system and related methods

Abstract

A method for conducting interviews using an artificial intelligence (AI) system may include a. obtaining training data from one or more databases, b. fine-tuning a base AI model of the AI interview system using the obtained training data, c. generating and outputting, to an interviewee, one or more interview questions for an interview based on the executed fine-tuned AI model, d. receiving answers, from the interviewee, to the output questions, e. generating prompt-response pairs, executing the fine-tuned AI model using the received answers, f. generating and outputting, to the interviewee, one or more additional interview questions for the interview based on the executed fine-tuned AI model; g. repeating steps e. and f. until the interview ends; and h. generating an updated fine-tuned AI model for use in future interviews.

Description

ARTIFICIAL INTELLIGENCE INTERVIEW SYSTEM AND RELATED METHODSCross Reference to Related Application

[0001] The application claims benefit of priority from U.S. Provisional Patent Application No. 63 / 627,852, filed February 1 , 2024, which is hereby incorporated by reference in its entirety.Technical Field

[0002] The present disclosure generally relates to the fields of artificial intelligence (Al), computational speech and language processing, and interview techniques from the social sciences. More particularly, the invention relates to systems and methods fortraining and using large language models to conduct generative tasks that mimic human language and interviewing techniques, producing an Al Interview System that conducts interviews with human interviewees on a particular topic of interest.Background

[0003] Al refers to a wide range of computer programs that learn from experience to complete a task. Al models learn when their performance in a given task improves with more experience. Interviews are a qualitative method in the social sciences where researchers conduct thorough, open-ended interviews to gather information about an individual’s experiences, perceptions, and interpretations. These interviews provide extraordinarily rich information about an individual, but a human interviewer can only conduct one interview at time. This makes it difficult to scale interviews to achieve a representative understanding of a larger group of people. In view of the foregoing, there is a need for an improved solution for conducting large-scale studiesof populations using qualitative interviewing techniques, such as structured or semistructured approaches, among others. In particular, there is a need for systems and methods for effective qualitative interviewing at scale.Summary

[0004] The present disclosure provides methods and systems for conducting qualitative interviews at scale by using a series of Al models configured to mimic the techniques that social scientists use in the process of conducting an interview. By virtue of the present invention, an Al Interview System may conduct hundreds or thousands of interviews on a set of hundreds or thousands of people (e.g., of a population) in a small amount of time, resulting in a set of rich data.

[0005] Generating interview data to teach an Al model how to conduct interviews that conform to best practices of social science is one component of the overall system. When available, transcripts of previously completed interviews recorded as part of social science research serve as one source of high-quality training data. An alternative source are interview transcripts generated using a human-in-the-loop system. This involves two Al-models - one that generates a list of options for messages from the interviewer and one that generates a list of options as responses from an interviewee. The social scientist interviewer at each step in the interview can either choose one of the Al-generated messages or enter their own text. This human-machine teaming can create quality interview transcripts in a relatively shorter amount of time as compared to the creation of interview transcripts using human-to-human interviews, and can be useful to provide an Al model with basic examples of interviews. Another source of interview data can originate from social scientists interviewing humans over a chat interface. These human-to-humaninterviews — which may follow best practices from the fields of social science — can provide examples of expert-level interviewing techniques that may arise as the social scientist addresses the spontaneity and eccentricities that human interviews often exhibit. Together, a large and diverse body of training data provides an Al model with sufficient experience to capably generate interview questions dynamically throughout an interview with an interviewee in a wide range of circumstances, including, e.g., when interviewees respond to questions with one-word answers, argue with the Al Interview System, or ask questions far outside the scope of the interview.

[0006] Interview transcripts without further enrichment are not suitable for use in training an Al model to conduct interviews, since they lack key pieces of context as well as an instruction for an Al model to follow. Specifically, a method for training an Al model of an Al Interview System to conduct high-quality interviews may include training data that may include, for each prompt-response pair in an interview transcript, (i) a context that generally describes the set of topics the interviewer should cover, (ii) an instruction for the Al Interview System, (iii) information about the interviewee, (iv) the overall goals of the interview, (v) a summary of what data has thus far been collected in the interview, and (vi) the transcript of the interview up to that point.

[0007] A method of using the Al interview system to conduct an interview is created by incorporating these pieces of contextual information and instructions into the prompt-response pairs contained in an interview transcript result in a dataset that provides an Al Interview System with a set of examples in which the inputs — interview summary, interview goals, incremental interview history, and transcript- are matched with the outputs — the question the social scientist asked at that point in the interview.

[0008] This sequence of inputs and outputs establishes an Al Interview System and provides the experiences required by an Al Interview System to learn how to ask questions at each question and answer pairing in ways that mirror an expertly trained social scientist. This enrichment process can be applied to any interview transcript, provided that the goals of the interview and biographic information about the interviewee are available or readily apparent from the transcript.

[0009] The training data created through the training process may be used to adjust the parameters of a pre-trained Large Language Model in a process commonly referred to as fine-tuning. Fine-tuning algorithms allow a general LLM to undergo additional training to become highly proficient at a specific task by optimizing over a loss function based on the inputs and outputs in the training dataset.

[0010] One component of an Al Interview System is the way in which it generates instructions, or prompts, for the base large language model to follow when generating the next question in an interview. This is relevant both to decisions about how to enrich interview transcripts as described above and to the process of generating questions during interviews. The instructions provided to an Al Interview System can evolve over the course of the interview to allow for the interviewee to move through the topics of interest in a way that makes the most sense to them. This is one aspect of the richness of data that is collected by a social scientist in these settings, since it allows the interviewees to relay their experiences, perceptions, and interpretations to the interviewer as they come to mind rather than in a fixed question order that may not resonate with a particular individual. It also allows the interviewer to ask follow-up questions that explore these themes more deeply with an interviewee. With this relatively organic path of an interview expected, evolvinginstructions for an Al Interview System may allow it to achieve the interview goals in any order.

[0011] In one embodiment, a method for creating instructions for an Al interview system may include providing inputs, including the goals of the interview, a summary of the interview, information about the interviewee, and the transcript of the interview up to that point. Such information would allow the Al Interview System to compare the goals of the interview with what has already been discussed and provides it with high-level aspects of the interviewee’s demographic details to leverage when relevant. This set of information approximates the types of details and mental considerations that a social scientist would make when deciding which questions would be most appropriate to ask next in an interview to make progress toward achieving the goals of understanding an individual’s experiences, perceptions, and interpretations related to topic of interest. All large language models have a limit to the amount of text that can be passed as inputs and outputs, which is normally referred to as the token limit. Token limits could cause the model to lose some of the context of the history of the interview if earlier messages have to be removed from the prompt as the length of the interview increases. Including the summary of the interview at each step allows the Al Interview System to retain an understanding of the full interview up to that point even when some of the earlier messages cannot be included in the transcript because the token limit of the prompt has been reached. The process of creating instructions for the Al-lnterview System is iterative. At each prompt-response pair, the process of generating the instructions is repeated so that the existing context is combined with the new developments in the interview.

[0012] An alternative embodiment of this process is to build a separate Al model whose task is to generate these instructions at each prompt-response pair. One wayto achieve this would involve an expert human interviewer writing down the considerations that they are making when deciding to ask a particular question. A social scientist would draft such a statement for each response from an interviewee. A set of these written considerations, combined with the interview transcript, would serve as the teaching examples or training data for this second Al-model. Such a system would potentially lead the Al model to generate a more concise and direct set of instructions for the Al Interview System to use when generating the next question in an interview.

[0013] Conducting an interview with an Al Interview System could include several components working in concert to provide human interviewees with an experience similar to an interview with a human social scientist. The Al Interview System may include graphical user interfaces that mirror those used in widely-used human-to- human chat applications. Before being able to collect any information from a potential interviewee, the Al Interview System may ask them for their consent to collect information and to participate in an interview. The graphical user interface of the chat application could allow interviewees to respond to questions from an Al Interview System by typing their answers directly into a dialogue box or by sending a recording of their voice to the system. When receiving audio recordings, the Al Interview System could send a request to an external service, powered by a separate Al model, that transcribes the audio response into text. The Al Interview System may require a written description of the goals of the interview that could include a list of the types of information the system should ask the interviewee about over the course of the interview and the role that the Al Interview System should play in the interview. Before the interview begins, the Al Interview System could ask interviewees a set of biographic and demographic questions to gather key pieces ofinformation that subsequently could be incorporated into instructions for the Al Interview System, as well as downstream analysis of interview data to uncover trends and patterns. During the interview, a method could call an external Al model that would create a summary of the interview transcript at each prompt-response pair. Another method includes creating an evolving set of instructions for the Al Interview System to follow when formulating questions by combining the interview summaries, biographic and demographic information about the interviewee, a description of the goals of the interview, and the interview transcript up to that point. At each step of the interview, a method includes sending the assembled instruction to the Al Interview System, receiving the question from the Al Interview System, and displaying the question in a chat bubble. In addition, methods may include listening for messages that indicate the interviewee is ready to end the interview and, if detected, sending an instruction to the Al Interview System indicating that it is time to send a final comment and end the interview. In addition, methods may include checking the quality of responses from the respondents and sending an instruction to the Al Interview System to expel interviewees who refuse to give, or are incapable of giving, relevant responses to questions. A post-interview set of questions may include asking whether the experience of interacting with the Al Interview System was preferable to well-known alternatives, such as fixed format surveys.

[0014] An Al Interview System may be able to conduct hundreds or thousands of interviews in a short period of time, each interview containing potentially hours of questions and answers. Given this scale, an Interview Analysis System may include a wide range of methods to generate insights from a large body of interview transcripts. One method may include generating separate summaries of each interview that could be quickly read or used to create one summary of all theinterviews that fall into a particular demographic group. Summaries of a large set of interviews would dramatically reduce the time required to arrive at a high-level understanding of what interviewees talked about during the interviews. More detailed methods may include placing each interview in a semantic embedding, which may enable numeric comparisons that may facilitate grouping interviewees by personas, which are profiles that represent groups of similar people. Other methods may include analyzing trends in the appearance of similar phrases or semantic embeddings across demographics or time; or they may include finding specific ways that interviewees talked about a topic of interest and determining which biographic or demographic groups tend to use related language. All of the methods included in the Interview Analysis System may be useful for a wide range of business or research use cases, while preserving the privacy of interviewees since the use of interview summaries written by a large language model to generate down-stream analyses including aggregate summaries abstracts away from the specific style of language that each interviewee used when interacting with the Al Interview System.Brief Description of Drawings

[0015] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments and aspects of the present invention. In the drawings:

[0016] FIG. 1 is a flowchart of a method for fine tuning a large language model using interview transcripts in the public domain;

[0017] FIG. 2 is a flowchart of a method for fine tuning a large language model using interview transcripts generated by a human-AI teaming process;

[0018] FIG. 3 is a schematic diagram including a flowchart of a method for fine tuning a large language model using interview transcripts generated by human-to- human interviews over a chat interface;

[0019] FIG. 4 is a flowchart of a method for creating a prompted transcript from an interview transcript and a body of contextual information about the interview and interviewee;

[0020] FIG. 5 is a flowchart of a method for generating interview instructions;

[0021] FIG. 6 is a flowchart of a method for fine-tuning a large language model to generate interview questions;

[0022] FIG. 7 is a flowchart of a method for conducting a qualitative interview using the Al Interview System;

[0023] FIG. 8 is a flowchart of a method for generating insights from a large body of interview data;

[0024] FIG. 9 is a flowchart of a method for creating personas by clustering on semantic embeddings;

[0025] FIG. 10 is a flowchart of a method for creating a research project;

[0026] FIG. 11 is a flowchart of a method for generating interview arcs; and

[0027] FIG. 12 is a schematic diagram of an Al Interview system.Description of the Embodiments

[0028] The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments and features of the invention are described herein, modifications, adaptations and other implementations are possible, without departing from the spiritand scope of the invention. For example, substitutions, additions, or modifications may be made to the components illustrated in the drawings, and the exemplary methods described herein may be modified by substituting, reordering, or adding steps to the disclosed methods. Accordingly, the following detailed description does not limit the invention. Instead, the proper scope of the invention is defined by the appended claims.

[0029] This disclosure describes the methods and processes related to training, applying, and generating insights from an Artificial Intelligence (Al) Interview System. The method for Training an Al Interview System includes methods for generating training data for the Al Interview System, which may involve curating publicly available interview transcripts, creating interviews where a social scientist — aided by Al models — acts as the interviewer and interviewee, or human-to-human interviews conducted over chat. Translating raw interview transcripts into data suitable for fine- tuning an Al model to conduct qualitative interviews on a specified topic may involve the creation of instruction prompts, enriching the interview data with Al-generated summaries of the interview at each successive step, incorporating information about the interviewee, including a description of the overall goals of the interview, and integrating these pieces into prompt-response pairs suitable for tuning an Al model. The process of conducting a qualitative interview using an Al Interview System may involve graphical user interfaces that mirror those used in widely-used human-to- human chat applications, Al models that transcribe audio recordings of people answering interview questions into text, application programming interface (API) calls to an Al model to summarize the interview after each response from an interviewee, programmatically drafting an instruction prompt that provides an Al model with a blueprint of the interview to follow, collecting demographic and biographicinformation of the interviewee before the interview begins, securing the consent of the interviewee to engage in an Al-powered interview and use the interview data for training and analysis, and conducting a separate API call to an Al model that has been tuned to conducting qualitative interviews using the processes described above to generate interview questions. An Interview Analysis System generates insights from a large body of interview data and may involve the application of separate Al models that generate summaries of each interview, summarizing interviews from interviewees with similar demographic and biographic characteristics, group interviews using semantic embeddings to create personas, uncovering trends in the appearance of specific semantic embeddings over time, and collecting similar phrases that interviewees use when they talk about a particular topic.Definitions

[0030] Artificial intelligence (Al) models are computer programs that learn from experience to complete a task. The models most often contain a large number of parameters that enable the models to successively build up complex relationships found between the given inputs and outputs contained in the set of training data that represents the experience from which the model learns.

[0031] Large language model refers to a specific type of Al model that is trained on a vast quantity of textual data to perform language tasks, including summarizing text, answering questions, and generating text. Large language Models often contain billions of parameters, allowing the models to create high-level understandings of language and concepts. The process of fine-tuning a large language model, by providing it with additional specific examples, can expand the set of language tasks the model can perform.

[0032] Semantic embeddings are numerical representations of textual data such that similar words or phrases are located closer together. Algorithms that capture contextual relationships found in a large corpus of textual data produce these representations.

[0033] An Interview Transcript Enrichment System is a set of algorithms and scientific processes for incorporating contextual information and social science interview techniques into an interview transcript that results in a dataset containing prompt-response pairs suitable for fine-tuning a large language model.

[0034] A Method for Training an Al Interview System is a series of processes and algorithms that generate training data suitable for a large language model, finetune a large language model to generate interview questions that mimic a human interviewer, and assess the quality of the questions the fine-tuned model generates.

[0035] An Al Interview System is a collection of artificial intelligence models, procedures, processes, user interfaces, and related algorithms that work in concert to conduct Al-driven interviews with humans.

[0036] An Interview Analysis System refers to a suite of processes and models that create summaries, themes, and groupings of a large body of interview transcripts that allow users to create insights which can be transparently linked to interviewee phrases.Description of the Drawings

[0037] FIG. 1 is a flowchart of a method for fine tuning a large language model using a collection of interview transcripts that are in the public domain and is a component of A Method for Training an Al Interview System, according to one embodiment. In step 100, a search is conducted for high-quality interview transcripts that are in the public domain. In this process a social scientist trained in interviewtechniques may search for interview transcripts collected as part of social science research, whether from the non-profit, public, or private sectors. Interviews that follow best practices developed by social scientists and cover a wide range of topics are more likely to provide a diverse set of examples with which to tune large language models. After collecting interview transcripts, in step 110, the set of interview transcripts are converted into prompted transcripts. In this step, an instruction for a large language model may be created after each prompt-response pair in the transcript. The instructions may include the following pieces of information: a summary of the interview up to this stage or point in the interview, the research context, information about the interviewee, and a portion of the interview transcript. This process is described in more detail in FIG 4. In step 120, a large language model is fine tuned to generate interview questions using the body of prompted interview transcripts created in step 110. In step 120, the Al Interview System sends a set of prompted interview transcripts to the Al Interview System for the method that fine tunes a large language model to generate interview questions. By using the range of experiences reflected in a set of high-quality public domain interview transcripts, the large language model can learn how to generate improved interview techniques. After the fine-tuning process, the large language model will have an updated set of parameters (for example, updates to the weights between parameters in the neural network) that are more attuned to applying interview techniques from the social sciences, as compared to an initial set of parameters .

[0038] FIG. 2 is a flowchart of method for fine tuning a large language model with a collection of interview transcripts generated as part of a human-AI teaming method and is a component of a Method for Training an Al Interview System. In step 200, a research context is created that can include a description of the interview topics ofinterest, relevant background information about the topic, or any other pieces of information that provide contextual details relevant to the interview. In step 210, a human interviewer creates and inputs pertinent details about the fictional interviewee. These can include a variety of details such as demographic information, particular individual experiences relevant to the research context, or answers to hypothetical screening questions. In step 220, an instruction prompt is generated using programming method, which combines the research context from step 200; information about the fictional interviewee from step 210; and a transcript of the most recent questions and answers in the interview, if available. The instruction prompt is for a large language model being trained, also referred to as a first large language model, to generate a given number of interview questions that elicit information about the hypothetical individual’s experiences, perceptions, and interpretations related to topic of interest. In step 230, the instruction prompt from step 220 is sent to the first large language model, and the first large language model returns, to the human interviewer, a list of possible interview questions, each relevant to the interview but randomly generated. The human interviewer reviews the list of questions generated by the first large language model in step 240, and either chooses a question from this list or drafts a new, unique question. At this point in the interview, the method employs a second large language model to play the role of the fictional interviewee. In step 250, a prompt for the second large language model is generated by a programming method, based on details of the fictional interviewee and the transcript of the interview up to that point. The prompt also includes an instruction to answer questions from the interviewer as well as the question asked. In step 260, the prompt is sent to the second large language model, which then produces a list of possible responses — each relevant to the interview and randomlygenerated — to the previously chosen interview questions. In step 270, the same human interviewer reviews the list of potential responses, and chooses one from the list or composes an alternative response. The process cycles between steps 220, 230, 240, 250, 260, and 270, until the human interviewer assesses the interview to represent an appropriate length for the research context. The framework can repeat this process (for example, repeating, all steps shown in FIG. 2) to generate any number of interview transcripts that can cover a wide range of research topics and interviewee demographic groups and types, making the method depicted in FIG. 2 extremely adaptable to training the large language model of the Al Interview System, including, for example, ways to handle hostile or reticent interviewees, which involves instructing the second large language model to act as a hostile or reticent interviewee.

[0039] After generating interview transcripts, in step 280, the set of interview transcripts are converted into prompted transcripts. In this step, an instruction for the first large language model is created after each set of hypothetical interviewee responses in the transcript. The instructions can include the following pieces of information: a summary of the interview up to this stage in the interview, the research context, information about the interviewee, and a portion or the entirety of the interview transcript. This process is described in more detail in FIG 4. In step 290, the first large language model is fine tuned to generate interview questions, and the fine tuning process uses the body of prompted interview transcripts created in Steps 200 through 270. That is, in step 290, the Al interview system outputs a set of prompted interview transcripts for subsequent fine tuning of the first large language model to generate subsequent interview questions. The fine-tuning process is described in more detail in Figure 6.

[0040] FIG. 3 is schematic diagram including a flowchart of a method for fine tuning a large language model with a collection of interview transcripts generated from human-to-human interviews conducted over a chat interface and represents a component of a method for training an Al Interview System. Box 300 represents, as part of a system that performs the method, a web application that enables human-to- human interviews. The web application enables the human interviewer and human interviewee to be in different locations (that is, remote from each other) and to submit questions and responses asynchronously. This addresses a severe limitation of in- person interviewing, in which both the interviewer and interviewee must be in the same place at the same time. Chat interfaces are ubiquitous with people around the world sending billions of text messages every day. There are several user interface elements that people expect to see in a human-to-human chat interface, such as the ability to send multiple messages that represent one thought, to see when other people in the chat are typing, and to use audio transcription services that convert spoken messages into text which appears in the chat, among others. A web application for human-to-human interviewing may include one or more of these features to promote ease of use and provide an interface in which many people are comfortable sharing their experiences, perceptions, and interpretations related to a wide range of topics. Box 310 represents a person who has been recruited to engage in an interview as an interviewee. There are many places from which people can be recruited to take part in an interview, such as a list of email addresses representing members of an organization, students in a classroom, or people who have purchased a product. For the purposes of an Al Interview System, the source of interviewees has very few restrictions and largely depends on the particular research interest. Once recruited, an interviewee visits the site of the human-to-human interview interface. Box 320 symbolizes a method for securing the agreement of the interviewee to participate in an online human-to-human interview. The agreements can be adapted to the location of the respondent to reflect varying laws and regulations for the collection of data. If the data is being collected for the purposes of training an Al model, language covering this use case may be included. If the user agrees to the terms of interview and subsequent uses of the interview data, the process continues and the process proceeds to Box 330. If the user does not agree, the interview is not conducted.

[0041] Box 330 represents a method for conducting pre-screening questions to ensure interviewees meet a set of criteria — such as past use of a commercial product — or to collect demographic data of interest to a particular research project. Box 340 symbolizes an expert human interviewer who may have academic training in and extensive experience with conducting interviews. The human interviewer may be skilled in a large number of interviewing techniques designed to ethically collect information about a person’s experiences and interpretations. The human interviewer will design an arc for the interview, depicted in Box 350, that includes a research context and an overall plan for the interview. The arc may include several lines of questioning that could serve to baseline an experience, solicit detailed examples of an experience, or provide information about an interviewee’s preferences or pain points. The arc serves as a blueprint, allowing the human interviewer to collect a comparable set of empirical observations on a given topic of interest. The interview process shown in Box 360 involves the human interviewer submitting questions to the interviewee that follow the arc and the human interviewee providing responses. The web interview interface may allow the interviewee to input responses by typing them by hand or transcribed from an audio recording of their voice. The interviewcan take many turns, and, in some cases, veer far away from the arc. The expert human interviewer may employ several techniques to ensure the interview results in data that follows the desired arc. In so doing, the interview may provide valuable examples for an Al Interview System to learn from that allow it to perform adeptly in a wide range of interview situations. After the human interviewer concludes the interview, the web application in Box 300 generates a transcript of the interview, which is represented in Box 370.

[0042] A programming method depicted in Step 380 converts each interview transcript 370 into a prompted transcript. This process method involves combining the research context, which is a narrative description of the goals of the interview and partly based on the interview arc described in Box 350; information about the human interviewee collected in the prescreening portion of process, Box 330; a transcript of the most recent questions and answers in the interview; and an instruction to generate a question that elicits information about the interviewee’s experiences, perceptions, and interpretations related to topic of interest. Together this information represents one prompt. The method creates one prompt for each set of interviewee answers. This process is described in more detail in FIG 4.

[0043] In Step 390, the Interview T ranscript Enrichment System outputs a set of prompted interview transcripts to a method that fine tunes a large language model to generate interview questions. By using the range of experiences and interviewing techniques that are reflected in a body of human-to-human interview transcripts, the large language model can learn how to mimic the techniques that skilled human interviewers regularly deploy. After the fine-tuning process, the large language model will have an updated set of parameters that are more attuned to applying interviewtechniques from the social sciences, as compared to an initial set of parameters used by the base large language model.

[0044] FIG. 4 is a flowchart of a process for creating a prompted transcript from an interview transcript and a body of contextual information about the interview and interviewee, which is performed by an Interview Transcript Enrichment System as a component used in the method for training the Al Interview System. An interview transcript, depicted in Box 400, is based on a transcript of the questions the interviewer asked the interviewee and the corresponding responses from the interviewee over the entire course of a qualitative interview. While the transcript by itself can be useful in many contexts, it may lack information about the research context and the interviewee, as well as an explicit instruction for a large language model to follow. Large language models may be limited in the amount of text they can process as part of an instruction due to their underlying architectures and compute costs, and, as an interview transcript accumulates questions and answers, a size of the transcript may exceed the amount of text the large language model can process at one time.

[0045] The process outlined in FIG. 4 is one solution to the limitations that interview transcripts by themselves present to the goal of training a large language model to conduct qualitative interviews. This embodiment involves the creation of a prompt for a large language model at each stage of an interview. If, for example, an interview consists of ten sets of exchanges between the interviewer and interviewee, then the process depicted in FIG. 4 may include generating a prompted transcript that contains ten sets of prompts and responses, where the response is the question asked by the interviewer. Each prompt includes a set of data that replicates some of the key information that an expert human interviewer would have front of mind asthey conduct an interview. The prompt may include: the research context which contains information about the topics of interest, background information where relevant, and the goals of the interview, shown in Box 410; information about the interviewee, represented by Box 420, collected from intake questions in the Al Interview System or from summary information found along with a publicly available interview transcript; a summary of the interview created by a large language model, as shown in Box 430; a portion of the interview transcript that includes the most recent exchanges between the interviewer and interviewee; and an instruction to generate an interview question, depicted in Box 440.

[0046] The research context, Box 410, is a narrative description of the interview topic and goals. The narrative describing the topic can include contextual information related to its location, timeframe, people who are involved, and related behaviors. Text outlining the goals of the interview can include detailed descriptions of the experiences or perceptions of the interviewee the Al Interview System may ask about. Information about the interviewee, Box 420, can include a wide range of demographic attributes, attitudes, or past behaviors that are relevant to the interview topic.

[0047] Large language models have defined limits on the prompt size they are able to process, which present a constraint for an Al Interview System, since the defined limits on the prompt size directly limit the amount of context the system has access to when deciding what interview questions to ask next. This is most acute when interviews accumulate an amount of text containing all the questions and responses that exceeds the prompt size. One solution to this limit is to combine a summary of the interview with the most recent portions of the interview transcript. This ensures the prompt includes knowledge of the full interview within its defined size. Asummary of the interview, Box 430, provides a succinct record of what has been discussed in the interview at any given point, which condenses the size of the interview transcript. Summarizing the interview transcript at each step allows the process in FIG. 4 to adapt to different prompt sizes as needed. Many methods can generate summaries, one embodiment is to use a large language model to produce them. The prompt may include particular aspects of the interview to highlight in the summary, such as themes that the interviewer should follow up on at a later point in the interview.

[0048] A prompt may include an instruction, Box 440, so that the large language model can learn to relate sets of contextual information and instructions to types of responses. One component of the instruction is the definition of the role the large language model should take on. One embodiment of the role would be, “you are social scientist conducting an interview for research.” The next components of the instruction can further define or limit this role. One embodiment of such additional instructions would be to avoid long compound questions and including opinions or examples in the questions.

[0049] A programming method, represented in Box 450, combines the information in Boxes 400, 410, 420, 430, and 440 to produce a prompted transcript from a raw interview transcript. The method can produce prompts that fit within a certain token limit, so the Interview Transcript Enrichment System is able to adapt to different prompt sizes as needed. The method creates a container in a memory that holds a set of prompt-response pairs; where the prompt is a set of contextual information and instructions for a large language model, and the response is a set of interview questions asked by the interviewer. At each stage of the method, a prompt-response pair (also referred to as a question-response pair) is added to the container. Themethod starts by creating a prompt for the first interview question. Then, the text of the research context, Box 410; information about the interviewee, Box 420; and instructions, Box 440 are combined in step 450. In addition, in step 450, the prompt is combined with the first set of questions in the interview and the prompt-response pair are then added to the container. After this, the method iterates through the remainder of the interview transcript as follows: after each set of responses from the interviewee, the method includes creating a prompt by combining the text of the research context, Box 410; textual information about the interviewee, Box 420; a textual summary of the interview, Box 430; and textual instructions, Box 440. The method then computes the size of this combined text and compares it to the prompt size. If the size of this combined text is less than the prompt size, it adds the most recent exchanges in the interview, Box 400, until it reaches the prompt size or there are not more exchanges in the transcript to include. That is, the method includes combining the elements of the prompt and iteratively comparing the size of the combined text to the prompt size, while adding the message history, to ensure the prompt size does not exceed the token limit. After the method reaches this limit, construction of the prompt for this stage of the interview is complete. The method then couples this prompt with the next interviewer question and adds the promptresponse pair to the container. When the method processes all interviewer questions, it completes and outputs a prompted transcript that is suitable for fine- tuning a large language model to ask interview questions.

[0050] FIG. 5 is a flowchart of a method for generating interview instructions, which is another embodiment of an Interview Transcript Enrichment System and a component of a Method for Training an Al Interview System. As noted in reference to the method of FIG. 4, interview transcripts by themselves are not suitable forteaching a large language model to ask interview questions that mimic human interviewers, because they lack accompanying instructions and some information about the research context and interviewee characteristics. A possible alternative embodiment of the method shown in FIG. 4 is a method in which the large language model processes input information, including responses to interview questions, to decide how to continue the interview more directly. This method includes, among other steps, human interviewers annotating transcripts of interviews that they conducted with a set of continuation instructions.

[0051] In Box 500, information about the interviewee is obtained based on answers to intake questions previously submitted by the interviewee. In Box 510, a research context for the interview is obtained. The method creates a container in memory that holds a set of prompt-response pairs. The prompts each combine contextual and summary information about the interview. The responses each contain written instructions for continuing the interview. At each stage of the method, a promptresponse pair is added to the container. The method starts by creating a prompt for the first interview question, which combines the interviewee information, Box 500; and the research context, Box 510. The human interviewer then composes a set of instructions for how to ask the first questions of the interview and inputs them into the method, Box 540. The method pairs the previously outlined prompt with the human interviewer-written instructions for the first questions, which are the corresponding response, and adds the pair to the container. The method proceeds through subsequent interview segments as follows: after each set of responses from the interviewee, a prompt is created by combining the textual information about the interviewee, Box 500; text of the research context, Box 510; and a textual summary of the interview, Box 520. The method then computes the size of this combined textand compares it to a prompt size limit, which is a limit to the amount of text that can be passed as inputs and outputs, the prompt size limit being less than the token limit to ensure a remaining capacity for the Al Interview Model to produce an output. If the size of this combined text is less than the prompt size limit, it adds the most recent exchanges in the interview, Box 530, until it reaches the size limit of the prompt or until all exchanges are included in the prompt. That is, the method includes combining the elements of the prompt and iteratively comparing the size of the combined text to the prompt size limit, to ensure the prompt size limit is not exceeded. After the method reaches this limit, it is finished constructing the prompt for this stage of the interview. The method then asks the human interviewer to compose a set of instructions for how to continue the interview, Box 540. The human interviewer inputs these instructions into the method, and it couples the prompt and response and adds the pair to the container. When the method processes all portions of the interview transcript, it outputs a set of prompt-response pairs that is suitable for fine-tuning a large language model to write interview continuation instructions, Box 550.

[0052] When the body of prompted interview continuation instructions reaches a suitable size, the process calls a method that fine-tunes a large language model, Box 560. Upon completion of this process, a fine-tuned large language model exists that can be called to generate interview continuation instructions when given a prompt that contains the information about the interviewee collected from intake questions, Box 500; research context for an interview, Box 510; a summary of the interview, Box 520; and a portion of the most recent interview transcript, Box 530.

[0053] FIG. 6 is a flowchart of a method for fine-tuning a large language model to generate interview questions, which is a component of a Method for Training an AlInterview System. Large language models are trained on vast amounts of text data to be performant in completing a wide range of tasks. Information in prompts submitted to a large language model can provide examples and context that can increase the model’s performance on a more specific task, but this method has severe limitations when the completion task is more complicated, such as in the case of continuing an interview. The task of generating interview questions may involve many more examples than would fit into any available prompt size. Fine-tuning a large language model allows the Al Interview System to change the underlying structure of the large language model by providing it with a broad range of interview questions that human interviewers asked in a diverse set of circumstances, while employing varied expert interview techniques. The resulting fine-tuned large language model can generate human-like interview questions with fewer examples included in the prompt, leaving relatively more space to include contextual details about the interview goals and interviewee information.

[0054] The process begins with the collection of a body of high-quality interview transcripts and converting them into prompted transcripts, Box 600. The transcripts can originate from the public domain, a human-AI teaming process, or human-to- human interviews over a chat interface (see, for example, FIGs. 2, 3, and 4). Next, apply the method shown in FIG. 4 to each collected transcript to produce a body of prompted transcripts. An additional method builds one file that combines all the prompt-interview question pairs across all of the prompted transcripts included in the current fine-tuning process. This set of prompt-response pairs are sent to a fine- tuning process for a large language model, Box 610. In this process, a subset of the enormous number of parameters in large language model update in a way that reduces the model’s error in producing the types of questions that are asked byexpert human interviewers. Methods in this step track the progress of the fine-tuning process and provide a notification when it is complete. The resulting fine-tuned large language model, Box 620, incorporates the interviewing techniques contained in the provided examples into its parameters and can be directly called during subsequent interviews to produce interview questions. A quality check process, Box 630, occurs in which a human interviewer engages in an interview with a human interviewee over a chat interface. During the interview, a method calls the fine-tuned large language model and instructs it to generate interview questions. The human interviewer reviews the generated interview questions, and, if they pass widely used criteria used by human interviewers to assess quality, allows the questions to be sent to the human interviewee. Other additional embodiments of the quality check process may include allowing the fine-tuned model to ask interview questions and asking the human interviewees to rate the quality of the interview. Additional embodiments of quality check process may involve an expert human interviewer writing an evaluation of an interview conducted by the fine-tuned large language model. The fine-tuning process can be repeated at regular intervals as additional high-quality human-to- human interview transcripts become available.

[0055] FIG. 7 is a flowchart of a method for conducting a qualitative interview using an Al interview system. The Al interview system integrates a wide range of methods to conduct an Al-powered interview with a human interviewee using a web browser. The research context, Box 700, may provide the framework for an interview. This consists of an expertly constructed plan for an interview that is designed to elicit interviewees’ experiences, perceptions, and interpretations that are related to a set of topics of interest. The context can define a specific group of interviewees that the interview can be given to, including particular demographic characteristics or peoplewho have used a particular product or engaged in certain experiences. The context also includes a plan or arc that the interview should follow. This plan contains one or more topics that the interview should focus on, including a set of possible follow up questions for each topic. Related interviewing techniques can be included throughout the instructions for the interviewer. The research context provides a roadmap for human interviewers to follow as they conduct interviews on the platform and for the Al interview system to follow as a condensed version of the research context is included in the instruction prompt for the Al interview system.

[0056] Box 710 illustrates an Al interview system comprised of a web application that runs a chat interface for the interview and calls various methods to ask interviewees intake questions, conducts quality checks of interviewee responses, creates summaries of the interview, and generates interviewer questions. All methods subsequently detailed in this figure are called by a web application code. Box 720 represents a set of prespecified, pre-interview intake questions that the web application asks the interviewee. As examples, these questions may include requests for demographic details or for product usage patterns or experiences. After answering the interviewee intake questions, the web application moves the interviewee into the interview, which is presented in the chat interface. The chat interface may be any suitable chat interface known to one of skill in the art.

[0057] The interview requires a large language model that is fine-tuned to produce interview questions that mimic a human interviewer, Box 730. Once the interviewee enters the interview, the web application calls methods that create a prompt for the first interview question. The prompt for the first question includes the research context, information about the interviewee from intake questions, and an instruction to generate the first question in the interview, while avoiding long compoundquestions and opinions or examples. The web application displays the first set of questions from the Al interview system, and the interviewee can begin inputting their responses, Box 740. The interviewee can input their responses via text or voice. If by voice, the web application calls an Al model to transcribe the audio message and store it in a chat transcript. The web application listens for responses, and, when it receives them, it begins two asynchronous processes: a process to produce a summary of the interview this far, Box 750; and a process to quality check the interviewee answers, Box 770.

[0058] In a summary task, Box 750, the Al interview system takes the most recent interview transcript and sends it to a large language model, with the instruction to write a one paragraph summary of the interview and highlight what the interviewee has said and what the Al interview system has not followed up on yet but could later in the interview. Box 770 represents a collection of quality check methods. One of these methods passes the most recent parts of the interview to a large language model with an instruction to answer whether the interviewees’ responses are coherent and relevant to the questions generated by the Al interview system. A second method sends the most recent interview transcript along with a prompt that asks a large language model if the interviewee or interviewer has stated they want to end the interview. If the interviewee fails the quality check processes, or a request to end the interview is detected, then the interview completes. Otherwise, the web application calls the method in Box 760 which creates a prompt for the Al interview system that contains the research context, interview instructions, interviewee information, summary of the interview, and the most recent portion of the interview transcript. The web application sends the prompt to a large language model, Box 750, and it generates the next set of interview questions. The process of theinterview proceeds until the Al interview system or interviewee wishes to end the interview, the interviewee fails a certain number of quality checks, or the interviewee clicks a button to end the interview. Once the interview is complete, the web application asks the interviewee to answer a few questions about the quality of the interview. These data facilitate tracking the quality of the interviews conducted with the Al interview System over time and on specific topics.

[0059] FIG. 8 is a flowchart of a method for generating insights from a large body of interview data, which is a component of the Interview Analysis System. The present invention can generate hundreds or thousands of interview transcripts in a short period of time. Generating this wealth of data requires a plurality of methods, each focusing on deriving a particular type of insight. FIG. 8 outlines several of those methods for generating insights from a collection of interview transcripts and associated information about the interviewees, Box 800. Creating summaries of a large collection of interviews provides a condensed accounting of what interviewees discussed, and it can save enormous amounts of time when an overall understanding is required. Box 810 is an embodiment of a summary task for the large language model, that is an instruction to create summaries of interview transcripts. This method creates a summary of a large body of interviews by carrying out the following steps: the method iterates over each interview and asks a large language model to create a summary of the interview. It instructs the large language model to pay attention to the main themes, issues, interests, and experiences expressed by the interviewee, while being comprehensive. To preserve the privacy of each interviewee, the instruction also tells the large language model not to include information that could identify the respondent. The next step in the method divides the interview summaries into groups, such that the total token length of each groupof summaries combined with the summary prompt fits within some token limit that is less than the context window of the large language model that is creating the summaries. The method then iterates through the groups and instructs the large language model to create an aggregate summary of the main interests, experiences, and themes contained within each group of summaries. It also instructs the model to provide salient examples of how interviewees talked about each of these elements. The method then continues following the pattern of grouping the summaries and asking a large language model to create a single aggregate summary. This process repeats until the token size of the remaining summary text and prompt instruction fits into the token limit and then it creates the final summary. An alternative embodiment could first cluster the summaries by the similarity of embeddings generated by a large language model to reduce the variance in underlying summaries for early model calls.

[0060] Box 820 depicts a method that identifies themes from a collection of interviews. The method starts with an aggregate summary of the interviews, which could be the result of the process outlined in Box 810 or the result of an alternative method. The method instructs a large language model to create a list of the main themes identified in the summary. The method then iterates over the list of themes, and for each, the method iterates over the interviews and combines the transcript of the interview with an instruction that asks a large language model to determine whether the interviewee discussed the particular theme. An alternative embodiment may use an embedding of each interview or message generated by a Large Language Model trained for sentence similarity and limit the set of candidate interviews for the theme using the distance between the embedding of the theme and the embedding of the interviews or messages. The method then iterates overthe transcripts of each interview that the previous method determined discussed that theme and asks a large language model to describe what the interviewee said about that theme. The instruction tells the model to see whether the interview discussed the theme, and, if it did, it returns what they said about the theme. The way an interview discussed a theme may not use the exact words of the theme definition, so the instruction includes language telling the large language model to interpret the theme broadly, representing an improvement over existing text-based analysis systems that look for more exact matches of phrases. It follows the same process of iteratively breaking the summaries into groups and writing aggregate summaries as described in Box 810 until the resultant final summary is produced.

[0061] Box 830 shows a method that produces summaries of interviews for subsets of the interviewees. The subsets can be defined by demographic variables, such as people from a certain state with a given level of educational attainment, or by product use or preferences. Once the set of interviewee characteristics are defined, the method collects all the interviews and produces a summary using a method similar to the one defined in Box 810.

[0062] FIG. 9 is a flowchart of a method for creating personas by clustering on semantic embeddings, which is a component of the Interview Analysis System. The Al interview system allows for a large number and wide range of people to be interviewed. The method of FIG. 9 is for creating a group of composite sketches or personas which are representations of people who voiced similar concerns or had similar experiences or perspectives about the interview topic. The personas can be useful in understanding how segments of a population are viewing a topic or how those segments relate to each other. Because they are representations of a group of people, personas also provide actionable insights while preserving privacy. Thismethod uses as its input a collection of interview transcripts and associated information about the interviewees, Box 900.

[0063] The method first creates a summary of each interview using a large language model, Box 910. The method iterates over each interview and asks a large language model to create a summary of the interview. It instructs the large language model to pay attention to the main themes, issues, interests, and experiences expressed by the interviewee, while being comprehensive. To preserve the privacy of each interviewee, the instruction also tells the large language model not to include information that could identify the respondent. Next, the method generates a numerical embedding, which is a vector of numbers that represent the language contained in the interview summary, for each interview summary, Box 920. The current embodiment uses a pre-trained large language model to create the embeddings for each interview summary, but the method is general to other implementations.

[0064] After creating numeric vector representations of the interview summaries, the method proceeds to finding groups of interviewees that have similar interview summaries, Box 930. This method begins by computing a measure of distance between the embedding vectors. The method is general to any properly defined distance metric, such as Euclidean distance. The method computes the distance between each pair of interviews in the dataset, resulting in a distance matrix. Next, the method creates a weighted network derived from the distance matrix where the weights are the distance between the interview summaries. The method then prunes the network by removing any edges below a certain weight percentile and applies a community detection algorithm to the resulting network, which results in a set ofinterview summaries that are relatively more similar to each other than to summaries outside of the group.

[0065] The method takes these communities of interview summaries and creates a summary for each, Box 940. For each community of interviews, the method divides the interview summaries into groups of interview summaries, such that the token length of each group of summaries combined with the summary prompt fits within some token limit that is less than the context window of the large language model that is creating the summaries. The method then produces a summary using a method similar to the one defined in Box 810. This results in a summary of the interviews that fall into each community and that represent a persona found among the interview transcripts. Finally, these summaries are given to a large language model, which is asked to create a descriptive name for each persona.

[0066] FIG. 10 is a flowchart of a method for creating an interview project, which is a component of the Al interview system. This process reduces the barriers to creating an interview project that leverages the Al interview system outlined in the present invention to answering a set of questions in natural language. The questions inform a customized line of questioning that will encourage interviewees to open up about their experiences, enabling the Al interview system to ask insightful follow-up questions and assess their responses against the interview goals laid out in the questions people fill out. The system is run via a web application, Box 1000, that collects input from users.

[0067] The first set of questions for creating an interview project revolve around defining a target audience for the interviews, Box 1010. The target audience defines the characteristics of people who will be selected for an interview and the number of people to interview. Users have the option to source interviews from a representativepanel of interviewees and select based on geography and demographics or by including customized screening questions. Users can also opt to bring their own interview audience by uploading an email list of customers, employees, or others the user is interested in interviewing. The platform also allows users to provide a set of intake questions the Al interview system will ask before entering the interview, such as the number of months someone has used a project or their income bracket. Next, the Al interview system guides the user through a series of questions about their research goals, Box 1020. The Al interview system starts by asking users to describe the interview’s purpose in broad terms, which provides high-level contextual information for the construction of an interview arc. Next, it asks users to provide a series of questions that they would like to ask interviewees to achieve their goals, which surfaces more specific user goals for the interviews. These questions may or may not be included directly in the interview. After the user provides these inputs, they have a chance to review and make changes to their inputs.

[0068] FIG. 11 is a flowchart of a method for generating interview arcs, which is a component of the Al interview system. An interview arc is a set of instructions for the model fine-tuned in FIG. 6 and may include several lines of questioning. An interview arc may include several lines of questioning that could serve to baseline an experience, solicit detailed examples of an experience, or provide information about an interviewee’s preferences or pain points. The arc serves as a blueprint, allowing the human interviewer to collect a comparable set of empirical observations on a given topic of interest. The arc provides general structure for the interview, but may also prescribe specific questions to ask in some cases. The Al interview system takes information about the desired goals of an interview as acquired in the methodshown in FIG. 10 as inputs and human designed interview arcs as the targets to learn how to construct interview arcs programmatically.

[0069] Box 1100 depicts a web application that collects information from users which is described in detail in reference to FIG. 10. Box 1110 represents a series of questions that the web application asks users about their goals for the interview. After receiving these inputs, a human interviewer develops an interview arc, Box 1120, that employs expert human interviewing techniques and incorporates contextual information provided by the user. The inputs collected in Box 1110 and the outputs created in Box 1120 represent an example prompt-completion pair that large language models can use to update their model parameters to increase performance on a specific task. Once the set of user-provided inputs and human- created interview arc reaches a sufficient size, a fine-tuning process for a large language model, Box 1130, can commence.

[0070] A method builds one file that combines all the interview input-interview arc pairs. Next, the prompt-response pairs are sent to a fine-tuning process for a large language model, Box 1130. In this process a subset of the enormous number of parameters in the large language model update in a way that reduces the model’s error in producing interview arcs that mimic those that are created by expert human interviewers. Methods in this step track the progress of the fine-tuning process and provide a notification when it completes. The resulting fine-tuned large language model incorporates the interview arc creation techniques contained in the provided examples into its parameters and can be directly called to produce new interview arcs.

[0071] FIG. 12 is a schematic diagram of an Al interview system. The Al interview system may include one or more Al models or large language models, each of whichmay include at least one processor and a memory that stores instructions to be executed by the processor. The Al interview system may also include at least one processor and a memory that stores other instructions to the executed by the processor. The Al interview system is configured to receive inputs from one or more of a human interviewer, a human interviewee, and one or more databases. The inputs from the human interviewer and the human interviewee may be received via a web application or a chat application, as described herein. The databases may include, for example, a database of interview transcripts. Although the Al interview system is described and shown as including and being in communication with the above-noted components, the Al interview system is not so limited, and other components may be included and / or in communication with the Al interview system.

[0072] While illustrative embodiments of the invention have been described herein, the scope of the invention includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and / or alterations as would be appreciated by those in the art based on the present disclosure. The claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application, which examples are to be construed as non-exclusive. Further, the steps of the disclosed methods may be modified in any manner, including by reordering steps and / or inserting or deleting steps, without departing from the principles of the invention. It is intended, therefore, that the specification and examples be considered as exemplary only with a true scope and spirit of the invention being indicated by the following claims and their full scope of equivalents.

Claims

CLAIMSWhat is claimed is:

1. A method for conducting interviews using an artificial intelligence (Al) system, the method comprising: a. obtaining training data from one or more databases; b. fine-tuning a base Al model of the Al interview system using the obtained training data; c. generating and outputting, to an interviewee, one or more interview questions for an interview based on the executed fine-tuned Al model; d. receiving answers, from the interviewee, to the output questions, generating prompt-response pairs; e. executing the fine-tuned Al model using the received answers; f. generating and outputting, to the interviewee, one or more additional interview questions for the interview based on the executed fine-tuned Al model; g. repeating steps e. and f. until the interview ends; and h. generating an updated fine-tuned Al model for use in future interviews.

2. The method of claim 1 , wherein the training data comprises transcripts of previously completed interviews.

3. The method of claim 2, further comprising converting the transcripts into prompted transcripts, wherein generating the trained Al model includes creating an instruction for the model after each prompt-response pair in the transcript, theinstruction including at least: a summary of the interview so far, a research context, information about the interviewee, and at least a portion of the interview transcript.

4. The method of claim 1 , wherein generating a trained Al model and generating a trained updated Al model each include updating the parameters of the Al model.

5. The method of claim 1 , wherein the training data comprises interview transcripts in which an interviewer and an interviewee are the same person, and a first Al model generates one or more questions and a second Al model generates answers.

6. The method of claim 1 , further comprising creating a research context including a description of one or more interview topics and relevant background about the topics.

7. The method of claim 1 , further comprising receiving, from the interviewer, information about the interviewee, including one or more of demographic information, experiences of the interviewee relevant to a research context, and answers to one or more screening questions.

8. The method of claim 1 , wherein the training data comprises interview transcripts from human-to-human interviews conducted via a chat application.

9. The method of claim 1 , wherein the interview is terminated when a separate large language model detects that either the interviewer or interviewee said that they want to end the interview.

10. The method of claim 1 , wherein, in a case in which a large language model determines that the received answers include one or more of (i) one-word answers, (ii) argumentative responses, and (iii) questions from the interviewee outside the scope of the interview, over multiple prompt-response pairs, the interview is terminated.

11. A method for training an artificial intelligence (Al) model of an Al interview system to conduct an interview, the method comprising: a. providing, to the Al model, inputs including: a context describing a task for the Al interview system to complete, an instruction to the Al interview system, information relating to an interviewee, one or more goals of the interview, a summary of data collected so far during the interview, and a transcript of the interview so far; b. generating, using the Al model, a plurality of prompt-response pairs for the interview based on the inputs; c. outputting the generated prompt-response pairs to an interviewer to decide which prompt-response pairs to use or inputs other prompt-response pairs; and d. repeating steps a.-c. at each prompt-response pair of the interview.

12. The method of claim 1 1 , wherein a stage of the interview is defined as a prompt-response pair of messages.

13. The method of claim 11 , wherein the context includes a narrative description of the interview topic.

14. The method of claim 13, wherein the narrative description of the interview topic includes information related to a location of the interview, a timeframe of the interview, people who are involved in the interview, and behaviors related to the interview.

15. The method of claim 11 , wherein the information about the interviewee includes one or more demographic attributes, attitudes, and past behaviors relevant to the interview topic.

16. The method of claim 11 , further comprising using a large language model to produce a summary of the interview at each prompt-response pair of the interview.

17. The method of claim 11 , wherein the generating the plurality of promptresponse pairs includes generating a prompt-response pair, of the plurality of prompt-response repairs, and adding the generated prompt-response pair to a container of a memory.

18. The method of claim 17, wherein, for each prompt-response pair generated, the method further comprises: e. combining the inputs, as a prompt, and an initial set of questions; f. computing a size of the combined inputs and the initial set of questions;g. comparing the computed size of the combined inputs and the initial set of questions to a size of a prompt; h. adding a number of most recent exchanges of the interview to the combined inputs and the initial set of questions, if the computed size of the combined inputs and the initial set of questions is less than the size of the prompt; i. repeating steps e. to h., with a subsequent set of questions, if the computed size of the combined inputs and the initial set of questions is less than the size of the prompt; and j. coupling the prompt with a next question if the computed size of the combined inputs and the first set of questions is not less than the size of the prompt.

19. The method of claim 18, further comprising h. outputting, when steps a.-j. are completed for all interviewer questions, a prompted transcript for use in fine tuning the large language model to ask interview questions.

20. An artificial intelligence (Al) interview system comprising: a memory storing a set of instructions; and at least one processor configured to execute the instructions to: a. use the context information about the interview and information about the interviewee to produce an initial question asked to the interviewee and transmit that message to the interviewee’s device; b. receive a response from the interviewee; c. produce a new prompt for the fine-tuned Al-model comprising at least the context of the interview, information about the interviewee, a summary ofthe interview up to the current stage, and at least some of the transcript of the interview; d. use the prompt from (c) to execute the fine-tuned Al-model to generate a new message to send to the interviewee; e. transmit the message to the interviewee; f. repeat steps c.-e. until the interview is completed; g. execute a general large language model to determine whether the interviewer or interviewee has said they want to end the interview and terminate the interview if they have; and h. execute a general large language model to determine whether the interviewee has given low quality responses as defined in 10 and terminate the interview if they have.

Images