System and method for generative artificial intelligence based on real-time search

Abstract

Embodiments described herein provide systems and methods for a customized generative AI platform that provides a user with tools to generate responses to various formats input by the user that incorporate results from a search made by the generative AI platform. The system may use a neural network to utilize input data and contextual information to identify potential search queries, collect relevant data, rank the information, generate text-based responses to user input, and present the responses and search results via user-accessible elements.

Description

[0001] Cross-references

[0002] This application is a non-provisional application to co-pending and co-owned U.S. Provisional Application No. 63 / 390,134 filed on July 18, 2022 and co-pending and co-owned U.S. Provisional Application No. 63 / 476,917 filed on December 22, 2022, and claims priority under 35 U.S.C. 119 to these applications, each of which is hereby expressly incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments relate generally to search engines and machine learning systems, and more particularly to real-time generative artificial intelligence (AI) dialogue-based applications. Background Art

[0004] Generative AI techniques have recently been developed to assist intelligent agents in conducting conversations with human users. For example, large language models (LLMs) such as ChatGPT, GPT-4, Google Bard, etc. have provided conversational AI platforms for many natural language processing (NLP) tasks. However, these LLMs typically require multiple stages of pre-training, training, and fine-tuning using carefully curated training datasets to perform NLP tasks. In other words, these LLMs may typically only extract knowledge from existing knowledge from the training data they have been exposed to, and provide NLP output based on that knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A is an example of an implementation according to an embodiment described herein. Figure 2-Figure 11 Simplified diagram of the data flow between the entities of the process described in .

[0006] Figure 1B is an example of an implementation according to another embodiment described herein Figure 2-Figure 11 Simplified diagram of the data flow between the entities of the process described in .

[0007] Figure 2 is an example of an implementation according to an embodiment described herein. Figure 1A and Figure 1B 2 is a simplified diagram of a computing device 200 of a text generation server 110 as described in FIG.

[0008] Figure 3 is an example of an implementation according to an embodiment described herein. Figure 2 A simplified diagram of the neural network structure of the text generation module 230 described in FIG.

[0009] Figure 4 is an example of an embodiment according to the present invention Figure 2A simplified diagram of an example architecture of the NL pre-processing module 231 and the search module 232 is shown in FIG.

[0010] Figure 5 is an example of an embodiment according to the present invention Figure 2 A simplified block diagram of an example architecture of the generation submodule 233 is shown in FIG.

[0011] Figure 6 is suitable for implementing Figure 1A and Figure 1B and a simplified block diagram of a networked system of a customized generative AI platform framework as described in other embodiments described herein.

[0012] Figure 7 1- Figure 6 An example logical flow chart of a method of customizing search of the framework and architecture shown in FIG.

[0013] Figure 8 1- Figure 6 An example logical flow chart of a method of customizing search of the framework and architecture shown in FIG.

[0014] Fig. 9 1- Figure 6 An example logical flow chart of a method for a generative AI system of the framework shown in .

[0015] Figures 10A-10E An example UI diagram is provided that illustrates an embodiment of implementing a generative AI system incorporated into a search system according to embodiments described herein.

[0016] Fig.11 An example UI diagram is provided that illustrates an embodiment of a text generation tool that implements set user parameters according to embodiments described herein.

[0017] Embodiments of the present disclosure and their advantages can be best understood by referring to the detailed description below. It should be understood that the same reference numerals are used to identify the same elements illustrated in one or more figures, wherein the illustrations in the figures are for the purpose of illustrating embodiments of the present disclosure, rather than for the purpose of limiting them. DETAILED DESCRIPTION

[0018] The present application relates generally to search engines and machine learning systems, and more particularly to systems and methods for language model-based search assistance tools.

[0019] As used herein, the term "network" may include any hardware or software based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or implemented with the same.

[0020] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0021] Generative AI techniques and LLMs have recently been developed to assist intelligent agents in conducting conversations with human users. However, existing LLMs can usually only provide generative output based on the knowledge they have obtained from previous training data. For example, if a user types a question that requires real-time knowledge, such as "which stock should I buy today", these LLM-based chat agents (such as ChatGPT) cannot provide a satisfactory answer because such chat agents cannot obtain the latest time-varying information about the stock market.

[0022] On the other hand, search engines allow users to provide search queries and return search results as responses. Users use search functions to learn about topics, keep up with the latest news, conduct research, and complete other tasks. In this way, search results can provide significant utility in preparing written content (e.g., writing articles or papers). For example, a user may wish to write an overview that highlights Abraham Lincoln and the Civil War. However, it is time-consuming for users to check hundreds, thousands, millions, or more search results returned from search queries. Although traditional search engines provide utility in conducting research on specific topics and aggregating large amounts of information, checking search results, identifying relevant information, and writing materials that incorporate information from searches may be time-consuming work. In view of the need for improved generative AI systems that reflect time-varying information such as real-time news events, the embodiments described herein provide systems and methods for generative AI platforms based on real-time search. Specifically, in response to user input for performing NLP tasks, the generative AI platform can trigger real-time search based on user input, and then aggregate search results to generate output. Searches can be performed as Internet web searches based on web indexes and / or dedicated searches from several selected data sources. In one embodiment, the generative AI platform may parse the content following the web link in the search results and use the parsed content as part of the input to generate a text answer in response to the user input.

[0023] In another embodiment, the generative AI platform may include a vision-language model such that the vision-language model can obtain and generate a summary of visual content (such as images or video content) from searched web links, and use such a summary as input to generate a text answer in response to user input.

[0024] For example, specific outputs can be generated based on user input queries to summarize results, answer questions, draft emails, articles, newsletters, posts, etc., and efficiently extract information for users. In one embodiment, the generative AI system uses a machine learning module to receive plain text input, generate dialogue output, and provide citations to search results that support the generated content. In this way, users can receive concise answers to queries and citations that support and confirm output without having to visit and check many web pages, thereby reducing the time to find answers while providing certainty of correctness. In some embodiments, the generative AI system can provide users with suggested follow-up responses to generate additional dialogue responses, as well as suggested search results, applications, or other information that can further help users.

[0025] For another example, a text-generated search tool provides text based on user-provided parameters. Specifically, the user can provide parameters such as use cases, desired tone, target audience, and topics for the generated text. The system uses a machine learning module to receive user parameters, search based on topics, and merge the results and user parameters to generate text output. The generative AI system can also provide citations or related search results to allow users to verify information and conduct further research. In some embodiments, the generative AI system can also provide suggestions for additional topics, alternative parameters that may be useful to the user, or other information that can help the user use the generated text or generate more text.

[0026] In this way, the generative AI system can generate text output based on real-time search results reflecting the latest information according to the user query. Generative AI technology is thus improved.

[0027] Figure 1A is an example of an implementation according to an embodiment described herein. Figure 2-Figure 111. A user 130 interacts with a user device 120, for example, by providing natural language (NL) input 126, which in turn interacts with a text generation server 110 hosting one or more natural language processing (NLP) models 115 via input 122. Input 122 may include user NL input 126, such as a user question, user parameters, user context or other context. User parameters may include any user-configured parameters for a natural language task, such as the intended audience, the type of task (writing an email, an overview and / or a research paper, etc.), and / or the tone of the output. User context may include any user preferences, such as user preferred search data sources, user activity on social media, etc.

[0028] In one embodiment, the text generation server 110 interacts with various data sources 103a-n (collectively referred to as 103). For example, the data sources 103a-n can be any number of available databases, web pages, servers, blogs, content providers and / or cloud servers, etc.

[0029] In one embodiment, upon receiving input 122, text generation server 110 may determine whether real-time search is required, and / or which data sources may be searched. In one embodiment, text generation server 110 may employ at least NLP model 115 to generate search queries. For example, in one implementation, text generation server 110 may extract key terms from text input 126 as search queries. For another example, in one implementation, text generation server 110 may generate text embeddings and perform vector search based on the text embeddings. For another example, text generation server 110 may perform a combined search based on both text queries and vector embeddings.

[0030] In one embodiment, the text generation server 110 may employ at least the NLP model 115 to generate NLP output.

[0031] In one embodiment, the text generation server 110 can utilize the neural network-based AI model 115 to predict relevant data sources for the search based on the user input. Additional details on determining specific data sources based on the search query can be found in Figure 5 and in co-pending and co-owned U.S. non-provisional application 17 / 981,102, filed on November 4, 2022.

[0032] For another example, when the text generation server 110 receives the input 122 from the user device 120, the text generation server 110 may determine data sources that have been predefined as being associated with keywords, phrases, topics, parameters, or other elements of the search input 122. Prior user interactions may also be performed on the determined data sources, such as user disapproval of search results from certain data sources and / or user preconfigured preferred data sources, etc.

[0033] In one embodiment, the text generation server 110, upon receiving the input 122, may determine what to generate as a search query, how many search queries to generate, and other parameters related to conducting a search. Figure 2-Figure 6 As further described, the text generation server 110 can host one or more neural network-based prediction modules. When the prediction module determines that a search should be conducted for one or more specific elements from the input 122, the prediction module can generate a query based on keywords or phrases in the input 122, user parameters, and / or other contextual information.

[0034] In one embodiment, the text generation server 110 can then convert the search query into a customized search query 111a-n that complies with the format requirements specific to the data source 103a-n, respectively. The customized search query 111a-n is sent to the corresponding data source 103a-n through the corresponding API 112a-n. In response, the data source 103a-n can return the query results 113a-n to the text generation server 110 in the form of links to web pages and / or cloud files.

[0035] Instead of merely presenting a link to the search results (e.g., a web page) to the user device 120, the text generation server 110 may utilize one or more NLP models 115 to extract information from the search results, generate text based on the search results and any parameters specified in the input 122, generate a natural language response, and return it as NL output 125 for display at the user device 120.

[0036] For example, at least one NLP model 115 can be used to parse the web content behind the link provided in the search results 113a-n. In one embodiment, at least one NLP model 115 can generate an overview of the web content from at least one search result 113a, and use the overview generated in this way to generate the final NL output 125.

[0037] In one embodiment, at least NLP model 115 may be used to generate output NL output 125 based on the parsed content from search results 113a-n and / or additional user-configured parameters. For example, the user-configured parameters may specify the type of NLP task (e.g., composing an email, writing a legal memorandum, and / or conducting a conversation, etc.), the intended audience of NL output 125, the tone of voice of NL output, etc.

[0038] It should be noted that the text generation server 110, NL output 125 and / or NLP model 115 are for illustrative purposes only. The framework 100 can be applied to any type of generation and / or generative model, and / or generate any type of output, such as but not limited to code snippets, images, etc.

[0039] For example, input 122 can take various formats, such as text input, audio input, image input, and / or video input. For example, input 122 may include two images and a text question, "which photo is taken at the BTS2023 tour at Barclay Center, New Yor City?". The image encoder together with the NLP model 115 may then be used to encode the image input to facilitate search based on the image encoding. In another implementation, a caption model may be used to generate a caption for the input image so that the server 110 can use the text caption of the input image for search.

[0040] For another example, input 122 may include an audio clip (e.g., of a music song). Server 110 may generate an audio signature from the audio clip and, for example, search at a data source storing a library of music songs. When the name and / or title of a musical work associated with the audio clip is received from the data source, server 110 may further generate a search based on the name and / or title of the musical work obtained to obtain search results associated with the musical work. For example, a music clip recorded by a user at Times Square, New York may be uploaded to a generation server, which in turn may identify that the music clip belongs to the Broadway musical Phantom of the Opera, and may generate an output (with both text and / or images associated with the musical) containing a brief description of the musical and / or the available schedule and tickets for the musical and a purchase link.

[0041] For example, a visual language model may be employed at the text generation server 110 together with or in place of the NLP model 115 to parse image and / or video content from at least one search result 113b and generate an overview of the image and / or video content. Such an overviewed multimedia content may be used to generate the NL output 125. For another example, a visual language model and / or other multimodal model employed at the text generation server 110 may generate a text description of an image retrieved from a web page following a search result link. The text description may be fed to the NLP model along with other text inputs from the search results to generate the NL output 125.

[0042] As another example, when user input 122 involves a query about code, such as "what is the difference to compile a list in Python and C#?", text generation server 110 can perform a code search and can generate output based on search results 113a-n. The output can include a code snippet in Python and a code snippet in C#, as well as a text portion explaining the difference between the two code snippets. Additional details about performing a code search to obtain code snippets can be found in co-pending and co-owned U.S. non-provisional application Ser. No. 18 / 330,225.

[0043] For another example, the generation server 110 may also use the NLP model 115 as a code generation model to generate code segments based on the search results 113 a - n .

[0044] For another example, the generation server 110 may also insert one or more images retrieved from the web page following the search result link into the NL output 125 for illustration. For example, when the input 122 includes the request "please write a paragraph about the history of direct current vs. alternate current", in addition to generating a text summary based on various search results, the NLP model 115 may also insert web images of Thomas Edison and Nikola Tesla into the output 125 for illustration.

[0045] For another example, the generation server 110 may also employ an image generation model that may generate image content based on text content and / or image content obtained from the search results 113a-n to form the output 125. For example, when the input 122 includes the question "why is Hilary Step at Mount Everest so famous?", the image generation and / or editing model may edit the photo of Hilary Step obtained from the search results 113a-n by adding measurement tags that overlay the photo showing the height, slope, temperature, and / or wind speed, and use the edited photo in the generated output 125 for illustration.

[0046] In one embodiment, the NL output 125 may include references to the data sources 103a-n at relevant portions based on the corresponding search results 113a-n, respectively. In this manner, the NL output 125 automatically includes the reference authority.

[0047] In one embodiment, the client component at the user device 120 may display the NL output 125 via a user interface. For example, the NL output 125 may be displayed at a side panel within the search browser, such as Figures 10A-10E In another example, the NL output 125 may be displayed at a mobile UI on the mobile user device 120 in the form of a dialog with the cloud agent.

[0048] Figure 1B is an example of an implementation according to another embodiment described herein Figure 2-Figure 11 A simplified diagram of the data flow between the entities of the process described in . Figure 1B as indicated, instead of and / or in addition to hosting Figure 1A In addition to its own NLP model 115 shown, the text generation server 110 may communicate with a plurality of external LLMs 116a-n hosted at external servers.

[0049] In one embodiment, LLMs 116a-n may be hosted at an external server that can be accessed by text generation server 110 via a network. In another embodiment, LLMs 116a-n (or copies thereof) may be hosted at text generation server 110. Text generation server 110 may communicate with LLMs 116a-n via their respective APIs 117a-n. In one embodiment, as described with respect to Figure 1AAs described, the text generation server 110 processes the input 122 and interacts with the data source 103a-n in a similar manner to obtain the search results 113a-n. When the text generation server receives the results 113a-n from the data source 103a-n, the text generation server 110 can generate an NLP input including the search results 113a-n and send the NLP input to one or more LLMs 116a-n via corresponding APIs 117a-n. For example, in one implementation, the NLP input for one or more LLMs 116a-n can be a cascade of search results 113a-n including corresponding links and / or prompts indicating the type of NLP task.

[0050] In one embodiment, the text generation server 110 may select an LLM from the candidate LLMs 116a-n for forwarding the NLP request based on the type of NLP task. For example, LLM 116a may be used to write a paper, while another LLM 116b may be selected to generate a system response in a conversation. The text generation server 110 may then generate an NL output 125 based on the output from the LLMs, and return it to the user device 120 for display. In one embodiment, the text generation server 110 may select an LLM from the candidate LLMs 116a-n based on the type of data source from which the search results are obtained. For example, when the input 122 queries "what's up with the latest tour of BTS?", the search module (e.g., Figure 2 232) may determine that the search of social media sources is prioritized (e.g., because of the query terms "tour", "BTS"). The generation server 110 may then determine that due to the nature of the data source, the search results 113a-n may contain a large amount of image and / or video content. The generation server 110 may then select an LLM capable of handling visual data to forward the request to generate an output.

[0051] Figure 2 is an example of an implementation according to an embodiment described herein. Figure 1A and Figure 1B A simplified diagram of the computing device 200 of the text generation server 110 described in FIG. Figure 2As shown, the computing device 200 includes a processor 210 coupled to a memory 220. The operation of the computing device 200 is controlled by the processor 210. And although the computing device 200 is shown as having only one processor 210, it should be understood that the processor 210 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and / or graphics processing units (GPUs), etc. in the computing device 200.

[0052] The computing device 200 may be implemented as a standalone subsystem, a board added to a computing device, and / or implemented as a virtual machine. In various embodiments, the communication device may include a personal computing device capable of communicating with a network (e.g., a smart phone, a computing tablet, a personal computer, a laptop computer, a wearable computing device such as glasses or a watch, a Bluetooth device, etc.). The service provider may utilize a network computing device capable of communicating with a network (e.g., a network server). It should be understood that the various devices utilized by the user and the service provider may be implemented as a computer system 200 in the following manner.

[0053] Memory 220 may be used to store software executed by computing device 200 and / or one or more data structures used during operation of computing device 200. Memory 220 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media from which a processor or computer is adapted to read.

[0054] The processor 210 and / or the memory 220 may be arranged in any suitable physical arrangement. In some embodiments, the processor 210 and / or the memory 220 may be implemented on the same board, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-in-chip), etc. In some embodiments, the processor 210 and / or the memory 220 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 210 and / or the memory 220 may be located in one or more data centers and / or cloud computing facilities.

[0055] In some examples, memory 220 may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 210), may cause one or more processors to perform methods described in further detail herein. For example, as shown, memory 220 includes instructions for a search platform module 230, which may be used to implement and / or simulate systems and models, and / or to implement any of the methods described further herein. Search platform module 230 may receive input 240 via data interface 215, such as an input search query (e.g., a term, sentence, or other input provided by a user) and / or user parameters, and / or the like, and generate output 250, which may be a response based on the conversation text presented in an element and supporting links to different data sources, suggestions for future user input, and / or suggestions for further user exploration, and the like. Examples of input data may include those in FIG. 1 and regarding Figure 4 Any input 122 is described in more detail, such as a user query 402, a user context 404, and / or other context 406 (such as parameters and / or current events, etc.).

[0056] The data interface 215 may include a communication interface, a user interface (such as a voice input interface and / or a graphical user interface, etc.). For example, the computing device 200 may receive input 240 (such as a training data set) from a networked database via a communication interface. Or the computing device 200 may receive input 240 from a user via a user interface, such as a search query or parameter entered by the user.

[0057] In some embodiments, the text generation module 230 is configured to generate a text message for a user device (eg, Figure 1A and Figure 1B 120). The text generation module 230 may also include an NL preprocessing submodule 231, a search submodule 232, a generation submodule 233, and (optionally) an LLM interface submodule 234. The NL preprocessing submodule 231 may perform processing steps to assist in searching, generating NL output, or other steps taken by the text generation module 230. For example, the NL preprocessing submodule 231 may generate a text-based dialog response based on received input, user parameters, or other information (e.g., Figure 1A and Figure 1B122 in ) to tokenize and generate search queries. The NL preprocessing submodule 231 can also determine how many searches to conduct and what words, topics, or phrases to search for. The search submodule 232 can determine one or more data sources for the search based on, for example, a user configuration of preferences, past user behavior indicating preferences, search queries, and / or source types. The search submodule 232 can also generate custom queries based on various data sources and transmit the custom queries to the corresponding API (e.g., Figure 1A and Figure 1B 112a-n) and receive search results from the API. Additional details of the operation of the search submodule 232 can be found in Figure 5 and co-pending and co-owned U.S. non-provisional application No. 17 / 981,102 filed on November 4, 2022. The generation submodule 233 can generate text-based output for the user based on the received and processed search results, user input, and information determined by the NL pre-processing submodule 231. In some embodiments, the generation submodule will complete and generate output based on data received and processed by the NL pre-processing submodule 231 and the search submodule 232. In other embodiments, the generation submodule 233 will generate results as output 250 based on data received from the optional LLM interface submodule 234. The LLM interface submodule 234 can interact with an LLM external to the text generation module 230, prepare input for an API associated with an external LLM, prepare prompts based on search results received by the search submodule 232, and / or prepare prompts based on input processed by the NL pre-processing submodule 231, etc. The LLM interface submodule 234 can also process results received from an external LLM and provide the results to the generation submodule 233. It can be about Figure 4 Additional functionality of the search submodule 232 is further described.

[0058] Some examples of computing devices, such as computing device 200, may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 210), may cause the one or more processors to perform the processes of the method. Some common forms of machine-readable media that may include the processes of the method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is suitable for reading.

[0059] Figure 3 is an example of an implementation according to an embodiment described herein. Figure 2A simplified diagram of the neural network structure of the text generation module 230 described in FIG. In one embodiment, the text generation module 230 and / or one or more of its submodules 231-234 may be connected via Figure 3 The artificial neural network structure shown in is implemented. The neural network includes a computing system built on a collection of connected units or nodes called neurons (e.g., 244, 245, 246). Neurons are usually connected by edges, and adjustable weights (e.g., 251, 252) are usually associated with the edges. Neurons are usually clustered into layers so that different layers can perform different transformations on corresponding inputs and output the transformed input data to the next layer.

[0060] For example, the neural network architecture may include an input layer 241, one or more hidden layers 242, and an output layer 243. Each layer may include multiple neurons, and the neurons between layers are interconnected according to a specific topology of the neural network topology. The input layer 241 receives input data (e.g., Figure 2 210), such as a user input query (e.g., Figure 1A and Figure 1B 122) and / or user-entered preferences, etc. The number of nodes (neurons) in the input layer 241 can be determined by the dimension of the input data (e.g., the length of the vector of a given input example). Each node in the input layer represents a feature or attribute of the input.

[0061] Hidden layer 242 is an intermediate layer between the input layer and the output layer of the neural network. Note that Figure 5 Two hidden layers 242 are shown for illustration purposes only, and any number of hidden layers may be used in the neural network structure. The hidden layer 242 may extract and transform input data through a series of weighted calculations and activation functions.

[0062] For example, Figure 2As discussed in , the text generation module 230 receives an input 210 including user input, parameters and / or other information, and transforms the input into an output 250 of a text-based response formulated for the input. To perform the transformation, each neuron receives an input signal, performs a weighted sum of the input according to the weights assigned to each connection (e.g., 251, 252), and then applies the activation function associated with the corresponding neuron (e.g., 261, 262, etc.) to the result. The output of the activation function is passed to the next layer of neurons or as the final output of the network. The activation function can be the same or different on different layers. Example activation functions include, but are not limited to, sigmoid function, hyperbolic tangent function, rectified linear unit (ReLU), leaky ReLU and / or Softmax, etc. In this way, after multiple hidden layers, the input data received at the input layer 241 is transformed into quite different values ​​indicating data characteristics corresponding to the task that the neural network structure has been designed to perform.

[0063] The output layer 243 is the final layer of the neural network structure. It generates the output or prediction of the network based on the calculations performed in the previous layers (e.g., 241, 242). The number of nodes in the output layer depends on the nature of the task being addressed. For example, in a binary classification problem, the output layer can consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer can have multiple nodes, each node representing the probability of belonging to a specific class.

[0064] Therefore, the text generation module 230 and / or one or more of its submodules 231-234 may include a transformative neural network structure of a neuron layer and a weight and activation function describing a nonlinear transformation at each neuron. Such a neural network structure is usually implemented on one or more hardware processors 210 (such as a graphics processing unit (GPU)).

[0065] In one embodiment, the text generation module 230 and its submodules 231-234 can be implemented by hardware, software and / or a combination thereof. For example, the text generation module 230 and its submodules 231-234 can include a specific neural network structure implemented and run on various hardware platforms 550, such as but not limited to a CPU (central processing unit), a GPU (graphics processing unit), an FPGA (field programmable gate array), an application-specific integrated circuit (ASIC), a dedicated AI accelerator such as a TPU (tensor processing unit), and a dedicated hardware accelerator designed specifically for the neural network calculations described herein. Example specific hardware for the neural network structure may include, but is not limited to, Google Edge TPU, a deep learning accelerator (DLA), and / or NVIDIA's GPU focused on AI, etc. The hardware 550 for implementing the neural network structure is specifically configured according to factors such as the complexity of the neural network, the scale of the task (e.g., training time, input data scale, size of the training data set, etc.), and the expected performance.

[0066] In one embodiment, the neural network-based text generation module 230 and one or more of its submodules 231-234 can be trained by iteratively updating the underlying parameters of the neural network (e.g., weights 251, 252, etc., bias parameters and / or coefficients in activation functions 261, 262 associated with neurons) based on a loss target. For example, during forward propagation, training data such as past code activity is fed into the neural network. The data flows through the layers 241, 242 of the network, where each layer performs calculations based on its weights, biases, and activation functions until the output layer 243 produces the output 250 of the network (such as generated text).

[0067] The output generated by the output layer 243 is compared to the expected output (e.g., "ground-truth", such as corresponding examples given ground-truth labels) (e.g., actual text from the training data). For example, the loss function can be cross entropy and / or mean squared error (MSE), etc. Given the loss, the negative gradient of the loss function is calculated separately for each weight of each layer. This negative gradient is calculated one layer at a time, iteratively from the last layer 243 of the neural network back to the input layer 241. These gradients quantify the sensitivity of the network output to parameter changes. The calculus chain rule is applied to efficiently calculate these gradients by propagating the gradients from the output layer 243 back to the input layer 241.

[0068] The parameters of the neural network are updated from the last layer back to the input layer (back propagation) based on the negative gradient calculated using the optimization algorithm to minimize the loss. Back propagation from the last layer 243 to the input layer 241 can be performed for multiple training samples in multiple iterative training epochs. In this way, the parameters of the neural network can be gradually updated to obtain a smaller or minimized loss, indicating that the neural network has been trained to generate a predicted output value that is closer to the target output value with improved prediction accuracy. Training can continue until a stopping criterion is met (such as reaching a maximum number of epochs or achieving satisfactory performance on validation data). At this point, the trained network can be used to predict new, unseen data (such as specific parameters of user queries and responses).

[0069] Thus, the training process transforms the neural network into an “updated” trained neural network with updated parameters (such as weights, activation functions, and biases). Thus, the trained neural network improves the neural network technology in the cloud-based generative AI system.

[0070] Figure 4 is an example of an embodiment according to the present invention Figure 2 231 and the search module 232. Figure 1A and Figure 1B The text generation server 110 and / or Figure 6 For example, the text generation module 230 can be based on the software and hardware platform implemented at the server 330 in Figure 3 It is implemented using the neural network structure shown in .

[0071] In one embodiment, the NL pre-processing module 231 receives input data. The input data may include natural language user input 402 (which is similar to Figure 1A and Figure 1BNL input 126 in ), user context 404 and one or more of other contexts 406. Natural language user input 402 may include a word, multiple words, sentence or any other type of search query provided by a user using platform 230 to search. In some embodiments, natural language user input 402 may be in the form of a dialogue statement or question. For example, natural language user input 402 may be a search term such as "what is a quasirecurrent neural network", "who is Richard Socher", etc. User context 404 may include input representing the user, including user ID, user preferences, user click logs or other information collected or provided by the user, user selected preferred data sources and / or "like" or "dislike" search results or search sources of the user's past activities, etc. Other contexts 406 may include input representing other useful input information, including information about world events, searches that have been conducted near the same time, searches that have been conducted near the same area, searches that have increased in volume over a period of time, or other potential context information that can help platform 410 provide appropriate search responses to users. Other contexts 406 may also include input selections included by the user using typed input, such as a desire for a response having "professional," "causal," "comedy," etc. and / or a desire to indicate a desired output such as "paragraph," "sentence," "article," etc.

[0072] In one embodiment, other context 406 may include user-configured generation parameters, such as the type of output (e.g., a paragraph, an email, a legal memorandum, an article, and / or a dialogue response, etc.), the intended audience (e.g., elementary school students, professionals, friends, social media, etc.), the tone (e.g., informative, persuasive, and / or analytical, etc.), and / or the length, etc.

[0073] In one embodiment, the NL pre-processing module 231 can cascade input information such as natural language user input 402, user context 404 and other context 406 into the input sequence of tokens and generate one or more predictive text queries. When new natural language user input 402 is received, predictions can be made. In some embodiments, the NL pre-processing module 231 can also consider previous natural language user input 402, user context 404 and other context 406 received from the same user or different users to generate one or more predictive text queries. In addition, in some embodiments, the NL pre-processing module 231 can alternatively reuse previous predictive text queries based on the similarity between the current input information and the previous input information.

[0074] The NL pre-processing module 231 may be trained on a dataset of previous natural language user input 402 , previous user context 404 , and / or previous other context 406 , and corresponding ground truth queries.

[0075] The search module 232 may receive one or more search queries from the NL preprocessing module 231 and subsequently determine a list of data sources for searching. In one implementation, the search module 232 may retrieve a predefined list of data sources that have been pre-classified based on the subject matter determined by the NL preprocessing module 231. In another implementation, in a manner similar to that described in co-pending and co-owned U.S. non-provisional application Ser. No. 17 / 981,102, filed on Nov. 4, 2022, the search module 232 may use a prediction module 431 to predict prioritized data sources for searching based on a cascade of natural language user input 402, user context 404, and / or other contextual information 406.

[0076] The search module 232 and its corresponding search module 433 may then send one or more search queries tailored for each identified data source to the corresponding search API 422a-n and receive a list of search results from the corresponding search API 422a-n.

[0077] In some embodiments, the ranking module 434 may optionally rank the list of search applications 422a-n for searching. Figure 1A and Figure 1B 406 . For example, when receiving an input query related to code, the ranking module 434 can select a data source so that the search application 422a corresponds to a search application configured to search within a database of "StackOverflow"; and / or the search application 522b corresponds to a search application configured to search within a database of "Tutorial Point", etc. The ranking module 434 uses the input sequence processed by the NL preprocessing 231 (including the natural language user input 404, the user context 404, and the other context 406) to score multiple search applications 422a-n by running the input sequence through the neural network model once for each search application 422a-n. In this way, the ranking module 434 can rank the search results from the list of data sources via the search APIs 422a-n.

[0078] In some embodiments, if the user has indicated a prior preference for search results from particular sources, as reflected in user context 404, ranking module 434 can rank search results from those particular sources higher than search results from other sources. Additionally, other context 406 can indicate that other users value particular sources related to the term identified in user query 402; thus, ranking module 434 can incorporate this other context when ranking search results. In some embodiments, ranking module 434 can prioritize sources so that sources are reused in subsequent follow-up responses related to earlier user queries 402 from earlier related conversations with the same user.

[0079] Search results from the search APIs 422a-n are typically in the form of links to web pages or cloud files in the corresponding data sources. A ranked list of search results may be passed from the ranking module 434 to the generation module 432.

[0080] The generation module 432 may follow the links of the search results and extract information from the content of the web page or cloud file. This information may then be processed by the generation module 432 and incorporated into the generated text response provided to the user as the result 430. In one implementation, the generation module 432 may also incorporate links to the web pages or cloud files that utilize the information in the result 430 as citations to support other information or to provide other information to the user.

[0081] For example, the results 430 are transmitted to the user device for display via a graphical user interface or some other type of user output device. Although the results 430 are primarily text-based responses, related search applications can be further incorporated, such as providing visual depictions of related data in addition to the text responses (e.g., to show weather information and / or stock charts, etc.). The results 430 can also include suggested follow-up inputs provided by the generation module 432, suggested URLs ranked by the ranking module 434 and provided by the generation module 432, or any other information of interest to the user. In other embodiments, such as with respect to Figure 5 As further described, the results 430 are transmitted to the generation submodule 233 for processing and preparation of output to the user.

[0082] Figure 5 is an example of an embodiment according to the present invention Figure 2 A simplified block diagram of an example architecture of the generation submodule 233 is shown in FIG. The generation submodule 233 may include Figure 1A and Figure 1B The text generation server 110 and / or Figure 6 For example, the generation module 233 can be based on the software and hardware platform implemented at the server 330 in Figure 3It is implemented using the neural network structure shown in .

[0083] In one embodiment, the generation submodule 233 (which is similar to Figure 2 233) receives one or more inputs 501a-n, such as a search input 501a from a data source (e.g., Figure 1A-1B 113a-n in the search results), user input 501b from the NL pre-processing submodule 231 (e.g., original user-asked questions and / or user-entered parameters, etc.) and / or context input 501n (e.g., previous conversation context, etc.), etc. These inputs may also include search results, links to search results, web content, photos, videos, PDF documents, natural language user input, user context, other context, and / or other data or related information.

[0084] The generation submodule 233 can process various data inputs and generate output 503. For example, the output 503 can be similar to Figure 1A-1B The NL output in is 125.

[0085] In one implementation, the generation submodule 233 may also guide content generation using guidance prompts containing user-configured parameters such as output type (e.g., email, legal memorandum, paragraph, news article, and / or conversation, etc.), intended audience (e.g., professionals, social media, friends, education, etc.), tone (e.g., informative, persuasive, and / or cautionary, etc.). In one implementation, training on a corpus of text documents annotated with tone and / or intended audience may include one or more generation submodules 233 of NLP and / or multimodal models. In one implementation, during training, prompts guiding the generation submodule 233 to generate relevant text according to the user-configured parameters may be updated accordingly.

[0086] For example, generation submodule 233 can prepare the search result that will be passed to the user, generate the summary of text, image, sound or other content.In certain embodiments, generation submodule 233 can be inserted into the link of search result in the generated context, to be used as the citation to the generated content.

[0087] In some embodiments, the generation submodule 233 will operate in conjunction with the optional LLM submodule 234 to request and process data from an external LLM based on the search results or generated text. In other embodiments, the functionality of the optional LLM submodule 234 can be integrated into the generation submodule 233.

[0088] Figure 6 is suitable for implementing Figure 1A and Figure 1B300 shows a simplified block diagram of a networked system of a customized generative AI platform framework as described in and other embodiments described herein. In one embodiment, block diagram 300 shows a system including a user device 310 that can be operated by a user 340, data provider servers 345a and 345b-345n, a server 330, and other forms of devices, servers, and / or software components that operate to perform various methods according to the described embodiments. Example devices and servers may include the following devices, stand-alone and enterprise-level servers, which may be similar to Figure 2 The computing device 200 described in the embodiment of the present invention operates as follows: Or other suitable OS based on the OS of the device and / or server. It is understood that Figure 6 The devices and / or servers shown in the figure may be deployed in other ways, and for a given embodiment, the operations performed by such devices and / or servers and / or the services provided may be combined or separated and may be performed by a greater or lesser number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0089] The user device 310, the data provider servers 345a and 345b-345n, and the server platform 330 (e.g., similar to the search server 110 in FIG. 1 ) can communicate with each other via a network 360. The user device 310 can be used by a user 340 (e.g., a driver, a system administrator, etc.) to access various features available to the user device 310, which may include processes and / or applications associated with the server 330 to receive output data anomaly reports.

[0090] The user device 310, the data sources 345a and 345b-345n, and the platform 330 may each include one or more processors, memories, and other suitable components for executing instructions (such as program codes and / or data stored on one or more computer-readable media) to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media (such as memory or data storage devices internal and / or external to the various components of the system 300) and / or may be accessed via the network 360.

[0091] The user device 310 may be implemented as a communication device that may utilize appropriate hardware and software configured for wired and / or wireless communication with the data source 345 and / or the platform 330. For example, in one embodiment, the user device 310 may be implemented as an autonomous vehicle, a personal computer (PC), a smart phone, a laptop / tablet computer, a watch with appropriate computer hardware resources, glasses with appropriate computer hardware (e.g., GOOGLE ), other types of wearable computing devices, implantable communication devices, and / or other types of computing devices capable of transmitting and / or receiving data, such as from of Although only one communication device is shown, multiple communication devices may function similarly.

[0092] Figure 6 The user device 310 includes a user interface (UI) application 312 and / or other applications 316 that may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 310 may receive a NL response (e.g., Figure 1A and Figure 1B 125 in ), and displays the message via UI application 312 (e.g., see Figures 10A-10E ). In other embodiments, the user device 310 may include additional or different modules having dedicated hardware and / or software as needed.

[0093] In various embodiments, the user device 310 includes other applications 316 that may be desirable in a particular embodiment for providing features to the user device 310. For example, the other applications 316 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate APIs over the network 360, or other types of applications. The other applications 316 may also include communication applications, such as email, text, voice, social networking, and IM applications that allow a user to send and receive emails, calls, texts, and other notifications over the network 360. For example, the other applications 316 may be an email or instant messaging application that receives prediction result messages from the server 330. As described below with respect to Figures 10A-10E As further described in detail, server 330 may provide user 340 with an email, text message, or other use case specific text response based on the query, which may be directly incorporated into other applications 316 and sent or processed without further input from user 340. Other applications 316 may include device interfaces and other display modules that may receive input and / or output information. For example, other applications 316 may include a software program executable by a processor for asset management (including a graphical user interface (GUI) configured to provide an interface to user 340 to view and interact with user-engageable elements of displayed text or other elements based on search results).

[0094] The user device 310 may also include a database 318 stored in a transient and / or non-transitory memory of the user device 310, which may store various applications and data and be utilized during the execution of various modules of the user device 310. The database 318 may store user profiles related to the user 340, predictions previously viewed or saved by the user 340, and / or historical data received from the server 330, etc. In some embodiments, the database 318 may be local to the user device 310. However, in other embodiments, the database 318 may be external to the user device 310 and accessible to the user device 310, including a cloud storage system and / or database accessible via the network 360.

[0095] The user device 310 includes at least one network interface component 319 adapted to communicate with the data sources 345a and 345b-345n and / or the server 330. In various embodiments, the network interface component 319 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency, infrared, Bluetooth, and near field communication devices).

[0096] Data sources 345a and 345b-345n may correspond to servers hosting one or more of search applications 303a-n (or collectively 303) to provide search results including web pages, posts, or other online content hosted by data sources 345a and 354b-345n to server 330. Search application 303 may be implemented by one or more relational databases, distributed databases, and / or cloud databases, etc. Search application 303 may be configured by platform 330, by data source 345, or by some other party.

[0097] In one embodiment, one or more data sources 345a-n may be similar to data sources 103a-n. In one embodiment, a hosted LLM (e.g., Figure 1B One or more additional external servers (in 116a-n) can communicate with server 330 via network 360.

[0098] In one embodiment, the platform 330 may allow various data sources 345a and 345b-345n to cooperate with the platform 330 as new data sources. The generative AI system provides an application programming interface (API) for each data source 345a and 354b-345n to plug the generative AI system into the service. For example, California Bar Association can register as a data source with the generative AI system. In this way, the data source "California Bar Association" can appear in the list of available data sources on the generative AI system. Users can select or deselect California Bar Association as the preferred data source for their search. In a similar manner, additional data sources 345 can cooperate with the platform 330 to provide additional data sources for the search so that users can understand where the search results are collected.

[0099] The data sources 345a-n (collectively, 345) include at least one network interface component 326 adapted to communicate with the user device 310 and / or the server 330. In various embodiments, the network interface component 326 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency, infrared, Bluetooth, and near field communication devices). For example, in one embodiment, the data source 345 may send asset information from the search application 303 to the server 330 via the network interface 326.

[0100] Platform 330 can accommodate Figure 2 In some implementations, the platform 330 may receive data from the search application 303 and / or the network interface 326 at the data source 345 via the network 360 to generate a user-engageable element incorporating search results. The generated user-engageable element may also be sent to the user device 310 via the network 360 for inspection by the user 340.

[0101] The database 332 may be stored in a transient and / or non-transitory memory of the server 330. In one implementation, the database 332 may store data obtained from the data provider server 345. In one implementation, the database 332 may store parameters of the search platform model 230. In one implementation, the database 332 may store user input queries, user profile information, search application information, search API information, or other information related to the ongoing search or previously conducted searches.

[0102] In some embodiments, database 332 may be local to platform 330. However, in other embodiments, database 332 may be external to and accessible to platform 330, including a cloud storage system and / or a database accessible via network 360.

[0103] The platform 330 includes at least one network interface component 333 adapted to communicate with the user device 310 and / or the data sources 345a and 345b-345n via the network 360. In various embodiments, the network interface component 333 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency (RF), and infrared (IR) communication devices).

[0104] Network 360 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 360 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 360 may correspond to a small-scale communication network (such as a private or local area network) or a larger-scale network (such as a wide area network or the Internet) accessible by various components of system 300.

[0105] Example Workflow

[0106] Figure 7 1- Figure 6 An example logical flow chart of a method for customizing search of the framework and architecture shown in . One or more of the processes of method 700 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which executable code, when executed by one or more processors, may cause one or more processors to perform one or more of the processes. In some embodiments, method 800 corresponds to code text generation module 230 (e.g., Figure 2-Figure 6 ) operation, the code text generation module 230 searches based on user input and uses the NL model to provide a natural language response to the conversational search query.

[0107] At step 702, natural language input is received at a server and from a user interface on a user device. Figure 4As shown, according to some embodiments, the input may include one or more of natural language user input 402, user context 404 or other context 406. In some embodiments, natural language user input 402 is a dialog or natural language query provided by a user. In some embodiments, the natural language input may be one or more of text input, audio input, image input, and video input.

[0108] At step 704, the server generates one or more search queries based on the natural language input received at step 702. In some embodiments, this step may be performed by a parser network. The generation of the search query may be performed, for example, by (above in Figure 2 The preprocessing stage performed by the NL preprocessing submodule 231 (discussed in Figure 2 The search submodule 232 discussed in ) is performed as part of the search operation.

[0109] In some embodiments, the server may also determine one or more potential search objects in the generated search query. This may be used to determine specific data sources that will be beneficial to the search based on the input and / or specific APIs to be accessed, etc. In some embodiments where the server is performing more than one search, this step may be used to determine that the previous search results are related to the search object currently input. In this way, resources may be saved by avoiding the need to perform multiple overlapping searches, in which case the previous search results may be sufficient to provide natural language output for the most recent input.

[0110] At step 706, the server obtains one or more search results based on the one or more search queries by real-time searching at one or more data source servers. More information about how to search is shown in co-pending and co-owned U.S. non-provisional application 17 / 981,102, which is incorporated herein by reference. In some embodiments, the search is performed based at least in part on the search object identified in step 704.

[0111] At step 708, the server generates a natural language output based at least in part on the one or more search results and includes a reference to at least one data source server. In some embodiments, the output is generated entirely by the server that receives and processes the input and performs the search. In other embodiments, the output is generated at least in part by using (in Figure 2 The LLM interface submodule 234 discussed in the above is used to generate an external NL network or LLM connected thereto.

[0112] In this manner, as shown above with reference to FIG. 1- Figure 6 discussed and referenced below Figure 8-Figure 9As further discussed, the server can provide one or more natural language outputs that include indications of web pages, PDFs, images, videos, and other sources used to generate the natural language outputs.

[0113] Figure 8 1- Figure 6 An example logical flow diagram of a method for customizing search of the framework and architecture shown in . One or more of the processes of method 800 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which executable code, when executed by one or more processors, may cause one or more processors to perform one or more of the processes. In some embodiments, method 800 corresponds to (for example, Figure 2-Figure 6 The code text generation module 230 performs a search based on the user input and uses the NL model to provide a natural language response to the conversational search query.

[0114] At step 802, the search server receives an input query via a data interface. Figure 4 As shown, according to some embodiments, the input query may include one or more of natural language user input 402, user context 404 or other context 406. In some embodiments, natural language user input 402 is a dialog or natural language query provided by a user.

[0115] The generative AI system may also consider user context when generating a dialog response to a user query. In some embodiments, user context 404 may include user profile information (e.g., user ID, user gender, user age, user location, zip code, device information, mobile application usage information, etc.), user-configured preferences or dislikes for one or more data sources (e.g., as shown in co-pending and co-owned U.S. non-provisional application 17 / 981,102), and any combination of user past activities that approved or disapproved search results from a particular data source. For example, if the user previously disapproved certain websites, the generative AI system will prioritize other websites to collect information and provide a dialog response. Conversely, if the user previously approved certain websites, the generative AI system may instead look for results from the approved website before searching other web pages to provide a dialog response.

[0116] In addition to the context related to the user, the generative AI system may also consider other contexts, such as information related to previous user interactions with the generative AI system. In some embodiments, other context 406 may include one or more of the following: previous user queries, previous responses by the generative AI system, previous searches conducted by the generative AI system, and any other contextual information related to the current or previous interactions between the generative AI system and the user or the generative AI system and the Internet.

[0117] At step 804, the search server may convert the input query into a modified query. In some embodiments, the search server may convert the input query using an NL model implemented on one or more hardware processors at the search server. In this manner, the search server may convert the input query into an input sequence of tokens. In some embodiments, user context and other context (such as user preferences) may also be incorporated into the modified query and the corresponding token sequence.

[0118] At step 806, the search server determines one or more potential search objects in the modified query. This may be a keyword or phrase within the modified query that is identified as being of particular interest.

[0119] At step 808, the search server searches based on the identified potential search objects from the modified query. In some embodiments, when multiple potential search objects are identified in step 806, multiple searches may be performed so that searches are performed for each potential search object. In other embodiments, a single search may be performed based on more than one potential search object. Before performing a search, the search server may identify potential data sources related to the modified query. The search server may then transmit search input based on the potential search object to the identified potential data sources and merge the results into a set of search results. Relevant data sources may be identified based on one or more tokens generated from user input, user context, and other contexts, and the relevant data sources may be based on previously identified relevant data sources, or the relevant data sources may be provided directly by the user.

[0120] At step 810, the search results from the search conducted at step 808 are incorporated into the NL model and processed. For example, the NL model may rank the search results, or the search results may already be ranked when they are received by the generative AI system.

[0121] At step 812, the NL model generates a response to the user query. The NL model utilizes the search results and corresponding information, tokens that incorporate the user query, user context and other context, and other relevant information to generate a text-based response corresponding to the desired output.

[0122] At step 814, the search server inserts the citation into the response generated at step 812. The search server may insert one or more citations at the end of the sentence to indicate where in the search results the information contained in the sentence can be found, provide additional links to search results of interest, and otherwise provide context to the user.

[0123] At step 816, the search server transmits the response to the user query and the set of search results obtained as a result of the user query to the user device. In some embodiments, the results may also incorporate one or more search applications, one or more interactive graphical elements, or other data related to the generated response or application-based content.

[0124] In some embodiments, the search server determines in steps 806-808 that one or more applications are related to the potential search object in the user query. Therefore, when the first search result set is obtained in step 808 or after the response is generated by the NL model in step 812, the search server can identify the APIs corresponding to the one or more applications related to the potential search object identified in step 806. These applications can then be included as output with the response to the first query, so that the information obtained through the API is provided to the user related to the first query.

[0125] As the user responds to the output provided by the search server, asks additional questions, or otherwise engages in a back-and-forth conversation, the above steps of method 800 may be repeated an indeterminate number of times. In this way, the generative AI system is able to maintain a history of the conversation. This allows the generative AI system to maintain context as the user asks questions or otherwise interacts with the generative AI system, allowing responses to be more accurately formulated for what the user is looking for. Additionally, this may allow the generative AI system to reduce resource usage by allowing reuse of searches conducted for previous user input when it is determined that the current user input is sufficiently similar or otherwise can rely on the same set of search results that have already been collected.

[0126] For example, when a second (or third, or later) user input is provided, the generative AI system will convert the input query into a modified query as detailed in step 804. When the search server determines one or more potential search objects in the modified query as detailed in step 806, the search server will determine whether any potential search object in the user input is related to one or more potential search objects in the earlier input.

[0127] If the search server determines that the potential search object is related to the potential search object from an earlier input, the search server may use the search results related to the potential search object from the earlier input rather than initiating a new search based on the search object.

[0128] However, if the search server determines that the potential search object is not relevant to the potential search object from the earlier input, the search server can instead conduct a new search based on the potential search object of the current user input. This ensures that if the user asks the search server an irrelevant question or otherwise changes the context of the user input, the search server can maintain a response that is relevant to the current user input. Although the previous input can be used to determine additional relevant context for the current user input, conducting a new search may still generate additional relevant information.

[0129] Fig. 9 1- Figure 6 An example logical flow chart of a method of a generative AI system of the framework shown in . One or more of the processes of method 900 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which executable code, when executed by one or more processors, may cause one or more processors to perform one or more of the processes. In some embodiments, method 900 corresponds to (for example, Figure 2-Figure 6 ) operation of the text generation module 230, which searches based on user input and uses the NL model to provide a formulated written response to the user query.

[0130] At step 902, an input query and one or more user constraints are received by a search server via a data interface. Figure 4 As shown, according to some embodiments, the input query may include one or more of natural language user input 402, user context 404, or other context 406. In some embodiments, the user query 402 is a dialog or natural language query provided by the user. The user constraints may include one or more parameters, such as use case, desired tone, and target audience.

[0131] At step 904, the search server may convert the input query and user constraints into a modified query. In some embodiments, the search server may convert the input query and user constraints using an NL model implemented on one or more hardware processors at the search server. In this manner, the search server may convert the input query and user constraints into an input token sequence. In some embodiments, user context and other context (such as user preferences) may also be incorporated into the modified query and corresponding token sequence.

[0132] At step 906, the search server determines one or more potential search objects in the modified query. This may be a keyword or phrase within the modified query that is identified as being of particular interest. When modifying the output to satisfy the provided user constraints, the search server may also identify key user constraints to be used in step 914.

[0133] At step 908, the search server searches based on the potential search objects identified from the modified query. In some embodiments, when multiple potential search objects are identified in step 906, multiple searches can be performed so that searches are performed for each potential search object. In other embodiments, a single search can be performed based on more than one potential search object. Before searching, the search server can identify potential data sources related to the modified query. In some embodiments, due to the provided user constraints determined in step 906, specific data sources can be identified. Then, the search server can transmit the search input based on the potential search object to the identified potential data source, and merge the results into the search result set. Relevant data sources can be identified based on one or more tokens generated from user input, user context, and other contexts, and the relevant data sources can be based on previously identified relevant data sources, or the relevant data sources can be directly provided by the user.

[0134] In step 910, the search results from the search conducted at step 908 are incorporated into the NL model and processed. For example, the NL model may rank the search results, or the search results may already be ranked when they are received by the generative AI system.

[0135] At step 912, the NL model generates a response to the user query. The NL model utilizes the search results and corresponding information, tokens that incorporate the user query, user context and other context, and other relevant information to generate a text-based response corresponding to the desired output. Although the response is modified based on the user constraints in step 914 (such as to ensure that the response meets the desired tone or audience), the user constraints can also be considered when generating the response to the user query. For example, for a social media post use case, the response may need to be shorter and more concise, while for an article use case, it may be desirable to generate a longer response.

[0136] The search server modifies the response based on the user constraints in step 914. This may involve ensuring that the language used in the response meets the desired tone of voice, or that the language is at the expected reading level of the target audience.

[0137] In step 916, the search server transmits the response to the user query and the set of search results obtained as a result of the user query to the user device. In some embodiments, the results may also incorporate one or more search applications, one or more interactive graphical elements, or other data related to the generated response or application-based content.

[0138] Figures 10A-10E An example UI diagram illustrating an embodiment of a generative AI system incorporated into a search system is provided. In an example embodiment, a text generation tool may be implemented as a search assistance tool (such as an artificial intelligence (AI) chatbot) that dialogues with a user during a search and provides an overview of search results in response to search topics of interest to the user. For example, when a user searches by typing in a search term, a search engine may return a list of search results. The text generation tool may then generate an overview in response to the list of search results, and present the search overview to the user in the form of a dialogue response. In other words, the input user query and the generated search results may be input to a generative AI system.

[0139] For example, Fig. 10A As shown, if a user searches for "best headphones", the search engine can generate a list of search results related to "best headphones". In one implementation, the data source most relevant to the search query "best headphones" (e.g., shopping sites such as Looria, electronic device rating sites such as PCMag, etc.) can be selected for searching. The search results within each data source can be presented to the user within a corresponding horizontal panel. In one implementation, each data source can perform its own independent search within its database in response to the search term. For example, the data source "PCMag" can search for "best headphones" based on review articles of different headphones. Similarly, the data source "Looria" can search for "best headphones" based on user ratings, etc.

[0140] In one embodiment, the search assistance tool can provide the user with a dialog statement outlining what the best headphones are using the entered search query and the returned search results. Fig. 10A As shown, the search tool may present a summary describing "Sony WH-1000XM5" as a search summary in a conversational format based on search results from multiple data sources related to the search term "best headphones".

[0141] In some embodiments, the search assistance tool may also provide references to specific results from the search within the search summary to allow the user to quickly access to view the results. Fig. 10A As shown, a user-clickable reference to "Sony WH-1000XM5" headphones is provided within the overview, allowing the user to choose to be redirected to specific search results related to "Sony WH-1000XM5." These references also provide a layer of transparency to the user to allow the user to see how specific search results affect the output overview from the search assistance tool.

[0142] In this manner, search assistance tools improve the user search experience by providing users with a more understandable and readable search result output environment as well as easy linking to related web pages based on the results.

[0143] In some embodiments, the search assistance tool may implement a pre-trained language model and utilize the pre-trained model to provide a dialog output to the user based on the input. However, the search assistance tool also utilizes the search results as input for reference to allow the search assistance tool to be up to date and avoid outdated answers due to updates after the training is completed. For example, if a new set of headphones is released after the training is completed, the search assistance tool can still provide the user with output that takes into account the updated headphones and is therefore relevant to the user despite the change in technology.

[0144] In another embodiment, when the user can provide additional dialog input, the search assistance tool can gradually update the search and search summary, which allows the search assistance tool to provide further refined dialog results. In response to the user dialog input, the search assistance tool can also determine whether the search engine should refine previous search results, generate additional output based on previous search results, and / or conduct a new search.

[0145] For example, Fig. 10B As shown, according to the above example, the user can further input "I only like overear headphones—what are the best ones of those?" Based on this input, the search assistance tool can determine to refine the previous search based on "ear headphones" and can accordingly provide input further tailored to the user's input and preferences while maintaining the dialogue element. In this instance, while the dialogue between the user and the AI ​​chatbot occurs, the search page can be slightly updated or maintained unchanged to allow the user to continue browsing the results while initiating the dialogue.

[0146] In other embodiments, the search assistance tool can determine whether a refined search is needed. For example, if the user's input begins to ask about a specific brand of headphones, the initial search results may not have significant input about that brand of headphones. Therefore, the AI ​​chatbot can initiate a refined search for that brand to provide further input information for consideration and output corresponding citations to the user. This allows the AI ​​chatbot to provide the user with contextually relevant, up-to-date information and output without the user initiating additional searches.

[0147] In another embodiment, when the user input switches to a new topic, the search assistance tool can determine to initiate a new search. For example, after a search query about "best headphones", if the user types another input such as "revert gitcommit", Fig. 10C As shown, the search aid may determine that this will require a new search because an entirely different topic is involved.

[0148] In this way, rather than conducting a new search each time input is generated by a user, to save computing resources and avoid overloading the search system, a generative AI system may instead determine that the context of the conversation has changed to the point where new search results are required or has become specific enough to benefit from additional results from a more detailed search.

[0149] In other embodiments, the search assistance tool may provide a response corresponding to the requested input. Figure 10C-10E As shown, a user can search for "revert git commit" for specific help on writing code, and the search assistance tool can produce output that includes an explanation of how to write code, code blocks with citations, and examples that implement the code for performing the task. The user can continue to ask follow-up questions (e.g., Figure 10D-10E ), and the search assistance tool can refine its search results based on user input and generate an updated summary.

[0150] In another implementation, a user may ask for pictures of animals, such as cats, and the search aid may generate the image directly next to the text in the output. Thus, the search aid may utilize input information from both the user and the search results to provide a tailored output that is relevant and timely to the user.

[0151] Fig.11Provided is an example UI diagram illustrating an embodiment of a text generation tool based on user parameters set. In some example embodiments, the text generation tool can be built on a search engine to automatically generate text based on user customization. For example, the text generation tool can include a generative model provided via a web-based platform. The web-based platform interface allows users to provide customized user inputs related to generated text, such as text type, expected audience, tone and / or content, etc.

[0152] A user may interact with the web-based platform interface to provide a query in the form of a desired topic 1140 for which a response should be generated. The generative AI system may take as input user constraints such as use case 1110, tone of voice 1120, target audience 1130, etc. Fig. 9 As described in step 902 of , the information may be provided to a search server where it is processed. Then, Fig. 9 Step 916 may generate a response 1150 that is provided to the user via the web-based platform.

[0153] Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. In addition, where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be separated into subcomponents comprising software, hardware, or both. In addition, where applicable, it is conceivable that software components may be implemented as hardware components, and vice versa.

[0154] Software (such as program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems networked and / or otherwise. Where applicable, the order of the various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide the features described herein.

[0155] The various features and steps described herein may be implemented as a system comprising one or more memories storing the various information described herein and one or more processors coupled to the one or more memories and a network (wherein the one or more processors are operable to perform the steps as described herein), as a non-transitory machine-readable medium comprising a plurality of machine-readable instructions (wherein the plurality of machine-readable instructions, when executed by the one or more processors, are suitable for causing the one or more processors to perform a method comprising the steps described herein), and as a method performed by one or more devices (such as a hardware processor, a user device, a server, and other devices described herein).

Claims

1. A processor-implemented method for neural network-based text generation using real-time search results, the method comprising: receiving, at the server and from a user interface on a user device, natural language input; generating, at the server, one or more search queries based on the natural language input; obtaining one or more search results based on the one or more search queries by searching in real time at one or more data source servers; as well as An output is generated at the server implementing a generative artificial intelligence neural network, i.e., a generative AI neural network, based at least in part on the one or more search results, wherein the output includes a reference to at least one data source server.

2. The method according to claim 1, wherein: The output in natural language is generated by the generative AI neural network at the server, and the generative AI neural network can include a language model.

3. The method according to claim 1, further comprising: transmitting a text generation input including the one or more search results to an external server hosting a language model; as well as The output is obtained from the external server.

4. The method according to claim 1, wherein: The output is also generated based on user-configured parameters, including one or more of the following: the type of said output in natural language; the intended audience of said output in natural language; the intonation of said output in natural language; the format of said output in natural language; as well as The length of the output in natural language.

5. The method according to claim 1, wherein: Obtaining one or more search results through the real-time search includes obtaining content from a web document via a link in the one or more search results.

6. The method according to claim 1, wherein: The output includes a text portion that is a summary of one or more search results and the reference to at least one data source server indicates that the text portion is associated with the at least one data source server.

7. The method according to claim 1, wherein: The natural language input includes one or more of text input, audio input, image input and video input.

8. A system for generating neural network-based text using real-time search results, the system comprising: a communication interface that receives natural language input via a user interface implemented on a user device; A server that implements a generative artificial intelligence neural network, namely a generative AI neural network and a plurality of processor executable instructions; as well as One or more processors that execute the instructions to perform operations including: generating, at the server, one or more search queries based on the natural language input; obtaining one or more search results based on the one or more search queries by real-time searching at one or more data source servers; and An output is generated at the server implementing a generative artificial intelligence neural network, i.e., a generative AI neural network, based at least in part on the one or more search results, wherein the output includes a reference to at least one data source server.

9. The system according to claim 8, wherein: The output in natural language is generated by the generative AI neural network at the server, and the generative AI neural network can include a language model.

10. The system according to claim 8, further comprising: transmitting a text generation input including the one or more search results to an external server hosting a language model; as well as The output is obtained from the external server.

11. The system according to claim 8, wherein: The output is also generated based on user-configured parameters, including one or more of the following: the type of said output in natural language; the intended audience of said output in natural language; the intonation of said output in natural language; the format of said output in natural language; as well as The length of the output in natural language.

12. The system according to claim 8, wherein: Obtaining one or more search results through the real-time search includes obtaining content from a web document via a link in the one or more search results.

13. The system according to claim 8, wherein: The output includes a text portion that is a summary of one or more search results and the reference to at least one data source server indicates that the text portion is associated with the at least one data source server.

14. The system according to claim 8, wherein: The natural language input includes one or more of text input, audio input, image input and video input.

15. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for neural network-based text generation using real-time search results, the instructions being executed by one or more processors to perform operations comprising: receiving, at the server and from a user interface on a user device, natural language input; generating, at the server, one or more search queries based on the natural language input; obtaining one or more search results based on the one or more search queries by searching in real time at one or more data source servers; as well as An output is generated at the server implementing a generative artificial intelligence neural network, i.e., a generative AI neural network, based at least in part on the one or more search results, wherein the output includes a reference to at least one data source server.

16. The processor-readable non-transitory storage medium of claim 15, wherein: The output is generated by the generative AI neural network at the server, and the generative AI neural network can include a language model.

17. The processor-readable non-transitory storage medium of claim 15, further comprising: transmitting a text generation input including the one or more search results to an external server hosting a language model; as well as The output is obtained from the external server.

18. The processor-readable non-transitory storage medium of claim 15, wherein: The output is also generated based on user-configured parameters, including one or more of the following: the type of said output in natural language; the intended audience of said output in natural language; the intonation of said output in natural language; the format of said output in natural language; as well as The length of the output in natural language.

19. The processor-readable non-transitory storage medium of claim 15, wherein: Obtaining one or more search results through the real-time search includes obtaining content from a web document via a link in the one or more search results.

20. The processor-readable non-transitory storage medium of claim 15, wherein: The output includes a text portion that is a summary of one or more search results and the reference to at least one data source server indicates that the text portion is associated with the at least one data source server.

Images