A machine learning architecture for multi-step querying of a database

Abstract

A method of querying a database comprises receiving a natural language query comprising a request for driver data regarding a plurality of drivers. The method includes generating, using a first generative language model, a first query to a database based on the natural language query, the first query comprising first parameters for retrieving data from the database. The method includes generating a temporary database comprising driver data regarding at least one of the plurality of drivers retrieved from the database. The method includes generating, using a second generative language model, a second query based on the natural language query, the second query comprising second parameters for retrieving data from the temporary database. The method includes generating a data structure containing the data retrieved from the temporary database. The method includes presenting a visual representation of the data structure at the client device.

Description

A MACHINE LEARNING ARCHITECTURE FOR MULTI-STEP QUERYING OF A DATABASECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to Indian Provisional Patent Application No. 202311063795, filed September 22, 2023, and Indian Provisional Patent Application No. 202411035178, filed May 3, 2024, the entireties of which are incorporated by reference herein.TECHNICAL FIELD

[0002] This application relates generally to using generative artificial intelligence for querying a database.BACKGROUND

[0003] Automated driving systems have garnered significant interest and investment in recent years due to their potential to revolutionize transportation by enhancing safety, efficiency, and convenience. These systems utilize various sensors, such as cameras, LiDAR, and radar, in conjunction with sophisticated algorithms, to perceive the surrounding environment and make driving decisions autonomously.

[0004] The large amount of data that sensors on vehicles can collect can create significant challenges for querying databases that store the data, particularly for large language models that query the stored data. For instance, the sheer volume and diversity of sensorgenerated data can cause generating queries for querying the databases to involve navigating complex data schemas and varied data formats. Given this complexity, a system that is operating a fleet of vehicles that collects data using such sensors may have difficulties generating a query that will result in retrieving all of the data that is necessary and relevant for generating a response to a data request.SUMMARY OF THE DISCLOSURE

[0005] A large language model (LLM) may query data storage devices to generate responses to queries. However, given an increase in complexity and size of a database from which the LLM retrieves data to generate a response, the LLM may face several technical difficulties when generating responses, based on such queries. For instance, existing LLMs often have a propensity to write plausible but incorrect answers to questions, generate biasedanswers, and / or generate inappropriately confident answers, which are also known as hallucinations. Databases that have larger sizes and / or have a higher complexity can increase the likelihood that an LLM retrieves incorrect or incomplete data when generating a response to a natural language query. Incorrect or incomplete data retrieval may be a cause or may contribute to hallucinations in the response.

[0006] A computer can solve the aforementioned technical challenges using a two-step querying process. For instance, the computer can generate a first query to retrieve data that is relevant for generating a response to a natural language query for data or a data analysis using one or more LLMs. The computer can generate the first query to have parameters that causes the computer to retrieve data that is relevant based on a first set of characteristics of the query. For instance, the first one or more LLMs can generate the first query to have parameters for querying specific tables that contain relevant data for responding to the query and / or tables for which a user is permissioned to access, and / or a time period of the data. The computer can generate a database (e.g., a temporary database) with the retrieved data. The computer can execute the one or more LLMs to generate a second query based on a second set of parameters to retrieve data from the temporary database. The second query may have more refined or specific parameters, such as for retrieving data of specific driver attributes from the tables of the temporary database. This two-step querying process can facilitate the computer retrieving data that is more relevant (e.g., accurate and / or necessary) for generating a response, thus aiding the LLMs in avoiding hallucinations that can arise under other approaches, such as a standard retrieval-augmented generation (RAG) database retrieval process that involves augmenting a user’s query so that retrieved documents are appended and then processed directly by an LLM.

[0007] In one embodiment, a method of querying a database includes receiving, by one or more processors, an access credential of a user from a client device; receiving, by the one or more processors from the client device via a chat interface, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers; responsive to the access credential corresponding to the plurality of drivers, generating, by the one or more processors using a first generative language model, a first query to a database based on the natural language query, the first query comprising first parameters for retrieving data from the database; generating, by the one or more processors, a temporary database comprising driver data regarding at least one of the plurality of drivers retrieved from the database using the first query; generating, by the one or more processors using a second generative language model, asecond query based on the natural language query, the second query comprising second parameters for retrieving data from the temporary database; generating, by the one or more processors, a data structure containing the data retrieved from the temporary database based on an execution of the second query in the temporary database; and presenting, by the one or more processors, a visual representation of the data structure at the client device.

[0008] The method can further include storing, by the one or more processors, a hierarchical data structure comprising nodes representing a parent profile and a plurality of profiles, wherein nodes of the plurality of profiles are linked with a node of the parent profile within the hierarchical data structure, and wherein the natural language query identifies the parent profile; and identifying, by the one or more processors, an identifier of each of at least one of the plurality of profiles from the hierarchical data structure by identifying, by the one or more processors, the node of the parent profile based on the natural language query and then identifying, by the one or more processors, the nodes of the plurality of profiles based on the links between the nodes of the plurality of profiles and the node of the parent profile.

[0009] Generating the first query can include determining, by the one or more processors using the first generative language model, a date range based on the natural language query, the date range comprising a start date and an end date; and including, by the one or more processors, the date range in the first parameters for retrieving data from the plurality of profiles.

[0010] The method can further include determining, by the one or more processors using the first generative language model, a parent profile based on the natural language query; and including, by the one or more processors, an identifier of the parent profile in the first parameters for retrieving data from the database.

[0011] The method can further include using, by the one or more processors, an application programming interface (API) to query the database using the first query.

[0012] The method can further include executing, by the one or more processors, the first generative language model to identify one or more tables containing the driver data regarding the plurality of drivers; and determining, by the one or more processors using the first generative language model, that the access credential corresponds to the plurality of drivers responsive to determining the access credential provides access to the identified one or more tables.

[0013] The method can further include determining, by the one or more processors using the first generative language model, a requested driver characteristic based on the natural language query, wherein identifying the one or more tables containing the driver data can include identifying, by the one or more processors, the one or more tables based on each of the one or more tables corresponding to a driver that corresponds to the requested driver characteristic.

[0014] Determining the requested driver characteristic can include determining, by the one or more processors using the first generative language model, a criterion regarding a driver score. Identifying the one or more tables can include identifying, by the one or more processors, the one or more tables based on the one or more tables containing driver data regarding drivers that correspond to driver scores that satisfy the criterion.

[0015] The driver scores that satisfy the criterion can exceed a threshold.

[0016] The method can further include querying, by the one or more processors, the database by using, by the one or more processors, a plurality of application programming interfaces (APIs) to query each of the one or more tables.

[0017] The method can further include discarding, by the one or more processors, the temporary database responsive to presenting the visual representation of the data structure at the client device.

[0018] Generating the second query can include generating, using the second generative language model, a structured query language (SQL) query. The method can further include querying, by the one or more processors, the temporary database using the SQL query.

[0019] In one embodiment, a system for querying a database includes one or more processors configured by computer-readable instructions. The one or more processors can be configured to receive an access credential of a user from a client device; receive, from the client device via a chat interface, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers; responsive to the access credential corresponding to the plurality of drivers, generate, using a first generative language model, a first query to a database based on the natural language query, the first query comprising first parameters for retrieving data from the database; generate a temporary database comprising driver data regarding at least one of the plurality of drivers retrieved from the database using the first query; generate, using a second generative language model, a second query based onthe natural language query, the second query comprising second parameters for retrieving data from the temporary database; generate a data structure containing the data retrieved from the temporary database based on an execution of the second query in the temporary database; and present a visual representation of the data structure at the client device.

[0020] The one or more processors can further be configured to store a hierarchical data structure comprising nodes representing a parent profile and a plurality of profiles, wherein nodes of the plurality of profiles are linked with a node of the parent profile within the hierarchical data structure, and wherein the natural language query identifies the parent profile; and identify an identifier of each of at least one of the plurality of profiles from the hierarchical data structure by identifying the node of the parent profile based on the natural language query and then identifying the nodes of the plurality of profiles based on the links between the nodes of the plurality of profiles and the node of the parent profile.

[0021] The one or more processors can be configured to generate the first query by determining, using the first generative language model, a date range based on the natural language query, the date range comprising a start date and an end date; and including the date range in the first parameters for retrieving data from the plurality of profiles.

[0022] The one or more processors can further be configured to determine, using the first generative language model, a parent profile based on the natural language query; and include an identifier of the parent profile in the first parameters for retrieving data from the database.

[0023] The one or more processors can further be configured to use an application programming interface (API) to query the database using the first query.

[0024] The one or more processors can further be configured to execute the first generative language model to identify one or more tables containing the driver data regarding the plurality of drivers; and determine, using the first generative language model, that the access credential corresponds to the plurality of drivers responsive to determining the access credential provides access to the identified one or more tables.

[0025] The one or more processors can further be configured to determine, using the first generative language model, a requested driver characteristic based on the natural language query. The one or more processors can be configured to identify the one or more tablescontaining the driver data based on each of the one or more tables corresponding to a driver that corresponds to the requested driver characteristic.

[0026] The one or more processors can be configured to determine the requested driver characteristic by determining, using the first generative language model, a criterion regarding a driver score. The one or more processors can be configured to identify the one or more tables by identifying the one or more tables based on the one or more tables containing driver data regarding drivers that correspond to driver scores that satisfy the criterion.

[0027] The one or more processors can be configured to generate the second query by generating, using the second generative language model, a structured query language (SQL) query, and the one or more processors can be further configured to query the temporary database using the SQL query.

[0028] In one embodiment, a method can include mapping, by one or more processors, a plurality of types of driver data to a plurality of pages of an online platform, each page configured to present a different type of driver data stored in a data structure linked with the page; receiving, by the one or more processors from a chat interface presented on a user interface at a client device, a natural language query, the natural language query comprising a request for driver data regarding at least one of a plurality of drivers, and the user interface displaying a first page of the plurality of pages of the online platform; determining, by the one or more processors using a generative language model, a type of driver data requested in the natural language query; retrieving, by the one or more processors, a second page from the plurality of pages based on the type of driver data corresponding to the second page in the mapping; and presenting, by the one or more processors, a visual representation of the type of driver data on the second page at the client device.

[0029] The method can further include subsequent to retrieving the second page, receiving, by the one or more processors, a second natural language query; and filtering, by the one or more processors using a second generating language model, a set of driver data depicted on the second page in the visual representation of the type of driver data based on the second natural language query.

[0030] Filtering the set of driver data can include generating, by the one or more processors using the second generative language model, a query using the second natural language query, the query comprising parameters for retrieving the requested driver data fromthe set of driver data of the visual representation of the type of driver data; and generating, by the one or more processors, a data structure containing the requested driver data retrieved from the set of driver data based on an execution of the query on the visual representation of the type of driver data.

[0031] The method can further include presenting, by the one or more processors, a visual representation of the data structure containing a filtered version of the set of data on the second page.

[0032] The method can further include determining, by the one or more processors using the second generative language model, a requested driver characteristic based on the second natural language query; and generating, by the one or more processors, the visual representation of the data structure based on the requested driver characteristic.

[0033] Determining the requested driver characteristic can include determining, by the one or more processors using the second generative language model, a criterion regarding a driver score.

[0034] Generating the visual representation of the data structure can include generating, by the one or more processors, a list of a defined number of drivers ordered based on performance score of each of the drivers of the list, the defined number extracted from the second natural language query by the second generative language model.

[0035] The method can further include generating, by the one or more processors, a shadow data structure containing the set of driver data of the second page; subsequent to filtering the set of driver data based on the second natural language query, receiving, by the one or more processors, a third natural language query; and generating, by the one or more processors using the second generating language model, a second visual representation of the set of driver data based on the set of driver data of the second page stored in the shadow data structure.

[0036] The method can further include detecting, by the one or more processors, navigation from the second page to a third page; and discarding, by the one or more processors, the shadow data structure responsive to the detection.

[0037] The method can further include determining, by the one or more processors using the generative language model, an action type to change from the first page to the secondpage based on the natural language query, wherein retrieving the second page from the plurality of pages comprises retrieving, by the one or more processors, the second page based on the determined action type.

[0038] In one embodiment, a system can include one or more processors configured by machine-readable instructions. The one or more processors can be configured to map a plurality of types of driver data to a plurality of pages of an online platform, each page configured to present a different type of driver data stored in a data structure linked with the page; receive, from a chat interface presented on a user interface at a client device, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers, and the user interface displaying a first page of the plurality of pages of the online platform; determine, using a generative language model, a type of driver data requested in the natural language query; retrieve a second page from the plurality of pages based on the type of driver data corresponding to the second page in the mapping; and present a visual representation of the type of driver data on the second page at the client device.

[0039] The one or more processors can be further configured to, subsequent to retrieving the second page, receive a second natural language query; and filter, using a second generating language model, a set of driver data depicted on the second page in the visual representation of the type of driver data based on the second natural language query.

[0040] The one or more processors can be configured to filter the set of driver data by generating, using the second generative language model, a query using the second natural language query, the query comprising parameters for retrieving the requested driver data from the set of driver data of the visual representation of the type of driver data; and generate a data structure containing the requested driver data retrieved from the set of driver data based on an execution of the query on the visual representation of the type of driver data.

[0041] The one or more processors can be further configured to present a visual representation of the data structure containing a filtered version of the set of data on the second page.

[0042] The one or more processors can be further configured to determine, using the second generative language model, a requested driver characteristic based on the second natural language query; and generate the visual representation of the data structure based on the requested driver characteristic.

[0043] The one or more processors can be configured to determine the requested driver characteristic by determining, using the second generative language model, a criterion regarding a driver score.

[0044] The one or more processors can be configured to generate the visual representation of the data structure by generating a list of a defined number of drivers ordered based on performance score of each of the drivers of the list, the defined number extracted from the second natural language query by the second generative language model.

[0045] The one or more processors can be further configured to generate a shadow data structure containing the set of driver data of the second page; subsequent to filtering the set of driver data based on the second natural language query, receiving, by the one or more processors, a third natural language query; and generate, using the second generating language model, a second visual representation of the set of driver data based on the set of driver data of the second page stored in the shadow data structure.

[0046] The one or more processors can be further configured to detect navigation from the second page to a third page; and discard the shadow data structure responsive to the detection.

[0047] The one or more processors can be further configured to determine, using the generative language model, an action type to change from the first page to the second page based on the natural language query, wherein the one or more processors are configured to retrieve the second page from the plurality of pages by retrieving the second page based on the determined action type.

[0048] In one embodiment, a method of querying a multi-access database (set of data sources) comprises receiving an access credential of a user; receiving a natural language query from the user; detecting, using a first generative language model, that the natural language query can be satisfied, at least in part, by a parameterization of a pre-defined filtering operation of the multi-access database (set of data sources); generating a temporary database, based on the parameterized pre-defined filtering operation and the access credential of the user applied to the multi-access database; generating, using a second generative language model, a database query based on the natural language query; and generating a response to the natural language query based on the database query applied to the temporary database.

[0049] The parameterized pre-defined filtering operation can be a database query corresponding to a specified date range, wherein the parameterization of the pre-defined filtering operation comprises a start date and an end date. The parameterized pre-defined filtering operation can be a database query corresponding to a specified driver group, wherein the parameterization of the pre-defined filtering operation comprises a driver group indicator.

[0050] The method can further include detecting, using a third generative language model, that the natural language query can be satisfied by a parameterization of a pre-defined database filtering operation of the temporary database. The parameterized pre-defined filtering operation can be a database query corresponding to a driver collision risk assessment, wherein the parameterization of the pre-defined filtering operation comprises an indication of a number of drivers about whom risk assessment data should be returned.

[0051] Certain aspects of the present disclosure generally relate to providing, implementing, and using a large language model to assist drivers and safety managers. Certain aspects are directed to overcoming challenges and limitations of existing LLMs so that they may be used effectively in driving applications. Such a system may be referred to as a Safety Manager Assistant (SMA). The SMA may feature a language model and other related chatbot functionality, such as may be incorporated into a conversational chatbot, or such as may make use of one or more components of a conversational chatbot. The present application addresses the use of LLMs to provide assistance to a safety manager.

[0052] In one embodiment, a method comprises receiving a natural language query from a user of a web portal, wherein the web portal comprises a plurality of pages, each page of the plurality containing one or more interactive views into driver safety information; processing the natural language query with a large language model to yield an identified page, the identified page corresponding to a page of the plurality of pages within the web portal where driver safety information responsive to the query can be found; generating interaction instructions, wherein the interaction instructions cause an interactive view contained within the identified page to highlight driver safety information responsive to the natural language query; and refreshing the identified page with interaction instructions.

[0053] The method can further include generating navigation instructions, wherein the navigation instructions cause the web portal to present the identified page to the user. Processing the natural language query with a large language model can further yield a time period. The method can further include configuring the interactive view with the time period.The method can further include summarizing the highlighted driver safety information with a second large language model. The method can further include identifying a driving behavior that a driver could improve based on the summarized driver safety information. The method can further include providing a recommendation for a coaching action based on the identified driving behavior.

[0054] The method can further include generating navigation instructions, wherein the navigation instructions may be used by the user to guide the user to navigate within the web portal to the identified page.BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. Unless indicated as representing the background art, the figures represent aspects of the disclosure. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0056] FIG. 1 illustrates an example environment showing a computing system for a machine learning architecture for querying a database, according to an embodiment;

[0057] FIG. 2 illustrates a flowchart of a method for implementing a machine learning architecture for querying a database, according to an embodiment;

[0058] FIG. 3 illustrates a sequence diagram for implementing a machine learning architecture for querying a database, according to an embodiment;

[0059] FIG. 4 illustrates an example hierarchical data structure, according to an embodiment;

[0060] FIG. 5 illustrates a flowchart of a method for implementing a machine learning architecture for navigating a web platform, according to an embodiment; and

[0061] FIGS. 6 and 7 illustrate example user interfaces for implementing a machine learning architecture for navigating a web platform, according to an embodiment.DETAILED DESCRIPTION

[0062] Reference will now be made to the illustrative embodiments depicted in the drawings, and specific language will be used here to describe the same. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is thereby intended. Alterations and further modifications of the inventive features illustrated herein, and additional applications of the principles of the subject matter illustrated herein, which would occur to one skilled in the relevant art and having possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The illustrative embodiments described in the detailed description are not meant to be limiting of the subject matter presented.

[0063] A system may attempt to generate responses to natural language queries by implementing large language models. Such a system may do so, for example, by inputting a natural language query into a large language model and executing the model. Based on the execution, the large language model can convert the natural language query into a database query (e.g., an embedding or vector) and search a database for data to use to respond to the query. The large language model can retrieve data to use to respond to the query based on a similarity between the database query and the data. The large language model can then formulate a response based on the retrieved data. However, such a data retrieval and response generation system may face multiple difficulties in accurately and efficiently generating responses to queries. For example, large language models are prone to “hallucinations.” Typical causes of hallucinations may be a lack of context, misleading prompts, rare or uncommon inputs, exploiting biases, incorrect and incomplete data retrieval, noise in the input data, and attempts at creative writing. If an LLM attempts to query a large database with data in varying formats, any of these causes can be compounded and result in incorrect or incomplete responses (e.g., such as because there may be multiple types of data in the database that are similar to the query used to query the database). Another challenge that may arise is that the LLM may attempt to inject data into the database or otherwise change the data within the database during the query. An LLM may attempt to do so, for example, due to programming errors, improper synchronization, application programming interface (API) issues, etc.

[0064] To address these technical challenges, a computing system or computing device implementing the systems and methods described herein may implement a two-step querying machine learning architecture. For instance, the computing system can receive a natural language query (e.g., a string of text requesting information or data) from a client device. The natural language query can include a request for driver data regarding a plurality of drivers. Based on the natural language query, the computing system can execute one or more generative language models to generate a first query comprising parameters for retrieving the requested data from a database. The computing system can generate the first query to have a first set of parameters. The first set of parameters can include broad features identifying data to retrieve, such as to retrieve data from a defined time (which the generative language models may determine from the natural language query) and / or the tables from which to retrieve the data. The one or more generative language models can use the first query and one or more APIs to retrieve data from the database that is associated with the plurality of drivers (e.g., associated with generating a response that is relevant to the natural language query). The computing system can generate a temporary database with the data retrieved using the first query. The computing system can then execute the one or more large language models using the natural language query to generate a second query in a schema that is compatible with the temporary database (e.g., a SQL schema). The computing system can generate the second query to have a second set of parameters. The second set of parameters can include more narrow features than the first set of attributes, such as identifying specific types of driver data to retrieve from the temporary database. The computing system can query the temporary database using the second query to retrieve data to use to generate a response to the natural language query. The one or more large language models can generate the response in a format based on the natural language query (e.g., if the natural language query requests the data in a table, the one or more large language models can generate the response as a table). The computing system can present the response at the client device that transmitted the query to the computing system. In this way, the computing system can use a dual-query method that breaks up the data retrieval process to initially retrieve relevant data to generate a response and then retrieve data is more closely related to generating an accurate and complete response. Doing so can reduce the noise in the data that the large language models use to generate the response, thus reducing hallucinations in the response while also reducing the likelihood of making improper changes or providing improper access to the underlying database storing the data.

[0065] Another technical challenge that arises when using an LLM to generate a response to a natural language query is that the LLM may have difficulty differentiating between data a user has the appropriate access credentials for which to generate the response. For example, a user accessing a web platform may only be able to view or access data of drivers that fall under a particular corporate umbrella and / or that are located in a particular geographical area. The user may not input these restrictions into the LLM when submitting a query, so the LLM may not discriminate in the data that the LLM retrieves and uses to generate a response. Thus, the LLM may provide the user with data that the user is not authorized to access.

[0066] To avoid providing data that the user is not authorized to access, a computing system may use an authorization credential provided by an application (e.g., a browser) to provide to the computing system when submitting a query. For example, when a client device accesses a platform provided by the computing system, a user of the client device can submit login information to the platform for authentication. Responsive to determining a match with stored login information, the computing system can establish a session for the client device accessing the platform. The computing system can assign a token to the session, such as an access credential that corresponds to data that the user is authorized to access. The access credential can be or include a user identifier or a role of the user or one or more permissions of data the user is authorized to access within the database of driver data stored by the data processing system. Subsequently, the user can submit a natural language query to the computing system through the session established between the client device and the platform (e.g., the computing system). The client device can transmit the access credential to the computing system or the computing system can identify the access credential of the session. The computing system can identify data in a database that the client device is authorized to access based on the access credential and only use the identified data to generate a response to the natural language query. Thus, the computing system can automatically restrict or filter the data that the LLM may include in a response to a natural language query based on a user’s access credentials.

[0067] For example, FIG. 1 illustrates an example environment showing a computing system for implementing a generative artificial intelligence model for data storage and retrieval, according to an embodiment. Various other system architectures may include more or fewer features and / or may utilize the techniques described herein to achieve the results and outputs described herein. Therefore, the system depicted in FIG. l is a non-limiting example.

[0068] FIG. 1 illustrates a system 100, which includes components of a driver and / or driving analysis system involving a generative artificial intelligence model. The system 100 can include one or more vehicles 102, 104, and 106, an analytics server 108, and / or a client device 110. The components 102-110 can communicate with each other via a communications network (not shown). The system 100 is not confined to the components described herein and may include additional or other components, not shown for brevity, which are to be considered within the scope of the embodiments described herein.

[0069] The vehicles 102-106 can each be any type of vehicle, such as a car, truck, van, sport-utility -vehicle (SUV), motorcycle, semi-tractor trailer, or other vehicle that can be driven on a road or another environment. The vehicles 102-106 can each be operated by a user, or, in some implementations, can include an autonomous vehicle control system (not pictured) that navigates the vehicles 102-106 or provides navigation assistance to operators of the vehicles 102-106. The vehicle 102-106 can be vehicles of a fleet of vehicles that are owned and / or operated by an entity (e.g., a business or organization) to transport goods, materials, and / or individuals.

[0070] A client device 110 can be a computer that is configured to receive messages and / or present messages or other data on a user interface. The client device 110 can include a processor 112 and a memory 114. The client device 110 can be any type of computing device, such as a mobile phone, laptop computer, desktop computer, smart watch, gaming console, personal data assistant, dashboard computer, or other computing device. A driver associated with one of the vehicles 102-106 can own or otherwise access the client device 110 to access a platform provided by the analytics server 108 that manages accounts for different drivers (e.g., drivers of a driver fleet managed by the analytics server 108). The driver can access the platform through the account for the driver that the analytics server 108 stores in a storage 116.

[0071] The analytics server 108 can include the storage 116, which can store datasets indicating driving data regarding different drivers of the vehicles 102-106 or the vehicles 102- 106 themselves as well as any model and / or application needed to process the datasets and / or provide a platform through which users can access the different datasets. The storage 116 can additionally or instead include a large language model (LLM) 118, an application 120, and / or a database 122. The storage 116 can be or include a computer-readable memory that can store or maintain any of the information described herein that is generated, accessed, received, transmitted, or otherwise processed by the analytics server 108. The storage 116 can maintainone or more data structures, which may contain, index, or otherwise store each of the values, pluralities, sets, variables, vectors, numbers, or thresholds described herein. The storage 116 can be accessed using one or more memory addresses, index values, or identifiers of any item, structure, or region of memory maintained by the storage 116.

[0072] The storage 116 may be internal to the analytics server 108 or may exist externally to the analytics server 108 and accessed via a suitable bus or interface. In some implementations, the storage 116 can be distributed across many different storage elements. The analytics server 108 (or any components thereof) can store, in one or more regions of the memory of the storage 116, the results of any or all computations, determinations, selections, identifications, generations, constructions, or calculations in one or more data structures indexed or identified with appropriate values.

[0073] The analytics server 108 can include or be in communication with a communication interface 124 that can communicate wirelessly with other devices. The communication interface 124 of the analytics server 108 can include, for example, a Bluetooth communications device, a Wi-Fi communication device, or a 5G / LTE / 3G cellular data communications device. The communication interface 124 can be used, for example, to receive traffic data (e.g., indications of accidents, traffic tickets, or other driving incidents associated with different drivers or vehicles) from a remote computer. The communication interface can also be used, for example, to receive driving data from the vehicles 102-106. The driving data can be or include sensor data generated by one or more sensors of a vehicle. For example, the driving data can include accelerometer data, sensor data indicating the speed or velocity of the vehicle, gyroscopic data, and tire air pressure data. The driving data can additionally or instead include weather data indicating the weather that corresponds to the times in which the sensor data or other data was generated. The driving data can additionally or instead include global positioning system (GPS) data indicating the geographic location of the vehicle over time (e.g., at different points in time). The driving data can additionally or instead include data indicating the regulations in place on the surface or the geographical region in which the vehicle was driving when the sensor data or any other data was collected. For example, the driving data can include an indication of a speed limit, a stop sign, a stop light, a yield sign, etc. The driving data can include any type of driving data.

[0074] In some cases, computing devices of the vehicles themselves can analyze sensor data, images collected by cameras of the vehicles, and / or other data to determine drivingmetrics for a vehicle. For example, the vehicles 102-106 can respectively include computing devices 126-130. The computing devices 126-130 can collect and / or aggregate sensor data collected by one or more sensors of the respective vehicles 102-106 and into driving metrics. Driving metrics can be or include, for example, average speed, top speed, speed relative to a speed limit, average following distance, number of instances in which a vehicle is within a threshold distance (which may vary or be a function of the velocity of the vehicle), time during the day spent driving, etc. The computing devices 126-130 can collect sensor data and / or images from the sensors and / or images of the vehicles 102-106 and generate such metrics for the vehicles 102-106 based on the collected data. The computing devices 126-130 can do so using defined rules or functions on data that is relevant to the different metrics, such as by calculating an amount of time spent driving after 9 PM, by detecting the number of instances in which an image or sensor data (e.g., reflective sensor data) indicates an object or vehicle in front of one of the vehicles 102-106 is within a threshold distance of the vehicle 102, 104, or 106, or by calculating a magnitude and / or time in which the vehicle travels above a speed limit. The computing devices 126-130 can calculate any metrics for the vehicles. The computing devices 126-130 can calculate or generate such metrics with data periodically at set intervals or pseudo-randomly. The computing devices can transmit the driving metrics to the analytics server 108 responsive to calculating or generating the driving metrics. The analytics server 108 can receive the metrics and / or any other driving data from the vehicles 102-106 and store the driving metrics and / or driving data in the storage 116.

[0075] The computing devices 126-130 can include identifications of the respective vehicles 102-106 and / or drivers of the vehicles 102-106 in the messages to the analytics server 108 containing the driving data and / or driving metrics. The identifications can be identifications that correspond with accounts of the respective vehicles 102-106 or drivers. For example, the computing devices 126-130 may be configured to include identifications of the vehicles 102-106 for which the computing devices 126-130 can collect driving data and / or for drivers of the respective vehicles 102-106. The computing devices 126-130 can include the identifications in the messages to indicate the drivers and / or vehicles that correspond to the driving data and / or driving metrics. The analytics server 108 can receive the identifications and store the driving data and / or driving metrics in the storage 116 (e.g., in the database 122 in the storage 116) with the identifications such that the data can be queried or otherwise grouped based on the identifications.

[0076] The analytics server 108 can provide a platform to different users. The platform can be or include a set of web pages (e.g., a plurality of web pages) that a user may access to view different types of driver data and / or information about individual drivers, a group of drivers, a fleet of drivers, individual vehicles, a fleet of vehicles, etc. A user can use the client device 110 to access the platform. To do so, the user can input login credentials (e.g., username and / or password) into the platform. The analytics server 108 (e.g., via the application 120) can authenticate the user’s login credentials (e.g., by comparing the login credentials with stored login credentials or by hashing the login credentials and comparing the hash against a hashed version of the login credentials). Responsive to authenticating the user’s login credentials, the analytics server 108 can access their account with the platform. Responsive to logging into the platform, the analytics server 108 can establish a session with the client device, such as by storing a session cookie on the application (e.g., browser) the client device uses to communicate with the analytics server 108. The analytics server 108 can assign an access credential (e.g., a JavaScript Object Notation (JSON) web token) for the session, such as by identifying the role of the user (e.g., the user’s account) and / or permissions (e.g., data that the account is permissioned to access) of the user and including the identified role and / or permission in the access token. The analytics server 108 can transmit the access token to the client device 110.

[0077] The analytics server 108 can include a chat interface on the platform. The chat interface can be or include a form into which users can input natural language queries. When the client device 110 accesses different pages of the platform, the analytics server 108 can include the chat interface on the user interface presenting the pages. The analytics server 108 can present the chat interface as an overlay of the pages such that a user can move the chat interface around and / or hide or expand the chat interface.

[0078] The user can provide an input natural language query into the chat interface. The natural language query can be or include a string of text with a syntax and schema. The syntax and schema may be a question or a schema for a request for data of the platform. The input natural language query can include a request for driver data regarding a plurality of drivers. For instance, the natural language query can include a request for driver speed information. The request can specify for the speed information to be for a particular set of drivers, such as drivers in a specific geographical location and / or that are a part of a particular fleet of vehicles (e.g., a fleet of vehicles under a corporate umbrella). The user can provide such inputs into the chat interface and select a submit button on the user interface of theplatform. The selection of the submit button can cause the client device 110 to transmit the natural language query to the analytics server 108.

[0079] The analytics server 108 can receive the natural language query and generate a response to the natural language query. To generate the response to the natural language query, the analytics server 108 can first generate a first query. For instance, the application 120 can provide the platform to the client device 110. The application 120 can receive the natural language query from the client device 110. The application 120 can pass the natural language query to the LLM 118. The LLM 118 can be or include one or more large language models, neural networks that each may be or include a transformer model, or any other type of model that is configured to generate responses to natural language queries. Large language models can also be called generative language models when described herein. The LLM 118 can receive the natural language query as a prompt. Based on the prompt, the LLM 118 can generate a first query. The first query can be in a format or schema (e.g., JavaScript Object Notation (JSON) or Yet Another Markup Language (YAML)) that can be used to query the database 122. The first query can include parameters (e.g., a first set of parameters) for retrieving data from the database 122. The parameters can identify the tables that contain the relevant data for generating a response and / or a time period of data to retrieve from the tables. The LLM 118 can generate the first query to have parameters for querying data that matches or corresponds with the data requested in the natural language query. For example, the LLM 118 can determine a context (e.g., requested information) of the natural language query and generate the first query to have parameters that correspond with the context. In another example, the LLM 118 can generate an embedding of the natural language query and use a similarity search algorithm to identify data in the database 122 that is relevant to the query by identifying embeddings of the data stored in the database 122 that are the most similar, or that are similar above a threshold, with the embedding of the natural language query. Operations or actions performed by the LLM 118 as described herein can be by the application 120, and vice versa.

[0080] The LLM 118 can use the first query to query the database 122. For instance, the LLM 118 can execute or use one or more application programming interfaces (APIs) that are configured to query the database 122. The LLM 118 can do so by querying the database 122 using the first query. Based on the first query, the LLM 118 can retrieve data that is relevant to generating a response to the natural language query received from the chat interface from the client device 110. For example, the LLM 118 can retrieve all data included in tablesidentified by the LLM 118 and / or that corresponds with a timestamp within a defined time period identified within the parameters of the first query.

[0081] The application 120 can generate a temporary database. The application 120 can generate the temporary database based on the data retrieved based on the first query. The temporary database can include a format or schema for storing the retrieved data. For instance, the temporary database can include the retrieved data organized into columns and rows. The application 120 can organize the data based on the locations of the retrieved data in the database 122. For instance, the application 120 can maintain consistency of the data between the database 122 and the temporary database by including data in the same columns in the database 122 in the same columns of the temporary database. The application 120 can similarly maintain consistency in data of the rows of the database 122. For instance, the application 120 can maintain consistency of data retrieved for particular drivers by organizing the data for the driver into the same row and separated between columns based on the attributes of the data. The application 120 can similarly organize data between drives to generate the temporary database. The application 120 can generate the temporary database in any manner.

[0082] Subsequent to generating the temporary database, the application 120 can use the LLM 118 to generate a second query. The application 120 can generate the second query by inputting the natural language query into the LLM 118, in some cases, with language indicating to query the temporary database instead of the database 122. The LLM 118 can receive the input prompt and generate the second query. The LLM 118 can generate the second query to include parameters (e.g., a second set of parameters) for retrieving data from the temporary database, such as by retrieving the requested data of the natural language query. The second query may not specify a time period because the first query already retrieved data based on a time period that is relevant for generating a response. The parameters of the second query can be more refined than the parameters of the first query, such as by identifying specific types of data or attributes of driver data to retrieve from the temporary database. The LLM 118 can generate the second query in a structured query language (SQL) that can be used to query the temporary database.

[0083] The LLM 118 can query the temporary database. The LLM 118 can query the temporary database using the second query. For instance, the LLM 118 can retrieve data from the temporary database that includes the data regarding the plurality of drivers requested in thenatural language query. The LLM 118 can delete or discard the temporary database responsive to querying the temporary database using the second query.

[0084] The LLM 118 can generate a response using the data in the temporary database. The LLM 118 can do so by first generating a data structure containing the data retrieved from the temporary database. The LLM 118 can generate the data structure to have a format of a vector, a database, a JSON format, a structure object, a non- structured format, etc. The LLM 118 can process the data in the data structure to generate a response to the natural language query. In doing so, the LLM 118 can generate the response to include the requested data and / or a format identified in the natural language query (e.g., a table, slide deck, image, etc.). The response can include a summary, a report, brief bullet points, or any other type of response or visual representation of the data structure that is relevant to the natural language query.

[0085] The application 120 can present the response at the client device 110. The response can be a visual representation of the data structure generated based on the second query. The application 120 can present the response in the chat interface of the platform presented at the client device 110. Because of the two-stage querying technique, the analytics server 108 can generate a response to the natural language query from the client device 110 without modifying the database 122 and / or while reducing hallucinations in the accuracy of the response.

[0086] In some cases, the application 120 can use the access credential provided to the client device 110 to control the data that can be included in the response. For instance, when transmitting the natural language query to the analytics server 108, the client device 110 can include the access credential in the message containing the natural language query. For example, the client device 110 can include the access credential in the authorization header of the message (e.g., HTTP request), such as with a bearer prefix. The access credential can indicate the type of data that the user submitting the request can access. Such access credentials may limit the user to receiving data for a particular set of drivers, for a particular organization, for a particular fleet, etc. In some cases, the access credentials may restrict the user from receiving such data. The application 120 can identify the access credentials and feed the access credentials into the LLM 118. The LLM 118 can determine whether the user is authorized to access the data requested in the natural language query based on the access credential, such as by determining whether the access credential authorizes the user to request data regarding at least one driver that is the subject of the request for driver data. Responsive to determining theaccess credential corresponds to (e.g., authorizes access to) at least one, or only responsive to determining the access credential corresponds to all of the drivers for which the natural language query requested, the LLM 118 can generate the first query to initiate the response generation process. However, responsive to determining the access credential does not correspond any, or responsive to determining the access credential does not correspond to all, of the drivers for which the natural language query requests driver data, the LLM 118 can generate a message or alert indicating the query could not be performed.

[0087] In cases in which the access credential only authorizes access to driver data of a subset of drivers for which the user requested driver data, the LLM 118 may only retrieve driver data for the subset of drivers. For example, the user at the client device 110 may only be authorized to access driver data for drivers within a particular geographic region (e.g., India). The natural language query may request driver data for all drivers not limited to any geographic region. The LLM 118 may determine to generate a response to the natural language query based on the access credential authorizing access to driver data of drivers in the geographic region. However, the LLM 118 may generate the first query to have parameters to only retrieve data regarding drivers in the geographic region and retrieve the driver data of the drivers in the geographic region from the database 122 only. The LLM 118 can generate a response using retrieved driver data, such as by generating a temporary database with the retrieved driver data, generating a second query to retrieve data from the temporary database, generating a data structure from the data retrieved from the temporary database, and then generating a response including a visual representation of the data structure. The application 120 can present the response at the client device 110, with or without an indication as to whether the response excludes data that the user is not authorized to view. In this way, the LLM 118 and the application 120 may operate together to safeguard private data in the database 122 to stop users from accessing data that the users are not authorized to access.

[0088] Additionally, in some cases, the analytics server 108 can enable users of the platform provided by the analytics server to navigate and / or query the platform using large language model processing. For example, instead of responding to a natural language query for driver data in the chat interface of the platform, the analytics server 108 can respond to a natural language query by navigating to a page of the platform containing the requested driver data of the natural language query. The analytics server 108 can identify a page containing the driver data and change the page of the platform from which the user provided the natural language query to the identified page that contains or otherwise corresponds to the requesteddriver data. The user can repeatedly input natural language queries into the chat interface while accessing the platform to change pages in this manner to view different desired driver data. Using the LLM can significantly reduce the time required to identify the page because the user may not have to select different drop down menus or different links to access the pages, which can take a large amount of time in many cases.

[0089] To facilitate navigation of the platform, the analytics server 108 can store a mapping of driver types to different pages (e.g., page identifiers) of the platform. The mapping can indicate the pages of the platform that are configured to present or otherwise store the different types of data. For example, the analytics server 108 can store a table with rows containing adjacent cells indicating a data type and a page of the platform that is configured to present data of the data type. The mapping can include any type of relationship between pages and data types.

[0090] The pages can be correlated or mapped with data types because, for example, the pages may be linked (e.g., configured to retrieve data from) with a data structure (e.g., the database 122) or a particular location in a data structure to retrieve the mapped data type. For instance, a page can correspond to a pointer associated with the page that points to or is an address of a location in the database 122 containing a particular type of driver data, such as alerts. For example, the uniform resource locator (URL) for the page may include the pointer that the analytics server 108 or the LLM 118 may identify and use to query data in the database 122 that corresponds to the pointer. In another example, the analytics server 108 can include a stored association between the page and a pointer that can be used to retrieve data from the database 122 upon retrieving the page. When the page is called or accessed by a user (e.g., the user accessing the platform through the client device 110), the analytics server 108 may present the page and use the pointer to automatically retrieve the driver data from the location in the database 122 identified by the pointer. By operating in this way, the pages themselves may not store the data, which can make updating and storing the data less computationally expensive than in systems that store data on the pages themselves.

[0091] The LLM 118 and / or application 120 can use the mapping to navigate through the platform based on natural language queries by users. For example, the user accessing the platform through the client device 110 can provide an input natural language query into a chat interface requesting driver data or an analysis of driver data from the analytics server 108. The LLM 118 can process the natural language query and determine an intent of the naturallanguage query to access a page of the platform containing the requested driver data or the driver data to be analyzed. The LLM 118 can determine a type of the requested driver data using internal weights and / or parameters of the LLM 118. Based on the determination, the application 120 or the LLM 118 can identify a page (e.g., a second page) of the platform that corresponds with the determined type from the stored mapping. The application 120 can retrieve the identified page from memory. The application 120 can present the identified page at the client device 110 in place of the page (e.g., the first page) that the user was viewing when the user input the natural language query.

[0092] In some cases, the application 120 can change pages to the page corresponding to the requested type of driver data responsive to determining an action type to change pages. For example, the LLM 118 can determine whether an intent of the natural language query from the user corresponds to taking an action. Examples of actions that the LLM 118 can determine or identify can include, but are not limited to, changing pages, reporting subscriptions, raising requests, sending emails, or changing configurations. In some embodiments, the LLM 118 can determined to take an action when a natural language query requests that an action be performed, but the query relates to an action that is not possible within the context of the platform or the user’s access credential indicates the action cannot be taken. In such cases the LLM 118 can generate a text reply indicating that the requested action is not available or otherwise could not be taken.

[0093] The application 120 can change pages to the page corresponding to the requested data type responsive to the LLM 118 determining to take an action to change pages. For example, the LLM 118 can determine an action to change pages based on the natural language query from the user. Responsive to the determination, the application 120 can use the mapping of pages to types of driver data to identify a page to retrieve and present at the client device 110. The application 120 can retrieve the page and present the page at the client device 110. The application 120 can additionally retrieve the data for which the page is configured to present data, such as based on the link associated with the retrieved page.

[0094] Subsequent to presenting the new page and / or the requested data of the page, the user may input a new natural language query about the data of the new page. The natural language query can be used to query for specific data points of the data presented on the page and / or an analysis of the data on the page. For example, the page can include a table of drivers and driver safety scores of the drivers. The user viewing the page can input a natural languagequery into a chat interface overlaying the page requesting to view a defined number (e.g., five) of the drivers shown on the page with the highest scores.

[0095] Responsive to receiving the natural language query, the LLM 118 can determine an action. The action can be to filter the data shown on the page. The LLM 118 can determine the action based on the content and / or context of the natural language query, such as by determining the natural language query indicates to highlight or specify specific types of data shown on the page.

[0096] Responsive to the determination to filter the data shown on the page, the LLM 118 can generate a query. The LLM 118 can generate the query using the natural language query as input. For instance, the LLM 118 can generate the query to have attributes for retrieving data that is relevant to answering the natural language query. The LLM 118 can generate the query to retrieve data with attributes identified in the natural language query, such as specific types of data (e.g., scores, number of alerts, average speed of the speed limit, etc.). The LLM 118 can generate the query in a schema that enables querying the data on the page itself, such as in an SQL format if the data shown on the web page is in a table.

[0097] The LLM 118 can query the page. The LLM 118 can query the page using an API or another tool and the query that the LLM 118 generated. Based on the query, the LLM 118 can retrieve data that is relevant to generating a response to the natural language query. The application 120 can generate a data structure (e.g., a table, a vector, a matrix, etc.) with the retrieved data from the web page. The LLM 118 can process the data of the data structure to generate a visual representation (e.g., a graph, table, summary, list, etc.) of the data structure. The LLM 118 can generate the visual representation in a format identified in the natural language query or based on a format that the LLM 118 determines based on the natural language query. For example, the natural language query can indicate to generate a list of the drivers with the top five driver scores. Based on the natural language query, the LLM 118 can retrieve data from the data structure and generate a response in the requested list format. The application 120 can update the page with the visual representation of the data structure, such as by replacing what was originally presented on the page with the new visual representation of the data structure.

[0098] The LLM 118 and the application 120 can operate in this manner to update or filter the data on the page any number of times to respond to user queries. Because the query for filtered data is performed directly on the page rather than the database storing the data, thequery has similar technical advantages to the technical advantages discussed herein regarding generating a temporary database to generate a response to a query. For example, querying a page can avoid improper injections of data into a database storing data, can reduce the querying time to generate a response to the query because there is less data to query, and can additionally reduce hallucinations because the initially relevant data is already available and the large language model will not be biased by other data that is not relevant to generating a response.

[0099] Referring now to FIG. 2, FIG. 2 illustrates a flow of a method 200 executed by a data processing system. The data processing system can be or include a remote computing system (e.g., the analytics server 108) configured to store driver data of different vehicles and / or provide a platform through which users can view such driver data. The method 200 is shown to include steps 202-218. However, other embodiments may include additional or alternative steps, or may omit one or more steps altogether. Different steps can be performed by different computing systems (e.g., computing systems of vehicles can perform one or more of the steps 202-218 and / or the analytics server can perform one or more of the steps 202-218) and / or the different computing systems and can operate together to perform individual steps of the steps 202-218.

[0100] In step 202, the data processing system can receive an access credential. The data processing system can receive the access credential from a client device. The access credential can be an access credential of a user accessing the client device. The access credential can indicate any permissions that the user (e.g., an account of the user) has. The access credential can be, for example, a JSON web token. The data processing system can receive the access credential in an authorization header of a request (e.g., an HTTP request). For instance, the data processing system can identify or extract the access credential from the authorization header of a request that the client device transmits to the data processing system.

[0101] For example, the data processing system can host a platform. The platform can include multiple pages that can be navigated to view details regarding driver data of drivers for which the platform manages data. A client device can communicate with the data processing system to access the platform, such as by establishing a connection with the data processing system over a network. In doing so, a user of the client device can log in to the user’s account with the platform. The account can have various permissions and / or roles. Responsive to authenticating the user for the login to the account for a session (e.g., communication session) with the data processing system, the data processing system can generate an access token forthe session. The access token can indicate a role of the user and / or permissions of data that the user can access. The data processing system can determine such permissions and / or the role of the user by retrieving the data from a data structure storing data for the user’s account. The data processing system can transmit the access token to the client device. Subsequently, when the client device transmits requests and / or messages to the data processing system for the session, the client device can include the access token in the requests and / or messages.

[0102] In step 204, the data processing system can receive a natural language query. The data processing system can receive the natural language query from a chat interface that the data processing system provides to the client device on the platform. The natural language query can be or include a string of text. The natural language query can include or be a request for driver data regarding a set of drivers (e.g., a plurality of drivers). For example, the natural language can be a string of text requesting driver safety data for drivers in a particular geographic language (e.g., Europe, the United States, and / or India). The data processing system can receive the natural language query from the client device accessing the platform in a message.

[0103] In some cases, the data processing system can receive the natural language query in the same message as the access credential. For example, the client device can generate a message containing the natural language query in the payload of the message and insert the access credential in the authorization header of the message. The client device can transmit the message to the data processing system. The data processing system can identify or extract the natural language query and / or the access credential from their respective locations in the message.

[0104] In step 206, the data processing system can determine whether the access credential corresponds with the set of drivers. The data processing system can do so by determining whether the permissions and / or the role identified in the access credential authorize the user that initiated the query to access the requested data for the individual drivers of the set of drivers. For example, the access credential can authorize the user to query for data regarding specific types of driver data and / or driver data of specific drivers. Examples of different types of driver data can include numbers of accidents, number of tickets given, average speed, average speed above a speed limit, etc. Examples of drivers for which the user is authorized to access driver data can include drivers in particular geographic locations (e.g., Europe, the United States, North America, India, etc.) or drivers with specific demographicdata (e.g., drivers within certain age ranges, drivers of specific genders, etc.). The access credential can indicate the types of driver data and / or the drivers for which the user is authorized to receive data. The data processing system can execute a large language model using the natural language query as input to determine the drivers that are relevant for generating a response to the natural language query.

[0105] The data processing system can determine whether the user is authorized to retrieve data for each of the identified drivers based on the access credential. To do so, depending on the configuration, the data processing system can determine if the access credential enables retrieval of driver data for at least one of the identified drivers or retrieval of driver data for all of the identified drivers. For example, the data processing system may determine to generate a response to the natural language query responsive to determining the access credential authorizes data retrieval of at least one of the set of drivers or only responsive to the access credential authorizing access to data of all of the set of drivers identified for the natural language query.

[0106] In some cases, the data processing system can determine not to generate the response responsive to determining the access credential does not allow access to the requested type. The data processing system can do so, for example, by executing the large language model to determine the type of the requested data. The data processing system can determine whether the access credential allows access to the type of the requested data. The data processing system can determine whether to generate a response based on whether the access credential authorizes access to the requested type of data.

[0107] In some embodiments, the data processing system can determine whether the access credential authorizes access to the requested data for the set of drivers using a hierarchical data structure. For example, the data processing system can store a hierarchical data structure in memory. The hierarchical data structure can include one or more nodes that each represent a driver or otherwise an entity of an organization managing the drivers. For example, the hierarchical data structure can include leaf nodes that represent the individual drivers. The nodes representing the drivers can be connected within the hierarchical data structure (e.g., by links) to parent nodes. The parent nodes can each have a one-to-many connection profile in which individual parent nodes can be connected to multiple nodes representing the drivers. The parent nodes can represent different aspects of the organization, such as a grouping of the connected drivers, a segment of the organization, a location of thedrivers, etc. The parent nodes can be further connected to grandparent nodes in a tier above the parent nodes that further group different parent nodes together. The hierarchical data structure can include any number of tiers and / or connections between tiers.

[0108] The nodes of the hierarchical data structure can correspond to profiles of the drivers or entities that the nodes represent. For example, the data processing system can store data for the different drivers in a database. The data processing system can store the data in profiles for the respective drivers. The profiles can represent separate data structures within the database. The profiles can be, for example, separate lines within a table, separate tables, nodes of a graph, etc. The profiles can each store driver data for the driver represented by the profiles and / or the entities represented by the profiles. The profiles can include the name of the driver or entity represented by the profile and / or any attributes of the driver or entity.

[0109] The data processing system can use the hierarchical data structure to determine whether to generate a response for the query. For example, the different profiles and / or nodes of the hierarchical data structure can each correspond to a different role or permission. The data processing system can determine if the access credential received with the natural language query has the permission and / or role to retrieve data for the individual nodes of the hierarchical data structure. For example, the natural language query may request driver data for drivers in India. Based on the request, and using the large language model, the data processing system can identify a node of the hierarchical data structure that corresponds to India. The node may be linked to nodes of drivers that drive in India. The data processing system can determine whether the access credential matches or authorizes data retrieval for the node or profile representing the India geographic area. Responsive to determining a match with the node for India and / or that the access token corresponds with the node for India, the data processing system can retrieve data for the drivers linked with the India node and / or from the profiles for the drivers linked with the node for India. However responsive to determining there is not a match and / or that the access token does not correspond with the node for Indian, the data processing system can determine not to generate a response to the natural language query.

[0110] Responsive to determining not to generate a response to the natural language query (e.g., responsive to determining the access credential does not correspond with the set of drivers), at step 208, the data processing system can generate an alert. The data processing system can generate the alert to indicate a response could not be generated. In some cases, thedata processing system executes the large language model to generate the alert. In such cases, the large language model can generate a text response indicating why the alert could not be generated, such as because the user is not authorized to receive the requested data. The data processing system can transmit the alert to the client device. The client device can present the alert on a display or otherwise in the chat interface displayed on the client device.

[0111] Responsive to determining the access credential corresponds to the set of drivers, in step 210, the data processing system can generate a first query. The data processing system can generate the first query using a first large language model. The first large language model can be the same or a different large language model from the large language model that the data processing system used to determine whether to generate a response to the natural language query. The data processing system can input the natural language query into the first large language model. The data processing system can execute the first large language model based on the natural language query. Based on the execution, the first large language model can determine to retrieve data from a database stored in memory that is relevant for generating a response to the natural language query.

[0112] The first large language model can determine to retrieve data from the database using various internal decision-making mechanisms or policies. For example, the first large language model can determine a context of the natural language query to answer a question. The first large language model can determine, based on the context, that the large language model has not been trained with the necessary or relevant data for answering the query. Based on the determination, the first large language model can determine to retrieve data from the database to generate a response to the natural language query.

[0113] The first large language model can generate the first query. The first large language model can generate the first query to contain parameters (e.g., first parameters) for retrieving data from the database. The parameters can include, for example, an identification of one or more tables within the database to query, fields from which to retrieve data (e.g., name license number, vehicle type, safety information, average speed, number of tickets, number of accidents, etc.), conditions for the data to be retrieved (e.g., active status, location of the driver, etc.). The first large language model can configure the first query to have such parameters by identifying the contents of the natural language query and generating the parameters for the first query using internal weights and / or parameters of the first large language model the contents. In one example, the natural language query can request trafficviolation data regarding drivers in Utah. Based on the natural language query, the first large language model can generate the first query to include parameters identifying a table containing traffic data for drivers in Utah and identify or extract the data requested in the natural language query. In some cases, the first query can be to retrieve all of the data from tables identified for generating the first query.

[0114] In an example, the natural language query can include a date range. The date range can indicate a time range for which to retrieve data. The date range can include a start date and an end date. For instance, the natural language query may be “show me the number of traffic violations experienced by drivers in Indiana in the year 2021.” The first large language model can identify the date range to be from the beginning to the end of the year 2021. The first large language model can include the date range in the first query as a parameter to use for retrieval such that the first query only retrieves data from within the date range.

[0115] The first large language model may be trained or configured to generate the first query in a schema that enables or facilitates data retrieval from the database of the database that stores driver data for different drivers. For example, the first large language model can be trained based on data stored in the database or similarly structured databases. Based on the training, the first large language model can learn relationships between data and how the data can be queried. The first large language model can additionally be trained based on queries into such databases to learn how to format the queries into the database. The training can teach the first large language model the locations of the data within the database and / or the schema of the database that can be used to retrieve data to generate responses to natural language queries.

[0116] Subsequent to generating the first query, the first large language model can execute the first query within the database to retrieve data that corresponds with the parameters of the first query. For example, the first large language model can execute or use one or more APIs to query the database. In doing so, the first large language model can cause the one or more APIs to use the first query to retrieve data that is relevant to responding to the natural language query by retrieving data that corresponds to the parameters of the first query. The APIs can retrieve the data from the database using the first query.

[0117] In some cases, the access credential associated with the natural language query may only authorize access to data of a subset of the drivers for which the natural language query requested data. In such cases, the first large language model can adjust, update, orotherwise include, in the parameters, limitations for the drivers for which to retrieve data. The first large language model can identify drivers for which the user is authorized to receive data and tailor the parameters for the first query to only retrieve data for the identified drivers. In some cases, the first large language model can do so based on a prompt that initiated the generation of the first query indicating the drivers for which to retrieve data and / or not to retrieve data. In some cases, the first large language model can determine the parameters after determining the drivers for which to retrieve data and / or that are not restricted from retrieving data. In using parameters in queries in this way, the first large language model can avoid retrieving data that a user is not authorized to access.

[0118] In step 212, the data processing system can generate a temporary database. The data processing system can generate the temporary database from the data retrieved using the first query. The data can be stored in a structured format, such as JSON, Extensible Markup Language (XML), a structured object, etc. The temporary database can have an SQL schema such that the temporary database can be queried using a SQL query. The temporary database can include one or more rows and / or columns. The rows or columns can match or contain similar data and / or fields to the rows and / or columns of the database of the first query. In some cases, the temporary database can have a differing format or schema from the database of the first query. In such cases, the data processing system can reformat the data into a format that is compatible with a table including rows and columns such that the temporary database can be queried using an SQL query. The data processing system can do so, for example, by inputting the retrieved data into a large language model (e.g., the first large language model or a different large language model) trained to format and / or organize data into a table. The data processing system can store the temporary database in a cache or temporary memory such that the temporary database can be discarded subsequent to use.

[0119] In step 214, the data processing system can generate a second query. The data processing system can generate the second query based on the natural language query. For instance, the data processing system can input the natural language query into a second large language model. The second large language model can be trained to generate queries to use to retrieve data from temporary databases generated from retrieved data. For instance, the second large language model can be configured or trained to generate SQL queries. The second large language model can generate a second query in a SQL schema based on the natural language query. The second large language model can generate the second query to include attributes(e.g., second attributes) for retrieving data from the temporary database that is relevant for generating a response to the natural language query.

[0120] The second large language model or the data processing system can use the second query to query the temporary database. For instance, the second large language model or the data processing system can execute the second query within the temporary database to retrieve data from the temporary database according to the attributes of the second query. The second large language model or data processing system can use one or more APIs (e.g., the same APIs that were used to perform the first query or different APIs) to perform the second query in the temporary database. The second large language model can retrieve driver data from the temporary database based on the second query.

[0121] The data processing system can remove the temporary database from memory responsive to executing the second query within the temporary database. For instance, after the data processing system retrieves data from the temporary database, the data processing system can delete or discard the temporary database from memory. Doing so can help the data processing system reduce the memory resources that may be required to perform this process over time by avoiding maintaining new databases that are generated for each natural language query that the data processing system receives. In some cases, the data processing system can discard the temporary database responsive to presenting a response to the natural language query.

[0122] In step 216, the data processing system can generate a data structure. The data processing system can use the second large language model to generate the data structure to include the retrieved data in a structured format, such as in a JSON format, an XML format, and / or as structured objects. The data structure can be formatted in a manner that the second large language model or a different large language model can use the data to generate a response to the natural language query.

[0123] The data processing system can generate a visual representation based on the data in the data structure. To do so, the data processing system can process the data of the data structure using the second large language model to generate a visual representation of the data in the data structure according to the natural language query. For example, the second large language model can determine a format in which to generate a response based on the natural language query. Such formats can include, a table, a report, a graph, a text summary, etc. The second large language model can determine the format for the response by determining thecontext or intent in the natural language query. The second large language model can determine the format, retrieve the data from the data structure, and generate a visual representation of the data in the data structure in a format appropriate for responding to the natural language query (e.g., in a format requested in the natural language query).

[0124] In step 218, the data processing system can present the visual representation of the data structure at the client device. To do so, the data processing system can transmit the visual representation of the data structure to the client device. The visual representation can be or include a response to the natural language query. The visual representation can include the requested driver data or otherwise requested data that the user is authorized to access as indicated in the access credential of the request.

[0125] In some cases, the data processing system can generate recommendations. The data processing system can generate the recommendations based on previous queries. For example, the data processing system can input a thread of messages between the data processing system and the user, in some cases including the response generated by the most recent natural language query, into a large language model. The large language model can process the thread of messages can generate recommendations of further queries that may request further data similar to the last natural language query for driver data. In some cases, the large language model can generate recommendations of software or platform features that can be used to improve the speed of responding future queries. For example, the data processing system may store reports for different types of data that the large langue model generates at set intervals. The large language model can generate a recommendation to request such a report with a next natural language query or to otherwise request access to such reports to receive the data of the reports faster without relying on the system to regenerate the data of the reports.

[0126] In a non-limiting example, the data processing system can perform the method 200 to identify drivers that have a particular driver characteristic. For example, a user at a client device can input a natural language query requesting a list or identifications of a set of drivers that have one or more particular driver characteristics. Examples of such driver characteristics can include characteristics regarding drivers’ driving (e.g., number of tickets, number of accidents, average speed, average speed above the speed limit, etc.). In some cases, the characteristics can be criteria regarding driver characteristics, such as to identify a set of drivers that have experienced at least a threshold number of accidents within the past year. Thedata processing system can receive the natural language query and determine the driver characteristics from the natural language query. The first large language model can identify one or more tables in the database stored in memory of the data processing system that include data associated with or identifying the driver characteristics. For instance, if the driver characteristic is a number of accidents, the first large language model can identify any tables in the database that include values indicating the number of accidents different drivers have had.

[0127] The first large language model can generate a first query and configure the first query with parameters for querying the identified tables based on the identified driver characteristic or driver characteristics. The parameters can limit or indicate other restrictions or attributes that are relevant for responding to the natural language query. For instance, the natural language query can specify one or more characteristics of drivers that are relevant for the response. The first large language model can configure the first query parameters indicating to only retrieve data for drivers that have each characteristic. For example, the natural language query can request numbers of tickets for drivers located in North America. Based on the natural language query, the first large language model can configure the first query with parameters specifying to only retrieve data for drivers located in North America (e.g., have a North America value in a location field of a table containing the ticket data). The first large language model can configure the first query based on any number of characteristics depending on the natural language query for which the data processing system is generating a response.

[0128] The first large language model can use one or more APIs to query the identified tables in the database. The first large language model can do so, for example, using the first query. For example, the first large language model can call an API configured to query the database responsive to generating the first query and / or responsive to determining a database query is necessary for generating a response to the natural language query. The first large language model can use the API to retrieve data from the tables of the database that contain driver data of drivers that have a characteristic identified from the natural language query. For instance, the first large language model can use the first query to retrieve driver data of drivers generated within a defined time period based on the natural language query requesting driver data from the defined time period. The first large language model can do so by querying the database according to the parameters of the first query.

[0129] The data processing system can generate a temporary database from the retrieved driver data. The data processing system can do so using the first large language model. In doing so, the data processing system can format the temporary database into a schema that can be queried using an SQL format. For instance, the data processing system can format the temporary database to have rows containing driver data for respective drivers and / or columns containing driver data for attributes of the drivers. The temporary database can be significantly smaller than the database for which the data processing system stores all the driver data that is available to the data processing system because the temporary database only contains data that is relevant to the natural language query based on which the temporary database was generated.

[0130] The data processing system can generate a second query. The second query can be a query for querying the temporary database. The data processing system can generate the second query using the second large language model. For instance, the data processing system can input the natural language query (e.g., the same natural language query as was input into the first large language model to generate the first query) into the second large language model. The second large language model may be configured to generate queries for data that satisfy some criteria specified in the natural language query.

[0131] In one example, the natural language query may be a query for drivers that have a driver score (e.g., a driver’s safety score) above a threshold or above a defined number. The first large language model can retrieve driver scores for drivers from the database stored in memory of the data processing system and the data processing system can generate the temporary database to include the retrieved driver scores. The second large language model can generate the second query to retrieve the driver scores from the temporary database. In some cases, the first large language model can retrieve data from the temporary database and generate the driver scores as a function of the retrieved data for the respective drivers. Responsive to retrieving and / or generating the driver scores using the second query, the second large language model can compare the driver scores (e.g., by executing or using a tool configured to perform different computations). Based on the comparison, the second large language model can identify any drivers associated with the driver scores above the threshold or the defined number. The second large language model can generate a data structure containing the retrieved and / or generated data identifying the drivers associated with driver scores above the threshold. The second large language model can generate a visual representation of the data structure, such as a list or summary identifying drivers that satisfythe criterion of being associated with a score above the threshold or defined value. The data processing system can present the visual representation on the user interface of the platform from which the data processing system received the natural language query. The second large language model can use the temporary database to identify drivers based on data of the drivers satisfying any criterion or otherwise process or analyze data of the temporary database in any way to generate a response to a natural language query.

[0132] By implementing the dual-query process described herein, the data processing system can first perform a coarse query of data to use to respond to a natural language query for driver data from a large database and then generate a second more refined query to retrieve the data that is more applicable to generating a response to the natural language query. By doing so, the data processing system can reduce the likelihood of hallucinations and make the answers more precise compared with systems that do not first create a temporary database and perform a second query. These benefits arise because the second query can include parameters for retrieving data that is more relevant for generating a response, thus reducing any biasing by less relevant data and avoiding attempts by the large language model to “fill in the gaps” and make up data to use to generate the response.

[0133] FIG. 3 illustrates a sequence diagram for implementing a machine learning architecture for querying a database, according to an embodiment. FIG. 3 can depict a sequence 300 of responding to a natural language query using the systems and methods described herein. The data processing system can be or include a remote computing system (e.g., the analytics server 108) configured to store driver data of different vehicles and / or provide a platform through which users can view such driver data. The sequence 300 is shown to include various steps and / or devices and / or systems for generating a response to a query. However, other embodiments may include additional or alternative steps, or may omit one or more steps altogether. Additionally, different steps can be performed by different computing systems and / or components of the data processing system than illustrated in FIG. 3 and / or the different computing systems and can operate together to perform individual steps of the sequence 300. The data processing system can perform the steps of the sequence 300 when performing the method 200, for example.

[0134] In the sequence 300, a frontend system 302 can present a user interface of a platform hosted by the data processing system. The frontend system 302 can present the user interface at a client device being accessed by a user. The user interface can include a chatinterface through which a user can submit natural language queries to communicate with the data processing system. Such communication can involve requests for driver data stored by the data processing system, requests for analyses of such driver data, conversational messages, etc. The frontend system 302 can receive such natural language queries as the user provides the natural language queries and pass the natural language queries to a load balancer 304 of the data processing system. In one example, a user can ask a question at the frontend system 302 which may be passed through the load balancer 304 elastic load balancer to the data processing system architecture. Each LLM of the architecture may be provided with the user question and chat history (n number of previous question and answers between the data processing system and the user before this question).

[0135] The load balancer 304 can be or include executable instructions stored in memory of the data processing system to balance the number or amount of messages from different users that are being processed at one time. For instance, the data processing system can simultaneously host the platform for different users at once. The users can each transmit natural language queries through the frontend system 302. Because each natural language query can take time to respond, the load balancer 304 can create a queue of the messages such that the data processing system is not attempting to generate a response to multiple messages at once or more messages at once than the data processing system can handle. Such can be beneficial in this case, for example, to avoid using data retrieved to respond to one natural language query to generate a response to a different natural language query (e.g., avoid using the incorrect temporary database to respond to a natural language query). Responsive to determining the data processing system has the processing resources available to generate a response to a natural language query received from the frontend system, the load balancer can pass the natural language query to a router LLM 306.

[0136] The router LLM 306 can be a large language model that is configured to determine a domain for the natural language query. The domains can be one of the following: (i) a help domain 308 that indicates to answer a question, (ii) an external domain 310 that indicates that the answer cannot be answered because it is out of the scope of the platform, and (iii) a domain data domain 312 that indicates that to answer the natural language query, data needs to be queried from a database (e.g., the database 122) stored by the data processing system. In some cases, the router LLM 306 can determine the natural language query is a general query and the router LLM 306 can facilitate a conversation by generating responses314 to communicate with the user until determining a natural language query from the user falls in one of the domains 308-312.

[0137] The router LLM 306 can determine how to route the natural language query by determining an intent of the natural language query. For example, the LLM 306 can process the natural language query used as an input to generate the intent of the natural language query. The intent can be a purpose or objective of the natural language query. The LLM 306 can determine the intent based on keywords or phrases, contextual understanding, syntax and grammar, entity recognition, etc., of the natural language query. The LLM 306 can store a mapping of intents to the different domains and use the mapping to route the natural language query to the particular domain mapped to the intent of the natural language query.

[0138] The router LLM 306 can route the natural language query into one of the domains 308-312. In response to determining to route the natural language query into the external domain 310, the router LLM 306 can generate (e.g., automatically generate) a message 316 indicating the question is out of scope of what the data processing system can provide to the user. In response to determining to route the natural language query into the help data domain 308, the router LLM 306 can route the natural language query to a help architecture 318 including one or more large language models that can generate responses to the natural language query regarding assisting with use of the platform. In response to determining to route the natural language query into the domain data domain 312, the router LLM 306 can call an LLM tool 320 to initiate a process to query the database containing driver data stored in memory of the data processing system.

[0139] The LLM tool 320 can be or include one or more large language models or a set of instructions configured to generate a first query to generate a response to a natural language query. For example, the LLM tool 320 can process the natural language query using a first LLM to identify or determine a date range or time frame of data to use to generate the response. The first LLM can identify or determine the date range or time frame based on the context or content of the natural language query. For instance, the natural language query can include a string of text of “tell me the number of accidents that occurred in Colorado in 2019.” The first LLM can identify the date range or time frame to be between the first day of 2019 and the last day of 2019. In cases in which the first LLM does not identify or determine a time range from the natural language query, the first LLM can determine the date range or time range to be a default configuration (e.g., last week, last year, etc.) that the first LLM is configured to identifyor determine in such cases. The LLM tool 320 can use a second LLM to identify one or more tables within the database that contains the data needed to generate a response. The LLM tool 320 can input the natural language query to the second LLM and the second LLM can query the database to identify one or more tables that contain data to use to generate a response to the natural language query. The LLM tool 320 can generate the first query to contain parameters of the time range and / or identifications of the tables.

[0140] In some cases, prior to generating the first query, the LLM tool 320 can determine whether the user that submitted the natural language query is authorized to access data in the tables identified by the LLM tool 320 for generating a response. The LLM tool 320 can do so based on an access token that is included in the message that the frontend system 302 receives containing the natural language query. The LLM tool 320 can determine the tables, if any, from which the user is authorized to access data based on determining the access credential includes or corresponds to permissions authorizing access to the tables. Responsive to determining the user is authorized to access data from at least one of the tables or responsive to determining the user is authorized to access data from each of the tables, the LLM tool 320 can generate the first query. In cases in which the user is only authorized to access a portion of the tables, the LLM tool 320 can generate the first query with attributes that only query the tables for which the user is authorized to access data.

[0141] The LLM tool 320 can use the first query to query the database. To do so, the LLM tool 320 can use one or both of APIs 322 and / or 324. For instance, the API 322 can be or include a stats list API configured to retrieve stats from the database. The API 324 can be or include an alerts StatLite API that is configured to retrieve data regarding alerts that occurred for different drivers while driving. The APIs 322 and 324 may be configured to query different tables of the database. The LLM tool 320 can use the APIs 322 and / or 324 to query the database according to the parameters (e.g., the time range parameter and / or the table identification parameter) of the first query, such as by using the APIs 322 and 324 to query the tables for which the APIs 322 and 324 are configured to query. The LLM tool 320 can query the database according to the parameters of the first query to retrieve driver data that can be used to generate a response to the natural language query.

[0142] The LLM tool 320 can generate a temporary database 326. The LLM tool 320 can generate the temporary database 326 with the driver data retrieved using the first query. For example, the LLM tool 320 can organize the data in the temporary database 326 in rowsthat correspond to particular drivers and columns that correspond to specific types of data, or vice versa. The LLM tool 320 can generate the temporary database 326 using an LLM or otherwise using instructions of the LLM tool 320.

[0143] The LLM tool 320 can generate 328 a second query. The second query can be a query in a schema that can parse or retrieve data from the temporary database 326. The schema can be in a SQL querying format. The LLM tool 320 can generate the second query using the natural language query as input into an LLM of the LLM tool 320, in some cases with an example SQL query from which the LLM can copy the format of the query (e.g., in a single shot or few-shot learning technique). The LLM tool 320 can execute the LLM based on the input to generate the second query to have attributes for retrieving data from the temporary database 326 that is relevant for responding to natural language query. The LLM tool 320 can attempt to use the second query to retrieve data from the temporary database 326.

[0144] In some cases, execution of the second query may not result in retrieving data or enough data to respond to the natural language query. For instance, there may not be enough driver data available to generate a response regarding various driving characteristics about a driver or a group of drivers. In such cases, the LLM tool 320 can generate a standard response 330 (e.g., a preconfigured response) indicating that the natural language query could not be answered. The LLM tool 320 can generate the response using a large language model or by retrieving a template response from memory responsive to the determination. The LLM tool 320 can present the response at the chat interface through the frontend system 302.

[0145] In cases in which execution of the second query can result in retrieval of data that can answer the natural language query, the LLM tool 320 can perform 332 the second query on the temporary database 326 to retrieve data for generating the response and generate a data structure with the retrieved data. The LLM tool 320 can generate the data structure by inserting the retrieved data in one or more fields of the data structure. The data structure can be a vector, table, matrix, etc. The LLM tool 320 can use an LLM to generate a visual representation of the data in the data structure, such as in default format (e.g., a table, summary, graph, slide deck, etc.) or in a format indicated in the natural language query. The LLM tool 320 can present 336 the visual representation at the chat interface through the frontend system 302. In some cases, one or more, or all, of the LLMs used in the sequence 300 may receive the thread of messages between the user and the data processing system that were generatedbefore receiving the natural language query to provide additional context for any of the outputs generated by the respective LLMs.

[0146] In one example a natural language query may be routed to the domain data domain 312. Based on the natural language query, the LLM tool 320 can determine a required time frame for the natural language query. In the situation in which the natural language query does not specify a time frame, the LLM tool 320 can identify a default time frame, or the LLM tool 320 can infer (e.g., using an LLM of the LLM tool 320) a time frame based on a context of previous interactions with the user. Based on the time frame specified by the natural language query (or as the default, or using the context), the LLM tool 320 can identify the corresponding time range information: start date, end date, and interval. In some cases, the time ranges may be specified with respect to the user’s time zone, or alternatively, a time zone corresponding to a local time zone for the driver or drivers that are the subject of the query may be used.

[0147] The LLM tool 320 can extract the driver and alerts data that are relevant to the natural language query from the database of the data processing system containing driver and alerts data, in part by filtering the data using the time range information. In some cases, the driver and alerts data may be extracted in parallel using two different APIs. Using the extracted data, the LLM tool 320 can generate a temporary database (e.g., the temporary database 326) containing a table. In some cases, the LLM tool 320 can generate such temporary databases for each natural language query and then remove the temporary database after LLM tool 320 generates or presents responses to the queries based on which the temporary databases were generated. In some cases, the LLM tool 320 can allow the temporary databases to persist for a period of time (e.g., a defined period of time or for the duration of a login session by a user) such that subsequent queries by the same user (e.g., during the same login session) may be answered more efficiently by querying the temporary database.

[0148] Next, the LLM tool 320 (e.g., through a generate SQL LLM) can process the table schema of the generated temporary database together with the natural language query to determine if a responsive answer can be determined based on the temporary database or, alternatively, that the available data is not able to provide a relevant answer. Responsive to determining, by the LLM tool 320, that the available data is not sufficient to return a relevant answer may be referred to as an “out of scope” determination. In case the user query is out of scope, a corresponding disclaimer or message may be provided to the user through the frontendsystem 302. In other cases, the LLM tool 320 can generate an SQL query to fetch the data from the temporary database. The LLM tool 320 can generate the SQL query such that the SQL query can be executed on the local database to fetch data within the temporary database that is responsive to the natural language query. The data returned by the SQL query may be referred to as “fetched data.” The fetched data and the original natural language can then be processed by the LLM tool 320 (e.g., via a summarizer LLM) to provide a response to the natural language query. Optionally, the LLM tool 320 can add disclaimers and / or notes are added to the response and presented to the user.Non-limiting Examples

[0149] In a non-limiting example, a safety manager accessing a platform provided by a data processing system (e.g., the analytics server 108) implementing the systems and methods described herein may want to ask questions about the driver safety system and its configuration. The answers may help the safety manager configure and understand the system and make better use of it. Such safety manager questions may include, for example: i) When does the system trigger a speeding alert? ii) Tell me about Driver Stars? iii) What is virtual coaching? iv) Tell me about hard turn alerts? v) Tell me about the D430 device? vi) What are advisory alerts? vii) How do speed violations work? viii) Can I edit a custom role?

[0150] Answers to these questions may be sourced from help documentation, installation guide, or similar documentation resources.

[0151] Some questions of interest to a safety manager may not be available in help pages or similar resources. For example, answers to each of the following questions may depend on driving data that was recently generated by drivers in the fleet. i) Who is my safest driver? ii) How many severe alerts do we have today?

[0152] In cases in which the data processing system only has access to help pages and similar resources, the answers to these types of questions may return a response that indicates that the data processing system is only configured to answer questions based on help pages and similar resources. In some embodiments, the data processing system may indicate where such data may be accessed on the platform, since these questions may be answerable based on preconfigured views into fleet data.

[0153] Additionally, the safety manager may want to get information about drivers and their safety alerts. The information may help the safety manager evaluate risk across the fleet, prioritize coaching activities, and the like. According to certain aspects of the present disclosure, the data processing system may be configured to answer questions about driving data and alert statistics data.

[0154] Safety manager questions or requests regarding driving data and alert statistics data may include:1. Give me the list of drivers with speed violation last week but not this week.2. Who are the drivers with the top 5 GreenZone score today / last week?3. Who are the drivers that have been speeding in the rain for the last 30 days?4. Can you break down my alerts last week by alert type?5. Which top 5 drivers have the most GreenZone minutes last month not counting the unknown driver?6. (For a company having multiple driving groups) Who are my top 10 drivers with highest GreenZone score across group ABC and group DEF over the last 30 days.

[0155] To handle questions from a safety manager about driving data and other statistics data, the data processing system can return answers that are properly within scope. Broadly, the safety manager can query the data processing system about driving data and alert statistics data for which the safety manager already has access. The data processing system may not return information about drivers who may be associated with the same customer account, but which are not accessible to the safety manager who is making the query (e.g., such as based on the access credentials provided for the safety manager indicating the permissions the safety manager has when making queries).

[0156] Methods to enforce proper scoping may include reference to an implementation setup. Broadly, an implementation setup may be a “flat” model or a “multi-level” model. A flat model can refer to a single “tenant” (customer, fleet), that has all its assets (safety devices,vehicles) in a single level or group. A multi-level model can refer to a configuration for a customer or fleet that is arranged into multiple levels or groups. Further, a multi-level model may refer to an implementation setup with a “parent / child / child relationship,” “a group hierarchy,” and the like.

[0157] For a “parent / child / child relationship” configuration, there may be a single parent with one or more “child” accounts. Each parent and child account may be represented as a “tenant” within the database. For example, there may be two child accounts, referred to as Childl and Child2. In addition, each child account may have a child. In the current example, for Child2 there may be two child accounts, referred to as Child21 , Child22.

[0158] For a “group hierarchy” configuration, within a single “tenant” there may be multiple “groups.” Groups may be considered a group, subgroup, or leaf, depending on its relationship to a “root” account. For example, a tenant may be organized into 7 levels of groups. There may be 1 group (Leafl l l) that is associated with Subgroup 11 of Group 1. There may be 2 groups (Leaf 121, Leaf 122) that is associated with Subgroup 12 of Group 1. There may be 1 group (Leaf 21) that is associated with Group 2, where Group 2 is not further divided into subgroups. Finally, there may be 1 group (Leaf3), that is not associated with a group, but is instead itself on the same level as Group 1 and Group 2.

[0159] The implementation setups described above may be visualized as follows:1. Flat model a. 1 tenant has all its assets under 1 level.2. Multi-level model a. Parent / Child / child relationship i. Both Parent and child are different tenants1. Example a. Parent i. Childl ii. Child21. Child212. Child22 b. Group hierarchy i. Within the same tenant, a company could create 7 levels of groups. ii. Example1. Root a. Group 1 i. Subgroup 111. Leaf 111 ii. Subgroup 121. Leaf 1212. Leafl22 b. Group2 i. Leaf21 c. Leaf3

[0160] Using the above illustration as an example, a safety manager may be assigned to Subgroup 12. This safety manager may have permissions to receive answers to queries based on driving data associated with that Subgroup 12, but not other subgroups or groups (Subgroup 11, Group2, or Leaf3).

[0161] To handle safety manager queries regarding driving data and alert statistics data, the data processing system may be granted access to information about how such data is organized. Such data may be organized into a table schema.

[0162] A table schema may include a definition for a “tenantmaster” table, which may specify data fields corresponding to high-level details about each tenant, such as a unique tenant id, a unique name. The “tenantmaster” schema may further include a field for a “parent tenant id” which may facilitate a parent / child / child configuration, as described above.

[0163] A table schema may further include a definition for a “user” table, which may specify data fields corresponding to each user of the safety system. The “user” schema may specify a tenant id to which the user is associated, and may further specify a driver id, which would be set to a non-null value if the user is a driver about whom driving safety data may be collected. This table may be supported by a “usertenant” table which may maintain associations between each user and the tenant account to which the user is assigned.

[0164] A table schema may further include a definition for a “hierarchylevel” table, which may contain constraints to facilitate a multi-level implementation model. For example, the hierarchy level table may specify a constraint, so that each tenant id, hierarchy level pair is associated with a unique identifier. This unique identifier may then facilitate access to data associated with the correct tenant id and hierarchy level.

[0165] Similarly, a table schema may further include a definition of a “hierarchytenant” table, which may manage organization of a parent / child / child implementation setup. This table schema may work in conjunction with a “hierarchytenantmapper” table.

[0166] In addition to representing proper scoping for data fields, the database may include schemas for alert data, other driving data, and statistics. Alert data may be organized by an “alert” schema. The alert schema may include an alert id and a driver id, where the driver id field may be filled with a non-null value if the alert data has been associated with a driver. The alert schema may also specify a vehicle id, which may refer to the vehicle where the safety event was observed.

[0167] Various tables may further represent alert statistics. For example, a “greenezonescore” table may contain entries for a summary safety score (a Greenzone® score), where each entry contains a driver id, time interval information, and other relevant statistics, such as the number of driving minutes that were analyzed for the driver over the time interval.

[0168] To handle safety manager queries regarding driving data and alert statistics data, the data processing system may generate SQL queries in relation to table schemas described above.

[0169] In certain embodiments, the data processing system may have access to alert data for which a “confirmation status” indicates that the alert has been verified by one or more processing steps. For example, a confirmation status of 1 may indicate that a detected alert was not only detected by on-device processing, but that it was further processed by a cloud computing process that applies additional checks and tests.

[0170] In certain embodiments, a summary safety score table may be accessed with interval type specifications. In these cases, safety score entries having a suitable from time stamp and to time stamp may be accessed.

[0171] According to certain aspects of the present disclosure, the data processing system may formulate SQL queries in response to given natural language prompts from a safety manager user. In the following examples, there is one fleet, named “ACME Corp.” The fleet has a parent_tenant_id=1001. The fleet is organized into two groups, India (hierarchy_tenant_id=101) and US (hierarchy_tenant_id=102). Each of these two groups is further organized into sub groups. The India group is organized into Cloud ind (subtenant_id=5001), Device ind (subtenant_id=5002), and Analytics ind(subtenant_id=5002). The entire fleet / group architecture may be visualized as depicted in FIG.4 as hierarchical data structure 400.

[0172] In the following examples, the data processing system is responding to a safety manager who is asking about drivers in the Cloud-ind (sub_tenant_id=5001) and Device-ind (sub_tenant_id=5002) groups come under India (hierarchy_tenant_Id=101) group.

[0173] In certain embodiments, sub-queries may be formulated by the data processing system to fetch a list of sub tenant ids (groups) and driver ids based on a selected hierarchy. The list of driver id that are returned may then be used is subsequent q ueries. These subqueries may simplify later SQL queries if the driver id list may be supplied in place of hierarchy and group identifiers.

[0174] A first sub-query may be formulated to identify all sub tenant id entries that are associated with a known hierarchy tenant id. The following SQL query:SELECT sub tenant id FROM hierarchytenantmapper WHERE hierarchy tenant id = 101; is expected to return a list of sub tenant id values that are associated with the India group in the current example. Based on the above architecture, the above query is expected to return a list with three values: 5001, 5002, and 5003. A sub-query may further take into account a user accessibility level (not shown).

[0175] A second sub-query may be formulated to identify all driver id entries that are associated with the sub tenant id values from the first query. This second sub-query may further refine the query to exclude any data that may be associated with an unassigned driver.SELECT driver id FROM user WHERE driver id != 0 AND driver id IS NOT NULL AND sub_tenant_id in (5001, 5002, 5003).The above query may return a list of driver_id values, such as (467137, 90509, 330, 182, 174, 264, 2370, 39585, 1465241, 284, 1706346, 14512, 747798, 39587, 307, 1136950, 11698, 297). This list of driver id values may then be used in subsequent SQL formulations by the data processing system.

[0176] A safety manager may probe the data processing system with the following query: “Give me the list of drivers with speed violation current week but not last week.” In response to the query, using the list of driver id entries from the appropriate group, as describedin the previous section, the data processing system may formulate the following complex SQL query as part of responding to the natural language query. As can be seen in the example below, some of the query fields may be redundant with others, which may be expected when utilizing a generative pre-trained large language model. For example, even though a full driver id list is specified and all of the driver id values that are listed are non-null and are positive, the data processing system may still add conditions including “AND driver_id is not null” and “AND driver_id > 0.”SELECTDi stinct(driver_i d)FROM alertsWHERE alert type = < speeding >AND time_stamp>='2023-06-l 1 07:00:00'AND time_stamp <='2023-06-18 07:00:00'AND driver_id is not nullAND driver_id in(467137,90509,330,182,174,264,2370,39585,1465241,284,170 6346,14512,747798,39587,307,1136950,11698,297)AND driver_id > 0AND tenant_id=1001EXCEPTSELECTDi stinct(driver_i d)FROM alertsWHERE alert type = < speeding >AND time_stamp>='2023 -06-04 07:00:00'AND time_stamp <='2023-06-11 07:00:00'AND driver_id is not nullAND driver_id > 0AND driver_id in(467137,90509,330,182,174,264,2370,39585,1465241,2 84,1706346,14512,747798,39587,307,1136950,11698,2 97)AND tenant_id=1001

[0177] As can be seen in the examples provided herein, the data processing system may determine the appropriate timestamp for the query, as well as other features for the query.

[0178] A safety manager may probe the data processing system with the following query: “Who are the drivers with the top 5 Greenzone score last week.” The data processing system may then formulate the following SQL query:SELECT * from (SELECTDISTINCT ON (driver id) driver id, greenzone scoreFROM greenzonescoreWHERE driver id in (11,467137,1738921,16345,179,29,51,541308,90509,12 77734,202,275,1465241,16194,1706346,330,191,10532 95,2370, 11698,39585,545668,695094, 126805,445274, 1 3,300,14512,118708,143,12045)AND interval_type = 'WEEKLY'AND from_time_stamp = '2023-06-05 07:00:00.000') as greenzone scoreORDER BY greenzone score DESCLIMIT 5;

[0179] A safety manager may probe the data processing system with the following query: “Can you break down my alerts last week by alert type.” The data processing system may formulate different SQL queries depending on whether the context of the conversation with the safety manager would suggest that the query is intended at the driver level or the fleet level. Alternatively, or in addition, the data processing system may reply in the form of a clarification to the safety manager, so that the data processing system could generate the appropriate query.At driver level :SELECT alert type, count(*)FROM alertsWHERE time_stamp>='2023 -06-04' and time_stamp<='2023-06-10' and driver_id=281GROUP BY alert_type;- At tenant level:SELECT alert type, count(*)FROM ndalertsWHERE time_stamp>='2023 -06-04' and time_stamp<='2023 -06-09'AND driver_id in(11,467137,1738921,16345,179,29,51,541308,90509,1277734,202,275,1465241,16194,1706346,330,191,1053295,2370,11698,39585,545668,695094,126805,445274,13,300,14512,118708,143,12045)GROUP BY alert_type;

[0180] A safety manager may probe the data processing system with the following query: “Who are the top 5 drivers by most monitored minutes last month?”

[0181] The data processing system may then formulate the following SQL query:SELECT * from (SELECTDISTINCT ON (driver id) driver id, monitored minutes as monitored minutesFROM greenzonescoreWHEREAND interval type = 'MONTHLY'AND from_time_stamp = '2023-05-01 07:00:00.000'AND to_time_stamp > '2023-05-01 07:00:00.000'AND to_time_stamp <= '2023-06-01 07:00:00.000'and driver id > 0AND driver_id in (11,467137,1738921,16345,179,29,51,541308,90509,12 77734,202,275,1465241,16194,1706346,330,191,10532 95,2370, 11698,39585,545668,695094, 126805,445274, 1 3,300,14512,118708,143,12045)) as monitored minutesORDER BY monitored minutes DESCLIMIT 5;

[0182] A safety manager may want to gain a better understanding of the drivers that the safety manager manages, so that the safety manager can be better prepared to coach the drivers. To do so, the safety manager may probe the data processing system with the following query: “I am going to meet with Bob (a driver) this afternoon, can you summarize his performance over the last 7 days and recommend some coaching action to improve his GreenZone score?” or “Could you tell me who are my top 5 drivers most likely to get into a collision, please tell me why and could you provide a recommendation for each driver on how to reduce their risk?”. These types of complex queries may require complex natural language responses. Where the safety system already provides functionality, the data processing system may point the safety manager to this functionality. For example, the safety system may periodically analyze alert statistics for each driver to identify which types of driving scenarios (e.g., stop sign violations, speeding) are pulling down the driver’s greenzone score the most. The answer to the first query may then make use of this type of computation. In some embodiments, the data processing system might explain how a recommended coaching engine works, then identify that for driver “Bob” the recommended coaching engine recommends that Bob address certain driving scenarios and behaviors.

[0183] FIG. 5 illustrates a flowchart of a method for implementing a machine learning architecture for navigating a web platform, according to an embodiment. The data processing system can be or include a remote computing system (e.g., the analytics server 108) configured to store driver data of different vehicles and / or provide a platform through which users can view such driver data. The method 500 is shown to include steps 502-510. However, other embodiments may include additional or alternative steps, or may omit one or more stepsaltogether. Different steps can be performed by different computing systems (e.g., computing systems of vehicles can perform one or more of the steps 502-510 and / or the analytics server can perform one or more of the steps 502-510) and / or the different computing systems and can operate together to perform individual steps of the steps 502-510.

[0184] In step 502, the data processing system can map a plurality of types of driver data to a plurality of pages of a platform (e.g., an online platform). For example, the data processing system can host a platform. The platform can include multiple pages that can be navigated to view details regarding driver data of drivers for which the platform manages data. A client device can communicate with the data processing system to access the platform, such as by establishing a connection with the data processing system over a network. The user can provide inputs into the client device to view different pages and / or types of data that are hosted by the platform. The data processing system can receive the inputs and retrieve pages requested by the inputs and present the different pages at the client device. In some cases, the pages can correspond to different types of data. For instance, the pages can correspond to identifiers or pointers (e.g., addresses) to locations in a database stored and / or hosted by the data processing system. The identifiers or pointers can identify the locations in which specific types of data are stored in the database. The identifiers or pointers can be located in the URLs of the pages and / or stored in memory of the data processing system with mappings to page identifiers (e.g., page numbers) of the pages. When presenting such a page at the client device, the data processing system can identify such an address. Based on the identification, the data processing system can automatically retrieve the data from the database at the address.

[0185] For example, responsive to receiving a request to access a page that includes data regarding alerts that are generated for different drivers when the drivers are driving, the data processing system can retrieve the page according to the request. The page can correspond to an identifier indicating a location in the database of the data processing system that includes such alert data. The data processing system can use the identifier to access the location in the database and retrieve the alert data of the page from the location in the database. The data processing system can retrieve the data and populate the page with the retrieved data in locations of the page that are configured to present the retrieved data. The data processing system can similarly perform this process for each page that the data processing system presents to users of the platform that corresponds to identifiers identifying locations of the database from which to retrieve data for presentation.

[0186] The data processing system can store a mapping of the types of driver data that the different pages are configured to present (e.g., the types of driver data for which the pages include identifiers that the data processing system can use to retrieve driver data upon presentation of the pages). The mapping can be a relational table or any other type of data structure that indicates which types of data correspond to which page. For instance, a mapping between a type of driver data and a page can indicate a type of driver data (e.g., a string identifying the type of driver data) and a page identifier (e.g., a uniform resource locator, a page number, or any other page identifier) of the page. In one example, the type of driver data and the page identifier can be included in the same row of a table. The types of driver data and pages can be mapped in any manner. The data processing system can map the types of driver data and pages by generating and / or storing such a table.

[0187] At step 504, the data processing system can receive a natural language query. The data processing system can receive the natural language query from a chat interface presented on the client device accessing the platform provided by the data processing system. The natural language query can be or include a string of text. The natural language query can include or be a request for driver data regarding a set of drivers (e.g., a plurality of drivers). For example, the natural language can be a string of text requesting driver safety data for drivers in a particular geographic language (e.g., Europe, the United States, and / or India). The data processing system can receive the natural language query from the client device accessing the platform in a message. The data processing system can receive the natural language query while the client device is displaying a first page of the platform on a user interface.

[0188] For example, referring now to FIG. 6, a user interface 600 depicting a first page 602 of a platform overlayed by a chat interface 604. The user interface 600 can be displayed at a client device a user can use to access the platform. The first page 602 can be a home page of the platform or can be a page the user navigated to (e.g., manually or automatically) while accessing the platform with the client device. The user can input a natural language query into the chat interface 604. For instance, the user can request to navigate to an alerts page that is configured to show alert data regarding different drivers. In some cases, the request can include a type of data to display when presenting the alert data. The user can select a submit button on the user interface or another input / output device to transmit the natural language query to the data processing system.

[0189] Referring again to FIG. 5, in step 506, the data processing system can determine a type of driver data that is requested in the natural language query. The data processing system can determine the type of driver data using a large language model. For example, the data processing system can input the natural language query into the large language model. The data processing system can execute the large language model based on the natural language query to generate an output identifying the type of driver data that is being requested. Examples of types of driver data that may be requested can be driver alert data, driver statistic data, driver traffic violation data, driver safety data (e.g., a driver safety score), etc. Each type of driver data can correspond to a different page of the platform. Based on the natural language query, the large language model can generate an output identifying a type of driver data or otherwise determining the type of driver data.

[0190] The data processing system can determine an action type based on the natural language query. The data processing system can do so using a large language model (e.g., the same or a different large language model that the data processing system used to determine the type of driver data of the natural language query). For example, the data processing system can input the natural language query into the large language model. The data processing system can execute the large language model based on the input to cause the large language model to generate an action type for the natural language query. The action type can be, for example, changing pages, filtering or analyzing data presented on pages, reporting subscriptions, raising requests, sending emails, changing configurations, etc.

[0191] The large language model can determine the action type to be to change pages based on the natural language query indicating to view driver on a different page of the platform different from the page that the client device is currently presenting. For example, the large language model can identify the type of driver data of the page that the client device is currently viewing based on the mapping of types of driver data to pages that the data processing system stores in memory. The large language model can do so by querying the mapping using the page identifier of the page as a key and identifying the type of driver data that corresponds to the page in the mapping. In another example, the large language model can process the contents of the page and determine the type of data based on the processing. The large language model can compare the type of driver data of the current page with the type of driver data of the natural language query. Responsive to determining the two types of driver data (e.g., the two identifications of the types of driver data) do not match, the large language model can determine an action type to change pages of the platform.

[0192] In step 508, the data processing system can retrieve a second page of the platform. The data processing system can retrieve the second page of the platform based on the type of driver data that was determined for the natural language query. The data processing system can retrieve the second page of the platform responsive to the determination of the action type to change web pages from the first page currently being presented at the client device to a second page corresponding to the type of driver data of the natural language query.

[0193] The data processing system can identify the second page of the platform using the mapping of types of driver data to pages of the platform. For example, the data processing system can compare the type of driver data determined for the natural language query to the mapping. Based on the comparison, the data processing system can identify the page of the platform that corresponds to the type of driver data. The data processing system can retrieve the second page of the platform from memory responsive to identifying the second page based on the mapping.

[0194] In step 510, the data processing system can present a visual representation of the type of driver data of the second page at the client device. The data processing system can present the visual representation on the second page while presenting the second page at the client device. For example, responsive to retrieving the second page from memory, the data processing system can process the code of the second page. During the processing, the data processing system can identify an identifier of a location in the database stored by the data processing system that contains the type of driver data requested in the natural language query. The data processing system can use the identifier to retrieve the requested type of driver data from the database. The data processing system can present the retrieved driver data on the page in the manner or format in which the page is configured to present the data (e.g., as a graph, a table, a set of videos, a set of images, etc.). The data processing system can similarly facilitate the navigation of the platform for the user as the user requests to view or access different types of driver data. In this way, the data processing system can use the large language model to facilitate making navigation of the platform easier than scrolling through the platform and / or selecting options from different drop-down menus.

[0195] After navigating to the second page, the data processing system can filter the driver data that is shown on the second page. For example, while viewing the second page, the user at the client device can input a new or second natural language query. The second natural language query can be a request to filter the data shown on the second page, such as byrequesting to highlight specific types of the data or to reduce the number of drivers for which the second page is showing driver data. In one example, the second page can correspond to driver safety and include safety scores for different drivers of the system. The user can input a second natural language query requesting to view the drivers that are associated with the top five highest driver safety scores. The user can include any driver characteristic and / or criteria for such driver characteristics in the natural language query.

[0196] The data processing system can receive the second natural language query and filter the data shown on the second page according to the second natural language query. For example, the data processing system can input the second natural language query into a large language model (e.g., a second large language model). The large language model can be the same large language model or a different large language model from the model the data processing system used to navigate to the second page. Based on the input, the large language model can determine an intent or action type to filter the data displayed on the second page, such as by determining the data type of the requested data corresponds to the same data type of the second page or based on the context or a message history of the user’s input natural language queries. The large language model can generate a page query based on the natural language query. In doing so, the large language model can configure the page query with attributes or parameters for retrieving data that is relevant to generating a response to the natural language query. The page query can be in a schema that conforms to the structure of the second page. For instance, if the second page displays data in a table, the large language model can generate the page query in a SQL format identifying specific rows and / or attributes or terms to query to retrieve data to generate a response.

[0197] The large language model can query the second page with the page query to retrieve data that is relevant to generating a response to the second natural language query. The large language model can generate a data structure (e.g., a table, matrix, vector, etc.) including the retrieved data. The large language model can then process the data of the data structure according to the second natural language query to generate a visual representation of the response. For instance, if the second natural language query is a request for a list of a defined number of drivers with the highest scores, the large language model can process the data of the data structure to compare the scores of the drivers between each other and then generate a list according to the request based on the comparison (e.g., a list of drivers ordered based on the driver scores of the drivers). The data processing system can present the generated visual representation of the data structure on the second page (e.g., a filtered version of the data or setof data that was originally included on the second page), in some cases replacing the previous contents of the second page. In this way, the data processing system can use a page as a temporary database to generate responses to natural language queries to avoid multiple queries to the database storing all driver data of the platform, reducing the processing resources that may be required to respond to natural language queries regarding similar data, reducing hallucinations by reducing the amount of less data the large language model processes, and / or reducing the likelihood of injecting data or otherwise changing the data that is in the database.

[0198] In some cases, the data processing system can maintain a shadow data structure of the data of the second page. The shadow data structure can be a data structure that includes the data of a page in the format of the page, but is not deleted when filtering the data shown on the page. The data processing system can use the shadow data structure to generate multiple responses to natural language queries that were performed to identify specific types of data on the page. For example, when filtering the second page as described above, the data processing system can generate a copy of the second page or a data structure that replicates the data of the second page in the format of the second page. The data processing system can remove the shadow data structure from memory responsive to changing to a different page (e.g., responsive to detecting a change to a different page or detecting an action type to change to a different page). However, while displaying the second page, the large language model may query the shadow data structure when generating responses to natural language queries by the user. The large language model can query the second page to generate the response to the first natural language query and then query the shadow data structure for each query thereafter, or query the shadow data structure for each query. By generating and storing the shadow data structure in this way, the data processing system can enable the user to perform multiple queries of data on the same page without being restricted to the data that is currently visible on the web page after an initial filtering of the data. Thus, the user can avoid leaving the page and requesting the page again just to view all of the details on the page.

[0199] For example, referring now to FIG. 7, a user interface 700 depicting a second page 702 of a platform overlayed by the chat interface 604 is shown. The second page 702 may be displayed after navigating from the first page 602 shown in FIG. 6. The second page 702 can depict the type of data that the user requested with the input into the chat interface 604 from the first page 602. Additionally, the second page 702 can include data that has been filtered based on a second natural language query into the chat interface 604. For instance, the second page 702 can be configured to show driver alert data. While the data processing systempresents the second page 702, the user viewing the second page 702 can input a natural language query indicating the types of alerts to display on the second page. The types of alerts can be, for example, hard turn alerts, distracted driving alerts, and speeding alerts. The natural language query can further indicate to only include such data for the past month. Based on the natural language query, the data processing system can use a large language model to generate a query with parameters to retrieve the requested data from the second page 702. The data processing system can execute the query to retrieve the requested data and replace the previously presented set of data on the second page 702 with the data retrieved from the second page 702. The data processing system can present indications 704 of active filters on a side bar of the second page 702. Thus, the data processing system can filter the presented data on the second page 702 through the chat interface 604.

[0200] Referring again to FIG. 5, the data processing system can filter pages based on any types of parameters identified or determined from natural language queries. For instance, the data processing system can use the large language model to generate the query to filter the second page based on a defined time period, time range, or date range that is either identified in a natural language query of determined from a natural language query. If the large language model is not able to identify or determine a time period for the query from the natural language query, the large language model can use a default time period.

[0201] In one example, the data processing system can use the large language model to generate recommendations based on driver data on a page of the platform and / or otherwise in response to queries for data through the chat interface. For example, the data processing system may navigate to a page of the platform that includes driver safety data. The data processing system can use the large language model to query the driver safety data of different drivers, either automatically or in response to a natural language query requesting remedial actions and / or recommended coaching actions. Based on the query, the data processing system can generate a summary of driver safety data for each individual driver. In some cases, the data processing system can generate a driver score (e.g., a driver safety score) by aggregating or otherwise performing a function on the retrieved driver safety data of each driver. The data processing system can present the driver scores and / or summaries on a user interface.

[0202] In some cases, the data processing system can use the large language model on the summarized driver safety information for each driver to identify driver behavior that may be improved. In some cases, the data processing system may only do so for a driver responsiveto determining a driver safety score for the driver (determined as described above) is below a threshold. For example, the large language model may determine that a driver typically drives too high (e.g., above a threshold) above a speed limit. Responsive to the determination, the large language model can identify a driver behavior of speeding or driving too fast. The large language model can identify any such improper driver behavior in this manner. Responsive to the identification, the large language model can generate a recommendation (e.g., a text recommendation) for a coaching action to perform to remediate the improper driving behavior. Examples of coaching actions can be showing videos of the driver performing the improper driver behavior, writing a note to the driver informing the driver of the improper driver behavior, establishing meetings with the driver to discuss the improper driver behavior, etc. The data processing system can present the recommendations on a user interface and / or, in some cases, automatically perform the recommended coaching actions, such as by storing videos of the improper driving behavior in the profile of the driver that performed the improper driver behavior for subsequent viewing. The data processing system can similarly generate such recommendations using a temporary database as described with respect to FIG. 2.

[0203] In some cases, the data processing system can prefilter the data of a new page before presenting the new page to a client device accessing the platform. For example, a user can request a subset of a type of driver data from the platform with a natural language query input into a chat interface of the platform. The data processing system can identify the new page that contains the type of data of the subset based on the natural language query using a large language model. After retrieving the new page, the data processing system can retrieve driver data that the new page is configured to present. The data processing system can filter the retrieved driver data to identify the subset of the type of driver data. The data processing system can do so, for example, by generating a temporary data structure from the retrieved driver data and generating a visual representation of the requested subset of driver data in the manner described with reference to FIG. 2. The data processing system can present the new page with the filtered driver data to the user. Thus, the data processing system can both dynamically navigate to the new page and filter data configured to be presented on the new page using the chat interface of the platform and natural language processing techniques.Non-limiting Examples

[0204] In a non-limiting example, a data processing system (e.g., the analytics server 108 or a safety manager assistant (SMA)) implementing the systems and methods describedherein may include an “LLM tool” for table selector functionality. For instance, a user query (e.g., natural language query) may be routed to a data domain path if the natural language query is a data query related to driver and / or fleet related data (e.g., individual and / or aggregated driver safety data). A “GetTime LLM” tool may determine the required time frame by processing the user query. As an example, the GetTime LLM tool may determine whether a time frame mentioned in a user’s natural language query is valid for coaching pages, which may be relevant if coaching pages are subject to a data retention setting. A “table selector LLM” tool may determine the tables and / or pages of a dashboard that may be relevant to the user query. The dashboard may be a web portal, where individual pages in the web portal provide interactive views into driver safety data. Based on the user query, the table selection LLM tool may identify an individual page in the web portal, such as a coaching, alert, or driver stats page, that contains information that is responsive to the user query. Upon determining or identifying a relevant page, respective APIs (such as APIs that are used to populate the interactive views in the identified page) may be called to retrieve relevant data from the driver safety data database (e.g., using the identifiers of the locations of the relevant data in the relevant page itself). Optionally, the retrieved data from the database may be used to create a temporary database (e.g., SQLite database) by a table creation module. A prompt selection module may select relevant prompts for the “generate SQL LLM” tool based on the retrieved data / tables. The generate SQL LLM tool can create an SQL query based on the received prompt from the prompt selection module. The “summarize answer LLM” tool may create a final answer to the user query based on the data stored in the temporary database.

[0001] In a non-limiting example, the table creation module of the data processing system may create a temporary database based on data retrieved by relevant APIs in response to a user query. The API calls may be made by the table creation module. The table creation module may determine relevant APIs based on the input from upstream LLM modules (e.g., Get Time and Table selector).

[0205] In a non-limiting example in some cases, the data processing system may include one or more additional modules. For instance, the data processing system can include a “Router LLM” module to process user queries and determine the relevant domains for the user queries based on the processing, such as a data path domain. A “Configurations LLM” may refer to a database that stores static data related to the configurations of the LLM (e.g., LLM configurations). This example architecture can additionally include an LLM tool for Greenzone (GZ) statistics widget selection. The “GZLLM” tool may be used to select module(s) under a GZ statistics page along with parameters. That is, the GZ statistics page may be one of a plurality of pages in a web portal, the web portal providing various viewing into driving safety information, including a GZ statistics page. The GZ statistics page may include one or more interactive data visualizations that can be configured by parameters. The table selector LLM tool may provide inputs to (and thereby invoke) the GZ LLM tool, if, for example, the table selector LLM tool determines that the user query is related to GZ statistics. To that effect, the user’s query may be processed by the Table Selector LLM, a module that identifies the relevant tables (in the database or locally stored) or pages (in the web portal) needed to address the user’s natural language query. The identified tables and / or pages may then be used to specify calls to various APIs, such as driver stats APIs and any other relevant APIs. These API calls may fetch the necessary data based on the chain of LLMs that lead to the APIs being called.

[0206] The table creation module may collate the fetched data and use the collated fetched data to establish a temporary, user-specific database. The creation of a user-specific database may ensure that responses to queries are restricted to the individual user’s context, thus enhancing the accuracy and relevance of the information provided by the safety manager assistant.

[0207] To retrieve the data from the temporary database that will provide a basis for a responsive answer to the user’s natural language query, a Gen SQL LLM tool may be used to generate an SQL query. The gen SQL LLM tool may receive an input prompt from an example selector module, which utilizes techniques such as Maximal Marginal Relevance (MMR) and Similarity Search. The example selector module may select an example prompt that is relevant to the user's query and the specific pages or tables identified by the table selector LLM tool, and this example prompt may be used by the Gen SQL LLM tool to create an SQL query.

[0208] The SQL query may be executed on the temporary database to retrieve the data that will be used to answer the user query. Two LLM tools may be used to generate the final response. A “sum ans LLM” tool may create the final answer (and may alternatively or optionally include disclaimers) based on the user's query and the final data obtained from the SQL query. Optionally, and simultaneously or consequently, a “plot LLM” tool can take the table output from the SQL query and identify the plot type, x-axis, and y-axis variables, and then generate a chart to visually represent the data.

[0209] When both tools are invoked in response to a user’s query, the data processing system may combine the responses generated by the “sum ans LLM tool” and the “plot LLM tool” to create a single, multi-modal answer. The combined answer may provide a more complete response to the user's query, offering both textual and visual data.

[0210] In some cases, the data processing system can implement an example selection process using an example selector module. The example selector module may include a similarity search algorithm to select an example for prompt creation from one or more examples based on the user query. The similarity search algorithm may provide an example prompt to the gen SQL LLM tool to create an SQL query for the temporary database.

[0211] In a non-limiting example, the data processing system can support additional domains that enhance answering capabilities of the data processing system. In this example, a new domain called ‘take action’ is added as a possible option or determination for the router LLM. The take action domain may facilitate a variety of specific actions based on a user’s query.

[0212] For example, the “router LLM” can be configured to recognize and select the ‘take action’ domain if a user query pertains to one or more of the actions within the ‘take action’ domain. Upon the selection of this domain, a device action type LLM may identify the specific type of action required, which can range from report subscriptions, raising requests, sending emails, or changing configurations. In some embodiments, the ‘take action’ domain may be invoked when a user query requests that an action be performed, but the query relates to actions that are not possible within the context of the user’s web portal environment; in such cases, the ‘take action’ domain processing may result in a user reply indicating that the requested action is not available.

[0213] Each identified action type in the ‘take action’ domain may have one or more corresponding sub-actions. In a driving safety web portal environment, these sub-actions may include creating report subscription structure, creating request structure, creating email, identifying configuration key, and providing disclaimers. These sub-actions may facilitate the specific needs of the main action type, facilitating a comprehensive response to the user query.

[0214] To execute these sub-actions, the safety manager assistant may perform a series of specific tasks. These tasks may include triggering report subscription action, triggeringrequest action, triggering email action, and triggering configuration change. Each task corresponds to a specific sub-action, ensuring that every aspect of the user’s query is addressed.

[0215] Further enhancing the capabilities of the data processing system, a User Interface (UI) action functionality may be added to the ‘data’ domain. This feature may allow the data processing system to interact directly with the user interface based on the requirements of the user's query.

[0216] If a user query pertains to the ‘data’ domain, the data processing system may determine whether a UI action is possible based on the user query. If a UI action is possible, the data processing system may generate UI parameters based on the user’s query. These parameters may be sent to the front end of the UI.

[0217] In a non-limiting example, UI parameters sent to a front end system may include information that guides how the user interface should respond or adapt to the user's query or action. These parameters may vary depending on the specific requirements of the data processing system and the user's query. Some examples of possible UI parameters may include: a. Page ID / URL: This parameter directs the system to the specific page where the UI action needs to be performed. b. Element ID: This specifies the particular UI element (such as a button, text box, dropdown menu, etc.) that needs to be interacted with or manipulated. c. Action Type: This indicates the type of action that needs to be performed on the specified UI element (for example, click, enter text, select, etc.). d. Data / Input Value: If the action involves inputting data or making a selection, this parameter provides the specific data or value to be inputted or selected. e. Filter Parameters: These are used to apply specific filters or sorting methods to the data being displayed on the page. f. Navigation Instructions: These parameters provide instructions for navigating the UI, such as switching between tabs, scrolling, or expanding / collapsing sections. g. Refresh Rate: This parameter determines how frequently the UI should refresh or update the displayed data.

[0218] Upon receiving the UI parameters, the front-end system can check whether the page currently displayed to the user is the desired page for executing the UI action. If thecurrent page is not the desired one, the module may redirect to the appropriate page, open a new tab, switch to an existing tab, and the like.

[0219] Simultaneously, or just before or after, the system may set filter parameters and actions to be applied on the desired page. These parameters may be based on the specifics of the user query and are configured to ensure that the UI action is executed accurately and effectively.

[0220] Once the redirection and filter settings are complete, the UI may undergo a refresh. This refresh may involve fetching new data based on the updated parameters and subsequently updating the UI displayed to the user. This ensures that the user interface may reflect the most relevant and accurate information in response to the user query.

[0221] The data processing system may have the capability to perform certain tasks on an online portal associated with driver information. The data processing system may be configured to: a) Control user interface (UI): The data processing system may be configured to navigate pages, select filters, and input dates. This may be done by preparing a JSON structure that is converted into UI changes that need to be executed to perform a specific task requested by the safety manager in the form of a query. b) Raise Support Email: The data processing system may be configured to raise a support email on the safety manager’s behalf. The data processing system may select an email template and the body of text to be sent from a database that stores email templates. The selection of the email template may be based on the query made to the data processing system by the safety manager. c) Subscribe to Reports: The data processing system may be configured to subscribe users to available reports. It will provide inputs like interval frequency and any additional filters specific to the reports.The data processing system can generate JSON files containing such information. The JSON files can each include a header marking the type of the payload (e.g., email, report, navigation, etc.) of the information. The data processing system can use the headers of the JSON files to determine how to process the information in the payloads of the JSON files.

[0222] In a non-limiting example, the systems and methods described herein can be used to generate electronic messages (e.g., emails) and / or reports. For example, an actionrouter (e.g., an LLM configured to determine an action type for a natural language query) can determine a type of action for a natural language query received from a client device to be to generate an email containing different types of driver data from within a defined time period. The data processing system can use the systems and methods described herein to generate a first query to request data within the defined time period from tables in the database of the data processing system containing the different types of data. The data processing system can generate a temporary database with the retrieved data. The data processing system can generate a second query to retrieve data that is relevant to generating the email with the different types of data and / or perform any analysis on the data to generate the requested data types. The data processing system can include the requested data in an email formatted to either a person or entity indicated in the natural language query or to a placeholder name. The data processing system can include the requested data in the email according to a format indicated in the natural language query or according to a default format (e.g., in a table). The data processing system can transmit or send the email to the client device that transmitted the request.

[0223] In another non-limiting example, the data processing system can generate reports. For example, the action router can determine a type of action for a natural language query received from a client device to be to generate or subscribe to a report containing different types of driver data from within a defined time period. A subscription can be a subscription to receive specific types of driver data at set intervals. For example, a natural language query may be a request to receive driver scores of drivers over the last week at weekly intervals. Responsive to the request, the data processing system can generate a record in memory with parameters for the subscription, such as the data types of the subscription, time intervals of the subscription, the user or account associated with the subscription, etc., thus generating the subscription for the user that requested the subscription. Subsequently, the data processing system can monitor an internal clock and use the systems and methods described herein to automatically query the database of the data processing system for data that is relevant for generating a report (e.g., the types of data over the previous time interval) and generate the report according to the subscription request. The data processing system may transmit such reports to the user (e.g., an electronic address of the user, such as an email address) or store the reports in the user’ s account with the platform. The data processing system can generate and / or transmit such reports to the user until receiving a natural language query requesting to unsubscribe to the report or otherwise receiving a subscription request.

[0224] A computer program product may include a non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by one or more processors configured to carry out any of the systems and / or methods described herein.

[0225] The systems and methods are described herein using one or more large language models. The process can be performed using a single large language model for each step, different large language models for each step, or with large language models that are configured to perform different combinations of different steps. In some cases, the analytics server 108 or the data processing system performs parts of the processes without using a large language model.

[0226] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure or the claims.

[0227] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0228] The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the claimed features or this disclosure. Thus, the operation and behavior of the systems and methods were described without reference to thespecific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0229] When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processorexecutable software module, which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0230] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0231] The terms “data processing apparatus”, “data processing system”, “client device”, “client computing device”, “computing platform”, “computing device”, “computing system”, “user device”, or “device” can encompass all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA or an ASIC. The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack,a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.

[0232] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0233] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer include a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS receiver, a digital camera device, a video camera device, or a portable storage device (e.g., a universal serial bus (USB) flash drive), for example. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0234] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), plasma, or LCD monitor, for displaying information to the user; a keyboard; and a pointing device, e.g., a mouse, a trackball, or a touchscreen, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can include any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user.

[0235] In certain circumstances, multitasking and parallel processing system may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. For example, the computing devices described herein can each be a single module, a logic device having one or more processing modules, one or more servers, or an embedded computing device.

[0236] Having now described some illustrative implementations and implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed only in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

[0237] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” “characterized by,” “characterized in that,” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0238] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods,their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.

[0239] Any implementation disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternate implementation,” “various implementation,” “one implementation,” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0240] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms.

[0241] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included for the sole purpose of increasing the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.

[0242] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0243] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

CLAIMS1. A method of querying a database, comprising: receiving, by one or more processors, an access credential of a user from a client device; receiving, by the one or more processors from the client device via a chat interface, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers; responsive to the access credential corresponding to the plurality of drivers, generating, by the one or more processors using a first generative language model, a first query to a database based on the natural language query, the first query comprising first parameters for retrieving data from the database; generating, by the one or more processors, a temporary database comprising driver data regarding at least one of the plurality of drivers retrieved from the database using the first query; generating, by the one or more processors using a second generative language model, a second query based on the natural language query, the second query comprising second parameters for retrieving data from the temporary database; generating, by the one or more processors, a data structure containing the data retrieved from the temporary database based on an execution of the second query in the temporary database; and presenting, by the one or more processors, a visual representation of the data structure at the client device.

2. The method of claim 1, further comprising: storing, by the one or more processors, a hierarchical data structure comprising nodes representing a parent profile and a plurality of profiles, wherein nodes of the plurality of profiles are linked with a node of the parent profile within the hierarchical data structure, and wherein the natural language query identifies the parent profile; and identifying, by the one or more processors, an identifier of each of at least one of the plurality of profiles from the hierarchical data structure by identifying, by the one or more processors, the node of the parent profile based on the natural language query and then identifying, by the one or more processors, the nodes of the plurality of profiles based on the links between the nodes of the plurality of profiles and the node of the parent profile.

3. The method of claim 1, wherein generating the first query comprises: determining, by the one or more processors using the first generative language model, a date range based on the natural language query, the date range comprising a start date and an end date; and including, by the one or more processors, the date range in the first parameters of the first query.

4. The method of claim 1, further comprising: using, by the one or more processors, an application programming interface (API) to query the database using the first query.

5. The method of claim 1, further comprising: executing, by the one or more processors, the first generative language model to identify one or more tables containing the driver data regarding the plurality of drivers; and determining, by the one or more processors using the first generative language model, that the access credential corresponds to the plurality of drivers responsive to determining the access credential provides access to the identified one or more tables.

6. The method of claim 5, further comprising: determining, by the one or more processors using the first generative language model, a requested driver characteristic based on the natural language query, wherein identifying the one or more tables containing the driver data comprises identifying, by the one or more processors, the one or more tables based on each of the one or more tables corresponding to a driver that corresponds to the requested driver characteristic.

7. The method of claim 6, wherein determining the requested driver characteristic comprises determining, by the one or more processors using the first generative language model, a criterion regarding a driver score; and wherein identifying the one or more tables comprises identifying, by the one or more processors, the one or more tables based on the one or more tables containing driver data regarding driver scores.

8. The method of claim 7, wherein driver scores that satisfy the criterion exceed a threshold.

9. The method of claim 5, further comprising querying, by the one or more processors, the database by: using, by the one or more processors, a plurality of application programming interfaces (APIs) to query each of the one or more tables.

10. The method of claim 1, further comprising: discarding, by the one or more processors, the temporary database responsive to presenting the visual representation of the data structure at the client device.

11. The method of claim 1, wherein generating the second query comprises generating, using the second generative language model, a structured query language (SQL) query, the method further comprising: querying, by the one or more processors, the temporary database using the SQL query.

12. A system for querying a database, comprising: one or more processors configured by computer-readable instructions to: receive an access credential of a user from a client device; receive, from the client device via a chat interface, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers; responsive to the access credential corresponding to the plurality of drivers, generate, using a first generative language model, a first query to a database based on the natural language query, the first query comprising first parameters for retrieving data from the database; generate a temporary database comprising driver data regarding at least one of the plurality of drivers retrieved from the database using the first query; generate, using a second generative language model, a second query based on the natural language query, the second query comprising second parameters for retrieving data from the temporary database;generate a data structure containing the data retrieved from the temporary database based on an execution of the second query in the temporary database; and present a visual representation of the data structure at the client device.

13. The system of claim 12, wherein the one or more processors are further configured to: store a hierarchical data structure comprising nodes representing a parent profile and a plurality of profiles, wherein nodes of the plurality of profiles are linked with a node of the parent profile within the hierarchical data structure, and wherein the natural language query identifies the parent profile; and identify an identifier of each of at least one of the plurality of profiles from the hierarchical data structure by identifying the node of the parent profile based on the natural language query and then identifying the nodes of the plurality of profiles based on the links between the nodes of the plurality of profiles and the node of the parent profile.

14. The system of claim 12, wherein the one or more processors are configured to generate the first query by: determining, using the first generative language model, a date range based on the natural language query, the date range comprising a start date and an end date; and including the date range in the first parameters for retrieving data in the first query.

15. The system of claim 12, wherein the one or more processors are further configured to: use an application programming interface (API) to query the database using the first query.

16. The system of claim 12, wherein the one or more processors are further configured to: execute, the first generative language model to identify one or more tables containing the driver data regarding the plurality of drivers; and determine, using the first generative language model, that the access credential corresponds to the plurality of drivers responsive to determining the access credential provides access to the identified one or more tables.

17. The system of claim 16, wherein the one or more processors are further configured to:determine, using the first generative language model, a requested driver characteristic based on the natural language query, wherein identifying the one or more tables containing the driver data comprises identifying the one or more tables based on each of the one or more tables corresponding to a driver that corresponds to the requested driver characteristic.

18. The system of claim 17, wherein the one or more processors are configured to determine the requested driver characteristic by determining, using the first generative language model, a criterion regarding a driver score; and wherein the one or more processors are configured to identify the one or more tables based on the one or more tables containing driver data regarding drivers that correspond to driver scores that satisfy the criterion.

19. The system of claim 12, wherein the one or more processors are further configured to: discard the temporary database responsive to presenting the visual representation of the data structure at the client device.

20. The system of claim 12, wherein the one or more processors are configured to generate the second query by generating, using the second generative language model, a structured query language (SQL) query, and wherein the one or more processors are further configured to: query the temporary database using the SQL query.

21. A method, comprising: mapping, by one or more processors, a plurality of types of driver data to a plurality of pages of an online platform, each page configured to present a different type of driver data stored in a data structure linked with the page; receiving, by the one or more processors from a chat interface presented on a user interface at a client device, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers, and the user interface displaying a first page of the plurality of pages of the online platform; determining, by the one or more processors using a generative language model, a type of driver data requested in the natural language query;retrieving, by the one or more processors, a second page from the plurality of pages based on the type of driver data corresponding to the second page in the mapping; and presenting, by the one or more processors, a visual representation of the type of data on the second page at the client device.

22. The method of claim 21, further comprising: subsequent to retrieving the second page, receiving, by the one or more processors, a second natural language query; and filtering, by the one or more processors using a second generative language model, a set of driver data depicted on the second page in the visual representation of the type of data based on the second natural language query.

23. The method of claim 22, wherein filtering the set of driver data comprises: generating, by the one or more processors using the second generative language model, a query using the second natural language query, the query comprising parameters for retrieving the requested driver data from the set of driver data of the visual representation of the type of data; and generating, by the one or more processors, a data structure containing the requested driver data retrieved from the set of driver data based on an execution of the query on the visual representation of the type of data.

24. The method of claim 23, further comprising comprises: presenting, by the one or more processors, a visual representation of the data structure containing a filtered version of the set of data on the second page.

25. The method of claim 24, further comprising: determining, by the one or more processors using the second generative language model, a requested driver characteristic based on the second natural language query; and generating, by the one or more processors, the visual representation of the data structure based on the requested driver characteristic.

26. The method of claim 25, wherein determining the requested driver characteristic comprises determining, by the one or more processors using the second generative language model, a criterion regarding a driver score.

27. The method of claim 26, wherein generating the visual representation of the data structure comprises generating, by the one or more processors, a list of a defined number of drivers ordered based on performance score of each of the drivers of the list, the defined number extracted from the second natural language query by the second generative language model.

28. The method of claim 22, further comprising: generating, by the one or more processors, a shadow data structure containing the set of driver data of the second page; subsequent to filtering the set of driver data based on the second natural language query, receiving, by the one or more processors, a third natural language query; and generating, by the one or more processors using the second generating language model, a second visual representation of the set of driver data based on the set of driver data of the second page stored in the shadow data structure.

29. The method of claim 28, further comprising: detecting, by the one or more processors, navigation from the second page to a third page; and discarding, by the one or more processors, the shadow data structure responsive to the detection.

30. The method of claim 21, further comprising: determining, by the one or more processors using the generative language model, an action type to change from the first page to the second page based on the natural language query, wherein retrieving the second page from the plurality of pages comprises retrieving, by the one or more processors, the second page based on the determined action type.

31. A system, comprising:one or more processors configured by machine-readable instructions to: map a plurality of types of driver data to a plurality of pages of an online platform, each page configured to present a different type of driver data stored in a data structure linked with the page; receive, from a chat interface presented on a user interface at a client device, a natural language query, the natural language query comprising a request for driver data regarding a plurality of drivers, and the user interface displaying a first page of the plurality of pages of the online platform; determine, using a generative language model, a type of driver data requested in the natural language query; retrieve a second page from the plurality of pages based on the type of driver data corresponding to the second page in the mapping; and present a visual representation of the type of driver data on the second page at the client device.

32. The system of claim 31, wherein the one or more processors are further configured to: subsequent to retrieving the second page, receive a second natural language query; and filter, using a second generating language model, a set of driver data depicted on the second page in the visual representation of the type of driver data based on the second natural language query.

33. The system of claim 32, wherein the one or more processors are configured to filter the set of driver data by: generating, using the second generative language model, a query using the second natural language query, the query comprising parameters for retrieving the requested driver data from the set of driver data of the visual representation of the type of driver data; and generating a data structure containing the requested driver data retrieved from the set of driver data based on an execution of the query on the visual representation of the type of driver data.

34. The system of claim 33, wherein the one or more processors are further configured to: present a visual representation of the data structure containing a filtered version of the set of data on the second page.

35. The system of claim 34, wherein the one or more processors are further configured to: determine, using the second generative language model, a requested driver characteristic based on the second natural language query; and generate the visual representation of the data structure based on the requested driver characteristic.

36. The system of claim 35, wherein the one or more processors are configured to determine the requested driver characteristic by determining, using the second generative language model, a criterion regarding a driver score.

37. The system of claim 36, wherein the one or more processors are configured to generate the visual representation of the data structure by generating a list of a defined number of drivers ordered based on performance score of each of the drivers of the list, the defined number extracted from the second natural language query by the second generative language model.

38. The system of claim 32, wherein the one or more processors are further configured to: generate a shadow data structure containing the set of driver data of the second page; subsequent to filtering the set of driver data based on the second natural language query, receive a third natural language query; and generate, using the second generating language model, a second visual representation of the set of driver data based on the set of driver data of the second page stored in the shadow data structure.

39. The system of claim 38, wherein the one or more processors are further configured to: detect navigation from the second page to a third page; and discard the shadow data structure responsive to the detection.

40. The system of claim 31, wherein the one or more processors are further configured to: determine, using the generative language model, an action type to change from the first page to the second page based on the natural language query, wherein the one or more processors are configured to retrieve the second page from the plurality of pages by retrieving the second page based on the determined action type.

Images