System for searching surveillance records using natural language queries

Abstract

A method for generating by an application program an LLM query for transmission to an LLM tool to cause the LLM tool to generate a structured data source query for causing a searching of a data source of security records. It includes obtaining an input query, the input query comprising natural language; accessing a computer-readable memory storing therein a system prompt template library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least one but not all of the system prompt template elements in the system prompt template library; processing the subset of system prompt template elements to generate the system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query; generating the LLM query comprising the received input query and the system prompt; and transmitting the generated LLM query to the LLM tool to cause the LLM tool to generate the structured data source query for causing a searching of a data source of security records based on the received LLM query.

Description

SYSTEM FOR SEARCHING SURVEILLANCE RECORDS USING NATURAL LANGUAGE QUERIES

[0001] The present disclosure claims priority from U.S. provisional patent application No. 63 / 670,433 filed on July 12, 2024, incorporated herein by reference. The present disclosure claims priority from U.S. patent application No. 19 / 060,550, filed on February 21, 2025, incorporated herein by reference. The present disclosure claims priority from U.S. provisional patent application No. 63 / 714,295, filed on October 31, 2024, incorporated herein by reference.Technical Field

[0002] The present disclosure relates generally to physical security and surveillance, and more specifically to querying repositories of surveillance data.Background

[0003] A surveillance system is a network of various devices employed to monitor activities and behaviours of persons, vehicles, or the like in a particular area being surveilled. A surveillance system may collect information from a variety of sources, centralize that information, and make the information available to surveillance personnel to aid them in making decisions relating to the safety of persons or other objects within the area being surveilled. A surveillance network may include any suitable number of devices for collecting information, including cameras, microphones, access card readers, and the like, as well as any number of monitors or other interfaces for presenting information to operators of the surveillance system. In the case of a video surveillance system, information (e.g., video streams) acquired by cameras or similar devices may be centralized in a video management system (VMS). Other types of surveillance data, such as access control records, intrusion detection records, event records, records relating to intercom calls or other voice communications, and the like, may also be stored in suitable systems: in some cases, a unified security platform may serve as a VMS and to store other types of surveillance records.

[0004] The overwhelming majority of surveillance records are never accessed after initial recording and / or viewing. This is because most surveillance records contain no evidence of situations requiring investigation, and thus are of limited interest. However, for a variety of reasons, it is very uncommon for operators of the surveillance system to delete surveillance records before a predetermined retention period has elapsed. This means that operators searching the surveillance records for events where persons or vehicles of interest are present must comb through all the records produced by the surveillance system, which is an arduous and time-consuming task.

[0005] Thus, improved approaches for searching through surveillance records aredesirable.

[0006] Moreover, application programs can include an LLM search tool to assist the user in answering certain questions. For instance, a surveillance system application program may include an LLM tool to query data available in the system which may include vehicle information corresponding to a licence plate read (LPR), or to obtain additional information regarding a detection of a specific vehicle or individual. Certain LLMs are included in these application programs as chatbots. However, Al models such as LLMs are often trained in a broad manner which has the effect that their output may be broad, generalized or otherwise not specific enough to the problem the user seeks to solve.

[0007] Therefore, it would be advantageous to arrive at a solution for improving the outputs generated by these LLMs in the context of their operating in connection with an application program.Summary

[0008] The following presents a simplified summary of one or more implementations in accordance with aspects of the present disclosure in order to provide a basic understanding of such implementations, without limiting the embodiments presented within the present disclosure.

[0009] A surveillance repository may contain any suitable number of surveillance records, including video records, audio records, image records, text records, event records, and the like. In order to facilitate searching through the repository, an operator may provide a search query via a user interface of a computer. The search query make take any suitable form, including natural language: that is to say, in the form of everyday parlance, as spoken or written by people.

[0010] Prior art systems suffer from the disadvantage that they tend to be fixed. In the past, systems relied on deterministic algorithmic conversion of text to database system queries, which would involve parsing the text to extract label and using a thesaurus to identify searchable terms in the labels. With the advent of Al, more complex systems could be made by training Al models using sets of user input queries and corresponding valid search queries. Both of these may be suitable if the searchable things do not change. However, recent and complex surveillance systems include ever increasing capabilities and searchable things. For example, new hardware may be supported to detect and record new things, or cameras may support new analytics to identify new metadata (e.g. detect different types of objects (e.g. luggage), new characteristics to objects that have previously detected (e.g. level of damage on a car), or existing types of objects may be expanded (e.g. new vehicle makes and models)). Thepresent disclosure provides a system that can be easily updated to reflect new searchable parameters without requiring a reprograming, a complex Al re-training and without necessarily requiring a software update. To this end, a general-purpose or non-dedicated-purpose Al model (in the primary example, a Large Language Model (LLM)) may be used and be adapted on- the-fly. This has the further advantage of allowing the present subject matter to benefit from improvements in Al models provided by external providers and to avoid a need to train an Al model or program a deterministic algorithm. The LLM produces a structured data source query based on the input query, and the repository is searched using the structured data source query to identify one or more query results. The query results may then be displayed via the user interface, for instance for review by the operator.

[0011] Moreover, the present disclosure relates to systems and methods for improving outputs generated by LLMs that function in connection with an application program such as a surveillance system application program. When user input is received to generate a query for an LLM, the systems and methods described herein analyze context information surrounding a usage by the user of the application program to generate context query data from the context information. The context query data provides information and insights on the application program used by the user before and / or at a time of receipt (real time) of the user input for the LLM query. The context query data is then processed along with the user input to formulate or reformulate the user search query to be inputted into the LLM, to improve the output generated by the LLM, thereby providing additional information to the LLM for permitting the LLM to derive the meaning of the user input that is the basis for the search query. The context query data provides additional information surrounding the activities of the user using the LLM that prompted the user to query the LLM. As such, this context surrounding the user’s activity is information that can further qualify the meaning behind the user input provided by the user for querying the LLM. This added context supplies information to the LLM to improve the analysis performed by the LLM and to enable the LLM to derive additional meaning from the user search query, for purposes of outputting an answer to the user’s search query that is closer to the intention of the user submitting the question to the LLM.

[0012] In accordance with a broad aspect, there is provided a method for searching a repository of surveillance records. An input query comprising natural language is obtained via a user interface. The input query is provided to a large language model (LLM), the LLM having been trained to translate natural language into data source queries structured based on metadata categories associated with the surveillance records stored in the repository. A structured data source query, based on the input query, is obtained from the LLM. The repository is searchedusing the structured data source query to identify at least one query result. The at least one query result is displayed via the user interface.

[0013] In at least some embodiments according to any one or more of the previous embodiments, the LLM may have been trained to translate natural language into data source queries comprising of one or more query couplets composed of a metadata category and a metadata characteristic.

[0014] In at least some embodiments according to any one or more of the previous embodiments, the LLM may have been trained to translate natural language into data source queries without producing any intermediary elements.

[0015] In at least some embodiments according to any one or more of the previous embodiments, providing the input query to the LLM may include providing the input query to the LLM without prior parsing of the input query.

[0016] In at least some embodiments according to any one or more of the previous embodiments, the LLM may have been trained to translate natural language into data source queries by predicting terms for the structured data source query as future tokens for the input query.

[0017] In at least some embodiments according to any one or more of the previous embodiments, the LLM predicting terms for the structured data source query may be performed without parsing the input query.

[0018] A broad aspect is a method for causing a searching of a data source of surveillance records. The method includes obtaining an input query, the input query comprising natural language; providing the input query to a large language model (LLM), for translating the natural language into a structured data source query based on searchable categories associated with the surveillance records stored in the data source; obtaining, from the LLM, a structured data source query based on the input query; transmitting the structured data source query to the data source to perform a search of the data source in accordance with the structured data source query to identify at least one query result; and receiving the at least one query results.

[0019] In some embodiments, the method may include causing a display, via the user interface, of the at least one query result.

[0020] In some embodiments, the searchable categories may be metadata categories.

[0021] In some embodiments, the searchable categories may form with corresponding metadata characteristics query couplets.

[0022] In some embodiments, providing the input query to the LLM may include providing the input query to the LLM without prior parsing of the input query.

[0023] In some embodiments, the method may include generating and transmitting instructions to the LLM for generating an application programming interface (API) call for an API related to a data source to be queried with the structured data source query.

[0024] In some embodiments, the method may include transmitting the at least one query results to the LLM; and generating and transmitting instructions to the LLM to generate a summary of the at least one query result.

[0025] In some embodiments, the method may include providing a permission level to the LLM, wherein the obtained structured data source query may include information related to the permission level.

[0026] In some embodiments, the method may include transmitting the at least one query result to the LLM; generating and transmitting instructions to the LLM for causing the LLM to analyse the at least one query result; receiving the output of the analysis from the LLM; and providing a response to a user computing device having generated the input query, the response corresponding to the received output.

[0027] In some embodiments, the instructions for causing the LLM to analyse the at least one query results may further cause the LLM to compare the at least one query result to the input query or the structured data source query to verily if the at least one query result satisfies the input query or the structured data source query.

[0028] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if additional search of the data source, or of one or more additional data sources is required.

[0029] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if a category of information identified in the input query is not searchable.

[0030] In some embodiments, the instructions may further cause the LLM to generate as an output a suggestion of an alternative category of information instead of the category of information identified in the input query that is not searchable.

[0031] In some embodiments, the method may include generating and transmitting instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input query is not searchable; and receiving an output from the LLM regarding if one or categories of information related to the input query is not searchable.

[0032] In some embodiments, the instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input queryis not searchable, may be to further cause the LLM to generate one or more alternative data categories to be searched related to one or more not searchable categories of information of the one or more categories of information; and wherein the received output may include the one or more alternative data categories to be searched related to one or more not searchable categories of information of the one or more categories of information.

[0033] In some embodiments, the obtained input query may be accompanied by metadata providing context information surrounding the obtained input query.

[0034] In some embodiments, the input query may be provided to the LLM along with the metadata providing context information or information derived from the metadata providing context information.

[0035] In some embodiments, the method may include generating a system prompt defining instructions on a manner of structuring the structured data prompt for the data source, and providing the system prompt along with the input query to the LLM.

[0036] Another broad aspect is a system for causing a searching of a data source of surveillance records. The system includes a processor; and memory comprising program code that, when executed by the processor, cause the processor to: obtain an input query, the input query comprising natural language; provide the input query to a large language model (LLM), for translating the natural language into a structured data source query based on searchable categories associated with the surveillance records stored in the data source; obtain, from the LLM, a structured data source query based on the input query; transmit the structured data source query to the data source to perform a search of the data source in accordance with the structured data source query to identify at least one query result; and receive the at least one query results.

[0037] In some embodiments, the program code may further cause the processor to cause a display, via the user interface, of the at least one query result.

[0038] In some embodiments, the data categories may be metadata categories.

[0039] In some embodiments, the metadata categories may form with corresponding metadata characteristics query couplets.

[0040] In some embodiments, providing the input query to the LLM may include providing the input query to the LLM without prior parsing of the input query.

[0041] In some embodiments, the program code may further cause the processor to generate and transmit instructions to the LLM for generating an application programming interface (API) call for an API related to a data source to be queried with the structured data source query.

[0042] In some embodiments, the program code may further cause the processor to: transmit the at least one query results to the LLM; and generate and transmitting instructions to the LLM to generate a summary of the at least one query result.

[0043] In some embodiments, the program code may further cause the processor to provide a permission level to the LLM, wherein the obtained structured data source query may include information related to the permission level.

[0044] In some embodiments, the program code may further cause the processor to: transmit the at least one query result to the LLM; generate and transmitting instructions to the LLM for causing the LLM to analyse the at least one query result; receive the output of the analysis from the LLM; and provide a response to a user computing device having generated the input query, the response corresponding to the received output.

[0045] In some embodiments, the instructions for causing the LLM to analyse the at least one query results may further cause the LLM to compare the at least one query result to the input query or the structured data source query to verily if the at least one query result satisfies the input query or the structured data source query.

[0046] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if additional search of the data source, or of one or more additional data sources is required.

[0047] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if a category of information identified in the input query is not searchable.

[0048] In some embodiments, the instructions may further cause the LLM to generate as an output a suggestion of an alternative category of information instead of the category of information identified in the input query that is not searchable.

[0049] In some embodiments, the program code may further cause the processor to: generate and transmitting instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input query is not searchable; and receive an output from the LLM regarding if one or categories of information related to the input query is not searchable.

[0050] In some embodiments, the instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input query is not searchable, may be to further cause the LLM to generate one or more alternative data categories to be searched related to one or more not searchable categories of information of the one or more categories of information; and wherein the received output may include the one ormore alternative data categories to be searched related to one or more not searchable categories of information of the one or more categories of information.

[0051] In some embodiments, the obtained input query may be accompanied by metadata providing context information surrounding the obtained input query.

[0052] In some embodiments, the input query may be provided to the LLM along with the metadata providing context information or information derived from the metadata providing context information.

[0053] In some embodiments, the program code may further cause the processor to generate a system prompt defining instructions on a manner of structuring the structured data prompt for the data source, and providing the system prompt along with the input query to the LLM.

[0054] Another broad aspect is non-transitory computer-readable medium having stored thereon program instructions for causing a searching of a data source of surveillance records, the program instructions executable by a processing unit for: obtaining an input query, the input query comprising natural language; providing the input query to a large language model (LLM), for translating the natural language into a structured data source query based on searchable categories associated with the surveillance records stored in the data source; obtaining, from the LLM, a structured data source query based on the input query; transmitting the structured data source query to the data source to perform a search of the data source in accordance with the structured data source query to identify at least one query result; and receiving the at least one query results.

[0055] In some embodiments, the program instructions may be further executable by the processing unit for causing a display, via the user interface, of the at least one query result.

[0056] In some embodiments, the data categories may be metadata categories.

[0057] In some embodiments, the metadata categories may form with corresponding metadata characteristics query couplets.

[0058] In some embodiments, providing the input query to the LLM may include providing the input query to the LLM without prior parsing of the input query.

[0059] In some embodiments, the program instructions may be further executable by the processing unit for generating and transmitting instructions to the LLM for generating an application programming interface (API) call for an API related to a data source to be queried with the structured data source query.

[0060] In some embodiments, the program instructions may be further executable by the processing unit for transmitting the at least one query results to the LLM; and generating andtransmitting instructions to the LLM to generate a summary of the at least one query result.

[0061] In some embodiments, the program instructions may be further executable by the processing unit for providing a permission level to the LLM, wherein the obtained structured data source query may include information related to the permission level.

[0062] In some embodiments, the program instructions may be further executable by the processing unit for: transmitting the at least one query result to the LLM; generating and transmitting instructions to the LLM for causing the LLM to analyse the at least one query result; receiving the output of the analysis from the LLM; and providing a response to a user computing device having generated the input query, the response corresponding to the received output.

[0063] In some embodiments, the instructions for causing the LLM to analyse the at least one query results may further cause the LLM to compare the at least one query result to the input query or the structured data source query to verify if the at least one query result satisfies the input query or the structured data source query.

[0064] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if additional search of the data source, or of one or more additional data sources is required.

[0065] In some embodiments, the instructions may further cause the LLM to interpret the at least one query results to determine if a category of information identified in the input query is not searchable.

[0066] In some embodiments, the instructions may further cause the LLM to generate as an output a suggestion of an alternative category of information instead of the category of information identified in the input query that is not searchable.

[0067] In some embodiments, the program instructions may be further executable by the processing unit for: generating and transmitting instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input query is not searchable; and receiving an output from the LLM regarding if one or categories of information related to the input query is not searchable.

[0068] In some embodiments, the instructions to the LLM for causing the LLM to analyze the input query to determine if one or more categories of information related to the input query is not searchable, may be to further cause the LLM to generate one or more alternative data categories to be searched related to one or more not searchable categories of information of the one or more categories of information; and wherein the received output includes the one or more alternative data categories to be searched related to one or more not searchable categoriesof information of the one or more categories of information.

[0069] In some embodiments, the obtained input query may be accompanied by metadata providing context information surrounding the obtained input query.

[0070] In some embodiments, the input query may be provided to the LLM along with the metadata providing context information or information derived from the metadata providing context information.

[0071] In some embodiments, the program instructions may be further executable by the processing unit for generating a system prompt defining instructions on a manner of structuring the structured data prompt for the data source, and providing the system prompt along with the input query to the LLM.

[0072] Another broad aspect is a method for updating a surveillance records search tool for searching data source of surveillance records, the data source having searchable parameters. The method includes providing a data representation of a system prompt to be transmitted along with user query to an Al model, the system prompt defining instructions on a manner of structuring a structured data prompt for the data source in accordance with the searchable parameters of the data source; modifying, replacing or supplementing the data representation of the system prompt to define instructions on a manner of structuring a structured data prompt for the data source in accordance with different searchable parameters of the data source; obtaining an input query, the input query comprising natural language; accessing the data representation of the system prompt; and generating an LLM query for a large language model (LLM) comprising a portion made with the input query and a portion made with the data representation of the system prompt.

[0073] In some embodiments, the data representation of the system prompt may be a text containing the system prompt.

[0074] In some embodiments, the LLM query may include the input query and the system prompt.

[0075] In some embodiments, the data representation of the system prompt may contain template elements each comprising portions of a potential system prompt, and wherein to generate an LLM query may include building a system prompt with a subset of the template elements.

[0076] In some embodiments, building the subset of template elements may be selected on the basis of the user query.

[0077] Another broad aspect is a non-transitory computer-readable medium having stored thereon program instructions for updating a surveillance records search tool for searching datasource of surveillance records, the data source having searchable parameters, the non-transitory computer-readable medium having stored thereon a data representation of a system prompt to be transmitted along with user query to an Al model, the system prompt defining instructions on a manner of structuring a structured data prompt for the data source in accordance with the searchable parameters of the data source the program instructions executable by a processing unit for: providing a data representation of a system prompt to be transmitted along with user query to an Al model, the system prompt defining instructions on a manner of structuring a structured data prompt for the data source in accordance with the searchable parameters of the data source; modifying, replacing or supplementing the data representation of the system prompt to define instructions on a manner of structuring a structured data prompt for the data source in accordance with different searchable parameters of the data source; obtaining an input query, the input query comprising natural language; accessing the data representation of the system prompt; and generating an LLM query for a large language model (LLM) comprising a portion made with the input query and a portion made with the data representation of the system prompt.

[0078] In some embodiments, the data representation of the system prompt may be a text containing the system prompt.

[0079] In some embodiments, the LLM query may include the input query and the system prompt.

[0080] In some embodiments, the data representation of the system prompt may contain template elements each comprising portions of a potential system prompt, and wherein to generate an LLM query may include building a system prompt with a subset of the template elements.

[0081] In some embodiments, building the subset of template elements may be selected on the basis of the user query.

[0082] Another broad aspect is a system for updating a surveillance records search tool for searching data source of surveillance records, the data source having searchable parameters comprising: a processor; and memory comprising a data representation of a system prompt to be transmitted along with user query to an Al model, the system prompt defining instructions on a manner of structuring a structured data prompt for the data source in accordance with the searchable parameters of the data source; program code that, when executed by the processor, cause the processor to modify, replace or supplement the data representation of the system prompt to define instructions on a manner of structuring a structured data prompt for the data source in accordance with different searchable parameters of the data source; program codethat, when executed by the processor, cause the processor to: obtain an input query, the input query comprising natural language; access the data representation of the system prompt; and generate an LLM query for a large language model (LLM) comprising a portion made with the input query and a portion made with the data representation of the system prompt.

[0083] In some embodiments, the data representation of the system prompt may be a text containing the system prompt.

[0084] In some embodiments, the LLM query may include the input query and the system prompt.

[0085] In some embodiments, the data representation of the system prompt may contain template elements each comprising portions of a potential system prompt, and wherein to generate an LLM query may include building a system prompt with a subset of the template elements.

[0086] In some embodiments, building the subset of template elements may be selected on the basis of the user query.

[0087] Another broad aspect is a method for improving an output generated by an application program-associated large language model (LLM)-based search tool following a user search query. The method includes receiving user input for generating the user search query for the LLM-based search tool, the LLM-based search tool configured to operate with the application program; analyzing context information of the application program and generating context query data for the application program derived from the context information; and generating the user search query from the user input, the generated user search query adapted with the generated context query data, the user search query for providing to the LLM-based search tool, wherein the search query is adapted to the context information of the application program, whereby the output is generated by the LLM-based search tool in response to the user search query.

[0088] In some embodiments, the context information may include historical use of the application program by the user.

[0089] In some embodiments, the context information may include information on a recent action performed by the user using the application program.

[0090] In some embodiments, the context information may include information on actions performed by the application program during a current session of running the application program.

[0091] In some embodiments, the context information may include information on a current use of the application program by the user.

[0092] In some embodiments, the context information may include information on modules of the application program that are open when the user input is received.

[0093] In some embodiments, the user search query may be a refined search query, where the method may include generating a crude search query from the user input, and wherein the refined user search query is generated from the crude search query that has been generated from the user input, the context query data refining the crude search query to arrive at the refined search query.

[0094] In some embodiments, the user input may include a string of characters defining a series of words.

[0095] In some embodiments, the user search query may include instructions to call an application program interface (API) for communicating with the application program.

[0096] In some embodiments, the application program may be a surveillance system application program.

[0097] Another broad aspect is a system for improving an output generated by an application program-associated large language model (LLM)-based search tool following a user search query. The system includes a processor; and memory storing program code that, when executed by the processor, cause the processor to: receive user input for generating the user search query for the LLM-based search tool, the LLM-based search tool configured to operate with the application program; analyze context information of the application program and generate context query data for the application program derived from the context information; and generate the user search query from the user input, the generated user search query adapted with the generated context query data, the user search query for providing to the LLM-based search tool, wherein the search query is adapted to the context information of the application program, whereby the output is generated by the LLM-based search tool in response to the user search query.

[0098] In some embodiments, the context information may include historical use of the application program by the user.

[0099] In some embodiments, the context information may include information on a most recent action performed by the user using the application program.

[0100] In some embodiments, the context information may include information on actions performed by the application program during a current session of running the application program.

[0101] In some embodiments, the context information may include information on a current use of the application program by the user.

[0102] In some embodiments, the context information may include information on modules of the application program that are open when the user input is received.

[0103] In some embodiments, the user search query may be a refined search query, and wherein the program code, when executed by the processor, may further cause the processor to generate a crude search query from the user input, and wherein the refined user search query is generated from the crude search query that has been generated from the user input, the context query data refining the crude search query to arrive at the refined search query.

[0104] In some embodiments, the user input may include a string of characters defining a series of words.

[0105] In some embodiments, the user search query may include instructions to call an application program interface for communicating with the application program.

[0106] In some embodiments, the application program may be a surveillance system application program.

[0107] Another broad aspect is a non-transitory computer-readable medium having stored thereon program instructions for improving an output generated by an application program- associated large language model (LLM)-based search tool following a user search query, the program instructions executable by a processing unit for receiving user input for generating the user search query for the LLM-based search tool, the LLM-based search tool configured to operate with the application program; analyzing context information of the application program and generating context query data for the application program derived from the context information; and generating the user search query from the user input, the generated user search query adapted with the generated context query data, the user search query for providing to the LLM-based search tool, wherein the search query is adapted to the context information of the application program, whereby the output is generated by the LLM-based search tool in response to the user search query.

[0108] In some embodiments, the context information may include historical use of the application program by the user.

[0109] In some embodiments, the context information may include information on a most recent action performed by the user using the application program.

[0110] In some embodiments, the context information may include information on actions performed by the application program during a current session of running the application program.

[0111] In some embodiments, the context information may include information on a current use of the application program by the user.

[0112] In some embodiments, the context information may include information on modules of the application program that are open when the user input is received.

[0113] In some embodiments, the user search query may be a refined search query, wherein the program instructions may be further executable by the processing unit for generating a crude search query from the user input, and wherein the refined user search query is generated from the crude search query that has been generated from the user input, the context query data refining the crude search query to arrive at the refined search query.

[0114] In some embodiments, the user input may include a string of characters defining a series of words.

[0115] In some embodiments, the user search query may include instructions to call an application program interface for communicating with the application program.

[0116] In some embodiments, the application program may be a surveillance system application program.

[0117] Another broad aspect is a method for improving an output generated by an application program-associated large language model (LLM)-based search tool following a user search query. The method includes transmitting context information pertaining to at least one of use and properties of the application program to the LLM-based search tool at least when the LLM-based search tool receives user input related to a query for the LLM-based search tool, whereby the output is generated by the LLM-based search tool in response to the user input, adapted with the context information.

[0118] In some embodiments, the context information may be transmitted continuously to the LLM-based search tool.

[0119] Another broad aspect is a method of generating by an application program an LLM query for transmission to an LLM tool to cause the LLM tool to generate a structured data source query for causing a searching of a data source of security records. The method includes obtaining an input query, the input query comprising natural language; accessing a computer- readable memory storing therein a system prompt template library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least one but not all of the system prompt template elements in the system prompt template library; processing the subset of system prompt template elements to generate the system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query; generating the LLM query comprising the received input query and the system prompt ; and transmitting thegenerated LLM query to the LLM tool to cause the LLM tool to generate the structured data source query for causing a searching of a data source of security records based on the received LLM query.

[0120] In some embodiments, the method may include: receiving the structured data source query in response to the LLM query; and searching the data source of security records according to the structured data source query.

[0121] In some embodiments, the one or more system prompt template elements in the subset of system prompt template elements may include usage logic including instructions defining when to use the respective system prompt template elements, wherein determining a subset of system prompt template elements may include interpreting the usage logic for respective system prompt elements in the system prompt template library to determine whether to use the respective system prompt template elements to generate the system prompt.

[0122] In some embodiments, the analysing the input query to determine an input query type; wherein interpreting the usage logic may include determining on the basis of the usage logic if the determined input query type corresponds to an input query type of the respective system prompt element defined in the usage logic of the respective system prompt element.

[0123] In some embodiments, the LLM query may include chat history of the LLM including one or more previous LLM queries, and one or more structured data source queries generated by the LLM tool in respective to the respective one or more previous LLM queries.

[0124] In some embodiments, the method may include further updating the system prompt template library.

[0125] In some embodiments, the updating may be performed following a receipt of an error message received from the searched data source following a receipt by the searched data source of the structured data source query.

[0126] In some embodiments, the updating may be performed following a software update of the searched data source, or of a system related to the searched data source.

[0127] In some embodiments, the updating may be performed following a passing of a predetermined amount of time.

[0128] In some embodiments, the updating may include changing a pointer location in memory of one or more of system prompt template elements of the system prompt template element library.

[0129] In some embodiments, the searched data source may be related to a video monitoring system.

[0130] In some embodiments, the security records may be image streams includingmetadata generated following an analysis of the corresponding image stream by a camera that generated the corresponding image stream.

[0131] In some embodiments, the system prompt template element library may include a system prompt template element with a variable field that is configured to be adapted in accordance with the determined input query type of the received input query.

[0132] In some embodiments, the determining of a subset of system prompt template elements may include determining an input query type of the received input query.

[0133] In some embodiments, the method may further include obtaining context data of the application program.

[0134] In some embodiments, the system prompt template elements library may include at least one system prompt template element related to the context data, wherein processing the subset of system prompt template elements to generate a system prompt may include processing the at least one system prompt template element to generate a portion of a system prompt instructions on how to use at least a portion of the context data.

[0135] In some embodiments, the usage logic of at least one system prompt template element of the subset of system prompt template elements may include logic indicating whether to use a respective system prompt template based on the context data.

[0136] In some embodiments, determining a subset of system prompt template elements may include interpreting the usage logic of the at least one system prompt template element in light of the context data and selecting the at least one system prompt template element as a result of this interpreting.

[0137] In some embodiments, the context query data may be generated from context information surrounding a usage by the user of the application program.

[0138] In some embodiments, the library of system prompt template elements may include at least a first set of system prompt template elements including instructions to the LLM on how to generate a search query, and at least a second set of system prompt template elements not related to searching including instructions to the LLM on how to perform a non-searching function.

[0139] In some embodiments, the second set of system prompt template elements may be related to device configuration and include instructions instructing the LLM on how to create a device configuration command for a device external to the application program.

[0140] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the method may further include receiving the device configuration command and transmitting the device configuration command to thedevice.

[0141] In some embodiments, the second set of system prompt template elements may be related to an application program configuration and includes instructions instructing the LLM on how to create an application program configuration command instructing the application program to change configuration.

[0142] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the method may further include receiving the application program configuration command and configuring the application program accordingly.

[0143] In some embodiments, the change of configuration including at least a modification to a graphical user interface (GUI) of the application program, and where the subset of system prompt template elements may include the second set of system prompt template elements, the method may further include receiving the application program configuration command, modifying the GUI accordingly, and causing the displaying of the modified GUI.

[0144] Another broad aspect is a method of upgrading a security application program having a search interface for searching through a data source of security records. The method including: accessing a computer-readable memory storing therein an original system prompt library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; and modifying at least a portion of the system prompt template element library to generate a modified system prompt template element library to change the system prompts generated using the security application program, the modified system prompt template element library comprising at least a modified portion of the original system prompt template element library.

[0145] In some embodiments, the method may include after modifying the system prompt template element library: obtaining an input query; accessing the computer-readable memory to access the modified system prompt library; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least a part of the modified portion of the original system prompt template element library; processing the subset of system prompt template elements to generate a system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query, the system prompt being different from the system prompt that would have been generated using the original system prompt template element library; generating the LLM query comprising the received input query and the system prompt ; and transmitting the generated LLM query to the LLM tool.

[0146] In some embodiments, modifying the system prompt template element library may include modifying at least a first system prompt template element of the library to change the system prompts that would be generated using the first system prompt template element.

[0147] In some embodiments, modifying the system prompt template element library may include adding at least one new system prompt template element to the library to permit new system prompt text to be generated.

[0148] In some embodiments, modifying the system prompt template element library may include deleting at least one system prompt template element from the library to prevent its respective system prompt text from being generated.

[0149] In some embodiments, the LLM tool may be a tool to generate the structured data source query for causing a searching of a data source of surveillance records based on the received LLM query, the method may further include: receiving the structured data source query in response to the LLM query; and searching the data source of surveillance records according to the structured data source query.

[0150] In some embodiments, the modified portion of the original system prompt template element library may include system prompt instructions for searching for new kinds of video metadata not supported in the original system prompt template element library.

[0151] In some embodiments, the method may be executed while the security application program is running, where the modifying at least a portion of the system prompt template element library may be performed during normal operation of the security application program such that security operations performed by the security application program do not need to be interrupted for the upgrading.

[0152] In some embodiments, the security application program ay include a pointer to the memory location in the computer-readable memory of the original system prompt template element library, and where the modifying at least a portion of the system prompt template element library may include: storing at a new location in the computer-readable memory as new system prompt template element library, the modified system prompt template element library, comprising the modified portion of the original system prompt template element library and a copy of any unmodified portion of the original system prompt template element library; and changing the pointer to the new location in the computer readable memory.

[0153] Another broad aspect is a system for generating by an application program an LLM query for transmission to an LLM tool to cause the LLM tool to generate a structured data source query for causing a searching of a data source of security records. The system including: a processor; and memory comprising program code that, when executed by the processor, causethe processor to: obtain an input query, the input query comprising natural language; access a computer-readable memory storing therein a system prompt template library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; determine a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least one but not all of the system prompt template elements in the system prompt template library; process the subset of system prompt template elements to generate the system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query; generate the LLM query comprising the received input query and the system prompt ; and transmit the generated LLM query to the LLM tool to cause the LLM tool to generate the structured data source query for causing a searching of a data source of security records based on the received LLM query.

[0154] In some embodiments, the memory includes program code that, when executed by the processor, may further causes the processor to: receive the structured data source query in response to the LLM query; and search the data source of security records according to the structured data source query.

[0155] In some embodiments, the one or more system prompt template elements in the subset of system prompt template elements may include usage logic comprising instructions defining when to use the respective system prompt template elements, where determining a subset of system prompt template elements may include interpreting the usage logic for respective system prompt elements in the system prompt template library to determine whether to use the respective system prompt template elements to generate the system prompt.

[0156] In some embodiments, the memory may include program code that, when executed by the processor, further causes the processor to analyse the input query to determine an input query type; wherein interpreting the usage logic may include determining on the basis of the usage logic if the determined input query type corresponds to an input query type of the respective system prompt element defined in the usage logic of the respective system prompt element.

[0157] In some embodiments, the LLM query may include chat history of the LLM comprising one or more previous LLM queries, and one or more structured data source queries generated by the LLM tool in respective to the respective one or more previous LLM queries.

[0158] In some embodiments, the memory may include program code that, when executed by the processor, further causes the processor to update the system prompt template library.

[0159] In some embodiments, the updating may be performed following a receipt of an error message received from the searched data source following a receipt by the searched datasource of the structured data source query.

[0160] In some embodiments, the updating may be performed following a software update of the searched data source, or of a system related to the searched data source.

[0161] In some embodiments, the updating may be performed following a passing of a predetermined amount of time.

[0162] In some embodiments, the updating may include changing a pointer location in memory of one or more of system prompt template elements of the system prompt template element library.

[0163] In some embodiments, the searched data source may be related to a video monitoring system.

[0164] In some embodiments, the security records may be image streams including metadata generated following an analysis of the corresponding image stream by a camera that generated the corresponding image stream.

[0165] In some embodiments, the system prompt template element library may include a system prompt template element with a variable field that is configured to be adapted in accordance with the determined input query type of the received input query.

[0166] In some embodiments, the determining of a subset of system prompt template elements may include determining an input query type of the received input query.

[0167] In some embodiments, the system may further include obtaining context data of the application program.

[0168] In some embodiments, the system prompt template elements library may include at least one system prompt template element related to the context data, wherein processing the subset of system prompt template elements to generate a system prompt may include processing the at least one system prompt template element to generate a portion of a system prompt instructions on how to use at least a portion of the context data.

[0169] In some embodiments, the usage logic of at least one system prompt template element of the subset of system prompt template elements may include logic indicating whether to use a respective system prompt template based on the context data.

[0170] In some embodiments, determining a subset of system prompt template elements may include interpreting the usage logic of the at least one system prompt template element in light of the context data and selecting the at least one system prompt template element as a result of this interpreting.

[0171] In some embodiments, the context query data may be generated from context information surrounding a usage by the user of the application program.

[0172] In some embodiments, the library of system prompt template elements may include at least a first set of system prompt template elements including instructions to the LLM on how to generate a search query, and at least a second set of system prompt template elements not related to searching comprising instructions to the LLM on how to perform a non-searching function.

[0173] In some embodiments, the second set of system prompt template elements may be related to device configuration and include instructions instructing the LLM on how to create a device configuration command for a device external to the application program.

[0174] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the method may further include receiving the device configuration command and transmitting the device configuration command to the device.

[0175] In some embodiments, the second set of system prompt template elements may be related to an application program configuration and include instructions instructing the LLM on how to create an application program configuration command instructing the application program to change configuration.

[0176] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the program code, when executed by the processor may further cause the processor to receive the application program configuration command and configuring the application program accordingly.

[0177] In some embodiments, the change of configuration including at least a modification to a graphical user interface (GUI) of the application program, and wherein the subset of system prompt template elements may include the second set of system prompt template elements, the program code, when executed by the processor may further cause the processor to receive the application program configuration command, modifying the GUI accordingly, and causing the displaying of the modified GUI.

[0178] Another broad aspect is a system for upgrading a security application program having a search interface for searching through a data source of security records. The system including: a processor; and memory comprising program code that, when executed by the processor, cause the processor to: access a computer-readable memory storing therein an original system prompt library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; and modify at least a portion of the system prompt template element library to generate a modified system prompt template element library to change the system prompts generated using the security applicationprogram, the modified system prompt template element library comprising at least a modified portion of the original system prompt template element library.

[0179] In some embodiments, the program code, when executed by the processor may further cause the processor to after modifying the system prompt template element library: obtain an input query; access the computer-readable memory to access the modified system prompt library; determine a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least a part of the modified portion of the original system prompt template element library; process the subset of system prompt template elements to generate a system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query, the system prompt being different from the system prompt that would have been generated using the original system prompt template element library; generate the LLM query comprising the received input query and the system prompt ; and transmit the generated LLM query to the LLM tool.

[0180] In some embodiments, modifying the system prompt template element library may include modifying at least a first system prompt template element of the library to change the system prompts that would be generated using the first system prompt template element.

[0181] In some embodiments, modifying the system prompt template element library may include adding at least one new system prompt template element to the library to permit new system prompt text to be generated.

[0182] In some embodiments, modifying the system prompt template element library may include deleting at least one system prompt template element from the library to prevent its respective system prompt text from being generated.

[0183] In some embodiments, the LLM tool may be a tool to generate the structured data source query for causing a searching of a data source of surveillance records based on the received LLM query, the program code, when executed by the processor may further cause the processor to: receive the structured data source query in response to the LLM query; and search the data source of surveillance records according to the structured data source query.

[0184] In some embodiments, the modified portion of the original system prompt template element library may include system prompt instructions for searching for new kinds of video metadata not supported in the original system prompt template element library.

[0185] In some embodiments, the program code may be executed while the security application program is running, wherein the modifying at least a portion of the system prompt template element library may be performed during normal operation of the security application program such that security operations performed by the security application program do notneed to be interrupted for the upgrading.

[0186] In some embodiments, the security application program may include a pointer to the memory location in the computer-readable memory of the original system prompt template element library, and wherein the modifying at least a portion of the system prompt template element library may include: storing at a new location in the computer-readable memory as new system prompt template element library, the modified system prompt template element library, comprising the modified portion of the original system prompt template element library and a copy of any unmodified portion of the original system prompt template element library; and changing the pointer to the new location in the computer readable memory.

[0187] Another broad aspect is a non-transitory computer-readable medium having stored thereon program instructions for generating by an application program an LLM query for transmission to an LLM tool to cause the LLM tool to generate a structured data source query for causing a searching of a data source of security records. The program instructions executable by a processing unit for: obtaining an input query, the input query comprising natural language; accessing a computer-readable memory storing therein a system prompt template library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least one but not all of the system prompt template elements in the system prompt template library; processing the subset of system prompt template elements to generate the system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query; generating the LLM query comprising the received input query and the system prompt ; and transmitting the generated LLM query to the LLM tool to cause the LLM tool to generate the structured data source query for causing a searching of a data source of security records based on the received LLM query.

[0188] In some embodiments, the program instructions may be further executable by the processing unit for: receiving the structured data source query in response to the LLM query; and searching the data source of security records according to the structured data source query.

[0189] In some embodiments, the one or more system prompt template elements in the subset of system prompt template elements may include usage logic including instructions defining when to use the respective system prompt template elements, wherein determining a subset of system prompt template elements may include interpreting the usage logic for respective system prompt elements in the system prompt template library to determine whether to use the respective system prompt template elements to generate the system prompt.

[0190] In some embodiments, program instructions may be further executable by the processing unit for analysing the input query to determine an input query type; wherein interpreting the usage logic may include determining on the basis of the usage logic if the determined input query type corresponds to an input query type of the respective system prompt element defined in the usage logic of the respective system prompt element.

[0191] In some embodiments, the LLM query may include chat history of the LLM including one or more previous LLM queries, and one or more structured data source queries generated by the LLM tool in respective to the respective one or more previous LLM queries.

[0192] In some embodiments, the program instructions may be further executable by the processing unit for further updating the system prompt template library.

[0193] In some embodiments, the updating may be performed following a receipt of an error message received from the searched data source following a receipt by the searched data source of the structured data source query.

[0194] In some embodiments, the updating may be performed following a software update of the searched data source, or of a system related to the searched data source.

[0195] In some embodiments, the updating may be performed following a passing of a predetermined amount of time.

[0196] In some embodiments, the updating may include changing a pointer location in memory of one or more of system prompt template elements of the system prompt template element library.

[0197] In some embodiments, the searched data source may be related to a video monitoring system.

[0198] In some embodiments, security records may be image streams including metadata generated following an analysis of the corresponding image stream by a camera that generated the corresponding image stream.

[0199] In some embodiments, the system prompt template element library may include a system prompt template element with a variable field that is configured to be adapted in accordance with the determined input query type of the received input query.

[0200] In some embodiments, the determining of a subset of system prompt template elements may include determining an input query type of the received input query.

[0201] In some embodiments, the program instructions may be further executable by the processing unit for obtaining context data of the application program.

[0202] In some embodiments, the system prompt template elements library may include at least one system prompt template element related to the context data, wherein processing thesubset of system prompt template elements to generate a system prompt may include processing the at least one system prompt template element to generate a portion of a system prompt instructions on how to use at least a portion of the context data.

[0203] In some embodiments, the usage logic of at least one system prompt template element of the subset of system prompt template elements may include logic indicating whether to use a respective system prompt template based on the context data.

[0204] In some embodiments, determining a subset of system prompt template elements may include interpreting the usage logic of the at least one system prompt template element in light of the context data and selecting the at least one system prompt template element as a result of this interpreting.

[0205] In some embodiments, the context query data may be generated from context information surrounding a usage by the user of the application program.

[0206] In some embodiments, the library of system prompt template elements may include at least a first set of system prompt template elements comprising instructions to the LLM on how to generate a search query, and at least a second set of system prompt template elements not related to searching comprising instructions to the LLM on how to perform a non-searching function.

[0207] In some embodiments, the second set of system prompt template elements may be related to device configuration and may include instructions instructing the LLM on how to create a device configuration command for a device external to the application program.

[0208] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the program instructions may be further executable by the processing unit for receiving the device configuration command and transmitting the device configuration command to the device.

[0209] In some embodiments, the second set of system prompt template elements may be related to an application program configuration and may include instructions instructing the LLM on how to create an application program configuration command instructing the application program to change configuration.

[0210] In some embodiments, the subset of system prompt template elements may include the second set of system prompt template elements, the program instructions further executable by the processing unit for receiving the application program configuration command and configuring the application program accordingly.

[0211] In some embodiments, the change of configuration including at least a modification to a graphical user interface (GUI) of the application program, and wherein the subset of systemprompt template elements may include the second set of system prompt template elements, the program instructions may be further executable by the processing unit for receiving the application program configuration command, modifying the GUI accordingly, and causing the displaying of the modified GUI.

[0212] Another broad aspect is a non-transitory computer-readable medium having stored thereon program instructions for upgrading a security application program having a search interface for searching through a data source of security records. The program instructions executable by a processing unit for : accessing a computer-readable memory storing therein an original system prompt library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; and modifying at least a portion of the system prompt template element library to generate a modified system prompt template element library to change the system prompts generated using the security application program, the modified system prompt template element library comprising at least a modified portion of the original system prompt template element library.

[0213] In some embodiments, the program instructions may be further executable by the processing unit after modifying the system prompt template element library for: obtaining an input query; accessing the computer-readable memory to access the modified system prompt library; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least a part of the modified portion of the original system prompt template element library; processing the subset of system prompt template elements to generate a system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query, the system prompt being different from the system prompt that would have been generated using the original system prompt template element library; generating the LLM query comprising the received input query and the system prompt ; and transmitting the generated LLM query to the LLM tool.

[0214] In some embodiments, modifying the system prompt template element library may include modifying at least a first system prompt template element of the library to change the system prompts that would be generated using the first system prompt template element.

[0215] In some embodiments, modifying the system prompt template element library may include adding at least one new system prompt template element to the library to permit new system prompt text to be generated.

[0216] In some embodiments, modifying the system prompt template element library may include deleting at least one system prompt template element from the library to prevent its respective system prompt text from being generated.

[0217] In some embodiments, the LLM tool may be a tool to generate the structured data source query for causing a searching of a data source of surveillance records based on the received LLM query, the program instructions may be further executable by the processing unit for: receiving the structured data source query in response to the LLM query; and searching the data source of surveillance records according to the structured data source query.

[0218] In some embodiments, the modified portion of the original system prompt template element library may include system prompt instructions for searching for new kinds of video metadata not supported in the original system prompt template element library.

[0219] In some embodiments, the program instructions may be executed by the processing unit while the security application program is running, wherein the modifying at least a portion of the system prompt template element library may be performed during normal operation of the security application program such that security operations performed by the security application program do not need to be interrupted for the upgrading.

[0220] In some embodiments, the security application program may include a pointer to the memory location in the computer-readable memory of the original system prompt template element library, and wherein the modifying at least a portion of the system prompt template element library may include: storing at a new location in the computer-readable memory as new system prompt template element library, the modified system prompt template element library, comprising the modified portion of the original system prompt template element library and a copy of any unmodified portion of the original system prompt template element library; and changing the pointer to the new location in the computer readable memory.

[0221] According to at least one embodiment, a method carried out by a computer comprises: receiving user input entered into a query interface associated with an application program running on the computer; and processing the user input and information on usage of the application program to generate a query for a large language model (LLM)-based promptresponse system.

[0222] In some embodiments, the processing comprises: structuring the received user input into structured user input compatible with the prompt-response system; and structuring the information on usage of the application program into structured usage data compatible with the prompt-response system. The query may include the structured user input and the structured usage data.

[0223] In some embodiments, the method further comprises: providing the query to the prompt-response system; and receiving an output generated by the prompt-response system in response to the query.

[0224] In some embodiments, the user input comprises restrictions and / or requirements on a format of the output generated by the prompt-response system, the query provided to the prompt-response system includes the restrictions and / or requirements, and the output generated by the prompt-response system is in a format that corresponds to the restrictions / requirements.

[0225] In some embodiments, the user input or the query provided to the prompt-response system includes instructions to cause the prompt-response system to generate an API call or content for producing an API call.

[0226] In some embodiments, the method further comprises causing the output generated by the prompt-response system to be conveyed to a user via the query interface associated with the application program.

[0227] In some embodiments, the computer is a first computer and wherein the query interface is implemented by a second computer connected to the first computer over a data network.

[0228] In some embodiments, the prompt-response system is executed by a third computer different from both the first computer and the second computer.

[0229] In some embodiments, the prompt-response system is executed by a plurality of computers.

[0230] In some embodiments, the prompt-response system comprises a plurality of LLM- based prompt-response systems.

[0231] In some embodiments, the plurality of prompt-response systems comprises a first prompt-response system operable to provide an output of a first type and a second promptresponse system operable to provide an output of a second type, the second type different from the first type.

[0232] In some embodiments, the method further comprises accessing a memory of the computer to obtain the information on usage of the application program.

[0233] In some embodiments, the information on usage of the application program includes information on at least one of current and historical usage of the application program.

[0234] In some embodiments, the information on usage of the application program includes information on at least one of current and historical usage of the application program by the user.

[0235] In some embodiments, the historical usage of the application program includes information on user actions performed by the application program during a current session of the application program.

[0236] In some embodiments, the information on usage of the application program includesinformation on actions performed by the application program during a current session of the application program.

[0237] In some embodiments, the information on usage of the application program includes information on a recent action performed by a user using the application program.

[0238] In some embodiments, the information on usage of the application program includes a version of the application program.

[0239] In some embodiments, the application program is associated with a plurality of software modules and the information on usage of the application program comprises information on those of the software modules that are open when the user input is received.

[0240] In some embodiments, the application program is associated with a plurality of software modules and the information on usage of the application program comprises information on those of the software modules for which an associated window is caused to be displayed by the computer on a screen when the user input is received.

[0241] In some embodiments, the application program is associated with at least one plugin and the information on usage of the application program comprises information on the at least one plugin.

[0242] In some embodiments, the information on usage of the application program comprises information on an indicator indicating a current status of at least one software feature of the application program.

[0243] In some embodiments, the indicator comprises a flag.

[0244] In some embodiments, the application program is a surveillance system application program.

[0245] In some embodiments, the surveillance system application program is operable to perform video surveillance of an area.

[0246] In some embodiments, the surveillance system application program is operable to control access to an area.

[0247] In some embodiments, the user input includes a string of characters defining a series of words.

[0248] In some embodiments, the information on usage of the application program comprises information on the user and the method further comprises obtaining the information on the user.

[0249] In some embodiments, the information on the user comprises information on access privileges of the user.

[0250] In some embodiments, the method further comprises: receiving a response from theprompt-response system, the response including instructions to generate commands for causing an action in the application program; and carrying out the instructions to cause said action in the application program.

[0251] In some embodiments, the method further comprises: receiving a response from the prompt-response system, the response including instructions for querying at least one data store associated with the application program; and querying the at least one data store based on the instructions.

[0252] In some embodiments, the data store comprises at least one of a database, a table and a spreadsheet.

[0253] In some embodiments, generating the query comprises refining the user input based on the information on usage of the application program.

[0254] In some embodiments, the processing comprises determining a meaning of the user input based at least in part on the information on usage of the application program.

[0255] In some embodiments, determining the meaning of the user input comprises determining an intention of a user of the application program.

[0256] In some embodiments, the processing comprises determining at least one relationship between the user input and the information on usage of the application program.

[0257] In some embodiments, the processing constrains the user input based at least in part on the information on usage of the application program.

[0258] In some embodiments, the information on usage of the application program comprises an output of the application program.

[0259] In some embodiments, the output of the application program comprises an image frame.

[0260] In some embodiments, the output of the application program limits the user input.

[0261] According to at least one embodiment, a computer program product comprises a computer readable memory storing computer executable instructions thereon that when executed by a computer perform any method described herein.

[0262] According to at least one embodiment, a system comprises a processor, the processor configured to, at least: receive user input entered into a query interface associated with an application program running on the processor; and process the user input and information on usage of the application program to generate a query for a large language model (LLM)-based prompt-response system.

[0263] Other aspects and features will become apparent to those ordinarily skilled in the art upon review of the following description of illustrative embodiments in conjunction withthe accompanying figures.Brief Description of the I

[0264] The invention will be beter understood by way of the following detailed description of embodiments of the invention with reference to the appended drawings, in which:

[0265] FIGs. 1A-B are exemplary graphical user interfaces for a search application;

[0266] FIGs. 2, 3, and 4 are exemplary graphical user interfaces for a search application;

[0267] FIG. 5 is a block diagram of an exemplary computing system;

[0268] FIG. 6 is a block diagram of an exemplary surveillance software platform or security system platform;

[0269] FIG. 7 is a flowchart illustrating an exemplary method for searching a repository of surveillance records;

[0270] FIG. 8 is a block diagram of an exemplary system for querying one or more data sources using an LLM, in communication with one or more client computing devices, one or more data sources and one or more servers;

[0271] FIG. 9 is a block diagram of exemplary software architecture for querying one or more data sources using a structured data source query generated an LLM; and

[0272] FIG. 10 is a flowchart diagram of an exemplary method for querying one or more data sources using a structured data source query generated by an LLM;

[0273] FIG. 11 is a block diagram of an exemplary system for improving LLM outputs in communication with an external computing device and an exemplary server;

[0274] FIG. 12 is a block diagram of an exemplary computing device;

[0275] FIG. 13 is a block diagram of exemplary software architecture for improving an LLM output; and

[0276] FIG. 14 is a flowchart diagram of an exemplary method of improving an output generated by an LLM in response to a user search query;

[0277] FIG. 15 is a block diagram of an exemplary agent module of a security application program in communication with an exemplary LLM tool;

[0278] FIG. 16 is a flowchart diagram of an exemplary method of adapting a system prompt for an LLM query;

[0279] FIG. 17A is a block diagram of exemplary memory of an exemplary security system storing an exemplary system prompt template element library;

[0280] FIG. 17B is a block diagram of an exemplary security system in communication with memory storing an exemplary system prompt template element library;

[0281] FIG. 18 is a flowchart diagram of an exemplary method of upgrading a systemprompt template element library;

[0282] FIG. 19 is a schematic illustration of a computer system according to one embodiment;

[0283] FIG. 20 is a schematic illustration of an example output of the computer system of FIG. 19;

[0284] FIG. 21 is a block diagram showing program code for obtaining an output from an LLM-based prompt-response system according to one embodiment;

[0285] FIG. 22A is a schematic illustration of a query for an LLM-based prompt-response system according to one embodiment; and

[0286] FIG. 22B is a schematic illustration of a query for an LLM-based prompt-response system according to another embodiment.Detailed Description

[0287] The present disclosure relates to, inter alia, methods, systems, devices, and computer-readable media for searching a repository of security records, such as surveillance records, which may include video records, audio records, image records, text records, event records, and the like. The repository may be structured in any suitable fashion, including as a relational data source or other type of data source, and may associated various metadata with each entry. In addition, although reference will be made herein to searching of the repository, it should be understood that searching of real-time or near-real-time events and entities is also considered.

[0288] EXEMPLARY SYSTEM FOR GENERATING A STRUCTURED DATA SOURCE QUERY USING AN LLM:

[0289] Reference is made to Figure 8, illustrating an exemplary server system 100 for generating a structured data source query using a Large Language Model (LLM), for querying a data source. The system 100 is in communication with one or more external servers 120 and one or more client computing devices 130 for a user, and one or more data sources 110.

[0290] The system 100 includes a processor 102, memory 101 and an I / O interface 103.

[0291] The processor 102 may be a general-purpose programmable processor. In the example of Fig. 8, the processor 102 is shown as being unitary, but the processor 102 may also be multicore, or distributed (e.g. a multi-processor).

[0292] The computer readable memory 101 stores program instructions and data used by the processor 102. The memory 101 stores, for instances, user input queries, structured data source queries, query results, etc. The computer readable memory 101, though shown as unitary for simplicity in the example of Fig. 8, may comprise multiple memory modules and / orcaching. In particular, the memory 101 may comprise several layers of memory such as a hard drive, external drive (e.g. SD card storage) or the like and a faster and smaller Random Access Memory (RAM) module. The RAM module may store data and / or program code currently being, recently being or soon to be processed by the processor 102 as well as cache data and / or program code from a hard drive. A hard drive may store program code and be accessed to retrieve such code for execution by the processor 102 and may be accessed by the processor 102 to store and access data. The memory 101 may have a recycling architecture for storing, for instance, structured search queries, search history, input queries, etc., where older data fdes are deleted when the memory 101 is full or near being full, or after the older data fdes have been stored in memory 101 for a certain time.

[0293] The I / O interface(s) 103 is in communication with the processor 102. The I / O interface(s) 103 may include a network interface and may be a wired or wireless interface for establishing a remote connection with, for example, one or more data sources 110 (e.g. through application program interface(s) - API(s)) via a network 140 (e.g. the Internet), and / or one or more servers 120 and / or one or more client computing devices 130. For instance, the I / O interface(s) 103 may be an Ethernet port, a WAN port, a TCP port, etc.

[0294] The processor 102, the memory 101 and the I / O interface(s) 103 may be linked via bus connections.

[0295] The client computing device 130 may be a remote computing device (i.e. client). One or more client computing devices 130 may be provided.

[0296] The client computing device 130 is in communication with the I / O interface(s) 103 of the server system 100. The computing device 130 has a processor 132, a memory 131, I / O interface(s) 133 that may be linked via bus connections. The computing device 130 may have (or be connected to) any user input interface 135, for example, such as a keyboard, a mouse, a touchscreen, etc. The computing device 130 may be a desktop computer, a laptop, a smartphone, a tablet, etc. The computing device 130 has (or is adapted to connect to) a display 134 (e.g. a screen, a tactile display, etc.). The processor 132, the memory 131 and the I / O interface(s) 133 may be similar to the processor 102, the memory 101 and the I / O interface(s) 103, respectively.

[0297] A client application program may be stored in memory of the computing device 130 that is associated with the server system 100, the client application program providing the user with an interface to interact with the server system 100. The client application program may be for interacting with the application program that is associated with the LLM tool (e.g. for inputting the input query as a string of characters, a selection of one or more options fordifferent fields of information to be queried, etc.)

[0298] The client computing device 130 may be operated by user(s) to access a webpage or graphical user interface (GUI) for interacting with the application program associated with the LLM tool, the application program managed by the system 100.

[0299] The system 100 may be in communication with one or more data sources 110 via the network 140. The system 100 may be in communication with one or more client computing devices 130 via the network 140. The system 100 may be in communication with one or more external servers 120 via the network 140.

[0300] The one or more external servers 120 are configured to run the LLM. The one or more external servers 120 may also cause the running of the application program that is associated with the LLM tool. The system 100 may be configured to interact with the external computing device 130 to receive from the external computing device 130 the user input for generating the user search query. The system 100 may be configured to interact with one or more of the servers 120 and / or the external computing devices 130 to receive the context information. The system 100 may be then configured to transmit to the one or more servers 120 the refined user search query for querying the LLM tool.

[0301] In some instances, the functionality of the one or more servers 120 may be integrated into the system 100, where the system 100 is configured to run the LLM tool and the application program that is associated with the LLM tool. More specifically, it is possible to run locally hosted Al models, including LLMs. In alternate embodiments, the function of the LLM described herein is implemented by a local LLM.

[0302] In some instances, the system 100 may be the same device as the client computing device 130, where the program code for improving the output of the LLM is stored in memory of the client computing device 130. For example, in the instant example, a cloud-based system physical security system may be provided, including for example a video management system (VMS), access control system (ACS) or the like, whereby the system is implemented principally on server system 100 which may be a cloud-bases system. In this example, the client computing device 130 may be running locally to the user on a computer or like device where a client application runs that allows access to the cloud-based system 100. Data sources 110 may be implemented in Cloud storage. Server 120 may host an external service, typically also Cloud-based. For example, GPT services from Azure Open Al services may be used. These are typically accessed via API calls to the service which the system 100 creates and transmits.

[0303] Other architectures are possible. In other examples, for example, a local “on premises” system may be provided. In such systems, you a server-client architecture may bepresent where the functionality of server system 100 is provided on a server and the client computing device 130 is another local computing device communicating with the server system over a network. In such an example, the data source 110 tends to be server storage. It is possible that in simpler systems, the server system 100 and client computing device 130 are implemented by the same physical device, however this is not common in complex deployments or ones requiring complex searching systems.

[0304] The data source(s) 110 includes or stores surveillance records to be queried following receipt of the structured data source query generated by the LLM. The data source(s) 110 may be repositories of surveillance data such as media. It may be, for example, a data source of surveillance videos, audio fdes, biometric data, licence plate read events, a criminal data source, biometric data, calls to law enforcement (e.g. 911 calls), etc. The data source(s) may also be one or more devices for generating surveillance records, such as a camera that generates images or video, a badge reader that generates badge read events, a microphone that generates audio fdes, etc. The data source(s) may be queried by the system 100, the client computing device 130 and / or the LLM running on the external servers 120.

[0305] The data source(s) 110 may have searchable categories (also referred to herein as searchable parameters), which are the things you can search in the data sources 110. In some systems, these searchable categories may change over time. As described above, searchable parameters may increase if, for example, new types of video metadata become available thanks to new analytics. For example, cameras running analytics modules (or server software applying analytics to received video) may today be able to detect a vehicle make and model. Later, these cameras may be upgraded, or new more capable cameras may be connected to the system that can detect new types of makes and models, or new characteristics to vehicles such as damage, the presence of roof racks, etc. In such a case, the data source(s) 110 may have new searchable parameters meaning that new things can be searched. Searchable parameters can define what can be searched in the data source. For example, the different types of metadata in video that can be searched or the vehicle characteristics that can be searched are searchable categories.

[0306] EXEMPLARY SOFTWARE ARCHITECTURE FOR GENERATING A STRUCTURED DATA SOURCE QUERY USING AN LLM:

[0307] Reference is made to Figure 9, illustrating an exemplary software architecture 300 for generating a structured data source query using an LLM.

[0308] For purposes of illustration, the system 100 is described herein as having program code, stored in memory 101, that includes the input query module 310, the prompt module 320 and the query module 330. Each of the input query module 310, the prompt module 320 andthe query module 330 includes program code configured to implement the functionality of the modules as are described herein.

[0309] The input query module 310 includes program code stored in memory 101 that, when executed by the processor 102, causes the processor 102 to receive an input query in natural language inputted by the user or generated by a system. The input query may be provided as text (a string of characters) entered by the user in a graphical user interface displayed on, e.g., a display 134 of the remote computer 130, an audio file, etc. For the purpose of illustration, it will be presumed herein that the input query is text entered by the user, but it will be appreciated that other known input methodologies could be employed, such as voice input, implementing for example speech-to-text. The input query is the search request provided by the user, defining the nature of search to be conducted in one or more data sources of surveillance records. For instance, the input query may be the following string of characters: “look for videos of a man with a skull t-shirt in the building at 1 Square Samantha today between 1 PM and 3 PM”.

[0310] In some embodiments, the input query module 310 may cause the processor 102 to parse the string of characters received as the input query.

[0311] The prompt module 320 includes program code stored in memory 101 that, when executed by the processor 102, causes the processor 102 to generate an LLM query for transmission and processing by an LLM to generate a structured search query. The LLM query is the instructions and information for transmission to the LLM for causing the LLM to generate an output when processing the LLM query. The prompt module 320 causes the processor 102 to receive the input query from input query module 310. The prompt module 320 causes the processor 102 to generate an LLM query, including the input query and the prompt (e.g. system prompt), for transmission to the LLM, instructing the LLM to generate a structured search query for querying one or more data sources in accordance with the received input query in natural language. As such, the prompt module 320 may cause the processor 102 to generate a system prompt comprising instructions for causing the LLM to perform one or more actions in accordance with the system prompt, such as instructions to generate a database system query such as an API call for communicating with the one or more data sources 110, for including a permission level in the outputted structured search query associated with the user, to confirm that the user has security clearance for accessing the one or more data sources 110, etc. The system prompt may be parametrised according to the searchable parameters. More specifically, the system prompt may define what constitutes a valid database system query within the searchable parameters. For example, the system prompt may provide the LLM with thesearchable parameters and the proper format for searching through the searchable parameters, e.g. the proper format for an API call to search the data source(s) 110. The prompt module 320 causes the processor 102 to transmit the LLM query to the LLM. In some embodiments, the prompt module 320 may cause the processor 102 to generate an API call for communicating with an application programming interface (API) of the LLM.

[0312] The query module 330 includes program code stored in memory 101 that, when executed by the processor 102, causes the processor 102 to receive the structured data source query generated by the LLM and transmit the structured data source query to the one or more data sources 110 to be queried. When the structured data source query includes an API call, the query module 330 may cause the processor 102 to transmit the structured data source query to the API of the data source 110, for accessing and querying the data source 110.

[0313] The query module 330 may cause the processor 102 to receive the search results that have been queried from the one or more data sources 110 (e.g. one or more surveillance records retrieved in response to the query). The processor 102 may then be caused by the query module 330 to transmit the search results to the client computing device 130 from which originated the initial input query. In some examples, the query module 330 may cause the processor 102 to generate a prompt for the LLM with the queried search results, the prompt instructing the LLM to summarize the queried search results. In some examples, the query module 330 may cause the processor 102 to generate a prompt for the LLM with the queried search results, the prompt instructing the LLM to compare the queried search results to the structured data source query and / or the input query to analyze whether the received queried search results adequately answer the structured data source query and / or the input query. In some instances, the processor 102 may be caused to include in the prompt to the LLM instructions to generate a further structured data source query from the input query if the LLM determines that the received queried search results do not adequately answer the structured data source query and / or the input query.

[0314] In some embodiments, the query module 330 may cause the processor 102 to transmit the credentials of the user or system, tied to the permission level of the user or the system, with the structured database query. The data source 110 receives the credentials along with the structured database query (for verification if the user or system has the proper credentials for accessing the surveillance records found within the data source 110).

[0315] EXEMPLARY METHOD OF QUERYING A DATA SOURCE USING A STRUCTURED DATA SOURCE QUERY GENERATED BY AN LLM:

[0316] Reference is now made to Figure 10, illustrating an exemplary method 2000 ofquerying a data source using a structured data source query generated by an LLM. The method 2000 may be performed by the system 100. However, it will be understood that the method 2000 may be performed by any system in accordance with the present teachings.

[0317] An input query provided by a user is received at step 2100. The input query specifies the nature of the search that the user or system is looking to perform through one or more data sources. For instance, the input query may be entered on a graphical user interface generated by a client application program as a series of characters forming one or more phrases. In some instances, the input query may be or include a selection of one or more options provided in a series of fields listing categories of characteristics to define the scope of the query (e.g. categories may include (date), (location), (time), (type of record), (type of crime), etc.) In some embodiments, the input query may be or include an audio file, where the input query is given as instructions provided orally.

[0318] In some instances, the input query may be accompanied by metadata providing related information on the input query. The metadata may be fields containing information related to the input query in natural language, such as an identifier related to the user account that generated the unique identifier, an IP address of the remote computing device that generated the unique identifier, a time for when the unique identifier was generated, a permission level corresponding to a security clearance level for the user or system that generated the unique identifier, etc.

[0319] An LLM query is generated for an LLM, containing instructions and data to cause the LLM to convert the received input query into a structured data source query for querying one or more data sources at step 2200. The LLM query includes the input query and a prompt, where the prompt includes instructions to be processed by the LLM to generate the structured data source query from the input query. The prompt may include other instructions for processing by the LLM. For instance, the prompt may cause the LLM to generate an API call for the API of the data source to be queried by transmitting the structured data source query to the data source. As such, the API call that accompanies the structured data query may call upon the API in order to access the related data sources, for retrieving the surveillance records corresponding to the structured data source query. In some instances, the LLM query may include information on credentials or a clearance level of the user who or system that originated the input query (e.g. a unique identifier or passcode for that user or system). The unique identifier may be provided as part of the LLM query, where the prompt of the LLM query may instruct the LLM to provide the unique identifier as accompanying information with the structured data source query, such that the data source or API can evaluate the unique identifierto determine if the user or system has authorization to access some or all of the surveillance records of the data source.

[0320] In some instances, the metadata that may accompany the input query may also be provided as part of the LLM query to be provided to the LLM, where the metadata may provide additional context information on the input query for the LLM to generate the corresponding structured data source query. The metadata may be fields containing information related to the input query in natural language, such as an identifier of the user account that generated the unique identifier, an IP address of the remote computing device that generated the unique identifier, a time for when the unique identifier was generated, a permission level corresponding to a security clearance level for the user or system that generated the unique identifier, etc.

[0321] In some instances, LLM query may include a system prompt for causing the LLM to generate the structured data source query in accordance with a format or style based on the configuration of the one or more data sources to be queried.

[0322] The following is an example of a system prompt generated to be transmitted including instructions to generate content for an API call in the output:System:You are responsible for creating a JSON document used for an API to search for objects and individuals within CCTV recordings. Do not answer with anything other than JSON.The output must contain a suitable start and end detection time range in the field 'detectionTimeRange' based on user input and current time, if the user specifies a duration (e.g. 'in the last 4 hours' or '4pm to 5pm), use exactly that duration as a time range, if the user specifies a point in time (e.g. '15min ago' or y ' esterday)', an time range should be assumed using an appropriate margin around the specified time.For example: when the user asks for the last 10 minutes the time range be from 15 minutes to 5 minutes in the past.The current time is 2024-07-11 T13:30:02.0586560Z. if the user specifies a past point in time rather than a duration, make sure to add a suitable duration before and after.For example: when the user asks for results at 4am today, select a time range between 3am and 5am.The output must contain filters suitable to the user query.Valid people clothing and vehicle colors are 'White', 'Green', 'Red', 'Black', 'Blue', 'Yellow'.Valid vehicle types are 'Bike', 'Car', 'Truck', 'Bus’.Consider the vehicle type 'Bike' to include any common 2 -wheeled vehicle like motorcycles.Not directly supported colors should be mapped to the doses valid colors.Not directly supported vehicle types should be mapped to the closest valid vehicle types.Consider using multiple alternative colors or vehicle types in the "oneOf" field.Try to map well-known brand names of vehicles to their respective vehicle types.Try to map names of people to the 'Person' class.If no object class can be derived from the user input, the filter should match 'Person' or 'Vehicle' with respective attributes (if applicable). If no attribute filter can be derived from the user input for a given class, an empty object should be used to search for that class with any attribute.If the clothing type (top / bottom) can't be derived from the user input, the response should contain two person filters. One for the respective top clothing color and another person filter with the respective bottom clothing color.

[0323] The system prompt may also specify a format for the output generated by the LLM(in the present example, the format is a JSON schema):The output must conform to following JSON schema: json{"type ": "object", "properties ":{"reasoning":{"type ": "string"}, "pa yioad":{" type ": "object", "properties ": {"person ".-{"type ": "array", "items ":{"type ": "o bject", "properties ":{"topCiothingCoior":{"type ": "object", "properties ":{" oneOf":{"type ": "arra y ", "items ":{"type ": "string", "enum ": ["White ", "Black" , "Green", "Blue", "Red", "Yellow"]}}}}, "bottomCiothingCoior":{"type":"obj ect", "properties ":{"oneOf":{"type ": "array", "items ":{"type ": "string", "enu m": ["White", "Black", "Green", "Blue", "Red", "Yellow"]}}}}}}}, "vehicie":{"ty pe ": "arra y ", "items ":{"type ": "object", "properties ":{"vehicieCoior":{"type " : "object", "properties ":{"oneOf":{"type ": "array", "items ":{"type ": "string", " enum":["White", "Black", "Green", "Blue", "Red", "Yellow"]}}}}, "vehicieTyp e ":{"type ": "object", "properties ":{"oneOf":{"type ": "arra y ", "items ":{"type " :"string", "enum": ["Bus", "Bike", "Car", "Truck"]}}}}}}}, "detectionTimeRan ge ":{"type ": "object", "properties ":{"start":{"type ": "string", "format": "date -time "}, "end": {"type ": "string", "format": "date-time "}}}}}}}[additional details for other types of requests such as LPR related request or access control related requests]

[0324] The instructions for generating the prompt may cause the prompt to include part of the user search query may be history of past user search query(ies) and corresponding output(s) generated by the LLM:Prior search 1:User: Find the Woman in the Red dressContext: {Software version: 1.2.3; modules: video, access control, LPR; active pane: live video; selected device: camera 01, camera 02; current action: viewing live video; recent pane: live video; recent selected devices: camera 01, camera 02, camera 06, camera 09; recent actions: object tracking}Assistant: {"reasoning":"Not supported: classification (woman), clothing type (dress)¥nSearching for a *person* with a *top clothingcolor* containing*red*. ", "payioad":{"person ":[{"topCiothingCoior":{"oneOf":["Red"J}}J}} Prior search 2:User: intruder with blue shirt and grey trousers, 30 minutes ago Context: {Software version: 1.2.3; modules: video, access control, LPR; active pane: live video; selected device: camera 01, camera 02; current action: viewing live video; recent pane: live video; recent selected devices: door 02, camera 06, camera 09; recent actions: access control, door view, object tracking}Assistant: {"reasoning":"Not supported: classification (intruder), clothing type (trousers), color (grey), door forced open (door 02)¥nSeiected Time Range: today [40min ago, 20min ago]¥nSearching fora *person* with a *top clothing color* containing *biue* and *bottom clothing color* containing * white* or *biack* or a door forced open event on door02. ", "payload":{"person":[{"topClothingColor":{"oneOf":["Blue"]}, "botto mCiothingCoior":{"oneOf":["White ", "Black"]}}], "event": ["doorforcedope n ":("door02") ], "detection TimeRange ": {"start": "2024-07-11 T12:50:02.058656Z", "end": "2024-07-11 T13:10:02.058656Z"}}}Prior search 3User: red hondaContext: {Software version: 1.2.3; modules: video, access control, LPR; active pane: hit list read; selected device: LPR camera 03; current action: LPR list; recent pane: access control, LPR hitlist; recent selected devices: LPR camera 03, badge reader 04; door 02; recent actions: LPR viewer; badge read viewer]Assistant: {"reasoning":"Not supported: vehicle brand(honda), hitiist(yes)¥nSearching for a *vehicie* with a *vehicie type* containing *car* or *bike* and *vehicie color* containing *red* and listed in the hitlist. ", "payload":{"vehicle":[{"vehicleColor":{"oneOf":["Red"]}, "vehicle Type ":{"oneOf":["Car", "Bike "]}, "hitlist": {["yes "]}}]}}

[0325] The LLM query includes the input query (e.g. the string of characters received asuser input for the current user search query:)USER: “find the guy in the red shirt”

[0326] The generated prompt is transmitted to the LLM at step 2300. In some instances, the LLM may have an API, where calling the API permits interaction with the LLM. In these instances, the prompt may be accompanied with an API call for communicating with the API of the LLM. The LLM then executes the action in accordance with the prompt, generating the structured data source query and, optionally, any other output in accordance with the received prompt. The output of the LLM may be in the form of a string of characters, forming the structured data source query and, optionally, any other output in accordance with the received prompt.

[0327] The structured data source query generated by the LLM may be received at step 2400 by the LLM. The structured data source query defines one or more searchable categories or parameters (categories of information related to the surveillance record, where the one or more searchable categories are searchable when performing the query). Different types of surveillance records may include different searchable categories. Different data sources may include different searchable categories. For instance, the one or more searchable categories may be metadata tags or categories, where corresponding values for each of the metadata tags or categories may be related, as couplets, to the metadata tags or categories. For instance, exemplary metadata tags may be {time}, {date}, {location}, {object_type}, etc.

[0328] The structured data source query is transmitted to the one or more data sources to be queried at step 2500. In some embodiments, where there are a plurality of data sources to be queried, the structured data source query may be transmitted to a server that, subsequently, directs the structured data source query to the appropriate data source(s), in accordance with the instructions provided in the structured data source query (e.g. based on the nature of the surveillance records to be queried and the type of surveillance record stored in a given data source).

[0329] In some embodiments, when the structured data source query is accompanied by an API call generated by the LLM, the API call may call upon the API of the related data source, the called API then processing the received structured data source query for causing a performance of the corresponding query in the data source based on the configuration of the data source. In some instances, when the structured data source query is accompanied by information on a permission level of a user or system that provided the initial input query to access surveillance records based on security clearance, the data source, or API corresponding to the API, may perform an analysis to determine if the user or system has authorization toaccess the surveillance records contained within the data source based on the permission level. If a determination is made that the user or system does not have authorization to access the surveillance records, a response may be generated and transmitted back indicating the lack of clearance.

[0330] In some instances, a verification of the permission level may be performed by the system that generates and transmits the prompt for the LLM at steps 2200 and 2300, and transmitted the structured data source query to the LLM at step 2500. The system may verify the permission level of the user or system (e.g. by comparing the unique identifier of the user or system to a table or records of permission levels based on unique identifiers). If a determination is made that the user or system does not have authorization to access the surveillance records, a response may be generated and transmitted back to the computing device of the user or to the system that generated in the initial input query, indicating the lack of clearance.

[0331] The results are received from the one or more queried data sources at step 2600. The results may be transmitted to the computing device of the user or to the system that generated the initial input query. The received results may be stored in a data source in association with the related input query or structured data source query (e.g. as a data set) to archive ahistory of queries (e.g. related to a user account, a system, etc.)

[0332] The results may be caused to be displayed on a graphical user interface generated on a display of a computing device of a user who generated the input query.

[0333] In some instances, the received results of the query may be transmitted (with a prompt) to the LLM for analysis. For instance, the generated prompt that is transmitted with the queried results may cause the LLM to analyze the queried results to generate a summary of the search results. The summary generated by the LLM may then be received from the LLM.

[0334] In some instances, the prompt that accompanies the queried results and that is transmitted to the LLM may be to cause the LLM to analyze the search results. In some embodiments, the prompt may be accompanied by a history of search results (corresponding to previous searches), along with their related input queries and / or structured data source queries, and / or a chat history with the LLM, with instructions to the LLM as part of the prompt to cause the LLM to interpret the results. The instructions defined within the prompt to interpret the queried results may be to cause an identification of elements of interest within the queried results. The nature of the elements of interest may be defined by the user or system (as part of, derived from or accompanying the input query that is the basis of the generated structured data source query that causes the retrieval of the queried results). The nature of the elements ofinterest may also be determined by the LLM, in accordance with the instructions defined by the prompt, such as an identification of a suspicious character, or an identification of criminal activity.

[0335] In some instances, the instructions defined within the prompt to interpret the queried results may be to cause the LLM to suggest additional search options or a subsequent action in accordance with the queried results. The suggestion generated by the LLM may be in the form of a string of characters. The suggestion generated by the LLM may be transmitted to the computing device of the user or to the system that generated the input query. In some instances, the suggestion generated by the LLM may be processed by logic in communication with the LLM, where the logic may cause the generation of a new prompt to cause the generation of a new structured data source query by the LLM in accordance with the suggestion generated by the LLM. In some instances, instead of a suggestion, the output by the LLM may be new structured data source query to perform an additional search.

[0336] For instance, the prompt may cause the LLM to determine, by analyzing the received queried results, that the user or system generating the input query, did not have sufficient clearance to access surveillance records in one or more data sources, thereby resulting in the received query results not answering the initial input query. The response generated by the LLM may include an output that identifies a type of surveillance record that could not be accessed due to insufficient clearance, or to identify one or more data sources that can be accessed by the user or the system, based on the input query, that contains surveillance records to respond to the input query, that the user or system has permission to access based on the security permission level of the user or system.

[0337] In some instances, the prompt to cause the LLM to interpret the search results may cause the LLM to generate as an output an indicator that a feature of the initial input query cannot be searched. For instance, if the initial input query is “find a bald man with an anchor tattoo that was in Jimmy’s Shopping Mall on October 28”, the LLM, analyzing the queried results and the input query, may determine that the feature of a tattoo type cannot be searched. The LLM may generate as an output a message indicating that a tattoo type cannot be searched. The LLM output may include an alternative search feature to be queried instead of the tattoo type.

[0338] EXEMPLARY METHOD OF UPDATING A SURVEILLANCE RECORDS SEARCH TOOL:

[0339] Reference is now made to an exemplary method of updating a surveillance records search tool for searching data source of surveillance records, the data source having searchableparameters. The method may be performed by system 100. It will be understood that the method may be performed by any other system in accordance with the present teachings.

[0340] A data representation of a system prompt is provided, to be transmitted along with user query to an Al model, the system prompt defining instructions on a manner of structuring a structured data prompt for the data source in accordance with the searchable parameters of the data source. Searchable parameters are categories of information that can be searched in the data source. For example, searchable parameters may be the different types of metadata in video that can be searched or the vehicle characteristics that can be searched (when the surveillance record is a video). The data representation of the system prompt may contain template elements each including portions of a potential system prompt, and wherein to generate an LLM query includes building a system prompt with a subset of the template elements. The building the subset of template elements may be selected on the basis of the user query. A data representation is data from which the system prompt can be produced in a form that is acceptable by the LLM (e.g. which can simply be the system prompt itself as a string, or the form could contain several parts (templates) from which to build a system prompt based on the input query).

[0341] A modification, replacement or supplementation of the data representation of the system prompt is performed to define instructions on a manner of structuring a structured data prompt for the data source in accordance with different searchable parameters of the data source (e.g. when the searchable parameters change, e.g. we support new metadata from new video analytics tools). The data representation may be a text containing a system prompt.

[0342] An input query is obtained (e.g. from a user, from a system), the input query including natural language (“identify a stolen red Toyota on Stanley Street.”)

[0343] The data representation of the system prompt may be accepted.

[0344] The LLM query may be generated for a large language model (LLM) including a portion made with the input query and a portion made with the data representation of the system prompt. The LLM query may include the input query and the system prompt.

[0345] It will be understood that the memory may be distributed, where the application program and the system prompt may not be located in the same physical memory.

[0346] The following example is provided to enable the skilled person to better understand the present disclosure. As it is but illustrative and representative examples, it should not limit the scope of the present disclosure. It is only added for illustrative and representative purposes. It will be understood that other examples may be used to further illustrate and represent the present disclosure without departing from the present teachings.

[0347] EXAMPLE OF QUERYING A DATA SOURCE USING A STRUCTURED DATA SOURCE QUERY GENERATED BY AN LLM:

[0348] With reference to FIG. 1A, a search window 1000 is illustrated. The search window 1000 is an interactive graphical user interface which can receive input from and display output to an operator and may be one application amongst many in a surveillance software platform. To perform a search for surveillance records, the operator interacts with category selectors 1100, via which the operator can specify characteristics for the search query. In some cases, interacting with a category selector 1100 causes a dropdown menu 1120 to be displayed, via which the operator selects from a number of preset characteristics. In some other cases, interacting with a category selector 1100 causes a different type of input graphical element to be displayed, such as a slider, a text box, or the like. When the operator completes entry of their desired characteristics for the search, the operator interacts with a search element 1140, which launches the search, using the characteristics provided by the operator. With reference to FIG. IB, once the search is complete, search results 1200 are displayed in the search window 1000.

[0349] In the embodiment illustrated in FIGs. 1A-B, the search window 1000 is used to obtain a search query from the operator that is structured based on metadata categories associated with the surveillance records stored in the repository. For example, each of the category selectors 1100 is indicative of one type of metadata for each of the surveillance records: temporal metadata, location metadata, object-type metadata, visual presentation metadata, and the like. Thus, by inputting their desired characteristics via the dropdown menus 1120 and / or via other graphical input elements, the operator is constructing a structured data source query which can be used to search the repository of surveillance records. The structured data source query can take any suitable form or format: in some cases, the structured data source query is a textual string; in some other cases, the structured data source query is an array or other data structure. In some cases, the structured data source query is composed of query couplets, in which one element of a couplet is a metadata category, and the other element of the couplet is a metadata characteristic. Other approaches are also considered.

[0350] With reference to FIG. 2, a search window 200 is illustrated. The search window 200 is also an interactive graphical user interface which can receive input from and display output to an operator and may be another application amongst many in the surveillance software platform. Much like the search window 1000, the search window 200 is used by the operator to perform a search for records within the surveillance repository which match desired characteristics indicated by the operator. The search window 200 includes a query box 210 in which the operator enters an input query. In the embodiment illustrated in FIG. 2, the querybox 210 receives input in the form of text, which may be of any suitable type, including natural language text. The search window 200 also includes a voice input button 212, which can enable the operator to input a query using voice commands. In some embodiments, the search window 200 (or another element of the surveillance software platform) is configured to interpret voice commands and transcribe them to text. In such embodiments, the transcribed voice command may be displayed as text within the query box 210. In some other embodiments, the search window 200 (or another element of the surveillance software platform) is configured to process voice commands as-is, without transcription, as will be described in greater detail hereinbelow. Whether as text, transcribed text, a voice command, or some other form of input, the operator provides an input query to the search window 200, which is used to search the repository of surveillance records.

[0351] To perform a search within the repository of surveillance records, the input query, which is not in and of itself a structured data source query, is translated into a more suitable query using a large language model (LLM). The LLM is trained to translate natural language into data source queries which are structured based on metadata categories associated with the surveillance records stored in the repository. The LLM may be trained to translate natural language of various forms, including natural language as provided by the operator via the query box 210, via voice command (using the voice input button 212), and / or in any other suitable format. In some cases, the LLM is also trained to translate natural language of various different languages, such that input to the query box 210 and / or via the voice input button 212 need not be in English (or whichever language is considered the default language for the surveillance software platform). Thus, in the example of FIG. 2, the input query “search for white trucks last week in daytime” is translated into a structured data source query.

[0352] By way of a non-limiting example, the structured data source query may take the form of query couplets composed of a data category such as a metadata category and a metadata characteristic:{location:N / A}{date:after[2020.01.01] AND before[2020.01.08]}{time:after

[0600] AND beforefl 800]}{object_type: vehicle}{sizedarge}{colorwhite}

[0353] In this example, the input query does not specify a location; as a result, the “location” metadata category is assigned a metadata characteristic of “not applicable” (or“N / A”). The input query indicates “last week” and “in the daytime”, which means the metadata category “date” is assigned a metadata characteristic indicative of the previous week, and the metadata category “time” is assigned a metadata characteristic indicative of a time range of daytime. The indication of “trucks” in the input query is translated into two separate metadata categories: a first, “object_type”, which is assigned the metadata characteristic of “vehicle” (as opposed to “persons”, “luggage”, or the like), and the second, “size”, which is assigned the metadata characteristic of “large” (as opposed to “small”, which might return results of cars or other smaller vehicles). Finally, the input query indicates “white trucks”, which results in the metadata category “color” being assigned the metadata characteristic of “white”. Of course, the structured data source query may be structured in any other suitable fashion, as appropriate.

[0354] The LLM may be any suitable type of machine learning (ML) model trained to process text (whether written or in some other form), including transformer ML models with self-attention, rather than using convolutional neural networks. For example, the LLM may be trained to interpret the input query from the operator as a series of tokens, and the LLM may then predict a structured data source query as future tokens for the input query. In some embodiments, the LLM may be configured to interpret an audio recording or clip as the input query; in some other embodiments, audio recordings or clips provided by the operator as an input query may be transcribed to text prior to being provided to the LLM.

[0355] Training may be performed in any suitable manner, for example first to interpret natural language inputs, and then to produce structured data source queries as an output. In some cases, the LLM may be provided with examples of structured data source queries, which the LLM leams to then produce. Examples of structured data source queries may be obtained from operator interaction with the search window 1000, which generates structured data source queries via the category selectors 1100, or may be generated by simulating operator interaction with the search window 1000. Alternatively, a repository of previously used structured data source queries, for instance produced by past operator interaction with the search window 1000, may be used to train the LLM. Other approaches are also considered.

[0356] Advantageously, employing the LLM to produce the structured data source query permits for the input query to be translated into the structured data source query in a single operation. It is not required for the input query to be converted into an intermediary product, nor do any terms in the input query need to be converted to account for irregular vocabulary. Thus, there is no need for a dictionary of preferred terminology or for a parser to be applied to the input query. This also means that the LLM can adapt to changes in the vocabulary used by operators without the need for such a dictionary of preferred terminology or parser to beupdated accordingly, nor is there a need to rely on external sources for such a dictionary.

[0357] Additionally, the structured data source query is not generated by converting the input query; in other words, the LLM does not rearrange or reprise terms from the input query when generating the structured data source query. Instead, the LLM generates a wholly new query, in the form of the structured data source query, based on the input query. This ensures that typographical mistakes, irregular formulations or terminology, or other potential issues with the input query are not replicated in the structured data source query produced by the LLM. Additionally, this approach ensures that the structured data source query, as produced by the LLM, is more consistently inline with the metadata categories established for the repository of surveillance records.

[0358] With additional reference to FIG. 3, the LLM translates the input query into a structured data source query which is then used to search the repository of surveillance records to identify query results 220, which can be displayed within the search window 200 in any suitable fashion. In the embodiment illustrated in FIG. 3, a plurality of query results 220 are displayed in a grid, with each query result 220 corresponding to a particular surveillance record stored in the repository. The query results 220 can present various information from their respective surveillance records, including an image of the vehicle, related metadata, and the like.

[0359] With additional reference to FIG. 4, in some embodiments the query results 220 are interactive, and when a user selects a particular one of the query results 220, a sidebar 230 may pop-out to provide additional information from the related surveillance record. In the illustrated embodiment, the sidebar 230 provides a video frame 232 which displays a portion of a surveillance video recording from which the surveillance record was generated. The sidebar 230 also provides an image frame 234 which displays the image of the vehicle to which the surveillance record relates. Other information, including relevant metadata, the vehicle’s license plate, and the like, may also be displayed in the sidebar 230, as appropriate.

[0360] With reference to FIG. 5, there is illustrated a schematic diagram of an example computing device 500. As depicted, the computing device 500 includes at least one processor 510, a memory 520, and program instructions 530 stored within the memory 520, as well as input and output interfaces (I / O interfaces) 502 and 504, respectively. For simplicity, only one computing device 500 is shown; the various computing devices described herein may be embodied by one or more implementations of the computing device 500, which may be the same or different types of devices. The components of the computing device 500 may be connected in various ways including directly coupled, indirectly coupled via a network, anddistributed over a wide geographic area and connected via a network, for instance via a cloud computing implementation.

[0361] The I / O interfaces 502, 504 may include one or more media interfaces, via which removable media or other data sources may be coupled, one or more network interfaces, or any other suitable type of interface. The I / O interfaces 502, 504 of the computing device 500 may additionally, in some embodiments, provide interconnection functionality to one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker, for instance devices via which a user may interact with a server. In embodiments in which the I / O interfaces 502, 504 include one or more network interfaces, the network interface(s) of the computing device 500 may enable the computing device 500 to communicate with other components, to exchange data with other components, to access and connect to network resources, to serve applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these.

[0362] The processor 510 may be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof. The processor 510 may be configured for executing the instructions 530 stored within the memory 520. The memory 520 may include a suitable combination of any type of computer memory that is located either internally or externally such as, for example, random-access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like.

[0363] In certain embodiments, the computing device 500 is operable to register and authenticate users (using a login, unique identifier, and password for example) prior to providing access to applications, a local network, network resources, other networks, and network security devices. The computing device 500 may serve one user or multiple users.

[0364] For example, and without limitation, the computing device 500 may be a server, network appliance, set-top box, embedded device, computer expansion module, personalcomputer, laptop, personal data assistant, cellular telephone, smartphone device, UMPC tablets, video display terminal, gaming console, electronic reading device, and wireless hypermedia device or any other computing device capable of being configured to carry out the methods and / or implementing the systems described herein.

[0365] With reference to FIG. 6, a block diagram of an exemplary surveillance software platform is illustrated at 600 and is composed of a user interface 610, a search application 620, a LLM 630, and optionally of a surveillance record repository 605. The surveillance software platform 600 can receive input from input device 602, which may include various hardware (e.g., keyboards, touchscreens, microphones, etc.) and / or software (e.g., copy-paste input, input from remote users, etc.) input elements. The input is received at the user interface 610, which may display the input to an operator for confirmation, and may also be used to display other information to the operator.

[0366] The input received from the operator at the user interface 610 includes a search query, for instance the input query described hereinabove. The user interface 610 may pass the input query to the search application 620, which may be one of many applications forming part of the surveillance software platform 600. The search application 620 may then provide the input query to the LLM 630 for the LLM to translate the input query into a structured data source query, as described hereinabove. In some embodiments, the LLM 630 may provide the structured data source query to the surveillance record repository 605 directly; in some other embodiments, the LLM 630 returns the structured data source query to the search application 620, which in turn queries the surveillance record repository 605 using the structured data source query.

[0367] The search application 620 then receives query results from the surveillance record repository 605 and may then present the query results via the user interface 610. Additionally, the user interface may receive further input from the operator to interact with the query results, for instance to obtain additional information of context relating thereto, or to perform a new query by providing a new input query.

[0368] With reference to FIG. 7, there is illustrated a flowchart for a method 700 for searching a repository of surveillance records. An input query comprising natural language is obtained via a user interface. The input query is provided to a large language model (LLM), the LLM having been trained to translate natural language into data source queries structured based on metadata categories associated with the surveillance records stored in the repository. In some embodiments, instructions are generated and provided with the input query (e.g. as a system prompt) to generate an application programming interface (API) call forcommunicating with an application programming interface related to a data source to be queried by the LLM. These instructions for generating an LLM call may also include instructions to generate the content for producing an API call in a specific format (such as JSON schema).

[0369] The LLM may generate the structured data source query, corresponding to the natural language of the input query, along with the API call corresponding to the API call instructions, for querying the data source related to the API that is being called by the API call generated by the LLM. In some embodiments, the called API may modify the data source query received from the LLM in a format suitable for the data source to be queried, such as by converting the metadata category of the structured data source query to a corresponding and suitable metadata category of the data source to be queried.

[0370] In some instances, instructions may be generated and provided with the input query, as a system prompt, to cause the LLM to generate a query regarding how to generate a data source query for a given data source or query, based upon the configuration of the data source or system to be queried. Once the LLM receives the response regarding how to query the data source or system, the LLM may generate the structured data source query corresponding to the input query, structured accordingly to the response regarding how to query the data source or system.

[0371] In some examples, instructions may be generated and provided along with the input query to define or include permission information in the structured data source query generated by the LLM. The permission level defines an availability of data or a data source to be queried by the LLM based on a security clearance of the user or computing device of the user generating the input query. For instance, an input query may include, as e.g. metadata, a unique identifier for the user or the computing device of the user that generated the input query. The unique identifier may be included as information to be shared in the structured data source query to be transmitted to the data source to be queried, or the unique identifier may be analyzed by external logic to generate instructions for the LLM to include information on the clearance or accessibility of data to be queried by the user, e.g. based on the security clearance tied to the unique identifier, in the structured data source query. Results received in response to the structured data source query may be limited depending on the permission level tied to the user or computing device of the user that is the originator of the input query.

[0372] In some instances, instructions may be generated and provided to the LLM, along with the input query, to verify if the queried results are limited due to a security issue, where certain data in the data source is not accessible due to security reasons. The instructions maycause the LLM receiving the queried results to generate a response for the user or user computing device that generated the input query indicative that a security or permission restriction impeded the receiving of results in response to the structured data source query.

[0373] A structured data source query, based on the input query, is obtained from the LLM. The repository is searched using the structured data source query to identify at least one query result. The at least one query result is displayed via the user interface. In some instances, instructions may be generated and transmitted to the LLM (e.g. as a system prompt), along with the natural language of the input query, to cause the LLM to, once the LLM receives the one or more query results (e.g. from the data source or the user’s computing device that originated the input query), generate a summary of the search results (e.g. in a text format, as an audio message, etc.) for transmission to the computing device of the user from which the input query originated. The instructions to summarize the query results transmitted to the LLM may include instructions regarding the format of the summary response that is to be generated by the LLM (e.g. audio, images, a text, etc.), a length of the summary response, etc.

[0374] In some instances, instructions may be generated and transmitted to the LLM, along with the natural language of the input query, to cause the LLM to analyze the query results received from the queried data source or the user computing device that originated the input query, in view of the input query or the structured data source query generated by the LLM. In some embodiments, the LLM may generate an adapted structured data source query following the analysis of the query results, to further query the data source to supplement or adapt the query results, to better respond to the initial input query originating from the computing device of the user.

[0375] The embodiments of the methods, systems, devices, and computer-readable media described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface.

[0376] Program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements may be combined, the communication interface may be a software communication interface, such as those for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implementedas hardware, software, and combination thereof.

[0377] Throughout the foregoing discussion, numerous references have been made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a server can include one or more computers operating as a web server, data source server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

[0378] The foregoing discussion provides many example embodiments. Although each embodiment represents a single combination of inventive elements, other examples may include all possible combinations of the disclosed elements. Thus, if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, other remaining combinations of A, B, C, or D, may also be used.

[0379] The terms “connected” or "coupled to", as well as any similar terms, may include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements).

[0380] The use of numerical ranges by endpoints in the present disclosure should be understood as including all numbers within that range (e.g., 1 to 5 includes 1, 1.25, 2, 2.5, 3, 3.69, 4, 4.33, 5, etc.). Where a range of values is qualified as being “greater than”, “less than”, etc., of a particular value, that value may or may not be included within the range, as appropriate.

[0381] Any direction or orientation described in the present disclosure, including but not limited to “top”, “bottom”, “left”, “right”, “upper”, “lower”, “above”, below”, as well as other directions and orientations, are described herein for clarity, and should be understood in reference to the drawings. These and other similar terms should not be understood as limiting of an actual device or system or of use of the device or system. Many of the devices, articles, or systems described in the present disclosure may be used in a number of suitable directions and orientations.

[0382] Any citation to references in this disclosure and during the prosecution thereof is made out of an abundance of caution. No citation should be construed as an admission that the cited reference qualifies as prior art or comes from an area that is analogous or directly applicable to the present teachings.

[0383] To aid the Patent Office, as well as any readers of any patent issued from this application, in interpreting the claims appended hereto, it is noted that none of the appended claims or elements of the appended claims, as pending or as granted, are intended to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim or claim or claim element.

[0384] The technical solution of embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory computer-readable storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.

[0385] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements. The embodiments described herein are directed to electronic machines and methods implemented by electronic machines adapted for processing and transforming electromagnetic signals which represent various types of information. The embodiments described herein pervasively and integrally relate to machines, and their uses; and at least some of the embodiments described herein have no meaning or practical applicability outside their use with computer hardware, machines, and various hardware components. Substituting the physical hardware particularly configured to implement various acts for non-physical hardware, using mental steps for example, may substantially affect the way the embodiments work. Such computer hardware limitations are clearly essential elements of the embodiments described herein, and they cannot be omitted or substituted for mental means without having a material effect on the operation and structure of the embodiments described herein. The computer hardware is essential to implement the various embodiments described herein and is not merely used to perform steps expeditiously and in an efficient manner.

[0386] EXAMPLE OF AUGMENTING A QUERY TO IMPROVE AN ANSWER PROVIDED BY AN LLM:

[0387] The present disclosure describes a system and method of use thereof for improving an output generated by an LLM that is associated to an application program. For instance, the LLM may be an engine for a chatbot feature of an application program, a search or help functionality that has an LLM implementation, etc. Exemplary application programs includesecurity surveillance systems, word processors, or any application program including, or connected to, a LLM tool.

[0388] It has been found that when an LLM-powered chatbot is provided, the outputted answers from these LLMs can lack accuracy and / or completeness, thereby undermining their utility. Therefore, in some instances, it may be advantageous to reformulate the user input from the user for generating the LLM query and can cause a reformulation of a string of characters based on the user input to generate a search query that is adapted for the LLM, such that the query is formulated in a manner to assist the LLM in outputting an answer that is responsive of the query. Input reformulation can include augmenting the query with additional information instructing the LLM on how to respond. For instance, the additional information may include information for causing a generation of content in the output to cause an application program interface (API) call for communicating with an application program (e.g. to execute a query in response to the LLM output, such as to query a database of license plate reads, to query a felon database, etc.).

[0389] In some instances, the output from the LLM may be a string of characters (such as a chat response). In some instances, the output from the LLM may be, or may be adapted to be, instructions to generate commands (when processed by logic) for causing an action in the application program (e.g. for controlling the application program).

[0390] An application program interface (API) may be used, where the API is configured to receive the output from the LLM, or to receive an API call generated from content found in the output from the LLM, and convert the output into commands in a language suitable for execution by an application program, carrying out the commands (such as querying a database), thereby causing the application program to carry out one or more actions corresponding to the commands.

[0391] The user search query may include instructions to cause the LLM to generate content for producing an API call, in order to answer the user search query as output. The user search instructions may also include instructions to generate the content for producing an API call in a specific format (such as JSON schema). The received content for producing an API call can then be processed by the application program to cause an API call (e.g. to query a database).

[0392] In some examples, programmatic logic may be used to analyze the search query to determine if an API call is needed. If an API call is needed, then the search query may be adapted to include instructions to generate content for causing an API call (for communicating with or instructing software).

[0393] In some instances, the API call can cause the application program to perform an action. In some instances, the API call can prompt a response from the application program. The API can then receive the response from the application program and arrange the response in a defined manner. In some instances, a response from the API may be transmitted back to the LLM, where the LLM can generate a further output response with the information received from the API (e.g. in the form of a text chat, an audio message as a voicebot, etc.) For instance, in response to the user input, as an initial query, “who is the owner of that car?” (where a licence plate is displayed in an image viewed in a window of the application program opened by the user), the response can be “Marie Tremblay”, the outputted response provided in a chatbot or voicebot.

[0394] In some instances, the output generated by the LLM may cause an answer that is a combination of information presented in a chatbot and / or voicebot, with actions performed by the application program, such as the display of a video or other information, moving a digital map, etc. For instance, a piece of identity for Marie Tremblay may be displayed on the screen, along with related information on Marie Tremblay.

[0395] In another example, in response to the user input “who came through this door today”, the answer generated from the LLM output may be a string of characters appearing in a chat, formulating “100 people came through the door today, most recently Mike Hunt and Harry Lasagna. Please see your screen for the full list of names.” The output may then cause the application program to display the list of names on a window of the application program.

[0396] The system and method described herein transmit or analyze context information on activity related to the application program by the user before and / or at a time of receipt of user input for generating a search query for inputting into the LLM. The user input may correspond to a series of words that serve as a basis for the query (provided as a string of characters, one or more images, an audio stream, a video stream, etc.) Context query data may be generated from the analyzed context information, the context query data providing information regarding characteristics of the user’s use of the application program, the context query data included in the generated user search query. The formulated user search query is generated from the user input and from the context query data in a formulation adapted to improve the answer outputted by the LLM in response to the search query. The weighing or inclusion of the context information or context query data when generating the formulation of the user search query permits an addition into the user search query of additional meaning to improve the LLM’s ability to generate an answer that is tailored to the circumstances surrounding the user input which the user may not have included in their user input forgenerating the user search query. This inclusion assists the LLM to compensate for a lack of information provided by the user for generating the user search query for the LLM.

[0397] Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense, that is as “including, but not limited to.”

[0398] Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0399] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise.

[0400] From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the teachings. Accordingly, the claims are not limited by the disclosed embodiments.

[0401] EXEMPLARY SYSTEM FOR IMPROVING AN OUTPUT GENERATED BY AN LLM:

[0402] Reference is made to Figure 11, illustrating an exemplary server system 100 for improving outputs (e.g. answers) provided by an LLM associated to an application program. The system 100 is in communication with one or more external servers 120 and one or more external computing devices 130 for a user, and optionally one or more data sources 110.

[0403] The system 100 includes a processor 102, memory 101 and an I / O interface 103.

[0404] The processor 102 may be a general-purpose programmable processor. In the example of Fig. 1, the processor 102 is shown as being unitary, but the processor 102 may also be multicore, or distributed (e.g. a multi-processor).

[0405] The computer readable memory 101 stores program instructions and data used by the processor 102. The memory 101 stores, for instances, context query data, user input for generating a user search query, LLM outputs, etc. The computer readable memory 101, thoughshown as unitary for simplicity in the example of Fig. 1, may comprise multiple memory modules and / or caching. In particular, the memory 101 may comprise several layers of memory such as a hard drive, external drive (e.g. SD card storage) or the like and a faster and smaller Random Access Memory (RAM) module. The RAM module may store data and / or program code currently being, recently being or soon to be processed by the processor 102 as well as cache data and / or program code from a hard drive. A hard drive may store program code and be accessed to retrieve such code for execution by the processor 102 and may be accessed by the processor 102 to store and access data. The memory 101 may have a recycling architecture for storing, for instance, LLM outputs, context query data, etc., where older data files are deleted when the memory 101 is full or near being full, or after the older data files have been stored in memory 101 for a certain time.

[0406] The I / O interface(s) 103 is in communication with the processor 102. The I / O interface(s) 103 may include a network interface and may be a wired or wireless interface for establishing a remote connection with, for example, one or more data source 110 (e.g. through application program interface(s) - API(s)) via a network 140 (e.g. the Internet), and / or one or more servers 120 and / or one or more external computing devices 130. For instance, the I / O interface(s) 103 may be an Ethernet port, a WAN port, a TCP port, etc.

[0407] The processor 102, the memory 101 and the I / O interface(s) 103 may be linked via bus connections.

[0408] The client computing device 130 may be a remote computing device (i.e. client). One or more client computing devices 130 may be provided.

[0409] The client computing device 130 is in communication with the I / O interface(s) 103 of the server system 100. The computing device 130 has a processor 132, a memory 131, I / O interface(s) 133 that may be linked via bus connections. The computing device 130 may have (or be connect to) any user input interface 135, for example, such as a keyboard, a mouse, a touchscreen, etc. The computing device 130 may be a desktop computer, a laptop, a smartphone, a tablet, etc. The computing device 130 has (or is adapted to connect to) a display 135 (e.g. a screen, a tactile display, etc.). The processor 132, the memory 131 and the I / O interface(s) 133 may be similar to the processor 102, the memory 101 and the I / O interface(s) 103, respectively.

[0410] A client application program may be stored in memory of the computing device 130 that is associated with the server system 100, the client application program providing the user with an interface to interact with the server system 100. The client application program may be for interacting with the application program that is associated with the LLM tool.

[0411] The client computing device 130 may be operated by user(s) to access a webpage or graphical user interface (GUI) for interacting with the application program associated with the LLM tool.

[0412] The system 100 may be in communication with one or more data sources 110 via the network 140. The system 100 may be in communication with one or more external computing device 130 via the network 140. The system 100 may be in communication with one or more external servers 120 via the network 140.

[0413] Similarly, the external computing device 130 may be in communication with the one or more external servers 120 via the network 140. The external computing device 130 may also be in communication with the server system 100 via the network 140.

[0414] The one or more external servers 120 are configured to run the LLM. The one or more external servers 120 may also cause the running of the application program that is associated with the LLM tool. The system 100 may be configured to interact with the external computing device 130 to receive from the external computing device 130 the user input for generating the user search query. The system 100 may be configured to interact with one or more of the servers 120 and / or the external computing devices 130 to receive the context information. The system 100 may be then configured to transmit to the one or more servers 120 the refined user search query for querying the LLM tool.

[0415] In some instances, the functionality of the one or more servers 120 may be integrated into the system 100, where the system 100 is configured to run the LLM tool and the application program that is associated with the LLM tool.

[0416] In some instances, the system 100 may be the same device as the external computing device 130, where the program code for improving the output of the LLM is stored in memory of the external computing device 130.

[0417] The data source(s) 110 may include information that can be retrieved and / or analyzed by the LLM tool for generating an output to the user search query. The data source(s) 110 may include information, such as context information relating to use of the application program with the LLM tool, that may be used by the system for generating a refined user search query from the received user input for the LLM query. The data source(s) 110 may include data related to the operation of the application program associated with the LLM tool. For instance, when the application program is a security surveillance system, the data source(s) 110 may include licence plate read (LPR) events, biometric data on perpetrators, facial recognition data for identified perpetrators, camera locations, etc.

[0418] EXEMPLARY SOFTWARE ARCHITECTURE FOR IMPROVING ANOUTPUT GENERATED BY AN LLM:

[0419] Reference is made to Figure 13, illustrating an exemplary software architecture 300’ for improving an output generated by an LLM tool that is associated to an application program.

[0420] For purposes of illustration, the system 100 is described herein as having program code, stored in memory 101, that includes the context query data module 310’, the user input analysis module 320’ and the user search query module 330’. Each of the context query data module 310’, the user input analysis module 320’ and the user search query module 330’ includes program code configured to implement the functionality of the modules as are described herein.

[0421] The context query data module 310’ includes program code stored in memory 101 that, when executed by the processor 102, causes the processor 102 to receive context information surrounding use of the application program by the user. The context information may include historical use of the application program by the user, and / or current use of the application program by the user. The context query data module 310’ then causes the processor 102 to analyze the context information to generate the context query data for purposes of refining the user search query for use as input for the LLM tool. The context query data module 310’ may cause the processor 102 to took for trends or patterns of use of the application program by the user in the context information to generate the context query data. The context query data module 310’ may cause the processor 102 to receive the search input provided by the user for producing the search query, or a crude user search query generated from the user input by the user input analysis module 320’. The generating of the context query data may be further adapted as a function of information derived from the user input or crude search query. The context query data may be provided as text (a string of characters) to be included in the user search query.

[0422] The user input analysis module 320’ includes program code stored in memory 101 that, when executed by the processor 102, causes the processor 102 to receive the user input provided by the user for generating the user search query for inputting into the LLM. In some instances, the user input analysis module 320’ may cause the processor 102 to analyse the user input to generate a crude user search query by adapting the user input as a function of parameters for improving the processing of the query by the LLM, for improving the quality of the outputted answer. The crude search query may be in the form of a string of characters for querying the LLM.

[0423] The user search query module 330’ includes program code stored in memory 101that, when executed by the processor 102, causes the processor 102 to generate the refined user search query. The refined user query is generated from the user input (and / or the crude search query received from the user input analysis module 320’), and adapted with the context query data. The generated refined user search query may be a string of characters that can be parsed by the LLM tool. The user search query may include instructions to include content for causing an API call in the output generated by the LLM. The user search query module 330’ may then cause the processor 102 to transmit the generated user search query to the LLM tool (e.g. via network 140).

[0424] EXEMPLARY METHOD OF IMPROVING AN OUPUT GENERATED BY AN LLM:

[0425] Reference is now made to Figure 14, illustrating an exemplary method 400 of improving an output generated by an LLM that is associated with an application program. The method 400 may be performed by the system 100. However, it will be understood that the method 400 may be performed by any system in accordance with the present teachings.

[0426] User input is received at step 410 for generating a user search query for use as input for the LLM tool associated with the application program. For instance, user input may include a string of characters, an audio recording, a video stream, one or more images, input generated on a touchscreen, etc.

[0427] In some examples, a crude search query for the LLM may be generated at step 420 from the user input. The user input may be analyzed to determine one or more meanings behind the user input, and generate a crude search query that has been adapted based on the functionality of the LLM to improve the output generated by the output in response to the user input. In some instances, a query may be transmitted to the user to provide additional information in order to improve the generated crude search query. The generated search query may be a string of characters formulating a series of words with a given meaning derived from the user input.

[0428] Context information surrounding the use of the application program by the user is received at step 430. A context determining program code module is provided that, when executed by the processor, accesses and gathers the context information described herein or other information indicative of the context information to generate the context query data (e.g. the program code may be that of a scraping tool for the application program).

[0429] The received context information may include information regarding the historical use of the application program by the user. For instance, the context information may include previous queries entered by the user while using the application program, prior functions oroperations carried out by the application program in response to requests inputted by the user, window panes and tools opened or used by the user while running the application program, a geographical location of the user when running the application program, times of use of the user while running the application program, prior search results generated by the application program in response to queries generated in response to user input, etc. The context information may relate to a current session of use of the application program. The context information may relate to one or more prior sessions of use of the application program by the user. The context information may include information on the most recent action performed by the user using the application program, or information on a number of recent actions performed by the user using the application program.

[0430] The received context information may also include information regarding the current use of the application program by the user, such as the identity and / or function of window panes (modules) of the application program that are currently opened by the user, information currently being displayed on a graphical user interface generated by the application program (e.g. a region of a digital map displayed on the graphical user interface, a timestamp of a video that is playing, the content of a video that is being played, etc.), etc.

[0431] In some instances, the context information may include one or more of information on what a user is in the process of accessing using the application program (e.g. door scans for a particular door), what the user has accessed in the past via the application program (e.g. a license plate read hotlist, etc.), the current actions performed by the application program, the current actions performed by the user using the application program (e.g. what pane or module is opened, what plugin is open, what window is opened), actions performed in the past by the application program, actions performed by the user in the past using the application program (e.g. what features or functions were opened or used recently, what module of the software has been used to date, recently, ever, most commonly, etc.), etc.

[0432] In some instances, the context information includes information on the plugins and modules of the application programs that have been purchased (e.g. by the user) during a given period of time or that are currently purchased or activated or installed.

[0433] In some instances, the context information may include information on plugins and / or programs that are running, or have run, in association with the application program that is associated with the LLM. The running plugins and / or program codes may provide information on what is relevant to the user, based upon the nature of the plugins (e.g. a plugin for detecting access to a door in a building, for monitoring video feeds, etc.)

[0434] In some instances, the context information may include version information aboutthe application program, related modules, plugins and / or programs that are running, which may include information about the capabilities of these things or their API and / or API-related content and formatting requirements, available data, etc.

[0435] In some embodiments, context query data may be generated from the context information at step 440. The context query data includes trends, values, patterns, etc., derived from the context information, that quantify or qualify the use of the application program by the user. This context query data can then be used to provide context regarding the actions carried out by the user that led the user to provide the user input for generating the user search query (information indicative of the motivation(s) or reason(s) prompting the user to seek to query the LLM). The context query data may be in the form of one or more values, one or more strings of characters, etc., to be included as part of the user search query.

[0436] In some instances, the context information may be transmitted directly to a module interacting with the LLM, where the module processes the context information to determine the relevance of the context information with respect to the user input for formulating the user search query (no context query data may be generated in these examples).

[0437] A refined user search query for the LLM is generated at step 450. The refined user search query is generated from the user input (and / or the crude user search query), and adapted by processing the context information or context query data, to further refine the search query with the information surrounding the use of the application program by the user or the running of the application program that could contextualize the motivation of the user to query the LLM. In some embodiments, the refined search query may add characters (meaning) to the crude search query or replace certain words, characters and / or instructions in the crude search queries with other words.

[0438] The refined search query may be a string of characters defining a series of words (e.g. one or more complete sentences), defining a meaning that can be determined and analyzed by the LLM for purposes of the LLM outputting an answer that corresponds to the user search query.

[0439] In some examples, the user search query may include commands for causing the generation of content for producing application program interface (API) call instructions, where the API can cause the carrying out of actions (e.g. to retrieve information) of an application program, associated plugins or related programs, or even external applications that the LLM determines as being required to answer the user search query. In some instances, an API may be used to optionally translate the user search query such that the LLM can process the user search query that is has received.

[0440] The refined search query is transmited to the LLM at step 460 using, in some instances, an application program interface (API) for communicating with the LLM.

[0441] The LLM generates an output (an answer) corresponding to the user search query that the LLM has received and transmits the output to a computing device of the user. The answer may include content for generating the API call.

[0442] In some instances, the LLM itself may be associated with a software module that is adapted (instead of method 400 being performed by a separate application program) to perform method 400, or receive the context information.

[0443] The following examples are provided to enable the skilled person to beter understand the present disclosure. As they are but illustrative and representative examples, they should not limit the scope of the present disclosure. They are only added for illustrative and representative purposes. It will be understood that other examples may be used to further illustrate and represent the present disclosure without departing from the present teachings.

[0444] EXAMPLE 1: IMPROVING OUTPUT GENERATED BY AN LLM:

[0445] A user is running a surveillance system, monitoring activity of a given immovable property (e.g. a building). The user is using the application program to monitor activity of a given door of the property.

[0446] The user is running in the surveillance system a first module for viewing an image stream of a camera capturing the door. A second module is running relating to access control for the door (logging individuals who have recently accessed the door using a unique identifier, such as an access card or badge, a passcode, a biometric reading, etc.)

[0447] The user may submit as user input, for querying a chatbot of the surveillance system running an LLM, “who was the last person to come through?”, received as a string of characters by the application program.

[0448] The context information includes information on the modules of the surveillance system that are active in the current session, namely the module for receiving the video stream of the door of interest and the module for access control of the door of interest.

[0449] In some instances, the context information is processed to generate context query data. As the context information relates to the door of interest, the context query data includes information regarding the identity of the door, as the context information is indicative that the user is interested by that specific door, based upon the activity of the modules that are currently running on the application program.

[0450] In some instances, the context information is transmited to a module associated with the LLM along with the string of characters as the user input, the module associated withthe LLM analysing the context information to derive information that qualifies the nature and subject of the user’s initial query (for generating the finalized user search query) from the received context information.

[0451] In some instances, an application program or separate logic may be used to analyze the context information to generate context query data therefrom, thereby including data that is relevant for qualifying the intention behind the user search for the search query.

[0452] The information sent to the LLM may also send API call instructions.

[0453] The LLM receives the user input and the context information (or in some instances, the context query data), and processes the user search query with the context derived from the context information, that the search query pertains to the door of interest. As such, the LLM determines that the user input “who was the last person to come through?” relates to providing the identity (e.g. the badge read at the badge reader for the door of interest) of the last person to access the door of interest.

[0454] The LLM then generates content for initiating an API call for causing a running of a search to identify the badge information of the last badge that has been read by the badge reader associated with the door. The LLM receives the badge information, and generates an output with the badge information.

[0455] EXAMPLE 2: AN EXEMPLARY USER SEARCH QUERY FOR IMPROVING AN OUTPUT GENERATED BY AN LLM:

[0456] The following is an example of an exemplary user search query generated by the systems or methods of the present disclosure for transmission to the LLM, for purposes of improving the output generated by the LLM in response to the user search query.

[0457] In this example, an LLM like GPT may be used. This example uses the LLM, not a chatbot program that uses the LLM, so queries to the LLM contain all the required information, including context but also history and other instructions every time since the LLM in this example does not have memory of past interactions. In this example, three “characters” are included in the query: a system that provides information on how to respond, a user which corresponds to user queries, and an assistant, which represents the LLM’s own responses. Since a history of interactions with the LLM may be provided with the user search query, previous LLM outputs may be included as coming from the assistant.

[0458] A) first part of the user search query: instructions for generating the output, including instructions to generate content for an API call in the output:System:You are responsible for creating a JSON document used for an API to search for objects and individuals within CCTV recordings. Do not answer with anything other than JSON.The output must contain a suitable start and end detection time range in the field 'detectionTimeRange' based on user input and current time. If the user specifies a duration (e.g. 'in the last 4 hours' or '4pm to 5pm), use exactly that duration as a time range. If the user specifies a point in time (e.g. '15min ago' or 'yesterday'), an time range should be assumed using an appropriate margin around the specified time.For example: when the user asks for the last 10 minutes the time range be from 15 minutes to 5 minutes in the past. The current time is 2024-07 -11T13:30:02.0586560Z.If the user specifies a past point in time rather than a duration, make sure to add a suitable duration before and after. For example: when the user asks for results at 4am today, select a time range between 3am and 5am.The output must contain filters suitable to the user query.Valid people clothing and vehicle colors are 'White', 'Green', 'Red', 'Black', 'Blue', 'Yellow'.Valid vehicle types are 'Bike', 'Car', 'Truck', 'Bus'.Consider the vehicle type Bike' to include any common 2-wheeled vehicle like motorcycles.Not directly supported colors should be mapped to the closes valid colors. Not directly supported vehicle types should be mapped to the closest valid vehicle types.Consider using multiple alternative colors or vehicle types in the ''oneOf field.Try to map well-known brand names of vehicles to their respective vehicle types.Try to map names of people to the 'Person' class.If no object class can be derived from the user input, the filter should match 'Person' or 'Vehicle' with respective attributes (if applicable). If no attribute filter can be derived from the user input fora given class, an empty object should be used to search for that class with any attribute.If the clothing type (top / bottom) can't be derived from the user input, the response should contain two person filters. One for the respective top clothing color and another person filter with the respective bottom clothing color.

[0459] B) a second part of the user search query: specifying a format for the output generated by the LLM (in the present example, the format is a JSON schema):The output must conform to following JSON schema: "json{"type "object", "properties {"reasoning ":{"type "string "payload":{"type "obj ect", "properties ":{" person ":{"type "array", "items ":{"type "object", "properties ":{ "topClothlngColor":{"type "object", "properties":{"oneOf":{"type "array", "items" :{"type":"string", "enum":["White", "Black", "Green", "Blue", "Red", "Yellow"]}}}}, "bot tomClothingColor":{"type":"object","properties":{"oneOf":{"type":"array", "items": {"type":"string", "enum":["White", "Black", "Green", "Blue", "Red", "Yellow"]}}}}}}}, "v ehicle":{"type":"array","items":{"type":"object","properties":{"vehicleColor":{"typ e ": "object", "properties":{"oneOf":{"type ": "array", "items":{"type ": "string ", "enum ":[ "White", "Black", "Green", "Blue", "Red", "Yellow"]}}}}, "vehicleType":{"type":"objec t", "properties":{"oneOf":{"type":"array", "items":{"type":"string", "enum":["Bus", "B ike", "Car", "Truck"]}}}}}}}, "detectionTimeRange":{"type":"object", "properties":{"st art":{"type ": "string ", "format": "date-time "}, "end": {"type ": "string ", "format": "date- time"}}}}}}}[additional details for other types of requests such as LPR related request or access control related requests]

[0460] C) a third part of the user search query can be instructions to account for context information or context query data as part of the user search query:Context is provided as follows: {SW:;modules:;active pane:; selected devices: ;current action:; recent panes:;recent selected devices:; recent actions:;}Where:- SW is the version of the software being used [include details of what different version mean / can do and its impact on the JSON schema]- modules is the list of active optional modules and add-ons in the system [include details of what each module is and can do, and its impact on the JSON schema]- active pane is the current pane in the software which can be, e.g. the access control device properties pane or the live video viewing paen [include details of each pane and what can be done or seen in them]- selected devices indicates what device or item is currently selected or being viewed, such as a door, camera, badge reader etc [include details of each possible selected device and what can be done with them and its impact on the JSON schema]- current action is the tool currently selected or action currently undertaken [include list of all possible current actions and related information and its impact on the JSON schema]- recent action are the tools recently used or action recently undertaken [include list of all possible current actions and related information and its impact on the JSON schema]- recent pane are the recent pane in the software which can be, e.g. the access control device properties pane or the live video viewing paen [include details of each pane and what can be done or seen in them]- recent selected devices indicates what devices or items were recently selected or being viewed, such as a door, camera, badge reader etc [include details of each possible selected device and what can be done with them and its impact on the JSON schema]

[0461] D) a fourth part of the user search query may be history of past user search query(ies) and corresponding output(s) generated by the LLM:Prior search 1 :User: Find the Woman in the Red dressContext: {Software version: 1.2.3; modules: video, access control, LPR; active pane: live video; selected device: camera 01, camera 02; current action: viewing live video; recent pane: live video; recent selected devices: camera 01, camera 02, camera 06, camera 09; recent actions: object tracking] Assistant: {''reasoning'':''Not supported: classification (woman), clothing type (dress)\nSearching for a *person* with a *top clothing color* containing *red*.","payload":{"person":[{"topClothingColor":{"oneOf":["Red"]}}]}}Prior search 2:User: Intruder with blue shirt and grey trousers, 30 minutes agoContext: {Software version: 1.2.3; modules: video, access control, LPR; active pane: live video; selected device: camera 01, camera 02; current action: viewing live video; recent pane: live video; recent selected devices: door 02, camera 06, camera 09; recent actions: access control, door view, object tracking}Assistant: {''reasoning'':''Not supported: classification (intruder), clothing type (trousers), color (grey), door forced open (door 02)\n Selected Time Range: today [40min ago, 20min ago]\nSearching for a *person* with a *top clothing color* containing *blue* and *bottom clothing color* containing *white* or *black* or a door forced open event on door 02. ", "payload":{"person ":[{"topClothingColor":{"oneOf':["Blue "]}, "bottomClothin gColor":{"oneOf":["White", "Black"]}}], "event":["doorforcedopen":("door02")], "det ectionTimeRange":{"start":"2024-07-11 T12:50:02.058656Z", "end": "2024-07- 11 T13: 10:02.058656Z"}}}Prior search 3User: red ho nd aContext: {Software version: 1.2.3; modules: video, access control, LPR; active pane: hit list read; selected device: LPR camera 03; current action: LPR list; recent pane: access control, LPR hitlist; recent selected devices: LPR camera 03, badge reader 04; door 02; recent actions: LPR viewer; badge read viewer}Assistant: {"reasoning":"Not supported: vehicle brand(honda),hitlist(yes)\nSearching for a *vehicle* with a *vehicle type* containing *car* or *bike* and *vehicle color* containing *red* and listed in the hitlist. ", "payload":{"vehicle ":[{"vehicleColor":{"oneOf":["Red"]}, "vehicle Type ":{"o neOf":["Car", "Bike"]}, "hitlist":{["yes"]}}]}}

[0462] E) a fifth part of the user search query is the string of characters received as user input for the current user search query:USER: “find the guy in the red shirt”

[0463] For prior search 2, the context information indicating that the user had been previously investigating access to door 002 provides context to the LLM that the user is looking for an intruder on the premises. This information is reflected in the output generated by the LLM, where the content outputted by the LLM for an API call is to search for “door forced open” events relating to door 002.

[0464] For prior search 3, the context information is indicative that the user is looking through a hitlist. This context information is reflected in the output generated by the LLM, where the output indicates that the user is searching for “red Hondas” in the hitlist. The content outputted by the LLM for the API call is adapted accordingly, to cause an API call to perform a search for red Hondas in the hitlist.

[0465] In some instances, the output generated by the LLM does not take into account thereceived context information, such as in prior search 1.

[0466] EXEMPLARY SYSTEM FOR ADAPTING A SYSTEM PROMPT OF AN LLM QUERY:

[0467] The security system 100 may, in some implementations, be configured to generate a system prompt that is specific to a received input query, generated from a system prompt template element library, storing a plurality of system prompt template elements, and / or manage (to update) a plurality of system prompt template elements stored in a library of system prompt template elements 1504.

[0468] As shown in Figure 17A, in some embodiments, the system 100 may include, stored in its memory 101, the system prompt template element library 1504 and the program data 1702 related to the system prompt template element library 1504 for managing the system prompt template element library 1504 through the program code 1701 of the system 100 (for running the system 100). The program code 1701 of the system 100 includes program code for causing the running of the security system 100 (e.g. for receiving and analyzing image streams and data generated from sensors in communication with the system 100, running security operations such as managing security guards and access to buildings by personnel, etc.) The program code 1701 of the system 100 also includes program code for causing the generating of a query tool (e.g. a chat function, such as GPT chat function) for receiving the input queries and transmitting LLM queries to an LLM (e.g. running on external server 120). The program code 1701 may also include program code for causing a generating of the LLM query from the received input query and by including the system prompt that accompanies the LLM query, where the system prompt is generated from instructions retrieved from the system prompt template element library 1504. In some embodiments, the program code 1701 may also include program code for causing an updating of one or more system prompt template elements stored in the system prompt template element library 1504, as explained herein.

[0469] In some embodiments, as shown in Figure 17B, the system prompt template element library 1504 and the program data 1702 may be stored in memory 1502 that is separate (a different data source) from memory 101 which stores the program code 1701. System 100 may be in communication with memory 1502 to retrieve, for instance, one or more of the system prompt template elements, from the system prompt template element library 1504.

[0470] In both Figure 17A and Figure 17B, the system prompt template element library 1504 is separate from the program code 1701, where the system prompt template element library 1504 may be updated (modified) separately and independently from the program code 1701. For instance, a pointer of the application program related to a location in memory 101 ormemory 1502 may be changed for the system prompt template element library or one or more of the system prompt template elements of the system prompt template element library 1504 when the system prompt template elements are updated (resulting in a modified subset of system prompt template elements), where the new pointer may point to a location of one or more modified system prompt template elements or a modified system prompt template element library 1504.

[0471] System 100 may be in communication with one or more devices such as security cameras and / or sensors (such as microphone(s), licence plate reader(s), badge reader(s), facial scanner(s), etc.). System 100 may be a video monitoring system. In some embodiments, the system 100 may be an access control system, where the system may be an operational software which runs operations related to security, such as guard duties, investigations, etc.). In the present disclosure, when reference is made to “security records”, it is meant data (e.g. files) related to security, such as surveillance records (e.g. image streams, audio files, license plate reads, badge reads, etc.), operational records (such as guard assignments information, investigation files or information, dispatching files or information, etc.), etc. Increasingly, as demands for greater unification grow, security application programs may include or take the form of operations management programs for managing operations such as investigations, dispatching security personnel, tracking tasks particularly security-related tasks, and tracking equipment such as vehicles, security devices and the like.

[0472] Herein the disclosure refers to an LLM tool. It should be noted that the state of the art in the field of artificial intelligence is changing rapidly and with this change, the meaning of terms can change overtime. Likewise new forms of artificial intelligence (Al) models appear that share some commonality with old forms and these new forms may be given different names in spite of having some overlap in capability or purpose. For this reason, the reader should not view the choice of terminology, particularly for Al -model related terms as more restrictive than the purpose for which they serve. Thus, as the reader is encouraged to explore what other Al paradigms, particularly new ones invented since the present publication, may be used as a substitute for the technologies used in the present examples, so too is the user encouraged not to read the terms here as unnecessarily restrictive. For example, an Al model that can perform the functions of the LLM tool should not be overlooked or avoided merely because it is not called an LLM or because the term LLM has fallen out of favor in the industry. The term should be interpreting as encompassing such an option.

[0473] EXEMPLARY SOFTWARE ARCHITECTURE FOR GENERATING AN LLM QUERY WITH AN ADAPTED SYSTEM PROMPT:

[0474] Reference is now made to Figure 15, illustrating an exemplary application program 1500 (the application program 1500 that is the result of the processor 102 executing the program code 1701) for running the security system 100. The application program 1500 includes an agent module 1501 for generating the LLM query from the received input query and a system prompt, and transmitting the generated LLM query to the LLM 1503 (e.g. running on external server 120).

[0475] The agent module 1501 causes a managing of the system prompt template library 1504 including the plurality of system prompt template elements. By managing the system prompt template library 1504, the agent module 1501 causes the processor to retrieve one or more system prompt template elements from the system prompt template element library, in response to a received input query (e.g. where the retrieval may involve an analysis of the received input query to classify the input query as a type of input query), and assemble or generate a system prompt to be provided with the input query, as an LLM query, the system prompt assembled from the retrieved one or more system prompt template elements of plurality of system prompt template library. In some embodiments, the agent module 1501 may cause the processor to modify the string of characters of the retrieved one or more system prompt template elements in accordance with the nature of the received input query.

[0476] In some embodiments, the agent module 1501 may cause an updating of the system prompt template element library by replacing, modifying, deleting or adding to the one or more system prompt template elements of the plurality of system prompt template elements.

[0477] The exemplary agent module 1501 receives the input query for querying an LLM, structures the input query into an LLM query, and transmits the LLM query to the LLM for querying the LLM. The agent module 1501 includes program code that causes the processor to access the system prompt template element library to identify and include a system prompt with the received input query. The program code of the agent module 1501 may further cause the processor to select portions of system prompt components to include as part of the system prompt (e.g. selected from a data source or list of system prompt components) of the LLM query, where the system prompt may be adapted or customized in accordance with the nature of the input query.

[0478] For instance, an exemplary list of system prompt template elements of a system prompt template element library may be structured as follows:Template Element 1: [system prompt text] [usage logic] [variable]Template Element 2: [system prompt text] [usage logic] [variable]Template Element 3: [system prompt text] [usage logic] [variable]

[0479] Each “template element” includes its own system prompt text as the “system prompt text”, which is a string of characters resulting in the system prompt text. The usage logic may be code that provides instructions regarding when, and optionally how, the specific template element is to be selected by the processor executing the agent module 1501 for generating the system prompt that correspond to the input query (e.g. following a determination or classification of a type of input query). The “variable” field is an optional field and includes information that can be adapted depending on the nature of the received input query (e.g. based on a type of received input query). In some instances, the content of the “variable” field may be defined from contextual information surround the input query, the contextual information as described herein. In some implementations, instead of having a plurality of template elements to select, the memory may store a single system prompt template element with a variable field that can be adapted in accordance with the characteristics of the received user query (e.g. the type of the user query).

[0480] As mentioned, given the rapidly changing Al-related lexicon, terminology must be interpreted according to its meaning, purpose and intent herein. For example, the term system prompt is used herein to describe instructions to the Al model provided in addition to a user query but is not part of the user query and that provides the Al model with instructions on how to service the user query. The term “system prompt” is used commonly in the context of GPT, such as Microsoft Azure GPT services proposed in the present examples. It is possible that other Al models have similar instruction possibility but call it by a different name. These should not be seen as outside of the scope of the term system prompt. Typically, the system prompt instructions are not generated by the user. Typically, the system prompt instructions are not shown to the user and are invisible to the user. Typically, the system prompt instructions accompany the user query, and typically are sent together with the user query in a same request (e.g. API call) to the Al model, e.g. LLM. Typically, a system prompt is sent with every request to the Al model.

[0481] In one example, a template element’s system prompt text may be instructions as described herein, and specific to a certain type of action expected of the LLM, such as a particular type of search. For example, it may be adapted specifically for searching for, or for generating an API call for searching for, access control events:System:You are responsible for creating a JSON document used for an API to search for access control events within an access control system. Do not answer with anything other than JSON.The output must contain a suitable start and end detection time range In the field 'detectionTimeRange' based on user input and current time. If the user specifies a duration (e.g. 'in the last 4 hours' or '4pm to 5pm), use exactly that duration as a time range. If the user specifies a point in time (e.g. '15min ago' or 'yesterday'), an time range should be assumed using an appropriate margin around the specified time.For example: when the user asks for the last 10 minutes the time range be from 15 minutes to 5 minutes in the past. The current time is 2024-07 -11 T13:30:02.0586560Z.If the user specifies a past point in time rather than a duration, make sure to add a suitable duration before and after. For example: when the user asks for results at 4am today, select a time range between 3am and 5am.The output must contain filters suitable to the user query.Valid event types are: ‘alarm’, ‘door open’, ‘door close’, ‘door lock’, ‘door unlock’, ‘badge read’, ‘access granted’, ‘access denied’, ‘door forced open’, ‘movement detection’, ‘new badge registration’, ‘new user registration’, ‘user modification’, ‘user deletion’.The output must conform to following JSON schema: '"json{"type ": "object", "properties ": {"reason I ng ":{"type ": "string "payload":{"type ": "obj ect", "properties ":{" person ":{"type ": "array", "items ":{"type ": "object", "properties ":{ "eventType ":{"type ": "object", "properties":{"oneOf":{"typ ": "array", "items":{"type '""string", "enum":["Alarm", "DoorOpen", "DoorClose", "DoorLock", "DoorUnlock", " badge read", "access granted", "access denied", "door forced open", "movement detection", "new badge registration", "new user registration", "user modification","userDeletion "]}}}}}}}, "detection TimeRange ":{"type ": "object", "properties":{"start":{"type "string", "format": "date-time "end":{"type "string ", "format": "date- time"}}}}}}}

[0482] Other system prompt template elements can concern other types of searches, and may accordingly contain different system prompt text, such as given in earlier examples, or portions thereof (e.g. split to separate vehicle searching and person searching).

[0483] The usage logic can be in any suitable format provided that it can be interpreted to resolve whether a system prompt template element is to be used to generate an LLM query. The usage logic may depend on the user input (including the user query), context data, or other information. In one example, the usage logic is interpreted directly as a function of the input query such as (in pseudocode): IF user_query CONTAINS keywords indicating access control search THEN SELECT system_prompt_segment = "Access Control Events Search Instructions". Alternatively, the user input, context data, or other information could be provided to an LLM with another system prompt (the assessment system prompt), which may itself be stored in memory as a system prompt template element as described herein and updated in the same manner as needed, instructing the LLM to assess which template element is(are) needed and return data to use against the usage logic. Alternatively still, such an LLM could be used to identify directly which system prompt template elements to use (and, optionally, how if the system is variable) and directly cause the right template elements to be identified and / or used. For example, the LLM may be provided with all the useable system prompt template elements as inputs and its own system prompt (removed from the list provided to the LLM, if saved as a template element) and tasked with generating the system prompt. In such a case, usage logic may be absent from the system prompt template elements, being replaced by instructions to the LLM, e.g. in the assessment system prompt. Finally, usage logic may be hard coded in the system and therefore absent from system prompt template elements.

[0484] Variables may be included in the system prompt template elements, within the usage logic or in its own area of memory. For example, variables may include portions of system prompt text which may be different in certain circumstances, e.g. for different user queries or, more commonly, for different context information such as for different versions of a system. E.g. the portions of text describing supported vehicle types, for example, may be different for different versions of the system (which may have different analytics engines) and so the portion of the system prompt text in the system prompt template element may be a variable, the different values of which are provided in the system prompt template, such as in the variable portion. (Of course, from the foregoing, the reader will understand thatalternatively, the different capabilities of different versions of the system could be accounted for by providing different system prompt template elements for each possibility).

[0485] In some embodiments, the program code of the agent module 1501 may cause the processor to retrieve and include a history of previous input queries (and corresponding LLM queries sent to the LLM, as well as the response given by the LLM to the LLM query, such as the structured data source queries generated by the LLM).

[0486] In some embodiments, the program code of the agent module 1501 may cause the processor to classify the input query as a type of input query (e.g. from a predetermined list of input query types). The analysis of the input query to determine the input query type of the input query may be performed using an algorithm, or using an artificial intelligence model (e.g. an LLM).

[0487] The agent module 1501 may be an algorithm, or may be or include an artificial intelligence module (e.g. an LLM) for generating the LLM query from the received input query.

[0488] In some embodiments, the agent module 1501 may be adapted to respond to certain events to update the system prompt that is added as part of the LLM query to the received input query (e.g. by updating the data source of possible system prompt elements to add as part of the system prompt). Exemplary events that can trigger an update of the stored system prompt elements include, but are not limited to, an implementation of a new patch for the software system related to the agent module 1501, an update of the video monitoring system that is in communication with the LLM, a time event that is to occur after the lapse of a predetermined amount of time, a warning message received from the system with the data source(s) to be queried using the structured data source generated by the LLM, etc.

[0489] In some embodiments, when generating the LLM query, the agent module 1501 may cause the processor to perform a selective culling of system prompt components and / or LLM chat history when generating the LLM query, in order to, for instance, avoid overwhelming the LLM 1503, reduce the number of bad results, reduce the number of tokens, etc.

[0490] In some instances, the content of a system prompt template element, or the system prompt template element library made available to the agent module 1501, may be corrected through software interacting with the agent module 1501 or the memory 1502 storing the system prompt template element library. For instance, the stored system prompt template element library for generating the system prompt may be modified to include or remove a system prompt template element. In some instances, the stored system prompt template element library for generating the system prompt may be modified to change the content of one or moreof the system prompt template elements (e.g. logic, variable, etc.). The stored system prompt template element library for generating the system prompt may be modified to modify the system prompt text and / or usage logic for one or more of the system prompt template elements. In some instances, at least a subset of the system prompt template elements library may be adapted following receipt of feedback from the surveillance system (e.g. the video surveillance system), such as an error message (e.g. when a colour that is queried by the LLM 1503 is invalid, when a search category - e.g. type of vehicle - is not recognized by the system that is queried by the query generated by the LLM 1503, etc.). In some embodiments, the system prompt template element library may be adapted following receipt of user feedback (e.g. a “like” or “dislike” selection made by the user following receipt of the results queried using the query generated by the LLM 1503).

[0491] In some implementations, the system prompt template elements may be updated in memory 1502 by changing lines of characters stored in memory corresponding to the available system prompt template elements and / or through a pointer change for a location or address in memory for one or more of the system prompt template elements.

[0492] In some instances, the LLM 1503 (e.g. running on external server 120) may be in communication with a video monitoring system 100 and the agent module 1501, where the LLM 1503 queries a data source containing image streams generated by the camera(s) of the video monitoring system. In some implementations, the image stream(s) generated by the cameras of the video monitoring system may include metadata that provides information on the image stream generated following an image analysis performed by the camera of the video monitoring system (e.g. such as with the ONVIF (Open Network Video Interface Forum) standard). For instance, the metadata may include information on objects appearing in the image stream, information on a time of the video, etc.

[0493] In some instances, the agent module 1501 may analyze the received input query to categorize the input query into a type of input query, depending on the nature of the query of the input query. Exemplary input query types include, but are not limited to, an API search call (e.g. “find red car”), an API configuration request (e.g. set outdoor cameras to 15 FPS), an API call with graphical user interface instructions (e.g. show me the outdoor camera feeds), etc.

[0494] EXEMPLARY METHOD OF UPDATING A SYSTEM PROMPT THAT IS INCLUDED AS PART OF AN LLM QUERY:

[0495] Reference is now made to Figure 16, illustrating an exemplary method 1600 of updating a system prompt that is included as part of an LLM query. Method 1600 may be performed by agent module 1501, or any other algorithm or artificial intelligence model inaccordance with the present teachings.

[0496] An input query is received at step 1610. The input query may have been provided as user input using a search interface of a security system (e.g. via a graphical user interface of a video monitoring system). In some embodiments, the input query may be generated by software (e.g. generated by the video monitoring system following an occurrence of a trigger event, such as the identification of a vehicle with a specific licence plate identified in one of the image streams; in some instances, the trigger event may be defined by a user, where the identification of an event with certain defined properties is to initiate a querying of a database).

[0497] The input query may be categorized at step 1620, to establish a type of the received user input. The categorization of the received user input may be performed via an algorithm or via a trained artificial intelligence model (e.g. an LLM) adapted to establish a type of the received input query For example, for a video monitoring system, exemplary input query types include, but are not limited to, an API search call (e.g. “find red car”), an API configuration request (e.g. set outdoor cameras to 15 FPS), an API call with graphical user interface instructions (e.g. show me the outdoor camera feeds), etc.

[0498] A query is then performed to retrieve from memory one or more system prompt template elements from a system prompt template element library that correspond to the received input query at step 1630. In some embodiments, the memory may include a plurality of system prompt template elements that are retrievable for generating the system prompt corresponding to the received input query. An exemplary list of possible system prompt template elements of a system prompt template element library may be as follows:- Template Element 1: [system prompt text] [usage logic] [variable]- Template Element 2: [system prompt text] [usage logic] [variable]- Template Element 3: [system prompt text] [usage logic] [variable]

[0499] Each “template element” includes its own system prompt text. The usage logic may be code that provides instructions regarding when the specific template element is to be selected (e.g. in accordance with a determination of the type of received input query).

[0500] In some embodiments, instead of providing a plurality of system prompt template elements stored in memory, a single system prompt template element may be stored in memory with a variable field. The “variable” includes information that can be adapted depending on the nature of the received input query (e.g. based on a type of received input query). In some instances, the content of the “variable” field may be defined from context data surrounding theinput query and the use of the application program 1500 (and / or, in some instances, other application programs operating on the computer of the search, the browsing history of the user, etc.), the context data as described herein. In some instances, the system prompt template element contains the context data, or can be processed or adapted to retrieve or include the context data based on use of the application program 1500 related to the received input query.

[0501] In some instances, wherein the usage logic of one or more of the system prompt template elements included in the library of system prompt template elements includes logic indicating whether to use a respective system prompt template based on the context data. The context data, related to the usage of the application program 1500, may inform the selection of the appropriate system prompt template element of the plurality of system prompt template elements contained within the library of system prompt template elements. For instance, the context data may provide information on the use of the application program 1500 which led to the generating of the input query (e.g. if the system was displaying recorded image streams related to a given time or geographical area, which could indicate that the received input query would further relate to retrieval of a related image stream, and the system prompt template element to be retrieved to generate the LLM query would similarly be related to retrieval of an image stream from a specific data source). The usage logic may provide information regarding if its system prompt template element is appropriate based on the context data (e.g. if the context data is indicative that the input query is to retrieve and display an image stream, then the usage logic may include information, such as a field defining a type for the system prompt template element, indicative of if the system prompt template element is appropriate for retrieval of image streams). In some embodiments, usage logic of system prompt template elements may be interpreted in light of the context data to determine if a system prompt template element is appropriate in light of this interpretation of the context data. The interpreting of the context data and the usage logic of the system prompt template elements may be performed by an algorithm, or by a trained artificial intelligence model. For instance, the context data may be interpreted to classify the received input query as a type of input query (e.g. where context data including search history of a user through a list of available cameras would indicate that the input query may involve turning on an available camera, and the interpreting is to logic usage logic of a system prompt template element related to causing a configuration of a camera or sensor).

[0502] The retrieved system prompt stored in memory that corresponds to the received query input is received, in order to generate the LLM query.

[0503] In some embodiments, a verification is performed if the received input query is partof a prior exchange with the LLM tool, where previous related LLM queries were sent to the LLM tool, resulting in a chat history, at step 1640. In those instances, this chat history may be included as part of the LLM query to be generated, along with the input query and the system prompt. The chat history may include a series of previous one or more LLM queries, and the corresponding response(s) received from the LLM tool (the structured data source query) in response to the LLM query(ies). The chat history may also include information received from the data source following transmitting the structured data source query to the one or more data sources.

[0504] The LLM query is generated at step 1650 from the input query, the system prompt and optionally, the chat history.

[0505] The LLM query is transmitted to the LLM tool, where the LLM tool generates a response (e.g. a structured data source query), in response to the LLM query, the response from the LLM adapted in accordance with the instructions contained within the selected system prompt.

[0506] Following receipt of the response from the LLM (e.g. the structured data source query) and / or the information retrieved following the transmission of the structured data source query generated by the LLM tool to the target data source(s), feedback may be received (as user input) from the user on the response from the LLM or the information retrieved following the transmission of the structured data source query generated by the LLM tool.

[0507] Depending on the nature of the input query received, the structured action executed by the LLM may not always be to generate a structured data source query (to query one or more data sources). For instance, the structured action may be to generate instructions for causing a modification of a configuration of one or more sensors or devices (e.g. a camera, an alarm system, a motion detector, etc.) when the input query requests a change (such as a setting change) of the sensor or device (an example of an input query may be: “turn on camera A” or “silence alarm” or “zoom camera B on the person standing in the parking lot”). In some instances, the structured action may be to cause a change of the information displayed on a graphical user interface appearing on the display of the system 100 (or another display in communication with the system 100), in response to input query that may be, for instance “display a video on the graphical user interface”, “identify that man appearing at time “X” on the video”, etc.

[0508] It will be understood that the structured action generated by the LLM may be a string of characters adapted in a standard for processing by the system, device and / or data source that is to be the target of the structured action. The nature of the structured actiongenerated by the LLM may depend on the nature of the instructions or request provided in the input query.

[0509] In some embodiments, depending on the nature of the input query that is received, the LLM may generate more than one structured action, if more than one structured action would respond to the received input query (e.g. such as querying a data source for a recorded image stream while turning on a camera to receive an image stream generated by that camera in real time).

[0510] For instance, one structured action may be configuring a device related to the system. Exemplary device configuration actions may be instructing a camera to change resolution, encoding or framerate (e.g. using an existing standard, such as the ONVIF standard). An exemplary device configuration may be instructing a badge reader to change its behavior (e.g. performing using the badge reader’s own standard, such as OPSDP). When the structured action is a device configuration action, the system prompt may include instructions for instructing the LLM to create a set of compliant instructions (e.g. ONVIF or OSDP) or API calls to cause the configuration in accordance with the received input query. The examples above show how to provide system prompt instructions for generating a properly structured API call. Similar instructions can be provided to generate a properly structured ONVIF or OSDP message or a message according to another standard or API. These will include instructions providing the LLM with the syntax and format of a properly structured message (e.g. according to a standard or API) and likely also instructions on how to choose the content of such a message. In response, the LLM generates and returns the properly structured message which may be device configuration action instructions. The application program receives the device configuration action instructions received from the LLM, once the LLM has executed the LLM query generated from the received input query. The application transmits the received device configuration action instructions from the LLM and transmits the received device configuration action instructions to the target device (e.g. at the IP address of the target device).

[0511] Therefore, in some instances, a system prompt template element may contain instructions for causing the LLM to generate the device configuration action instructions in accordance with the instructions of the system prompt template element, and the received input query, part of the transmitted LLM query, where the device in external from the application program of the security system.

[0512] In some instances, a structured action may be an application program configuration action. Exemplary application program configuration actions may include, for instance, instructing the program to switch modules (e.g. switch from access control to videosurveillance or vice versa), instructing the program to change the camera feeds on display (e.g. “put camera 1,2, 4, and 6 on the screen” or “switch to a camera view”). The system prompt of the LLM query includes instructions for causing the LLM to create a properly formulated API call to the application program itself using the program’s API, instructing the program to perform the change. Again, as in the examples above, instructions may be provided to generate a properly structured message instructing the application program to configure a certain way, such as an API call. These will include instructions providing the LLM with the syntax and format of a properly structured message (e.g. according to the API) and likely also instructions on how to choose the content of such a message. In response, the LLM generates and returns the properly structured message which may be application program configuration instructions. The program receives the application program configuration instructions from the LLM and executes the API call in conformity with the received application program configuration instructions.

[0513] Therefore, in some instances, a system prompt template element may contain instructions for causing the LLM to generate the application program configuration instructions in accordance with the instructions of the system prompt template element, and the received input query, part of the transmitted LLM query.

[0514] In some instances, a structured action may be instructions to cause a change in configuration of a graphical user interface (GUI) of the application program (e.g. a change of appearance). Exemplary changes to the graphical user interface of the application program may be, for instance, causing one or more tiles to appear on the graphical user interface to receive and display corresponding one or more image streams, opening or closing one or more windows appearing on the graphical user interface of the application program, changing a language of the text appearing on the graphical user interface of the application program, etc. Other reconfigurations may include arming or disarming an alarm, or the like, which may involve a change in the GUL

[0515] Therefore, in some instances, a system prompt template element may contain instructions for causing the LLM to generate instructions to cause a change of a configuration of a graphical user interface of the application program. The instructions of the LLM are generated from the instructions of the system prompt template element, and the received input query, part of the transmitted LLM query. A modified graphical user interface is resulting displayed, once the instructions received from the LLM have been processed by the system.

[0516] In some embodiments, the system prompt template elements of the system prompt template element library stored in memory, and retrievable to generate the system prompt thatis part of the LLM query, may be updated at step 1670. For instance, the system prompt template elements may be updated following receipt of user input, provided as feedback, related or to cause a change of the system prompt template elements. In some embodiments, messages or feedback received from the system with the data sources to be queried using the structured data source query generated by the LLM. Such messages or feedback may include, for instance, an error message, such as indicating that a category of a characteristic of object to be queried is not recognized by the data source. The received error message can be analyzed, and the system prompt template elements may be modified in order to avoid a similar error message in response to a future query by correcting the system prompt template element(s) in accordance with the received error message.

[0517] In some examples, when the system (or the data source(s) of the system) that is queried using the structured data source query generated by the LLM is updated, the system prompt template elements may also be correspondingly updated (such as when the system is added to include some new search fields, those search fields can be added to the system prompt template element(s) stored in memory).

[0518] In some embodiments, the system prompt template element stored in memory may be updated periodically following a passing of a pre-determined amount of time.

[0519] In some examples, where the system prompt template element are updated, a pointer location for the system prompt element stored in memory can be updated, the pointer, following the update, pointing to a new location in memory. In some embodiments, when the system prompt template element(s) is updated, the string of characters corresponding to that system prompt template element may be modified.

[0520] EXEMPLARY METHOD OF UPGRADING A SECURITY APPLICATION PROGRAM:

[0521] Reference is now made to Figure 18, illustrating an exemplary method 1800 of upgrading a security application program. The method 1800 may be performed by system 100, or any other system in accordance with the present teachings.

[0522] The security application program may have a search interface (e.g. a chatbot) for searching through a data source of security records, where the chatbot interacts with an LLM to produce a structured action in accordance with the received input query.

[0523] A computer-readable memory storing an original system prompt library including a plurality of system prompt template elements, where each of the system prompt template elements may include instructions for generating at least a portion of a system prompt, as described herein, is accessed at step 1810.

[0524] One or more of the system prompt template elements stored in the system prompt template element library is modified at step 1820, resulting in a modified system prompt template element library. The system prompts generated by the modified system prompt template element library are different from the system prompts that could be generated by the system prompt template element library prior to the change. The modified system prompt template element library includes a modified portion of the original system prompt template element library, where certain of the original system prompt template elements stored in the system prompt template element library are modified.

[0525] In some instances, a modification to the system prompt template element library may include modifying a system prompt template element contained in the system prompt template element library, such that the system prompt generated from the modified system prompt template element is different from the system prompt generated by the system prompt template element prior to the modification. For instance, the “system prompt text” of a system prompt template element may be modified. In some instances, the usage logic of the system prompt element may be modified. In some instances, more than one system prompt template element may be modified.

[0526] In some instances, a modification to the system prompt template element library may include adding one or more new system prompt template elements to the system prompt template element library. A new system prompt may be generated from the newly added system prompt template element, to generate a new LLM query.

[0527] For instance, in one example, the newly added system prompt template element may be to support, and to permit a search, for new kinds of video metadata not supported by the security system prior to the modification, and where system prompts generated using the original system prompt template element library did not permit generating a LLM query that would cause the generating of instructions by the LLM to search through the new kind(s) of metadata.

[0528] In some embodiments, a modification of the system prompt template element library may include deleting one or more system prompt template elements included in the former system prompt template element library. A system prompt corresponding to the one or more deleted system prompt template elements may no longer be generated following the deletion.

[0529] The modification of the system prompt template element library may be triggered following a receipt of an error message from the security application program, an external device, a data source, following a processing of structured instructions generated by the LLMtool. The error message may be indicative that one or more of the system prompt template elements contained in the system prompt template element library results in the generating of inexecutable instructions. In one example, ahistory may be retrieved and analyzed to determine which system prompt template element of the system prompt template elements of the system prompt template element library was used to generate the system prompt that led to the structured action that caused the error message. The identified system prompt template element of the system prompt template element may accordingly be modified.

[0530] In some instances, the modification of the system prompt template element library may follow an update of the security application program (e.g. a patch for the security application program), an update of one or more of the searchable data sources related to the security application programs, the addition of anew category or model of device (e.g. camera, badge reader, etc.) that is in communication with the security application program. In some instances, the modification of the system prompt template element library may following a modification of the security application program to support a new category of security device (e.g. such as a new biometric reader), etc.

[0531] In some embodiments, the modification of the system prompt template element library may be performed following receipt of user input at the security application program to cause a modification of the system prompt template element library.

[0532] In some instances, a punctual verification may be performed to determine if a modification of the system prompt template element library is required (e.g. based on the status of the security application program, feedback received from one or more users, a number of error messages received in response to input query received at the search interface of the security application program).

[0533] In some instances, the modification of step 1820 may be performed while the security application program is running. The modifying of at least a portion of the system prompt template element library is performed during normal operation of the security application program such that security operations performed by the security application program do not need to be interrupted for the upgrading of the system prompt template element library.

[0534] In some embodiments, the security application program includes a pointer to the memory location in the computer-readable memory of the original system prompt template element library. The modifying of one or more system prompt template elements of the system prompt template element library may include storing at anew location in the computer-readable memory as a new system prompt template element library, the modified system prompttemplate element library (or the one or more modified system prompt template elements). The memory at the new location may include the modified portion of the original system prompt template element library and a copy of any unmodified portion of the original system prompt template element library. The pointer of the security application program may be changed to the new location in the computer readable memory.

[0535] Once the system prompt template element library has been modified, an input query may be received at step 1830.

[0536] The modified system prompt template element library may be accessed at step 1840.

[0537] One or more system prompt template elements of the system prompt template element library may be determined and retrieved at step 1850. The determined one or more system prompt template elements may be that of the modified system prompt template element library. The determination of the one or more system prompt template elements may be performed based on a determination of a type of the input query, based on context data surrounding use of the security application program, the usage logic corresponding to each of the system prompt template elements of the modified system prompt template element library, etc.

[0538] The system prompt is generated from the retrieved one or more system prompt template elements at step 1860. The system prompt includes instructions for the LLM on how to respond to an LLM query, the system prompt being different from the system prompt that would have been generated using the original system prompt template element library.

[0539] The LLM query is generated form the system prompt and the input query.

[0540] The generated LLM query is transmitted to the LLM tool at step 1870. The LLM tool generates a structured action when executing the LLM query. For instance, the LLM tool may be a tool to generate the structured data source query for causing a searching of a data source of surveillance records based on the received LLM query. In this example, the structured data source query may be received from the LLM, and a search of one or more data sources containing surveillance records may be performed in accordance with the received structured data source query.

[0541] In some instances, the LLM may generate instructions to cause a change in configuration of a graphical user interface (GUI) of the security application program. In some instances, the LLM may generate instructions to cause a change in configuration of a device external to the security application program. It will be understood that the structured action instructions generated by the LLM may be to cause an action with respect to the application program, server, device or data source related or connected to the security system.

[0542] EXEMPLARY SYSTEM TO IMPROVE AN LLM QUERY WITH CONTEXT DATA:

[0543] With reference to FIG. 19, there is shown a computer system 10 according to one example embodiment. In the illustrated embodiment, the computer system 10 comprises a security (or surveillance) system 12. One or more devices 14 are communicatively coupled to the security system 12. In the illustrated embodiment, a plurality of devices 14 (i.e., the devices 14-1, 14-2 and 14-3) are communicatively coupled to the security system 12.

[0544] The devices 14 are configured to (or operable to) maintain or monitor security of a desired area by collecting surveillance data from the area and / or control access to the area. For example, the one or more devices 14 may include image capture devices configured to capture still or video images (such as cameras, for example), access control devices (such as, for example, access card readers, access control pin pads, electronically controlled locks, etc.), audio capture devices (such as microphones, for example), one or two-way communication devices (such as intercoms, for example), etc. Different ones of the devices 14 may be the same or different. Suitable devices 14 may be based on a variety of commercially available models made by a variety of manufacturers.

[0545] The security system 12 is configured to receive and manage data from the devices 14. For example, the security system 12 may be caused to run a data management system (such as a video management system, for example) to receive and manage data from the devices 14. The security system 12 may comprise at least one processor 13 configured to (or operable to) run the data management system. In some embodiments, the security system 12 is, or includes, a server computing device. The data management system may be based on a commercially available or existing system such as Genetec™ Security Center. In some embodiments, the data management system is a cloud-based system.

[0546] A user may monitor data from the devices 14 (e.g., monitor still image or video feeds from cameras, monitor access card scans, etc.) and / or control settings of the devices 14 (e.g., which cameras are actively recording, which access cards should be given access to the area, etc.) using an application program. The application program may, for example, be a surveillance system application program. The application program may be run on a user computing apparatus 15 which is communicatively coupled to the security system 12. The user computing apparatus 15 may be caused to obtain data from the security system 12 and provide the obtained data to a user through the application program running on the user computing apparatus 15.

[0547] The user computing apparatus 15 is a computing device and may, in variousembodiments, include a user computing device, a server computing device, a personal computer, a laptop computer, a tablet computer, a smartphone, a mobile device or one or more other devices including one or more other computing devices or a combination of two or more thereof.

[0548] The user computing apparatus 15 includes a processor circuit 16 which includes a central processing unit (CPU) 17. However, alternative embodiments may include one or more alternatives to the CPU 17, such as one or more microprocessors, one or more analog circuits, one or more configurable logic blocks, one or more application-specific integrated circuits (ASICs), or one or more field programmable gate arrays (FPGAs), for example. The processor circuit 16 also includes an input / output (I / O) interface 18 and a data-storage device 19 in communication with the CPU 17.

[0549] The I / O interface 18 may include various signal interfaces, analog-to-digital converters (ADCs), receivers, transmitters, and / or circuitry to receive, produce, and transmit signals as described herein, for example. In general, signals as described herein may include one or more radio signals, one or more optical signals, one or more electronic signals, or a combination of two or more thereof. In the embodiment shown, the I / O interface 18 is operable to transmit signals to, and receive signals from, a communication network 23 using one or more networks such as the Internet, one or more wired networks, one or more wireless networks or a combination of two or more thereof, for example.

[0550] The I / O interface 18 may be operable to receive signals from one or more input devices, such as an input device 20 that may include a keyboard, a mouse, a touchscreen, a microphone, another user-input device, another input device or a combination of two or more thereof to receive inputs, such as user inputs from a user of the user computing apparatus 15, for example. Also, the I / O interface 18 may be operable to transmit signals to one or more output devices such as an output device 21 that may include a display screen, an audio speaker, a projector, another user-output device, another output device or a combination of two or more thereof to control one or more such output devices to produce output, for example, for the user of the user computing apparatus 15. The input device 20 and the output device 21 are shown as separate devices, but may be collectively one device such as a touchscreen, for example. Also, the input device 20 and the output device 21 are shown in FIG. 1 as part of the user computing apparatus 15, but in alternative embodiments one or more input devices, one or more output devices, or both may be separate from the user computing apparatus 15.

[0551] The data-storage device 19 may include one or more of the same or different computer-readable and / or computer-writable data-storage media, which in variousembodiments may include one or more of a read-only memory (ROM), a random access memory (RAM), a hard disc drive (HDD), a solid-state drive (SSD), and other computer- readable and / or computer writable data-storage media.

[0552] The data-storage device 19 includes a program-code store 22 storing program code that, when executed by the CPU 17, cause the processor circuit 16 to implement functions of the user computing apparatus 15 such as those described herein, for example.

[0553] The processor circuit 16 is an example only, and alternative embodiments may differ. For example, alternative embodiments may include more, fewer or different components. Also, in alternative embodiments, components described herein may be combined or separated into separate components. Alternative embodiments may include one or more alternatives to components as described herein. Further, an alternative to the user computing apparatus 15 may include multiple devices that collectively function as the user computing apparatus 15.

[0554] The user computing apparatus 15 may be communicatively coupled to the security system 12 with the communication network 23. The communication network 23 may be a secure communication network. The communication network 23 may include routers, switches, splitters, buffers and any other components needed to communicate between the security system 12 and the user computing apparatus 15. In some embodiments, the communication network 23 comprises one or more components or devices operable to couple the user computing apparatus 15 to the cloud (e.g., to a cloud-based implementation of the security system 12). In some embodiments, the communication network 23 is a local network.

[0555] The devices 14 may be communicatively coupled to the security system 12 with the communication network 23 or a different communication network. At least one of the devices 14 may be a direct-to-cloud device (such as a direct-to-cloud camera, for example). The direct- to-cloud device 14 may, for example, directly couple to a cloud-based security system 12 (i.e., no other network components are required to couple the direct-to-cloud device to the security system 12). One or more devices 14 which may not have a direct-to-cloud feature may nevertheless still be coupled to the cloud (and / or a cloud-based security system 12). For example, a device 14 may be coupled to the cloud using a cloud gateway device.

[0556] The computer system 10 also includes a large language model (LLM)-based prompt-response system 25 (alternatively may be referred to as an LLM-based promptresponse tool). In the illustrated embodiment, the LLM-based prompt-response system 25 is communicatively coupled to the security system 12 and the user computing apparatus 15 with the communication network 23. In some embodiments, the LLM-based prompt-responsesystem 25 includes, or is at least partially hosted on, a server computing device-based system. For example, the LLM-based prompt-response system 25 may be hosted on a local server computing device. The local server computing device may be the same device as the server device which is also operable to host the security system 12, for example. In some embodiments, the LLM-based prompt-response system 25 is a cloud-based system. In some embodiments, the LLM-based prompt-response system 25 is executed by, or hosted on, a plurality of computing devices.

[0557] The LLM-based prompt-response system 25 may comprise a plurality of LLMs. Different LLMs of the plurality of LLMs may have different functionalities. The LLM-based prompt-response system 25 may be configured to, or may be caused to, decide which ones of the plurality of LLMs to use in response to a query (such as a query 46 described herein) that is input into the LLM-based prompt-response system 25. Additionally, or alternatively, the query may at least partially instruct, or provide information to, the LLM-based promptresponse system 25 on which ones of the plurality of LLMs to use.

[0558] In the illustrated embodiment, the LLM-based prompt-response system 25 is shown as being a separate computing device from the security system 12 and the user computing apparatus 15. In alternate embodiments, the security system 12 or the user computing apparatus 15 may be caused to run the LLM-based prompt-response system 25, for example.

[0559] As discussed above, a user may interact with the security system 12 through the application program running on the user computing apparatus 15. The user interacting with the security system 12 may comprise the user viewing data (such as by using the output device 21, for example) from the security system 12 and / or the devices 14 (such as, for example, one or more camera feeds from devices 14 that are cameras, viewing recent access card scans, etc.). Additionally, or alternatively, interacting with the security system 12 may comprise controlling or varying one or more settings of the security system 12 and / or the devices 14 (such as varying which cameras are connected to the security system 12, for example).

[0560] A query interface may be associated with the application program. Through the query interface, the user of the user computing apparatus 15 may seek out or request further information about a current and / or past output of the application program.

[0561] The query interface being “associated with” the application program can encompass any or all of the following:In one example, “associated with” means that information on usage of the application program provides context that is necessary for interpreting a word or expression input into the query interface. For instance, a word or expression input into the query and requiringcontext for interpretation may include a determiner (such as “that”, “this”, etc.) referring to an object in a displayed scene, or a seemingly incomplete query that is only resolvable with knowledge of the displayed scene (“who was the last person to come through?”), among other examples. The input need not be limited to text input;- In another example, “associated with” means “part of’, i.e., the query interface is part of the application program. Since the query interface is part of the application program, data from the query interface and information on usage of the application program may be readily accessible together. Permission to access one of the data from the query interface and information on usage of the application program may provide implicit permission to access the other one of the data from the query interface and the information on usage of the application program;- In another example, “associated with” means that data from the query interface and information on usage of the application program are obtained by the same computing device. In other words, the computing device has the necessary permission to access both data from the query interface and information on usage of the application program.

[0562] The further information that the user is interested in obtaining may at least in part be provided, or caused to be provided, by an output of the LLM-based prompt-response system 25. User input entered into a query interface may be supplemented with information (or data) on usage of the application program. The user input and the information on usage of the application may be processed to generate a query for the LLM-based prompt-response system 25. Based on the generated query, the LLM-based prompt-response system 25 may generate an output. The output of the LLM-based prompt-response system 25 may provide the further information that the user requested. Alternatively, the output of the LLM-based promptresponse system 25 may be an application programming interface (API) call (or content for an API call). Running the API call may provide the further information the user requested.

[0563] By supplementing the user input with the information on usage of the application program, accuracy and / or completeness of the output of the LLM-based prompt-response system 25 is increased.

[0564] In some embodiments, the query interface is, or includes, a chatbot interface. The chatbot interface may be a text chatbot interface, a voice or audio chatbot interface, a video chatbot interface or a combination of two or more thereof.

[0565] With reference to FIG. 20, there is shown an example output 30 of the application program that is displayed by the output device 21. In the illustrated example, the application program is a security center video management / access control application program. The output30 may be, or include, a graphical user interface (GUI), for example. The output 30 includes a display box or window 31. The display box 31 displays data from a license plate reader (LPR) module of the security system 12. In the illustrated embodiment, the display box 31 displays license plate (or license) readouts 32 corresponding to cars entering a parking lot. The output 30 also includes a display box or window 33 which displays image data 34 from a live video surveillance feed of the parking lot. The image data 34 may, for example, be captured by a device 14 that is a surveillance camera. In the illustrated example, the user is interested in locating the car with license plate (or license) 886BBS in the parking lot. The user highlights or selects the license plate readout corresponding to license plate 886BBS with selection window 35. The output 30 also includes a query interface 36. The query interface 36 is a query interface as described elsewhere herein. Through an input interface 37 of the query interface 36, the user may input an input 38 (such as by using input device 20, for example) requesting a location of the car with license plate 886BBS. In the illustrated embodiment, the user’s input 38 is “Where is this one?”. As described elsewhere herein, the user’s usage of the application program may be used to provide context for the user input 38 such that it can be determined (e.g., by an LLM-based prompt-response system (such as the LLM-based prompt-response system 25)) that based on the user’s monitoring of the parking lot the user is interested in locating something in the parking lot, based on the user’s use of the license plate reader module the user is interested in locating a car in the parking lot and that the user is interested in locating the car with the selected license plate of 886BBS based on the user’s selection of the license plate readout of 886BBS. Once an answer to the user’s inquiry is determined as described herein, the answer may be displayed for the user (or otherwise provided to the user). In the illustrated embodiment, the output 39 is displayed for the user as a response to the user input 38 in the query interface 36 and includes the message “The car with license 886BBS is located in stall 25” and an image 39A showing the car with license 886BBS in stall 25 of the parking lot.

[0566] Referring to FIG. 21 , program code stored in the program-code store 22 may include blocks of program code shown generally at 40 that, when executed by the CPU 17, cause the processor circuit 16 to obtain an output from the LLM-based prompt-response system 25.

[0567] The blocks 40 may begin at a block 41 including program code that, when executed by the CPU 17, cause the processor circuit 16 to receive user input 42 from a user of the user computing apparatus 15. The user input 42 comprises at least one request from the user. The request typically is, or includes, a request for further information about at least one observation the user has made through interacting with or using the application program. The request forfurther information may, for example, include a request to identify and / or locate a person, identify and / or locate an object (such as a car, for example), etc. The user input 42 may be received through a query interface as described elsewhere herein.

[0568] The user input 42 may be, or include, human language words (or natural language words). In some embodiments, the user input 42 is organized in human language sentences (or natural language sentences). The user input 42 may include at least one determiner (e.g., a word that is used before a noun to show which particular example of the noun you are referring to).

[0569] In some embodiments, the user input 42 comprises a string of characters. The string of character may define a series of words.

[0570] In some embodiments, the user input 42 comprises audio data. In some embodiments, the user input 42 comprises video data. In some embodiments, the user input 42 comprises one or more images. In some embodiments, the user input 42 comprises input generated on a touch screen.

[0571] As described elsewhere herein, it may be desirable for the LLM-based promptresponse system 25 to generate an API call or content for producing an API call. The user input 42 may include instructions to cause the LLM-based prompt-response system 25 to generate the API call or the content for producing the API call. For example, the user input 42 may include instructions to generate the content for producing an API call in a specific format (such as a JSON schema, for example).

[0572] In some embodiments, the user input 42 comprises restrictions and / or requirements on a format of an output (such as output 49, for example) of the LLM-based prompt-response system 25. For example, the user input 42 may specify what language (e.g., human or coding language) the output of the LLM-based prompt-response system 25 should be provided in. As another example, the user input 42 may specify requirements such as a maximum length or size of the output of the LLM-based prompt-response system 25, a file type (e.g., jpeg for images, PDF for documents, etc.) that the output of the LLM-based prompt-response system 25 should be provided in, etc.

[0573] The blocks 40 may continue at a block 43 including program code that, when executed by the CPU 17, cause the processor circuit 16 to obtain information or data 44 on usage of the application program (which may be referred to as “usage information” or “usage data”). The usage information 44 may comprise data representing one or more characteristics of the application program or its usage by the user. The usage information 44 may provide contextual information for the user input 42. The program code of block 43 may, for example, be that of a scraping tool for the application program.

[0574] The usage information 44 may comprise information regarding a historical use of the application program by the user. For example, the usage information 44 may comprise previous inputs entered by the user while using the application program, prior functions or operations carried out by the application program in response to requests inputted by the user, window panes and tools opened or used by the user while running the application program, a geographical location of the user when running the application program, times of use of the user while running the application program, prior search results generated by the application program or the LLM-based prompt-response system 25 in response to prior user input, etc. The usage information 44 may relate to a current session of use of the application program. In some embodiments, the usage information 44 relates to one or more prior sessions of use of the application program by the user. The usage information 44 may include information on a most recent action performed by the user using the application program, or information on a plurality of recent actions (such as, for example, the last 2, 5, 10, etc. actions) performed by the user using the application program. The information regarding a historical use of the application program (or information on usage of the application program by the user generally) may at least in part be retrieved from memory (such as from the data store 19, for example).

[0575] Additionally, or alternatively, the usage information 44 may comprise information regarding a current use of the application program by the user. Information regarding the current use of the application program may, for example, comprise information such as an identity and / or function of window panes (or modules) of the application program that are currently opened by the user or running, information currently being displayed by the application program such as a graphical user interface (GUI) generated by the application program (e.g., a region of a digital map displayed on the GUI, a timestamp of a video that is playing, content of a video that is being played, etc.), etc.

[0576] In some embodiments, the usage information 44 comprises one or more of information or data on what the user is in the process of accessing using the application program (e.g., door scans for a particular door), what the user has accessed in the past using the application program (e.g., a license plate read hotlist, etc.), current actions performed by the application program, current actions performed by the user using the application program (e.g., what pane or module is opened, what plugin is open, what window is opened), past actions performed by the application program, past actions performed by the user using the application program (e.g., what features or functions of the application program were opened or used recently, what module(s) of the application program has been used (the measure of use may be a measure to date, recently, ever, most commonly, etc.), etc.

[0577] In some embodiments, the usage information 44 comprises information or data on plugins and / or programs that are running, or have run, in association with the application program. The running plugins and / or programs may provide information on what is relevant to the user, based upon a nature of the running plugins or programs (e.g., a plugin for detecting access to a door in a building may suggest that the user is interested in finding out who had access to the door, a plugin for monitoring video feeds may suggest that the user is interested in monitoring activity in an area captured by the video, etc.), for example.

[0578] In some embodiments, the usage information 44 comprises information or data about the application program, related software modules, plugins and / or programs that are running, which may include information about their capabilities, their API and / or API-related content and formatting requirements, available data, etc. The information or data about the application program may comprise a version of the application program (such as version 3.6.2, for example).

[0579] In some embodiments, the usage information 44 comprises information or data about one or more of the devices 14 (such as, for example, cameras, access card readers, door controllers, license plate reading cameras, etc.) that the security system 12 is in communication with. The information or data may include information about the capabilities of the one or more devices 14, information about the one or more devices 14 (such as, for example, a model number, year of production, what software version is being run by the device, etc.), what data is available from the one or more devices 14, etc.

[0580] In some embodiments, the usage information 44 comprises information or data about the user of the application program. The information or data about the user may include the user’s name, job title, department, etc. In some embodiments, different users have different access privileges (or permissions). For example, a user who is an administrator may have full access while another user who is not an administrator may have only limited access to the features of the application program (e.g., a user with limited privileges may not have the necessary privileges (or permissions) to edit a hotlist, add or remove cardholders, etc.). The usage information 44 may include information about a user’s access privileges.

[0581] The application program may be associated with one or more software modules. The usage information 44 may comprise information on those of the software modules that are open when the user input 42 is received. In some embodiments, the usage information 44 comprises information on those of the software modules for which an associated window is caused to be displayed (such as by using output device 21, for example) by the user computing apparatus 15 when the user input 42 is received.

[0582] In some embodiments, a software module described herein and / or a feature may have two or more possible current statuses (such as enabled or disabled, whether a third part implementation or a native implementation is used, etc.). If a software module is enabled, the software module may be relied upon by the LLM-based prompt-response system 25, relied upon by the security system 12, accessed by the user, etc. In contrast, if the software module is disabled, the software module may not be relied upon by the LLM-based prompt-response system 25, relied upon by the security system 12, accessed by the user, etc. An indicator may indicate a current status of a particular software module and / or feature (e.g., whether a particular software module and / or feature is enabled or disabled, whether a particular software module and / or feature is available or not available, etc.). For example, a software module or feature may be enabled only if a subscription includes the software module. As another example, to conserve computational resources, to reduce expense (e.g., for pay-per-use services), etc. a software module or feature may be disabled. In some embodiments, the indicator is, or includes, a feature flag which may be toggled based on a current status of a software module or feature (such as whether the software module or feature corresponding to the feature flag is currently enabled or disabled, for example). The usage information 44 may include one or more indicators (such as feature flags, for example) indicating the current status of a corresponding software module or feature.

[0583] In some embodiments, the usage information 44 comprises one or more characteristics (e.g., trends, values, patterns, etc.) that quantify or qualify use of the application program by the user. The one or more characteristics may be derived from or based on other data of the usage information 44. The one or more characteristics may provide context regarding actions carried out by the user that led the user to provide the user input 42 (e.g., may provide information indicative to the motivation(s) or reason(s) prompting the user to make a query). The one or more characteristics may, for example, be in the form of one or more values, one or more strings of characters, etc. In some embodiments, the usage information 44 comprises data quantifying or qualifying a user’s use of one or more fdters (such as to narrow or expand what is being displayed or provided for the user, for example) during the user’s use of the application program.

[0584] In some embodiments, the usage information 44 comprises an output of the application program (such as an image frame being displayed for the user, for example). The output of the application program may, for example, impose at least one limit on a meaning of the user input 42 (e.g., if the user input 42 is “Who owns that car?” and the output shows only one car, the meaning of the user input 42 is limited to the owner of the single car that is shown).

[0585] The usage information 44 may include historical information such as information related to past searches. The historical information may be time limited (e.g., limited to a specified time period). In some embodiments, the historical information may be retrieved at least in part from the data store 19. The historical information may be short term (e.g., limited to a recent or short time period, discrete inquiry, etc.). Additionally, or alternatively, the historical information may be tong term (e.g., may cover a time period of several days, a week, several weeks, a month, several months, a year, several years, etc.). For example, if a user typically regularly accesses parking tot and / or license plate data, then it may be inferred from the corresponding tong term historical information that the user likely is interested in parking security and the user input 42 has a higher likelihood of being related to parking security.

[0586] If a user is part of a team of users, the usage information 44 may include historical use information for the team or at least some of the other team members.

[0587] In some embodiments, the usage information 44 is at least partially related to a past interaction a user had with the LLM-based prompt-response system 25 (e.g., current usage information 44 may be related to the immediately previous usage information 44 and may be built upon that, etc.). In some embodiments, the usage information 44 is not related to a past interaction a user had with the LLM-based prompt-response system 25.

[0588] In some embodiments, the usage information 44 includes metadata corresponding to data collected by one or more of the devices 14. The metadata may be generated by analyzing the data collected by the one or more devices 14. For example, if a device 14 is a video surveillance camera, the metadata may be generated by performing video analytics on the video image data. In such example, the metadata may, for example, include information on what the camera sees such as people, objects, etc.

[0589] In some embodiments, the usage information 44 may at least in part be provided directly to the LLM-based prompt-response system 25.

[0590] Blocks 41 and 43 may occur concurrently. In some embodiments, block 43 commences in response to the user input 42 being received at least in part. For example, block 43 may commence in response to a first character of the user input 42 being received.

[0591] Once both the user input 42 and the usage information 44 have been acquired, the blocks 40 may continue at a block 45 including program code that, when executed by the CPU 17, cause the processor circuit 16 to generate a query 46 for the LLM-based prompt-response system 25. Generating the query 46 may comprise processing the user input 42 together with the usage information 44 to refine the user input 42 in view of the usage information 44. As described elsewhere herein, the usage information 44 may provide contextual information forthe user input 42. Refining the user input 42 in view of the usage information 44 may increase the likelihood that the LLM-based prompt-response system 25 will provide an accurate or complete output. Additionally, or alternatively, refining the user input 42 in view of the usage information 44 may provide additional information transforming an otherwise unclear or unusable input into an input that may be received by the LLM-based prompt-response system 25 and used by the LLM-based prompt-response system 25 to produce a desired output. The user input 42 and the usage information 44 may be refined or reformulated into the query 46 in a manner to assist the LLM-based prompt-response system 25 in outputting an answer that is responsive to the user input 42. Refining or reformulating the user input 42 in view of the usage information 44 into the query 46 may assist the LLM-based prompt-response system 25 to compensate for a lack of information provided by the user.

[0592] In some embodiments, the LLM-based prompt-response system 25 is operable to at least partially refine the user input 42 in view of the usage information 44. The LLM-based prompt-response system 25 may then process the generated query 46 (which was generated using the at least partially refined user input 42) as described elsewhere herein (e.g., to interpret the query 46).

[0593] In some embodiments, generating the query 46 comprises augmenting the user input 42 and / or the usage information 44 with additional information instructing the LLM-based prompt-response tool 25 on how to respond. For example, the additional information may include information for causing generation of content in the output of the LLM-based promptresponse tool 25 to cause an API call for communicating with an application program (e.g., to execute a query in response to the output of the LLM-based prompt response system 25, such as to query a database of license plate reads (“LPR”s), to query a felon database, etc.).

[0594] In some embodiments, generating the query 46 comprises concatenating the user input 42 with the usage information 44. For example, with reference to FIG. 22A, the query 46 may be like example query 46 A which comprises a user input portion 50A which corresponds to the user input 42 and a usage information portion 50B which corresponds to the usage information 44. One or both of the user input 42 and the usage information 44 may be processed or refined at least in part pre or post concatenation. For example, additional information may be added to the user input 42 based on the usage information 44 prior to concatenating the user input 42 and the usage information 44 together into the query 46.

[0595] The query 46 may be segmented into, or may include, a plurality of segments. The query 46 may include at least a user input segment (or user query segment) which includes input for the LLM-based prompt-response system 25 provided by a user. The query 46 mayalso include a system segment (or a system prompt segment) which includes instructions, prompts, etc. for the LLM-based prompt-response system 25 intended to set one or more characteristics of the LLM-based prompt-response system 25 such as output format, search parameters, a behavior or personality of the LLM-based prompt-response system 25, etc. The user input 42 may be provided to the LLM-based prompt-response system 25 as part of the user input segment. The usage information 44 may be provided to the LLM-based prompt-response system 25 as part of the system segment. By segmenting the query 46 such that the user input 42 is separate from the usage information 44, risk of the usage information 44 being leaked to the user (such as by the LLM-based prompt-response system 25 leaking a portion of the usage information 44 in it’s output, for example) is reduced or eliminated.

[0596] In some embodiments, the different segments of the query 46 are provided to the LLM-based prompt-response system 25 together (such as in a string of characters including both the user input segment and the system segment, for example). In some embodiments, the different segments of the query 46 are provided to the LLM-based prompt response system 25 separately (such as the user input segment being provided to the LLM-based prompt response system 25 as a string of characters and the system segment being provided to the LLM-based prompt response system 25 as a separate fde).

[0597] As described above, generating the query 46 may comprise concatenating the user input 42 with the usage information 44. Concatenating the user input 42 with the usage information 44 may comprise populating the user input segment of the query 46 with the user input 42, populating the system segment of the query 46 with the usage information 44 and concatenating the user input segment and the system segment into the query 46.

[0598] In some embodiments, generating the query 46 comprises at least in part structuring the user input 42 into structured user input which is in a format compatible with the LLM- based prompt-response system. Structuring the user input 42 may at least in part include adding the user input 42 to a user input segment of the query 46. Additionally, or alternatively, generating the query 46 may comprise at least in part structuring the usage information 44 into structured usage information which is in a format compatible with the LLM-based promptresponse system. Structuring the usage information 44 may at least in part include adding the usage information 44 to a system segment of the query 46.

[0599] In some embodiments, generating the query 46 comprises rephrasing the user input 42 and the usage information 44 into a single query. For example, with reference to FIG. 22B, the query 46 may be like example query 46B which comprises a rephrased query for the LLM- based prompt response system 25. In the rephrased query 46B, the user input 42 may not bereadily separable from the usage information 44. In some embodiments, the content of both the user input 42 and usage information 44 are analyzed and are combined or rephrased into a single query. Alternatively, in some embodiments the rephrased query may comprise a plurality of segments as described above (such as a user input segment and a system segment, for example).

[0600] Generating the query 46 may at least in part comprise ascertaining a meaning of the user input 42 in combination with the usage information 44. The meaning may be ascertained in many ways or combinations thereof.

[0601] For example, capabilities of the application program may be ascertained at least in part by processing the usage information 44. Characteristics of the application program such as the version of the application program, one or more modules (e.g., a license plate recognition module, a facial recognition module, etc.) associated with the application program, one or more actively run modules, one or more modules run in the past, one or more active subscriptions (e.g., what services of the application program does the user have access to or would be interested in (e.g., if the user is monitoring an airport can provide airport related data but if the user is monitoring a coffee shop should not provide airport related data)), etc. may at least in part define the capabilities of the application program. The meaning of the user input 42 may be constrained to be a meaning within the capabilities of the application program (e.g., if the application program cannot determine an owner of a car, then the user input 42 cannot be interpreted to mean that the user would like to determine an owner of a car). Associations between the user input 42 and the usage information 44 may be drawn based on the capabilities of the application program.

[0602] Additionally, or alternatively, keywords in the user input 42 may at least in part be used to ascertain the meaning of the user input 42. For example, if the user input 42 includes the word “car”, it may be inferred that the user input 42 requires the use of a license plate reading module to determine an owner of a car or to locate a specific car within an area. As another example, if the user input 42 includes the word “lady”, it may be inferred that the user input 42 requires the use of a facial recognition module.

[0603] Additionally, or alternatively, an output of the application program may be used to at least in part determine a meaning of the user input 42. As described above, the usage information 44 may include an output of the application program. The output may be processed to determine what the output displays. For example, the output of the application process may be input into an image recognition process. The image recognition process may determine that the image displays a car. If the user input 42 is “who is that”, it may be ascertained fromknowing that the output the user saw was a car, that the user is interested in knowing who is the owner of that displayed car. The user input 42 and the usage information 44 may be refined such that the query 46 indicates that the user is interested in finding out who owns the car depicted in the output of the application program and that a license plate recognition module should be used to determine the owner of the displayed car.

[0604] As described above, the application program is associated with the LLM-based prompt-response system 25. Although the LLM-based prompt-response system 25 may assist a user to interact with the application program and / or to obtain further information or data about something the user is interested in, the application program may be used without the LLM- based prompt-response system 25. In other words, the application program has at least one intended or primary purpose that is separate or different from facilitating user interaction with the LLM-based prompt-response system 25. For example, the application program may be intended to be used to monitor and / or control access to an intended area. Such application program may be used to achieve its intended purpose of monitoring and / or controlling access to the intended area with or without the LLM-based prompt-response system 25.

[0605] In some embodiments, the processor circuit 16 is caused to run at least one machine learning model to at least in part generate the query 46. For example, the at least one machine learning model may receive as input the user input 42 and the usage information 44 and may output the query 46.

[0606] In some embodiments, generating the query 46 comprises removing unnecessary information from the user input 42 and / or the usage information 44. Removing unnecessary information may, for example, improve computational efficiency and reduce an amount of computational power that is required.

[0607] In some embodiments, generating the query 46 may comprise adding characters (such as from the user input 42 and / or the usage information 44, for example), removing characters (such as from the user input 42 and / or the usage information 44, for example), replacing one or more words (such as from the user input 42 and / or the usage information 44, for example), replacing one or more instructions, etc.

[0608] In some embodiments, the query 46 is, or includes, a string of characters defining a series of words (e.g., one or more complete sentences) which define a meaning that can be determined and analyzed by the LLM-based prompt-response system 25 for the purposes of the LLM-based prompt-response system 25 outputting an output (or answer) that corresponds to the user input 42 (or a query posed by the user of the application program).

[0609] In some cases, the user input 42 includes commands for causing the generation ofcontent for producing API call instructions, where the API can cause the carrying out of actions (e.g., to retrieve information) of an application program, associated plugins or related programs, or even external applications that the LLM-based prompt-response system 25 determines as being required to answer the user input 42. In some cases, an API may be used to optionally translate at least in part the user input 42 such that the LLM-based promptresponse system 25 can process the query 46.

[0610] In some cases, the user input 42 includes restrictions and / or requirements on a formation of an output of the LLM-based prompt-response system 25. The generated query 46 may include such restrictions and / or requirements such that an output of the LLM-based prompt-response system 25 is in a format that corresponds to the restrictions and / or requirements.

[0611] In some embodiments, the user input 42 may be analyzed to determine if an API call is needed. In some embodiments, programmatic logic may be used to analyze the user input 42 to determine if an API call is needed. If an API call is needed, then the query 46 may be adapted to include instructions to generate content for causing an API call (e.g., for communicating with or instructing software).

[0612] In some embodiments, generating the query 46 comprises weighing which of the usage information 44 is included in the query 46. The weighing or inclusion of the usage information 44 when generating the query 46 permits an addition into the query 46 of additional meaning to improve the LLM-based prompt response system 25 ’s ability to generate an answer that is tailored to the circumstances surrounding the user input 42 which the user may not have included in their user input.

[0613] If it is determined that the user input 42 is insufficient or inadequate (e.g., user input 42 cannot be answered within the system capabilities, the user input 42 is illegible, etc.), the user may be prompted to provide additional user input 42 or revise the user input 42. The user may, for example, be prompted to provide additional user input 42 or revise the user input 42 as part of block 41 and / or 45.

[0614] Referring back to FIG. 21, once the query 46 is generated, the blocks 40 may continue at a block 47 including program code that, when executed by the CPU 17, cause the processor circuit 16 to provide the generated query 46 to the LLM-based prompt-response system 25. In some embodiments, an API for communicating with the LLM-based promptresponse system 25 is used to provide the query 46 to the LLM-based prompt-response system 25.

[0615] In some embodiments, the blocks 40 end or terminate after the block 47.

[0616] In some embodiments, the blocks 40 may continue at a block 48 including program code that, when executed by the CPU 17, cause the processor circuit 16 to process an output 49 generated by the LLM-based prompt-response system 25. The LLM-based prompt-response system 25 may generate the output 49 in response to receiving the query 46. The LLM-based prompt-response system 25 may transmit the output 49 to, for example, the user computing apparatus 15 which may be caused to receive the output 49. In some embodiments, processing the output 49 comprises causing the output 49 to be at least in part displayed for the user (e.g., using the output device 21). The output 49 may include an API call or content for generating an API call. Processing the output 49 may comprise causing the processor circuit 16 to run at least one API call included in the output 49 (or to generate at least on API call for which content was included in the output 49).

[0617] In some embodiments, the output 49 of the LLM-based prompt-response system 25 is, or includes, a string of characters (such as a set of words to be displayed in a chat response). In some embodiments, the output 49 of the LLM-based prompt-response system 25 may be, or may be adapted to be, instructions to generate commands (when processed (e.g., by logic)) for causing an action in the application program (e.g., for controlling the application program).

[0618] An application program interface (API) may be used, where the API is configured to receive the output 49 of the LLM-based prompt-response system 25, or to receive an API call generated from content found in the output 49, and convert the output 49 into commands in a language suitable for execution by the application program carrying out the commands (such as querying a database, for example) thereby causing the application program to carry out one or more actions corresponding to the commands.

[0619] As discussed above, the user input 42 may include instructions to cause the LLM- based prompt-response tool 25 to generate content for producing an API call in order to answer the user input 42 as output. The user input instructions may also include instructions to generate the content for producing an API call in a specific format (such as JSON schema). The received content for producing an API call received at least in part as the output 49 of the LLM-based prompt-response tool can then be processed by the application program to cause an API call (e.g., to query a database).

[0620] In some embodiments, the API call received as at least part of the output 49 may cause the application program to perform an action. In some embodiments, the API call may prompt a response from the application program. The API may then receive the response from the application program and arrange the response in a defined manner. In some embodiments, a response from the API may be transmitted back to the LLM-based prompt-response system25, where the LLM-based prompt-response system 25 may generate a further output response with the information received from the API (e.g., in the form of a text chat, an audio message as a voicebot, etc.). For example, in response to the user input 42, as an initial query, “who is the owner of that car?” (where a licence plate is displayed in an image viewed in a window of the application program opened by the user), the response can be “Marie Tremblay”, the outputted response provided in a chatbot or voicebot.

[0621] In some embodiments, the output 49 generated by the LLM-based prompt-response system 25 may cause an answer that is a combination of information presented in a chatbot and / or voicebot, with actions performed by the application program, such as the display of a video or other information, moving a digital map, etc. For instance, a piece of identity for Marie Tremblay may be displayed on the screen, along with related information on Marie Tremblay.

[0622] In another example, in response to the user input 42 “who came through this door today”, the answer generated from the LLM output may be a string of characters appearing in a chat, formulating “100 people came through the door today, most recently Marie Tremblay and Pierre Grolet. Please see your screen for the full list of names.”. The output 49 (e.g., through the use of an API call) may then cause the application program to display (such as using the output device 21, for example) the list of names on a window of the application program.

[0623] Since the LLM-based prompt-response system 25 processes the query 46 which is a combination of the user input 42 and the usage information 44, the LLM-based promptresponse system 25 can generate a response to the user’s input more efficiently. Without the additional contextual information provided by the usage information 44, the LLM-based prompt-response system 25 may need to run significantly more searches or queries to generate the output that the user is looking for. Improving efficiency also reduces the required computational power. Additionally, or alternatively, without the contextual information provided by the usage information 44, it may be technically impossible for the LLM-based prompt-response system 25 to generate the output that the user is asking for (e.g., since the user input may be meaningless without context). By generating the query 46 as described herein and providing the query 46 to the LLM-based prompt-response system 25, the LLM-based prompt-response system can generate the output that the user is asking for.

[0624] In some embodiments, the output 49 is, or includes, one or more suggested upgrades which the user may make to add new software modules or features to increase the capabilities of the security system 12. The suggested upgrades may alert the user to new potentially useful software features or software features which may better tailor the output 49 to what the user islooking for. Additionally, or alternatively, the suggested upgrades may be, or include, suggested upgrades and / or modifications to one or more hardware devices (such as one or more of the devices 14, for example) in communication with the security system 12. Likewise, the suggested upgrades may alert the user to new potentially useful hardware devices which may better tailor the output 49 or the security system 12 more generally to what the user is looking for. The suggested upgrades may, for example, be in response to the LLM-based promptresponse system 25 being able to provide a suitable response to a user input 42.

[0625] In some embodiments, if a user does not have the required access privileges for the LLM-based prompt-response system 25 to provide a response that the user is looking for, the output 49 of the LLM-based prompt-response system may provide one or more proposed solutions. For example, the output 49 may include a prompt to contact a person who has the required access privileges such as an administrator or more specifically a particular name of an administrator such as “John Fitzwell”. The output 49 may include contact information for the administrator. Additionally, or alternatively, the output 49 may include an API call which initiates communication (e.g., a call, video call, email, text message, etc.) with the administrator.

[0626] In some embodiments, the LLM-based prompt-response system 25 detects misuse. For example, the LLM-based prompt-response system 25 may detect that a user is repeating actions causing them to go around in a circle based at least partially on the usage information 44. The output 49 of the LLM-based prompt-response system 25 may include a suggested solution or manner of using the application program to achieve the result that the LLM-based prompt-response system 25 thinks the user would like to achieve. For example, the output 49 may include a hyperlink intended to take the user to a location in the application program that the LLM-based prompt-response system 25 thinks the user wanted to reach. As another example, the output 49 may include instructions for the user on how to reach a location in the application program that the LLM-based prompt-response system 25 thinks the user wanted to reach. As discussed above, the query 46 may include a system segment (or system prompt segment). In some embodiments, the system segment comprises a list of different places (e.g., pages, panes, etc.) in the application program that are suited for different user intentions (e.g., “change camera frame rate”, “add new cardholder”, etc.) as well as hyperlinks to these places or a navigation guide for how to reach them.

[0627] The LLM-based prompt-response system 25 may be a large language model (LLM) artificial intelligence model that has been trained from a large dataset of texts (such as texts sourced from media-hosting platforms). In some embodiments, the LLM-based prompt-response system 25 may be, or include, a Generative Pre-Trained Transformer (GPT) model. For example, a GPT model may include millions or billions of parameters trained on vast amounts of data (e.g. gigabytes or terabytes of text). A GPT is, or includes, a kind of neural network that uses a transformer architecture to leam from large amounts of text data. The model may have two main components: an encoder and a decoder. The encoder processes the input text and converts the text into a sequence of vectors, named embeddings, representing the meaning and context of each word. The decoder generates the outputted text by predicting the next word in the sequence of words, based on the embedding and the previous words. The model may use a technique called attention to focus on the most relevant parts of the input and outputted texts, and to capture long-range dependencies and relationships between words. The model may be trained using a large corpus of text for both the input and the output, and by aiming to minimize over time the differences between the predicted words and the actual words. The model may be refined for specific domains or tasks by using smaller and / or more specific datasets. Other types of LLMs include, but are not limited to, a Transformer-Based Model, a Bidirectional Encoder Representations from Transformers (BERT) model, a Unified Language Model, a Text-to-Text Transfer Transformer (T5) model, etc.

[0628] In some embodiments, the LLM may include a neural network that may be a computational network of a system of artificial neurons, arranged in a plurality of lawyers or nodes. Each Layer of the plurality of layers may include an input layer, one or more hidden layers, and an output layer. Each layer of the plurality of layers may include one or more nodes (or artificial neurons). Outputs of all nodes in the input layer may be coupled to at least one node of a hidden layer (or layers). Inputs of each hidden layer may be coupled to outputs of at least one node in other layers of the neural network. Outputs of each hidden layer may be coupled to inputs of at least one node in other nodes of the neural network. A node (or nodes) in the final layer may receive inputs from one or more hidden layers to output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the corresponding neural network model. Such hyperparameters may be set before or while training the corresponding neural network model on a training dataset.

[0629] The neural network may correspond to a mathematical function (such as a sigmoid function or rectified linear function, for example) with a specific set of parameters. These parameters may be tuned during training of the network. The set of parameters may include, for example, a weight parameter, a regularization parameter, etc. Each node may use the mathematical function to compute the output based on one or more inputs, e.g. from nodes of other layers. Some or all of the nodes may correspond to the same, or a distinct, mathematicalfunction.

[0630] While training the neural network, one or more parameters of each node may be modified based on whether an output of the final layer for a given input (e.g., from the training dataset) matches an expected result for the neural network. The training process may be repeated until the neural network has achieved an acceptable margin of error or loss. Exemplary training methods include, but are not limited to, gradient descent, stochastic gradient descent, batch gradient descent, meta-heuristics, etc.

[0631] Each neural network may include electronic data, such as one or more software programs, data libraries, scripts or other logic for execution by a processor. Each of the neural networks may include code and routing configured to enable a system, such as the LLM-based prompt-response system 25, to perform one or more operations. In some embodiments, the neural network may be implemented via hardware which includes a processor, a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), etc., or a combination of software and of hardware, to execute one or more operations.

[0632] A deep neural network with a plurality of hidden layers between the input lawyer and the output layer may be the most representative type of artificial neural network (or ANN) which enables deep learning, which is one machine learning technique.

[0633] A machine learning model (e.g., an ANN) may be trained by using a training dataset. Herein, the training may mean a process of determining a parameter of the ANN in order to achieve the objects such as classification, regression, clustering, etc. of input data. As a representative example of the parameters of the ANN, there can be a weight given to a synapse of a bias applied ...

Claims

What is claimed is:

1. A method of generating by an application program an LLM query for transmission to an LLM tool to cause the LLM tool to generate a structured data source query for causing a searching of a data source of security records, comprising: obtaining an input query, the input query comprising natural language; accessing a computer-readable memory storing therein a system prompt template library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least one but not all of the system prompt template elements in the system prompt template library; processing the subset of system prompt template elements to generate the system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query; generating the LLM query comprising the received input query and the system prompt; and transmitting the generated LLM query to the LLM tool to cause the LLM tool to generate the structured data source query for causing a searching of a data source of security records based on the received LLM query.

2. The method according to claim 1, further comprising: receiving the structured data source query in response to the LLM query; and searching the data source of security records according to the structured data source query.

3. The method according to claim 1 or claim 2, wherein the one or more system prompt template elements in the subset of system prompt template elements comprise usage logic comprising instructions defining when to use the respective system prompt template elements, wherein determining a subset of system prompt template elements comprises interpreting the usage logic for respective system prompt elements in the system prompt template library to determine whether to use the respective system prompt template elements to generate the system prompt.

4. The method according to claim 3, further comprising analysing the input query to determine an input query type; wherein interpreting the usage logic comprises determining on the basis of the usage logic if the determined input query type corresponds to an input query type of the respective system prompt element defined in the usage logic of the respective systemprompt element.

5. The method according to any one of claims 1 to 4, wherein the LLM query includes chat history of the LLM comprising one or more previous LLM queries, and one or more structured data source queries generated by the LLM tool in respective to the respective one or more previous LLM queries.

6. The method according to any one of claims 1 to 5, further comprising further updating the system prompt template library.

7. The method according to claim 6, wherein the updating is performed following a receipt of an error message received from the searched data source following a receipt by the searched data source of the structured data source query.

8. The method according to claim 6, wherein the updating is performed following a software update of the searched data source, or of a system related to the searched data source.

9. The method according to claim 6, wherein the updating is performed following a passing of a predetermined amount of time.

10. The method according to any one of claims 5 to 9, wherein the updating comprises changing a pointer location in memory of one or more of system prompt template elements of the system prompt template element library.

11. The method according to any one of claims 1 to 10, wherein the searched data source is related to a video monitoring system.

12. The method according to claim 11, wherein the security records are image streams including metadata generated following an analysis of the corresponding image stream by a camera that generated the corresponding image stream.

13. The method according to any one of claims 1 to 12, wherein the system prompt template element library includes a system prompt template element with a variable field that is configured to be adapted in accordance with the determined input query type of the received input query.

14. The method according to any one of claims 1 to 13, wherein the determining of a subset of system prompt template elements comprises determining an input query type of the received input query.

15. The method according to any one of claims 1 to 14, further comprising obtaining context data of the application program.

16. The method according to claim 15, wherein the system prompt template elements library comprises at least one system prompt template element related to the context data, wherein processing the subset of system prompt template elements to generate a system promptcomprises processing the at least one system prompt template element to generate a portion of a system prompt instructions on how to use at least a portion of the context data.

17. The method according to claim 15, wherein the usage logic of at least one system prompt template element of the subset of system prompt template elements comprises logic indicating whether to use a respective system prompt template based on the context data.

18. The method according to claim 15, wherein determining a subset of system prompt template elements comprises interpreting the usage logic of the at least one system prompt template element in light of the context data and selecting the at least one system prompt template element as a result of this interpreting.

19. The method according to any one of claims 15 to 18, wherein the context query data is generated from context information surrounding a usage by the user of the application program.

20. The method according to any one of claims 1 to 19, wherein the library of system prompt template elements comprises at least a first set of system prompt template elements comprising instructions to the LLM on how to generate a search query, and at least a second set of system prompt template elements not related to searching comprising instructions to the LLM on how to perform a non-searching function.

21. The method according to claim 20, wherein the second set of system prompt template elements is related to device configuration and comprises instructions instructing the LLM on how to create a device configuration command for a device external to the application program.

22. The method according to claim 21, wherein the subset of system prompt template elements comprises the second set of system prompt template elements, the method further comprising receiving the device configuration command and transmitting the device configuration command to the device.

23. The method according to claim 20, wherein the second set of system prompt template elements is related to an application program configuration and comprises instructions instructing the LLM on how to create an application program configuration command instructing the application program to change configuration.

24. The method according to claim 23, wherein the subset of system prompt template elements comprises the second set of system prompt template elements, the method further comprising receiving the application program configuration command and configuring the application program accordingly.

25. The method according to claim 23, wherein the change of configuration including at least a modification to a graphical user interface (GUI) of the application program, and wherein the subset of system prompt template elements comprises the second set of system prompttemplate elements, the method further comprising receiving the application program configuration command, modifying the GUI accordingly, and causing the displaying of the modified GUI.

26. A method of upgrading a security application program having a search interface for searching through a data source of security records, the method comprising: accessing a computer-readable memory storing therein an original system prompt library comprising a plurality of system prompt template elements each comprising data for generating at least a portion of a system prompt; and modifying at least a portion of the system prompt template element library to generate a modified system prompt template element library to change the system prompts generated using the security application program, the modified system prompt template element library comprising at least a modified portion of the original system prompt template element library.

27. The method according to claim 26, further comprising after modifying the system prompt template element library: obtaining an input query; accessing the computer-readable memory to access the modified system prompt library; determining a subset of system prompt template elements required to generate a system prompt to accompany the query, the subset comprising at least a part of the modified portion of the original system prompt template element library; processing the subset of system prompt template elements to generate a system prompt, the system prompt comprising instructions to the LLM on how to respond to an LLM query, the system prompt being different from the system prompt that would have been generated using the original system prompt template element library; generating the LLM query comprising the received input query and the system prompt; and transmitting the generated LLM query to the LLM tool.

28. The method according to claim 26 or claim 27, wherein modifying the system prompt template element library comprises modifying at least a first system prompt template element of the library to change the system prompts that would be generated using the first system prompt template element.

29. The method according to claim 26 or claim 27, wherein modifying the system prompt template element library comprises adding at least one new system prompt template element to the library to permit new system prompt text to be generated.

30. The method according to claim 26 or claim 27, wherein modifying the system prompt template element library comprises deleting at least one system prompt template element from the library to prevent its respective system prompt text from being generated.

31. The method according to claim 26 or claim 27, wherein the LLM tool is a tool to generate the structured data source query for causing a searching of a data source of surveillance records based on the received LLM query, the method further comprising: receiving the structured data source query in response to the LLM query; and searching the data source of surveillance records according to the structured data source query.

32. The method according to claim 26 or claim 27, wherein the modified portion of the original system prompt template element library comprises system prompt instructions for searching for new kinds of video metadata not supported in the original system prompt template element library.

33. The method according to any one of claims 26 to 32, executed while the security application program is running, wherein the modifying at least a portion of the system prompt template element library is performed during normal operation of the security application program such that security operations performed by the security application program do not need to be interrupted for the upgrading.

34. The method according to claim 33, wherein the security application program comprises a pointer to the memory location in the computer-readable memory of the original system prompt template element library, and wherein the modifying at least a portion of the system prompt template element library comprises: storing at a new location in the computer-readable memory as new system prompt template element library, the modified system prompt template element library, comprising the modified portion of the original system prompt template element library and a copy of any unmodified portion of the original system prompt template element library; and changing the pointer to the new location in the computer readable memory.

Images