Method, program, apparatus for handling user knowledge base queries

Abstract

Embodiments include a system comprising: an interface to receive a live knowledge base query; a retrieval augmented generative (RAG) Al component operable to process input knowledge base queries by: a) retrieving, from accessible knowledge base documentation, information relevant to the input query; b) creating a prompt comprising the retrieved information and the input query and submitting the prompt to a ELM; c) outputting a response from the ELM as an answer to the input query; the software service being operable in a live mode to process a query with accessible knowledge base documentation being restricted to the live version; and in a development mode to receive one or more pending edits to a development version of the knowledge base documentation, to receive a simulated knowledge base query, and to process a simulated query with accessible knowledge base documentation being restricted to the development version, incorporating the pending edits.

Description

[0001] Method, Program, Apparatus for handling user knowledge base queries

[0002] FIELD

[0003] Embodiments are in the field of software service user help provision. In particular, embodiments relate to automated generation of responses to input help requests.

[0004] BACKGROUND

[0005] End users of a software service may seek guidance on how to properly utilize the functionality of the software service, or how to access one or more functions or settings, by submitting a help request to a help or search interface of the software service or equivalent.

[0006] At the software service provider, the help requests are processed either manually by customer service operatives, or automatically by identifying articles from a knowledge base relevant to the help request and pointing to the identified articles in an automated response. In the case of a search query, an automated searching algorithm may encode the query, compare the encoded query with an indexed version of the software system and one or more knowledge bases, and return links to the most relevant elements in a response to the search query.

[0007] Manually processing help requests is desirable since a manual operator may be capable of better understanding a help request and providing a more tailored response than may be generated by an automated system. However this incurs a cost both in terms of manual effort and in terms of a latency period between help request and reply.

[0008] Automatically processing help requests is desirable since it provides almost instantaneous response to an end user and does not incur manual effort cost at the software service provider. However the response may not meet the precise needs of the end user and may lead to end user frustration.

[0009] It is desirable to address one or more shortcomings in the present end user knowledge base query handling techniques.

[0010] SUMMARY

[0011] In one aspect: a system operable to provide a software service for a plurality of end users, the system comprising: an end user interface operable, at runtime, to receive a live knowledge base query from an end user; knowledge base documentation; a retrieval augmented generative (RAG) Al component operable to process input knowledge base queries by: a) retrieving, from accessible knowledge base documentation, information relevant to the input knowledge base query; b) creating a prompt comprising the retrieved information and the input knowledge base query and submitting the prompt to a LLM; c) receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the input knowledge base query.

[0012] The knowledge base documentation is stored as a live version of the knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time. The software service is operable in a live mode to receive the live knowledge base query via the end user interface, and to input the live knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the live version at step a). The software service is operable in a development mode running concurrently with the live mode, to receive one or more pending edits to the development version of the knowledge base documentation via the developer interface, to receive a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface, and to input the simulated knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the edited development version at step a).

[0013] Optionally, upon approval by the developer, the one or more pending edits to the development version are replicated in the live version.

[0014] Optionally, upon rejection by the developer, the one or more pending edits in the development version are discarded.

[0015] Optionally, the prompt further comprises task specific instructions, comprising one or more from among: a) generic task specific instructions, wherein the generic task specific instructions are generic to all input knowledge base queries; and b) selected task specific instructions from a set of classification-based task specific instructions, being selectable according to a classification of the live knowledge base query, and are included in the prompt as task specific instruction for the LLM in generating the response, the task specific instructions being a rule, constraint, or other instruction to the LLM.

[0016] Optionally, the system comprises a live version of the task specific instructions and a configurable development version of the task specific instructions, wherein the RAG Al component is configured: in the live mode to include the live version of the task specific instructions in the prompt, and in the development mode to accept one or more edits to the configurable development version of the task specific instructions from a developer, to include the edited configurable development version of the task specific instructions in the prompt, and upon approval from the developer, to replicate the edited configurable development version of the task specific instructions in the live version of the task specific instructions.

[0017] Optionally, the developer interface is operable to provide an interface to access and make pending edits to the development version of the knowledge base documentation and / or the edit the task specific instructions, to review the answer to the simulation knowledge base query output by the RAG Al component at step c), and to select whether to approve the pending edits to the development version of the knowledge base documentation and / or the task specific instructions for replication in the respective live versions.

[0018] Optionally, the software system comprises a user interface, the user interface being configured to receive text or voice data input by a user, the text data representing a live knowledge base query, and to submit the live knowledge base query for processing by the RAG Al component as an input knowledge base query.

[0019] Optionally, the user interface displays the response output by the RAG Al component as the answer to the input knowledge base query.

[0020] Optionally, the user interface provides a classification input field, to receive a classification of the live knowledge base query, and to submit the classification to the RAG Al component in association with the live knowledge base query for filtering the live version of the knowledge base documentation at information retrieval step a), the classification being a label, descriptor, rule, constraint, or other information.

[0021] Optionally, the RAG Al component is operable to automatically assign a classification to an input knowledge base query, based on a tool, web page, application page, or interface, of the software service from which the input knowledge base query originates, or based on a classification of the input knowledge base query by a classification algorithm, and to submit the classification to the RAG Al component in association with the input knowledge base query for filtering the live version of the knowledge base documentation at information retrieval step a), the classification being a label, descriptor, rule, constraint, or other information.

[0022] Optionally, the user interface is directly accessible by at least one of an operative of the software system provider and an end user. Optionally, the simulation knowledge base query is an historical live knowledge base query previously submitted to the system by an end user.

[0023] Optionally, the RAG Al component includes a pre-trained information retrieval algorithm, and the RAG Al component is configured to perform retrieving information at step a) by executing the pre -trained information retrieval algorithm.

[0024] Optionally, the development version of the knowledge base documentation to which pending edits are made by a developer is a restricted-access version of the knowledge base documentation being a replication of the live version of the knowledge base documentation at a point in time from which one or more documentation elements are removed and / or superseded by replacement documentation elements comprising pending edits, the development version being accessible to the RAG Al component only in processing a simulation knowledge base query input by the said developer, the documentation elements each being an article, page, or document.

[0025] Optionally, upon a pending edit being made to a development version of the knowledge base documentation by a developer via the development interface, an unedited form of the knowledge base documentation remains accessible to the RAG Al component in processing live knowledge base queries, the unedited form of the knowledge base documentation is inaccessible to the RAG Al component in processing a simulation knowledge base query from the same developer, and the edited form of the knowledge base documentation is accessible to the RAG Al component only in processing a simulation knowledge base query input by the said same developer.

[0026] Optionally, the live version of the knowledge base documentation is stored as a replicated document repository comprising plural replicas stored on distinct server apparatus, each replica being operable to be accessed simultaneously by the RAG Al component in processing multiple input queries in parallel.

[0027] Optionally, pending edits to the knowledge base documentation, upon approval at the developer interface, are committed to the replicated database by a replicated database update procedure; and during the replicated database update procedure, the developer interface blocks any new pending edits from being made.

[0028] In another aspect: a computer-implemented method in a system providing a software service for a plurality of end users, the computer-implemented method comprising: storing a live version of knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time. The method further comprising, in a live mode: at an end user interface, at runtime, receiving a live knowledge base query from an end user; at a retrieval augmented generative (RAG) Al component processing the live knowledge base query by: retrieving, from the live version of the knowledge base documentation, information relevant to the live knowledge base query; creating a prompt comprising the retrieved information and the live knowledge base query and submitting the prompt to a LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the live knowledge base query. The method further comprising, in a development mode running concurrently with the live mode: at the developer interface, receiving one or more edits to the development version of the knowledge base documentation via the developer interface, and receiving a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface. The method further comprising, at the retrieval augmented generative (RAG) Al component processing the simulated knowledge base query by: retrieving, from the edited development version of the knowledge base documentation, information relevant to the simulated knowledge base query; creating a prompt comprising the retrieved information and the simulated knowledge base query and submitting the prompt to the LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the simulated knowledge base query.

[0029] In another aspect: a computer program comprising processing instructions which, when executed by a computing apparatus comprising memory hardware and processor hardware, cause the computing apparatus to perform a method comprising: storing a live version of knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time; in a live mode: at an end user interface, at runtime, receiving a live knowledge base query from an end user; at a retrieval augmented generative (RAG) Al component processing the live knowledge base query by: retrieving, from the live version of the knowledge base documentation, information relevant to the live knowledge base query; creating a prompt comprising the retrieved information and the live knowledge base query and submitting the prompt to a LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the live knowledge base query; in a development mode running concurrently with the live mode: at the developer interface, receiving one or more edits to the development version of the knowledge base documentation via the developer interface, and receiving a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface, at the retrieval augmented generative (RAG) Al component processing the simulated knowledge base query by: retrieving, from the edited development version of the knowledge base documentation, information relevant to the simulated knowledge base query; creating a prompt comprising the retrieved information and the simulated knowledge base query and submitting the prompt to the LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the simulated knowledge base query. Embodiments include a system comprising: an interface to receive a live knowledge base query; a retrieval augmented generative (RAG) Al component operable to process input knowledge base queries by: a) retrieving, from accessible knowledge base documentation, information relevant to the input query; b) creating a prompt comprising the retrieved information and the input query and submitting the prompt to a LLM; c) outputting a response from the LLM as an answer to the input query; the software service being operable in a live mode to process a query with accessible knowledge base documentation being restricted to the live version; and in a development mode to receive one or more pending edits to a development version of the knowledge base documentation, to receive a simulated knowledge base query, and to process a simulated query with accessible knowledge base documentation being restricted to the development version, incorporating the pending edits.

[0030] DETAILED DESCRIPTION

[0031] Embodiments are set out, by way of example, in the following detailed description. Reference is made to the following illustrations of embodiments:

[0032] Figure 1 A illustrates a part of a software system architecture;

[0033] Figure IB illustrates a part of a software system architecture;

[0034] Figure 2 illustrates a process;

[0035] Figure 3A illustrates a process;

[0036] Figure 3B illustrates a process;

[0037] Figure 3C illustrates a process;

[0038] Figure 4 illustrates a process;

[0039] Figure 5 illustrates a process;

[0040] Figure 6 illustrates a hardware arrangement; and

[0041] Figure 7 illustrates an accounting system.

[0042] Embodiments include a system operable to provide a software service for a plurality of end users. The system may comprise: an interface to receive a live knowledge base query from an end user; and a retrieval augmented generative (RAG) Al component operable to process input knowledge base queries.

[0043] The RAG Al component may be operable to process input queries by: a) retrieving, from accessible knowledge base documentation, information relevant to the input query; b) creating a prompt comprising the retrieved information and the input query and submitting the prompt to a LLM; and c) outputting a response from the LLM as an answer to the input query. The knowledge base documentation may be stored in a live version and a development version, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time.

[0044] The software service may be operable in a live mode to receive the live knowledge base query, and to input the live knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the live version at step a).

[0045] The software service may be operable in a development mode running concurrently with the live mode to receive one or more pending edits to the development version of the knowledge base documentation via the developer interface, to receive a simulated knowledge base query from the developer interface, and to input the simulated knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the development version, incorporating the pending edits, at step a).

[0046] Figure 1A is a schematic illustration of functionality performed by an RAG Al component 100 in a software system. It is noted that the term RAG Al component 100 is used broadly to define not only the specific RAG Al functionality but also the associated interfaces, data storage, and data processing functions. The user interface 200 and developer interface 300 may be considered to be elements of the RAG Al component 100 or may be considered to be elements forming part of the software system and external to the RAG Al component. The RAG Al component may be one or more from among: an RAG Al module, RAG Al service, RAG Al algorithm, RAG Al program, RAG Al system, RAG Al engine. The RAG Al component may be a mechanism for receiving a knowledge base query, creating an LLM prompt based on the received knowledge base query, receiving a response to the prompt from the LLM, and outputting a reply to the received knowledge base query based on the received response from the LLM.

[0047] It is noted that the term ‘software service’ is used to represent a data storage and / or data processing service provided by the software system to end users. The end users may be subscribers, for example, accessing the software service in exchange for a periodical or up-front payment. The overriding purpose of the RAG Al component 100 is to provide information, help, and guidance, to users of the software service regarding how to use the software service. The software service may be an online accounting service.

[0048] A knowledge base interface 110 provides the RAG Al component 100 with access to the knowledge base documentation. The knowledge base documentation (not illustrated) comprises content (i.e. articles, elements, documents) to which access is selectively controllable, including a live version which is accessible in processing live knowledge base queries (i.e. live means a knowledge base query received at runtime of the software service from an end user or from a customer service operative), and a configurable development version being an editable replica of an element of the live version for edit by a developer and which is accessible only in processing knowledge base queries from the said developer. The replica is understood to be a replica at the point in time at which the replica was made, noting that since the development version is editable, once an edit is made it ceases to be a precise replica. Nonetheless, the development version is a replica of the live version except for edited elements in the development version. The knowledge base documentation is, for example, a document repository such as a plurality of logically interconnected articles, the logical interconnections being expressed within the articles by links such as hyperlinks to one or more other of the articles. The knowledge base documentation may be a proprietary knowledge base belonging to the software service provider and comprising articles expressing information about the services provided for users by the software system. The knowledge base documentation may be freely accessible by users, for example via an element of a user interface of the software service. The knowledge base documentation comprises component elements, which could be articles, documents, text passages, etc.

[0049] A knowledge base query may also be referred to as a help query, or help request, for example. The term knowledge base query is applied because the knowledge base documentation is leveraged in composing a response by the LLM, in the form of an RAG Al model. The knowledge base query may be received as a text string, or as an audio file containing speech which is converted to text after receipt as part of a preprocessing step. The received knowledge base query may be parsed for downstream processing.

[0050] The knowledge base interface 110 may be, for example, an API. The knowledge base interface 110 is accessible to the RAG Al component 100, and may or may not be accessible to other elements of the broader software system. Hence, the knowledge base interface 110 is illustrated as a component of the RAG Al component 100.

[0051] The knowledge base documentation is a data store storing information which may be logically divided into articles, documents, or equivalent (the term ‘document’ will be used hereinafter), wherein one document generally relates to one topic, issue, or factor relating to the software service.

[0052] The knowledge base documentation is replicated for parallelization of information retrieval query handling of live queries. The redundancy provides backup in the case of failure of one or more replicas, and also improves capability to parallelise processing of live information retrieval queries. Each replica may be stored on a distinct data storage apparatus with respective distinct query handling interfaces so that the software service has increased bandwidth to process live queries. Furthermore, redundancy allows each replica to be taken offline and updated in turn whilst ensuring that at least one full copy of the knowledge base documentation remains online.

[0053] The live version of the knowledge base documentation 400a is one or more full copies of the knowledge base documentation, to which access is restricted. Access is restricted to the processing of live knowledge base queries. In some cases the live version of the knowledge base documentation may be accessed in the processing of simulated knowledge base queries. This is illustrated and explained in more detail with reference to Figure 5.

[0054] The configurable development version 400b of the knowledge base documentation may be a full replica of the knowledge base or a portion thereof, replicated for editing. The knowledge base documentation being edited may be duplicated one or more times so that each developer may work on their own replication of the knowledge base documentation, flagged so that it is utilized only in processing development knowledge base queries from the same developer, whilst an unedited (i.e. pre-edit) version of the knowledge base documentation remains live and is accessible to the information retrieval for all other knowledge base queries (i.e. live knowledge base queries).

[0055] The configurable development version of the knowledge base documentation is editable via the developer interface 300, via which a developer may make one or more edits 310 to a document being edited.

[0056] When a developer via the developer interface 300 makes edits 310 to a document of the knowledge base documentation, the pertinent development version (i.e. the copy assigned to the developer) of the documentation is edited, but the live version remains unedited. In the development version, the element (i.e. individual document) to which edits have been made replaces the unedited copy of the same element. The development version may be edited in other ways, such as by the deletion of an element altogether, or the addition of a new element. The edited development version is flagged as having a pending edit by the specific developer, that is, the flag includes a developer ID or equivalent, or in some other way the development version is recorded as being assigned to the specific developer. The flag is an access control. It allows the version of the knowledge base documentation to remain on the data storage server(s) where it is accessible by the information retrieval algorithm (in processing simulation knowledge base queries from the specific developer), but still allows the live knowledge base queries to be processed by access to the live (unedited) version. The unedited version of the documentation is flagged as being accessible to the information retrieval algorithm in processing live knowledge base queries. Therefore regardless of edits, there remains a full unedited replica of the documentation available for processing live knowledge base queries (i.e. in the information retrieval step 220). A simulation knowledge base query 320 received from the developer interface 300 instance associated with the specific developer identified in the flag is processed by the information retrieval step S202 using the development version of the knowledge base documentation assigned to the same specific developer, which is a replica of the live version from a point in time plus any edits made by the developer in the meantime. This is further discussed below with reference to Figure 5.

[0057] For example, an indexer may generate an index of the knowledge base documentation specific to each developer upon the developer making edits 310 to their respective development version of the documentation. The indexer may also generate an index of the unedited replica. The developer may be able to trigger indexing via the developer interface, or the indexing may be automated upon any edits to the documentation by the developer. The indexes may be used at the information retrieval step S202, with the specific index to be used in handling a knowledge base query dependent upon the source: a live knowledge base query 220 from a user is handled using the index of the live version of the knowledge base documentation, a simulation knowledge base query 320 from developer 1 is handled using an index of the development version of the knowledge base documentation assigned to developer 1. A simulation knowledge base query 320 from developer 2 is handled using an index of the development version of the knowledge base documentation assigned to developer 2. Edits made to the developer 1 knowledge base documentation but not yet committed to the live version (or committed to the live version after the replication of the live version forming the development version assigned to developer 2) are not visible to developer 2. The access by the information retrieval algorithm may be to an index representing an article rather than to the article itself.

[0058] The term ‘developer’ is used to refer to a human operative with responsibility for maintaining the knowledge base of the software service. The developer interface 300 is effectively a user interface accessible by logged-in users with appropriate permissions (namely particular employees, contractors, etc of the provider of the software service with a duty or responsibility to administrate the knowledge base) to configure the knowledge base documentation. The term developer is not intended to imply that the operative be a software developer. The term ‘developer’ is intended to be interpreted in relation to developing the ability of the RAG Al component to respond effectively and accurately to knowledge base queries from users.

[0059] The developer interface 300 may be configured to facilitate editing of the knowledge base documentation by subject matter experts, that is, individuals with expertise in topics covered by one or more articles or documents.

[0060] The developer interface 300 provides a mechanism for a developer to submit a development knowledge base query 320 to the RAG Al component 100 and also to review the answer 330 generated by the component 100. Furthermore, the developer interface 300 provides a mechanism for the developer to make edits 310 to a development version of the knowledge base documentation.

[0061] The cycle of submitting a simulation knowledge base query 320, reviewing the answer 330, making an edit 310 to the development version of the knowledge base documentation, and then re-running the same simulation knowledge base query 320 and comparing the answer 330 with the previous answer (i.e. pre-edit) may be repeated on an iterative basis. This iterative approach enables the developer to make one or a series of edits until they obtain a desired outcome in terms of a change in the answer 330 output by the RAG Al component.

[0062] Figure 2 illustrates a knowledge base query handling process executed by the RAG Al component. Figure 1A illustrates the interactions between the RAG Al component 100 and the associated user interface 200 and developer interface 300.

[0063] The RAG Al component includes an executable knowledge base query handling procedure S201-S206 which includes an RAG Al component. The process illustrated in Figure 2 is generic in terms of applying to both live knowledge base queries (i.e. received from users in real life use cases via the user interface) and development or simulation knowledge base queries (i.e. received from developers via the developer interface 300).

[0064] The knowledge base query handling process S201-S206 is performed for live knowledge base queries 220 received from a user of the RAG Al component 100 via the user interface 200, and development knowledge base queries 320 received from operatives of the software service provider via the developer interface 300. The generic process is the same in both cases, with the difference being in the version of the knowledge base documentation accessible in the information retrieval step at S202, as illustrated in Figure 1A.

[0065] The term user is employed broadly. The RAG Al component 100 may be implemented as a tool that is directly accessible to end users of the software service. In that case, the user interface 200 of the RAG Al component may be an element of the software service that is provided to end users, so that the live knowledge base query 220 is submitted to the RAG Al component 100 via the RAG Al component user interface 200 directly by an end user of the software system.

[0066] Alternatively, the user of the RAG Al component user interface 200 may be an operative of the software service provider such as a helpdesk operative, customer service provider, or any operative with a duty to handle, process, or otherwise respond to, knowledge base queries from end users. The user may have access to the RAG Al algorithm via a user interface from a customer service program or application distinct from the software service of the software service provider to which the knowledge base relates. That is, the user interface may be from a program distinct from the RAG Al component and the software service.

[0067] A knowledge base query may be received in a mailbox or via an end user interface or by some other method from an end user, the customer service operative of the software service provider (referred to as a customer service operative for ease of reference) then accesses the RAG Al component user interface 200 to submit the received knowledge base query to the RAG Al component 100. In this use case the user of the user interface 200 is the customer service operative. The end user knowledge base query may be considered to be submitted to the RAG Al component 100 indirectly since the customer service operative effective relays the knowledge base query to the RAG Al component 100.

[0068] At step S201 a knowledge base query is input to the RAG Al component 100. The knowledge base query may be a natural language knowledge base query, and may be input as speech (which is then converted to text by the interface) or text. The knowledge base query may be a question, or may be one or a series of words or terms. The RAG Al component 100 may be operable to handle live knowledge base queries 220 from users via the user interface 200 as illustrated by Figure 1A, and to handle simulation knowledge base queries 320 received from developers via the developer interface 300 also as illustrated in Figure 1A.

[0069] The RAG Al component 100 may comprise or be associated with a single interface such as an API which is called by the user interface 200 and developer interface 300 to facilitate information exchange, in particular the submission of knowledge base queries at S201 and receipt of responses at S206.

[0070] The RAG Al component 100 may parse and encode, or otherwise process, the input knowledge base query for use in an information retrieval query at step S202. An information retrieval mechanism of the RAG Al component 100 is executed, which may be a pre-trained Al information retrieval algorithm, or may be a set of rules for converting an input knowledge base query into an information retrieval query or queries for searching by accessing an accessible version of the knowledge base documentation (or an index thereof). In fact, the knowledge base may be chunked and indexed, or otherwise indexed, for information retrieval processing at S202. Different indexes are generated based on flags or other access control fields in the stored versions of the knowledge base documentation, as discussed below with reference to Figure 5. When new edits are committed to the knowledge base documentation it is updated (a process illustrated by the arrow between the two versions in Figure 1A), and re-indexed. Or, reindexing may be performed periodically. Similarly, new edits made to development versions of the knowledge base documentation specific to a particular developer may trigger a re-index so that the simulation knowledge base query from the particular developer is handled in a manner which reflects the edits 310 being trialled by the developer. That index is not accessible to other users.

[0071] Figure 5 illustrates conceptually a process that may be followed when pending edits are made to a development version of the knowledge base documentation A. Pending edits are edits for testing or simulation purposes that have not yet been committed to the live version of the knowledge base documentation by a developer and so are not present in the accessible knowledge base documentation for processing live knowledge base queries. The knowledge base documentation may comprise a plurality of individually and independently editable elements, such as articles or documents. The developer may, in the development version, delete an article or document, add a new article or document, or update an existing article or document (either by deletion of an existing version and replacement by an edited version, or in some other way). Figure 5 illustrates in particular the information retrieval step S202.

[0072] In the process of Figure 5, for simplicity the knowledge base documentation is illustrated as being composed of two elements A and B, which is a replica of the live version of the knowledge base documentation at a point in time (i.e. the point in time at which the replica is made). Initially both the live version and the development version comprise A and B. At S501 developer 1 makes a pending edit to element A. For example, developer 1 may have been notified via a feedback mechanism that a live knowledge base query (knowledge base query X) to which element A may provide useful contextual information has been handled in a manner leading to negative feedback. This may be, for example, because element A is formulated in such a way that it was not retrieved by the information retrieval step in the processing of knowledge base query X, or because it was retrieved but it contained information that was incorrect, out of date, etc.

[0073] At S502 the pending edits made to element A by developer 1 are stored in element A’, a new edited version of element A. Element B remains unedited. The development version of the knowledge base documentation now comprises A’ and B. The developer 1 may achieve this by removal of element A and addition of element A’, or in some other way.

[0074] Sample knowledge base queries will now be discussed to illustrate the access control to the different versions of the knowledge base documentation.

[0075] Element A is accessible to the RAG Al component 100 in processing an input knowledge base query from a user, that is, a live knowledge base query. Element A’ is accessible to the RAG Al component 100 in processing an input knowledge base query from developer 1, that is, a simulation knowledge base query, but since the edits are pending, it is inaccessible for all other cases. Element A’ is inaccessible to the RAG Al component in processing live knowledge base queries input by users and simulation knowledge base queries input by developers other than developer 1. Other developers may be making parallel edits to element A, but they are independent and to a different version. So the element A’ having been added to the development version of the knowledge base documentation assigned to developer 1 is accessible only in handling simulation knowledge base queries input by developer 1.

[0076] Element B is unedited and belongs to the live version of the knowledge base documentation to which access is restricted to live knowledge base queries, so is accessible to the RAG Al component 100 in processing a live knowledge base query. In fact, since element B is unedited, it also remains in the same form in the development version of the knowledge base documentation assigned to developer 1. It is feasible that another developer, developer 2, is editing their own version of element B, but the edits have not been committed and so this is not currently reflected in the live version of the knowledge base documentation nor in the development version of the knowledge base documentation assigned to developer 1.

[0077] Accessibility may be controlled by an access field in the metadata of each replica of the knowledge base documentation, or in the index thereof, such as a flag indicating whether or not the version is accessible in processing live knowledge base queries, and if not, identifying a developer (via a user ID or otherwise) to which the version is assigned and responsible for the pending edits, so that the version is only accessible in processing simulation knowledge base queries from the identified developer. Wherein access to versions may be via an index of the version.

[0078] At S503, developer 1 inputs a simulation knowledge base query, for example, it may be a replication of an historical live knowledge base query, knowledge base query X. As illustrated by the dashed lines, the RAG Al component 100 processes the input simulation knowledge base query by accessing element A’ and element B in the information retrieval, since the development version of the knowledge base documentation assigned to developer 1 comprises elements A’ and B. It is noted that one or both may be considered irrelevant by the algorithm, but they are accessible nonetheless. It is further noted that accessed in this context may relate to a comparison of a representation of the input knowledge base query with an indexed form of the knowledge base element. Full text of elements considered relevant by the information retrieval algorithm may then be accessed for inclusion in the contextual information component of the prompt to be provided to the LLM. At S504, a user inputs a live knowledge base query. As illustrated by the dashed lines, the RAG Al component 100 processes the input simulation knowledge base query by accessing the live version of the knowledge base documentation, which comprises element A and element B, in the information retrieval. If the developer 1 commits the development version (or one or more elements thereof) to the live version, the live version would then comprise A’ and B.

[0079] Were S504 to be a simulation knowledge base query input by a developer other than developer 1, element A’ would be inaccessible, and the accessible elements would be dependent upon the development version assigned to the said developer.

[0080] The developer 1 may iteratively repeat S501 to S503 until the contextual information retrieved in the processing of the simulation knowledge base query is satisfactory, or until some other end point. Should the developer 1 wish to commit the pending edits in A’ to the live version of the knowledge base, then A would be edited to match A’ at the next database update, whether that be event-based or scheduled.

[0081] The development versions of the knowledge base documentation may be individualized per developer (or there may only be a single developer authorized to edit the knowledge base documentation), so that a replica of the knowledge base documentation being edited by a first developer is personal to that developer. Simulation knowledge base queries input to the RAG Al component 100 by the said first developer will utilize the edited version of the knowledge base documentation, whereas live knowledge base queries will utilize the live version.

[0082] Optionally, rather than being treated as a whole, the knowledge base documentation may be configured so that a developer is assigned a replica of a single element or group of elements (that is, a portion or subset of the knowledge base documentation). In such an example, the accessible knowledge base documentation for knowledge base query handling comprises the developer edited version of the portion or subset, and the remainder of the live version. The index is compiled accordingly.

[0083] Returning to Figure 2, the RAG Al component 100, and specifically the information retrieval component, accesses a different version of the knowledge base documentation in dependence upon whether the input knowledge base query is a live knowledge base query 220 or a simulation knowledge base query 320. This differential information retrieval processing is illustrated by the arrows joining the live knowledge base query 220 and the simulation knowledge base query 320 to different knowledge base documentation in Figure 1A. An information retrieval step for a live knowledge base query 220 is performed by retrieving information from the live version 400a of the knowledge base. An information retrieval step for a simulation knowledge base query 320 is performed by retrieving information from the development version 400b of the knowledge base. The input knowledge base query may be pre-processed, for example by parsing and encoding, before being submitted as an information retrieval query to the knowledge base. The query is marked with a flag or other data element indicating that it is a live knowledge base query (and so should be processed by reference to live knowledge base documentation at S202) or a different flag indicating that it is a simulation knowledge base query from a specific developer (and so should be processed by reference to the development knowledge base documentation assigned to the developer). The exact model for the formation and processing of the information retrieval query may vary across implementations, and may be an Al information retrieval algorithm, or may be predefined rules and processing logic.

[0084] The output of the information retrieval component is retrieved information. The retrieved information is termed as being relevant information, indicating that, by the information retrieval process (whether that by an Al model, or by predefined rules and processing logic, or otherwise) the retrieved information is deemed to be relevant to the input knowledge base query. The retrieved information may be in the form of natural language, encoded natural language, or may be textual information represented or encoded in some other way.

[0085] Some processing may be applied to the retrieved information at this step, for example, to consolidate all of the retrieved information into a single document, page, or other form, for inclusion in the LLM prompt as contextual information.

[0086] At step S203, the RAG Al component 100 creates a prompt 120 composed of the input knowledge base query and the retrieved information, and metadata. Prompt creation may be in accordance with predefined rules and processing logic. The prompt 120 may be created by software external to the RAG Al component 100 so that S203 is performed by submitting the input knowledge base query and the retrieved information to the external software (i.e. also forming part of the software system but not forming part of the RAG Al component 100) and receiving the response, wherein such submitting and receiving may be via an interface such as an API. The prompt 120 may be created by the RAG Al component itself using predefined logic for prompt creation.

[0087] The RAG Al component 100, upon receipt of the live knowledge base query or simulation knowledge base query, may be configured to execute processing instructions to automatically retrieve, by reference to the knowledge base, contextual information for inclusion in the prompt created at step b), the contextual information being information available to the LLM in generating a response to the knowledge base query. The prompt 120 comprises the input knowledge base query as a query, and the retrieved information from S202 as contextual information. Metadata indicates which of the payload data is query and which is contextual information. The prompt 120 is submitted to a large language model (LLM) at S204. The prompt 120 is configured so that the LLM addresses the query and the answer is constrained by the contextual information. The prompt 120 is configured so that the answer generated by the LLM is logically consistent with the retrieved information, that is, the contextual information. The prompt 120 is metadata and payload text (including input knowledge base query and retrieved information) sent to the LLM to trigger the LLM to generate a natural language response to the input knowledge base query.

[0088] The LLM may be a part of the software system or may be external. The LLM is accessible to the RAG Al component 100 via an LLM interface 130 such as an LLM API via which to submit the prompt at S204 and to receive an LLM response 140 at S205. The LLM response 140 includes text data representing a natural language response to the knowledge base query as constrained by the contextual information.

[0089] At S206 output processing is performed. The output processing may comprise relaying the LLM response 140 received from the LLM to the interface via which the input knowledge base query was received, as a simulation knowledge base query answer 330 (in case the input knowledge base query was a simulation knowledge base query 320) or a live knowledge base query answer 230 (in case the input knowledge base query was a live knowledge base query 320). Alternatively the answer may be transmitted via some other messaging or communication interface to communicate the response 140 to the developer or end user submitting the source input knowledge base query. In the case of a live knowledge base query, the output processing may further comprise submitting the response 140 to an operative of the software system provider (such as a customer service representative or helpdesk operative) for a manual check before being relayed to the user as a live knowledge base query answer 230. Knowledge base query responses 140 submitted for manual checks may be randomly selected or may be flagged by the RAG Al component based on, for example, a classification of the subject matter.

[0090] Figure IB illustrates a variant of Figure 1A. Figure IB illustrates a further component, task specific instructions 150, that the RAG Al component 100 may include in the prompt 120. The task specific instructions 150 are editable via the developer interface 300. Task specific instructions 150 are a set of generic instructions that are included in the prompt, for example, as context specific instruction, regardless of the input knowledge base query. Task specific instructions 150 are instructions to the LLM for use in generating the LLM response 140. They may be a set of constraints or rules that define the content of the LLM response 140 generated by the LLM. Task specific instructions may be generic, or may be classification-specific. Classification relates to a classification label assigned to an input query either manually by a customer service operative or by a developer, depending on the use case, or automatically. For example, a pre-trained Al classification algorithm may classify input knowledge base queries as relating to one of a predefined set of subject matter classifications based on the semantics of the input knowledge base query. Alternatively, a section, component, web page, interface, application page, being used by the end user at a time of submitting the live knowledge base query may determine the classification. The classification attaches to the query as metadata, for example.

[0091] Further, the classification may be leveraged by the information retrieval algorithm as a filter. For example, knowledge base elements may be tagged as relevant to one or more of the classifications (the tag being editable via the development interface), and the information retrieval algorithm may be configured to retrieve only information from knowledge base elements being tagged with a tag matching a classification assigned to the pertinent knowledge base query.

[0092] Task specific instructions 150 may also be present in the prompts 120 generated by the RAG Al component 100 of Figure 1A. That is, the distinction between Figures 1A and IB is not the presence or absence of the task specific instructions 150, but the configurability thereof. For example, the task specific instructions in Figure 1A may not be editable via the developer interface 300, whereas in the system illustrated in Figure IB the task specific instructions 150 are editable via the developer interface 300.

[0093] In a similar manner to the knowledge base documentation, edits 310 made to the task specific instructions 150 by a developer are accessed only in handling knowledge base queries 320 submitted to the RAG Al component by that particular developer. For example, the task specific instructions are replicated, with an unedited version being accessible by the RAG Al component 100 in handling knowledge base queries from all users (including developers who do not have pending edits 310 to the task specific instructions 150), and an edited version being flagged (the flag being an access control data element) with an identifier of the particular developer and which is accessible in handling development knowledge base queries 320 from the said particular developer.

[0094] Pending edits 310 made to configurable components, including at least one of knowledge base documentation and task specific instructions 150, have a pending status from a time at which the edit 310 is initially made by the developer and a time at which the edit 310 is either committed (confirmed), rejected, or modified. A committed edit is replicated in the live version of the knowledge base documentation or the live version of the task specific instructions 150, as appropriate. Once an edit is committed, it is no longer pending. It is noted particularly with respect to the knowledge base that there may be some latency between commitment and replication into the live versions. Latency will depend upon how the knowledge base has been configured.

[0095] Unedited versions of knowledge base documentation are replaced by the edited versions during an update process. The edited versions become accessible as live knowledge base documentation. During the update process, developers are blocked or otherwise stopped from making new pending edits to the knowledge base. This addresses issues with version control that may be caused by pending edits made during an update process. Clashes may be handled on an element by element basis, so that if two developers have development versions simultaneously and commit edits to different elements, the updates are allowed with no clash handling process. Whereas if the edits are to the same element or are otherwise inconsistent, a clash handling process is initiated. Clash handling may comprise, for example, rejecting the edits from one developer (for example based on a status or by reverting to one or both developers for further instruction).

[0096] The purpose of pending edits is to enable developers, via the developer interface 300, to simulate the effect their edits 310 will have on the answer 330 to a knowledge base query 320 output by the RAG Al component 100, relative to the answer output pre-edit, or output with a different form of the edit. The iterative process of: submitting a knowledge base query 320, reviewing the answer 330, making pending edits 310, re-submitting the knowledge base query 320, reviewing the answer 330, deciding whether to commit, discard, or modify the pending edits; may continue for any number of iterations.

[0097] Committed edits may be stored in a queue to be executed in the true version of the knowledge base and / or task specific instructions. The knowledge base and the task specific instructions may be configured with routines for handling clashing edits (for example, both developer A and developer B had concurrent pending knowledge base edits and both chose to commit), for example, enabling a first in the queue to take precedence, determining precedence based on identity of the developer, not making any edits until both developers have reviewed the clash and agreed upon a solution.

[0098] Figure 3A illustrates a processing flow. The processing flow illustrated in Figure 3 originates at a customer service operative using the RAG Al component user interface 200. At S301 the customer service operative initiates a session with the RAG Al component 100, for example by opening a case record. The case record may be a record of a helpdesk ticket, and includes either a knowledge base query received from an end user and / or some other description of a problem relating to an end user’s interaction with the software system. At S302 the RAG Al component 100 loads in response to the initiation at S301. S302 is an example of S201. In the particular instance of S302, the input knowledge base query is the knowledge base query received from the end user and / or the other description of a problem relating to an end user’s interaction with the software system. At S303 the RAG Al component has loaded and the information retrieval S202 and prompt creation step S203 are performed as discussed above. In the present example the input knowledge base query is a live knowledge base query so the live knowledge base documentation 400a is accessed in the information retrieval process.

[0099] At S304 the prompt is submitted to the LLM API endpoint 130 as an LLM query request. The LLM API endpoint reformats the prompt for submission to the LLM 132. An example format is JSON. S304 is exemplary of S204 of Figure 2. The prompt is an instruction or request issued to the LLM, and may include task specific instructions, which are included for every query being handled (or may be classification-specific for each of a finite set of classifiers), and also contextual information, which is context-specific.

[0100] At S305 the LLM 132 embeds the LLM query request. That is, the LLM 132 receives the LLM query request and embeds it into a vector. The embedded LLM query request is then transmitted via a vector database endpoint 134 to a vector database 136 for a vector search at S306, the results of which are used to generate the text response to the prompt by the LLM at S307. The LLM formats the generated text, for example, as JSON data, and transmits the text response to the LLM API endpoint 130, which generates an LLM query response at S308 as a response to the prompt submitted at S304. The LLM query response is received by the RAG Al component 100 at S309 which performs output processing to create a response, comprising the LLM query response, to the input knowledge base query. S309 is exemplary of S205 of Figure 2.

[0101] The RAG Al component 100 provides the created response to the user interface 200 as an answer to the input knowledge base query at S310. S310 is exemplary of S206 of Figure 2. The user interface 200 is loaded with the answer to the input knowledge base query.

[0102] Depending on the implementation scenario, the answer to the input knowledge base query may simply be output to the end user at S311. Alternatively, the customer service operative may update the answer before outputting to the end user at S311.

[0103] Processes may include a feedback step S312 at which the customer service operative or the end user provides feedback assessing the output answer from S311. At S313 the feedback from S312 is used to flag knowledge base queries which may benefit from consideration by a developer, in case the developer wishes to edit the knowledge base and / or the task specific instructions in response to the feedback. The feedback may be quantitative or qualitative or both. The quantitative feedback may be binary. Figure 3B illustrates a variation of Figure 3A. In the example of Figure 3B, at S302Athe user, such as a customer service operative or equivalent, via the user interface 200, is provided a previous answer to the same knowledge base query, and may include a mechanism (such as an element of the user interface) to manually add contextual information to the prompt for a subsequent submission of the knowledge base query. For example, S302A may be triggered by negative feedback at S312. The contextual information is an element of the prompt that is included only in the single execution of the process of Figure 2 for processing the resubmission of the knowledge base query. The contextual information may be composed of information retrieved from the knowledge base documentation (based on the content of the knowledge base query) and optionally may also include additional context provided manually by the user based on knowledge base query documentation or end user characterization (including end user type, end user business type, end user geographic location, end user system user status, or another descriptor or label pertaining to the specific end user or the specific knowledge base query). Addition of the additional context may be manual, via a manual input field provided to a user (noting that user in this context may be an operative of the software service provider tasked with responding to end user knowledge base queries). Addition of the additional context may be automated, by executing processing logic to extract or otherwise derive the additional context from the input knowledge base query, from an origin point of the knowledge base query within the software service, and / or end user identity.

[0104] The additional context included in the prompt may be the addition of a keyword such a descriptor applying to the end user (i.e. small business, corporation, sole trader) and then the LLM API endpoint 132 in providing the JSON input to the embed request can specify the descriptor which causes a corresponding vector namespace mapped to the descriptor to be utilized in the embedding.

[0105] As illustrated in Figure 3C, a developer may configure the processing logic for the automatic determination of context specific instructions.

[0106] The process of Figure 3C begins at S312 (see Figure 3B) in which an answer generated by the system receives negative feedback via the user interface 200. The knowledge base query and answer are stored for access by a developer via the developer interface to review instances of negative feedback. S314 is a review via the developer interface 300 of cases (i.e. live knowledge base query plus answer) that received negative feedback and S315 is the selection, via the developer interface S314 of a particular case.

[0107] At S316 the developer, via the developer interface, edits the contextual information included in the prompt by editing a relevant element in a development version of the knowledge base documentation. The knowledge base query along with the modified context specific instruction is submitted to the RAG Al component 100 and at step S303a a development API call is generated. Step S303a is an alternative version of step S303. At S303a the information retrieval S202 and prompt creation step S203 are performed as discussed above. In the present example the input knowledge base query is a simulation knowledge base query so the accessible knowledge base documentation is a development version of the knowledge base documentation assigned to the specific developer, which is accessed in the information retrieval process. The modified contextual information resulting from the information retrieval and the edited knowledge base documentation is included in the prompt. Modified in this context is relative to the previous iteration of the same input knowledge base query.

[0108] Steps S304 to S309 continue per Figures 3A and 3B.

[0109] S310a is an alternative version of S310. At S310a the RAG Al component 100 provides the created response to the developer interface 300 as an answer to the input simulation knowledge base query. S310a is exemplary of S206 of Figure 2. The developer interface 300 is loaded with (i.e. displays) the answer to the simulation knowledge base query.

[0110] Step S317 is a commit (confirm), reject, or modify step for the edits to the knowledge base. The modify step is illustrated by the dashed line between S316 and S317. The modify step rejects (i.e. does not commit to the live version) the edits made at the previous iteration of S316 but enables the developer via the developer interface S316 to make modifications to the edits. Alternatively the developer may choose to commit edits so that, for example in the case of knowledge base edits 310, they are committed to the live knowledge base documentation for access in processing live knowledge base queries (or flagged for access for live knowledge base query handling).

[0111] The commits made at S317 are reflected in the corresponding vector data at S318, which makes a save / delete context API call to the vector database endpoint to make a database upsert / delete at S319 and the database record itself is updated at S320.

[0112] Figure 4 illustrates a process executed by a software system comprising an RAG Al component 100 and one or more interfaces. The process corresponds to the process illustrated at Figure 2. Steps S401 to S405, for example, are not represented in Figure 2.

[0113] S401, fetch documents, may be event-triggered, or may be polling. S401 relates to the preparation of the knowledge base documentation for knowledge base query handling. The documents or articles composing the knowledge base documentation are fetched via an API. S401 may be event-triggered, such as by commiting an edit to the live version of the knowledge base documentation for handling live knowledge base queries, and / or S401 may be performed periodically.

[0114] The software system includes a model for preprocessing and chunking the fetched documentation at S402 and S403 to make the fetched documentation accessible for information retrieval step S202, S410 etc.

[0115] At S204 the model embeds as vectors the chunked preprocessed documentation. Document metadata is included in the fetched documentation at S401. At S405 the API stores both the embedding (i.e. vector) for lookups and the chunk text as payload.

[0116] Receive case at S406 is an example of S201, receive input knowledge base query. An API passes the request to the model from the user interface 200 or developer interface 300. Optional step S407 anonymises the knowledge base query by masking personal information. In the case of developers some identifying information is required to control access to different development versions of the knowledge base 400b.

[0117] Steps S408 to S410 describe a process for retrieving information relevant to an input knowledge base query, and therefore correspond to S202. At S408 a case received at the model via an API is transformed to an information retrieval query. The case may be a natural language question submited from a user via a messaging interface or via a GUI or some other interface, and at S408 is parsed and otherwise processed using processing logic to generate a corresponding information retrieval query for submission to the knowledge base. At S409 the information retrieval query is embedded and the embedded information retrieval query is compared with the indexed embedded knowledge base at S410. The payload chunks relating to the embeddings identified as relevant are retrieved.

[0118] Step S411, create prompt, corresponds to S203.

[0119] Step S412, call LLM with prompt, corresponds to S204.

[0120] Steps S413 to S414, post-process LLM response and return response, correspond to S205 to S206.

[0121] For example, the RAG Al component 100 may execute the steps of S401 to S413 including cooperating with one or more endpoints, interfaces, APIs, to perform the functions illustrated in Figures 2 and 4 and discussed above. The RAG Al component is a model, program, sub-program, or other software artefact, for executing a retrieval augmented generation artificial intelligence process and associated input processing, output processing, such as illustrated in Figures 2 and 4. Figure 6 is a schematic illustration of a hardware arrangement of a computing apparatus. The methods, processes, protocols, techniques, etc, described herein may be performed by apparatus having an arrangement such as illustrated in Figure 6. Apparatus having processor hardware and memory hardware described in the present specification may include one or more devices having an arrangement such as illustrated in Figure 6. A plurality of such devices may be interconnected over a network such as a Local Area Network or the internet. A cloud service including performing one or more of the methods, processes, protocols, techniques, etc, described in the present specification may be performed by one or more devices having an arrangement such as illustrated in Figure 6.

[0122] The computing apparatus comprises a plurality of components interconnected by a bus connection. The bus connection is an exemplary form of data and / or power connection. Direct connections between components for transfer of power and / or data may be provided in addition or as alternative to the bus connection.

[0123] The computing apparatus comprises memory hardware 991 and processing hardware 993, which components are essential regardless of implementation. Further components are context-dependent, including a network interface 995, input devices 997, and a display unit 999. The display unit 999 and the processing hardware 993 may cooperate to implement a graphical user interface. The display unit 999 may be a touchscreen display unit.

[0124] The computing apparatus may be a smartphone or tablet having a touchscreen display unit.

[0125] The computing apparatus may be a laptop or other form of personal computer having a touchscreen display unit.

[0126] The memory hardware 991 stores processing instructions for execution by the processing hardware 993. The memory hardware 991 may include volatile and / or non-volatile memory. The memory hardware 991 may store data pending processing by the processing hardware 993 and may store data resulting from processing by the processing hardware 993.

[0127] The processing hardware 993 comprises one or a plurality of interconnected and cooperative CPUs for processing data according to processing instructions stored by the memory hardware 991.

[0128] A computing apparatus may comprise one computing device according to the hardware arrangement of Figure 6, or a plurality of such devices operating in cooperation with one another. For example, in a client: server arrangement. A network interface 995 provides an interface for transmitting and receiving data over a network. Connectivity to one or more networks is provided. For example, a local area network and / or the internet. Connectivity may be wired and / or wireless.

[0129] Input devices 997 provide a mechanism to receive inputs from a user. For example, such devices may include one or more from among a mouse, a touchpad, a keyboard, an eye-gaze system, and a touch interface of a touchscreen. Inputs may be received over a network connection. For example, in the case of server computers, a user may connect to the server over a connection to another computing apparatus and provide inputs to the server using the input devices of the another computing apparatus.

[0130] A display unit 999 provides a mechanism to display data visually to a user. The display unit 999 may display user interfaces by which certain locations of the display unit become functional as buttons or other means allowing for interaction with data via an input mechanism such as a mouse. A server may connect to a display unit 999 over a network.

[0131] As illustrated by Figure 7, an example of a software service (as described in relation to Figures 1 to 6) is an accounting system 231. The accounting system 231 may comprise one or more computing devices and / or server devices, such as one or more servers, databases, and / or processing devices in communication over a network. The accounting system 231 may be configured to provide accounting services to users, such as entities and accounts, and to maintain accounts for a plurality of entities, such as businesses, individuals and organisations. For example, the accounting system 231 may be used by an accounting services provider such as an accountant, and used to track payer data and invoice data generated with respect to clients of the accounting services provider, such as business entities.

[0132] The RAG Al component 100 is a component of the accounting system 231 and may be accessible to other components via an API 239, and may access the functionality and data of other components via respective APIs 239. An end user device for submitting live knowledge base queries may be a client device 900.

[0133] According to some embodiments, the accounting system 231 may comprise a cloud-based server system. The cloud-based server system may comprise one or more computing apparatus such as illustrated in Figure 6. The accounting system 231 may further comprise a processor 992 in communication with a memory 991. The processor 992 may comprise one or more data processors for executing instructions, and may comprise one or more microprocessor-based platforms, central processing units (CPUs), application specific instruction set processors (ASIPs), application specific integrated circuits (ASICs), suitable integrated circuits, or other processors capable of fetching and executing instruction code as stored in the memory. The processor 992 may include an arithmetic logic unit (ALU) for mathematical and / or logical execution of instructions, such as operations performed on the data stored in internal registers of the processor.

[0134] A communications interface 237 controls input and output of data to / from the accounting system. The interface may comprise plural individual interfaces. For example, a messaging interface such as a virtual assistant, chatbot, or Al assistant. Via such interfaces, live knowledge base queries may be received and answers output.

[0135] APIs 239 are application programming interfaces, which may be RESTfiil APIs. APIs control flow of data within the accounting system 231 and instruct execution of software services, microservices, and other processing instructions forming part of the accounting system 231. One or more messaging interfaces may have an underlying API 239 to control the flow of information from the messaging interface to the accounting system 231 and from the accounting system 231 to the messaging interface.

[0136] The accounting system 231 may be configured to receive and / or store data related to one or more invoices issued by an entity to a client or customer. Data relating to clients (i.e. users) may be stored by the accounting system 231 internally or at a remote database 236 accessible via a network 215. Invoice data may include a unique invoice identifier, such as an invoice number. Invoice data may also include one or more of a payment date, payment deadline, payment amount, discount amount, tax amount and unique client or invoice identifier. The unique client identifier may include one or more of the client name, client contact information such as a telephone number, a company registration number (such as an ABN or ACN) or a number generated by the accounting system 231 to uniquely identify the client.

[0137] The accounting system 231 may also be configured to store data relating to payers associated with the business entity, such as clients and customers to whom invoices are issued. Payer data may include one or more of the payer name, payer contact information such as a telephone number, a company registration number (such as an ABN or ACN) or a payer identifier such as a payer account number.

[0138] The accounting system 231 may be configured to execute functions such as reading and writing invoice and / or payer data and communicating retrieved data to segmentation rules determination server 217. This data may be communicated between the accounting system 231 and the segmentation rule determination server 217 via network 215, wired connection (not shown) and / or may be comprised within the same computer system or server (not shown).

[0139] The accounting system 231 may communicate with one or more financial institute or banking servers, such as server 233. In some embodiments, the accounting system 231 receives, from such server(s) 233, records or documents associated with data being monitored by the accounting system 231. For example, the accounting system 231 may be arranged to receive bank feeds associated with transactions to be reconciled by the accounting system 231. The financial or banking data may be imported through a bank feed and / or a user- or accountant-created document. In some embodiments, the accounting system 231 may communicate with third-party tools of the server(s) 233 via an application protocol interface (API) to receive the banking data. Third party server(s) may include external Al models such as generative Al models including LLMs, to which the accounting system 231 may issue prompts and receive responses in reply.

[0140] The accounting system 231 may comprise a bookkeeping system 235 for maintaining bookkeeping account information on behalf of the user based on data input by the user, or retrieved from a bank feed or another third party source 233, such as a payment processing service, relating to financial transactions in which the user is a payer or payee.

[0141] A client device 900 may have an arrangement as illustrated in Figure 6. The memory 991 of the client device 900 may store an application 234 which provides one or more interfaces for an end user to at least one of: view, add to, edit, manipulate, process, assimilate, generate reports based on, data stored by the accounting system 231 in relation to a user account of the particular user (or any data that the user is authorised to access, such as an accountant user accessing client data). The interfaces may include a messaging interface such as a chatbot, virtual assistant, or Al assistant. A communications interface 995 controls flow of data in and out of the client device 900.

Claims

CLAIMS1. A system operable to provide a software service for a plurality of end users, the system comprising: an end user interface operable, at runtime, to receive a live knowledge base query from an end user; knowledge base documentation; a retrieval augmented generative (RAG) Al component operable to process input knowledge base queries by: a) retrieving, from accessible knowledge base documentation, information relevant to the input knowledge base query; b) creating a prompt comprising the retrieved information and the input knowledge base query and submitting the prompt to a LLM; c) receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the input knowledge base query; the knowledge base documentation being stored as a live version of the knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time; the software service being operable in a live mode to receive the live knowledge base query via the end user interface, and to input the live knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the live version at step a), the software service being operable in a development mode running concurrently with the live mode, to receive one or more pending edits to the development version of the knowledge base documentation via the developer interface, to receive a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface, and to input the simulated knowledge base query to the RAG Al component to process with accessible knowledge base documentation being restricted to the edited development version at step a).

2. The system according to claim 1, wherein upon approval by the developer, the one or more pending edits to the development version are replicated in the live version.

3. The system according to any of the preceding claims, wherein upon rejection by the developer, the one or more pending edits in the development version are discarded.

4. The system according to any of the preceding claims, wherein the prompt further comprises task specific instructions, comprising one or more from among: a) generic task specific instructions, wherein the generic task specific instructions are generic to all input knowledge base queries; and b) selected task specific instructions from a set of classification-based task specific instructions, being selectable according to a classification of the live knowledge base query, and are included in the prompt as task specific instruction for the LLM in generating the response, the task specific instructions being a rule, constraint, or other instruction to the LLM.

5. The system according to claim 4, comprising: a live version of the task specific instructions and a configurable development version of the task specific instructions, wherein the RAG Al component is configured: in the live mode to include the live version of the task specific instructions in the prompt, and in the development mode to accept one or more edits to the configurable development version of the task specific instructions from a developer, to include the edited configurable development version of the task specific instructions in the prompt, and upon approval from the developer, to replicate the edited configurable development version of the task specific instructions in the live version of the task specific instructions.

6. The system according to any of the preceding claims, wherein the developer interface is operable to provide an interface to access and make pending edits to the development version of the knowledge base documentation and / or the edit the task specific instructions, to review the answer to the simulation knowledge base query output by the RAG Al component at step c), and to select whether to approve the pending edits to the development version of the knowledge base documentation and / or the task specific instructions for replication in the respective live versions.

7. The system according to any of the preceding claims, wherein the software system comprises a user interface, the user interface being configured to receive text or voice data input by a user, the text data representing a live knowledge base query, and to submit the live knowledge base query for processing by the RAG Al component as an input knowledge base query.

8. The system according to any of the preceding claims, wherein the user interface displays the response output by the RAG Al component as the answer to the input knowledge base query.

9. The system according to claim 7 or 8, wherein the user interface provides a classification input field, to receive a classification of the live knowledge base query, and to submit the classification to theRAG Al component in association with the live knowledge base query for filtering the live version of the knowledge base documentation at information retrieval step a), the classification being a label, descriptor, rule, constraint, or other information.

10. The system according to claim 7 or 8, wherein the RAG Al component is operable to automatically assign a classification to an input knowledge base query, based on a tool, web page, application page, or interface, of the software service from which the input knowledge base query originates, or based on a classification of the input knowledge base query by a classification algorithm, and to submit the classification to the RAG Al component in association with the input knowledge base query for filtering the live version of the knowledge base documentation at information retrieval step a), the classification being a label, descriptor, rule, constraint, or other information.

11. The system according to any of claims 7 to 10, wherein the user interface is directly accessible by at least one of an operative of the software system provider and an end user.

12. The system according to any of the preceding claims, wherein the simulation knowledge base query is an historical live knowledge base query previously submitted to the system by an end user.

13. The system according to any of the preceding claims, wherein the RAG Al component includes a pre-trained information retrieval algorithm, and the RAG Al component is configured to perform retrieving information at step a) by executing the pre -trained information retrieval algorithm.

14. The system according to any of the preceding claims, wherein the development version of the knowledge base documentation to which pending edits are made by a developer is a restricted-access version of the knowledge base documentation being a replication of the live version of the knowledge base documentation at a point in time from which one or more documentation elements are removed and / or superseded by replacement documentation elements comprising pending edits, the development version being accessible to the RAG Al component only in processing a simulation knowledge base query input by the said developer, the documentation elements each being an article, page, or document.

15. The system according to claim 14, wherein upon a pending edit being made to a development version of the knowledge base documentation by a developer via the development interface, an unedited form of the knowledge base documentation remains accessible to the RAG Al component in processing live knowledge base queries, the unedited form of the knowledge base documentation is inaccessible to the RAG Al component in processing asimulation knowledge base query from the same developer, and the edited form of the knowledge base documentation is accessible to the RAG Al component only in processing a simulation knowledge base query input by the said same developer.

16. The system according to any of the preceding claims, whereinThe live version of the knowledge base documentation is stored as a replicated document repository comprising plural replicas stored on distinct server apparatus, each replica being operable to be accessed simultaneously by the RAG Al component in processing multiple input queries in parallel.

17. The system according to claim 15, wherein pending edits to the knowledge base documentation, upon approval at the developer interface, are committed to the replicated database by a replicated database update procedure; and during the replicated database update procedure, the developer interface blocks any new pending edits from being made.

18. A computer-implemented method in a system providing a software service for a plurality of end users, the computer-implemented method comprising: storing a live version of knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time; in a live mode: at an end user interface, at runtime, receiving a live knowledge base query from an end user; at a retrieval augmented generative (RAG) Al component processing the live knowledge base query by: retrieving, from the live version of the knowledge base documentation, information relevant to the live knowledge base query; creating a prompt comprising the retrieved information and the live knowledge base query and submitting the prompt to a LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the live knowledge base query; in a development mode running concurrently with the live mode: at the developer interface, receiving one or more edits to the development version of the knowledge base documentation via the developer interface, and receiving a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface, at the retrieval augmented generative (RAG) Al component processing the simulated knowledge base query by:retrieving, from the edited development version of the knowledge base documentation, information relevant to the simulated knowledge base query; creating a prompt comprising the retrieved information and the simulated knowledge base query and submitting the prompt to the LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the simulated knowledge base query.

19. A computer program comprising processing instructions which, when executed by a computing apparatus comprising memory hardware and processor hardware, cause the computing apparatus to perform a method comprising: storing a live version of knowledge base documentation and a development version of the knowledge base documentation, the development version being configurable via a developer interface from a baseline corresponding to the live version at a point in time; in a live mode: at an end user interface, at runtime, receiving a live knowledge base query from an end user; at a retrieval augmented generative (RAG) Al component processing the live knowledge base query by: retrieving, from the live version of the knowledge base documentation, information relevant to the live knowledge base query; creating a prompt comprising the retrieved information and the live knowledge base query and submitting the prompt to a LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the live knowledge base query; in a development mode running concurrently with the live mode: at the developer interface, receiving one or more edits to the development version of the knowledge base documentation via the developer interface, and receiving a simulated knowledge base query or a selection of a simulated knowledge base query from the developer interface, at the retrieval augmented generative (RAG) Al component processing the simulated knowledge base query by: retrieving, from the edited development version of the knowledge base documentation, information relevant to the simulated knowledge base query; creating a prompt comprising the retrieved information and the simulated knowledge base query and submitting the prompt to the LLM; receiving from the LLM a response to the submitted prompt and outputting the response as an answer to the simulated knowledge base query.

20. A computer program comprising processing instructions which, when executed by a computing apparatus having memory hardware and processor hardware, cause the computing apparatus to implement the system according to any of claims 1 to 17.

Images