Method and device for rule-based integration of a large language model

Abstract

The invention relates to a method for rule-based integration of a large language model, LLM, with comprehensive rule trees, rule maps, the inner nodes of which represent logical links and on the leaves of which prompts for the handover to the LLM are stored, wherein the prompts are each composed of a system prompt and one or more user prompts, wherein user prompts comprise a specific substantive matter and a query definition, which determine the form of the result, and are used in the rule tree in the next inner node as a prompt in the logical link, wherein the system prompt specifies the context from globally predetermined settings and information that is predetermined for the specific rule map, wherein the prompts thus created are handed over sequentially to the LLM in order to create a logical response that is suitable for being evaluated in the rule tree, wherein the rule tree is run through from the leaves to the roots in order to create a logical end result.

Description

[0001] 24032SNBPTEP - Rule based AI 08 / 29 / 2024

[0002] Applicant:

[0003] Rulemapping Group GmbH

[0004] Friedrichstr. 123, 10117 Berlin

[0005] Method and apparatus for rule-based integration of a Large Language Model

[0006] Rule-based artificial intelligence is a technical procedure for machine-based and automated testing to determine whether all the prerequisites specified by the underlying rule are met in a specific case.

[0007] Field of invention:

[0008] Currently, rule-based systems are used for machine-based rule-based checks. Attempts are also being made to support rule-based decisions using generative artificial intelligence (AI) in the form of a Large Language Model (LLM). However, there is currently no technical method that combines these two approaches. Possible LLMs include, but are not limited to, ChatGPT 4, MS Azure, Mistral, Aleph Alpha, and LLama, which are also supported by the

[0009] Invention to be used. 24032SNBPTEP - Rule-based AI 29.08.2024

[0010] 2 / 56

[0011] An LLM alone cannot reliably and reliably conduct a rules-based review. When using an LLM to review rules-based contexts, there is a risk that it will hallucinate (see, for example, "Hallucinating AI in legal review": Affidavit of Steven Schwartz in Mata v. Avianca, Inc., 1:22-cv-01461, (SDNY May 25, 2023) ECE No. 32, District Court, SD New York, https: / / w.courtlistener.com / docket / 63107798 / 32 / l / mata-v-avianca-inc / ) and invent its own content. Users have no control over how the LLM behaves or the rules it uses to reach its review outcome.

[0012] Rule-based systems cannot perform independent assessments. All characteristics and prerequisites requiring evaluation are either data-driven or must be manually checked, which is very time-consuming and precludes full automation of rule-based systems.

[0013] Decision support is excluded wherever a fully data-driven decision cannot be made due to a lack of data.

[0014] Overview of the invention:

[0015] The technical method described in the Rule-based AI combines these two technologies. A rule-based check is performed along a rule tree (the rule map) while simultaneously applying a semantic correlation-based LLM (Large Learning Model) using artificial intelligence.

[0016] What kind of rule-based test is the "result", i.e., the case question, and what characteristics and prerequisites are required for its examination or answering based on the justifying 24032SNBPTEP - Rule based AI 29.08.2024

[0017] The rules that are necessary are defined in the rule map of the rule-based AI. A rule map is fundamentally abstract for a multitude of cases, i.e., an examination matrix. A rule map can be defined once, specific to a particular application, and then used repeatedly in an automated manner.

[0018] Application areas include all rule-based systems. Besides classic rule-based contexts such as law and administration, these include, for example, procedural rules, process rules, rule-based machine automation, medical rules (guidelines), operating instructions, and technical standards, without this list being exhaustive.

[0019] At each end node (leaf) of the rule map, so-called query definitions can be stored, which communicate with the LLM during the actual processing.

[0020] In the invention, a rule-based test is performed automatically based on the rule architecture defined in the (visually representable) rule map (causal relationship), using the only correlatively defined

[0021] LLMs .

[0022] Using this technical method of rule-based AI, the individual features are checked separately and independently. The queries in the end nodes / leaves are passed to the LLM via automatically generated prompts. The LLM check is then performed separately for each feature. This technically "forces" the AI ​​to follow the rule architecture and perform a rule-based evaluation. The user retains complete control over which rules are provided to the AI ​​and can adjust them as needed. 24032SNBPTEP - Rule-based AI 29.08.2024

[0023] 4 / 56

[0024] Another part of the invention is a device that performs the necessary calculations to carry out the method. Parallel AI systems from, for example, Nvidia or other manufacturers are particularly relevant here.

[0025] Character description:

[0026] The characters are briefly described below.

[0027] Figure 1 shows a simple rule map with the three basic logical operators. The start node represents the result. This occurs when A, B, AND C are given. A occurs when 1 OR 2 OR 3 (or more than one of these) are given. 1 occurs when either 1a or 1b are given.

[0028] Figure 2 describes the individual elements of the rule map. a) is the node title, b) symbolizes the logical operators.

[0029] Figure 3 describes the logical connectives / operators. a) describes the AND operation, b) the OR operation (non-exclusive OR), c) the XOR operation (exclusive OR).

[0030] Figure 4 describes the examination sequence within the rule map.

[0031] Figure 5 describes how a definition is stored at an end node, which is then fed into the automatic prompt.

[0032] Figure 6 describes how the automatic test result (here "yes") is visualized using color (true / green). 24032SNBPTEP - Rule-based AI 29.08.2024

[0033] 5 / 56

[0034] Figure 7 describes how the automatic test result (here “no”) is visualized in color (false / red).

[0035] Figure 8 describes how the logical sum of the three child nodes a, b and c linked in an AND operation, all of which are given, defines the result (given, green).

[0036] Figure 9 describes how the logical sum of the three child nodes linked in an AND operation, of which c) is not given (false / rot), defines the result (not given, rot).

[0037] Figure 10 describes the sequence of automatic checks and result representation in an AND operation. Along the rule map, the first end node of the first path (here

[0038] A) checked. If this is given (true / green), the check continues with the next end node of the path, (here

[0039] B) Since this value is not given (false / red), the logical sum already yields the result "not given" (false / red) and the check is complete. Node C no longer needs to be checked.

[0040] Figure 11 describes the result representation in an OR operation with a positive outcome. Node 1 is "not given" (false / red), node 2 is "given" (true / green), node 3 is "not given" (false / red). In the OR operation, a single positive logical result is sufficient for the parent node (A) to be given.

[0041] Figure 12 describes the result representation in an OR operation and an AND operation with a negative result. Nodes 1, 2, and 3 are "not given" (false / 24032SNBPTEP - Rule based AI 29.08.2024).

[0042] 6 / 56 (red), which is why node A is "not given" (false / rot) in the logical sum. Because A is linked to its parent node ("result") in an AND operation, the logical sum is also "not given" (false / rot) here.

[0043] Figure 13 describes the sequence of automatic checks and result representation in an OR operation with a positive result. Along the rule map, the first end node of the first path (here 1) is checked. This node is not present (false / red), so the check continues with the next end node of the path (here 2). Since this node is present (true / green), the result of the parent node (A) is already "present" (true / green) from the logical sum. Nevertheless, the check continues with node 3 because this node could also be present (non-exclusive OR).

[0044] Figure 14 describes the logical behavior in the XOR operation with a positive result (true / green). Since la is "given" (true / green), 1 is also "given" (true / green). 1b and 1c are no longer checked (gray) because they are exclusively linked with la. Because 1 is linked to its parent node A in an OR operation, the logical sum is also already "given" here (true / green).

[0045] Figure 15 describes the logical behavior in the XOR operation with a negative result (false / rot). Because 1a, 1b, and 1c are "not given" (false / rot), 1 is also "not given" (false / rot).

[0046] Figure 16 describes the sequence of automatic testing and result representation in an XOR operation with a positive result. The first step is along the rule map. 24032SNBPTEP - Rule based AI 29.08.2024

[0047] 7 / 56

[0048] The end node of the first path (here la) is checked. This node is not given (false / red), so the check continues with the next end node of the path (here 1b). Since this node is given (true / green), the result of the parent node (1) is already "given" (true / green) from the logical sum. 1c is no longer checked (grey) because it is exclusively linked to la and 1b.

[0049] Figure 17 describes the logical behavior in the AND operation with a positive result (true / green). Since A, B, and C are "given" (true / green), the "result" is also "given" (true / green).

[0050] Figure 18 describes the logical behavior in the AND operation with a negative result (true / green). Since A and B (true / green) are "given", but not C (false / red), the "result" is also "not given" (false / red).

[0051] Figure 19 illustrates the structure of a damages claim assessment. Six criteria must be met cumulatively (AND). The sixth criterion has two mutually exclusive (XOR) options.

[0052] Figure 20 describes how definitions / descriptions of the feature to be checked are stored at an end node.

[0053] Figure 21 describes how the (positive test) result of the first automatic prompt is returned to the rule map (true / green).

[0054] Figure 22 describes how the (positive test) result of the first five automatic prompts is sequentially returned to the rule map (true / green). 24032SNBPTEP - Rule-based AI 29.08.2024

[0055] 8 / 56

[0056] Figure 23 describes how the (positive test) result of a node in the XOR operation is returned to the rule map (true / green) and that the node exclusively linked to it is no longer displayed as testable (grey).

[0057] Figure 24 describes how the (positive test) result of all child nodes of the starting node leads to the positive final result (true / green).

[0058] Figure 25: describes the representation of the rule structure of § 130 StGB (incitement to hatred) in a rule map and shows that in the specific examination carried out, three alternative elements of the offense are not present and the examination is currently in the fourth alternative element of the offense.

[0059] Figure 26: describes the overall process of Rule Based AI in a diagram. A scenario (extendable by 1+n) is loaded into a Rule Based AI rule map, where query definitions for the individual rules are stored in the respective end nodes. Each query definition, along with the scenario and general text for prompt optimization, is passed individually to an LLM (Large Language Management) module. The LLM then returns a binary result (true / false) and a textual result to the end node from which the query definition originated. This process is repeated for each subsequent end node according to the logic of the rule map.

[0060] Description of the execution form based on the figures:

[0061] Procedure steps of Rule Based AI 24032SNBPTEP - Rule based AI 29.08.2024

[0062] 9 / 56

[0063] The procedural steps of the rule-based AI's review follow the path derived from the rule architecture of the rule map (Error! Reference source could not be found.). The rule map is a decision tree representing the entirety of the nodes (Error! Reference source could not be found., a) and their logical connections (lines, Error! Reference source could not be found., b). There can be various types of connections, corresponding to the possible logical relationship between different elements. These connection types are: "AND" (Error! Reference source could not be found., a), "Non-exclusive OR" (Error! Reference source could not be found., b), "Exclusive OR" (Error! Reference source could not be found., c), and "NOT".The rule map is not only a visualization of the relationships between different elements, but it is also machine-readable and interpretable without further coding. The type of linking used determines how the rule map is technically read.

[0064] The starting node of the rule map contains the result, i.e., the exam question. The characteristics and prerequisites necessary for this result are represented as child nodes. Each child node can, in turn, be a parent node for child nodes at a further level. The depth of a rule map's architecture depends on the complexity of the rule set it represents.

[0065] During the check along the rule map, each node is individually checked to determine whether the corresponding characteristic is present in a specific case. The check follows the path from top to bottom and right to left from the starting node, as shown in [Error! Reference source not found 24032SNBPTEP - Rule based AI 29.08.2024]

[0066] 10 / 56 are displayed. A different order from bottom to top or based on priorities is conceivable. The end nodes of a path (lowest child nodes) are checked and either confirmed and set to green or rejected and set to red. Other visually easily grasped representations are conceivable. The respective parent node is set to green or red according to the logical sum of the examined child nodes by the algorithm built into the rulemap software.

[0067] When applying "rule-based AI," artificial intelligence is integrated into this rule-based check. The AI ​​is used to decide whether an individual end node / leaf node should be set to green or red, that is, whether the feature represented in the node is present in the specific case being checked. This is technically achieved as follows:

[0068] For each end node, a prompt is passed to the LLM via the interface. The query passed in this way consists of the following parts:

[0069] System prompt

[0070] User prompt o Situation o Query definition

[0071] : openai

[0072] ( : token "xxxxxxx" : config

[0073] ( : model " aaa" : temperature b :max_tokens ccc : top-p d) 24032SNBPTEP - Rule based AI 08 / 29 / 2024

[0074] 11 / 56

[0075] The system states: "You are a German criminal judge and must examine the elements of an offense. We proceed step by step, examining each element of the offense in separate prompts."

[0076] Note that the [statement of facts] may contain errors due to OCR recognition. Furthermore, the statement of facts lacks any formatting.

[0077] Think step by step. If the [element of the offense] involves several steps, address them sequentially in a causal order. If an [element of the offense] includes different variations, examine each one individually for its applicability.

[0078] For your analysis, use in particular the following interpretation methods:

[0079] “Grammatical interpretation” (or also “interpretation from the wording”)

[0080] “Systematic interpretation” (or also “interpretation from the context” in which a legal provision is placed)

[0081] "Historical interpretation" (or also "interpretation based on the history of its creation")

[0082] “Teleological interpretation” (or also “interpretation according to the meaning and purpose of a legal provision”)

[0083] This list of interpretations is not exhaustive.

[0084] Your task will be to solve the following [element of the offense] in relation to the [facts].

[0085] Your answer must contain 2 parameters. 24032SNBPTEP - Rule-based AI 29.08.2024

[0086] 12 / 56

[0087] 1. A binary answer, 1 for yes, 0 for no.

[0088] 2. A brief explanation of why you made this decision. The explanation must be short, concise, and to the point, and should not exceed 250 characters. It must refer to the specific reasons for the decision arising from the facts of the case and simultaneously identify which arguments related to the elements of the offense led to the decision. If the reasoning is completely obvious, meaning you are very certain about the outcome, keep the explanation brief or omit it altogether. If you are less certain, provide a more detailed explanation.

[0089] Syntax: 0 / 1 "REASONING". Your answer must look exactly like this. Do not enclose the 0 / 1 in quotation marks or similar. Furthermore, the word "REASONING" should not precede the word. The reasoning should be written in the style of a judgment. A text written in the style of a judgment is created in two steps:

[0090] 1. State the result

[0091] 2. Statement of the supporting considerations, i.e. the justification"

[0092] : Facts #variable field value#

[0093] :userresult "0 = No; 1 = Yes. The mandatory syntax is 1 OR 0 REASON.

[0094] Example :

[0095] 0 B's liability under Section 823 Paragraph 1 of the German Civil Code (BGB) is not applicable, as he is not responsible for his conduct pursuant to Section 276 Paragraph 2 of the German Civil Code (BGB).") 24032SNBPTEP - Rule based AI 29.08.2024

[0096] 13 / 56

[0097] The system prompt, which precedes the actual user prompt, transmits the aforementioned variables and instructs the LLM (Large Logical Management) regarding the context. The system prompt is preferably specific to the entire rule tree / rulemap. However, it can also be defined only for sections of the rule tree / rulemap, but it applies to at least two or more nodes to distinguish it from node-specific information. This means that global settings for the rulemap are stored in the background, which it uses abstractly and for any number of situations.

[0098] Initially, the parameter "temperature," which controls the randomness or creativity of the outputs generated by the AI, should be set to the lowest possible level (0-10%). In a legal context, the system prompt would then read, for example, "The assistant is a German judge and must examine elements of a crime." The definition of an element of a crime is equivalent to the query definition. A crime can regularly consist of a multitude of elements. Each end node references only one element. The sum of the elements (parent nodes) then constitutes the crime.

[0099] The defining characteristic is defined in the query definition of an end node. For example: "The content must somehow be related to a union."

[0100] An association is an organized union of more than two entities intended to last for a longer period of time, independent of the definition of roles of the members, the continuity of membership and the specific form of the structure. 24032SNBPTEP - Rule based AI 29.08.2024

[0101] 14 / 56

[0102] persons for the pursuit of a higher common goal

[0103] Interest.

[0104] Knowledge from the Large Language Model can also be used for this purpose.

[0105] Positive examples: political parties, associations.

[0106] Negative examples: The association is only of short duration, or only a loose connection between people, or there is no common organization, or there are fewer than 3 people.

[0107] Furthermore, the syntax for the assistant's output is already defined here, e.g., "If the characteristic is present, respond exclusively with '1'; if it is not present, respond exclusively with '0'." Other logical representations are of course conceivable, such as True or False, etc. This setting is preferably stored in the system prompt but can also be defined in one or more user prompts.

[0108] The downstream user prompt contains the facts that are checked against the rule model stored in the rule map, as well as a definition of the characteristic (query definition) itself (Figure 5).

[0109] The user prompt is the combination of the variable fact, which is passed via a field value, and the query definition of the end node.

[0110] If the information is not originally available in text form, it is converted into text. For example, if the information is only available as an image, it is first processed using OCR and LLM 24032SNBPTEP - Rule-based AI 29.08.2024

[0111] 15 / 56 interprets, meaning the amount of text contained in an image is extracted (OCR) and / or the image is described textually by the LLM. If the LLM used for the rule-based AI also accepts images as input, this conversion step can be skipped.

[0112] Depending on the type of rule map, the query definition can be a natural language description of the characteristic / prerequisite to be checked. It is formulated as a yes / no question, meaning it can only be answered with "yes" or "no," for example, "Is characteristic X present?" or are the values ​​within specific ranges or intervals?

[0113] The LLM determines whether the information passed in the specific case matches the query definition of the node. If so, the result of the logic check is, for example, "1", and the node is set to "given" (Error! Reference source not found). If the result of the logic check is "0", the node is set to "not given" (Error! Reference source not found). The colors shown here are only examples; other representations can also be used.

[0114] After a node is set to either "given" / true or "not given" / false, the rule map automatically checks, based on the logical link with the preceding parent node / inner node, whether the parent node is "given" or "not given". Depending on the link type used, the AI ​​determines which end node / leaf node must be checked next. The technical procedure for these three logical link types is as follows: 24032SNBPTEP - Rule-based AI 29.08.2024

[0115] 16 / 56

[0116] Technical procedure for "AND" conjunction:

[0117] If all child nodes are set to given / true / 1 in the "AND" operation, the parent node is set to "given" as a logical sum (Error! Reference source not found.). If even one node is set to "not given", the parent node is also set to "not given" (Error! Reference source not found.).

[0118] The automated check for the "AND" conjunction preferably follows these steps:

[0119] Step 1: The top node is checked and, if affirmed, set to "given".

[0120] Step 2: The next node is checked and, if confirmed, set to "given".

[0121] Step 3: The process continues until a logical result is found for the parent node: If a node is set to "not given", no further check takes place because the parent node is already set to "not given" (Error! Reference source not found.).

[0122] Technical procedure for the "non-exclusive OR":

[0123] If at least one node in the "non-exclusive OR" operation is set to "given", the parent node is considered the logical node.

[0124] The sum is set to "given" (Error! Reference source not found). If all child nodes are set to "not given", the parent node is also set to "not given" (Error! Reference source not found). The technical procedure follows the following: 24032SNBPTEP - Rule based AI 29.08.2024

[0125] 17 / 56

[0126] Steps (Error! Reference source not found.) :

[0127] - All end nodes in a "non-exclusive OR" are checked and either set to "given" or "not given".

[0128] Then, according to the logical sum of the child nodes, the parent node is set to either "given" or "not given".

[0129] Technical procedure for the “Outflowing Oder”:

[0130] With a "separate OR" logic, only one child node can logically be set to "given". If a child node is set to "given", further child nodes become inactive (grayed out) and are no longer checked; the parent node is set to "given" (error!).

[0131] (Reference source could not be found.)

[0132] If all child knots are set to "not given", the

[0133] Mother node also set to "not given" (Error!

[0134] (Reference source could not be found.)

[0135] The technical procedure follows these steps.

[0136] (Error! Reference source could not be found.)

[0137] The children's knots are checked from top to bottom.

[0138] If a node is set to "not given", the next node is checked.

[0139] If a node is set to "given", no further nodes will be checked. 24032SNBPTEP - Rule-based AI 29.08.2024

[0140] 18 / 56

[0141] The parent node is set to either "given" or "not given" according to the logical sum of the child nodes.

[0142] The process of identifying the nodes and passing the explanatory definitions to the AI ​​via prompt is repeated until all the end nodes of the rule map necessary for an overall test result have been checked, so that the starting node is automatically set to "given" (Error! Reference source could not be found.) or "not given" (Error! Reference source could not be found.), meaning the test question can be answered yes or no.

[0143] In principle, other operators can also be used, such as range operators that determine whether a result lies within a specific range. Such a query could, for example, be given as a token or query at a leaf node. The invention is not limited to the Boolean operators AND, OR, XOR, and NOT.

[0144] Rule-based AI can be used for automated checks of facts in rule-based contexts. Typical rule-based contexts include compliance checks, legal rules of all kinds, process rules, medical rules (guidelines), technical descriptions and procedures, operating instructions for machines, and other scientific rule-based procedures.

[0145] The primary advantage of the invention lies in the fact that, with the aid of Rule-based AI, rule-based systems can be combined with LLM. Automated testing is performed (24032SNBPTEP - Rule-based AI, August 29, 2024).

[0146] 19 / 56 can be carried out transparently and comprehensibly based on predefined rules. The examination of the individual characteristics and prerequisites of a goal-oriented rule is performed individually and atomically by the artificial intelligence for each case passed to the machine. The process is controlled by a rule map as a visual decision tree. Thus, one possibility of the invention is to enable visual representation on a computer screen. The rule tree is displayed graphically and contains all its results for the specific case being examined, also visually. The logical connections and their results are displayed in different colors, so that it is immediately clear to a user how the result was obtained. They can quickly understand the logic.In contrast to purely rule-free AI, this approach creates transparency and traceability of the results, which can also lead to better outcomes. The invention allows the AI ​​to answer questions where necessary and then determines the result rule-based, using the logical sum of the answered questions (logical sum of affirmative and negative elements).

[0147] The functionality of rule-based AI can be illustrated using the following example:

[0148] In a legal case with the following facts, it must be examined whether B has a claim for damages against A:

[0149] A slaps B hard in the face during an argument. Attempting to dodge the slap, B falls and is seriously injured. Due to this unexpected serious injury, B incurs treatment costs of €5000. 24032SNBPTEP - Rule based AI 29.08.2024

[0150] 20 / 56

[0151] Step 1: To examine the claim in the given case, a rule map is created as a first step (Error! Reference source not found.) that visualizes the desired rules for examining the claim. The rule map begins with the result in the start node (Error! Reference source not found., No. 1). The features for this claim are represented as child nodes (Error! Reference source not found., Nos. 2-7). The features in (Error! Reference source not found., Nos. 2-7) are linked as cumulative prerequisites (Error! Reference source not found., No. 8), because the rule question in the start node can only be answered affirmatively if all prerequisites are met. The prerequisites of Figure 19 No. 6 ("fault") in (Error! Reference source not found., No. 9) and (Error! Reference source not found., No. 1) are linked as cumulative prerequisites.10 are linked with “exclusive or” (Error! Reference source could not be found., No. 10) because fault can be either intent or negligence, but not both.

[0152] Step 2: Definitions are inserted as query / token definitions for all end nodes. For example, the error message "Error! Reference source could not be found" shows a definition (Error! Reference source could not be found. No. 12) of the end node "Vorsatz" (Error! Reference source could not be found. No. 9).

[0153] Step 3: A process is initiated that performs a machine-based, automated check. The check is carried out from top to bottom and from right to left, i.e., from leaf to root. It should be noted that the tree may have multiple roots, which must then be processed sequentially or in parallel. 24032SNBPTEP - Rule-based AI 29.08.2024

[0154] 21 / 56 are. First, the end node marked with number 2 (Error! Reference source could not be found) is checked. The query definition of the node, along with the situation, is passed to the LLM as a prompt. The LLM compares the situation with the query definition and returns that the characteristic is present in this case. The end node is marked as "present" (Error! Reference source could not be found).

[0155] Step 4: The same procedure is repeated for the nodes marked with numbers 3, 4 and 5 indicating an error! Reference source could not be found: The respective query definition is passed to the LLM step by step, along with the relevant information, as a prompt.

[0156] The LLM compares the situation with the query definitions and returns that these characteristics are present in this case. The checked end nodes are marked as "present" one after the other (Error! Reference source could not be found.).

[0157] Step 5: To decide whether the node marked with No. 6 (Error! Reference source not found) should be set to "given" or "not given," its child nodes are checked. First, the definition from the node marked with No. 8 (Error! Reference source not found) is passed to the LLM along with the factual information. The LLM compares the factual information with the definition and returns that this feature is given. The node marked with No. 9 (Error! Reference source not found) is set to given. The node marked with No. 9 (Error! Reference source not found) is deactivated due to the applied "exclusive OR" operation (Figure 20 No. 11). 24032SNBPTEP - Rule based AI 29.08.2024

[0158] 22 / 56

[0159] The node marked with number 6 (Error! Reference source could not be found) is set to "given" (Error! Reference source could not be found.).

[0160] Step 6: The node marked with the error "Reference source could not be found" (No. 7) is checked. Its query definition, along with the factual information, is passed to the LLM via prompt. The LLM compares the factual information with the query definition and returns that this characteristic is present in the case being checked. The node is then marked as "present" (Error! Reference source could not be found, No. 1).

[0161] Step 7: The starting node 1 is set to "given" (Error! Reference source not found., No. 2). The check by "Rule-based AI" is complete.

[0162] Example handover of Rulemap AI:

[0163] The criminal relevance of the following matter will be examined:

[0164] On March 15, 2024, at 2:45 PM, a user on the social media platform TikTok posted the following: "Merkel How did this bitch even come up with the idea of ​​changing things so drastically? She should have just let all those stupid foreigners die. #Merkelmustgo refugeesnotwelcome". This post received a total of 5,000 likes and was shared 1,200 times by March 20, 2024. The comments under the post were mixed, with some users signaling agreement. 24032SNBPTEP - Rule based AI 29.08.2024

[0165] 23 / 56 while others vehemently protested against the choice of words and the opinion expressed therein.

[0166] A Trusted Flagger, who regularly reviews content on the platform, reported the post on March 21, 2024, at 10:30 a.m. on suspicion of illegality. The user who created the post had 20,000 followers at that time. The Trusted Flagger argued that the post was illegal due to hate speech and violations of the German Criminal Code's provisions against incitement to hatred. This assessment referred to the derogatory term used against a public figure and the derogatory statement directed at a group of people based on their origin.

[0167] TikTok decided on March 25, 2024, not to remove the post, arguing that the platform was a space for free expression and that the post did not violate its internal freedom of expression guidelines. This decision led to further debate and discussion within the community.

[0168] The rule map depicts – among numerous other offenses – the offense of incitement to hatred according to § 130 of the German Criminal Code. (Figure 27)

[0169] Through the automatic interaction of Ruelmap and LLM, the special variants of § 130 StGB were already examined and denied, the element of the offense “incitement to hatred” was affirmed, as was the element of the offense “mode” of the superior element “suitable action”.

[0170] The rule map now specifies that the second step is next.

[0171] The element of the offense, namely the "disturbance of public life" 24032SNBPTEP - Rule based AI 29.08.2024

[0172] 24 / 56

[0173] The "peace" status must be checked. For this purpose, the rule map automatically calls a prompt of the following type via API:

[0174] Option 1: Simple yes / no test

[0175] Global settings per rulemap. The global settings per rulemap in the context of interest here are the settings for the model, system prompt, and user prompt—that is, the entirety of the components of the query described below.

[0176] Definition of the element of the offense (end node, leaf) "disturbance of the public peace" against which the facts are examined. The following is an example of a node call. Here, `list:model` represents the LLM, and `max_token` defines the size of the return value. 1 token represents 1-3 characters; `:system` represents the system prompt; and `:user` represents the user prompt.

[0177] (ask-openai

[0178] (list :model "gpt-4-1106-preview"

[0179] : temperature 0

[0180] :max_tokens 1)

[0181] The system states: "You are a German judge and must examine the elements of a crime. We will proceed step by step, examining each element of the crime in separate prompts."

[0182] Note that the [case] may contain errors due to OCR recognition. Furthermore, the case lacks any formatting. 24032SNBPTEP - Rule-based AI 29.08.2024

[0183] 25 / 56

[0184] Your task will be to solve the following [element of the offense] in relation to the [facts].

[0185] Think step by step. If the [element of the offense] involves several steps, address them sequentially in a causal order. If an [element of the offense] includes different variations, examine each one individually for its applicability.

[0186] - Your answer must consist exclusively of '1' for YES or '0' for NO."

[0187] : user

[0188] [Start of the case]

[0189] On March 15, 2024, at 2:45 p.m., a user on the social media platform TikTok posted the following: "#Merkel How did this bitch even come up with the idea of ​​changing the country so drastically? She should have just let all those stupid foreigners die. #Merkelmustgo refugeesnotwelcome"

[0190] This post received a total of 5,000 likes and was shared 1,200 times by March 20, 2024. The comments under the post were mixed, with some users signaling agreement, while others vehemently protested against the wording and the opinion expressed.

[0191] A Trusted Flagger, who regularly checks content on the platform, reported the post on March 21, 2024 at 10:30 AM due to suspected violation of 24032SNBPTEP - Rule based AI 29.08.2024

[0192] 26 / 56

[0193] Illegality. The user who created the post had 20,000 followers at the time. The Trusted Flagger argued that the post was illegal due to hate speech and violations of provisions against incitement to hatred under the German Criminal Code. This assessment referred to the derogatory term used against a public figure and the derogatory statement made about a group of people based on their origin.

[0194] TikTok decided on March 25, 2024, not to remove the post, arguing that the platform is a space for free expression and that the post did not violate its internal freedom of expression guidelines. This decision led to further debate and discussion within the community.

[0195] [End of facts]

[0196] [Start element of the offense]

[0197] It is questionable whether the content is likely to disturb the public peace.

[0198] Public peace encompasses the state of general legal certainty and peaceful coexistence among citizens, as well as the population's awareness of living in tranquility and peace.

[0199] Content is likely to disturb the public peace if there is reason to fear that it will undermine confidence in public legal certainty. Actual disturbance of the public peace is not required. 24032SNBPTEP - Rule-based AI 29.08.2024

[0200] 27 / 56

[0201] A high intensity of the attack, the size and homogeneity of the group affected by the statement, and especially the lack of social integration of the affected segment of the population can indicate a disturbance of the public peace. The younger the addressees of the statement, the more likely a disturbance of the public peace becomes. For a statement to be considered capable of disturbing the public peace, it is sufficient if the insulting statement reaches only a single adherent of the affected religious denomination, provided that, according to the specific circumstances, it can be expected that the insult will become known to a wider public.

[0202] Good social integration of the targeted group, as well as the lack of receptiveness of critically minded recipients of the statement, can argue against a disturbance of public peace. This is the case, for example, with a hate-inciting letter to the editor whose publication was hardly to be expected, or if the statement comes from a young person or an adult not considered serious. A critical article published in a generally critical journal, whose readership showed no discernible risk of promoting intolerance, is not likely to disturb public peace.

[0203] [End of element of the offense] 24032SNBPTEP - Rule-based AI 29.08.2024

[0204] 28 / 56

[0205] The prompt, as shown above, includes the LLM (model) "gpt-4-1106-preview", the temperature, and the number of tokens passed. The temperature represents the randomness of the result; a high temperature indicates high randomness. For most examples, the randomness is set to 0. However, there may be other situations that require a different value. Furthermore, the prompt includes the system prompt and the user prompt, each introduced by `system:` and `user:`, respectively. The user prompt is further divided into the sections "Fact" and "Characteristic." Characteristics are predefined at each leaf node, while the system prompt is the global setting that applies to the entire rule map / rule tree. In this case, the system prompt references both the fact and the characteristic. The fact is redefined for each iteration of the rule map and automatically inserted into the prompt.

[0206] The LLM returns the requested test result via API, the rule map stores it, and, following the rule map's logic, generates the next prompt in the same way, i.e., from global settings, the specific situation, and the definition of the logically next node in the rule map. In this way, the rule map calls an automatic prompt of the structure described above end node by end node via API, stores the test result, and determines the next test program.

[0207] Variant 2: Justified review 24032SNBPTEP - Rule-based AI 29.08.2024

[0208] 29 / 56

[0209] To also record the rationale for each individual test step in the rule map, a prompt with the following structure is generated:

[0210] (ask-openai

[0211] (list :model "gpt-4-1106-preview"

[0212] : temperature 0

[0213] :max_tokens 900)

[0214] The system says: "You are a German criminal judge and must examine the elements of an offense. We will proceed step by step, examining each element of the offense in separate prompts."

[0215] Please note that the [case details] may contain errors due to OCR recognition. Furthermore, the case details lack any formatting.

[0216] Think step by step. If the [element of the offense] involves several steps, address them sequentially in a causal order. If an [element of the offense] includes different variations, examine each one individually for its applicability.

[0217] Your task will be to solve the following [element of the offense] in relation to the [facts].

[0218] Your answer must contain 2 parameters.

[0219] 1. A binary answer, 1 for yes, 0 for no.

[0220] 2. A brief explanation of why you made this decision. The explanation must be short and 24032SNBPTEP - Rule based AI 29.08.2024

[0221] The written justification (30 / 56) must be concise and no longer than 250 characters. It must address the specific reasons for the decision arising from the facts of the case and identify which arguments related to the elements of the offense led to the decision. If the reasoning is entirely obvious, meaning you are very certain of the outcome, keep the justification brief or omit it altogether. If you are less certain, provide a more detailed explanation.

[0222] Syntax: 0 / 1 'REASONING'. Your answer must look exactly like this. There must be no quotation marks or similar around the 0 / 1.

[0223] userresult: "If several different sub-issues are present in the facts of a case, consider only the facts as a whole and do not divide them into different sub-issues in your decision or in the reasoning. 0 = No; 1 = Yes. The mandatory syntax is 1 OR 0 followed by the reasoning."

[0224] Example :

[0225] 0 B's liability under Section 823 Paragraph 1 of the German Civil Code (BGB) is not applicable, as he is not responsible for his conduct in accordance with Section 276 Paragraph 2 of the German Civil Code (BGB).

[0226] The difference lies solely in the global settings; the facts and the definition of the elements of the offense remain the same. The system prompt specifies that the response must consist of a binary value (logical value) and a textual explanation. These are global parameters, 24032SNBPTEP - Rule-based AI 29.08.2024

[0227] 31 / 56 are passed to each system prompt in the rule map. The rule map stores not only the return result but also the justification in an object at the end node. From the sum of these justifications, the rule map then generates a fully reasoned decision.

Claims

24032SNBPTEP - Rule based AI 08 / 29 / 2024 32 / 56 Claims:

1. A method for rule-based integration of a Large Language Model (LLM) comprising a rule tree (rulemap) whose internal nodes represent logical links and whose leaves contain prompts for passing to the LLM, wherein each prompt is composed of a system prompt and one or more user prompts, wherein user prompts comprise a specific situation and a query definition that determine the form of the result and are used in the rule tree at the next internal node to the prompt in the logical link, wherein the system prompt defines the context from globally specified settings and information that are predefined for the specific rulemap, wherein the prompts thus generated are passed to the LLM to generate a logical response that is suitable for evaluation in the rule tree, wherein the rule tree is traversed from the leaves to the one or more roots.to produce a logical final result.

2. The method according to the preceding claim, wherein the user prompt consists of the facts and the query definition, wherein the query definition is specifically specified for each sheet and the facts are automatically added during the processing of the rule tree.

3. The method according to the preceding claim, wherein the LLM determines whether the passed-in fact matches the query definition of the sheet. 24032SNBPTEP - Rule based AI 08 / 29 / 2024 33 / 56 4. The method according to the preceding claim, wherein if the facts are not in text form, they are automatically converted into text form, preferably by OCR.

5. The method according to any of the preceding claims, wherein the logical connections within the rule tree are one or more of: and, or, x- or, not.

6. The method according to one of the preceding claims, wherein the system prompt refers to the facts and / or the query definition.

7. The method according to one of the preceding claims, wherein a visual representation of the decision tree is displayed on a computer screen, which reveals which results are present at the nodes, in particular by means of text or graphic representations.

8. The method according to one of the preceding claims, wherein the rule tree has several roots which are processed successively from the leaves to the root in a predetermined sequence, preferably also in parallel.

9. Computer device with at least one Processing unit and storage area configured to perform a method according to any one of claims 1 to 8.

Images