Machine learning legal natural language processing and generation

Abstract

A machine learning legal natural language generation computing system for developing a pre-trained and instruction fine-tuned MLLNLG generative language model that aligns with human instructions and preferences using causal language modeling for next-token prediction. The training method employs a scaled version of RoPE to compress token position indices, to manage longer sequences beyond the computing hardware devices (GPU's) DRAM memory capacity. This system is optimized for various legal NLP and NLG tasks by receiving input tasks from users through an I / O component. The system enables efficient pre-training of a generative legal language model from scratch on a single computing hardware device (GPU), achieving an efficient MFU of 41.35 and through weight tying, the system reduces model parameters compared to LLMs, eliminating the need for multiple GPUs. Tailored for the legal domain, the model excels in processing legal jargon and demonstrates advanced reasoning capabilities, outperforming existing models in summarization tasks.

Description

TITLE OF THE INVENTIONMACHINE LEARNING LEGAL NATURAL LANGUAGE PROCESSING AND GENERATIONFIELD OF INVENTION

[0001] The present invention relates to method, system, devices, computer readable media for Machine Learning Legal Natural Language Generation (MLLNLG) model for executing multitude of legal natural language processing (NLP) and natural language generation (NLG) tasks.CROSS REFERENCE TO RELATED APPLICATIONS

[0002] The present application claims the benefit of Indian Patent Application No. 202431039738 filed 21 May 2024 (21-05-2024), said application being hereby incorporated herein in its entirety by reference.BACKGROUND OF THE INVENTION

[0003] The domain of legal language presents a distinct landscape compared to everyday conversation. It's characterized by a unique lexicon, legal jargon, the presence of Latin phrases, and the utilization of words in unconventional ways, all of which contribute to its distinctive syntax and semantics. Conventional language models, designed to function across various domains, fall short when it comes to meeting the demands of the legal field. A specialized legal domain language model is essential for efficiently addressing legal tasks. Attempting to employ conventional language models for legal purposes results in challenges such as prolonged inference latency, heightened training costs, high requirements of multiple computing hardware devices (GPUs), and issues with tokenization not optimized for specialized legal terminology.

[0004] One approach to address this gap is through the domain adaptation method applied to large language models (LLMs) for legal language processing. However, this approach is not without its own set of challenges. Problems such as training set underfitting, test set inseparability, training-test misalignment, and the misalignment of domain specialised tokenizers and embeddings with the existing embeddings of large language models (LLMs) via continual pre-training with vocabulary expansion of the existing LLMs tokenizers hinder the effectiveness of this adaptation. The primary issue lies in Data Distribution Mismatch, where there exists a significant difference in data distribution between the source and target domains. This incongruity leads to models performing admirably within the source domain but faltering when applied to the target domain. Aside, LLMs often suffer from catastrophic forgetting due to continual pre -training of LLMs for domain adaptation. LLMs can indeed suffer from catastrophic forgetting when they are continually fine-tuned or pre-trained on new data, especially in the context of domain adaptation.

[0005] In the context of domain adaptation, if a LLM is continually pre-trained on data from various domains without proper mechanisms like experience replay to retain knowledge from previous domains, it may forget crucially linguistic patterns or domain-specific knowledge acquired during earlier pre-training phases. This can lead to a degradation in performance when the model is applied to task in those previous domains.

[0006] The prior art [CN 116822591] discloses a method that trains a legal domain generative model by initially pre-training it on unstructured legal and general Chinese text, producing a fine-tuned model. Thismodel is then refined for specific legal subfields using a specialized instruction dataset and low-rank matrix increment weight training. A reward model is created with a grading sequencing dataset and the refined model.

[0007] The prior art [CN118093851] discloses an automatic question-answering and legal task processing method that includes acquiring and analyzing task description information to extract key details. These key details help in retrieving relevant reference data from a pool of candidates. The retrieved data is then fed into a task processing model to generate results for the target analysis task.

[0008] The non patent literature prior arts like Chinese legal Llawyer Llama (Huang et al., 2023), a LLM like Llama continually pre-trained on a large-scale legal dataset, and ChatLaw (Cui et al., 2023), a legal LLM with integrated external knowledge bases. Both said models rely on retrievers to extract relevant external knowledge and generation via retrieval augmented generation using LLMs

[0009] The evaluation of law-specific LLMs necessitates benchmarks that focus on legal tasks. The LegalBench (Guha et al., 2023) and LawBench (Fei et al., 2023) are collaboratively constructed benchmarks designed for legal reasoning in the contexts of U.S. law and Chinese law, respectively. However, given that legal laws and processes differ significantly across countries, developing a single global evaluation benchmark for LLMs is impractical for precise legal evaluation of LLMs.

[0010] Instead of using domain adaptation methods for modeling legal language in legal documents, the MLLNLG model offers a novel solution. This specialized, lightweight language model is trained from scratch specifically for the legal domain. It features significantly fewer parameters and requires considerably less training time compared to large language models (LLMs). As a result, it is more environmentally friendly and addresses the limitations of existing LLMs in the legal field. This approach effectively reduces high inference latency, lowers training costs, and overcomes the challenges associated with traditional LLMs. and the challenges associated with non-specialized tokenizers, and the misalignment of domain specialised tokenizers and embeddings with the existing embeddings of large language models (LLMs). Additionally, it supports the training of larger models on a single computing hardware device (GPU), thereby reducing training costs and minimizing the catastrophic forgetting issues that are common with LLMs. Tailored for various legal NLP and legal NLG tasks, this model operates with fewer parameters (under 7 billion) to function effectively as an end- to-end legal artificial intelligence assistant.

[0011] The primary objective of the present invention is to develop a legal domain-specific MLLNLG model from scratch that efficiently performs various legal NLP and legal NLG tasks. Instead of using domain adaptation methods of LLMs for modeling legal language in legal documents, this novel solution introduces a specialized, lightweight generative language model tailored specifically for the legal domain.

[0012] Another objective of the present invention is to develop a method to train MLLNLG models with longer sequence length context size on a single computing hardware device (GPU) to overcome the existing challenges of the existing LLMs.

[0013] A further objective is to design a method that requires significantly fewer model parameters compared to current LLMs, thereby reducing inference latency.

[0014] Yet another objective of the present invention is to develop a method that requires limited physical DRAM memory during the pre-training process. This innovation enables CPU inference instead of relying on multiple computing hardware devices (GPUs), making the generative process lightweight, faster, more cost- effective, and efficient, even in constrained resource environments with a single computing hardware device.

[0015] Yet another objective of the present invention is to develop an end-to-end legal Al assistant that is capable of excelling in both generative and discriminative tasks.SUMMARY OF THE PRESENT INVENTION

[0016] The present invention discloses a system tied to computer hardware designed for developing both a pre-trained machine learning legal natural language generation (MLLNLG) model and an instruction fine-tuned MLLNLG model aligning with human instructions and preferences using causal language modeling of nexttoken prediction, optimized for various legal natural language processing (NLP) and legal natural language generation (NLG) tasks by receiving input tasks and queries from human users via an I / O component. The system is further configured with a detailed data processing pipeline that includes specific engines for data collection, cleaning, deduplication, tokenization, and vectorization, followed by a structured pre-training phase that involves hyperparameter tuning and performance evaluation.

[0017] This system enables the efficient pre-training of a MLLNLG model from scratch on a single computing hardware device (GPU). It achieves a high Model FLOPs Utilization (MFU) of 41.35 through a scaled version of RoPE embedding technique, which compresses the position IDs or indices of tokens using a shrinking factor. Additionally, it employs weight tying by sharing weights between embeddings and so f tmax layers of the MLLNLG model. This reduces the number of model parameters compared to existing LLMs, thereby lowering both training and inference costs and eliminating the need for multiple computing hardware devices (GPUs), for training and inference from MLLNLG model. Unlike traditional LLMs inference, this approach eliminates the need of model’s weight quantization to lower numeric precision for inference on a single hardware device (GPU), preserving model performance. It supports low-latency inference, even on CPUs, without requiring the need for multiple computing hardware device (GPU). Tailored specifically for the legal domain, the model excels in processing legal j argon, predicting legal case judgment explanation outperforming LLMs like Gemini Pro, Llama and ensuring consistent long-form summarization for sequences ranging from 1024 to 8192 tokens. Additionally, the MLLNLG model demonstrates advanced capabilities in legal reasoning, quantitative reasoning and comprehension, making it a powerful tool for legal professionals seeking efficient, high- performance natural language processing solutions within a constrained computing hardware environment. Overall the system functions as an end-to-end legal Al assistant, excelling in both generative and discriminative tasks

[0018] The present invention also discloses a computer implemented method to train MLLNLG model for performing both legal NLP and legal NLG tasks. The method comprising: a. data curation process through pretraining data curation engine and tokenization engine further comprising; collecting a pretraining data corpus using a legal data crawling engine that crawls legal specific data, such as court case documents, country specific legislative constitution documents, and legal penal code documents; storing the collected data in a legal data storing device; cleaning and pre-processing the stored data through a legal data cleaning engine, which removes noisy text, including unreadable characters, special characters, excessive white spaces, case numbers, judge names, and dates, through the application of regular expressions; removing duplicate entries in the web scrapped pretraining corpora using a legal data deduplication engine; storing the cleaned and deduplicated curated data in the legal data storage; tokenization, by a data tokenization engine, performing training of BytePair encoding (BPE) algorithm using Sentence piece module, developing specialized tokenizer for legal domain to learn intricate legal jargon and terminology, in the process treating data as a sequence of bytes considering each bytes as tokens and eliminating duplicate tokens from the tokenizer, thereby forming specialized, compact, effective and optimized tokenizer; the size of the legal domain specialized tokenizer can vary between 2000 and 64000; data vectorization, preprocessed data converted into numerical vectors using scaled version of Rotary Position Embedding (RoPE); b. pre-training process through pre-training engine where the pre-training of autoregressive MLLNLG models with processed vectorized training / pre-training data using language modelling objective of next token prediction, i.e., predicting the next token in a sequence given the preceding tokens, by minimizing cross-entropy loss between the predicted probability distribution of tokens over the vocabulary and the actual next token, the MLLNLG model learns to assign higher probabilities to tokens that are more likely to occur next in the sequence; performing hyperparameter-tuning on smaller MLLNLG model (less than 50 million parameters in size) to determine optimal vocabulary size, learning rate and warmup ratio, and transferring the learned hyperparameters to the bigger (less than 7 billion parameters) MLLNLG models through maximum update parameterization (pP); and undergoing performance evaluation based on validation perplexity metric, on model FLOPs Utilization (MFU) metric for computing hardware resource (GPU) utilization and CPU inference speed of MLLNLG model in FP32 (floating point) numeric precision; second evaluation of pretrained MLLNLG legal model on zero-shotcase judgement prediction with explanation task, and legal case summarization task at varying degree of length (<=8192 tokens) of abstractive summary generation; c. instruction fine-tuning process the pre-trained MLLNLG models using instruction fine-tuning dataset which is generated from legal instructions data triplet generation engine, results in a instruction finetuning dataset comprising of 10,763 unique instructions covering multitude of legal NLP tasks, legal instructions dataset was split into training and testing datasets respectively; and reporting of training and test loss while training with different learning rate scheduler such as cosine, constant and linear learning rate scheduler on the legal instructions training dataset respectively; and reporting of training and test loss while training with different learning rate scheduler such as cosine, constant and linear learning rate scheduler on the legal instructions training dataset; and d. deploying process to deploy the pre-trained and instruction fine-tuned MLLNLG models in humancomputer interface for execution of various legal NLP and legal NLG tasks based on human instructions and preferences through a I / O component of the computer hardware system.BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a schematic representation of the exemplary system, according to an embodiment of the present invention.

[0020] Figure 2 is a schematic representation of the operations involved in the data curation process, according to an embodiment of the present invention.

[0021] Figure 3 is a schematic representation of the pre -normalization of transformer block for MLLNLG model used in one of the embodiments.

[0022] Figure 4 is a schematic representation of the grouped query attention mechanism used in one of the embodiments.

[0023] Figure 5 is a schematic representation of the legal instruction data triplet generation engine.

[0024] Figure 6 is a schematic representation of the MLLNLG model end-to-end system, according to an embodiment of the present invention.

[0025] Figure 7 is a schematic representation of the MLLNLG model method flowchart, according to an embodiment of the present invention.

[0026] Figure 8 is a schematic representation of the tasks performed by the MLLNLG model, according to an embodiment of the present invention.

[0027] Figure 9 is a schematic representation of the computing hardware system, according to an embodiment of the present invention.

[0028] Figure 10 is a schematic representation of the implementation of scaled version of RoPE embedding method.

[0029] Figure 11 is a graphical representation of the training loss v / s training steps for one of the MLLNLG pretrained model, according to an embodiment of the present invention.

[0030] Figure 12 is a graphical representation of the training perplexity v / s number of tokens for one of the MLLNLG pretrained model, according to an embodiment of the present invention.DETAILS DESCRIPTION OF INVENTION

[0031] With reference to Figure 1, the MLLNLG model system (100) interacts with user devices (150) over networks (160) that may include wide area networks (WANs), local area network (LANs), or a combination of both. These networks can span intranets, the Internet, or enterprise networks. The system may consist of multiple servers working together to deliver the specified functions. User devices can range from personal computers and tablets to mobile devices, each equipped with one or more processors. The human-computer interface (150) includes an application - either a web browser or a specialized app - that allows users to interact with the MLLNLG model system (100).

[0032] The present invention discloses a computer implemented method for machine learning legal natural language generation (MLLNLG) model for executing various legal NLP and legal NLG tasks, comprising operations such as data curation process through legal data crawling engine, legal data cleaning engine, legal data deduplication engine, legal data storage device, data tokenization engine and vectorization through scaled version of Rotary Positional Embedding (RoPE); data pre-training process through pre-training engine for training and evaluation of the pretrained MLLNLG models; instruction fine-tuning process through supervised instruction fine-tuning; and deploying process to deploy the pre-trained and instruction fine-tuned MLLNLG models in human-computer interface for execution of various legal NLP and legal NLG tasks based on human instructions and preferences through a I / O component of the computer hardware system.

[0033] In data curation process (110), includes the utilization of a pre-training / training data curation engine and data tokenization engine (110A), which is responsible for processing a corpus of legal domain data, which includes court case documents, country-specific constitution documents, and legal penal code documents. These documents are collected using a legal data crawler engine (111) and stored in a legal data storage device (112) for subsequent processing.

[0034] In data curation process (110), the pre-training / training data are collected through the legal data crawling engine (111) that crawls legal specific digital data, such as court case documents, country specific legislative constitution documents, and legal penal code documents using public sources.

[0035] In a preferred embodiment, the pre-training data comprises an extended version of the Indian Legal Document Corpus (ILDC) (referred from article titled “ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation.” (Malik et al., 2021)), supplemented by additional data sourced from various court case documents, country constitutions documents, and diverse laws and codes documents accessible in the public domain using the legal data crawler engine (111).

[0036] In another preferred embodiment, the pre-training data are collected from Supreme Court of India court cases, Constitution of India in English and the Indian Penal Code in the public domain using the legal data crawler engine (111).

[0037] Figure 2 elaborates various operations of pre-training data curation engine and data tokenization engine (110A) of the present method.

[0038] In a preferred embodiment, subsequently after data crawling using the legal data crawler engine (111), the data cleaning and pre-processing of legal domain data are conducted using a legal data cleaning engine (113). Initially, noisy text elements, such as unreadable characters, special characters, extra white spaces, and meta information (e.g., case numbers, judge names, dates), are identified. These noisy elements are then removed using regular expressions. Subsequently, a data deduplication process is performed using a legal data deduplication engine (114) to eliminate duplicate entries and storing of deduplicated cleaned curated data forming pre-training models for data tokenization process.

[0039] Furthermore, in a preferred embodiment, the Tesseract tool is utilized to extract text from PDF files. Sections such as title pages, acknowledgments, table of contents, and other irrelevant information are identified and subsequently removed from the dataset. This comprehensive pre-processing approach ensures that the data is cleansed and formatted optimally for further analysis and causal language modelling pre-training phase (120).

[0040] In another preferred embodiment, tokenization of data is performed through tokenization engine (116) where training of greedy-based subword algorithm, Byte-Pair encoding (BPE) for tokenizer using Sentencepiece module on the curated and cleaned pre-trained data happens, thereby developing specialized tokenizer for legal domain to learn intricate legal jargon and terminology; during the process data is treated as a sequence of bytes considering each of the 256 bytes as tokens and removing duplicated tokens in the legal domain specialized tokenizer. During pre -tokenization, NFC normalization was performed on the processed data, digits are split into individual tokens and fall back unknown UTF-8 characters to byte granularity for improving the arithmetic learning ability of the pretrained legal MLLNLG model. A token, as used herein, may refer to a unit or element of text that is used as a basic building block in natural language processing (NLP) or other machine learning tasks. A text token may represent a separate unit of meaning or linguistic component within a phrase or sentence. For example, a piece of text, such as a word, phrase, sentence, paragraph, or document, may be broken down into smaller units (e.g., text tokens) for further analysis. Text tokens may include, e.g., individual words, punctuation marks, or even smaller sub-word units (e.g., a portion of a word), depending on the specific tokenization strategy employed. Tokenization may refer to a process of splitting a text into tokens, which may be helpful in completing machine learning tasks including text classification, sentiment analysis, machine translation, and named entity recognition. In some embodiments, a token may be generated based on natural language input, such as by transforming the natural language input into a numerical vector representation format understandable to a computerized model.

[0001] In a preferred embodiment, the data vectorization (117), the process of converting the preprocessed data into numerical vectors representation through a scaled version of Rotary Position Embedding (RoPE), enabling efficient numerical processing for subsequent stages of analysis and causal language modelling. The scaled version of RoPE is a form of static relative positional embedding, which rotates the token embedding ina high dimensional space, hence preserving the original information while also incorporating positional knowledge. The intuition behind scaled version of RoPE is that representing the token embeddings as complex numbers and their positions as pure rotations that is applied to them. If both the query and key vectors are shifted by the same amount, changing absolute position but not relative position, this will lead both representations to be additionally rotated in the same manner. Thus, the angle between them will remain unchanged and, thus, the dot product will also remain unchanged.

[0041] By exploiting the nature of rotations, the dot product used in self- attention will have the property for preserving relative positional information while discarding absolute position. In one of the embodiments, the value of theta is set to 10,000.

[0042] The preprocessed tokenized training data gets converts into numerical vectors representation through scaled version of Rotary Position Embedding (RoPE) embedding through a shrinking factor by dividing the target sequence length context size for pre-training / training of MLLNLG model by maximum sequence length context size feasible / permissible on single computing hardware device (GPU) that the MLLNLG model can process on a single computing hardware device for predictions keeping all other hyperparameters fixed such as batch size, vocabulary size; thereby, allowing every positional id’s / indices of the tokens to be divided by the shrinking factor, and be bounded within the maximum feasible / permissible sequence length context size of single computing hardware device; thus, enabling the MLLNLG model to pretrain / train MLLNLG models at much higher / longer sequence length context size than the maximum equivalent DRAM memory availability for single computing hardware device (GPU), making training MLLNLG model at infinite (theoretically maximum bound) sequence length context size on single computing hardware device (GPU) without requiring more than single computing hardware device (GPU). For example, if the maximum permissible sequence length context size for a given single computing hardware device (GPU) physical DRAM memory hardware is 256 keeping all other hyperparameters fixed which means the position / indices of tokens are bounded within 256, then applying shrinking factor of 16 to reach target sequence length context size of 4096 on single computing hardware device (GPU) during pre-training or training for higher / longer context size on single computing hardware device than the equivalent physical DRAM memory required. Then, a token with position ids’ / index = 4000 becomes 4000 / 16 = 250, and the neighbouring token with position ids’ / index = 4001 becomes 4001 / 4 = 250.25, to be within 0 to 256. This novel technique in the implementation of scaled version of RoPE embedding function enables to capture higher context sequence length of tokens that the MLLNLG model can consider for pretraining during pretraining on limited physical DRAM memory required to pre-train / train model at higher / longer context size outside the maximum permissible sequence length context size on single computing hardware device. This novel modification allows to pre-train MLLNLG models from scratch at much higher context size than the physical DRAM memory required for pre-training. Hence, with limited physical DRAM memory and limited computing hardware devices (GPUs), this present architecture canpretrain / train Legal language models from scratch at much higher desired context size. The following is the computer code function definition for the novel modification to scaled version of RoPE: query states’ k y tates= rotarypOsemb(querystates> key states, cos, sin, positionids / shrinking factor

[0043] In the pre-training process (120), causal language modelling objective of next token prediction is performed. The objective of the language modeling is to maximizing the probability of a sequence of tokens w1, w2, ••• , wNwhere P w wq, w2, ••• , W;^) is the probability of token wtgiven the sequence of previous tokens w0, ■" >wt-i 'c-- predicting the next token in a sequence given the preceding tokens, by minimizing crossentropy loss between the predicted probability distribution of tokens over the vocabulary and the actual next token, the MLLNLG model learns to assign higher probabilities to tokens that are more likely to occur next in the sequence.

[0044] The causal language modeling of next token prediction objective by minimizing the cross-entropy loss between the predicted probability distribution of tokens over the vocabulary and the actual next token, i.e., the negative log-likelihood of the observed data under the model under consideration, which for a given dataset is defined as:

[0045] Following the pre-training process (120), the pre-training processed data corpus is utilized to generate pre-trained MLLNLG model through pre-training engine (121). During this phase, hyperparameter-tuning is conducted to determine the optimal settings for vocabulary size, learning rate, and warm-up ratio, particularly focusing on smaller language models (less than 50 million parameters in size). Subsequently, a first performance evaluation process is initiated, which includes assessing validation and training perplexity, measuring the Model FLOPs Utilization (MFU) metric of computing hardware device (GPU) utilization during pre-training / training phase, and evaluating CPU inference speed of MLLNLG model in FP32 numeric precision. Second evaluation of pretrained MLLNLG model on zero-shot case judgement prediction with explanation task, and legal case summarization task at varying degree of length (<=8192 tokens) of abstractive summary generation; This evaluation provides insights into the model's training efficiency, computational resource utilization, legal reasoning, legal documents understanding, legal case judgement prediction with explanation, legal document abstractive summary generation and various legal NLP real-time processing capabilities.

[0046] Lower training and validation cross-entropy loss indicates better performance of the MLLNLG model; however, simply computing the loss may not provide an intuitive understanding of the model's effectiveness. Therefore, perplexity serves as a valuable metric for evaluating the performance of a given legal language model. It is defined as the exponent of the average negative log-likelihood loss across the training and validation samples.

[0047] Linear warmup in conjunction with other learning rate scheduling techniques, such as learning rate decay or cyclical learning rates, to further optimize the training process and enhance model performance (e.g., a 1000 step linear warmup and a cosine learning rate decay). During the warmup phase, the learning rate may initially be set to a low value and may then be linearly increased over a certain number of training steps or epochs. Non-linear learning rate schedulers may also be used. This may allow the model to start with smaller learning rates, which helps it explore the parameter space more effectively and avoid large and potentially harmful updates at the beginning of training when the model's parameters are randomly initialized. By increasing the learning rate linearly (or with any increasing trend), the model may quickly adjust its parameters and adapt to the training data while avoiding abrupt and potentially destabilizing changes. Once the warmup phase is completed, the learning rate may be decreased according to a predefined schedule, such as using a learning rate decay or employing adaptive optimization methods like Adam / Adagrad / Adadelta / Adamax / Adaf actor / Nadam / SGD / RMSprop,- and performing 95%-5% train and test split and run the evaluation loop preferably every 2000 steps of training on the test split of the pretrained data. The machine learning model may be trained for a total of other amounts of tokens, such as millions of tokens (e.g., 1 million, 10 million, 100 million) or even billions of tokens (e.g., 1 billion, 10 billion, 100 billion) or even trillion of tokens.

[0048] The pre-trained machine learning legal language models proceed to undergo an instruction fine-tuning process (130). This involves fine-tuning the pre-trained models using an instruction fine-tuning dataset generated from a Legal Instructions Data Triplet Generation Engine (132). This process yields an instruction fine-tuning dataset containing 10,763 unique instructions that span various legal natural language processing (NLP) tasks. Throughout this phase, training and validation loss are monitored and reported through training using cosine, constant, and linear learning rate schedulers, providing insights into the model's performance across different learning rate schedulers strategies.

[0049] Following the instruction fine-tuning process (130), a second performance evaluation process is initiated to assess various metrics such as clarity, relevance, completeness, and legal reasoning in the responses generated by both the pretrained and Instruction Fine-Tuned MLLNLG models. These metrics are evaluated on a scale ranging from 0 (indicating the poorest performance) to 10 (representing the highest performance). This evaluation aims to gauge the improvements achieved through instruction fine-tuning and to ascertain the overall effectiveness of the models in generating responses for multitude of legal NLP and legal NLG tasks. Thus, demonstrating abilities of MLLNLG models having strong legal reasoning, vast legal domain knowledge,intricacies of legal language understanding, high quality legal language generation capabilities, and high performance in multitude of legal NLP tasks.

[0050] In a preferred embodiment, the MLLNLG models undergo training using pre-training data to produce pre-trained machine learning legal language models via causal language modelling objective by minimizing cross-entropy loss. During this stage, hyperparameter-tuning is conducted to determine the optimal settings for vocabulary size, learning rate, and warm-up ratio. This meticulous tuning process ensures the models are finely calibrated for optimal performance in multitude of legal language processing and legal language generation tasks. A hyperparameter is a machine learning parameter whose value is chosen before a learning algorithm is trained. For pre-training a 95%-5% data split is performed for training and testing. A first performance evaluation process was conducted based on validation perplexity of MLLNLG models, on model FLOPs Utilization (MFU) metric and CPU inference speed in FP32 numerical precision. Large language models (LLMs) are expensive for hyperparameter-tuning due to their humongous number of model parameters and large-scale dataset makes hyperparameter tuning difficult and expensive. This problem is eliminated by transferring the learned hyper tuning from small to bigger models using maximal update parameterization ( / / P). In a preferred embodiment, a batch size of 8, gradient accumulation steps of 8, and the maximum sequence length set to 8192., i.e., 524,288 tokens per iteration. Using the pP transfer, the learned hyper parameters transferred to our bigger model from smaller models (less than 50 million parameters).

[0051] In another preferred embodiment, the learning rate (Zr) decay steps is set to maximum steps of training and the minimum Ir is set nearly to 0. 1 • Ir . The Ir schedule starts with a linear warm-up from 0 to the maximum Ir at 1000 steps, followed by a cosine decay to the minimum Ir until the end of an epoch of training. The lrdecayratiois defined as follows:where t is the current training step.The maximum learning rate (Zr) is set to 3 X 10-3(max), weight decay to 1 X 10-1.

[0052] In a preferred embodiment, the instruction fine-tuning process (130) involves splitting the dataset into a 90%-10% training and testing set. During this stage, supervised full fine-tuning of the pre-trained language models is executed for three epochs using the accumulated 10,763 instructions. This instruction triplet-labelled dataset corpus is generated from the Instructions Data Triplet Generation Engine (132), encompassing 10,763 unique instructions covering various legal NLP and legal NLG tasks. The training and validation loss are reported using cosine, constant, and linear learning rate schedulers, with the cosine learning rate scheduler resulting in the lowest validation loss being selected. Furthermore, a second performance evaluation process is conducted, which assesses the responses generated by the MLLNLG models focusing on clarity, relevance, completeness, and legal reasoning metrics of both the pre-trained and Instruction Fine-Tuned models in a scaleof 0 (worst) to 10 (best) by Open Al GPT-3.5-Turbo Al Engine. This evaluation provides insights into the effectiveness of the instruction fine-tuning process in enhancing the MLLNLG models' performance across these key metrics and aligning with the specific requirements and nuances of multitude of legal NLP and legal NLG tasks.

[0053] In a preferred embodiment, to enhance training stability, pre -normalization is applied to the input of each transformer sub-layer, the attention layer, and the feedforward layer. This normalization process standardizes the input vector and corresponding gradients, resulting in accelerated pre-training and training of the MLLNLG model. Moreover, this approach offers computational simplicity while effectively improving training stability. To improve the training stability, normalization of the input of each transformer sub-layer, the attention layer and the feedforward layer was performed with RMSNorm. RMSNorm accelerates the training and inference with similar performance in these large models. RMSNorm can achieves comparable performance and save training and inference time by 7% - 64% (referred from article titled “Do transformer modifications transfer across implementations and applications?” (Narang et al., 2021)). RMSNorm improves the pre-training speed by 5% compared with the LayerN orm baseline. RMSNorm normalizes the activations based on their root mean square (RMS) value instead of normalizing the inputs based on their mean and variance. RMSNorm rescales the input vector and the corresponding gradients. It is defined by the following equation,where epsilon is a small factor to avoid division by zero. In the current embodiment, £ is set to 1 X 10-5.

[0054] In a preferred embodiment, the pre-trained and instruction fine-tuned MLLNLG model uses RMSNorm as pre-normalization layer, uses SwiGLU activation function as non-linearity by replacing the standard Re LU nonlinearity activation function. With reference to figure 4, it uses grouped query attention (GQA) divides query heads into G groups, each of which shares a single key head and value head. GQA-G refers to grouped-query with G groups, and total self-attention heads is divided into G groups, where key and value heads are shared within each group (the value of G is less than the number of self-attention heads) or may use Multi-head self-attention mechanism or may use Multi-Query attention where different attention heads share a single set of keys and values; GQA-1, with a single group and therefore single key and value head, is equivalent to multi-query attention (MQA), while GQA-H, with groups equal to number of heads, is equivalent to multi-head attention (MHA).

[0055] In the present invention, the MLLNLG models sharing weights between embeddings and softmax layers through weight-tying mechanism, thereby reducing the number of MLLNLG model parameters compared to existing LLMs, and thus lowering training and inference costs, and eliminating the requirements of multiple computing hardware devices (GPUs) for training and inference from MLLNLG model.

[0056] In the deploying process (140), the pre-trained and instruction fine-tuned MLLNLG models are deployed for real-time usage in a human-computer interface (150) such as computer hardware, cloud servers, edge-devices, mobile phones, smart watches and other natural user interface (NUIs). This interface facilitates the execution of multitude of legal NLP and legal NLG tasks on human instructions and preferences through an I / O component ( 185) of the computer hardware system (180). Through seamless interaction with the system, users can access the advanced capabilities of the MLLNLG models to address various legal challenges efficiently and effectively in real-time scenarios.

[0057] With reference to figure 6, MLLNLG model capable of taking feedback from the user via the UX interface (150) and this feedback goes through the model’s learned weights / abilities through Reinforcement Learning from Human Feedback mechanism (170) and improves the model’s learning abilities and alignment with human users preferences through the feedback from the user. Reinforcement learning is a paradigm in which an agent learns to make decisions by receiving feedback from its environment. In the context of MLLNLG models, this feedback is provided by human reviewers who assess and rate the model’s responses. By leveraging human expertise and judgments, reinforcement learning facilitates the iterative improvement of the model’s performance and fine-tunes its responses.

[0058] The process of reinforcement learning by human feedback involves several important steps: (i) Guidelines are defined to guarantee unique criteria when deciding what is a good and a bad answer to an input; (ii) A Reward Model (RM) should be trained, which will evaluate each of the responses in terms of accuracy, relevance, and adherence to guidelines; (iii) Train the RM, some prompts are selected and sent to human reviewers generally referred as Preference Data (PD); (iv) The reviewers then interact with the model and manually evaluate and rate the corresponding outputs; (v) The collected feedback, in the form of ratings or rankings, is used to train the RM; (vi) With the trained RM, train a Policy Optimizer, a required component which will guide the further fine-tuning of the instruction fine-tuned model to align with human preferences; (vii) fine-tune the instruction fine-tuned model with Policy Optimization; (viii) This iterative feedback loop allows the model to gradually learn from human guidance and refine its behavior accordingly.

[0059] In another preferred embodiment focused on instruction fine-tuning process (130), particular attention was dedicated to addressing the nuances of multitude of legal NLP and legal NLG tasks. This involved customizing instructions to suit the intricacies of legal tasks and meticulously evaluating the models’ proficiency in executing these tasks. Through this process, the MLLNLG models were trained to accurately interpret and respond to natural language legal instructions provided via a human-computer interface (150), through an I / O component (185) of the computer hardware system (180). The execution of instructions encompassed a diverse array of legal tasks, ranging from document analysis to legal reasoning and decisionmaking. Each instruction presented a unique challenge, requiring the models to navigate through complex legal language and contexts to generate precise and relevant responses.

[0060] By leveraging the present invention, the MLLNLG models were honed to effectively interpret and respond to legal instructions with a high degree of accuracy and comprehension. This embodiment reflects a commitment to developing sophisticated Al systems tailored specifically for the demands of legal applications, ultimately enhancing efficiency and productivity in legal workflows such as• Analyze and explain the legal reasoning behind the judgment in the given case• Identify and summarize the key legal issues in the provided case• Draft an argument appealing the decision of the given case• Identify the legal precedent s used in the presented case• Draft a summary of a given Indian law or statute, outlining its purpose, main provisions, and implications• Develop a legal strategy for a hypothetical client based on the facts of the provided case• Draft a hypothetical dissenting opinion for the provided case• Identify potential policy changes that could be advocated for in light of the given case• Draft a hypothetical legal notice based on the facts of the provided case• Discuss potential legal reforms suggested by the decision in the provided case• Summarize the primary dissenting arguments in the provided case• Identify areas of disagreement between judges’ opinions in the presented case• Specify the terms of termination, including the notice period and grounds for termination• Create a legal condition for Confidentiality Obligation for Employment Contract in Banking industry• Add a clause to the contract that specifies the consequences of termination for convenience• Draft a condition related to property damage for Land Lease Agreement in Agriculture industry• Draft a summary of the given case, highlighting its purpose, main provisions, and implications• Establish the employee’s confidentiality obligations.

[0061] Figure 8 shows various legal NLG and legal NLP tasks execution capabilities performed using the present novel and efficient method.

[0062] With reference to figure 9, the computing hardware system (180) features a bus (187) connecting various components: memory (181), one or more processors (182), presentation components (183), I / O ports (184), I / O components (185), and a power supply (186). The bus (187) may include different types such as address and data buses. The diagram in figure 9 illustrates a typical setup, where presentation components like display devices also function as I / O components. The processors include memory, and the example provided is representative of various devices like workstations, servers, laptops, and hand-held devices that fall under the category of computing hardware system (180).

[0063] The computing hardware system (180) includes various types of computer-readable media, which are accessible storage and communication mediums. These can be volatile or non-volatile, removable or nonremovable, and include both storage and communication mediums. Computer storage media, used for storinginstructions, data, and program modules, encompasses RAM, ROM, EEPROM, flash memory, CDs, DVDs, magnetic tapes, and other similar devices. Communication media, which deliver data through modulated signals, includes wired connections, such as direct links or networks, and wireless methods like infrared, RF, and acoustic signals. Combinations of these media types are also covered.

[0064] In the computing hardware system (180), memory (181) includes both volatile and non-volatile types, which can be removable or fixed. Examples of memory include optical disc drives, solid-state drives, and hard drives. One or more processors read data from memory (181) and I / O components (185). Data is presented to users or devices via presentation components (183), such as display screens, speakers, printers, or vibrating elements.

[0065] The computing hardware system (180) connects to various I / O components (185) through I / O ports (184). Examples include microphones, joysticks, printers, and wireless devices. A Natural User Interface (NUI) may handle inputs such as speech, touch, and gestures. The NUI can include speech recognition, touch and stylus recognition, facial recognition, biometric identification, and gesture tracking. Depth cameras (RGB, stereoscopic, infrared) and motion sensors (accelerometers, gyroscopes) may also be used for gesture detection and motion tracking.QUANTITATIVE RESULTS

[0066] The present MLLNLG models are evaluated on perplexity metric, and zero-shot evaluation on two tasks: case judgement prediction with explanation and abstractive legal case summarization. . Additionally, the instruction fine-tuned MLLNLG models are evaluated by the OpenAI GPT-3.5-Turbo Al Engine (ChatGPT) as an external evaluator. This engine assesses the models' responses to a variety of legal instructions, encompassing both legal NLP and NLG tasks such as drafting clauses, making modifications, and creating legal contracts across different industries. The evaluation is based on four key metrics: clarity, relevance, completeness, and legal reasoning.Table 1: Perplexity and MFU Metrics of the legal language pre-trained models at context size of 8192, and 4096 respectively and CPU (AMD epyc 7252 8-core processor, RAM: 128GB RAM) inference speed.

[0067] The evaluation of instruction fine-tuned MLLNLG models utilizes data generated from the Legal Instructions Data Triplet Generation Engine (132), which accepts human-annotated legal instructions data triplets in form of (instructions, input, response) or legal instruction data tuples in the form of (input, response) or (input, output) or (instruction, response) or (instruction, output) and / or machine generated legal instructions data triplet comprising instructions, input, and response. In case if the input is null, the triplet becomes instruction-response pairs or input-response pairs if the instruction is null.Table 2: Open Al GPT-3.5-Turbo Al Engine’s evaluation of Instruction Fine-Tuned Embodiment with 8192 context size responses on scale of 10 to legal instructions of drafting legal clauses and various types of legal contracts.Table 3: GPT-3.5-Turbo evaluation of Instruction Fine-Tuned Embodiment with 8192 context size responses on scale of 10 to legal instructions (legal NLP and legal NLG tasks) related to Supreme Court cases documents.

[0068] Legal documents are typically lengthy, and existing transformer decoder-based large language models (LLMs) struggle to effectively encode the semantic and syntactic intricacies of their content within limited context windows during pre-training. This often results in poor comprehension of complex legal jargon and verbose documents, leading to subpar and incoherent summarization capabilities. In contrast, the present invention is designed to encode complicated legal information and understand long legal documents more effectively, as one of its embodiments was pre-trained from scratch with a context size of 8192. This enhancement allows the model to better comprehend and analyze intricate legal documents, acquiring extensive domain-specific knowledge and enabling it to generate state-of-the-art legal abstractive summaries and legal NLG. These tasks are particularly challenging in legal NLP due to the average length of legal documents. One embodiment, a 97 million parameter legal MLLNLG pre-trained model, demonstrates consistency in generating long summaries, achieving high ROUGE- 1 and ROUGE-L metrics across summary generations of varying lengths between 1024 and 8192 tokens. ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, is a standard lexical evaluation metric that measures the overlap between model-generated summaries and reference summaries. ROUGE- 1 assesses the textual overlap of unigrams, while ROUGE-L evaluates the longest matching sequence of words using the Longest Common Subsequence (LCS) method. Additionally, BERTScore (Zhang et al., 2020) utilizes the BERT model to compute similarity scores between the token-level embeddings of generated and reference summaries. The MLLNLG model was evaluated through zero-shot evaluation on the Supreme Court of India abstractive summarization test dataset (Shukla et al., 2022). Table 4 reporting the test evaluation metrics for summarizationTable 4: Zero shot evaluation on the IN-Abs Test Dataset: Indian Supreme Court case documents (Shukla et al. 2022) & their ' abstractive1summaries.

[0069] Zero-shot evaluation means evaluating the pre-trained model without any additional fine-tuning on a further training set. During inference time, top-p sampling was used and set to the value of 0.9 and temperature hyperparameter was set to le-2. N-shot evaluation refers to evaluating a machine learning model's ability to generalize and perform tasks using only a limited number (N) of labeled examples or instances per class or task. In this context:• N typically represents the number of examples (shots) available for each class or task during evaluation.• Shot refers to a single example or instance used for learning or evaluation within this framework.

[0070] The goal of N-shot evaluation is to assess how well a model can learn and adapt to new tasks or classes unseen during pre-training with minimal supervision, mimicking real-world scenarios where obtaining large labeled datasets may be impractical or expensive. This technique is commonly applied in various machine learning tasks where the model is tested on classes or tasks that were not part of its training data, using only a small number of examples per class to measure its generalization capability.Table 5: Case Judgement Prediction: Zero Shot evaluation on the test set, i.e., without fine-tuning training set.

[0071] Table 6 compares the zero-shot performance on legal case judgement prediction with explanation task (Nigam et al., 2024) of existing LLMs such as Llama 7B, Gemini Pro, and one of the embodiment of the present invention. The 97M MLLNLG model outperformed both Gemini Pro by 1.2% points, and Llama-2 7B by 1.75% points despite being smaller by 74 times in size compared to 7B (billion) parameters LLMs. This demonstrates the powerful legal reasoning and legal knowledge understanding capabilities of one of the embodiments, the 97M MLLNLG model.

[0072] Furthermore, the MLLNLG was further evaluated through zero-shot evaluation on various commonsense reasoning benchmark (ARC-challenge (Clark et al., 2018), causal reasoning task - COPA (Roemmele et al. 2011), PIQA (Bisk et al., 2020), logical reasoning - MMLU (Hendrycks et al. 2020), mathematical reasoning - AGIEVAL (Zhong et al. 2023), WSC (Levesque et al. 2012), Truthfulq (Lin et al. 2022), Truthful QA (Lin et al., 2021a), question and answering (Bisk et al. 2019) and natural language inference - QNLI (Wang et al. 2018)).

[0073] TruthfulQA (Lin et al., 2021a) evaluates the model to provide truthful answers, focusing on its understanding of facts and its ability to avoid deception. MMLU (Hendrycks et al., 2020) measures the model’s broad knowledge across numerous subjects such as, humanities, science, technology, engineering and management. ARC Challenge (Clark et al., 2018) tests complex reasoning with science questions. PIQA (Bisk et al., 2020) evaluates the model’s physical commonsense knowledge, requiring understanding of everyday physical processes.

[0074] One of the embodiments underwent testing for LEGISLATIVE RESPONSE, yielding the following outcome by the pre-trained and instruction fine-tuned MLLNLG model.

[0075] Another embodiment was tested for KEY LEGAL ISSUE, yielding the following outcome by the pretrained and instruction fine-tuned MLLNLG model.

[0076] Yet another embodiment underwent testing for DRAFTING LEGAL ARGUMENTS, yielding the following outcome by the pre-trained and instruction fine-tuned MLLNLG model.

[0077] Yet another embodiment was tested for LEGAL PRECEDENTS, yielding the following outcome by the pre-trained and instruction fine-tuned MLLNLG model.

[0078] Yet another embodiment was tested for RAISING OF HYPOTHETICAL QUESTIONS, yielding the following outcome by the pre-trained and instruction fine-tuned MLLNLG model.

[0079] Yet another embodiment was tested for POTENTIAL POLICY CHANGE, yielding the following outcome by the pre-trained and instruction fine-tuned MLLNLG model.

Claims

CLAIMS1. One or more computer storage media storing computer-executable instructions that, when used by a computing hardware system (180), cause the computing hardware system (180) to perform operations, the operations comprising: a data curation process (110) through a data crawling engine (111), a data storage device (112), a data cleaning engine (113), a data deduplication engine (114), a tokenization engine (116) and a data vectorization (117) through scaled version of Rotary Positional Embedding (RoPE) by receiving an input task / query from real human; a data pre-training process (120) through a pre-training engine (121) for pretraining, training and evaluation of a pretrained MLLNLG models; an instruction fine-tuning process (130) for evaluating pre-trained MLLNLG models utilizes data generated from a legal instructions data triplet generation engine (132) that accepts human -annotated legal instruction data triplets in the form of (instructions, input, response) or legal instruction data tuples in the form of (input, response) or (input, output) or (instruction, response) or (instruction, output) and / or machine-generated legal instruction data triplets, which also consist of instructions, input, and response; and a deploying process (140) to deploy the pre-trained / instruction fine-tuned legal language models in a human-computer interface (150) to execute legal NLP and legal NLG tasks.

2. A computer implemented method (100) for machine learning legal natural language generation (MLLNLG) model for performing legal natural language processing and legal natural language generation tasks, comprising: the data curation process (110) through pre-training data curation engine and tokenization engine (110A) further comprising; collecting a pretraining data corpus using a legal data crawling engine (111) that crawls legal specific data, such as court case documents, country specific legislative constitution documents, and legal penal code documents; storing the collected data in a legal data storing device (112); cleaning and pre-processing the stored data through a legal data cleaning engine (113), which removes noisy text, including unreadable characters, special characters, excessive white spaces, case numbers, judge names, and dates, through the application of regular expressions; removing duplicate entries in the web scrapped prefraining corpora using a legal data deduplication engine (114); storing the cleaned and deduplicated curated data in the legal data storage (115); tokenization, by a data tokenization engine (116), performing training of greedy merge-based sub-word algorithm, Byte-Pair encoding (BPE) tokenizer, developing specialized tokenizer forlegal domain to learn intricate legal jargon and terminology, in the process treating data as a sequence of bytes considering each bytes as tokens and eliminating duplicate tokens from the tokenizer, thereby forming specialized, compact, effective and optimized tokenizer; the size of the legal domain specialized tokenizer can vary between 2000 and 64000; data vectorization (117), preprocessed data converted into numerical vectors using scaled version of Rotary Position Embedding (RoPE); the pre-training process (120) through pre-training engine (121) where the pre-training of legal autoregressive MLLNLG models with processed vectorized training / pre -training data using causal language modelling objective of next token prediction, i.e., predicting the next token in a sequence given the preceding tokens, by minimizing cross-entropy loss between the predicted probability distribution of tokens over the vocabulary and the actual next token, the MLLNLG model learns to assign higher probabilities to tokens that are more likely to occur next in the sequence; performing hyperparameter-tuning on smaller MLLNLG model (less than 50 million parameters in size) to determine optimal vocabulary size, learning rate and warm-up ratio, and transferring the learned hyperparameters to the bigger (less than 7 billion parameters) MLLNLG models through maximum update parameterization (pP); and undergoing performance evaluation based on validation perplexity metric, on model FLOPs Utilization (MFU) metric for computing hardware device (GPU) utilization and CPU inference speed of legal MLNLG model in FP32 (floating point) numeric precision; second evaluation of pretrained MLLNLG legal model on zero-shot case judgement prediction with explanation task, and legal case summarization task at varying degree of length (<=8192 tokens) of abstractive summary generation; third evaluation of pretrained MLLNLG on zero-shot evaluation on various common-sense reasoning benchmark (ARC-challenge, COPA, PIQA, MMLU, WSC, TruthfulQA), mathematical reasoning such as AGIEVAL, and natural language inference (QNLI). the instruction fine-tuning process (130), the pre-trained MLLNLG models using supervised instruction fine-tuning dataset which is generated from legal instructions data triplet generation engine (132), results in a instruction fine-tuning dataset comprising of 10,763 unique instructions covering multitude of legal NLP tasks, legal instructions dataset was split into training and testing datasets respectively; and reporting of training and test loss while training with different learning rate scheduler such as cosine, constant and linear learning rate scheduler on the legal instructions training dataset; and the deploying process (140) to deploy the pre-trained and instruction fine-tuned MLLNLG models in human-computer interface (150) for execution of various legal NLP and legal NLG tasks based on human instructions and preferences through a I / O component (185) of the computer hardware system (180).

3. The computer-implemented method (100) as claimed in claim 2, wherein to enhance training stability, the method employs RMSNorm for pre-normalization of the input in each transformer and feedforward layer, andincorporates residual and skip connections and utilizes the SwiGLU activation function to introduce nonlinearity in feedforward dense layers, alternatively using ReLU or an approximate GeGLU activation function to introduce non-linearity in the feedforward dense layers, and the method architecture incorporates Grouped- Query Attention (GQA) where query heads are divided into G groups, each sharing a single key and value head, with GQA-G referring to grouped-query with G groups, or alternatively employs Multi-head self- attention or Multi-Query Attention where different attention heads share a single set of keys and values.

4. The computer-implemented method (100) as claimed in claim 2, wherein weights are shared between embeddings and layers to reduce the size of the pre-trained MLLNLG model, and hyperparameter-tuning is performed to determine optimal values for vocabulary size, learning rate, learning rate scheduler, and warm-up ratio in smaller models (less than 50 million parameters), with these hyperparameters transferred to larger models (less than 7 billion parameters) using maximum update parameterization (pP).

5. The computer-implemented method (100) as claimed in claim 2, wherein during the pretraining phase, the learning rate (Zr) decay steps are set to max_steps of pretraining phase and the minimum Ir is set to 0.

1. Ir, and the Ir schedule starts with a linear warm-up from 0 to maximum Ir at 1000 steps, followed by a cosine decay to the minimum Ir until the end of an epoch of training.

6. The computer-implemented method (100) as claimed in claim 2, wherein during the pretraining phase, the maximum learning rate Ir is to 3 X 10-3(max), weight decay of 1 X 10-1, the MLLNLG model is trained with fused AdamW optimizer or may use other optimizers such as Adam / Adagrad / Adadelta / Adamax / Adaf actor / Nadam / SGD / RMSprop,- and performing 95%-5% train and test split and run the evaluation loop preferably every 2000 steps of training on the test split of the pretrained data.

7. The computer-implemented method (100) as claimed in claim 2, wherein in the decoding process uses Top- p nucleus sampling to generate predictions of the inputs whereas the probability distribution of decoded tokens (Pt) is a function of temperature hyper parameter (T) and it may also use other decoding methods such as greedy decoding, beam search, Top-k sampling, while running inference from MLLNLG models.

8. A computing hardware system (180) for managing MLLNLG models, the system comprising: a processor (182); and a memory (181) coupled with the processor ( 182), the memory (181) storing programmed instructions that, when executed by the processor (182), causes the system (180) to perform operations comprising: data crawling, data storing, data cleaning, data deduplication, storing of deduplicated data, data tokenization and data vectorization through a scaled version of Rotary Positional Embedding (RoPE) embedding in the data curation process (110);pre-training the legal language models from scratch using causal language modeling objective of next token prediction by minimizing cross-entropy loss between the predicted probability distribution of tokens over the vocabulary and the actual next token, in the pre-training process (120); fine-tuning of the pre-trained legal language models in the instruction fine-tuning process (130); and deploying the pretrained / instruction fine-tuned legal MLNLG models in human-computer interface (150) in the deploying process (140).

9. The system as claimed in claim 8, capable of doing various legal NLP and legal NLG task including draft legal notice, formulate legal advice, drafting legal arguments, raising hypothetical questions, draft legal contract condition, draft legal clauses, draft NDA, work contracts, constitutional QA, legal question answering (QA), draft legal contracts, legal case summarization, formulate legal strategy, formulate potential policy changes, identify legal precedents, draft legal petition, legislative response, and identify key legal issue.

10. A reinforcement learning by human feedback mechanism (170) to further improve the MLLNLG model learning ability and its responses to the humans query or input, the reinforcement learning process comprising: defining guidelines to guarantee unique criteria to decide a good and a bad answer to an input; developing a Reward Model (RM) to evaluate each of the responses in terms of accuracy, relevance, and adherence to guidelines; training of the RM wherein some prompts or preference data are selected and sent to human reviewers; interaction with the reward model by human reviewers to manually evaluate and rate the corresponding outputs; collection of feedback, in the form of ratings or rankings, is used to further train the RM; training a Policy Optimizer with the trained reward model wherein this policy optimizer is responsible for the further fine-tuning of the instruction fine-tuned legal language model; and reinforcement learning process is performed in iterative feedback loop allowing the legal language model / instruction-tuned legal language model to gradually learn from human guidance and refine its behavior accordingly.

Images