System and method for private generation of code

Abstract

An example system for private generation of code is provided. The system includes a client side including a vocabulary unit and a tuning unit. The vocabulary unit receives prompts from a client and outputs token embeddings, and the tuning unit executes an encoder to encode the token embeddings and generate prompt embeddings. The system includes a cloud side including a large language model (LLM) executed to generate enriched embeddings from the prompt embeddings. The decoder of the tuning unit is executed to decode the enriched embeddings to generate code corresponding to the prompts from the enriched embeddings.

Description

Attorney Docket No. 94350.00296 (NUT Ref. 25-002)SYSTEM AND METHOD FOR PRIVATE GENERATION OF CODECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of a co-pending, commonly assigned U.S. Provisional Patent Application No. 63 / 685,371, which was filed on August 21, 2024. The entire content of the foregoing provisional application is incorporated herein by reference.STATEMENT OF GOVERNMENT SUPPORT

[0002] This invention was made with government support under Grant Nos. 1935923 and 1935928, awarded by the National Science Foundation (NSF). The government has certain rights in the invention.FIELD OF THE DISCLOSURE

[0003] The present disclosure relates to a system for private generation of code and, in particular, to a system for providing intellectual property and data security protection for code generation using large language models (LLM).BACKGROUND

[0004] Recent ground-breaking developments of large language model (LLM) powered code generation have significantly improved software developer productivity with commercial tools. Commercial LLM-powered code generation tools (such as Microsoft GitHub Copilot, Google Gemeni, Amazon Code Whisperer, or the like) have significantly improved software developer productivity. (See, e.g., GitHub, Research: Quantifying GitHub copilot’s impact on code quality, available at https: / / github.com / features / copilot (2023)). However, a critical concern raised by over 80% of companies using cloud-hosted generative Al is intellectual property (IP) leakage and data security, with nearly 45% companies encountering unintended data exposure. See, e.g., CNBC, The biggest risk corporations see in gen Al usage isn’t hallucinations, available at https: / / www.cnbc.com / 2024 / 05 / ! 6 / the-no- 1 -risk-companies-see-in-gen-ai-usage-isnt- hallucinations.html (2024); T-Minus365, Microsoft 365 Copilot, Security Risks & How to Protect Your Data, available at https: / / tminus365.com / microsoft-365-copilot-security- risks-how-to-protect-your-data / (2024); Concentric. ai, Too Much Access? Microsoft Copilot Security Concerns Explained, available at https: / / concentric.ai / too-much-access- microsoft-copilot-data-risks-explained / (2024)). Real world incidents of proprietary code1MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) leakage have also been reported. (See, e.g., PC Mag, Samsung Software Engineers Busted for Pasting Proprietary Code Into ChatGPT, available at https: / / www.pcmag.com / news / samsung-software-engineers-busted-for-pasting- proprietary-code-into-chatgpt (2023)). These concerns and risks are rooted in the prompts a client sends to the cloud-hosted LLMs to obtain generated code. The prompts and the generated code allow the cloud operators to observe the exact functions and techniques used to develop commercial systems. Thus, addressing these concerns and risks is critical given the proliferation of LLM-powered code generation across industry sectors. (See, e.g., Fortune Business Insights, Low Code Development Platform Market Size, Share & Industry Analysis, available at https: / / www.fortunebusinessinsights.com / low-code- development-platform- market- 102972 (2024)).SUMMARY

[0005] In accordance with embodiments of the present disclosure, an exemplary system for private generation of code is provided. The exemplary system includes a client side and a cloud side, with specific separation between the respective client and cloud sides to ensure privacy of input prompts from the client and code generated by the system in response to the input prompts. The client side includes a vocabulary unit that outputs token embeddings corresponding to the prompts input by the client, and a tuning unit that encodes the token embeddings to generate prompt embeddings. The prompt embeddings are transmitted to a large language model (LLM) of the cloud side, which prevents the cloud side from detecting the raw input prompt data. The LLM generates as output enriched embeddings, which a decoder at the client side uses to generate code corresponding to the input prompts. The exemplary system therefore prevents detection by the cloud side of the input prompts and the generated code, ensuring privacy of the client information and work product.

[0006] In accordance with embodiments of the present disclosure, an exemplary system for private generation of code is provided. The system includes a client side including a vocabulary unit configured to receive as input one or more prompts from a client. The vocabulary unit is executed to output token embeddings corresponding to the one or more prompts. The client side includes a tuning unit including an encoder and a decoder. The encoder is executed to encode the token embeddings and generate as output prompt embeddings. The system includes a cloud side including a large language model2MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)(LLM) configured to receive as input the prompt embeddings from the tuning unit. The large language model (LLM) is executed to generate as output enriched embeddings. The decoder of the tuning unit receives as input the enriched embeddings and is executed to decode the enriched embeddings to generate code corresponding to the one or more prompts from the enriched embeddings.

[0007] The vocabulary unit can include a tokenizer unit. In some embodiments, the tokenizer unit can be a local randomized tokenizer. The tokenizer unit can be configured to assign every token of the token embeddings to a random index in a vocabulary of the vocabulary unit. The vocabulary unit can be configured to adaptively randomize the token embeddings based on vocabulary of the vocabulary unit. In some embodiments, the vocabulary unit can be a local differential privacy (LDP) preserving vocabulary unit. In some embodiments, the token embeddings can be local differential privacy (LDP) preserving token embeddings.

[0008] In some embodiments, the client side can include a processing device configured to execute the vocabulary unit and the tuning unit. In some embodiments, the cloud side can include a processing device configured to execute the large language model (LLM). The decoder can generate the code on the client side. The vocabulary unit and the tuning unit can prevent observation of the one or more prompts and the generated code by the cloud side such that the one or more prompts and the generated code are only visible on the client side. The encoder and the decoder can be executed to operate only on the client side.

[0009] The tuning unit can be configured to receive fine-tuning input from the client to fine-tune at least one of the encoder or the decoder. The fine-tuning input can directly align the encoder and the decoder while bypassing a fine-tuning of the large language model (LLM) of the cloud side. The prompt embeddings can prevent a cloud on the cloud side from reconstructing the one or more prompts.

[0010] In accordance with embodiments of the present disclosure, an exemplary method for private generation of code is provided. The method includes receiving as input one or more prompts from a client at a vocabulary unit at a client side. The method includes executing the vocabulary unit to output token embeddings at the client side corresponding to the one or more prompts. The method includes executing an encoder of a tuning unit at the client side to encode the token embeddings and generate as output prompt embeddings.3MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)The method includes receiving as input the prompt embeddings at a large language model (LLM) of a cloud side. The method includes executing the large language model (LLM) at the cloud side to generate as output enriched embeddings. The method includes receiving as input at a decoder of the tuning unit at the client side the enriched embeddings. The method includes executing the decoder of the tuning unit at the client side to generate code corresponding to the one or more prompts from the enriched embeddings.

[0011] The vocabulary unit can include a tokenizer unit. The tokenizer unit can be configured to assign every token of the token embeddings to a random index in a vocabulary of the vocabulary unit. The vocabulary unit can be configured to adaptively randomize the token embeddings based on vocabulary of the vocabulary unit.

[0012] In accordance with embodiments of the present disclosure, an exemplary non- transitory computer-readable medium storing instructions for private generation of code is provided. The instructions are executable by at least one processing device. Execution of the instructions by the at least one processing device causes the at least one processing device to receive as input one or more prompts from a client at a vocabulary unit at a client side, and execute the vocabulary unit to output token embeddings at the client side corresponding to the one or more prompts. Execution of the instructions causes the at least one processing device to execute an encoder of a tuning unit at the client side to encode the token embeddings and generate as output prompt embeddings, and receive as input the prompt embeddings at a large language model (LLM) of a cloud side. Execution of the instructions causes the at least one processing device to execute the large language model (LLM) at the cloud side to generate as output enriched embeddings, receive as input at a decoder of the tuning unit at the client side the enriched embeddings, and execute the decoder of the tuning unit at the client side to generate code corresponding to the one or more prompts from the enriched embeddings.

[0013] Any combination and / or permutation of the embodiments is envisioned. Other objects and features will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed as an illustration only and not as a definition of the limits of the present disclosure.4MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)BRIEF DESCRTPTTON OF THE DRAWINGS

[0014] To assist those of skill in the art in making and using the exemplary system for private generation of code, reference is made to the accompanying figures, wherein:

[0015] FIG. l is a block diagram of an exemplary system for private generation of code in accordance with embodiments of the present disclosure.

[0016] FIG. 2 is a table of a summary of differences between the exemplary system for private generation of code and related works.

[0017] FIG. 3 is a block diagram of a threat model of an honest-but-curious cloud on an exemplary system for private generation of code.

[0018] FIG. 4 is an example of an algorithm for an exemplary system for private generation of code.

[0019] FIGS. 5A-5D are examples of a threat model from a cloud on an exemplary system for private generation of code, including a client’s prompt (FIG. 5A), the client’s generated code from the system (FIG. 5B), the cloud’s reconstructed code in a defense-free environment (FIG. 5C), and the cloud’s reconstructed code under the system (FIG. 5D).100201 FIG. 6 is a table of PromptRA and CodeRA in a defense-free environment.

[0021] FIG. 7 is a table of a comparison of original prompts and promptRA on clean (defensive-free), and system generated code with CodeQwenl.5-7B and 6=27.

[0022] FIG. 8 is an example of cloud’s reconstructed code under clean (defense-free) with CodeQwenl.5-7B, and ARy , AR ™ , ARyUy,1are the privacy, proprietary, and fusi attack result, respectively, between the ground-truth code y and the reconstructed code y’.

[0023] FIG. 9 is a table of examples of cloud’s reconstructed code under the system with CodeQwenl.5-7B and 6=27, and AR , AR^° , ARyUy,1are the privacy, proprietary, and fusi attach result, respectively, between the ground-truth code y and the reconstructed code y’.

[0024] FIG. 10 is a block diagram of a token reconstruction security game during testing of the system.

[0025] FIG. 11 is a graph of varying G with | | =90,000 and |x |=100.5MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0026] FIGS. 12A-12C are graphs of the cloud’s probability of reconstructing a clear gist of prompts, i.e., p > 20, as a function of LDP budget 6, vocabulary size |V|, prompt size |x|, and the threshold p, with the second line in the key indicating the probability of 0.5, and |F| with |x| = 100 (FIG. 12A), |x| with |F| = 90,000 (FIG. 12B), and p with |F| = 90,000 and |x| = 100 (FIG. 12C).

[0027] FIG. 13 is a graph of LLMs for code generation vs. reconstruction probability, G= 13, |x| = 50.

[0028] FIGS. 14A-14B are graphs of the percentage of token changes in a prompt (FIG. 14A), and the Li-norm distance between original and LDP -preserving token embeddings on the MBPP dataset (FIG. 14B), G= 13.

[0029] FIGS. 15A-15B are graphs of the exemplary system gain in terms of bi-gram angular change in every prompt (FIG. 15 A), and angular change among tokens in the vocabulary compared with T2 and SnD (FIG. 15B).

[0030] FIG. 16 is a graph of adaptive probabilities.

[0031] FIG. 17 is a block diagram of a local randomized tokenizer of an exemplary system for private generation of code.

[0032] FIGS. 18A- 18B are graphs of Pass @ 1 and LDP budget e for the MBPP dataset (FIG. 18A) and the HumanEval dataset (FIG. 18B).

[0033] FIGS. 19A-19B are graphs of Pass@ lr 6 {1,5,10} and LDP budget e for the MBPP dataset (FIG. 19A) and the HumanEval dataset (FIG. 19B).

[0034] FIGS. 20A-20E are graphs of the exemplary system against PromptRA for ASR* in training set (MBPP) for prompt (FIG. 20A) and prompt and testcase (FIG. 20B), ASR*™ in test set (MBPP) for prompt (FIG. 20C) and prompt and testcase (FIG.HumanEval (FIG. 20E).

[0035] FIGS. 21 A-21C are graphs of the exemplary system against CodeRA for MBPP training (FIG. 21A), MBPP test (FIG. 21B), and HumanEval (FIG. 21C).

[0036] FIGS. 22A-22C are graphs of token-based frequency analysis attacks on different datasets, including MBPP test using 376k dataset on token-based attack (FIG.6MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)22A), HumanEval using 376k dataset on token-based attack (FTG. 22B), and MBPP test using MBPP train dataset on token-based attack (FIG. 22C).

[0037] FIGS. 23A-23C are graphs of frequency analysis attacks (1=3) for MBPP test using 376k dataset on sequence-based attack (FIG. 23A), HumanEval using 376k dataset on sequence-based attack (FIG. 23B), and MBPP test using MBPP train dataset on sequence- based attack (FIG. 23C).

[0038] FIGS. 24A-24B are graphs of varying LLM models, including an MBPP dataset (FIG. 24A) and a HumanEval dataset (FIG. 24B).

[0039] FIGS. 25A-25B are graphs of enlarging the training data (CodeQwenl.5-7B- Chat), including enlarging the training dataset (FIG. 25A) and increasing the privacy budget G (FIG. 25B).

[0040] FIGS. 26A-26D are graphs for enlarging training data to 370k data points and LDP G to 27 under PromptRA and Code RA for the exemplary system, including PromptRA (FIG. 26A), CodeRA priv (FIG. 26B), CodeRA prop (FIG. 26C), and CodeRA fusi (FIG. 26D).

[0041] FIGS. 27A-27D are graphs for enlarging training data to 370k data points and LDP G to 27 under PromptRA and Code RA for the exemplary system without LTokenizer, including PromptRA (FIG. 27 A), CodeRA priv (FIG. 27B), CodeRA prop (FIG. 27C), and CodeRA fusi (FIG. 27D).

[0042] FIGS. 28A-28D are graphs for enlarging training data to 370k data points and LDP G to 27 under PromptRA and Code RA for the exemplary system without LDPVocab, including PromptRA (FIG. 28A), CodeRA priv (FIG. 28B), CodeRA prop (FIG. 28C), and CodeRA fusi (FIG. 28D).

[0043] FIGS. 29A-29B are graphs of the exemplary system with and without LoRA on the cloud, including a MBPP dataset (FIG. 29 A) and a HumanEval dataset (FIG. 29B).

[0044] FIG. 30 is a graph of LoRA and LTokenizer of the exemplary system.

[0045] FIGS. 31A-31B are graphs of client’s fine-tuning cost on 1 A100 GPU (80GB) for GPU memory (FIG. 31 A) and time and cost on AWS (FIG. 3 IB).

[0046] FIGS. 32A-32B are graphs of varying the size of the decoder, including a MBPP dataset (FIG. 32A) and a HumanEval dataset (FIG. 32B).7MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0047] FIG. 33 is a block diagram of an exemplary system for private generation of code in accordance with embodiments of the present disclosure.

[0048] FIG. 34 is a block diagram of an exemplary computing device for implementing the exemplary system for private generation of code in accordance with the present disclosure.

[0049] FIG. 35 is a block diagram of an exemplary private generation of code system environment in accordance with the present disclosure.DETAILED DESCRIPTION

[0050] Example systems, methods, and apparatus are disclosed herein for private generation of code (referred to herein as “PGCode”). In particular, an exemplary system for private generation of code and an exemplary method of using a system for private generation of code are discussed herein. The PGCode introduces a first framework to protect clients' prompts and generated code from being observed by the cloud. In PGCode, an encoder, and a decoder, which are lightweight as few attention blocks, exclusively run on the client side, such that a client can send their prompts’ embeddings derived from the encoder to the cloud-hosted LLM to receive enriched embeddings in return. The decoder subsequently generates code privately using the enriched embeddings on the client side.

[0051] The exemplary system provides an advantageous concept of achieving local differential privacy (LDP) in the vocabulary (LDPVocab) consisting of all words (also called tokens) used in prompts and code, on the client side, and from that PGCode establishes an effective defense against reconstruction attacks carried out by a privacy untrusted cloud to infer clients' prompts and generate code. To optimize the utility of the generated code, the system develops a cost and performance-effective learning approach to finetune. The lightweight encoder and decoder are on the client side. Extensive theoretical analysis and empirical results in code generation shows that PGCode achieves nearly lossless utility (a marginal utility drop of about 1% on average compared with conventional LLMs) in the high privacy guarantee regime against reconstruction attacks.

[0052] Although boosting software development performance, large language model (LLM)-powered code generation can introduce intellectual property and data security risks rooted in the fact that a service provider (cloud) can observe a client’s prompts and generated code, which can be proprietary in commercial systems. To mitigate this problem,8MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) an exemplary system (PGCode) is provided. The exemplary system includes a first framework to protect the client’ s prompts and generated code from the cloud. The system uses an encoder and a decoder at the client to encode and send the prompts’ embeddings to the cloud to get enriched embeddings from the LLM, which are then decoded to generate the code locally at the client. Since the cloud can use the embeddings to infer the prompt and the generated code, the system introduces a new mechanism to achieve local differential privacy in the vocabulary used in the prompts and code, and a data-independent and randomized tokenizer on the client side. These components effectively defend against reconstruction and frequency analysis attacks by an honest-but-curious cloud. Extensive analysis and results discussed herein show that the system significantly outperforms conventional baselines on benchmark datasets, including the Evalplus, consisting of MBPP and HumanEval with a Pass@l of 76.7 and 77.4 in code generation and completion, and the BigCodeBench with a Pass@ l of 31.1 (a marginal drop of 1.8% compared with the original LLM) under strong privacy against attacks.10053 J In the exemplary system, the key focus of privacy protection is that an encoder and a decoder (which can be basic and lightweight attention blocks) exclusively run on the client, enabling the client to interact with the cloud-hosted LLM through the embedding space. The encoder on the client side encodes a prompt before sending the encoded prompt embedding to the cloud-hosted LLM to enrich the embedding through the knowledge in the latent space learned by the LLM. The enriched embedding offered by the cloud-hosted model enhances model performance since the cloud-hosted LLM has learned features from large datasets. The decoder decodes the enriched embedding, which the cloud returns, to generate code on the client side.

[0054] The exemplary system for private generation of code discussed herein provides a solution for protecting the functions and techniques during code generation while using LLMs. The system provides a vital solution to the problem of cloud operators observing the exact functions and techniques used to develop code by protecting both the client’s prompts and the corresponding generated code from being observed by the cloud. A direct approach to achieve this objective is to build a data center on the client side to train, host, operate, and maintain LLMs for code generation. However, this option is generally financially unaffordable to most clients and enterprises. (See, e.g., McGregor, J., Forbes, Generative Al Breaks the Data Center: Data Center Infrastructure And Operating Costs Projected To Increase To Over $76 Billion By 2028, available at9MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) https: / / www.forbes.eom / sites / tiriasresearch / 2023 / 05 / 12 / generative-ai-breaks-the-data- center-data-center-infrastructure-and-operating-costs-projected-to-increase-to-over-76- billion-by-2028 / (2023); Potter, B., Construction Physics, How to Build an Al Data Center, available at https: / / www.construction-physics.eom / p / how-to-build-an-ai-data-center (2024)). Such solution is also impractical for the cloud to send its proprietary LLMs to the client for hosting, even the compressed ones since the cloud will risk losing its model’s ownership. (See, e.g., Xiao, G. et al., Offsite-Tuning: Transfer Learning without Full Model, arXiv, 2302.04870 (2023)).

[0055] Prior approaches have included prompt tuning see, e.g., Duan, H. et al., Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models, Advances in Neural Information Processing Systems, Volume 36 (2023); Wu, T. et al., Privacy-Preserving In-Context Learning for Large Language Models, ICLR (2024); Hong, J. et al., DP-OPT: Make Large Language Model Your Privacy Preserving Prompt Engineer, ICLR, (2024)) and example ensembling (see, e.g., Tang, X. et al., Privacy-Preserving InContext Learning With Differentially Private Few-Shot Generation, ICLR (2024)) under differential privacy (DP) to protect membership information of data points in the training set; that is, prevent the cloud from inferring whether a data point is used in training prompttuning models. Other methods to protect the client’s prompts have been to alter tokens (see, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)) or their embeddings (see, e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) in every client’s prompt to achieve metric DP (i.e., dx -privacy, as in Chatzikokolakis, K. et al., Broadening the Scope of Differential Privacy Using Metrics, Privacy Enhancing Technologies (2013)). These state-of-the-art (SoTA) approaches focus on either classification tasks (see, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023); Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) instead of code generation tasks (as pointed out in, for example, Mai, P. et al., Split-and-Denoise: Protect Large Language Model10MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)Inference With Local Differential Privacy, ICML (2024)) or protecting the training data for prompt tuning (such as in Duan, H. et al., Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models, Advances in Neural Information Processing Systems, Volume 36 (2023); Wu, T. et al., Privacy-Preserving In-Context Learning for Large Language Models, ICLR (2024); Hong, J. et al., DP-OPT: Make Large Language Model Your Privacy Preserving Prompt Engineer, ICLR, (2024); Tang, X. et al., Privacy-Preserving In-Context Learning With Differentially Private Few-Shot Generation, ICLR (2024)) without protecting the content of client’s prompts and generated outcomes.

[0056] One challenge in the private generation of code is that subtle changes in the clients’ prompts for privacy protection, either at the token level or its embedding, can significantly degenerate the generated code’s functionality since LLMs are sensitive to such changes. Achieving good generated code’s functionality typically results in loose dx - privacy protection in SoTA approaches (see, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023); Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)), making the clients’ prompts and generated code vulnerable to reconstruction attacks (see, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023); Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) by a privacy-untrusted cloud.

[0057] FIG. 1 is a block diagram of an exemplary system 100 for private generation of code. The system 100 provides the first framework to protect the client’s prompts 128 (x) and generated code from being observed by the cloud. The system 100 includes four components at the client side 112: an encoder 102 and decoder 104 pair, a fine-tuning unit 106 (including the encoder 102 and decoder 104), and an LDP -preserving vocabulary (vocabulary unit 108) associated with a local randomized tokenizer (tokenizer unit 114), integrated together. The prompts 128 are provided as input by the client 110 to the vocabulary unit 108, and the vocabulary unit 108 outputs local differential privacy (LDP) preserving token embeddings 130. The vocabulary associated with the language learning model (LLM) 118 is essential for the LLM 118 to operate, and it includes words (e.g.,11MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) tokens) and their token embeddings.

[0058] The main privacy -protection idea in the system 100 is that the pair of encoder 102 and decoder 104, which are lightweight attention blocks, exclusively runs on the client side 112, enabling the client 110 to interact with the cloud 116 hosted LLM 118 (on a cloud side 120) through the embedding space (e.g., client embedding 122) rather than sending the raw input prompts 128 and output code to the cloud 116. The client side 112 encoder 102 encodes a prompt 128 before sending the encoded prompt to the cloud-hosted LLM 118. The LLM 118 then enriches the prompt embedding 122 received from the client’s encoder 102 through the knowledge in the latent space learned by the LLM 118, outputting the enriched embedding 124. The enriched embedding 124 offered by the cloud-hosted LLM 118 enhances model performance since the LLM 1 18 has learned features from large datasets. The enriched embeddings 124 from the cloud 116 are transmitted to the client’s decoder 104, which decodes the embeddings 124 to generate the output code 126 (y) (via the vocabulary unit 108).

[0059] To optimize the encoder 102 and decoder 104 to client’ s local tasks, the system 100 includes the tuning unit 106, a cost and performance-effective split learning approach (see, e.g., Vepakomma, P. et al., Split Learning for Health: Distributed Deep Learning Without Sharing Raw Patient Data, CoRR (2018); Lin, Z. et al., Split Learning in 6G Edge Networks, IEEE Wireless Communications (2024); Pasquini, D. et al., Unleashing the Tiger: Inference Attacks on Split Learning, CCS (2021)) to fine-tune the encoder 102 and decoder 104 on the client side 112 and a lightweight low-rank adaptation (LoRA) (such as the LoRA discussed in Hu, E. et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR (2022)) of the cloud-hosted model on the cloud side 120 using a client’s local dataset. The tuning unit 106 recovers the small utility loss due to the use of encoder 102 / decoder 104 to protect privacy. The tuning unit 106 is cost-effective since the encoder 102, the decoder 104, and the LoRA are lightweight compared with the cloud-hosted LLM 118.

[0060] To prevent the cloud 1 16 from inferring sensitive and proprietary content from the prompts 128 and the generated code 126 by exploiting the prompt embeddings 122 and back-propagated gradients under SoTA reconstructing attacks (see, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023); Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based12MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) in both the tuning unit 106 and inference phases, the system 100 offers a privacy mechanism at the client side 112 in the form of the vocabulary unit 108 and the tokenizer unit 114. The vocabulary unit 108 adaptively randomizes the token embeddings in the vocabulary (with output of the embeddings 130) such that the probability for the cloud 116 to infer the ground-truth tokens, prompts 128, and code 126 given the prompt embeddings 122 is upper-bounded while notably reducing the randomness injected into the vocabulary for better utility.

[0061] To protect the one-hot vectors of tokens used in the client tokenizer unit 114, which the cloud 116 can exploit during the tuning unit 106 operation to reconstruct the client’s data from the back-propagated gradients (see, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)), the system 100 includes a local tokenizer (tokenizer unit 114), which uniformly assigns every token and its LDP-preserving embedding 130 to a random index in the vocabulary of the vocabulary unit 108, on the client side 112. The client 110 keeps this data-independent tokenizer (i.e., no extra privacy cost) secret from the cloud 116, misleading the cloud 116 to observe meaningless tokens from the back-propagated gradients.

[0062] The tokenizer unit 114 and the vocabulary unit 108 offer complete protection for the client’s prompts 128 and output code 126 in fine-tuning and inference phases. The vocabulary unit 108 keeps the tokens in prompts 128, the (prompt) instruction, the (prompt) template, and the code 126 unchanged while randomizing token embeddings only once with negligible noise. Thus, the vocabulary unit 108 maintains the correlation among prompts 128, instruction, template, and code 126 while providing LDP protection to every token, given its token embedding 130 in the client’s local data. This correlation improves the performance of the client’s tuning unit 106, mitigating the utility drops caused by the misalignment between the encoder 102, the cloud-hosted model (LLM 118), and the decoder 104 when using the vocabulary unit 108 and the tokenizer unit 114.

[0063] As a result, the system 100 generates code 126 with high functionality under strong LDP protection on the client side 112. Extensive analysis and experiments using LLMs (CodeLlama-7B, CodeQwenl.5-7B-Chat, and Llama3-8B-instruct) on benchmark datasets (Evalplus consisting of MBPP (see, e.g., Austin, J. et al., Program Synthesis With13MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)Large Language Models, arXiv:2108.07732 (2021)) and HumanEval (see, e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)), and BigCodeBench (see, e.g., Zhuo, T.Y. et al., Bigcodebench: Benchmarking Code Generation With Diverse Function Calls and Complex Instructions, arXiv:2406.15877 (2024))) show that the system 100 achieves a Pass@ l of 76.7 and 77.4 in code generation and completion on the MBPP (see, e.g., Austin, J. et al., Program Synthesis With Large Language Models, arXiv:2108.07732 (2021)) and HumanEval (see, e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)) and 31.1 on the BigCodeBench (a marginal drop of 1.8% compared with the original LLM) and outperforms SoTA baselines (T2T, discussed in, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023), or SnD, discussed in, e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) against reconstruction and frequency analysis attacks. (See, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023); Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)).

[0064] Related Work

[0065] Privacy-preserving Prompts. Split-and-Denoise (SnD) (such as in, e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) is one of the related works. The client injects independent draws of Laplace noise into the token embeddings of every prompt to achieve dx -privacy (relaxed LDP) (see, e.g., Chatzikokolakis, K. et al., Broadening the Scope of Differential Privacy Using Metrics, Privacy Enhancing Technologies (2013)) before sending the embeddings to the cloud. Then, the client side receives output embeddings in return and denoises them for downstream tasks. The common point among other studies (see, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)) is achieving dx -privacy by replacing tokens with randomized tokens, after injecting noise into token embeddings of a14MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) prompt.

[0066] The table of FIG. 2 summarizes key differences between the exemplary system and the methods of related works. The related works include TokEmbPriv (Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021)), Text2Text (Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020)), RAPT (Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)), and Split-and-Denoise (Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)). The existing methods were designed for classification tasks. As discussed in, e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024), generalizing these methods for (sequence-to-sequence) generation tasks is challenging. The error from the previous token prediction will compound the deviation of the following tokens and thus substantially degenerate model performance. These conclusions were observed in the experimentation provided when testing the existing methods, as noted in FIG. 2.

[0067] Departing from the SoTA, the exemplary system offers private code generation, which is significantly harder than classification tasks. The client must protect their prompts and the generated code while maintaining the generated code functionality. The exemplary system differs from the existing methods in achieving pure LDP of the vocabulary (vocabulary unit 108) and the tokenization unit 114. Specifically, the system adaptively randomizes token embeddings in the vocabulary only once. The system keeps them unchanged and consistent among all prompts and code to balance the trade-off between privacy and model performance. These properties reduce the randomness injected into the client’s prompts and code while maintaining their essential correlation in the client’s local data; hence, enabling the client to fine-tune the encoder and decoder for code generation at the client side.

[0068] Privacy-preserving Prompt Tuning (P3T). The most recent P3T (discussed generally in, e.g., Duan, H. et al., Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models, Advances in Neural Information Processing Systems, Volume 36 (2023); Wu, T. et al., Privacy-Preserving In-Context Learning for Large Language Models, ICLR (2024); Hong, J. et al., DP-OPT: Make Large Language Model Your Privacy Preserving Prompt Engineer, ICLR, (2024)) is DP-OPT (specifically15MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002) discussed in, e.g., Hong, J. et al., DP-OPT: Make Large Language Model Your Privacy Preserving Prompt Engineer, ICLR, (2024)), which fine-tunes a local LLM for ensemble prompt engineering, with DP to protect the membership information of prompts in the local LLM’s training set. This objective is different from protecting the content of the client’s prompts and generated code in both fine-tuning and inference phases of the exemplary system.

[0069] Preliminaries

[0070] LLM-based Code Generation. An LLM requires a vocabulary including a set of tokens (i.e., human-readable unique words) and their corresponding token embeddings denoted as V = {t, et], where |F |is the number of tokens in V and etG UY" and m is the number of features in et. A prompt % is a sequence of tokens, each of which is from the vocabulary the number of tokens in x.The prompt x is integrated with a task instruction it in a template T to guide an LLM in generating an output code y. For example, n can be “You are an expert in Python:” and T can be “Complete the request. ### Instruction: {instruction 71} {prompt x] ### Response:”. This process is presented as a conditional probability Ph[y\T(x, TT)] where the higher the temperature h(> 0), the more diverse the generated code. For clarity, the same instruction and template are used for prompts; thus, we reduce n and T, and consider x as the LLM’s input. The prompt x is represented as a sequence of token embeddings

[0071] Tokenization. Tokenization consists of the first and the last steps of text processing and modeling in LLMs. (See, e.g., Mistral Al, Tokenization, available at https: / / docs.mistral.ai / guides / tokenization / (2024)). A tokenizer breaks down text into tokens and assigns each token t G V a unique numerical index represented by a one-hot vector vtGwithout affecting the generality and correctness. Given the input] et[ , the LLM iteratively generates the one-hot vector vtof the next token t in the l Dy=1output code y. Finally, the tokenizer detokenizes the one-hot vectors {vt}teyback to human-readable text by mapping those vectors to their corresponding tokens using the vocabulary V.

[0072] Differential Privacy. Differential Privacy (DP) (discussed in, e.g., Dwork, C.16MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) et al., The Algorithmic Foundations of Differentia] Privacy, Found and Trends in Theoretical CS (2014)) is widely-used for data privacy, and LDP (discussed in, e.g., Kasiviswanathan, S. P. et al., What Can We Learn Privately?, FOCS, pages 531-540 (2008)) particularly protects the values of data inputs against an untrusted data curator (the cloud in the exemplary system). An LDP-preserving mechanism produces similar output distributions, preventing the curator from distinguishing the outcomes of the data inputs. As used herein, e-LDP refers at a randomized algorithm M which fulfills e-LDP, if for any two inputs x and x’, and for all possible outputs 0 of M (O G Range (M)), resulting in Pr [M(x) = O] < e£Pr [M(x’) = O], where e is a privacy budget. A smaller e enforces a stronger privacy guarantee controlling the difference of the distributions induced by x and x

[0073] Split Learning. Split Learning (discussed in, e.g., Vepakomma, P. et al., Split Learning for Health: Distributed Deep Learning Without Sharing Raw Patient Data, CoRR (2018)) enables distributed learning by decomposing a neural network, e.g., an LLM in the system, into non-overlapping segments among clients and a cloud typically. The exemplary system considers the setting of a client, who can be a small enterprise with limited resources, and a cloud for simplicity without affecting the framework generality. Common split learning frameworks adapt two architectures (see, e.g., Lin, Z. et al., Split Learning in 6G Edge Networks, IEEE Wireless Communications (2024); Pasquini, D. et al., Unleashing the Tiger: Inference Attacks on Split Learning, CCS (2021)), among which, the system leverages the following setting given the ability to protect both the input prompt x and the output code y from being observed by the cloud: The client owns the first and the last several attention blocks of the LLM while the cloud hosts the remaining middle blocks of the LLM.

[0074] Reconstruction Attacks (RAs). In RAs, an attacker aims to recover an input text from its embedding see, e.g., Ni, J. et al., Large Dual Encoders Are Generalizable Retrievers, available at https: / / arxiv.org / abs / 2112.07899 (2021); Li, Z. et al., Towards General Text Embeddings With Multi-Stage Contrastive Learning, arXiv:2308.03281 (2023); Song, C. et al., Information Leakage in Embedding Models, CCS (2020); Li, H. et al., Sentence Embedding Leaks More Information Than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence, ACL (2023)) or to recover both an input and output text from their associated back-propagated gradient. (See, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS17MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)(2024)). Vec2Text (discussed in, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023)) is an advanced embedding -based RA, consisting of base and refining steps. A trained conditional language model converts the embedding to a text corpus (the “base hypothesis”) in the base step. In the refining step, Vec2Text recursively re-embeds and corrects the base hypothesis such that the cosine similarity between the improved hypothesis’s embedding and the original text’ s embedding is better than in the previous step. The Vec2Text model is trained on this generated data. Meanwhile, BiSR (discussed in, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) is a gradient-based RA. BiSR initializes a dummy label, then iteratively improves the dummy label by using backward gradient matching and forward embedding matching.

[0075] Overview and Threat Model

[0076] Overview. Intellectual property (IP) leakage and data security motivate the development of the exemplary system, to protect data Confidentiality, information Privacy, code Confidentiality, and model Ownership (referred to herein as “CPCO” constraints).

[0077] Raw Data Confidentiality: The client’s local data, constructed by the client using (either or both) open and private sources, prompts and generated code in the inference phase, cannot be shared with the (privacy-untrusted) cloud.

[0078] Encoded-lnformation Privacy: Sensitive content in the client’s prompts and code in the training and inference phases should not be inferable by the cloud.

[0079] Code Confidentiality: The syntax, semantics, and functionality of the code in the client’ s data and the generated code in the inference phase should not be reconstructable by the cloud.

[0080] Model Ownership: The model ownership is a critical concern in practice. The parameters of the cloud-hosted model cannot be shared with the client, and thus, the client cannot generate code without using the cloud-hosted model.

[0081] General Design. To accommodate the CPCO constraints in the exemplary system 100 represented in FIG. 1, the cloud decomposes its LLM 0 into three parts: an encoder with the first few attention blocks 6e, a decoder with the last few attention blocks 6d , and the remaining middle blocks 0m. The lightweight encoder and the decoder18MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) associated with the vocabulary V and its tokenizer are shared with the client so that the client can use them at a negligible cost. Meanwhile, the cloud hosts the remaining blocks 9m. The pair of an encoder and a decoder satisfies the data confidentiality constraint. By integrating the vocabulary unit and the tokenizer unit, the system 100 design satisfies information privacy and code proprietary constraints. The cloud-hosted model 9m prevents the client from generating code using or fine-tuning only the lightweight encoder and the decoder disconnected by 9nv. therefore, satisfying the model ownership constraint. The general setting is feasible in practice because the LLM-provider has an incentive to give the encoder and decoder to the clients: (1) Clients who are sensitive to TP leakage and data security may choose LLM-providers that protect their prompts and code; LLM-providers adapt the design to get these clients as new customers; otherwise, they lose the clients; and (2) Future artificial intelligence (Al) regulation may force LLM-providers to offer such privacy protection. Using open-source LLMs naturally fits with the design of the exemplary system.

[0082] The client uses the vocabulary U and the shared tokenizer to represent every prompt x as a sequence of token embeddings 1 et. [ . This sequence is fed to the encoder t L7=19e to extract a prompt embedding £, denoted as £ = Enc ( ]( et. [ , 9e\ . The client sends D; =i / the prompt embedding 8 to the cloud-hosted model 9m and then receives an enriched embedding E, denoted as £ = femb(8, 9m~). The client feeds the enriched embedding £ to the decoder 9d to generate the output code y locally, denoted as y = Dec , 9dThe client finetunes (denoted as the tuning unit of system 100) the encoder and decoder to adapt the encoder and decoder to their local tasks primarily. The client can curate or construct the training data from open and / or the client’s private sources. The cloud has options to either (1) fine-tune a lightweight LoRA (discussed in, e.g., Hu, E. et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR (2022)) of the middle block m with the client to gain better model utility or (2) keep 9mas is to reduce the cloud’s computation complexity with a marginal utility drop as a trade-off.

[0083] Threat Model

[0084] In a defense-free environment, the client uses the cloud’s shared vocabulary and tokenizer during the fine-tuning and inference phases. FIG. 3 is a block diagram of a threat model 150 of an honest-but-curious cloud 152 on an exemplary system 150. The19MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002) honest-but-curious cloud 152 (on cloud side 160) aims to reconstruct prompts 154 and code 156 used in the fine-tuning phase of the client 158 (on client side 162), and code generated in the inference phase. To reconstruct the prompts, the cloud 152 applies the SoTA RAs (such as, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023)) to exploit the client’s prompt 154 embeddings (i.e., PromptRA unit 164).

[0085] To generate reconstructed versions of the client’s code 156 (i.e., reconstructed code 168), the cloud 152 feeds the reconstructed prompts 166 to the LLM model 9. However, given the noisy prompt embeddings, the reconstructed prompts 166 are typically imperfect, affecting the quality of the reconstructed code 170. To maximize the attack performance, the cloud 152 feeds the prompt embeddings 154 directly to the cloud-hosted model 9m concatenated with the original decoder 9d as an alternative approach. At the finetuning process, the cloud 152 can also feed the back-propagated gradient 172 of each training sample {x,yj through the middle block 9m, denoted as Vx,y9m, into BiSR (such as, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) to reconstruct the prompts x and their output code y in the client’s training set D. The system considers the best reconstructed code 170 with the maximal information leak in each metric, including information privacy, code proprietary, and code functionality, of every client’s prompt and generated code in both methods to assess the attack performance.

[0086] The analysis did not consider attacks from compromised employees in the client’s organization colluded with the cloud for the cloud to query the client’s encoder, disclose the input prompts x, the output code y, the vocabulary unit, the tokenizer unit, the fine-tuned encoder and decoder, and their associated gradients. Such inside intruders can gather all the raw data in the training and inference phases (out-of-the-scope of the system). Token sequence-based frequency analysis attacks, e.g., the codebook attack(such as, e.g., Biryukov, A. et al., Codebook Attack, pages 216-216, Springer US, Boston, MA, ISBN 978-1-4419-5906-5, doi: 10.1007 / 978-l-4419-5906-5_563, available at https: / / doi.org / 10.1007 / 978-l-4419-5906-5_563 (2011)), are not directly applicable to the system since: (1) the same token that appears in the input prompt has varied output embeddings from the encoder due to the encoder’s attention mechanism; and (2) the cloud cannot query the client’s encoder or access the client’s draw inputs. Therefore, adaptive attacks (the cloud adaptively manipulates the input prompts to observe the differences at20MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) the generated outputs) are not applicable.

[0087] Exemplary System Defense

[0088] To protect the client’s prompts and code against RAs, the exemplary system has two key components. First, the vocabulary unit replaces the vocabulary V with an LDP- preserving V={ / , c(], where every is an LDP-preserving token embedding, derived from the original token embedding et. Hence, a prompt x is represented by a sequence of LDP- preserving token embeddings ] t et. l 1|x|defending against PromptRA. Second, the system l Dy=1develops a local tokenizer (tokenizer unit), which uniformly assigns every token and its embedding to a random index in the tokenizer, on the client side to defend against CodeRA. The client keeps this data-independently and randomized tokenizer (no extra privacy cost) secret from the cloud, misleading the cloud’s attacks to reconstruct meaningless tokens in the output code y of the client’s training data D. At the inference phase, the client uses the vocabulary unit V associated with the tokenizer unit, the fine-tuned encoder 9*, and the decoder 6^ to generate code for its prompts. The client does not share the fine-tuned 0* and 0^ with the cloud, and the cloud does not share the LoRA of the middle block 0mduring the fine-tuning and inference phases to maintain their model ownership. The system maintains the CPCO constraints, incentivizing the client to use LLM-based code generation under IP and data security protection and broadening the adoption of the cloud’s service. An example algorithm for the system operation is provided in FIG. 4.

[0089] Example. FIGS. 5A-5D provide examples of a threat model from a cloud on an exemplary system. Given the prompt in FIG. 5 A, the cloud reconstructs the client’s generated code with a clear gist in the defense-free environment (FIG. 5C). On the contrary, the cloud reconstructs meaningless code under the vocabulary unit and the tokenizer unit (FIG. 5D). Meanwhile, the client enjoys its desired code with the system on their side privately (FIG. 5B). Next, reconstruction attacks as security games between the client and the cloud are discussed below.

[0090] Reconstruction Attacks

[0091] Prompt Reconstruction A ttack

[0092] In PromptRA, the cloud feeds each prompt embedding E into Vec2Text (such as, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023)) or feeds the back-propagated gradients Vx,y0mto BiSR (such21MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) as, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)), the SoTA RAs to generate x approximating the ground-truth prompt x, formulated as PromptRA : ({ £ }, 0t) U ({V^.ySm}, 0) -^x, where 9ais the pre-trained parameters of Vec2Text. The more similar x and x are, the better the attack, resulting in more information privacy leaked. In the security game, the cloud can request the client to send the prompt embeddings 8 of all prompts x and their associated back-propagated gradientsx,y9min the training set D at any time, i.e., multiple requests at different times, during the client’s local fine-tuning. The sets of all prompt embeddings are denoted as IE and their reconstructed prompts as X.

[0093] To evaluate the attack success rate, we employ well-known token-level metrics, including Bleu score (see, e.g., Papineni, K. et al., Bleu: A Method for Automatic Evaluation of Machine Translation, ACL (2002)) and Rouge score see, e.g., Lin, C.Y. et al., ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, ACL (2004)), to quantify the similarity between the reconstructed prompt x. and the prompt x. The cloud wins the game for a given prompt x in the training set D if it returns a reconstructed prompt x G X with a clear gist: either max{Bleu(x, x}i6> pb, where pb = 20 (see, e.g., Google Cloud AutoML Translation, Evaluating Models - BLEU, AutoML Translation Documentation, “Evaluate” Section, Cloud Translation AutoML (2025)), or max {R ouge (x, x]yey Pr where pr= 0.4 (see, e.g., Lin, CY., ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, pages 74-81, Barcelona, Spain, Association for Computational Linguistics (2004)) The overall attack success rate (ASR) during the client’s fine-tuning is the cloud’s average winning rate over all prompts in the training set D. The cloud requests all prompt embeddings in the test set Dtest once in the inference phase, as multiple requests make no difference. In the study, Bleu and Rouge scores were used with the uni-gram by default since it yields the best scores in the results compared with bi- and longer- grams; otherwise, the specific n-grams used are specified.

[0094] Code Reconstruction Attack

[0095] In CodeRA, the cloud 1) feeds the prompt embeddings {£} to the cloud -hosted model 9m concatenated with the original decoder 9d, denoted as 9m © 9d, or 2) feeds the reconstructed prompts x to the LLM 9, or 3) feeds the back-propagated gradientsx,y9m to22MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)BiSR (see, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split- Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) to generate the output code y approximating the ground-truth code y, i.e., CodeRA: ({£}, 0m®0d)U({ %}, 0)U({Vx.y0m}, 0) ~^y ■ The more similar the code y to the reconstructed code y is, the better the attack and the higher the risk are. This similarity is quantified by: (1) Information Privacy: token-level metrics are employed, including Bleu score and Rouge score, to quantify the similarity between the reconstructed code and the ground-truth code, y and y correspondingly; (2) Code Proprietary: CodeBleu (see, e.g., Ren, S. et al., Codehleu: A Method for Automatic Evaluation of Code Synthesis, CoRR (2020)) is employed taking into account code syntax and semantic similarity between y and y; and (3) Code Functionality: The more similar unit tests y and y pass, the more similar functionality they share, defined as Equation 1 (below):where Uxis a set of unit tests u to access the functionality of the generated code, given a prompt x.pass(y,u)= 1 if the code y passes the unit test n; otherwise, pass(y,u)=0. D[.] = 1 if pass(y, u) = pass(y, u) = 1; otherwise, U[.] = 0.

[0096] The cloud wins the game for a given code y in the training set D if it returns a reconstructed code y with a clear gist regarding information privacy, code proprietary, and code functionality. Fusi is stricter than Bleu, Rouge, and CodeBleu since passing unit tests demands rigorous semantic and syntax correctness of all the tokens in the code. Incorrectly generating a single token or its order in the reconstructed code can cause the code to fail unit tests while negligibly affecting Bleu, Rouge, and CodeBleu scores. Hence, Fusi typically has smaller values than Bleu, Rouge, and CodeBleu, resulting in smaller ASRs in code functionality than information privacy and code proprietary. Passing only one similar unit test with the ground-truth code indicates a leak of the client’s code functionality caused by the reconstructed code. Therefore, the cloud’s favorable threshold is considered Fusi (y,y)>Pf = 0, of winning a security game (a leak or no leak) in code functionality.

[0097] The cloud wins the game for code y in terms of i) information privacy if either Bleu(y,y) > pb (= 20) (see, e.g., Google Cloud AutoML Translation, Evaluating Models - BLEU, AutoML Translation Documentation, “Evaluate” Section, Cloud Translation AutoML (2025)) or Rouge (y,y) > pr(= 0.4) (see, e.g., Walker, S.M. et al., What is the23MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)Rouge Score (Recall-Oriented Understudy for Gisting Evaluation)?, klu.ai, available at https: / / klu.ai / glossary / rouge-score), ii) code proprietary if CodeBleu(y,y) > pb (= 20), and iii) code functionality if Fusi (y,y) > pf (= 0). The overall ASR is the average cloud’s winning rate over all the code in D. The cloud conducts the same attack in the inference phase with one request for all prompt embeddings in the client’s test set.

[0098] Token-Level Privacy Metrics

[0099] In addition to Bleu, Rouge, CodeBleu, and Fusi metrics, correctly reconstructed tokens (CRT) are also reported in the prompt x and code y with associated metrics q uuantified as follows: CRTXxability of the cloud to reconstruct sensitive information, including imported packages, variable names, and function names, from the generated code, was further evaluated, which is quantified by leak = ^Xyen leak(y), where leak(y) = 1 if there exists a piece of sensitive information (i.e., imported packages, variable names, and function names) in the reconstructed code, and leak(y) = 0 vice-versa. These metrics and inspections provide a comprehensive evaluation of privacy leakage at the structural content and token levels in PromptRA and CodeRA.

[0100] Initial Assessment of PromptRA, Code RA

[0101] The following experiment was conducted to assess Ras in the defense-free environment using benchmark CodeAlpaca (see, e.g., Chaudhary, S. et al., Code Alpaca: An Instruction-Following Llama Model for Code Generation, available at https: / / github.com / sahil280114 / codealpaca (2023)) and MBPP (see, e.g., Austin, J. et al., Program Synthesis With Large Language Models, arXiv:2108.07732 (2021)) datasets from the Evalplus benchmark. Both CodeAlpaca and MBPP are datasets for Python code generation tasks. CodeLlama-7B was used as the LLM 9, such that the first attention block is the encoder 9C, the decoder 9d consists of the last four attention blocks, and the cloud- hosted model 9mconsists of the remaining middle attention blocks. The CodeAlpaca dataset, consisting of about 18k data points, was larger than the MBPP dataset, which consisted of 974 data points. The cloud initialized the BaseModel and Vec2Text from a T5- base checkpoint (see, e.g., Raffel, C. et al., Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer, arXiv: 1910.10683 (2023)) as in the discussion of, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages24MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)12448-12460 (2023). Then, the cloud trained its PromptRA on the (larger) CodeAlpaca as public data. The batch size was 24, and the max sequence length was 768 tokens since the original texts in CodeAlpaca are long. Meanwhile, the client fine-tuned its encoder and decoder on the (smaller) MBPP dataset.

[0102] Without fine-tuning and with no test cases in the client’s prompts, PromptRA achieved high Bleu scores, i.e., 34.17 and 35.53, for the prompts in the training and the test sets, respectively (see table of FIG. 6, and examples in the tables of FIGS. 7-9). Also, PromptRA achieves excellent Rouge scores, i.e., 0.66 and 0.67. As a result, the privacy attack success rates ASR1and ASR ™ are 96% indicating that the cloud can reconstruct meaningful content in most of the client’s prompts. Interestingly, when the client includes test cases into their prompts, PromptRA has moderate scores of Rouge, i.e., 0.4 and 0.39, resulting in lower ASRs, i.e., 50% and 54%, for the training and test sets, respectively. For example, given the prompt “Write a Python function to find the remainder of two numbers,” a test case is ‘assert find(3,3)=0.’ This is because Vec2Text (see, e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023)) and BiSR see, e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) have not been designed for lengthy prompts with code and test cases. Although PromptRA is less effective, CodeRA achieves excellent Bleu, Rouge, CodeBleu, and Fusi scores, i.e., 74.32, 0.86, 67.87, 0.46, respectively, when prompts have test cases; this results in high ASRs, i.e., over 97% privacy and code proprietary ASRs (98.25% on average), and an average of 46.5% code functionality ASR across training and test sets, i.e., code functionality leaks in 46.5% of all clients’ prompts.

[0103] The cloud still registers high ASRs in both PromptRA and CodeRA when the client fine-tunes its encoder and decoder and keeps them secret. Privacy ASRs are 51 % for prompts, 95.5% for prompts with no test cases, and 99% for code. Code proprietary ASR is 97%, and code functionality ASR is 45.75% in training and test sets on average. Similar results were achieved on other benchmark datasets, i.e., HumanEval (e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)), BigCodeBench (e.g., Zhuo, T.Y. et al., Bigcodebench: Benchmarking Code Generation With Diverse Function Calls and Complex Instructions, arXiv:2406. 15877 (2024)), and LLMs, i.e., CodeQwenl.5-7B-Chat, Llama3-8B-instruct. The BigCodeBench, an advanced25MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) evaluation of LLMs in programming, consists of 1 ,140 data points. The token-level metrics C / ?7\ and CRTyare 0.28 and 1.00 on the MBPP dataset, respectively. The same attack performance was observed on the BigCodeBench. The leak scores of sensitive information were high, i.e., 0.98 on both the MBPP and BigCodeBench.

[0104] The initial results show that a cloud can reconstruct the content of prompts and code with severely high Bleu, Rouge, Code-Bleu, Fusi, CRT, and leak scores, resulting in high ASRs. A single high ASR among these metrics is sufficient to cause remarkable IP and data security risks for the client. Therefore, developing an effective privacy-preserving mechanism (via the exemplary system) is crucial to protect clients’ prompts and code while maintaining high model performance.

[0105] LDP-Preserving Vocabulary

[0106] RAs infer tokens in the prompt x from its embedding 8 to reconstruct prompts and code. Consider a prompt x at the token level: x represented as a sequence oftoken embeddings where tj E V. The closer the reconstructed token embeddingsto the ground-truth ones in x are, the more precise the cloud can infer every token tj in %; thus, it can reconstruct the prompt x and corresponding code y. To provide privacy guarantees, the worst-case attack is considered, in which the cloud can losslessly reconstruct the ground-truth token embeddings: tj E x ■ et. = et, where et. is the reconstructed token embedding at the position of the token tj in x.

[0107] To defend against this worst-case, the client can apply existing methods to randomize tokens in the prompt x and the code y for local fine-tuning, resulting in replacing tokens with other tokens to preserve dx-privacy (see, e.g., Chatzikokolakis, K. et al., Broadening the Scope of Differential Privacy Using Metrics, Privacy Enhancing Technologies (2013)). Then, the client can either denoise the outcome from the cloud- hosted LLM (see, e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) or fine-tune the decoder to generate code privately as in the exemplary system. The classical purpose of preserving dx-privacy at the token level is to prevent a data curator from inference authorship of a piece of text (see, e.g., Mattern, J. et al., The Limits of Word Level Differential Privacy, ACL (2022)) rather than reconstruction attacks. Given this mismatched purpose, the following fundamental challenges are faced.26MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0108] Challenges. First, a subtle perturbation at the token level in the prompt x can significantly change the content of the prompt, such as code syntax, code functionality, function names, variables, or the like. LLMs are sensitive to these changes, generating irrelevant and poorly functional code. Also, if the token t appears in multiple prompts, perturbing the token embedding of t in these prompts with different draws of random noise to achieve dx-privacy at the token level amplifies the randomness introduced to these prompts. This is because a single token is represented by different randomized token embeddings, causing model performance degeneration. Second, the instruction is significant to the performance of LLMs. However, the randomness is applied and impacts the instruction n and the template T(x, TT). Poor instructions with notable randomness further degenerate model performance. Hence, achieving good model performance in code generation is challenging while protecting prompts and code under LDP against PromptRA and CodeRA from a cloud.

[0109] LDPVocab (Vocabulary Unit)

[0110] To address the challenge, the system includes a concept of LDP-preserving vocabulary (performed by the vocabulary unit). As discussed herein, e-LDPVOCAB (incorporated into the vocabulary unit) refers to a randomized algorithm AT fulfills e-LDP, if for any two tokens t and t' in the vocabulary V, and for all possible outputs O of M , i.e., O E Range (M), we have: Pr[M(et ) = (?] < e6Pr\M(et ) = (?].

[0111] e-LDPVOCAB ensures that the cloud cannot distinguish the outcomes M(et) of the original token embeddings et under e-LDP protection. Instead of using the original token embeddings { et} tev in the vocabulary V, the client applies algorithm M to randomize these token embeddings resulting in LDP-preserving token embeddings {et— M(et)}teIZin a new LDPVOCAB V . By locally replacing V with V to map a token t to an embedding et, the client prevents the cloud from inferring the ground truth tokens. This is because the cloud can only reconstruct the randomized token embeddings {et}t6lzinstead of the ground-truth token embeddings. Thus, the tokens and the prompt x are protected against PromptRA. Using the LDPVOCAB V to derive the prompt embedding by feeding the prompt x to the encoder de results in an LDP-preserving prompt embedding e following the post-processing property in DP (see, e.g., Dwork, C. et al., The Algorithmic Foundations of Differential Privacy, Found and Trends in Theoretical CS (2014)). The LDP-preserving tokens, prompts, and prompt embeddings e further prevent the cloud from27MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002) reconstructing the code y with CodeRA.

[0112] Achieving LDPVOCAB. The system includes a vocabulary unit having an adaptive randomized response (ARR) mechanism as the algorithm AT to preserve LDPVOCAB while maintaining high model utility with key advantages to overcome the challenges and limitations related to token-level LDP (see, e.g., Mattern, J. et al., The Limits of Word Level Differential Privacy, ACL (2022)) in code generation. The pseudocode of the ARR mechanism is in Lines 4-8 of the algorithm in FIG. 4. The ithfeature in a token embedding et is denoted as et'. The ARR’s key idea is flipping a probabilistic coin whether the original value of the feature etLis kept or changed to another possible feature value among other tokens {e / <}kel / ..t- where k is a token different from t in the vocabulary V, such that more similar feature values to etlhave higher probabilities to be selected as a replacement. The idea is formulated as discussed below.

[0113] Given an arbitrary token t G V, Vi'h-I'eature of er.token embedding, and pt+Ykevy Qi.k — 1- The larger qkindicates that ekis more similar to e^. Theorem 1 bounds Pi s.t. V / c G V\t: pt> qi kpreserving Gj-LDP as in typical RR definition (see, e.g., Warner, S.L. et al., Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias, JASA, 60(309):63-69 (1965)). Theorem 1 is that randomizing the ithfeature in a token embedding etpreserves Gj-LDP if 8; and Pi are bounded as:

[0114] Proofs of all Theorems. In practice, the upper-bound of ft is used in Theorem28MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)1, since larger f3t yields less noisy randomization probabilities. ARR is applied to independently randomize every / "‘-feature in a token embedding et with a privacy budget 6i. Thus, an e-LDP-preserving token embedding etwhere the total privacy budget is 6 = Sieetei following the composition theorem in DP (see, e.g., Dwork, C. et al., The Algorithmic Foundations of Differential Privacy, Found and Trends in Theoretical CS (2014)) reflected in Theorem 2 (discussed below). The total LDP budget e is lower- bounded as follows: ewhich is tiny ((< le - 3) in almost all LLMs, enabling the system to operate with a high privacy protection (if needed) in practice.

[0115] The proof of Theorem 1 is provided below. Given two possible values etland etlof the i,h-feature in the token embedding et, i.e., etl,andanY possible output z G Range (ARR), where Range(ARR) denotes every possible output of ARR, which gives:

[0116] Following the typical definition of RR mechanisms (see, e.g., Warner, S.L. et al., Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias, JASA, 60(309):63-69 (1965)), the probability pt must be larger than or equal to any probabilities qi.k. Having this condition, it is assumed: vk G V\t : Pt > qi k(5)From Equations 4 and 5, the following is obtained:Taking a natural logarithm of Equation 6, the following is obtained: (exp(ft))29MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)For Equation 7 to hold, the condition in Equation 5 must hold. Equation 5 can be rewritten as:

[0117] For Equations 7 and 9, which represent the upper-bound and the lower-bound of Pi respectively, to hold simultaneously, resulting in feasible Pi, the upper-bound needs to be larger than or equal to the lower-bound, as follows:From this,following is found:Consequently, from Equations 7, 9 and 11, Theorem 1 holds.30MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)

[0118] Post-Processing Property. The client applies LDPVOCAB V to represent every prompt x and the instruction n as a sequence of e-LDP-preserving token embeddings, e.g., the sequence of token embeddings given a prompt x is ] et. I . The number of timesa token t appears in one or more prompts does not affect the e-LDP guarantee of the token embedding of t following the post-processing property of DP (see, e.g., Dwork, C. et al., The Algorithmic Foundations of Differential Privacy, Found and Trends in Theoretical CS (2014)); that is, no extra information regarding the ground-truth token’s embedding is used afterward. The client freely uses the LDPVOCAB V without affecting the LDP protection.

[0119] Theorem 2 is applying ARR to independently randomize every ith-feature with a privacy budget et in a token embedding et preserves e-LDP, where G= XieetG; . In terms of the proof, given any two possible embeddings et= {et}ieetandof a token t, i.e., With-feature in et: = etl, e^' E {e^}lev, and any possible output 0 = [zlEin e> where RangeiARR) denotes every possible output of ARR, which gives:Consequently, Theorem 2 holds.

[0120] Prompt-Level Protection

[0121] In leaking authorship information of a piece of text (see, e.g., Chatzikokolakis, K. et al., Broadening the Scope of Differential Privacy Using Metrics, Privacy Enhancing Technologies (2013)), it is known that token-level dx-privacy has two major weaknesses (see, e.g., Mattern, J. et al., The Limits of Word Level Differential Privacy, ACL (2022)): (1) Privacy budget and authorship information leakage is accumulated across the number of tokens in a sentence; and (2) Privacy protection varies for different sentences; that is, sentences having other numbers of tokens have different privacy budgets. However, authorship leakage is not an issue in the study performed since the cloud knows the client’ s identity to provide the code generation service. Also, authorship information can be protected by using other well-known techniques, e.g., Tor, discussed in Dingledine, R. et al., Tor: The Second-Generation Onion Router, USENIX (2004). In the study performed,31MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) the aim was to protect prompts and code against reconstruction attacks from the cloud, a more complex and widely open problem. Therefore, the focus was on providing upper- bounded probability for a given prompt x to be reconstructed, hence, shedding light on the correlation between LDPVocab and reconstruction risk.

[0122] To achieve the goal, a security game was presented (FIG. 10), in which the client randomly picks an e-LDP-preserving token embedding o G {et}tEV. Then, the client sends o and the original vocabulary V associated with the ground-truth token embeddings {et}telzto the cloud. The cloud wins the game if it can infer the ground-truth token t of the 6-LDP -preserving token embedding o. The cloud returns one token in response to the client among all |7| possible tokens. The cloud does not observe the game’s outcome, that is, whether it wins the game or not. In practice, the client does not send any extra information, which is the game outcome, derived from the original ground-truth token embedding to the cloud, maintaining the CPCO constraints. Thus, the number of times the client plays the game and the number of times an c-LDP-preserving token embedding selected in a game does not affect the LDP protection (the post-processing property (e.g., Dwork, C. et al., The Algorithmic Foundations of Differential Privacy, Found and Trends in Theoretical CS (2014))). In other words, one game does not affect other games since the cloud receives no extra information about the LDPVocab V from playing the games. Theorem 3 bounds the cloud’s probability to infer that an arbitrary t G V is the ground-truth token of an observed outcome o, denoted as P[ARR(et) = o]. It is noted that Vt t': etet,.

[0123] Theorem 3 states that given an e-LDPVocab V, the probability that an arbitrary t G V is the ground-truth token of an observed e-LDP token embedding o G {et]tEV, denoted as P[ARR(et) = o], is bounded as follows:With respect to the proof, given an arbitrary e-LDP-preserving token embedding o G and the vocabulary V, the cloud will make an inference whether o is a result of randomizing one of the |V| token embeddings in the vocabulary. Therefore, given | V | possible outcomes from the cloud perspective, the result is:%tevP[ARR(et) = o] = 132MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)From the definitions herein and Theorems 1 -2, the result is that:Given the observed outcome o E O, the result is:By applying this result to all t’G V\t, the result is:From Equations 14 and 17, the following is found:100124J In addition, it is found that:Given the observed outcome o E O, it is found that:P[ARR(et,) = o] < eeP[ARR(et) = o], (20)By applying this result to all t’6 V\t, it is found that:1 - Zt'ev\t P[ARR(etP) = o] > 1 - (| F| - l)e£P[ARR(et) = o], (21)From Equations 14 and 21, it is found that:Consequently, from Equations 18 and 22, Theorem 3 holds.

[0125] From Theorem 3, it is evident that the “larger the number of tokens” in the vocabulary V and the “smaller the privacy budget e” is, the “lower the probability” for the cloud to infer the ground-truth token, offering rigorous privacy protection. The security|x| game can be extended by allowing the client to pick an arbitrary prompt x = {t7}represented as a sequence of e-LDP preserving tokens33MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) and the client sends to the cloud. The cloud wins the game if its reconstructedprompt x provides a clear gist of x by identifying ground-truth tokens [t7] in x. For instance, Bleu(x, x) > p(= 20) for a clear gist of x (see, e.g., Google Cloud AutoML Translation, Evaluating Models - BLEU, AutoML Translation Documentation, “Evaluate” Section,Cloud Translation AutoML (2025)), formulated Like theprevious security game, the client does not send the game outcome to the cloud. Therefore, the cloud observes no extra information derived from the ground-truth token embeddings in playing the games. As a result, the number of games, the number of times a token appears in one or more prompts, and the number of times a prompt is selected in these games do not affect the e-LDP protection of the token embeddings (the post-processing property of DP (e.g., Dwork, C. et al., The Algorithmic Foundations of Differential Privacy, Found and Trends in Theoretical CS (2014))).

[0126] In this security game, the numbers of tokens in the reconstructed and groundtruth prompts, x and x, are the same: |x| = | x |. Hence, Rouge precision is equal to Rouge recall: Rouge-precision(x, x), Rouge-recalKx, x)=C / |x|, where C is the number of correct reconstructed tokens in x. As a result, Rouge-F\ (x, x) (e.g., Lin, C.Y. et al., ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, ACL (2004)) is equal to C / |x|, which is equivalent to Bleutx, x)=C / |x|. Therefore, we focus on analyzing the correlation between the LDP budget e, the vocabulary size | V | , the size of the prompt |x|, and the threshold p in reconstructing a clear gist of x using the Bleu score below. The following proposition limits the cloud’s upper-bounded probability of winning the security game.

[0127] Proposition 1. The cloud’s probability to identify ground- truth tokens in x with a clear gist is upper-bounded as:where = |P | — 1.

[0128] In terms of the proof, the Bleu score definition is as follows:Bleu(x, x) = BP(x, x') X C / |x|, (24) where BP (x, x) stands for the brevity penalty given for the mismatched length between the34MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) reconstructed prompt x and ground-truth prompt x, and C is the number of correct reconstructed tokens in x. In the security game, BP(x, x) = I since x. and x have the same length.

[0129] The probability of having C correct reconstructed tokens, assuming (tj #= tj } ._c+1without loss of generality is:p (ft nll+where oyJ indicates the cloud’s probability to infer than tj is not theground-truth token of the observed e-LDP preserving token embedding oy.

[0130] From Theorem 3, it is found thataddition, it is found that:From Equations 13, 25 and 27, it is found that:where ip = |F| — 1.

[0131] From Equation 29, the probability to reconstruct a clearer gist of a prompt x, i.e., C increases, is reduced monotonically. It is because ip » 1 and C G [0, |x| ]. Therefore, it is found that:= / ’ (ft = CT^ Cw+I) {<1) O')35MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)Consequently, Proposition 1 holds.

[0132] Theorem 4. The cloud’s previously reconstructed token sequences t<7can enhance its probability of correctly reconstructing the next token t P(tj = his advantage is bounded by a constant y in practice, as follows:Given a bounded constant y, Proposition 1 is extended, bounding the probability to reconstruct a clear gist of the prompt x exploiting t<7, as follows:

[0133] In terms of the proof, the probability of having C correct reconstructed tokens,

[0134] From Equation 33, it is found that V / : P(t7= tj |t<7) < P(t7= t7) + y.Considering the worst-case for the client, in which the cloud has the maximal probability of correctly reconstructing every token tj, denoted as+ Y-The corresponding probability of incorrectly reconstructing a token tj is: =1 From Equation 36, it is found that:Consequently, Theorem 4 holds.

[0135] As discussed herein, in the study, the aim was to safeguard prompts and code against reconstruction attacks from the cloud, a complex and widely open problem. Thus,36MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) the focus was on providing upper-bounded probability for a given prompt x to be reconstructed in the security game described above, shedding light on the correlation between the token- level LDPVOCAB and the prompt- level privacy protection under reconstruction risk. In this security game, the numbers of tokens in the reconstructed and ground-truth prompts, x and x, are the same: |x| = |x|. Hence, Rouge- Fl (x, x)=Bleu(x, x)=C / \x\, where C is the number of correct reconstructed tokens in x. The analysis in Theorem 3 and Proposition 1 (above) upper-bounds the cloud’ s probability of identifying ground-truth tokens in x with the level of gist higher than or equal to p. as follows:where ip = | V | — 1. Proposition 1 is generalized to consider the reconstruction attack with token sequences, capturing correlation among tokens in real-world scenarios, as follows: The cloud’ s previously reconstructed token sequences t<7can enhance its probability of correctly reconstructing the next token t . Pr(tj = tj\t<j'). This advantage is bounded by a constant y G [0, 1] in practice, as follows:Given y, Theorem 4 (discussed above) tightens the upper-bounded reconstruction risk, as follows:

[0136] Reconstruction Probability Analysis

[0137] Proposition 1 shows that the reconstruction probability of a clear gist is upper- bounded by a function of the LDP budget e, the vocabulary size | V | , the size of the prompt |x|, and the threshold p. The analysis in FIG. 11 and FIGS. 12A-12C sheds light on the correlation between these factors.

[0138] LDP Budget e. FIG. 11 illustrates the correlation between the upper-bounded probability to reconstruct a clear gist of the prompt % as a function of the LDP budget e in the security game. It is understood that the smaller the privacy budget, i.e., stronger privacy protection, the lower the probability for a prompt x to be reconstructed. Specifically, up to 6 = 14, the probability for the cloud to reconstruct a clear gist (p = 20) of a prompt x (i.e., P ^Bleu(x, x) > p; = 0.23) is significantly lower than the probability for the37MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) cloud to observe a useless gist, i.e., P >p;=0-77, with the vocabulary size |F| = 90, 000 and the prompt size |x| = 100.The gap is remarkably increased when e decreases. At e = 13, the cloud observes a useless gist with a probability of at least 0.98 while the probability for it to infer a clear gist is smaller than or equal to 0.02. This result is encouraging since the client can use a wide range of the LDP budget up to e = 13 to optimize model performance better safely.

[0139] Vocabulary Size |7|. In this analysis, the upper-bounded reconstruction probability was examined as a function of the vocabulary size |V|. FIG. 12A shows that larger vocabulary sizes impose lower reconstruction probabilities of a clear gist P ^Bleu(x, x) > p; significantly lower than the reconstruction probabilities of auseless gist P ^Bleu(x, x) < p; • This result is reasonable since it is harder for thecloud to identify correct ground-truth tokens when there are more tokens in the vocabulary. Also, smaller privacy budgets result in lower reconstruction probabilities of a clear gist. It is worth noting that vocabulary sizes of state-of-the-art LLMs are notably larger than 15k, and some are up to 90k+ (FIG. 13); hence, reducing the reconstruction probabilities of a clear gist to be significantly lower than the reconstruction probabilities of a useless gist when e = 13. For instance, CodeQwenl.5-7B-Chat (e.g., Bai, J. et al., Qwen Technical Report, arXiv:2309.16609 (2023)) has a vocabulary size of 92,416 tokens with the exact reconstruct probability of 0.02.

[0140] Prompt Size |x|. FIG. 12B shows that, with 6 = 13, if the prompt has 20 or more tokens, a clear gist reconstruction probability (0.47) is smaller than the reconstruction probability of a useless gist (0.53), indicating a safe level. When a prompt has 50 tokens or more, the reconstruction probability of a clear gist reduces to 0.15, significantly smaller than the reconstruction probability of 0.85 for a useless gist. This result shows good practicability of LDPVocab since the number of tokens in a prompt for code generation and completion tasks is usually larger than 50. For instance, benchmark datasets, including MBPP for the code generation task and HumanEval for the code completion task, have prompts with more than 50. Also, it is intuitive that when e increases, the reconstruction probability of a clear gist also increases since it offers looser LDP protection.

[0141] Reconstruction Threshold p. In this analysis, the threshold p was varied while38MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) keeping the vocabulary and prompt sizes as 90k and 100, correspondingly. In FIG. 12C, the reconstruction probability for the cloud to infer a clear gist of the prompt is almost 0, given the LDP budget 6 = 13. When p < 20 indicates a vague or almost useless gist, the reconstruction probability is still lower than 0.5. Although the cloud can reconstruct more information when the LDP budget 6 increases to 15, the result shows that it is hard to reconstruct an understandable gist, i.e., p > 30.

[0142] FIGS. 11, 12A-12C, and 13 thoroughly analyze this upper-bounded reconstruction risk as a function of 6, prompt size |x|, vocabulary size |V|, and reconstruction threshold p to demonstrate the effectiveness of LDPVOCAB in protecting the input prompt x. It prevents the cloud from reconstructing a clear gist (p = 0.2) from a prompt and code having 50 or more tokens with reconstruction probability less than 0.05, which is tiny for modern LLMs (151k or more tokens in their vocabulary) in a wide range of 6 up to 13 (Equation 33). The small upper-bounded constant y < 0.146 derived from the Evalplus benchmark datasets further shrinks the reconstruction probability to less than 5e - 4 (Equation 35). The upper-bounded y was computed with Pr(t7= ty |t<j) for every prompt in the dataset: D = y= t / ) / |D|. It is noted that the longer the input prompt, the lower the upper-bounded reconstruction probability is. This range of LDP budget with small upper-bounded reconstruction probabilities for existing LLMs and typical prompts in practice enables the client to achieve high model utility when fine-tuning the encoder and the decoder under LDP protection against RAs.

[0143] Advantages of LDPVocab

[0144] LDPVOCAB has fundamental advantages to achieving good model utility while protecting prompts and code.

[0145] Unchanged Tokens and Negligible Noise. If tokens in prompts x, the instruction n, and the template T are replaced by other tokens as in T2T (see, e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)), the syntax correctness of the code will be destroyed since an inappropriate token can remarkably degenerate code functionality. Also, it distorts the essential correlation between a prompt x and its output code y. Fine-tuning a model on distorted correlations between prompts and their desired code results in a model39MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) that generates code with functionality deviated from the prompt’s requirement. Although voiding changing the tokens in prompts, the instruction, and the template, randomizing every token embedding with different draws of considerable noise such as in SnD (e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) still distorts the correlation among x, n, T, and code y since many token embeddings can represent one token. As a result, the model performance is notably degenerated.

[0146] LDPVOCAB keeps all the tokens in the prompts x, instruction TT, and the template T unchanged. Tokens’ embeddings are randomized once with negligible e-LDP noise. Thus, LDPVOCAB maintains the essential correlation among prompts x, instruction 7T, and the template T in the client’ s local data. This property lets the client fine-tune the encoder and decoder, better balancing model utility and privacy. To show this property, four experiments were conducted using the MBPP dataset: (1) Quantifying the percentage of changed tokens on average in the input prompts with test cases after LDP preservation: (2) Measuring the Zi-norm distance between the LDP-preserving and the original token embeddings on average using 32k tokens of the CodeLlama-7B model’s vocabulary: {|et— et\1}t£x x£D; (3) Measuring changes in terms of angles between bi-gram (i.e., two sequential) tokens tt and ti+i in prompts x: cosine similarityfunction; and (4) Measuring changes in terms of angles between any pairs of tokens in LDPVOCAB given their corresponding angles in the original vocabulary: {|cos(et, et / ) —For a fair comparison, the privacy budget r / dxin T2T and SnD (p is predefined) equals the e in the exemplary system.

[0147] FIG. 14A shows that T2T (e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility- Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)) changes almost all tokens (~ 100%) in every prompt and test case. Meanwhile, SnD (e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)) and the exemplary system do not change tokens in their prompts and test cases. Although not changing tokens, SnD injects large noise into its token40MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) embeddings with the average Li-norm distance is ~645k compared with 98.06 on average of the exemplary system (FIG. 14B). The Li-norm distance between the LDP-preserving and the original token embeddings in the exemplary system is notably smaller than T2T (102.13 on average), i.e., statistically significant, p-value < 3.3e-100 (2-tail t-test). Only the exemplary system keeps the tokens unchanged in prompts and test cases while injecting negligible c-LDP noise into token embeddings.

[0148] LDPVOCAB obtains a significantly smaller bi-gram angular change on average compared with T2T and SnD in a wide range of LDP budgets e, with an average of 33%+ improvement (FIG. 15A). The larger the e is, the better LDPVOCAB can maintain the bi-gram angles compared with the SoTA approaches. As a result, LDPVOCAB maintains better the angular correlation among bi-gram tokens under LDP protection. FIG. 15B shows that the exemplary system achieves the least angular change among tokens on average compared with T2T and SnD in a wide range of e, indicating that LDPVOCAB has better ability in maintaining the relative angular correlation among tokens.

[0149] Adaptive Randomization. To assess the adaptive randomization effectiveness, the ARR was compared with a uniform randomized response (URR), in which every token embedding feature has the same randomization probability (see “Prompt-Level Protection” section above). In this experiment, one token t and a feature t,hin its token embedding were randomly picked. FIG. 16 shows the correlation between the probability qi,k and the feature similarity qk given the ARR and the URR mechanisms. Features similar to eLlhave higher probabilities of being selected in the ARR than the uniform probability in the URR. In fact, features with similarity qk and probabilities qt,k larger than the ones of the URR mechanism are the most popular (i.e., the rectangle selection in FIG. 16). As a result, the exemplary ARR better maintains the utility of the token embeddings than the URR under the same LDP protection.

[0150] -LDP -Preserving URR

[0011] In this section, a uniform randomized response (URR) is discussed, in which every token embedding feature has the same randomization probability controlled by a uniform hyper-parameter / ?, as follows:Given an arbitrary token t 6 V, Yi E et41MEl\56762934.vlAttorney Docket No. 94350.00296 (NJIT Ref. 25-002)where / ? is upper-bounded as follows: 0 < / 3 < et.

[0152] The following theorems show that applying URR using Equation 36 preserves €i - and e-LDP on the whole token embedding. Theorem 5 is that randomizing the infeature in a token embedding et using Equation 36 preserves ei-LDP if / ? is upper-bounded, as follows:P < £i + ln(|R| - 1) (37)

[0153] Given two possible values e{ and ez' of the (‘“-feature in the token embedding any possible output z E Range(URR), where Range(URR)denotes every possible output of URR, we have:Given / ? > 0, we always have that: Wk : p > qk . Therefore, from Equation 38 it is found that:

[0154] Taking a natural logarithm of Equation 39, the following is obtained: f> < st in addition to / ? > 0. Consequently, Theorem 5 holds.

[0155] THEOREM 6. Applying URR to independently randomize every (‘“-feature with a privacy budget et in a token embedding et preserves c-LDP, where c

[0156] With respect to the proof, given any two possible embeddings et— {etl]iEetand e't=°fatoken t, i.e., Vi,h-feature in et: e^ e^' 6andany possible output denotes everypossible output of URR, which results in:42MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)Consequently, Theorem 6 holds.

[0157] STuning (Tuning Unit) and LTokenizer (Tokenizer Unit)

[0158] Model Misalignment. After deriving an effective LDPVOCAB preventing the cloud from reconstructing the client’s prompts and code, the focus is now on fine-tuning the encoder 9eand the decoder d such that the client can privately generate desirable code in the system. Although supported by the advantages of LDPVOCAB, fine-tuning the encoder and the decoder to achieve high performance in generating code can still be challenging. Per the privacy protection effect, the LDPVOCAB creates a misalignment between the encoder’s output distribution and the cloud-hosted LLM’s input distribution. This misalignment is amplified through the latent space of the cloud-hosted model 9m, creating significant distribution misalignment between the encoder output and the decoder input, consequently degenerating model performance as the trade-off for LDP protection.

[0159] STUNING. To address the model misalignment, the system employs split tuning, in which the client computes the loss function using the logits at the last layer and the raw one-hot vector vt of the next token t in the output code y, computing the gradient of the decoder 9d, sending the gradient up to the last layer of the middle block 9m back to the cloud that continues the back-propagation computing the gradient of the middle block 0m’s LoRA. To complete the iteration, the cloud sends the gradient up to the last layer of the client’ s encoder, and the client continues the back-propagation to compute the gradient of its encoder 9elocally. Given the computed gradients, the client updates the encoder and decoder while the cloud updates the middle block’s LoRA. This fine-tuning minimizes the average loss: arg mine is the number of data points[x,y] in D and 9 = { 9e, 9d, 9m,s LoRA}, by updating 9, as follows: 9 <— 9 -yg s.t. g =0) / d0, where Q is the aggregated gradient and y is a learningrate. STUNING enables the model to operate on a data point to derive the gradient dL {et.yj > y> 9) / d9 rather than aggregating the statistic of multiple prompts, which damages the prompts’ embeddings. STUNING updates the model parameters using the aggregated gradients, recovering the impact of the e-LDP noise until the model converges. It results in effective models for generating code.

[0160] The power of STUNING is its effectiveness and efficiency in model performance and fine-tuning cost. First, it does not require extra information from the43MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) cloud, maintaining the CPCO constraints. Second, it aligns the encoder and the decoder to mitigate the impact of the LDPVOCAB on model performance. Third, the cloud efficiently updates a lightweight LoRA at a negligible cost compared to the heavy middle blocks. Fourth, the client can easily fine-tune all the parameters in 0eand 6d to maximize the model performance without applying LoRA since few attention blocks are needed, e.g., one and four blocks for the encoder and the decoder correspondingly. As a result, the system achieves outstanding model performance for code generation while being cost-effective as a practical solution for many clients, such as but not limited to, small enterprises with limited resources to finetune and operate the encoder and decoder. The cloud has the option not to fine-tune its 9m’s LoRA to reduce the cloud’s computation complexity and cost with a marginal drop in terms of model utility as a trivial trade-off.

[0161] The system provides a complexity and cost analysis in operation. Let n be the number of tokens given an input, and d be the hidden size of attention layers.

[0162] Network Communication. In the system, the network communication between the client and the cloud consists of two phases. The first phase is that the client sends the output embedding 8 of the encoder to the cloud, which has the complexity of 0(nd). In the second phase, the cloud returns the enriched embedding 8 to the client, in which the cost is 0(nd). As a result, the total network transmission cost between the client and the cloud is 0(nd).

[0163] Client Computation and Tuning Cost. The computation process can be broken down into three stages on the client side. Firstly, the client obtains the privatized token embedding, which is O(n). In the second and the last stage, the client feeds embedding through the encoder and the decoder. These operations have the complexity of O(n-d) as mentioned in (e.g., Vaswani, A. et al., Attention Is All You Need, NeurlPS (2017)). Hence, the total computation cost of the client is 0(n2d). Regarding the fine-tuning cost, the client only needs to compute the decoder’s gradients twice and the encoder’s gradients once in each training iteration compared with the whole LLM’s gradients. Therefore, the tuning cost is significantly reduced compared with fine-tuning the entire of LLM model. For instance, in the experiment discussed herein, the client uses 1 and 4 attention blocks from CodeLlama-7B for the encoder and decoder, respectively, resulting in 71.9% less fine- tuning cost, i.e., 9 attention blocks’ gradients vs. 32 attention blocks’ gradients.

[0164] Cloud Prompting and Hosting Cost. Every prompt embedding feed to the44MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) cloud-hosted LLM goes through a smaller number of attention blocks, i.e., 27 instead of 32 blocks of the CodeLlama-7B. Hence, the prompting and LLM -hosting cost is reduced, e.g., using 27 blocks of the CodeLlama-7B model as the cloud-hosted LLM registers a 15.6% lower cloud prompting and hosting cost. Therefore, the system is a scalable and deployable solution since the client only needs limited resources to fine-tune and operate the encoder and decoder. The complexity and cost analysis discussed above, and the experimental results show that a single GPU is sufficient for the client to fine-tune the encoder and decoder, achieving high model utility in code generation and completion tasks.

[0165] LTOKENIZER. Although the effectiveness of STUNING, the cloud can exploit the back-propagated gradients of the middle block 0m’s LoRA derived from the raw one- hot vectors [vt]tey of tokens t in the output code y to reconstruct the training data through BiSR (e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split- Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)). The cloud maps every one-hot vector vt to the client’s token t using its tokenizer shared with the client and the vocabulary V. LDPVOCAB does not protect the raw one-hot vectors { jtey. To prevent this risk, the system develops a local tokenizer (LTOKENIZER), which uniformly assigns every token and its embedding in the LDPVOCAB V to a random index in the tokenizer, at the client side.

[0166] FIG. 17 illustrates an example of assigning an arbitrary token t,e.g., ‘function,’ and its embedding etiin the LDPVOCAB to a random index y. The one-hot vector vt for this token become [l7] , that is, the ylhelement of vt is 1 while all remaining elements are 0. Now, if t is the next token in the output code y the client’s decoder generates, the gradient is derived to minimize the loss function between the one-hot vector vt = [l7] with the decoder’s output logits. This gradient misleads the cloud’s RAs to reconstruct the token at the index y of the cloud’ s tokenizer with its vocabulary V. Hence, instead of observing the client’s token, which is ‘function,’ the cloud observes an irrelevant and incorrect token, which is ‘jump.’ FIGS. 5A-5 show an example of a meaningless outcome reconstructed by the cloud. LTOKENIZER is data-independent with no extra privacy cost.

[0167] The probability for the cloud to infer the index of every client’s token in the cloud’s tokenizer is 1 / 1 F|, which is tiny; thus, protecting the client’s secret one-hot vectors vt and its tokenizer from being exploited by the cloud. The client uses the LTOKENIZER with its LDPVOCAB in STUNING and in the inference phase (e.g., algorithm of FIG. 4).45MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0168] Experimental Results

[0169] The extensive experiments shed light on the trade-off between privacy and model utility on the client side against RAs from the cloud. The system significantly outperforms the SoTA with strong privacy against the cloud. The system is cost-effective with a minimal fine-tuning cost on the client side, representing a step forward in protecting a client’ s prompts and code in code generation and completion.

[0170] Configurations. In some embodiments, the system can use CodeLlama-7B, CodeQwenl.5-7B-Chat, and Llama3-8B-instruct models, which have vocab sizes of 32k, 92,416, and 128,256 tokens, respectively. If it is not specified, the system can use CodeLlama-7B. The client uses one and four attention blocks for each model as the encoder and the decoder. The cloud-hosted model consists of the remaining blocks. Each token embedding has m=4, 096 features. A uniform elm was set across all features in et. The temperature h is 0.25 (the highest performance across mechanisms).

[0171] Datasets. The system was evaluated using the Evalplus benchmark (e.g., Liu, J. et al., Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation, Thirty-Seventh Conference on Neural Information Processing Systems, available at https: / / openreview.net / forum?id=lqvx610Cu7 (2023)), Mostly Basic Python Problems (MBPP) (e.g., Austin, J. et al., Program Synthesis With Large Language Models, arXiv:2108.07732 (2021)) for code generation, and HumanEval (e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)) for code completion, and the Big-CodeBench benchmark (e.g., Zhuo, T.Y. et al., Bigcodebench: Benchmarking Code Generation With Diverse Function Calls and Complex Instructions, arXiv:2406.15877 (2024)). The client fine-tunes the encoder and the decoder on the (public) CodeAlpaca dataset (e.g., Chaudhary, S. et al., Code Alpaca: An Instruction- Following Llama Model for Code Generation, available at https: / / github.com / sahil280114 / codealpaca (2023)) consisting of ~ 18k data points for Python and then evaluates the system on the (private) HumanEval dataset. The system finetunes the encoder and decoder again on the MBPP dataset as the client’s private data. In the ablation study, the CodeAlpaca dataset was enriched to ~376k data points by adding code from the Stack dataset (e.g., Kocetkov, D. et al., The Stack: 3 TB of Permissively Licensed Source Code, TMLR (2023)).46MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0172] Datasets. The MBPP consists of ~ Ik Python programming problems covering the programming fundamentals, standard library functionality, or the like. The MBPP consists of a training set, a validation set, and a test set, which have 374, 90, and 500 data points, respectively. The HumanEval is only a testing dataset comprising 164 data points. The BigCodeBench benchmark consists of 1,140 data points utilizing and expanding upon the EvalPlus benchmark to create a more advanced and versatile evaluation of LLMs on programming tasks.

[0173] Frequency Analysis Attacks. By using prompt embeddings to reconstruct the mapping of sequences and LDP-preserving sequence embeddings in the LDPVOCAB, the cloud creates a sequence frequency distribution for each sequence position on all the data points in publicly available datasets, i.e., 376k data points. Next, the cloud forms a position- based sequence embedding frequency distribution using prompt embeddings received from the client during the STUNING and inference phases. Then, it maps the observed sequence embeddings and the sequences based on the frequency order in the two distributions and recovers the client’s tokens and their LDP-preserving embeddings.

[0174] Baselines. SoTA dx-privacy -preserving mechanisms were considered for protecting user prompts, which are T2T (e.g., Qu, C. et al., Natural Language Understanding With Privacy-Preserving Bert, CIKM (2021); Feyisetan, O. et al., Privacy- and Utility-Preserving Textual Analysis Via Calibrated Multivariate Perturbations, WSDM (2020); Li, Y. et al., Privacy-Preserving Prompt Tuning for Large Language Model Services (2023)) and SnD (e.g., Mai, P. et al., Split-and-Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)). Defense-free mechanisms include the fClean model, i.e., the exemplary system without LDPVOCAB and LTOKENIZER, and the Clean model, i.e., the original LLM model. Pass@r (e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)) was used to evaluate the model performance: Pass@r = —rn — c\ / "n\~\(r) I (r) I, where !Eprompt:sis the expectation over all prompts; for a given prompt: r is the requested number of generated code versions, n is the total number of generated code versions, and c is the total number of correct code versions passing all unit tests. As in (e.g., Chen, M. et al., Evaluating Large Language Models Trained On Code, arXiv:2107.03374 (2021)), we set n = 2r.47MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0175] Questions. The following questions were considered.

[0176] QI: What is the trade-off between privacy and model performance? FIGS. 18A-18B show that SnD and T2T do not generate code in both code generation and completion tasks across values of the LDP budget e. SnD and T2T start to generate codelike outcomes at significantly large e > 5, 000; however, their Pass@ l values are still 0. This result is consistent with an observation in the literature (e.g., Mai, P. et al., Split-and- Denoise: Protect Large Language Model Inference With Local Differential Privacy, ICML (2024)). Meanwhile, the exemplary system outperforms these SoTA baselines across all LDP budgets e. The larger the LDP budget e is, indicating a looser LDP protection, the better the Pass@ l is. When e = 13, the system achieves the Pass@ l of 41.71 and 45.76, registering marginal drops of 0.26% on MBPP (code generation) and 1.14% on HumanEval (code completion) compared with the upper-bounded performance of defense-free mechanisms. This good performance is attributed to three key components: the embeddingbased design of the system, the ability to maintain tokens in prompts with marginal and adaptive noise in LDPVOCAB, and the effectiveness of the STUNING.

[0177] Q2: What is the cost to significantly improve model performance with strongLDP protection? In the stronger protection (smaller e), the model performance in Pass@l drops more notably (FIGS. 18A-18B). For instance, when 6 = 1, a very strong LDP protection, the Pass@ l for the system in MBPP and HumanEval are 32.22 and 31.1. Fortunately, the client can compensate for this performance drop by increasing the number of generated code versions, r, for each prompt. The client then chooses the best-generated version based on the number of test cases it passes. In fact, even when e = 1, FIGS. 19A- 19B show that the system has significantly higher Pass@ 10 (r = 10), which are 50.01 and 49.24 (55% and 58.3% improvements) in MBPP and HumanEval, respectively compared with Pass@ l. The improvement is more significant when e increases. This improvement makes the Pass© 10 of the system surpass the Pass@ l of defense-free mechanisms with the cost of generating multiple code versions.

[0178] Q3: What is the cost for the performance improvement? The cost is marginal.The client generates multiple code versions of a prompt as the input. Thus, the local computational cost is (r - 1 ) multiplied by the number of test cases for each prompt to identify the best code to use, which is negligible given the lightweight decoder. The communication and financial costs are (r - 1 ) times for this extra code as a trade-off for48MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) the significant performance boost.

[0179] Q4: How effective is the exemplary system in defending against RAs? FIGS.20A-20E and FIGS. 21A-21C show the privacy ASRs of PromtpRA and CodeRA under the exemplary system defense, respectively. Given the MBPP’s training set with no test cases in prompts, the system significantly lowers the privacy ASR of PromptRA (FIGS. 20A-20B) from 96% (no test case) and 51% (with test cases) of the defense-free mechanisms to 7% on average given the LDP budget 6 = 1. This result registers an effectiveness improvement of 92.7% and 86.27% of the system in defending against PromptRA. Similar results in the test set of the MBPP strengthen our observation (FIGS. 20C-20D). Regarding the code completion in the HumanEval, the privacy ASR of PromptRA is 0.0, indicating that PromptRA does not reconstruct a clear gist of any prompts (FIG. 20E). This result is intuitive since Vec2Text (e.g., Morris, J. et al., Text Embeddings Reveal (Almost) As Much As Text, EMNLP, pages 12448-12460 (2023)) and BiSR (e.g., Chen, G. et al., Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack, CCS (2024)) have not been designed for prompts that are pure code with LDPVOCAB and LTOKENIZER, i.e., the HumanEval.

[0180] Although PromptRA is ineffective in the code completion task, CodeRA shows severe ASRs in privacy, proprietary, and functionality of 99%, 97%, and 46%, respectively, against the defense-free mechanism in the HumanEval dataset (FIG. 21C). Thanks to the system LDP protection and LTOKENIZER, the system significantly reduces the privacy and proprietary ASRs to 0% across LDP budgets 6, and the cloud cannot reconstruct any functioning code, i.e., Fusi ASR is 0.0%. Similar results were observed with code generation in the MBPP training and test sets (FIGS. 21A-21B). Like the HumanEval dataset, the cloud cannot reconstruct any functioning code, i.e., Fusi ASR is 0.0. TheCRTx andCRTyunder the system are 0.028 and 0.108 respectively, significantly smaller than 0.28 and 1.00 of the defense-free mechanism on the MBPP dataset. Also, the sensitive information leak under the system is 0.038, which is tiny compared with 0.98 of the defense-free mechanism. The lists of top-5 tokens in the prompts and code that have been reconstructed by the cloud are { ‘ s)’, " , ‘a’, ‘ • ’ , ‘to’ } and { ‘per’, ‘i’, ‘f’, ‘res’, ‘p’ }, respectively, which are meaningless. The client maintains high model performance under this strong LDP protection with marginal cost, as shown in Q3 (above).49MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)

[0181] Q5: Is PGCODE robust against frequency analysis attacks? In this experiment, the cloud conducts token-based (FIGS. 22A-22C) and sequence-based (FIGS. 23A-23C) frequency analysis attacks, inspired by (e.g., Naveed, M. et al., Wright, Inference Attacks on Property-Preserving Encrypted Databases, CCS (2015)) and (e.g., Biryukov, A. et al., Codebook Attack, pages 216-216, Springer US, Boston, MA, ISBN 978-1-4419- 5906-5, doi: 10.1007 / 978-l-4419-5906-5_563, available at https: / / doi.org / 10.1007 / 978-l- 4419-5906-5_563 (2011)), respectively. A token-based attack is a sequence-based attack where the sequence length k = 1. Details are discussed in the Datasets and Frequency Analysis Attacks sections (above). Among over 32k tokens in the vocabulary, the cloud only recovers a limited number of tokens in the instruction template, which is used in every prompt, without being able to recover any tokens and their embeddings in the client’s prompts x and code y. In particular, token-based attack reconstructed 6 tokens { ‘Write’, ‘a’, ‘Python’, ‘function’, ‘to’, ‘(|im_end|)’ [, whereas, sequence-based attack reconstructed 3 meaningful tokens { ‘Write’, ‘a’, ‘Python’ }. The privacy ASR in reconstructed prompts based on the recovered tokens is 0.0 after ignoring these tokens. Similar results were observed when the cloud uses the publicly available data, e.g., the MBPP training set, sharing a similar distribution with the client’ s local data, e.g., the MBPP test set. The results show that the exemplary system is effective against the frequency analysis attacks.

[0182] Q6: What is the model performance for different LLMs and vocabulary sizes?The LLM was varied by using CodeQwenl .5-7BChat and Llama3-8B -instruct models, in addition to the CodeLlama-7B. These models have different capabilities and vocabulary sizes. Using CodeQwenl.5-7B-Chat in PGCODE achieves the best model performance compared with Llama3-8B -instruct and CodeLlama- 7B models in both MBPP and HumanEval datasets (FIGS. 24A-24B). A gap was observed between the exemplary system with CodeQwenl.5-7B-Chat and the clean CodeQwenl.5-7B-Chat model. Two reasons explain this gap: insufficient training data and small privacy budgets. To address this gap, the training data and the privacy budget were increased, and the cost of doing so regarding RAs was analyzed.

[0183] Q7: What is the effect of enlarging the training data and privacy budget? When the client, e.g., a small enterprise, does not have sufficient training data, the client can supplement its data with relevant publicly available data to enhance model utility. The client does not disclose its augmented local data. To demonstrate this, the number of data points was gradually increased from ~ 18k in the CodeAlpaca dataset to ~68k, ~ 106k, and50MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)~376k by adding Python code from the Stack dataset (e.g., Kocetkov, D. et al., The Stack: 3 TB of Permissively Licensed Source Code, TMLR (2023)). FIG. 25A shows that the Pass@ l of the system significantly improves from 39.2 to 59.3 in the MBPP dataset, closing the gap from 42.3% to 12.7% compared with the Clean (CodeQwenl.5-7B-Chat) model when 6 = 13. After increasing the training data to 376k data points, increasing the privacy budget significantly improves the Pass@ 1 of the system further from 54.8 to 65.2 notably closing the gap with 67.9 of the defense-free mechanism Clean (a marginal drop of 3.9%) in the MBPP dataset (FIG. 25B). We observe a similar result given the HumanEval dataset with a Pass@ 1 ’s improvement from 46.3 to 65.9 when increasing e from 13 to 27. In addition, the system achieves 31.1 in Pass@ l compared with 31.7 of the clean model on the “Instruct” benchmark of the BigCodeBench given e - 27, registering a marginal 1.8% utility drop. It is worth noting that the system is still effective against both CodeRA and PromptRA since ASRs are 0.0.

[0184] 08: What is the cost of enlarging the training data and privacy budget regarding RAs? FIGS. 26A-26D show that increasing the privacy budget and enlarging the training data does not result in higher reconstruction ASRs (~ 0.0). It is because the combination of task instruction n of CodeQwenl.5-7B-Chat model and the prompt x is notably lengthier with a more significant number of tokens in the vocabulary than the CodeLlama-7B model. It lowers the upper-bounded reconstruction probability, making CodeQwenl.5-7B-Chat a more secure model under LDPVOCAB and LTOKENIZER protection. Therefore, increasing the privacy budget should be considered together with increasing the training data to address the balance between model performance and privacy.

[0185] Q9: What are the roles of LDPVOCAB and LTOKENIZER against RAs? The experiment sheds light on the defensive roles of these components in the system. CodeRA becomes very effective when the client disables the LTOKENIZER; that is, the client uses the cloud-shared tokenizer (FIGS. 27A-27D). Meanwhile, PromptRA takes its turn to be effective when the client disables the LDPVOCAB; that is, the client uses the cloud-shared vocabulary (FIGS. 28A-28D). Therefore, the roles of LDPVOCAB and LTOKENIZER are to effectively defend against PromtpRA and CodeRA correspondingly. It is crucial to combine LDPVOCAB and LTOKENIZER to generate high-quality code while defending against the attacks (FIGS. 26A-26D).

[0186] Q10: What is the effectiveness of STUNING, including LoRA and zeroth-order51MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)(ZO) optimization? It was observed that either training the decoder alone or training a shortcut model, which concatenates the encoder and the decoder, alone results in poor performance, i.e., Pass@ l is 0.0 (e = 13) with the CodeLlama-7B model. This poor result is intuitive since training the decoder alone cannot mitigate the LDP -preserving noise exaggerated through the huge latent space of the cloud-hosted model 6m. Meanwhile, training the shortcut model alone ignores significant features learned by 3m. Hence, integrating the encoder and the decoder with the cloud-hosted model 9min STUNING is essential to achieve good model utility. The cloud has the option not to fine-tune the 0m’s LoRA to reduce its computational complexity with a marginal drop ~2% of model utility as a trivial trade-off (FIGS. 29A-29B and FIG. 30). Applying LoRA on the encoder and the decoder does not improve the model performance since the benefit of LoRA is marginal, given the lightweight encoder and decoder. The SoTA ZO optimization (e.g., Zhang, Y. et al., Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark, ICML (2024)) was used to fine-tune the encoder by approximating its gradient through the enriched embeddings. It was found that ZO introduces 50 extra inferences for each data point to approximate the gradient in each training iteration without a notable performance improvement.

[0187] Qll: What is the client cost? An A100-80GB GPU was used to quantify the client’s cost of the system fine-tuning across training data sizes from 18k to 376k data points with the CodeQwenl.5-7B-Chat model. FIGS. 31A-31B show that the total GPU memory allocated, fine-tuning time, and total cost of equivalent hosting expenses on AWS grow sub-linearly as the dataset size increases. Hence, the fine-tuning is scalable on larger datasets without an explosion of the communication cost on the client side. When increasing the dataset size by 20.89 times, the GPU memory allocated only increases 1.36 times from 49.92GB to 67.70GB (FIG. 31 A), and the AWS hosting fee is affordable to most small enterprises, i.e., as low as $152 to as high as $2,394 (FIG. 31B). Therefore, the exemplary system is scalable with larger datasets at negligible costs.

[0188] QI 2: What are the best numbers of attention blocks? 6 = 13 was set and focus was on understanding the impact of the number of attention blocks in the encoder and the decoder on model performance. When the number of attention blocks of the encoder is fixed to 1, it was observed that increasing the number of the attention blocks of the decoder notably improves the system model performance (FIGS. 32A-32B). Increasing the number of attention blocks of the encoder does not improve model performance. This result is52MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) intuitive since the impact of privacy noise will propagate further to the latent space, thus degenerating the ability of the cloud-hosted model to enrich the prompt embeddings. Therefore, maintaining effectiveness in fine-tuning the decoder and the encoder becomes challenging.

[0189] Conclusion

[0190] The exemplary system provides a novel code generation framework for protecting clients’ prompts and code using LLMs. In the system, the clients send prompt embeddings derived from a local encoder to the cloud-hosted model and receive enriched embeddings in return. The clients generate code by feeding the enriched embeddings to a decoder locally. To prevent the cloud from reconstructing the client’s prompts and code, the system includes a novel concept of LDP-preserving vocabulary (LDPVOCAB or vocabulary unit) associated with a local randomized tokenizer (LTOKENIZER or tokenizer unit). The system can include a tuning unit (STUNING) to fine-tune the encoder and decoder on the client side.

[0191] Extensive theoretical and empirical results show that the exemplary system significantly outperforms the SoTA baselines while protecting data privacy, code proprietary, and code functionality against reconstruction and frequency analysis attacks from an honest-but-curious cloud. The system opens several directions for research and practice, as discussed below.

[0192] Embedding Alignment: The cloud must ensure that it will not respond to malicious prompt embeddings derived from harmful prompts, such as “generating malware code.” This alignment is widely open since identifying harmful embeddings is more complex than harmful prompts.

[0193] Proprietary LLMs The model architecture, vocabulary, and token embedding size are sensitive to the proprietary LLM-provider. One solution to protect this information is for the client to use open-source encoders and decoders. The cloud can align the embedding size of the encoder and decoder with its proprietary LLM using feature transformation techniques.

[0194] FIG. 33 is a block diagram of an exemplary system 200 for private generation of code in accordance with embodiments of the present disclosure. The system 200 generally includes a client side 202 and a cloud side 204. The client side 202 can include a processing device 206 and a user interface 208 (e.g., a graphical user interface) configured53MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) to receive input from a client. The client side 202 includes a vocabulary unit 210, a tokenizer unit 212 (which can be incorporated into the vocabulary unit 210), and a tuning unit 214. The tuning unit 214 can include an encoder 216 and a decoder 218.

[0195] A client inputs into the client side 202 one or more prompts 220, which can be electronically stored in a database 222 in communication with the client side 202. The database 222 (or the vocabulary unit 210) can include vocabulary 224 such that the vocabulary unit 210 is executed to generate token embeddings 226 (e.g., LDP-preserving token embeddings) from the prompts 220. The encoder 216 of the tuning unit 214 is executed to encode the token embeddings 226 and generates prompt embeddings 228. The client side 202 transmits the prompt embeddings 228 to the cloud side 204, and the prompt embeddings 228 can be stored as prompt embeddings 230 of a database 232 in communication with the cloud side 204. Separation of the data in the database 222 is thereby maintained from the cloud side 204.

[0196] The cloud side 204 can include a processing device 234 for executing a large language model (LLM) 236 and, in some embodiments, can include a user interface 238. The LLM 236 is executed to generate enriched embeddings 240 from the prompt embeddings 230. The enriched embeddings 240 are transmitted to the decoder 218 of the client side 202, which is executed to decode the enriched embeddings 240 to generate code 242 corresponding with the prompts 220. In some embodiments, the enriched embeddings 240 can be stored in the database 222 as enriched embeddings 248. However, the generated code 242 is only stored in the database 222 on the client side 202 such that the code 242 cannot be detected by the cloud side 204. The tuning unit 214 can allow for fine-tuning by the client using the interface 208 to adjust the encoder settings 244 and / or decoder settings 246.

[0197] FIG. 34 is a block diagram of an exemplary computing device 300 for implementing the exemplary system for private generation of code in accordance with the present disclosure. The computing device 300 includes one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments. The non-transitory computer- readable media may include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more flash drives), and the like. For example, memory 30654MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) included in the computing device 300 may store computer-readable and computerexecutable instructions or software for implementing exemplary embodiments of the present disclosure. The computing device 300 also includes configurable and / or programmable processor 302 and associated core 304, and optionally, one or more additional configurable and / or programmable processor(s) 302’ and associated core(s) 304’ (for example, in the case of computer systems having multiple processors / cores), for executing computer-readable and computer-executable instructions or software stored in the memory 306 and other programs for controlling system hardware. Processor 302 and processor(s) 302’ may each be a single core processor or multiple core (304 and 304’) processor.

[0198] Virtualization may be employed in the computing device 300 so that infrastructure and resources in the computing device 300 may be shared dynamically. A virtual machine 314 may be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines may also be used with one processor. Memory 306 may include a computer system memory or random access memory, such as DRAM, SRAM, EDO RAM, and the like. Memory 306 may include other types of memory as well, or combinations thereof.

[0199] A user may interact with the computing device 300 through a visual display device 318 (e.g., a personal computer, a mobile smart device, or the like), such as a computer monitor, which may display at least one user interface 320 (e.g., a graphical user interface) that may be provided in accordance with exemplary embodiments. The computing device 300 may include other I / O devices for receiving input from a user, for example, a camera, a keyboard, a microphone, or any suitable multi-point touch interface 308, and a pointing device 310 (e.g., a mouse). The interface 308 and the pointing device 310 may be coupled to the visual display device 318. The computing device 300 may include other suitable conventional I / O peripherals.

[0200] The computing device 300 may also include at least one storage device 324, such as a hard-drive, CD-ROM, eMMC (MultiMediaCard), SD (secure digital) card, flash drive, non-volatile storage media, or other computer readable media, for storing data and computer-readable instructions and / or software that implement exemplary embodiments of the systems described herein. Exemplary storage device 324 may also store at least one55MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) database 326 for storing any suitable information required to implement exemplary embodiments. The databases 326 may be updated by manually or automatically at any suitable time to add, delete, and / or update one or more items in the databases 326.

[0201] The computing device 300 can include a network interface 312 configured to interface via at least one network device 322 with one or more networks, for example, a Local Area Network (LAN), a Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, Tl, T3, 56kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, controller area network (CAN), or some combination of any or all of the above. The network interface 312 may include a built-in network adapter, a network interface card, a PCMCIA network card, Pa Cl / PCIe network adapter, an SD adapter, a Bluetooth adapter, a card bus network adapter, a wireless network adapter, a USB network adapter, a modem or any other device suitable for interfacing the computing device 300 to any type of network capable of communication and performing the operations described herein. Moreover, the computing device 300 may be any computer system, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer (e.g., the tablet computer), mobile computing or communication device (e.g., the smart phone communication device), an embedded computing platform, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.

[0202] The computing device 300 may run any operating system 316, such as any of the versions of the Microsoft® Windows® operating systems, the different releases of the Unix and Linux operating systems, any version of the MacOS® for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, or any other operating system capable of running on the computing device and performing the operations described herein. In exemplary embodiments, the operating system 316 may be run in native mode or emulated mode. In an exemplary embodiment, the operating system 316 may be run on one or more cloud machine instances.

[0203] FIG. 35 is a block diagram of an exemplary private generation of code system environment 400 in accordance with the present disclosure. The environment 400 can56MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) include servers 402, 404 configured to be in communication with at least one client side 406, at least one cloud side 408, at least one processing device 410, at least one user interface 412, and (optionally) a central computing system 414, via a communication platform 416, which can be any network over which information can be transmitted between devices communicatively coupled to the network. For example, the communication platform 416 can be the Internet, Intranet, virtual private network (VPN), wide area network (WAN), local area network (LAN), and the like. In some embodiments, the communication platform 416 can be part of a cloud environment.

[0204] The environment 400 can include repositories or databases 418, 420, which can be in communication with the at least one client side 406, the at least one cloud side 408, the at least one processing device 410, the at least one user interface 412, and the central computing system 414, via the communications platform 416.

[0205] In exemplary embodiments, the servers 402, 404, the at least one client side 406, the at least one cloud side 408, the at least one processing device 410, the at least one user interface 412, and the central computing system 414 can be implemented as computing devices (e.g., computing device 300). Those skilled in the art will recognize that the databases 418, 420 can be incorporated into at least one of the servers 402, 404.

[0206] While exemplary embodiments have been described herein, it is expressly noted that these embodiments should not be construed as limiting, but rather that additions and modifications to what is expressly described herein also are included within the scope of the invention. Moreover, it is to be understood that the features of the various embodiments described herein are not mutually exclusive and can exist in various combinations and permutations, even if such combinations or permutations are not made explicit herein, without departing from the spirit and scope of the invention.57MEl\56762934.vl

Claims

Attorney Docket No. 94350.00296 (NUT Ref. 25-002)CLAIMS:

1. A system for private generation of code, the system comprising: a client side including: a vocabulary unit configured to receive as input one or more prompts from a client, wherein the vocabulary unit is executed to output token embeddings corresponding to the one or more prompts; a tuning unit including an encoder and a decoder, wherein the encoder is executed to encode the token embeddings and generate as output prompt embeddings; and a cloud side including: a large language model (LLM) configured to receive as input the prompt embeddings from the tuning unit, wherein the large language model (LLM) is executed to generate as output enriched embeddings; wherein the decoder of the tuning unit receives as input the enriched embeddings and is executed to decode the enriched embeddings to generate code corresponding to the one or more prompts from the enriched embeddings.

2. The system of claim 1 , wherein the vocabulary unit includes a tokenizer unit.

3. The system of claim 2, wherein the tokenizer unit is a local randomized tokenizer.

4. The system of claim 2, wherein the tokenizer unit is configured to assign every token of the token embeddings to a random index in a vocabulary of the vocabulary unit.

5. The system of claim 1, wherein the vocabulary unit is configured to adaptively randomize the token embeddings based on vocabulary of the vocabulary unit.

6. The system of claim 1 , wherein the vocabulary unit is a local differential privacy (LDP) preserving vocabulary unit.

7. The system of claim 1, wherein the token embeddings are local differential privacy (LDP) preserving token embeddings.

8. The system of claim 1 , wherein the client side comprises a processing device configured to execute the vocabulary unit and the tuning unit.58MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002)9. The system of claim 1 , wherein the cloud side comprises a processing device configured to execute the large language model (LLM).

10. The system of claim 1 , wherein the decoder generates the code on the client side.

11. The system of claim 1 , wherein the vocabulary unit and the tuning unit prevent observation of the one or more prompts and the generated code by the cloud side such that the one or more prompts and the generated code are only visible on the client side.

12. The system of claim 1 , wherein the encoder and the decoder are executed to operate only on the client side.

13. The system of claim 1 , wherein the tuning unit is configured to receive fine-tuning input from the client to fine-tune at least one of the encoder or the decoder.

14. The system of claim 1 , wherein the fine-tuning input directly aligns the encoder and the decoder while bypassing a fine-tuning of the large language model (LLM) of the cloud side.

15. The system of claim 1 , wherein the prompt embeddings prevent a cloud on the cloud side from reconstructing the one or more prompts.

16. A method for private generation of code, the method comprising: receiving as input one or more prompts from a client at a vocabulary unit at a client side; executing the vocabulary unit to output token embeddings at the client side corresponding to the one or more prompts; executing an encoder of a tuning unit at the client side to encode the token embeddings and generate as output prompt embeddings; receiving as input the prompt embeddings at a large language model (LLM) of a cloud side; executing the large language model (LLM) at the cloud side to generate as output enriched embeddings; receiving as input at a decoder of the tuning unit at the client side the enriched embeddings; and59MEl\56762934.vlAttorney Docket No. 94350.00296 (NUT Ref. 25-002) executing the decoder of the tuning unit at the client side to generate code corresponding to the one or more prompts from the enriched embeddings.

17. The method of claim 16, wherein the vocabulary unit includes a tokenizer unit.

18. The method of claim 17, the tokenizer unit is configured to assign every token of the token embeddings to a random index in a vocabulary of the vocabulary unit.

19. The system of claim 16, wherein the vocabulary unit is configured to adaptively randomize the token embeddings based on vocabulary of the vocabulary unit.

20. A non-transitory computer-readable medium storing instructions for private generation of code that are executable by at least one processing device, wherein execution of the instructions by the at least one processing device causes the at least one processing device to: receive as input one or more prompts from a client at a vocabulary unit at a client side; execute the vocabulary unit to output token embeddings at the client side corresponding to the one or more prompts; execute an encoder of a tuning unit at the client side to encode the token embeddings and generate as output prompt embeddings; receive as input the prompt embeddings at a large language model (LLM) of a cloud side; execute the large language model (LLM) at the cloud side to generate as output enriched embeddings; receive as input at a decoder of the tuning unit at the client side the enriched embeddings; and execute the decoder of the tuning unit at the client side to generate code corresponding to the one or more prompts from the enriched embeddings.60MEl\56762934.vl