Detecting semantically invalid code generations

Abstract

System and method for detecting semantically invalid code, including receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.

Description

DETECTING SEMANTICALLY INVALID CODE GENERATIONSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Serial No. 63 / 663,430, filed June 24, 2024, the disclosure of which is hereby incorporated by reference in its entirety as if fully set forth herein.

[0002] The present application may be related to U.S. Provisional Serial No. 63 / 728,792, filed December 6, 2024, the disclosure of which is hereby incorporated by reference in its entirety as if fully set forth herein.FIELD

[0003] The present disclosure relates generally to devices, systems and methods for detecting semantically invalid code generations.BACKGROUND

[0004] Large Language Models (LLMs) are complex Natural Language Processing (NLP) models that, in certain general -purpose applications, have proved to have superior zero-shot performance. Utilizing these zero-shot models for generating computational process, such as SQL query, however, tend to have high rates of failures. Such failure can result in an invalid syntactic or semantic code. Invalid syntactic code is erroneous code that cannot be executed and may immediately generate an error (such as during compiling). Invalid semantic responses, which may be syntactically valid and thus executable, may not be semantically representative of the original query to the LLM. Such invalid semantic responses can thus be executed or relied upon by downstream processes and can have negative consequences, thereby representing a disadvantageous technological problem associated with using LLMs for generating computational processes.

[0005] Certain existing systems describe supervised quality measurements for evaluating model based generated responses. Roughly, these evaluation methods are of three types: content overlap, model based, and manual. Content overlap measures compute explicit similarity score between a model generated output and a fixed golden standard output. Common content overlap measures N-grams overlap in a fixed size, limited “window”, suboutputs, or overlap between semantic representation of output tokens. Model basedevaluations that utilize model-based token representations to compute semantic similarity between the generated and ground-truth reference outputs. Common types of model-based methods are embedding-average, word sentence movers, and using scoring models (e.g., based on BERT). One drawback of the content overlap and model-based approaches is that each require ground truth labeling. Finally, manual evaluation methods define how to score the generated outputs using manual human labor. One drawback of manual evaluation is that it requires human labor. Thus, all three approaches are not reliably scalable for large datasets.

[0006] Unsupervised evaluation methods include G-EVAL, an LLM led evaluation process in which LLM-Agent follows initial evaluation instructions and objective to compute a “star” scale measure such as coherence, measuring properties of LLMs such as toxicity by aggregating similarity scores of between outputs of original and perturbed input texts, implicit measure of response validity via the Perplexity of the next-token distribution of the generated text, wherein high perplexity indicates high chance for “hallucination,” and explicit measure of response validity by instructing the LLM to include such quantification in its response. One drawback of this method is that it does not measure or determine a semantic difference between an input query and an output computational code.BRIEF SUMMARY

[0007] The present application overcomes the disadvantages of the prior art by providing a system and method for detecting semantically invalid responses. Detecting semantically invalid responses represents an improvement in the function of a computer and, in particular, an improvement in the operation of LLMs, by improving the reliability of the LLM output for queries relating to generation of computational processes.

[0008] One aspect of the disclosure provides a method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.

[0009] In one example, the method further includes computing one or more first embedding vectors with respect to the first query; and computing one or more second embedding vectors with respect to the second query, wherein generating the reconstructionscore comprises generating a similarity score with respect to the one or more first embedding vectors and the one or more second embedding vectors.

[0010] In one example, the method further includes normalizing the similarity score; and calibrating the normalized similarity score.

[0011] In one example, the method further includes operating a classifier on the reconstruction score, the classifier defining a threshold for determining the semantic validity.

[0012] In one example, generating the reconstruction score is unsupervised.

[0013] In one example, the first query is a request to generate computation code.

[0014] In one example, the first query is textual input.

[0015] In one example, the computation code is Python or SQL.

[0016] In one example, the similarity score is computing a dot product between the one or more first embedding vectors and the one or more second embedding vectors.

[0017] Another aspect of the disclosure provides a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform a method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.

[0018] In one example, the method further comprises: computing one or more first embedding vectors with respect to the first query; and computing one or more second embedding vectors with respect to the second query, wherein generating the reconstruction score comprises generating a similarity score with respect to the one or more first embedding vectors and the one or more second embedding vectors.

[0019] In one example, the method further comprises: normalizing the similarity score; and calibrating the normalized similarity score.

[0020] In one example, the method further comprises: operating a classifier on the reconstruction score, the classifier defining a threshold for determining the semantic validity.

[0021] In one example, generating the reconstruction score is unsupervised.

[0022] In one example, the first query is a request to generate computation code.

[0023] In one example, the first query is textual input.

[0024] In one example, the computation code is Python or SQL.

[0025] In one example, the similarity score is computing a dot product between the one or more first embedding vectors and the one or more second embedding vectors.

[0026] Another aspect of the disclosure provides one or more non -transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform a method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] A more complete appreciation of the subject matter of the present disclosure and the various advantages thereof can be realized by reference to the following detailed description in which reference is made to the accompanying drawings in which:

[0028] Fig. 1 is a block diagram of a computing system according to one or more aspects of the disclosure;

[0029] Fig. 2 is a flow chart of a method of generating a reconstruction score;

[0030] Fig. 3 is a flow chart of a method of generating a detecting semantically invalid code;

[0031] Fig. 4 a flow chart of a method of generating one or more guided follow up clarifications according to one or more aspects of the disclosure; and

[0032] Fig. 5 graph depicting a sigmoid function representing calibrated score vs. probability.DETAILED DESCRIPTION

[0033] Fig. 1 is a block diagram showing a computing system 100, according to one or more aspects of the disclosure, that may implement one or more techniques described in the present application.

[0034] As shown, the system 100 can include at least one computing device 105. The at least one computing device 105 can be any type of computing device, such as a personalcomputer, laptop computer, mobile device, tablet computer, wearable device, AR / VR headset, etc. As shown, the computing device 105 can include one or more processor(s) 110 that communicates with one or more devices, modules, or components via a bus, such as one or more memory modules 120, and / or any other components typically present in general purpose computers. The memory 120 can be a tangible non-transitory computer-readable medium and can store information accessible by the processor(s) 110, such as program instructions 122 (e.g., computation code) that may be retrieved and executed by the processor and / or data 124 that may be retrieved, manipulated, or stored by the processor(s) 110 or any other component of the computing device 105. The processor 110 can be any type of processor, such as one or more CPU(s), GPU(s), TPU(s), and / or NPU(s). The memory 120 can be any type of memory, such as volatile or non-volatile types of memory. In particular, the memory can include one or more of the following: ROM, such as Mask ROM, PROM, EPROM, EEPROM; NVRAM, such as Flash memory; Early stage NVRAM, such as nvSRAM, FeRAM, MRAM, or PRAM, or any other type, such as, CBRAM, SONOS, RRAM, Racetrack memory, NRAM, Millipede memory, or FJG. Any of the methods, routines, procedures, steps, blocks, etc., discussed herein in the present disclosure can be implemented as a set of program instructions stored in the memory 120, such that, when executed by the processor 110, can cause the processor 110 to perform the corresponding method, routine, procedure, step, block, etc.

[0035] Although depicted as single elements in Fig. 1, the processor(s) 110 and memory 120 can respectively comprise one or more processors and / or one or more memory elements that can communicate by a wired / wireless connection. In this regard, the processor 110 can comprise a plurality of processors (e.g., one or more CPU(s), GPU(s), TPU(s), and / or NPU(s)) that can cooperate to execute program instructions. Moreover, the memory 120 can comprise a plurality of memories that can cooperate to store instructions and / or data.

[0036] The computing device 105 can also accept user input according to a number of methods, such as by a mouse, keyboard, trackpad, touchscreen interface, gesture, microphone, or the like (not shown). The computing device 105 can also be operably connected to a display (not shown).

[0037] The computing device 105 can also include one or more components to allow for wired or wireless communication via link 128 with any other computing device, such as server cloud 130 or computing device 140. The server cloud 130 can comprise one or moreserver computing devices, where such server computing devices can include similar components to those set forth with respect to device 105. Similarly, computing device 140 can include a processor 142, memory 144, instructions 146, and data 148, like the features set forth with respect to device 105 and can be connected directly or indirectly to computing device 105 via link 132 and server cloud 130.

[0038] Fig. 2 is a flow chart depicting a method 200 of generating a reconstruction score. The various steps described can be performed by one or more components of the system 100 described above. While the method 200 depicts various steps in a particular order, the steps can be performed in any order, steps may be added, and / or steps may be omitted.

[0039] At block 205, a first query is received. The first query may be provided by a user of a computing device, such as any of computing devices 105 and / or 140. The first query can be a natural language input of any type, for example a text query or an audio query. In the example of a text-based query, the query can be text entered directly by the user, or can be a file in the form of a PDF, worksheet, .csv file, Excel file, or any type of file. In the example of an audio query, the query can be in the form of a spoken query, a pre-recorded audio query, or an audio result of a text-to-speech input.

[0040] In some examples, the first query is a request or command made by the user, such as a request or command to generate a computation code capable of being executed by one or more processor(s). Take, for example, the first query of “Write a code to generate a random number between 1 and 10.” The first query can further indicate the desired programming language for the computation code, for example “Write a SQL code to generate a random number between 1 and 10.” While, in this example, the first query requests SQL code, the computation code can be any type of code executable by a processor, such as BASIC, Fortran, C, C++, Python, and / or SQL, or any other type of formal language. In some examples, the first query can be a request or command for a plurality of computation codes, such as a combination of one or more of BASIC, Fortran, C, C++, Python, and / or SQL, or any other type of formal language.

[0041] At block 210, a first computation code is generated from the first query. This can be performed according to any number of techniques, such as by a first model. The first model can be any type of generative text-to-code model capable of generating computation code from text. The first model can be a generally available general purpose large language model (LLM), such as open source or the commercially available GPT-4 Turbo or GPT-4developed by OpenAI. In another example, the LLM can be a special purpose model trained for the task of text-to-code generation.

[0042] Where the first query is a request or command for computation code, the first computation code may be syntactically valid or invalid. In the case of syntactically invalid computation code, such code may not be capable of being executed by a processor. In this regard, an attempt to compile or execute such code may result in an error. On the other hand, syntactically valid code may be capable of being executed by a processor.

[0043] The first computation code may also be semantically valid or invalid. In the case of semantically invalid code, the first computation may not be semantically representative of the first query. In the example, of the first query “Write a SQL code to generate a random number between 1 and 10,” the first computational code may generate a predetermined number or may generate a random number outside of the range.

[0044] At block 215, a second query is generated from the first computation code. This can be performed according to any number of techniques, such as by a second model. In some examples, the second model is the same model as the first model, while in other examples the first model and the second model are distinct. The second model can be any type of generative code-to-text model capable of generating text from computation code. The second model can be a generally available general purpose large language model (LLM), such as open source or the commercially available GPT-4 Turbo or GPT-4 developed by OpenAI. In another example, the LLM can be a special purpose model trained for the task of code-to-text generation. In one specific example, the first model can be a special purpose LLM for the purpose of text-to-code transformation, while the second model can be a separate special purpose LLM for the purpose of code-to-text transformation. The second query may be a request or command for computation code generated from the first computation code. Stated another way, the second query may represent a reconstruction of the first query that may be input into a LLM in order to arrive at the first computation code. For example, an input to the second model may be to reconstruct a query to arrive at the first computation code.

[0045] At block 220, one or more embedding vectors are generated from the first query and the second query. This can be performed according to any number of techniques, such as by a third model. In some examples, the third model is the same model as the second model and the first model, while in other examples the third, second, and first models aredistinct. In still other examples, a subset of the first, second, and third models may be the same while another subset of the models may be distinct. The third model can be any type of generative model capable of generating embedding vectors from textual input. The third model can be a generally available general purpose large language model (LLM), such as open source or the commercially available embedding models developed by OpenAI. In another example, the LLM can be a special purpose model trained for the task of generating embedding vectors.

[0046] In one example, one or more first embedding vectors are generated from the first query and one or more second embedding vectors are generated from the second query. The one or more embedding vectors can be numerical representations of the semantic of the first query and the second query. In this regard, the one or more embedding vectors capture the meaning of the text of the first and second queries.

[0047] The one or more first and second embedding vectors can be generated by any type of model, such as a LLM. The one or more first and second embedding vectors can comprise a set or list of real numbers, with each real number in the set or list being associated with a vector direction. In this regard, the one or more first embedding vectors can be represented as x and the one or more second embedding vectors can be represented as x’, with x and x’ being embedded into a high-dimensional geometric space Rd, as shown below.

[0048] x, x' G Rd

[0049] At block 225, a similarity score is generated from the one or more embedding vectors. Assuming that a proximity in the high-dimensional vector space Rdrepresents a semantic similarity between embedding vectors x and x ’, a distance between the embedding vectors x and x’ in the high-dimensional vector space can be computed. In this regard, a multi-dimensional distance is computed between vectors x and x’ . Any number of similarity metrics may be used to compute the similarity score. In one example, a cosine similarity is implemented, with the similarity score being a normalized dot product between the one or more first embedding vectors and the one or more second embedding vectors, as shown below. x ■ x' sim x,x') =IMIIlyll

[0050] In one example, the similarity score can be numeric and defined relative to a predetermined range. In one example, where the similarity metric generates normalized numeric values, the similarity score is a normalized similarity score in the range of -1 to 1.

[0051] At block 230, the similarity score is calibrated to compute the reconstruction score. The calibration of the normalized similarity score can be performed according to any number of calibration techniques. For example, calibration of the similarity score can be performed by dividing a group of similarity scores into a predetermined number of quantiles with respect to some reference group and the similarity scores can be assigned calibrated values based upon their associated quantile. In this example, each calibrated similarity score can be between - 1 and 1.

[0052] In another example, the similarity scores are calibrated using a reference set (or subset) of text queries. In this regard, a given input text query is compared with respect to the reference set (or subset) of text queries to generate reference scores. One or more statistical operations can be performed on the reference scores, such as mean, median, standard deviation, or standard deviation of the mean. To calibrate a similarity score of interest, the similarity score is then normalized based upon the mean and the standard deviation calculated from the reference scores. Based upon the reference set (or subset) of text queries, normalized and calibrated similarity scores can be computed.

[0053] In another example, adaptive score normalization can be performed on the normalized similarity score. A method for adaptive score normalization is described in “Analysis of Score Normalization in Multilingual Speaker Recognition” by Matejka et al., the entirety of which is hereby incorporated by reference. Adaptive score normalization uses a reference set (or subset) of text queries and reconstruction scores associated with each of the queries. In this regard, a given input text query is compared with respect to the reference set (or subset) of text queries to generate reference scores. To calibrate a similarity score of interest, the similarity score is then normalized based upon the the reference scores. This is done by assuming that the reference scores follow some distributional assumption. In one example, it is assumed that the reference scores follow a normal distribution, based upon the reference set (or subset) of text queries and the assumption that the reference scores follow a normal distribution, normalized and calibrated similarity scores can be computed.

[0054] In one example, the normalized and calibrated similarity scores can be in the range of -1 to 1, with -1 representing a complete dissimilarity between the first query and thefirst computational code and 1 representing similarity or identicalness between the first query and the first computational code.

[0055] In another example, the similarity scores are calibrated using one or more pivotal scores in the range of (0,1). In this example, a sigmoid function is selected as an underlying calibration function. An exemplary sigmoid function can be represented as:

[0056] As depicted in Fig. 5, which is a graph depicting calibrated score vs. probability, the C2 parameter corresponds to a probability of 0.5. In this regard, a controlled pivotal score can be generated with a user-selected business-oriented quantification, for example, by changing the prompt’s main objective. Assume, for example, such change matches a (non-calibrated) score value 87, hence, C2= 87. Another non-calibrated score and its matching controlled probability is identified, with both the non-calibrated score and its probability value being controlled by the prompt structure and / or selected business considerations. With the further non-calibrated score and matching controlled probability, ci can be calculated and the calibration function can be identified. For example, assume the noncalibrated score 92 should express probability 0.75. The calibration function can be represented as, 0.75 0.75 +1 -ci x (x0 75- c2) = log (-) -log (I) Cl = C2 -logC1= — - 0.219722192 -

[0057] The choice of non-calibrated score that fits 0.75 is user-selected and can be business oriented. In other examples, other non-calibrated scores can be selected. Forexample, a validity probability of SQL queries with invalid time range can be selected to correspond be 0.75.

[0058] At block 235, the calibrated and normalized similarity score can be output as the reconstruction score between the first query and the first computational code. In this regard, the reconstruction score represents a semantic validity (e.g., a semantic similarity) of the first computational code with respect to the first query.

[0059] As described above, a disadvantage of using LLM to generate computational code is the risk of semantically invalid code. One cause of semantically invalid code and / or syntactically invalid code is LLM hallucinations. LLM hallucinations are the generation of responses that are irrelevant, made-up, or otherwise inconsistent with the input.

[0060] In one example, the reconstruction scores can be ranked, with the highest scores indicating highest semantic confidence and the lowest indicating lowest semantic confidence. A subset of first queries and corresponding first computation code can be extracted based upon reconstruction scores, for example, the highest scores can be extracted, or the lowest scores can be extracted. The first model can be debugged based upon the extracted subset of first queries and corresponding first computation code to refine further reconstruction score generation. In another example, computation code can be filtered with respect to its corresponding reconstruction score to identify relevant query inputs resulting in certain reconstruction scores. In yet another example, the first queries and corresponding first computation code can be clustered with respect to reconstruction score. The clustered queries and codes can then be examined (automated or manually) to refine the first model.

[0061] Fig. 3 is a flow chart of a method 300 of detecting semantically invalid code generations. The various steps described can be performed by one or more components of the system 100 described above. While the method 300 depicts various steps in a particular order, the steps can be performed in any order, steps may be added, and / or steps may be omitted.

[0062] At block 305, the normalized and calibrated reconstruction score can be received by a classifier.

[0063] At block 310, the classifier operates on the reconstruction score, with the reconstruction score representing a normalized and calibrated similarity score.

[0064] The classifier can be any type of classifier, such as supervised, semisupervised, or unsupervised. In the example of supervised, the supervised classifier can be trained using labeled datasets. The labeled training set can include, for example, a set of inputqueries, with each input query being labeled as “valid” or “invalid.” In the example of semisupervised, any combination of labeled or unlabeled training data can be provided. For example, the training data can be labeled only, unlabeled only, or both labeled and unlabeled.

[0065] The classifier can incorporate any type of classification algorithm, such as a decision tree, naive Bayes classifier, linear regression, optimization techniques, K-nearest neighbors, support vector machines, and / or artificial neural networks.

[0066] In one example, the classifier implements using one or more semi-supervised learning methods. In this regard, the classifier can operate as an adaptive semi-supervised detector.

[0067] As described above, the reconstruction score is both normalized and calibrated. In this regard, the classifier can be a binary classifier that determines a threshold for determining semantic validity of the first computational code with respect to the first query. The threshold may be set manually by a user. For example, the threshold may be 0, with reconstruction scores above this representing semantically valid computational code and reconstruction scores below this representing semantically invalid. In other examples, the classifier may incorporate one or more machine learning techniques to determine an optimized threshold at least in part based upon prior reconstruction scores. In this regard, a prior query, its corresponding generated code, and the resulting reconstruction score can be used to train the classifier. In this example, one or more learning techniques (for example, contrastive learning techniques) can be implanted to train the classifier based upon the prior query, its corresponding generated code, and the resulting reconstruction score in order to determine if a second query is a valid reconstruction of a first query.

[0068] At block 315, the classifier outputs a validity determination based upon the normalized and calibrated reconstruction score. Such validity determine can be “valid” for semantically valid first computation code or “invalid” for semantically invalid first computation code.

[0069] Fig. 4 is a flow chart of a method 400 of generating one or more guided follow up clarifications according to one or more aspects of the disclosure.

[0070] At block 405, the first query and / or the first computation code are deconstructed into one or more subcomponents. In the example of the first query, the first query can be deconstructed into one or more first sub-queries that encapsulate sub-semantics relative to the overall first query. The first sub-queries and / or sub-semantics can be extractedfrom the first query according to any number of techniques, such as natural language processing, name entity recognition, and / or keyword recognition for one or more sub- semantic concepts, such as dates, times, business metrics and / or data sources. The first computation code can be deconstructed into first sub-codes encapsulating sub-semantics corresponding to the sub-queries, with the first sub-codes and / or sub-semantics being extracted from the first computation code according to any number of techniques, such as natural language processing, name entity recognition, and / or keyword recognition for one or more sub-semantic concepts, such as dates, times, business metrics and / or data sources.

[0071] At block 410, a reconstruction score is generated for at least one of the first sub-queries and corresponding first sub-codes. The reconstruction score for each of the first sub-queries and corresponding first sub-codes can be generated according to the method 200 described above depicted in Fig. 2. In this regard, each sub-query and corresponding subcode will have its own reconstruction score. Optionally, the reconstruction score can be validated according to the method 300 described above and depicted in Fig. 3.

[0072] At block 415, for any of the first sub-queries and corresponding first subcodes having a reconstruction score below a predetermined threshold, one or more guided follow-up clarifications can be generated and posed to the user for a response.

[0073] For example, where the sub-query relates to date or time, such as the phrases “last quarter,” “last month,” etc., the guided follow-up clarifications may request further clarification regarding the date or time. In the example of “last quarter,” the guided follow-up clarification can include “Did you mean Jan. 1 to March 31?”

[0074] Guided follow-up clarifications can also be generated for business metrics (e.g., clarification regarding mean or average, absolute values, revenue, average revenue, sum revenue, etc.). In the example of “revenue,” the guided follow-up clarification can include “Did you mean average revenue or sum revenue?”

[0075] Guided follow-up clarifications can also be generated for data sources (internal data sources, database, Snowflake, Microsoft drive, cloud accessible by API, etc.). In the example where data is retrieved from a first source, the guided follow-up clarification can include “Did you want data from a different source?”

[0076] At block 420, the first computation code can be refined based upon the user responses to the guided follow-up clarifications.

[0077] Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

CLAIMS1. A method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.

2. The method of claim 1, further comprising: computing one or more first embedding vectors with respect to the first query; and computing one or more second embedding vectors with respect to the second query, wherein generating the reconstruction score comprises generating a similarity score with respect to the one or more first embedding vectors and the one or more second embedding vectors.

3. The method of claim 2, further comprising: normalizing the similarity score; and calibrating the normalized similarity score.

4. The method of claim 3, further comprising: operating a classifier on the reconstruction score, the classifier defining a threshold for determining the semantic validity.

5. The method of claim 1, wherein generating the reconstruction score is unsupervised.

6. The method of claim 1, wherein the first query is a request to generate computation code.

7. The method of claim 6, wherein the first query is textual input.

8. The method of claim 6, wherein the computation code is Python or SQL.

9. The method of claim 2, wherein the similarity score is computing a dot product between the one or more first embedding vectors and the one or more second embedding vectors.

10. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform a method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.1 1 . The system of claim 10, wherein the method further comprises: computing one or more first embedding vectors with respect to the first query; and computing one or more second embedding vectors with respect to the second query, wherein generating the reconstruction score comprises generating a similarity score with respect to the one or more first embedding vectors and the one or more second embedding vectors.

12. The system of claim 11, wherein the method further comprises: normalizing the similarity score; and calibrating the normalized similarity score.

13. The system of claim 12, wherein the method further comprises: operating a classifier on the reconstruction score, the classifier defining a threshold for determining the semantic validity.

14. The system of claim 10, wherein generating the reconstruction score is unsupervised.

15. The system of claim 10, wherein the first query is a request to generate computation code.

16. The system of claim 15, wherein the first query is textual input.

17. The system of claim 16, wherein the computation code is Python or SQL.

18. The system of claim 11, wherein the similarity score is computing a dot product between the one or more first embedding vectors and the one or more second embedding vectors.

19. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform a method for detecting semantically invalid code, the method comprising: receiving a first query; generating a first computational code from the first query; generating a second query from the first computational code; and generating a reconstruction score based at least in part on the first query and the second query, wherein the reconstruction score represents a semantic validity of the first computational code with respect to the first query.

Images