Code generation using generative machine learning models

Abstract

A computer-implemented method of generating a computer program, the method comprising: receiving a structured program description comprising a plurality of nodes, each node having a node type assigned from a set of defined node types, each defined node type associated with a defined node structure; generating the computer program based on the structured program description, including: identifying a first node assigned a first node type, based on the first node and the first node type, deterministically generating a first code portion for the computer program, identifying a second node assigned a second node type, extracting from the second node, based on the defined node structure of the second node type, a requirements descriptor, and generating based on the requirements descriptor, using generative machine learning processing, a second code portion for the computer program.

Description

455128PCT Filing Copy Code Generation Using Generative Machine Learning Models Technical Field

[0001] The present disclosure pertains to code generation using generative machine learning models, such as large language models (LLMs). Background

[0002] In recent years, the field of software development and artificial intelligence (AI) has witnessed significant advancements with the integration of generative machine learning (ML) techniques (also known as ‘generative AI’) for automating or assisting in the process of program code generation. These advancements have the potential to revolutionize software development by accelerating the creation of code, enhancing developer productivity, and facilitating the realization of complex functionalities. Particular attention has been given to ‘large’ generative ML models, such as large language models (LLMs), with typically a billion or more parameters. Currently, the largest such models have of the order of hundreds of billions of parameters.

[0003] LLMs, such as GPT (Generative Pre-trained Transformer) variants, trained on massive amounts of training data, have demonstrated remarkable capabilities in understanding and generating human-like natural language text based on free-text prompts. However, the applications of such models extend far beyond natural language. Viewed more generally, such models take a sequence of data units as input (e.g., tokens of a text string, such as characters, words etc.) and output a sequence of data units in response. Neither sequence is constrained to have a fixed length. Increasing attention is being given to the use of such models to generate structured outputs that are 'machine-readable' in the conventional programmatic (deterministic) sense. Examples of such structured outputs include data structures conforming to a predefined data schema and program code conforming to a predefined programming syntax.

[0004] Researchers and developers have recognized the potential of these models for streamlining software development processes. By training these models on extensive repositories of existing code, they can learn programming syntax, semantics, and patterns, enabling them to produce functional code snippets in response to natural language prompts or high-level descriptions of desired functionalities.455128PCT Filing Copy

[0005] This approach holds promise for both automating routine coding tasks and providing developers with intelligent suggestions to guide their coding decisions. Developer time spent on repetitive and mundane tasks may be reduced. Moreover, this approach enables rapid prototyping and experimentation, allowing developers to iterate and refine their ideas more efficiently. This approach can also assist developers in learning new programming languages and paradigms by providing contextual examples and explanations. Such models can also potentially enhance the accessibility of software development, enabling individuals with limited coding experience to participate more effectively in the creation of software applications.

[0006] Despite the promise of large language models for code generation, there are several challenges and concerns that need to be addressed. While these models can produce syntactically correct code, ensuring semantic correctness, adherence to best practices, and absence of logical errors remains a challenge. The generated code may lack optimization and might not adhere to specific project requirements. Moreover, when prompted with a given task, large language models suffer from a lack of broader project context, making it difficult for them to produce code that aligns perfectly with the intended functionality and design goals. As such, code generated by LLMs might require extensive post-generation editing and maintenance, potentially negating some of the time-saving benefits initially gained. Existing LLM-based approaches also suffer from issues of explainability. Understanding how an LLM arrives at a specific code suggestion can be challenging, making it harder for developers to assess validity and debug code or program flows.

[0007] Recent work has explored “full application” development, meaning the development of full and potentially complex applications from start to finish using LLM code synthesis. A recent approach to code generation with LLMs can be characterized as a ‘conversation between experts’. In this approach, an LLM (or different LLMs) are assigned different roles, such as requirements engineer, architect, developer, test engineer etc. The LLM(s) operating in these different roles interact in a process analogous to a team of human experts. The idea behind this approach is that, by breaking the end-to-end development process into stages, with collaboration between those stages, reliable and scalable full application development will be possible using LLM(s). Summary455128PCT Filing Copy

[0008] Whilst the ‘conversation between experts’ approach can improve the quality and reliability of code generation for applications up to a certain size / complexity, the current limitations of generative ML models remain a barrier to scalability beyond a certain point.

[0009] An alternative code generation mechanism is provided herein, which uses deterministic code generation supported by a generative ML model(s), such as an LLM(s). In contrast to the ‘conversation between experts’ approach, the present code generation mechanism as a whole can be characterized as an ‘augmented’ compiler (referred to as the Cogna compiler herein) that augments a deterministic code generation process with targeted use of generative ML. Somewhat analogous to a conventional compiler, the augmented compiler receives as input a structured program description and compiles part(s) of it into code using deterministic processing. This exploits the realization that many aspects of development are largely ‘application agnostic’ and can therefore be feasibly automated using procedural (or ‘rule-based’) processing that leverages the structure of the program description. However, in contrast to a conventional compiler, the augmented compiler’s code generation capabilities are greatly extended through use of generative model(s), such as LLM(s), to generate one or more code components (portions) for more bespoke application elements. Such elements are accommodated by allowing ‘requirements descriptors’ to be embedded in the structured program description. The use of generative model(s) is more targeted than the ‘conversation between experts’ approach, being confined to more bespoke elements that are less susceptible to automation using classical deterministic techniques. Hence, this approach retains the enormous flexibility benefits of generative ML model(s), such as LLM(s), but with greatly improved reliability and scalability.

[0010] There are also significant benefits in increased computational efficiency and reduced latency in using deterministic processing in favour of LLMs where appropriate, as LLMs are computationally expensive to run and incur a significant latency penalty.

[0011] According to a first aspect of the present disclosure, a computer-implemented method of generating a computer program comprises: receiving a structured program description comprising a plurality of nodes, each node having a node type assigned from a set of defined node types, each defined node type associated with a defined node structure; generating the computer program based on the structured program description, including: identifying a first node assigned a first node type, based on the first node and the first node type, deterministically generating a first code portion for the computer program, identifying a455128PCT Filing Copy second node assigned a second node type, extracting from the second node, based on the defined node structure of the second node type, a requirements descriptor, generating based on the requirements descriptor, using generative machine learning processing, a second code portion for the computer program.

[0012] The first node may for example have a node type without a requirements field, with the consequence that generative M is not used to generate the first code portion. Instead, it is generated deterministically. The second node may for example have a node type with a requirements field(s). The requirements descriptor is extracted from this, and an LLM or other generative ML model is used to generate the second code portion.

[0013] For the avoidance of doubt, unless otherwise indicated, the term ‘compiling’ is used herein in a broad sense to refer to a process of converting a first computer program representation into a second computer program representation, rather than in the narrow sense of translating source code into machine code. The second representation could, for example, be source code (e.g. Python, JavaScript etc.) or some intermediate representation that would then be subject to one or more additional compiler stages to produce machine code. Note also that the terms ‘application’ and ‘computer program’ are used interchangeably herein, unless otherwise indicated.

[0014] In embodiments, the second code portion may be generated by: generating a candidate second code portion, identifying an ambiguity in the requirements descriptor based on the candidate second code portion, causing an indication of the ambiguity to be outputted at a user interface, generating the second code component based on a user response to the indication of the ambiguity.

[0015] Multiple candidate second code portions may be generated based on the requirements descriptor using generative ML-based processing, wherein the second code portion may be generated by: identifying from the multiple candidate code portions an ambiguity in the requirements descriptor, causing an indication of the ambiguity to be outputted at a user interface, receiving via the user interface a disambiguation input, generating based on the disambiguation input a modified requirements description; and generating the second code portion based on the modified requirements description using generative ML-based processing.455128PCT Filing Copy

[0016] The ambiguity may be identified by executing each candidate component on a test input, and comparing resulting outputs for consistency.

[0017] The test input may be generated based on the requirements descriptor using LLM based processing.

[0018] The method may comprise additionally performing a static analysis on the second code portion using non-generative ML-based processing.

[0019] The method may comprise generating the structured program description based on inputs received at an interactive user interface (UI) editor, wherein at least some nodes of the structured program description correspond to visual UI elements defined in the interactive UI editor.

[0020] For example, the requirements descriptor may be received in an input field rendered in the interactive UI editor in association with a visual UI element corresponding to the second node.

[0021] The method may comprise checking validity of the structured program description based on the defined node types and their defined node structures.

[0022] The structured program description may be generated using generative ML-based processing.

[0023] The structured program representation may be encoded in a domain specific language (DSL), and the first code portion may be generated using deterministic code generation rules defined in relation to the DSL.

[0024] The generative ML-based processing may be LLM-based processing.

[0025] A second aspect herein is directed to a computer-implemented method of generating a computer program, the method comprising: receiving a first structured program representation comprising a plurality of nodes, each node having a node type assigned from a set of defined node types, each defined node type associated with a defined node structure; generating a second structured program representation based on the first structured program representation, including: identifying a first node assigned a first node type, based on the first node and the first node type, deterministically generating a first portion for the second structured program representation, identifying a second node assigned a second node type,455128PCT Filing Copy extracting from the second node, based on the defined node structure of the second node type, a requirements descriptor, generating based on the requirements descriptor, using generative machine learning processing, a second portion for the second structured program representation.

[0026] The method may comprise generating code for the computer program based on the second structured program representation.

[0027] The method may comprise running a static or runtime analysis on the second code component.

[0028] Another aspect provides a computer-implemented method of generating code for a computer program, the method comprising: generating using generative ML-based processing a structured program description encoded in a domain specific language (DSL); and programmatically generating the code using predefined rules that leverage the DSL structure and syntax.

[0029] The code may be generated programmatically using a combination of generative ML processing and programmatic processing.

[0030] The method may comprise checking validity of the structured program description and prompting a large language model to correct the structured program description if invalid.

[0031] Further aspects provide a computer system comprising one or more computers configured to implement the method of any above aspect or embodiments, and non-transitory or transitory media embodying computer readable instructions configured, when executed in a computer system to cause the computer system to implement the same.

[0032] Illustrative embodiments will now be described, by way of example only. Brief Description of Drawings

[0033] Some examples will now be described, by way of non-limiting and illustrative example only, with reference to the accompanying Figures (FIGs.) in which:

[0034] FIG. 1 shows an example of a UI tree for an example application.455128PCT Filing Copy

[0035] FIG. 2A shows an example visual representation of an app UI builder in a first implementation.

[0036] FIG. 2B. shows an example of a separate list of textual requirements inputted in the first implementation.

[0037] FIG. 3 shows an example visual representation of an app UI builder in a second implementation.

[0038] FIG. 4 shows an example visual representation of an app UI builder in a third implementation.

[0039] FIG. 5 shows an example of an extended code synthesis pipeline. Detailed Description Overview

[0040] In the described embodiments, an augmented compiler generates a computer program from a structured program description. The structured program description represents a desired computer program as a collection of nodes with defined types and structures. ‘Deterministic’ in this context refers to the use of procedural (or ‘rules-based’) techniques that leverage the defined types and structures of the structured program description. This approach recognizes that significant parts of the application development process are, to a large degree, ‘application agnostic’. It is therefore feasible to automate these aspects of the development process by describing them in a structured format and using deterministic techniques somewhat analogous to a classical compiler to generate code component(s) for implementing them. In the following examples, the structured program description is encoded in a domain specific language (DSL), and deterministic code generation rules are defined in relation to the DSL. Such rules may, for example, embody associations between particular syntax elements of the DSL and predefined code ‘snippets’ that can be flexibly combined.

[0041] The power of generative models is still leveraged but can be confined to more bespoke elements of the application. Such elements are captured within the structured program description as requirements descriptors. For example, a requirements descriptor may be a text string expressing a requirement in natural language, or in some other unstructured / semi-structured form. In the following example, the DSL defines at least one node type with a requirements field that contains a natural language requirements descriptor.455128PCT Filing Copy Generative model(s) are used to interpret such requirements descriptors and generate code components for implementing them. In practice, generative model(s) might only be needed to generate small portions of code (such as the body of a function whose operation is described in natural language) to insert into a ‘backbone’ of deterministically generated code.

[0042] By way of further illustration, in one embodiment, a graphical tool – referred to as a user interface (UI) builder – is provided that enables a user to design an interactive graphical user interface. The design includes predefined components (such as fields, buttons, menus), their layout, as well as hierarchical associations between components (e.g. associating a button with a field and a drop-down menu). The components of the GUI and their relationships are represented as nodes, with predefined types and structures, in a hierarchical program description tree. For example, the tree might include a ‘button’ node associated with both a ‘field’ node and a ‘menu’ node. This node structure implies the need for some function, which is called when the button is selected, and which receives as arguments a first value inputted into the field and a second value selected from the menu. At that level of abstraction, the graphical user interface can be automatically coded using deterministic techniques, in a manner that is agnostic to the actual operation of the function. The actual operation of the function can be described by a bespoke textual requirement(s) contained in or otherwise associated with the button node in the program description tree. An LLM is used to generate code from the textual requirement(s) for only the body of the function, whilst all of the other code (defining the variables, defining the function, defining its arguments etc.) can be generated deterministically from the structure of the tree.

[0043] Generative ML processing can be used at various stages (for example, to identify and help a user resolve ambiguity in requirements, in addition to code generation). Such processing is implemented using one or more LLMs in the following examples. A generative ML model has the form of an algorithm and a set of parameters, which have been learned through structured training on a (typically large) dataset. An LLM is typically assigned a role and one or more tasks to perform in that role though one or more prompts inputted to the LLM. Different LLMs may be used to perform different types of tasks, or the same LLM may be used in different roles.

[0044] A distinction is drawn between ‘deterministic’ code generation, where the structure of the generated code is ‘hard coded’ in predefined code generation rules (e.g. rules for combining predefined template code), and code that is generated using generative ML455128PCT Filing Copy processing. In the latter case, a generative ML model has learned how to generate code structures from a training set, and that knowledge is captured in its learned parameters.

[0045] A multi-stage code generation architecture is described, which can be applied to multiple domains. Three core domains are considered, each with their own types of content: 1. Problem Domain including entities such as business process and related existing user flows, ‘pain points’ in the existing processes, business constraints, problem description etc. 2. Solution Requirements Domain including elements such as user / functional flows, visual requirements, functional requirements, non-functional requirements etc. 3. Solution Implementation Domain including generated code, and (in some cases) additional intermediate representation(s) such as data flows and data layer descriptions, technical architecture etc.

[0046] Requirements in the Solution Requirements Domain are generated from a problem description in the Problem Domain. Elements of the Solution Implementation Domain, including the code, are generated from the requirements in the Solution Requirements Domain. Various intermediate representations (IRs) of the program (such as problem description, requirements, flows, technical architecture etc.) are generated across the different stages, to facilitate eventual code generation in the Solution Implementation Domain.

[0047] A code synthesis platform incorporating a staged architecture of this kind is described in our co-pending UK Patent Application Nos. GB2313394.5, GB2313396.0, GB2313397.8 and GB2313395.2 and International PCT Publication Nos. WO2025046110A1, WO2025046113, WO2025046116 and WO2025046107, each of which is incorporated herein by reference in its entirety. In the aforementioned references, translation between the different stages (and feedback from later stages to earlier stages, e.g. to refine requirements etc.) is primarily performed using LLMs, with some level of classical / deterministic processing performed on LLM-generated code (such as static analysis to resolve dependencies). In one example therein, requirements are expressed in semi-structured Markdown format.455128PCT Filing Copy

[0048] A similarly staged code synthesis pipeline is described in the following examples. However, this places a greater emphasis on deterministic translation between stages, with LLMs / generative ML in a more controlled and more targeted manner. To facilitate this approach, more structured representations are used in earlier and intermediate stages. In particular, requirements are represented in stage 1 in a more structured form. The code generation architecture is supported by a DSL, or multiple DSLs used at different stages to define and enforce these representation structures.

[0049] A user interface is provided in communication with the code synthesis pipeline (the ‘platform’), via which a user may interact with the code synthesis. The user interface provides a graphical UI builder, and additionally provides an interactive mechanism for the code synthesis pipeline to provide feedback to the user (e.g. to request that requirements are added or clarified), and receive additional user input (e.g. a disambiguation input clarifying an existing requirement). Interactions between the user and the code synthesis pipeline (such as the user inputting a problem description, inputting, selecting, confirming or editing requirements, the pipeline requesting additional clarification or information from the user etc.) are conducted via the user interface. The UI builder tool is also rendered via the user interface, enabling a user to design visual and functional elements of a graphical user interface for an application to be synthesised. The term platform user interface may be used to distinguish from an application user interface of a synthesised application (e.g. designed using the platform user interface).

[0050] The code synthesis pipeline can perform whole application synthesis in a modular manner, with the synthesis process broken into smaller pieces of work.

[0051] Pre-built components are used in requirements to break down synthesis into smaller pieces of work such as customising pre-built code to the current use case. To enable this, the DSL used to describe the requirements in a structured form includes: a. A notion of components in the backend, and their requirements b. A notion of pages and views in the frontend, and their requirements c. A notion of widgets (component parts in a UI) in the frontend, and their requirements455128PCT Filing Copy

[0052] A technical design step is used to break down requirements. A description of the components in the requirements DSL can be generated using a WYSIWYG (what you see is what you get) user interface builder.

[0053] As discussed, one popular metaphor for code synthesis (and other applications of generative AI) is one of a textual conversation between multiple agents playing different roles, such as product manager, software architect, developer and tester. In this metaphor, synthesis is a natural language conversation, including snippets of source code.

[0054] By contrast, in the described embodiments, code synthesis is viewed as being a compilation problem. In the following approach, an LLM-augmented compiler (or ‘LLM compiler’ for short) translates between a source language and a target language.

[0055] The source language is a DSL that represents requirements (R). The DSL can be extended to incorporate other elements in addition to requirements. In one implementation, the DSL functions as a description layer across the IRs, with DSL encompassing all IRs generated / consumed within the pipeline (of which ‘requirements’ is one part).

[0056] The target language is a solution (S) in combination with one or more tests (T).

[0057] In some implementations, the LLM compiler transforms the source to the target via an intermediate representation (IR) or multiple IRs that include that DSL and also intermediate structures such as a technical architecture. As indicated, the use of such IRs is not essential.

[0058] At each stage, well-formedness conditions are defined that can be checked. For a typed DSL, these checks include type checking. Checks can be extended to other aspects, such as checking of safety properties expressed using logical formulas.

[0059] A DSL means a structured format, that could for example be represented in JSON, to represent textual and non-textual requirements, like UI descriptions, and also to capture information from analyses of these requirements, and to record links with generated code and data models. The DSL can embed sections (e.g. whole paragraphs) of text that specify particular requirements, e.g., the text for a schedule button requirement. It also embeds a tree structure that specifies different pages in the UI. These may be interleaved, e.g., the text describing a schedule button might be stored at the appropriate place in the tree for the UI. A unique feature of the DSL is that it contains natural language parts to be interpreted using LLMs / generative ML processing.455128PCT Filing Copy

[0060] The DSL is a general language that allows requirements to be captured on one end of the pipeline and an executable on the other end (optionally forming a hierarchy of IRs expressed in the same language). Similarly, the synthesis would gradually lower the representation, either algorithmically or via LLM. In the above example, the first IRs in the DSL exist in the problem domain. From those, the pipeline generates requirements in the solution requirements domain, which in turn are used to synthesise elements (IRs and code) in the solution implementation domain.

[0061] A key purpose of the DSL (and IR(s) where applicable) is to embed structured information to be processed by efficient deterministic code rather than rediscovered by processing text using LLMs. In principle, the results of any static or dynamic analyses of the requirements domain or solution domain can be recorded within the DSL.

[0062] In more detail, a partially-ordered hierarchy of IRs might be: ● requirements IR (pain points, functional, non-functional) ● business logic IR (user stories, business model diagrams) ● user interface IR (pages, subpages, widgets) ● component IR (high level behaviour description identifying components) ● data models IR (what should the data contain with respect to requirements) ● data structures IR (precise schemas of data, how they are stored and accessed) ● high function IR (signatures, pre / post conditions) ● low function IR (implementation, invariants, side-effects) ● validation IR (tests) ● code IR ● AST IR ● verification IR

[0063] This approach implicitly provides traceability and observability. Diffing / updating requirements would only require some caching, and introducing a new IR would amount to455128PCT Filing Copy implementing a lowering process. Viewing synthesis as a new form of compilation, a series of IRs corresponds to a series of transforms in the compiler.

[0064] An unusual aspect of this compiler is that its input is ambiguous, and generative AI is used to make sensible assumptions when selecting between the many valid outputs. Techniques can be used to verify with some confidence that a delivered app meets what a user was thinking of in terms of requirements, but much less confidence that the natural language requirements are accurate or complete.

[0065] A source language with unconventional ambiguity of this nature leads to two unusual requirements, set out below.

[0066] Conventionally, when re-compiling an app, there is an implicit assumption that the new build behaves like the previous one, unless there are conflicting bug reports or new requirements. Said otherwise, it is assumed the user accepted the previous version unless they explicitly indicate otherwise.

[0067] For example, the requirement “next month” is ambiguous because it could mean either “next whole calendar month” or “next thirty calendar days”. If the pipeline interprets it to mean “next whole calendar month” in one version, and the user has not flagged this as incorrect, an implicit assumption is that “next month” still means “next whole calendar month” for the next version, even if “next thirty calendar days” remains compatible with the ambiguous requirement.

[0068] As in agent-based approaches, the code synthesis pipeline generates low-level tests to check correctness of its generated code.

[0069] Preferably, when the pipeline identifies an ambiguous requirement, and makes a selection to resolve that ambiguity when compiling code, an ambiguous test is lifted to the level of a new requirement to be accepted or rejected by the user. For example, assuming today’s date of 14 February, the pipeline might ask the user to confirm ‘does the “next month” mean from now until the end of March, or from now until 14 March?’.

[0070] The compiler is therefore unusual in several other respects, in that it is incremental (partly because of the latency of calls to LLMs, but also because of the need to preserve assumptions made about ambiguous requirements) and also interactive (in that it suggests new requirements to resolve ambiguities and get agreement from the user).455128PCT Filing Copy

[0071] Having a structured DSL and IR gives several advantages over a conversation between agents.

[0072] The structure of a DSL allows synthesis to be modular: for example, in an air travel management application with a pilot dashboard and booking page, a page for the pilot dashboard and a page for booking trainings can be synthesized separately. The decomposition of specifications via lowering to lower IRs will allow this modularity at various levels of IR, which will build up cumulative gains as more applications are built.

[0073] The structure of the DSL allows synthesis to be incremental: changes to the dashboard need the dashboard to be re-synthesised but not the booking page.

[0074] The structure of the DSL allows for easy measurement of how many requirements within the DSL are completely synthesised. There are seven bullet points in our example: we can measure how many have led to code generation.

[0075] Given the DSL, a partial synthesis can be performed to get rapid feedback from users: for instance, the pipeline can generate code to check whether or not a pilot is eligible to fly on a given date, as well as example positive and negative tests, so as to show to the user and get confirmation. This partial synthesis can be done rapidly before generating the UI or backend parts of the app.

[0076] The DSL can be quickly checked for static issues such as being synthesizable within an Operational Design Domain, in the style of type-checking or static analysis. Consider a minimal ODD that allows display of spreadsheet-style calculations augmented with read-only access to external resources such as web pages or corporate databases. A synthesizable check on the previously mentioned example would pass all the requirements, apart from the last one, which needs to make external bookings, which cause updates to databases and likely a financial commitment. A fix might be to insert an explicit requirement that the app seek human approval for any bookings made.

[0077] The DSL, represented in JSON, is a natural storage format for synthesis tasks to allow benchmarking and regression testing.

[0078] The structure of a DSL with multiple IRs allows for precise identification of inconsistencies as the inconsistency can be traced through the various IR levels and hence can be traced up to the originating requirement.455128PCT Filing Copy

[0079] In future, improvements to code-generating models mean generative AI methods will get better in terms of code generation, but agent conversations will still be loosely structured. The benefits of the compiler structure, as listed above, will remain even as generative AI continues to improve. Used directly, these models can reliably generate perhaps dozens of lines of code. Even as context windows expand, it seems unlikely that models could implement a multi-pass compiler by prompt engineering alone. DSL Example

[0080] In one implementation, requirements extracted from the UI builder are held in a tree structure in a backend database. They are deliberately in a flexible structure with four kinds of nodes. An application programming interface (API) exporting these trees for consumption by the augmented compiler uses a proto description for the trees exported by the database.

[0081] The DSL allows a whole synthesis problem to be captured in a single JSON document, or at least a single top-level document that links to any subsidiary documents.

[0082] It also allows a UI structure to be captured (UI tree diagrams in JSON), including some textual requirements nested within the JSON.

[0083] Although the initial definition trees are deliberately flexible to allow for expansion, schema notation may be imposed to check DSL validity.

[0084] For example, a recursive schema notation like TypeScript types can be used to describe JSON values that are valid DSL. An example is described below and depicted in FIG. 1, which shows a structured intermediate representation of an application to be synthesised. In this example, a set of TypeScript type declarations provides a formal definition for the DSL and the structured intermediate representation is a JSON object adhering to the TypeScript definition.

[0085] FIG. 1 shows an example of a UI tree for an example application (‘App1’) to illustrate this concept. The UI tree comprises various nodes at different hierarchical levels, representing different aspects of the application. Top-level nodes 101a-d represent example pages for each of the top-level navigation. Node 103 represents an example sub-page for secondary navigation. Node 105 represents an example component part instance for table of existing schedules. Node 107 represents an example simple component instance for the button to add a new schedule. Node 109 represents an example pop-up for a form to create a455128PCT Filing Copy new one. Nodes 111a-d represent example view states within the pop-up for the four steps of the form. Nodes 113a-d show example component part instances for each step with the view state.

[0086] An example JSON representation of the UI tree is shown below. / / final example from "Requirements and components" const ViewState: ViewState = { step_1: {name:"Schedule form step 1"}, step_2: {name:"Schedule form step 2"}, step_3: {name:"Schedule form step 3"}, step_4: {name:"Schedule form step 4"} } const AddNewScheduleModal: PopUp = { name: "Add new schedule modal", viewstate: ViewState } const ScheduleTable: CognaComp = { name: "Schedule table"} const AddButton: SimpleComp = { name: "Add new button"} const Schedules:SubPage = { name:"Schedules", reqs:[], parts: [ScheduleTable, AddButton, AddNewScheduleModal] } const Upload:Page = { name:"Upload a submission", subPages:[]} const Rates:Page = { name:"Rates", subPages:[]} const Contributor:Page = { name:"Contributor management", subPages:[]} const FileUploadSchedule:Page = { name:"File upload Scheduling", subPages:[Schedules]} const App1:App = { name: "App1", reqs:[], pages: [Upload, Rates, Contributor, FileUploadSchedule] } Code Example 1

[0087] Code example 1 shows an example JSON representation.455128PCT Filing Copy

[0088] Values of the TypeScript type App are valid JSON in the DSL. type App = { name: string, reqs: Requirement[], pages : Page[] } type Page = { name: string, subPages: SubPage[] } type SubPage = { name: string, reqs: Requirement[], parts: ReqUI[] } type Requirement = string / / natural language description of a requirement type ReqUI = SimpleComp | CognaComp | PopUp type SimpleComp = { name: string } type CognaComp = { name: string } type PopUp = { name: string, viewstate: ViewState} type ViewState = { [key: string]: (SimpleComp | CognaComp)} Code 2

[0089] Code Example 2 defines TypeScript types for App, Page, SubPage, Requirement, ReqUI, SimpleComp, CognaComp, PopUp and ViewState. Values of the TypeScript type App are valid JSON in the DSL (Code Example 1 being one such example). In Code Example 2, the TypeScript type declarations (App, Page, SubPage and so on) act as an intrinsic schema that is verifiable using a TypeScript compiler. The JSON tree in Code Example 1 is an instance that must satisfy those declarations: every key and value in the tree must fit the shapes described by the types.

[0090] Note, the types App and SubPage include textual Requirements within the tree structure that describes the UI.

[0091] The JSON tree presented in Code example 1 is one example of a structured program description: every object in that tree—such as the FileUploadSchedule page or the ScheduleTable component—is a node, and its permissible keys and value types are those imposed by the TypeScript declarations in Code example 2. The declarations therefore form the grammar of the DSL, while the JSON instance is a concrete program description written in that DSL that the augmented compiler can ingest without further transformation.

[0092] The ability to check a tree schema like the type App is a powerful tool. In addition, more complex static checkers for the JSON could also check that any embedded requirements strings only refer to entities that are in scope, e.g., tables that exist.455128PCT Filing Copy

[0093] Whilst the above examples focus on code generation, a similar approach can be used to translate between any two stages within the code synthesis pipeline. For example, a structured set of requirements may be converted to a structured technical architecture using deterministic processing primarily, with LLM processing uses to interpret and resolve ambiguities in natural language requirements where necessary.

[0094] To further illustrate the concepts, a specific example is shown below. This example considers a GUI structure defined using a graphical UI builder, with the addition of textual requirements, provided in some form of input field presented within the UI builder. This can be implemented in the UI builder in various ways. A toy “banana counting” application is considered purely by way of conceptual illustration.

[0095] In one embodiment, illustrated by FIG. 2A, a visual representation of the app’s UI is provided. FIG. 2B illustrates a separate list of textual requirements that follow when the “Submit” button of FIG. 2A is clicked. This list may start with “Banana Eat Update” and may specify what is to happen.

[0096] In an alternative approach, illustrated by FIG. 3, the requirements for the button are embedded visibly within the UI representation itself. When “Submit” is clicked one or more of the following may occur: the banana count in the inventory may be updated; and / or a screen to confirm that the entry has been recorded and banana count has been updated may be presented.

[0097] In a third variation, instead of visibly placing the requirements in the UI, they are added to a button component’s side pane, illustrated by FIG 4.. When “Submit” is clicked the following may occur: the banana count in the inventory may be updated; and / or a screen to confirm that the entry has been recorded and banana count updated may be presented.

[0098] Code Example 3 applies the concept of mixed deterministic / LLM-based synthesis, an example compilation step from the high-level DSL in FIG. 3 down to executable code. Input: page with an embedded “Submit” button with attached textual requirement Algorithm 1: 1. Deterministically compile page layout into code (code portion C4) to generate page and button, and code (code portion C5) to call server when button clicked. Both C4 and C5 are TypeScript code to run on the client. 2. Deterministically compile function signature for server action, but use the LLM to compile the requirement into the body of that function (code portion S). Code in S is Python code to run on the server.455128PCT Filing Copy Code example 3: Comprising Input and Algorithm 1

[0099] The advantage of deterministic compilation is that it directly exploits the structure of the input DSL (that there is a button and where it sits on the page) to generate code reliably. The LLM is relied upon only for the part (the action of the button) where there is a need to interpret natural language, as the LLM is much more flexible than a traditional deterministic compiler in this respect, but at the cost of sometimes needing to have the code repaired after generation, e.g., using the Reflexion technique. The DSL structure allows parts to be deterministically generated (e.g. the static frontend code, stubs for the pages in the backend) and the rest filled in with LLM.

[0100] Code portions C4 and C5 depicted below are generated deterministically. To aid readability, variables and functions have been given intuitive names. This is not essential and does not alter the code structure or its functionality (arbitrary function / variable names can be used). However, if desired for interpretability, various techniques can be used to assign interpretable names to variables, functions etc. For example, in a rules-based approach, names may be assigned by rules dependent on a project name, or names given to elements in the UI builder.

[0101] Note that deterministic code generation does not preclude the use of ML methods altogether. Even if the structure of the code is generated deterministically, ML techniques could be used to perform secondary functions such as assigning intuitive names to variables or functions within the code that do not alter its code structure or functionality.

[0102] The following example considers client code for a button in a user interface deterministically compiled from a DSL. import { inventoryBananaConsume }from ‘.. / store.ts’; const BananaCounterForm = ({bananaCounter }: {bananaCounter: BananaCounter}) => { const [updateBananaCounter, { isLoading, error}] = inventoryBananaConsume(); const onSave = useCallback(() => { updateBananaCounter({ …bananaCounter, }) }, [bananaCounter, updateBananaCounter])455128PCT Filing Copy return ( <Box sx={{padding: ‘1rem’}}> … <Button variant=”contained” disabled={isLoading} onClick={onSave} > … < / Box> ) } Code Example 4 (referred to as code portion C4 in Algorithm 1 above): Client code for button in user interface where client code may be deterministically compiled from DSL.

[0103] Client code for calling the server may also be deterministically compiled from DSL: const api = createApi({ reducerPath: ‘api’, baseQuery, tagTypes: [‘BananaCounter’], endpoints: (builder) => ({ updateBananaCounter: builder.mutation<BananaCounter, BananaCounter>({ query: (body) => ({ url: ‘ / inventory / banana / consume’, method: ‘POST’, body, }), invalidatesTags: [‘BananaCounter’], }), }) }) export const { inventoryBananaConsume } = api Code example 5 (referred to as code portion C5 in Algorithm 1 above): Client code for calling the server whereby the client code may be deterministically compiled from DSL.

[0104] Code component S shown below is generated using an LLM, in combination with deterministic processing:

[0105] Server code for behaviour of the button.455128PCT Filing Copy @router.post(“ / inventory / banana / consume”) async def record_banana_consumption( banana_id: int, consumed_quantity: int, session: Session = Depends(get_db) ) -> None: banana = session.get(Banana1, banana_id) if not banana or banana.current_stock < consumed_quantity: raise HTTPException( status_code=400, detail=”insufficient banana stock or not found” ) banana.current_stock -= consumed_quantity session.add(banana) session.commit()

[0106] Code example S (referred to as code portion S in Algorithm 1 above): Server code for behaviour of the button.

[0107] Code component S is an example FastAPI router method for the POST query. The function signature (including the ‘router.post’ decorator) may be deterministically compiled from the DSL. The function body may be synthesised by the LLM from the requirement “Update the banana count in the inventory”.

[0108] Code component S can be characterized as two code components: the function body generated using an LLM, and the remaining code generated deterministically.

[0109] An example prompt used to generate the server code S is given below: "Your task is to design the required architecture for the backend to support the page:" "Generate a stub fastapi.APIRouter named `router` for the endpoints.", "Focus on correct pydantic models, type hints, and docstrings, but do not implement the method bodies.", "Use type hints. Do not use a 'response_model' attribute.", "Take advantage of existing sqlmodels where appropriate, as they can be used as directly as pydantic request / response models, or nested in other models. Do not create classes extending the SQLModels", "Here's a general example of how you should structure the router. It is just an example:", '''455128PCT Filing Copy class Foo(BaseModel): foo: str @router.post(" / baz") async def baz(foo: Baz, session: Annotated[Session, Depends(get_db)]) -> Foo: """ Description of endpoint """ pass '''

[0110] In an extension of the above, the LLM-augmented compiler is supported by applying static checkers, including for disambiguation, at lower levels and passing that information up to the end user.

[0111] One principle of the LLM-augmented compiler is that at each level the pipeline can run available static checkers to check and improve code or IR quality, and use the LLM to repair any issues discovered (as in for example Reflexion: Language Agents with Verbal Reinforcement Learning”, Shinn et al., 20 Mar 2023, accessible atwhich is incorporated herein by reference in its entirety). These static checkers can include ‘classical’ type-checkers or linters. In addition, the pipeline can apply disambiguation tools, an unusual but important new class of static checkers. A prominent early example is ClarifyGPT, which can detect ambiguities or edge- cases when a function or other piece of code has been generated from natural language. The ClarifyGPT algorithm (Reflexion: Language Agents with Verbal Reinforcement Learning”, Shinn et al., 20 Mar 2023, accessible at https: / / arxiv.org / abs / 2310.10996, which is incorporated herein by reference in its entirety) determines ambiguities in textual specs for single Python functions, and interacts with the user to disambiguate (see Figure 1 of ClarifyGPT in particular).

[0112] An example follows of the LLM-augmented compiler incorporating the ClarifyGPT algorithm as a static analysis tool.

[0113] Suppose a high level DSL spec includes a page with a dashboard, which includes an option for the data to be sorted, expressed in text.455128PCT Filing Copy

[0114] The compiler compiles this spec to code to be used when constructing the dashboard, including a low-level function with the spec “Write a function to sort a list of elements”. The compiler uses this low-level spec to generate the body of a Python function.

[0115] Additionally, the compiler can run an algorithm to find ambiguities or edge-cases in the spec for this low-level function. For example, the ClarifyGPT algorithm generates a question for the user, “Should the sorting be in ascending or descending order”.

[0116] The compiler can lift this question to the high-level for the user, by inserting context, “In the context of the dashboard, should the sorting be in ascending or descending order.” and then it passes this high-level contextualized question to the end user. This is conveyed to the user as output via the user interface.

[0117] When the user answers with “descending order”, the compiler can pass it back down to the low-level algorithm to disambiguate the ambiguity. The user’s response is provided as input via the user interface.

[0118] In the present context, Clarify GPT can be used as follows. Given a requirement(s) to be synthesised using generative ML, multiple candidate code components are synthesised, along with one or more tests for testing performance of the candidate code component against the requirement(s). The test(s) are run on each candidate code component, and the candidate code components are clustered based on consistency of their respective test results. If all code candidate components yield mutually consistent test results, there is a single cluster, and the requirement(s) is determined to be unambiguous. If different code components yield inconsistent test results, generative ML is used to generate a clarifying question(s) (e.g. via one or more LLM prompts generated based on the test results). The resulting clarification question(s) is passed to the user, and their response is used to refine the requirement(s) to remove the identified ambiguity. Because, in the present context, the use of generative ML is highly targeted, disambiguation tools (such as Clarify GPT) can perform more effectively, meaning that ambiguous requirement(s) can be identified with greater accuracy and consistency.

[0119] In addition, other forms of runtime analysis may be performed (in combination with disambiguation analysis and / or static analysis). For example, other forms of test(s) may be generated and run on the generated code and / or predetermined tests may be run. The results455128PCT Filing Copy of either type of analysis can be feedback through the synthesis pipeline to refine the output (code and / or IR) at any stage.

[0120] Whilst in the above, a structured program description is generated via a UI builder, in other implementations, a structured program description is generated using a generative model. For example, an LLM can be prompted with a DSL definition (e.g. by providing a TypeScript definition, as in the code example 2 above, in the prompt to the LLM), guidance on the desired structured program description, and instructions to generate the structured program description in adherence to the DSL definition.

[0121] Validity of LLM-generated structured program descriptions can be checked using the same mechanisms as described above, with automated feedback loops to either repair such outputs programmatically (if unambiguous) or prompting the LLM to provide a corrected output. Extended code synthesis pipeline

[0122] Whilst the above description focuses on a UI-centric example, the described techniques can be extended to many intermediate aspects of software synthesis, in which intermediate representations or “artefacts” are generated in accordance with an appropriately defined DSL, and which are further examples of structured program descriptions. The following example considers the use of LLMs to generate or partly generate such artefacts.

[0123] FIG. 1 shows a high-level block diagram of an extended program synthesis platform 500, which is capable of synthesising a computer program in the form of a software application 512.

[0124] The platform 500 comprises two main components: requirements discovery, and program synthesis. FIG. 1 shows a requirements discovery pipeline 502 and a synthesis pipeline 504. A user inputs a program description in the form of an initial problem statement 108, which is used to generate a set of program requirements 510.

[0125] The software requirements 510 are inputted to the synthesis pipeline 504, which uses those requirements 510 to synthesise the software application 512, which in turn involves synthesizing program code for the application 512. At this stage, code is typically synthesised in the form of source code in a defined program syntax (such as Python, JavaScript etc.).455128PCT Filing Copy

[0126] The requirement’s discovery pipeline 502 and synthesis pipeline 504 are supported by a generator 107 embodying at least one LLM 107. Herein, an ‘ML-based generator’ refers to a processing component which can receive a prompt (typically in the form of a variable- length character string, referred to as an input string) and generate a response (also typically in the form of a variable-length character string, referred to as an output string) using one or more generative machine learning models, such as LLM(s). References to ‘generated’ artefacts herein refer to outputs generated by a generative ML model, unless the context demands otherwise. Such components are typically implemented in software, but it is also feasible to implement such components using specialized hardware (such as application- specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) etc.). An LLM-based generator embodies one or more LLMs, and applies at least one of the LLMs to the inputted prompt. An LLM, has the form of an algorithm and a set of parameters, which have been learned through structured training on a very large dataset. A single LLM may be sufficient. Alternatively, multiple LLMs may be used (for example, multiple LLMs may be prompted and their outputs may be compared, or different types of prompt may be provided to different LLMs). As another example, an LLM might include a modifiable input parameter (such as ‘temperature’ which is a value that controls the model’s ‘creativeness’), and a prompt may be processed multiple times with different values of the input parameters. When multiple responses are generated (e.g. with different LLMs and / or different input parameters), the generator may include logic for selecting one of the responses. Alternatively or additionally, with multiple LLMs, in some cases, a generator may include logic for selecting an appropriate LLM (or subset of LLMs) to process a given prompt. The following description may refer to a single LLM for conciseness. However, such description applies equally to other forms of generator. The generator 107 may be internal to the platform 500, or external, or it may comprise a combination of internal and external models. For the avoidance of doubt, an ML-based generator can, for example, be a collection of generative ML models, not all of which have to be used at every stage of requirements discovery / synthesis. For example, a first generative model of the ML-based generator might be used for requirements discovery, and a second generative model of the ML-based generator might be used for code synthesis. In this case, the ML-based generator may include appropriate model selection logic, implemented within the platform 500.455128PCT Filing Copy

[0127] In the requirements discovery phase, the user is guided to produce a sufficiently precise and comprehensive set of requirements. Along-side the requirements, a high-level program structure 501 is generated according to a predefined DSL.

[0128] Feedback techniques can be applied to structured outputs generated within the platform 500 as these can be readily parsed. Feedback regarding a generated artefact which has been analysed may be passed to the generator 507 with an instruction to modify the generated artefact. Feedback loops of this nature are implemented throughout the synthesis pipeline 106. Deployed in this manner, the effect is to greatly reduce the rate at which generative model(s) become “stuck”. Automated feedback also increases the overall speed of code generation.

[0129] An example requirements discover flow is considered.

[0130] Requirement generation: the generator 507 is prompted to generate an additional set of software requirements, in the form of an intermediate requirement artefact. A check could be performed at this point within the platform 500, e.g. parsing the set of requirements to verify that it has a correct requirements structure. If an error is identified, this could be addressed by modifying the structure of the requirements directly, or feeding back to the generator 507.

[0131] Generating program structure: a DSL for a high-level program structure is specified. An LLM prompt to generate the high-level program structure 501 using this DSL (enabling the program structure 501 to be parsed) is constructed. The prompt indicates the program structure DSL, and includes a natural language explanation of its elements. The prompt also instructs the generator 107 to generate its output in the defined DSL. The output is the program structure 501, which could be parsed at this point to verify that it conforms to the program structure DSL, prompting corrections and / or feedback to the generator 507 if required.

[0132] Function requirements mapping: For the subsequent stages, it is useful to map specific requirements to specific functions. A new chat is initiated with the generator 107, in which both the generated program structure 500 and the requirements 510 (in their current state) are given. The generator 507 is asked to produce a requirements mapping (which requirements are satisfied by which functions of the program structure 501), which can then be parsed programmatically (which in turn may trigger direct modifications to the455128PCT Filing Copy requirements mapping, or feedback to the generator 507). This is useful to subdivide the problem, and makes it easier to identify any requirements that are not satisfied, etc.

[0133] Generating additional function requirements: A is used to pass the requirements mapping of the previous section to the generator 507, and ask it to generate any additional function requirements. Note that the requirements have been linked to the program structure at this point, hence the generator performs this task with knowledge of the program structure. The structured output can then be parsed, in order to add the additional requirements to the requirements document 510 programmatically.

[0134] Once adequately defined, the requirements 510 are passed to the synthesis pipeline 504.

[0135] Applying the DSL-based approach described above, generative ML techniques are used in combination with ‘classical’ programmatic techniques, such as programmatic code / artefact generation based on predefined templates or structures etc. LLMs are used to generate intermediate artefacts (e.g. requirements 510, program structure 501 etc.) conforming to a DSL, from which code and / or other artefacts can be deterministically generated, or at least partially generated deterministically.

[0136] Hence, in some such implementations, a code artefact or program design artefact may be generated using a combination of generative ML processing (e.g. based on one or more text-based prompts) and programmatic processing (e.g. predetermined rules applied to defined structures, templates etc.). For example, artefacts may be constructed in accordance with a domain specific language (DSL) or collection of DSLs that precisely defines their structure and syntax. For example, programmatic processing may be used to generate, from a first structured artefact (e.g. the set of requirements 510), a second structured artefact (e.g. technical design, software skeleton, code artefact etc.) using predefined rules that leverage the DSL structure and syntax, with generative ML techniques used to synthesise one or more artefact portions (e.g. via one or more prompts to an ML-based generator) that are incorporated in the second artefact. Portions of design artefacts and / or code artefacts that are synthesised using generative ML can be refined or modified within the pipeline using the techniques described herein.

[0137] In some implementations, code synthesis is performed in multiple further design stages, starting from those requirements. First, a technical design is generated from the455128PCT Filing Copy requirements. The technical design is used to generate a software skeleton, including a file structure for the program. Finally, program code is synthesised for each program file within the file structure. Although each file is synthesised individually, the wider context of the requirements, the technical design and the software skeleton guide the code synthesis, providing wider context. These higher-level elements guide the eventual program synthesis, and are examples of program “design artefacts” as that term is used herein. Generated artefacts may include, for example, program design artefacts and synthesised program components / tests (referred to as ‘code / test artefacts’). A design artefact could also be a code artefact, e.g. containing template or boilerplate code to be refined / completed. The term program artefact is used to refer to code artefacts, higher-level descriptive artefacts, text artefacts and the like. Program design artefacts (e.g. software requirements, technical design, software skeleton etc.) can take various structured forms, such as a graph or tree (e.g. defining individual elements of a program design artefact and hierarchical relationships between them).

[0138] The architecture of the synthesis pipeline 506 is guided by the manner in which a human engineer would write software. At its core, the synthesis pipeline 506 produces the following artefacts consecutively:

[0139] 1. [Input] The complete requirements structure 510 and optionally the high-level program structure 501.

[0140] 2. From the inputs of 1, a technical software architecture (technical design) is generated. This includes all components, API specifications, data formats, etc.

[0141] 3. From the technical software architecture, a software skeleton is generated. The software skeleton comprises an initial set of program files. Within each file, all classes and methods are defined, as are their types.

[0142] 4. Finally, an implementation is generated for each component. As noted, a component may be a single program file, or comprise multiple program files. Either way, in the present example, an implementation is generated per-component, but with per-file prompts. In this manner, an implementation is generated for each component in the form of one or more code artefacts generated by the generator 107 (where a code artefact may, for example, take the form of a program file containing complete code). Each code artefact is statically analysed to detect any issues that can be resolved automatically.455128PCT Filing Copy

[0143] As before, a code artefact or program design artefact may be generated using a combination of generative ML processing and programmatic processing to deterministically generate a code artefact or other intermediate artefact(s) from a structured intermediate artefact (e.g., technical design or software skeleton) conforming to a defined DSL.

[0144] The above code synthesis pipeline is implemented at the hardware level in a computer or system of multiple networked computers. A computer system refers to a computer or set of networked computers, where the / each computer comprises one or more hardware processors. A hardware processor may take the form of a central processing unit (CPU), graphical processing unit (GPU) or other accelerator processor. Such processors are so- called ‘general purpose’ processors that may be programmed. Alternatively or in addition other forms of programmable hardware processor, such as field-programmable gate arrays (FPGAs), may be used. Non-programmable hardware processors, such as application- specific integrated circuits (ASICs) may be used. Computer-readable instructions may be stored in non-transitory media (such as magnetic or solid-state storage).

Claims

455128PCT Filing Copy Claims 1. A computer-implemented method of generating a computer program, the method comprising: receiving a structured program description comprising a plurality of nodes, each node having a node type assigned from a set of defined node types, each defined node type associated with a defined node structure; generating the computer program based on the structured program description, including: identifying a first node assigned a first node type, based on the first node and the first node type, deterministically generating a first code portion for the computer program, identifying a second node assigned a second node type, extracting from the second node, based on the defined node structure of the second node type, a requirements descriptor, and generating based on the requirements descriptor, using generative machine learning processing, a second code portion for the computer program.

2. The method of claim 1, wherein the second code portion is generated by: generating a candidate second code portion, identifying an ambiguity in the requirements descriptor based on the candidate second code portion, causing an indication of the ambiguity to be outputted at a user interface, generating the second code portion based on a user response to the indication of the ambiguity.

3. The method of claim 1, wherein multiple candidate second code portions are generated based on the requirements descriptor using generative ML-based processing, wherein the second code portion is generated by: identifying from the multiple second candidate code portions an ambiguity in the requirements descriptor, causing an indication of the ambiguity to be outputted at a user interface, receiving via the user interface a disambiguation input,455128PCT Filing Copy generating based on the disambiguation input a modified requirements description; and generating the second code portion based on the modified requirements description using generative ML-based processing.

4. The method of claim 3, wherein the ambiguity is identified by executing each candidate component on a test input, and comparing resulting outputs for consistency.

5. The method of claim 4 wherein the test input is generated based on the requirements descriptor using LLM based processing.

6. The method of any of claims 2 to 5, comprising additionally performing a static analysis on the second code portion using non-generative ML-based processing.

7. The method of any preceding claim, comprising: generating the structured program description based on inputs received at an interactive user interface (UI) editor, wherein at least some nodes of the structured program description correspond to visual UI elements defined in the interactive UI editor.

8. The method of claim 7, wherein the requirements descriptor is received in an input field rendered in the interactive UI editor in association with a visual UI element corresponding to the second node.

9. The method of any preceding claim, comprising checking validity of the structured program description based on the defined node types and their defined node structures.

10. The method of claim any preceding, comprising generating the structured program description using generative ML-based processing.

11. The method of any preceding claim, wherein the structured program representation is encoded in a domain specific language (DSL), and the first code portion is generated using deterministic code generation rules defined in relation to the DSL.455128PCT Filing Copy 12. The method of any preceding claim, wherein the generative ML-based processing is LLM-based processing.

13. A computer-implemented method of generating a computer program, the method comprising: receiving a first structured program representation comprising a plurality of nodes, each node having a node type assigned from a set of defined node types, each defined node type associated with a defined node structure; generating a second structured program representation based on the first structured program representation, including: identifying a first node assigned a first node type, based on the first node and the first node type, deterministically generating a first portion for the second structured program representation, identifying a second node assigned a second node type, extracting from the second node, based on the defined node structure of the second node type, a requirements descriptor, generating based on the requirements descriptor, using generative machine learning processing, a second portion for the second structured program representation.

14. The method of claim 13, comprising generating code for the computer program based on the second structured program representation.

15. The method of any of claims 1 to 13, comprising running a static or runtime analysis on the second code portion.

16. A computer-implemented method of generating code for a computer program, the method comprising: generating using generative ML-based processing a structured program description encoded in a domain specific language (DSL); and programmatically generating the code using predefined rules that leverage the DSL structure and syntax.

17. The method of claim 16, wherein the code is generated programmatically using a combination of generative ML processing and programmatic processing.455128PCT Filing Copy 18. The method of claim 16 or 17, comprising checking validity of the structured program description and prompting a large language model to correct the structured program description if invalid.

19. A computer system comprising one or more computers configured to implement the method of any preceding claim.

20. Non-transitory or transitory media embodying computer readable instructions configured, when executed in a computer system to cause the computer system to implement the method of any of claims 1 to 18.

Images