Generation of toy construction models

Abstract

A computer-implemented method of creating a digital representation of a toy construction model, the method comprising: receiving a user prompt to create a digital representation of a toy construction model, the user prompt including one or more desired attributes of the toy construction model; using a generative machine-learning model to create, based on the user prompt, a structured representation of a toy construction model, the toy construction model being constructed from a set of mutually interconnected toy construction elements, the structured representation being indicative of the toy construction elements of said set and of the mutual interconnections between respective ones of said toy construction elements; translating the created structured representation into a digital 2D or 3D representation of a visual appearance of the toy construction model and / or into a set of building instructions for creating the toy construction model.

Description

[0001] Generation of toy construction models

[0002] Technical field

[0003] The present invention relates to the generation of toy construction models.

[0004] There are various known types of modelling concepts of physical construction toy sets. Especially, concepts using modular or semi-modular concepts are very popular as they provide an interesting and challenging play experience. Typically, these concepts provide a set of pre-manufactured toy construction elements that can be interconnected with each other in some predetermined way according to modules of the pre-manufactured elements. The pre-manufactured elements may resemble well-known objects adapted to a specific modelling task. Thus in e.g. building a model of a house the elements may resemble wall bricks, roof tiles, doors, and windows.

[0005] For example, the toy construction sets available under the name LEGO® comprise a plurality of different types of interconnectable toy construction elements having coupling members in the form of protrusions and corresponding cavities. The coupling members are arranged according to regular grid patterns, thereby allowing a wide variety of interconnections between the toy construction elements.

[0006] Typically, such toy construction sets comprise a set of toy construction elements configured for creating one or more toy construction models, e.g. an animal, a robot, or another creature, a car, an airplane, a spaceship, a building, or the like. Typically, a construction set further includes printed building instructions or assembly instructions that illustrate how to construct a certain model from the toy construction elements of the set. Typically, the building instructions enclosed in a toy construction set comprise a sequence of pictures illustrating step by step how and in which order to add the toy construction elements to the model. Such building instructions have the advantage that they are easy to follow, even for children without great experience in toy construction sets and / or without reading skills.

[0007] However, such building instructions have the disadvantage that they are static and only provide instructions to build one or more predetermined models.

[0008] More recently, building instructions have been generated in electronic rather than in printed form. In particular, animated building instructions where the more complicated building steps are animated. However, it remains a problem to provide toy construction sets that inspire children to create their own models or to rebuild the model in a different way, thereby increasing the play value of the toy construction set.

[0009] WO 2012 / 160057 discloses an augmented-reality-based interactive building process where the user is presented with an image of the partial model built so far and with a choice to select one or more subsequent construction elements from a set of possible alternative subsequent construction elements. While the number of possible construction paths, and possibly even the number of resulting toy models, available to the user is increased by this prior art process, this prior art process requires the user to operate a camera and only allows selection from predetermined construction paths resulting in one or more predetermined models.

[0010] The scientific dissertation “Hierarchical Style Modeling: A generative framework for Style-Centric Generation of 3D Models”, 2019, by Mazeika, Stella, University of California Santa Cruz, describes a framework for encoding generative models, called a Hierarchical Decomposition. Based on the philosophical school of mereology, this framework emphasizes the partwhole relationships of the artifacts that are encoded. In this way, style is a series of both constraints placed on the Hierarchical Decomposition and transformations that occur to the decomposition. While providing a scientific framework for hierarchical representations of artifacts, it remains desirable to provide a user-friendly method and system that assists users, including children or other users that are unfamiliar with abstract concepts like hierarchical decomposition, in creating and, optionally, in iteratively editing buildable toy construction models.

[0011] Despite these previous attempts, it thus remains desirable to provide a method and corresponding system, in particular for creating toy construction models, that address one or more of the above problems and / or other problems of existing solutions, or that may serve as an alternative to existing solutions.

[0012] Various aspects disclosed herein provide methods, apparatus and systems that facilitate the creation and, optionally, the iterative editing of buildable toy construction models.

[0013] According to one aspect, disclosed herein are embodiments of a computer- implemented method of creating a digital representation of a toy construction model, the method comprising:

[0014] - receiving a user prompt to create a digital representation of a toy construction model, the user prompt including one or more desired attributes of the toy construction model,

[0015] - using a generative machine-learning model to create, based on the user prompt, a structured representation of a toy construction model, the toy construction model being constructed from a set of mutually interconnected toy construction elements, the structured representation being indicative of the toy construction elements making up said toy construction model and of the mutual interconnections between respective ones of said toy construction elements making up said toy construction model,

[0016] - translating the created structured representation into a digital 2D or 3D representation of a visual appearance of the toy construction model and / or into a set of building instructions for creating the toy construction model.

[0017] Accordingly, the generative machine-learning model operates on a structured representation of the toy construction model, which represents the toy construction elements from which the model is constructed, including their mutual interconnections. A generative machine-learning model is capable of assisting users, including untrained users, in the creation of buildable toy construction models based on user-provided prompts and without requiring the user to have knowledge about structural attributes of the desired toy construction model.

[0018] The structured representation preferably represents all toy construction elements from which the toy construction model is constructed, i.e. the toy construction elements that are at least partly visible when viewing the model from at least one viewing point and the toy construction elements that are not visible when viewing the constructed model from any viewing point, e.g. because they are complete surrounded by other toy construction elements of the model. This allows the machine-learning model to create a digital representation of a realistic model that is constructible in the physical world. The translation of the created structured representation into a representation of the visual appearance of the model and / or into a set of building instructions allows the resulting model to be presented in a user-friendly manner, in particular in a manner that allows a user to readily perceive and appreciate how the toy construction model will look when constructed in the physical world and, preferably, how the toy construction may be constructed from the set of toy construction elements. Examples of digital representations include an image, a series of images, a 3D digital rendering, etc.

[0019] According to another aspect, disclosed herein are embodiments of a computer-implemented method of creating a digital representation of a toy construction model, the method comprising, repeatedly:

[0020] - presenting a visual representation of an intermediate toy construction model,

[0021] - receiving a user input indicative of a user-selected modification to said intermediate toy construction model,

[0022] - using a generative machine-learning model to create, based at least on the user input, a visual representation of at least a part of a modified intermediate toy construction model for presentation to the user as a modified intermediate toy construction model.

[0023] Accordingly, the process provides an iterative and interactive process, during which the user can make decisions and influence the appearance of the final model rather than merely being presented with a complete toy construction model. Hence, the process facilitates a creative, yet guided process and allows a more efficient creation of a new toy construction model while preserving a large degree of freedom in creating the model. In particular, the process is not limited to a predetermined set of toy construction models but is capable of assisting a user in creating new toy construction models.

[0024] In some embodiments, user feedback received by the process, in particular the input indicative of user-selected modifications, may be used for adapting the machine-learning process, e.g. by an incremental training process or otherwise. In some embodiments, the process may use the machine-learning model to create a structured representation, as described herein, of the modified intermediate toy construction model and then translate the created structured representation into a visual representation, e.g. as described herein or otherwise.

[0025] In some embodiments, the visual representation of the modified intermediate toy construction model may emphasize the modifications made. In some embodiments, the process may create two or more candidate modified intermediate toy construction models, i.e. two or more candidate modifications to be made to the intermediate toy construction model. The candidate modifications may include one or more alternative toy construction elements to be added to the intermediate toy construction model so as to arrive at respective candidate modified toy construction models, or otherwise. The process may then provide functionality for receiving a user input indicative of one or more user-selected modifications of the presented candidate modifications. Accordingly, the user input indicative of a user- selected modification may be indicative of a user-selection among a plurality of alternative choices. The user input may then be used as input for a subsequent iteration of the process.

[0026] When the creation of the toy construction model is performed iteratively as a series of modifications to intermediate toy construction models, the user may use the system to iteratively construct a toy construction model by implementing the creation of the series of intermediate construction models, e.g. using physical toy construction elements concurrently with the creation of the digital version of the model.

[0027] In some embodiments, the user is presented with an image of the intermediate toy construction model built so far and with a choice to select one or more subsequent construction elements from a set of possible alternative subsequent construction elements, or otherwise from a set of alternative continuations. Hence, the subsequent construction elements may be alternative subsequent construction elements. Consequently, while guidance is provided to the user, there is still a large number of possible construction paths available to the user, and even a large number of resulting toy construction models, thereby allowing for a variety of different building experiences. The available toy construction models and construction paths are not limited to a set of predetermined choices, but they are rather created by the generative machine-learning model responsive to the user input.

[0028] If desired, the iterative process may be stopped in a variety of ways, e.g. after a predetermined number of iterations, responsive to a user command, responsive to the current intermediate toy construction model fulfilling one or more completion criteria, after lapse of a predetermined time, and / or based on any other suitable completion criterion or based on a combination of criteria. Upon completion of the iterative process, the process may use the current iterative toy construction model as a final toy construction model, i.e. the process may output a digital representation of the final toy construction model and / or output building instructions for constructing the final toy construction model, and / or otherwise.

[0029] According to a further aspect, disclosed herein are embodiments of a computer-implemented method of training a generative machine-learning model for creating digital representations of toy construction models, the method comprising:

[0030] - providing digital representations of a plurality of training toy construction models, each training toy construction model being constructed from a respective set of mutually interconnected toy construction elements,

[0031] - obtaining structured representations of the training toy construction models from the digital representations, each of the structured representations being a representation of a respective one of the plurality of training toy construction models, the structured representations including a first structured representation of a first training toy construction model constructed from a first set of toy construction elements, the first structured representation being indicative of the toy construction elements of said first set and of the mutual interconnections between respective ones of said toy construction elements of said first set making up said first training toy construction model,

[0032] - using the created structured representations as a training set for creating a generative machine-learning model trained to create structured representations of new toy construction models.

[0033] Accordingly, the machine-learning model is trained on a set of training data that includes structured representations of the training toy construction models, which represent all toy construction elements from which the respective training models are constructed, including their mutual interconnections. It will be appreciated that different training toy construction models may be constructed from the same or from different sets of toy construction elements. The structured representation of a toy construction model only includes the toy construction elements from which said toy construction model is constructed, i.e. all the toy construction elements included in and making up said toy construction model. Basing the generative machine-earning model on structured representations of toy construction models allows the machine-learning model to be trained to create representations of new models that are constructible in the physical world.

[0034] According to yet another aspect, disclosed herein are embodiments of a toy construction system, which comprises a set of toy construction elements with coupling members for releasably interconnecting the toy construction elements with each other. Some embodiments of the toy construction system comprise a data processing system comprising a processing unit and a display, wherein the data processing system is adapted to perform the steps of an embodiment of at least one of the methods disclosed herein. Alternatively, or additionally, some embodiments may include a computer program product configured to be executed by a data processing system and, when executed by the data processing system, to cause the data processing system to perform the steps of an embodiment of at least one of the methods disclosed herein.

[0035] The various aspects disclosed herein can be implemented in different ways, including the methods and the toy construction system described above and in the following, a data processing system, further methods, and further product means, each yielding one or more of the benefits and advantages described in connection with the above-mentioned aspects, and each having one or more preferred embodiments corresponding to the preferred embodiments described in connection with one or more of the other aspects and / or disclosed in the dependent claims.

[0036] In particular, the features of the method described herein may be implemented in software and carried out on a data processing system or other processing means caused by the execution of computer-executable instructions. The instructions may be program code means loaded in a memory, such as a RAM, from a storage medium or from another computer via a computer network. Alternatively, the described features may be implemented by hardwired circuitry instead of software or in combination with software.

[0037] Accordingly, the invention further relates to a data processing system adapted to perform one or more of the methods described above and in the following. The invention further relates to a computer program comprising program code means for performing all the steps of one or more of the methods described above and in the following when said program is run on a computer. The invention further relates to a computer program product comprising program code means for performing the steps of one or more of the methods described above and in the following when said computer program product is run on a computer or other data processing system. The program code means may be stored on a computer readable medium and / or embodied as a propagated data signal.

[0038] The data processing system may be or include a suitably programmed computer, tablet computer, smartphone or another suitably programmed mobile device or other suitable computing device. The data processing system may include more than one computing device, e.g. a user terminal and a remote computing system, such as a cloud computing architecture, or otherwise.

[0039] Brief description of the drawings:

[0040] Various aspects disclosed herein will be explained more fully below in connection with preferred embodiments and with reference to the drawings, in which:

[0041] FIGs. 1A-D each show a prior art toy construction element.

[0042] FIG. 2 schematically illustrates an example of a toy construction system as described herein.

[0043] FIG. 3 shows a schematic block diagram of an example of a data processing system as described herein.

[0044] FIG. 4 shows a schematic block diagram of another example of a data processing system as described herein.

[0045] FIG. 5 shows an example of a process for creating a digital representation of a toy construction model. FIGs. 6A-6G illustrate an example of a user-interface provided by a data processing system that implements the process of FIG. 5.

[0046] FIG. 7 shows an example of a process for creating a digital representation of a toy construction model based on a user prompt.

[0047] FIG. 8 schematically illustrates an example of a structured representation of a toy construction model.

[0048] FIG. 9 illustrates an example of a process for translating a received digital representation into a graph representation.

[0049] FIG. 10 illustrates a process for training a generative machine-learning model to create graph representations of toy construction models.

[0050] FIG. 11 illustrates a process for training a generative machine-learning model to create graph representations of toy construction models using a diffusion model.

[0051] Detailed description:

[0052] Various aspects and embodiments will now be described that facilitate the creation and, optionally, the iterative editing of buildable toy construction models. Various disclosed embodiments operate on a structured representation of the toy construction model, such as a graph representation. Various disclosed embodiments use machine-learning to discover and record patterns of toy construction models. Various disclosed embodiments store and use the created toy construction model. Various disclosed embodiments respond to user interaction.

[0053] Digital representation Various embodiments disclosed herein relate to the creation of a digital representation of a toy construction model, the toy construction model being constructed from a set of mutually interconnected toy construction elements. The digital representation is preferably a visual representation, i.e. a representation of the visual appearance of the toy construction model that can be viewed by a user and that illustrates to the user what the toy construction model looks like and, optionally, which toy construction elements it is constructible from and / or how it is constructible from the toy construction elements. The digital representation may be in the form of a computer- generated image of the toy construction model, a series of computer- generated images of the toy construction model, a computer-generated animation of the toy construction model, a computer-generated 2D or 3D digital rendering of the toy construction model, a computer-generated 3D model of the of the toy construction model, or the like. The digital representation may be a 3D representation that includes information about the 3D shape and / or structure of the toy construction model. The digital representation may be a 2D representation, e.g. a view of the toy construction from a certain viewpoint.

[0054] In embodiments where the process creates a set of building instructions, the building instructions may be created as a series of images illustrating how the set of toy construction elements are to be assembled so as to construct the toy construction model. The images may be presented as digital building instructions, e.g. displayed on a display of a processing device, or they may be printable. The set of building instructions may also be provided in a different format e.g. as an animated video, as spoken or written instructions, or the like.

[0055] Based on the created digital 2D or 3D representation of a visual appearance of the toy construction model and / or based on the created set of building instructions for creating the toy construction model, the user may construct a physical toy construction model from physical toy construction elements that corresponds to the toy construction model represented by the created digital representation or by the created set of building instructions. construction model and toy construction elements

[0056] The toy construction elements may be physical, real-world toy construction elements. The digital representation may include virtual toy construction elements, i.e. digital representations of the physical, real-world toy construction elements.

[0057] The toy construction elements may be pre-manufactured elements that can be attached to each other by the user of the toy construction elements to form user-selected spatial structures. To this end, the toy construction elements include coupling members adapted for mechanically connecting the toy construction elements with each other in a detachable manner, so as to construct a toy construction model. The coupling members of the toy construction elements may include coupling members using friction engagement, snap-fit engagement, or a combination of both, in order to support a detachable connection between the toy construction elements. Alternatively, or in addition thereto, detachable mechanical connections may also be formed between toy construction elements using coupling members providing a detachable form-locking positive connection. Alternatively, or in addition thereto, detachable mechanical connections may also be formed between toy construction elements using magnetic coupling members, i.e. cooperating coupling members relying on an attractive magnetic force.

[0058] A toy construction model is a modular, spatial structure constructed from a plurality of the toy construction elements, where the toy construction elements of the toy construction model are mutually interconnected with each other via their respective coupling members so as to form a coherent modular spatial structure. The toy construction model is modular and the building process is reversible, i.e. the toy construction elements that have been interconnected with each other by means of their respective coupling members to form the toy construction model can again be disconnected from each other such that they can be interconnected again with each other or with other toy construction elements of the system, e.g. so as to form a different spatial structure. Accordingly, the toy construction models can be disassembled into the individual toy construction elements without destroying or damaging the toy construction elements. In various embodiments, the toy construction elements are the smallest modules of the modular structure that cannot themselves be disassembled into smaller elements and subsequently be re-combined by means of their coupling members.

[0059] Structured

[0060] At least some embodiments of the various aspects disclosed herein use a structured representation of a toy construction model. The structured representation is a digital representation that represents the structure of the toy construction model it represents. In particular, the structured representation may be indicative of all toy construction elements from which the toy construction model is constructed, including a digital representation of the mutual interconnections between respective ones of said toy construction elements making up the toy construction model, i.e. of how the toy construction elements are interconnected via coupling members of the respective toy construction elements. Accordingly, the structured representation includes a representation of the spatial structure of the toy construction model. The structured representation may include information of how the toy construction model can be constructed from the individual toy construction elements that are attached to each other in a user-selectable spatial configuration, in particular in a re-combinable manner where individual toy construction elements can be detached / disassmbled from each other and re-attached / re-assembled in a different spatial configuration.

[0061] The structured representation may represent the individual toy construction elements such that each toy construction element has one or more element attributes associated with it, e.g. an attribute indicative of the type of toy construction element, and / or one or more attributes indicative of respective properties of the toy construction element. Examples of such properties may include a color of the toy construction element, a weight or mass of the toy construction element, connectivity properties of the toy construction element, a type of material or a property of the material from which the toy construction element is made, e.g. whether the material is transparent or opaque, whether it is soft or hard, etc. Connectivity properties of the toy construction element may be indicative of how the toy construction element is connectable with other toy construction elements, e.g. indicative of the type, and / or number and / or placement of the coupling members of the toy construction element.

[0062] The structured representation may include representations of the respective mutual connections between the toy construction elements making up the toy construction model. The representation of each connection may represent which toy construction elements are interconnected with each other and, optionally, how they are interconnected with each other. To this end, the representation of each connection may have associated with it one or more connection attributes, e.g. indicative of attributes of the represented physical connection. Examples of such properties include a strength of the connection, e.g. as represented by a force or torque or the like required to separate the connection, or otherwise. Other examples include a digital representation of how the two toy construction elements are connected. For example, the toy construction elements may each comprise multiple connectors or other coupling members, and the edge may represent which coupling members of one of the toy construction elements engage which coupling members of the other toy construction element.

[0063] The structured representation may represent the spatial structure of the toy construction model at least in part by data items that are mutually linked or otherwise associated with each other, thereby forming a structured representation. The structure of the representation, i.e. how the data items are linked with each other, may thus carry information about the structure of the toy construction model represented by the structured representation. The data items may include element data items representing toy construction elements and connection data items that link respective element data items with each other to represent connections between respective toy construction elements. The mutual links or other associations between respective data items define a structure of the representation, which in turn may represent the spatial structure of the modular toy construction model.

[0064] The inventors have found that a graph representation of the toy construction model is a particularly suitable form of structured representation for operating a generative machine-learning model that creates digital representations of toy construction models. A graph representation is a digital data structure representing a plurality of mutually, in particular pairwise, interconnected nodes. In various embodiments, each node represents one of the toy construction elements from which the toy construction model is constructed. The graph representation further comprises one or more edges, each edge representing a physical connection between two toy construction elements of the model as represented by two nodes of the graph representation.

[0065] Each node may be represented by a node data structure, or otherwise. Each node may have associated with it one or more element attributes, e.g. element attributes described above or otherwise. Similarly, the edges may be represented by edge data structures, or otherwise. Each edge may have associated with it one or more connection attributes, e.g. indicative of attributes of the physical connection represented by the edge, for example as described above or otherwise.

[0066] The inventors have realised that, when the toy construction model is represented by means of a graph representation, machine-learning techniques that are particularly suitable for learning and generating graph structures and that have been developed in the field of drug discovery may be applied to the creation of toy construction models. Examples of such techniques are described in [1], Other examples of suitable structured representations include, for instance, SMILES (Simplified Molecular Input Line Entry System), a sequential notation that describes molecular structure, see e.g. [2],

[0067] Translation into a digital 2D or 3D representation of a visual appearance of the toy construction model

[0068] Various embodiments disclosed herein translate a created structured representation into a digital representation of the visual appearance of the model and / or into a set of building instructions, thereby allowing the resulting model to be presented in a user-friendly manner, e.g. as an image, a series of images, a 3D digital rendering, etc.

[0069] Examples of digital 3D representations include a surface model of the toy construction model, e.g. a 3D model based on vertices and edges. In contrast to the structured representation disclosed herein, such 3D representations of the visual appearance do not necessarily describe how a model is constructed from re-combinable toy construction elements in a particular configuration; they may merely describe the visual appearance of an object and not how the object is constructed from individual parts. The generated structured representation, through its structure and the attributes it represents, may uniquely, or at least to a sufficient degree, define a toy construction model so as to allow the structured representation to be translated into a virtual toy construction model or into a building instruction for constructing a physical toy construction model. For example, the structured representation of a toy construction model constructed from LEGO® bricks may be translated into an LXFML file. In cases where the graph or other structured representation does not contain the full information required for a fully automatic translation, the translation may be performed in a human assisted manner. This step can also serve as part of the building experience, allowing users to engage creatively and apply their own interpretation to the generated structure, enhancing personalization while still using the generated structure as a guide for building the toy construction model.

[0070] User interaction

[0071] Various embodiments of the aspects disclosed herein are interactive, i.e. they involve and are affected by user input. The user input may be in the form of user prompts to a generative machine-learning model, in the form of user feedback, in the form of control inputs, and / or the like.

[0072] Some embodiments receive a user prompt to create a digital representation of a toy construction model, the user prompt including one or more desired attributes of the toy construction model. In particular, some embodiments disclosed herein may create a digital representation solely based on one or more such user prompts, optionally in combination with one or more predetermined constraints. The user prompts and / or the predetermined constraints may define global attributes of the toy construction model, and various embodiments of the process do not require a detailed structural specification of a spatial model structure of the desired toy construction model as an input to the process, i.e. the user does not need to specify how toy construction elements are to be connected with each other or indeed spatially arranged relative to each other within the desired toy construction model. Alternatively, or additionally, some embodiments receive user prompts and / or other user input indicative of a user-selected modification to a created intermediate toy construction model during an interactive process for creating a digital representation of a toy construction model. For the purpose of the present disclosure, the term user prompt is intended to refer to a user input to a generative machine-learning process, where the generative machine-learning process operates responsive to the user input, i.e. the user prompt may be considered an instruction to the generative machine-learning process on which the generative machine-learning process is configured to react.

[0073] Generally, the user input, in particular the user prompt, may be provided in a variety of forms, including:

[0074] • Text or voice input indicative of a user-intent, in particular text or voice input indicative of a desired toy construction model or a desired modification of a previously created (e.g. intermediate) toy construction model. Examples of such text or voice inputs may be provided in natural language, such as “a big purple cat”, “more windows” “cheaper bricks, but also more rainbow colors and cooler”, etc.

[0075] • Inputs received via user-activatable control elements of a userinterface, e.g. of a graphical user-interface, such as sliders and / or other controls.

[0076] • Inputs received via a virtual toy construction environment, such as the LEGO® Digital Designer, and / or via a recorded user interaction with a physical toy construction model, and / or otherwise. Such inputs may be indicative of moving, adding or removing toy construction elements, either physically or digitally, or by otherwise manipulating a physical toy construction model. Accordingly, in some embodiments, the user prompts may be in the form of a detected modification or other manipulation of a virtual or physical toy construction model by the user. For example, such a manipulation of a physical toy construction model may be detected by a process as described in WO 2016 / 075081 , WO 2012 / 160057, WO 2020 / 152189, or otherwise, e.g. by using an Al-based description generator for generating a textual description of a series of recorded images depicting a user manipulating a physical toy construction model.

[0077] • Inputs received by a device for detecting physical or digital gestures, e.g. gesturing with hands how the toy construction model should be shaped.

[0078] • Inputs indicative of highlighting parts of a (physical or virtual) toy construction model and, optionally, using text, voice, gestures, controls, images or other forms of input to specify desired changes to the highlighted parts.

[0079] • Inputs in the form of received images, scanned objects, or other ways of specifying the goals of the model.

[0080] • Inputs in the form of received data from other interactive sources, like gameplay traces or unlocking certain pieces or build styles from winning a game.

[0081] • Inputs in the form of received environmental data or constraints, from digital or real-world data, such as covering a particular table in augmented reality with a forest, or making a racetrack that fits a landscape.

[0082] It will be appreciated that user input may be provided as a combination of two or more of the above and / or in a different manner.

[0083] User input indicative of user-selected attributes and / or a user-selected modification may be indicative of a variety of attributes / modifications and indicate attributes / modifications with varying degrees of specificity. For example, in some embodiments, the user input may be indicative of one or toy construction elements to be added to a toy construction model, rearranged within the toy construction model, and / or removed from the toy construction model. Alternatively, or additionally, the user input may be indicative of a desired attribute or property of the toy construction model, e.g. a size, shape and / or color of the toy construction model, the resemblance of the toy construction model with a real-world or fantasy object, and / or the like. For example, the user input may be indicative of a desired degree of an attribute of the toy construction model, e.g. that the toy construction model should be larger, appear cuter, resemble more a big lion, etc. One or more of the attributes may define or be indicative of a constraint to be fulfilled by the toy construction model, e.g. a constraint on the size, shape and / or color of the toy construction model, on the number of toy construction elements, on the pool or set of toy construction elements that may be included in the toy construction model and / or the like. In some embodiments, the user input may be indicative of a user-selection among a plurality of alternative choices presented by the process, e.g. a user-selection among a plurality of alternative toy construction elements, alternative toy construction models, alternative continuations of an interactive construction process, etc.

[0084] In some embodiments, responsive to receiving user input indicative of a desired toy construction model (e.g. a toy construction model having desired attributes) or user input indicative of a desired modification of an existing toy construction model (e.g. of an intermediate toy construction model previously created during an interactive model creation process), the process may use a machine-learning model to create a structured representation of the desired toy construction model (e.g. of a toy construction model having the desired attributes) or of the desired modified toy construction model, and then translate the created structured representation into a visual representation, e.g. as described herein or otherwise.

[0085] In some embodiments, the visual representation of the modified toy construction model (e.g. of a modified intermediate toy construction model during an interactive, iterative process) may emphasize the modifications made. In some embodiments, the process may create two or more candidate modified intermediate toy construction models, i.e. two or more candidate modifications to be made to a current intermediate toy construction model. The candidate modifications may include one or more alternative toy construction elements to be added to the current intermediate toy construction model so as to arrive at respective candidate modified toy construction models, or otherwise. The process may then provide functionality for receiving a user input indicative of one or more user-selected modifications of the presented candidate modifications. Accordingly, the user input may be indicative of a user-selection among a plurality of alternative choices.

[0086] The process of creating a digital representation of a toy construction model according to at least some embodiments disclosed herein is initiated based on a user prompt. The user prompt may be provided to the system in a variety of ways, e.g. as described above in the context of the user input in general, or otherwise. The user-prompt may be a verbal prompt, e.g. by typing instructions into the computer, by a voice interface having a speech-to- text converter, an image prompt, e.g. by the user capturing or otherwise inputting / selecting an image of an object, a prompt indicative of a user choice from a set of available choices and / or the like, or a combination thereof. The user prompt is indicative of one or more attributes of the toy construction model to be created, e.g. of the desired visual appearance of the toy construction model, e.g. that the toy construction model should resemble a user-selected object and / or of other attributes of the toy construction model to be created. Attributes related to what object the toy construction model should resemble may be represented by an image of the object and / or by text or verbal input, e.g. “a car”, “a huge fire truck”, “an angry dragon”, “a cute dragon”, etc. In some embodiments, the user may input a narrative, and the process may derive one or more objects (e.g. characters) appearing in the narrative and create a toy construction model resembling or otherwise representing the derived object. Other examples of attributes include constraints to be fulfilled by the toy construction model.

[0087] In some embodiments, user feedback received by the process during or after the creation of a digital representation of a toy construction model, in particular the input indicative of user-selected modifications during an interactive process, may be used for adapting the machine-learning process used by the process, e.g. by an incremental training process or otherwise.

[0088] Constraints

[0089] In various embodiments of one or more of the aspects disclosed herein, the generative machine-learning model may receive one or more constraints to be fulfilled by the created digital toy construction model. For example, the machine-learning model may receive the constraints as part of the user input. To this end, the machine-learning model may receive the constraints as part of a user prompt, e.g. a user prompt initiating the process or user prompts received during an iterative process. Alternatively, or additionally, the machine-learning model may be preconfigured with one or more constraints or otherwise receive such constraints, e.g. by reading a configuration file or other data structure, by receiving environmental data, or data from other sources, or otherwise. The constraints may be regarded as attributes of the toy construction model, as they may define constraints on one or more attributes of the toy construction model to be constructed, or constraints on sets or ranges of attribute values.

[0090] Examples of such constraints include:

[0091] - an identity of one or more toy construction elements to be included in the toy construction model, in particular as a pivotal element,

[0092] - a target size or a target range of sizes of the toy construction model; the target size may be expressed as a height, volume, weight, number of toy construction elements in the model, and / or as another one or more parameters indicative of the size of the model,

[0093] - one or more colors or a color scheme of the toy construction model,

[0094] - a target age or age range, or a target proficiency level or range of proficiency levels, of a user constructing the toy construction model,

[0095] - an indication of a type of object represented by the toy construction model,

[0096] - a target pool of toy construction elements from which the toy construction elements making up the toy construction model are to be selected.

[0097] Training

[0098] Various embodiments of the aspects disclosed herein use a trained machinelearning model, in particular a trained generative machine-learning model. A trained generative machine-learning model is a type of machine-learning model that has been trained based on training data to learn the underlying patterns or distributions of the data in order to generate new, similar data. In the context of the present disclosure, the generative machine-learning model may be trained based on a training set of structured representations of toy construction models, in particular to learn the underlying patterns or distributions of the structured representations of toy construction models, in order to generate new, similar structured representations of toy construction models.

[0099] When training the machine-learning model, the training may thus be based on a training set, which may be indicative of a plurality of training toy construction models. To this end, the training process may receive representations of the training toy construction models. At least some embodiments of the various aspects disclosed herein use a structured representation of a toy construction model as input to the machine-learning model. In particular, the structured representation may be indicative of all toy construction elements from which the toy construction model is constructed, including a digital representation of the mutual interconnections between respective ones of said toy construction elements making up the toy construction model. Accordingly, if the received representations are not structured representations to be used for the training of the machine-learning model, the training process may translate the received representations into corresponding structured representations.

[0100] The received representations may e.g. include non-structured data, e.g. including information about the toy construction elements included in the model and their respective positions within the model. The process may thus initially derive the interconnectivity of the toy construction elements to arrive at the structured representation.

[0101] The process of creating a digital representation of a toy construction model according to at least some embodiments disclosed herein includes receiving a user prompt or user input indicative of one or more attributes. The process may further receive one or more constraints to be fulfilled by the model. Some or all training toy construction models used as training data may thus have associated with it one or more suitable attributes allowing the machinelearning model to compare the training models with one or more desired attributes and / or constraints. For example, the training toy construction models may be labelled with one or more properties, such as an age range of users for which the model is suitable, a level of proficiency required to build the model, a theme or object represented by the model, a name of the model, a color theme, etc. Some labels may be manually assigned while others may be derivable from the received representation. Derivable attributes may include a total weight, color, color scheme, volume, shape and / or size of the model, which may be derivable based on training or estimated properties of the toy construction elements included in the model. In some embodiments, properties such as an identity of the object represented by the model may be automatically derivable, e.g. using machine vision or other known object recognition techniques.

[0102] In some embodiments, the training set includes graph representations of the training construction elements, thereby providing a particularly effective training. However, it is also possible to train the machine-learning model on other types of representations, e.g. a text format (such as the LXFML elements) or a list of element data.

[0103] The training data may be created in a variety of ways and be obtained from a variety of sources, including:

[0104] - Existing toy construction models, such as commercially available toy construction sets.

[0105] - User-constructed toy construction models.

[0106] - Toy construction models designed by professional model designers.

[0107] - Toy construction models built (e.g. by end users or professional model designers) using physical bricks, then uploaded or scanned digitally.

[0108] - Toy construction models built (e.g. by end users or professional model designers) in a virtual toy construction environment, e.g. in a digital app. The virtual construction may e.g. be part of a building or playing experience or puzzle solving experience (e.g. “Fold At Home”).

[0109] - Partial models of larger toy construction models, which can increase the training set. The partial models may be obtained as subgraphs of the graph representations of the larger toy construction models.

[0110] - Synthetic training data, which may include toy construction models generated by other algorithms or modified versions of existing toy construction models. Such synthetic data may be used to extend the training set. For example, such data may be created by a computer-implemented process for rebuilding an existing toy construction model with many different kinds of toy construction elements, e.g. using a checking algorithm to detect “correctness”, i.e. to check one or more criteria imposed on suitable toy construction models.

[0111] The training process then uses the obtained training data to build a trained machine-learning model. As discussed herein, the training is preferably based on structured representations, in particular graph representations, of the training toy construction models. Nevertheless, the training may also be based on a combination of textual representation of the model (e.g., an LXFML file), a list of elements, and / or a graph representation, or otherwise. In some embodiments, the training process also trains relationships between the training toy construction models and other kinds of data, such as:

[0112] - resulting 3D data about the toy construction model,

[0113] - text descriptions of the model (e.g. as described in

[0014] ), thereby facilitating text-based prompts / input such as “give me a purple car with wings,” “make the cat bigger,” “use fewer pieces”, etc,

[0114] - images of the resulting toy construction model,

[0115] - other resulting data about the toy construction model (strength, cost, weight, dimensions, etc.)

[0116] Various types of generative machine-learning models may be trained to create structured representations of toy construction models as described herein. The training process for training the generative machine-learning model may depend on the type of generative machine-learning model used.

[0117] Examples of generative machine-learning models that are suitable for creating graph-based structured representations of toy construction models include:

[0118] Variational Autoencoders (VAEs), see e.g. [3],

[0119] Generative Adversarial Networks (GANs), see e.g. [4], Normalizing Flows, see e.g. [5],

[0120] Diffusion-based models, see e.g. [6] or [7], The particular choice of training process may depend on the type of generative model used. For example, training of a generative diffusion model on graphs using a forward pass and a reverse denoising pass may be supervised using loss functions such as Mean Squared Error (MSE), Crossentropy, or more specific graph-based losses (considering adjacency matrices and node / edge sets, for instance). Models used in the reverse pass may include Graph Neural Networks (GNNs), Graph Attention Networks (GATs), or other suitable ML architectures designed for graph data.

[0121] Various aspects and embodiments of toy construction systems disclosed herein will now be described in more detail with reference to the drawings and with reference to toy construction elements in the form of bricks. However, the invention may be applied to other forms of construction elements used in toy construction sets.

[0122] FIGs. 1A-D each show a prior art toy construction element in the form of a toy construction brick with coupling members in the form of coupling studs 105 on their top surface and a cavity 102 extending into the brick from the bottom. FIG. 1A shows a top side of a toy construction brick, while FIG. 1 B shows the bottom side of the toy construction brick of FIG. 1A. FIG. 1C-D show examples of similar toy construction bricks of different sizes. The cavity has a central tube 103, and coupling studs of another brick can be received in the cavity in a frictional engagement as disclosed in US 3 005 282. The toy construction elements shown in the remaining figures may have this known type of coupling members in the form of cooperating studs and cavities. However, other types of coupling members may also be used. The coupling studs are arranged in a square planar grid, i.e. defining orthogonal directions along which sequences of coupling studs are arranged. Generally, such an arrangement of coupling members allows the toy bricks to be interconnected in a discrete number of orientations relative two each other, in particular at right angles with respect to each other. It will be appreciated that other geometric arrangements of coupling members may result in different orientational constraints. For example, the coupling members may be arranged in a triangular, regular grid allowing a toy construction element to be placed on another toy construction element in three different orientations.

[0123] Generally, the coupling members may include coupling members that may be grouped into different classes of coupling members, e.g. connectors, receptors, and mixed elements. Connectors are coupling members that may be received by a receptor of another toy construction element, thereby providing a connection between the toy construction elements. For example, a connector may fit between parts of another toy construction element, into a hole, or the like. Receptors are coupling members which can receive a connector of another toy construction element. Mixed elements are parts that can function both as a receptor and a connector, typically depending on the type of the cooperating coupling member of the other toy construction element.

[0124] Toy construction elements of the type illustrated in FIGs. 1A-D are available under the name LEGO® in a large variety of shapes, sizes, and colors. Furthermore, such toy construction elements are available with a variety of different coupling members. It is understood that the above toy construction element merely serves as examples of possible toy construction elements.

[0125] Embodiments of the method and system disclosed herein may be used in connection with a variety of toy objects and, in particular with construction toys that use modular toy construction elements based on dimensional constants, constraints and matches, with various assembly systems like magnets, studs, notches, sleeves, with or without interlocking connection etc. Examples of these systems include but are not limited to the toy constructions system available under the tradename LEGO®. For example, US3005282 and USD253711S disclose one such interlocking toy construction system and toy figures, respectively.

[0126] FIG. 2 schematically illustrates an example of a toy construction system as described herein. The toy construction system includes a set of conventional toy construction elements 120, e.g. of the type described in connection with FIGs. 1A-D, or otherwise, from which one or more toy construction models 131 , 132 can be constructed. In the example of FIG. 2, a toy figurine 131 and a toy car 132 constructed from the toy construction elements 120 of the set are shown. The toy construction system further comprises a suitably programmed data processing system 450, e.g. a tablet computer or smartphone or other portable computing device executing an app that implements a digital game of the toy system.

[0127] In particular, the data processing system 450 may be programmed to:

[0128] • receive a user prompt to create a digital representation of a toy construction model, the user prompt including one or more desired attributes of the toy construction model,

[0129] • use a generative machine-learning model to create, based on the user prompt, a structured representation of a toy construction model, the toy construction model being constructed from toy construction elements of the set, the structured representation being indicative of the toy construction elements from which the toy construction models is constructed and of the mutual interconnections between respective ones of said toy construction elements from which the toy construction models is constructed,

[0130] • translate, in particular fully or partly automatically translate, the created structured representation into a digital 2D or 3D representation of a visual appearance of the toy construction model and / or into a set of building instructions for creating the toy construction model, and

[0131] • to present the digital 2D or 3D representation and / or the building instructions to the user.

[0132] Alternatively, or additionally, the data processing system 450 may be programmed to, repeatedly:

[0133] - present a visual representation of an intermediate toy construction model, - receive a user input indicative of a user-selected modification to said intermediate toy construction model,

[0134] - use a generative machine-learning model to create, based at least on the user input, a visual representation of at least a part of a modified intermediate toy construction model for presentation to the user as a modified intermediate toy construction model.

[0135] For example, the data processing system may be programmed to perform the process of any of FIGs. 5 and 7, or of another embodiment of a process for creating a digital representation of a toy construction model.

[0136] FIG. 3 shows a schematic block diagram of an example of a data processing system, e.g. of the data processing system of the toy construction system of FIG. 2.

[0137] The data processing system of FIG. 3 is implemented as data processing device 450, e.g. a computer, smartphone, tablet computer or the like. The data processing device 450 comprises a central processing unit 455, a memory 456, and a user interface 457. The data processing device may comprise one or more additional components, e.g. an image capture device 459.

[0138] The user interface 457 may e.g. include a display, such as a touch screen, and, optionally input devices such as buttons, a touch pad, a pointing device, etc.

[0139] The central processing unit 455 may be a programmable microprocessor or another suitable type of processing device.

[0140] The memory 456 may include a rewritable portion and / or a read-only portion.

[0141] The memory may use any suitable memory technology for storing data. The image capture device 459 may include a digital camera, a depth camera, a stereo camera, and / or the like.

[0142] FIG. 4 shows a schematic block diagram of another example of a data processing system, e.g. of the data processing system of the toy construction system of FIG. 2. The data processing system of FIG. 4 is similar to the data processing system of FIG. 3, the only difference being that the data processing system of FIG. 4 comprises a data processing device 450 and a remote system 1170. To this end, the data processing device 450 further comprises a communications interface 460, such as a wireless or wired communications interface, allowing the data processing device 450 to communicate with the remote system 1170. The communication may be wired or wireless. The communication may be via a communication network. The remote system 1170 may be a server computer or other suitable data processing system which may be configured to implement one or more of the processing steps described herein, e.g. as a cloud computing architecture. For example, the remote system 1170 may implement a generative machinelearning model, or parts thereof, for creating a digital representation of a toy construction model. Yet alternatively or additionally, the remote system may implement at least a part of the digital game, e.g. in embodiments where the digital game includes a multi-player play experience or a networked play experience.

[0143] It will be appreciated that other examples of data processing systems may be implemented in a different manner, e.g. with components distributed among multiple devices in a different manner.

[0144] FIG. 5 shows an example of a process of creating a digital representation of a toy construction model. The process may be implemented by a data processing system, e.g. as described in connection with any one of FIGs. 2 - 4, or otherwise. In step S1 , the process receives a first user prompt indicative of a type of toy construction model the user wishes to construct. The process may receive the first user prompt in a variety of ways. For example, the process may display a number of object categories from which the user may select one. Examples of categories may e.g. include “car”, “animal”, “creature” etc. In other embodiments, the process may allow the user to provide text input describing the desired category, e.g. “I would like to build a fire truck”, etc. The text input may be provided via a keyboard, as audio input using a speech-to-text process, or in another suitable way. In some embodiments, the process may receive a narrative (e.g. spoken or as text) involving one or more objects, e.g. a story involving a fire truck. The process may perform a natural language processing step to derive one or more objects of which respective one or more toy construction models are to be constructed. In some embodiments, the process may allow the user to provide an image depicting an object to be built, e.g. a captured digital picture, a drawing and / or the like. In some embodiments, the choice of objects may be restricted to one or more classes of objects, e.g. to different types of cars. The class(es) of objects may be predetermined in accordance with the type of toy construction elements included in a toy construction set.

[0145] In step S2, the process receives a user selection of an initial toy construction element. For example, the process may allow the user to select a toy construction element from a set of toy construction elements, e.g. from a subset of toy construction elements of a toy construction set. Alternatively, the user may provide a typed text input, input an image or audio input descriptive of a selected toy construction element. In some embodiments, the user may select more than one initial toy construction elements, e.g. by selecting an initial intermediate toy construction model or part-model.

[0146] It will be appreciated that, in alternative embodiments, the user may only select a type of model to be constructed, without also selecting an initial toy construction element. Similarly, in some embodiments, the process may only receive an indication of the initial toy construction element, without receiving a user selection of the type of model to be constructed.

[0147] Yet alternatively or additionally, the process may receive an input indicative of one or more other attributes of the model to be constructed, in particular of one or more other constraints to be fulfilled by the toy construction model to be constructed. Examples of such other attributes include a color scheme, an indication of a size of model, an indication of the set of toy construction elements available for constructing the model, an age range or indication of the proficiency of the user, and / or the like. Such other attributes may be received by the process as part of the first user prompt, e.g. instead of or in addition to an indication of the desired type of toy construction model to be constructed. Alternatively, or additionally, the process may receive such other attributes, e.g. other constraints, in a different manner.

[0148] The process then uses the user-selected initial toy construction element as a current intermediate model 50. In embodiments, where the user does not select an initial toy construction element, the process may automatically select an initial toy construction element, e.g. randomly, by selecting a predetermined toy construction element, or the like. Yet alternatively, the process may initially start without initial toy construction element.

[0149] In step S3, the process uses a generative machine-learning model to create a set of alternative additions to (or other modifications of) the current intermediate toy construction model 50. To this end, the process may use the information received as part of the first prompt, e.g. information about the type of construction model to be built, as input. The process further provides a digital representation of the current intermediate toy construction model 50 as an input to the generative machine-learning model. The generative machine-learning model is trained to output a number of possible continuations of the construction process. In the example of FIG. 5, the process creates three alternative candidates 51 A, 51 B and 51 C, respectively. It will be appreciated that, in other embodiments, the process may create a different number of candidates. In some embodiments, in step S3, the process may apply a number of constraints 52 to the generative machinelearning model, e.g. predetermined or user-configured constraints and / or constraints based on the initial user prompts. The constraints may determine the size or complexity of the created candidates, the available toy construction elements, etc. and / or the like.

[0150] The process may present the created candidates to the user and allow the user to select one of the candidates. The process then creates a modified current intermediate toy construction model 54 by implementing the user- selected candidate continuation of the construction process.

[0151] The process may display the selected modified intermediate toy construction model, e.g. along with building instructions for guiding the user in creating a physical version of the modified intermediate toy construction model.

[0152] The process may then repeat step S3 with the modified intermediate toy construction model 54 as the new current intermediate toy construction model 50.

[0153] The process may repeat the iterative and interactive construction process until a completion trigger is reached, e.g. until the user terminates the process, until the modified intermediate toy construction model 54 fulfils one or more completion criteria, until the machine-learning model indicates completion of the construction process, and / or the like. Examples of completion criteria may include a total number of toy construction elements or another suitable attribute of the toy construction model.

[0154] It will be appreciated that other embodiments may implement a different iterative process, e.g. by only presenting one continuation of the building process and, based on user input modifying the continuation. In yet alternative embodiments, the process may not be iterative but present a complete model in response to the initial user prompt. FIGs. 6A-6G illustrate an example of a user-interface provided by a data processing system that implements the process of FIG. 5.

[0155] FIG. 6A-C illustrate examples of user-interfaces for providing user prompts. In this example, the user can choose a type of model, a favourite color, and one of the toy construction elements to be included in the model to be constructed.

[0156] FIG. 6D illustrates an example of an initial iteration of the construction process, where the user is presented with three alternative initial intermediate construction models, allowing the user to select one of the three alternatives.

[0157] FIGs. 6E and 6F illustrate examples of subsequent iterations of the construction process, where the user is presented with an illustration of the respective current intermediate construction model 60 and three alternative continuations.

[0158] For each iteration, the user-interface includes respective activatable userinterface elements 61 A, 61 B and 61 C, respectively, that allow the user to select one of three alternative continuations. It will be appreciated that other embodiments may provide a different number of alternative continuations. Optionally, the user-interface further provides a user-activatable element 62 for requesting further alternatives. The user-interface may further provide a user-activatable progress bar 63, allowing the user to rewind the process to return to a previous iteration.

[0159] FIG. 6G illustrates an example of a user interface showing the completed model. The process may provide functionality to execute a digital game involving the created toy construction model, functionality for presenting building instructions for constructing a physical version of the toy construction model, and / or the like. Fig. 7 shows an example of a process for creating a digital representation of a toy construction model based on a prompt, in particular a user prompt.

[0160] In the example of FIG. 7, the process creates a digital representation 74 of new toy construction model based on a digital representation 71 of a known toy construction model.

[0161] The digital representation 71 may include an image or a data structure descriptive of the known toy construction model. An example of a data structure includes an XML document describing the toy construction elements of the toy construction model, e.g. in LXFML format or otherwise. LXFML files represent toy construction models constructed from toy construction elements in the form of LEGO® bricks; they contain detailed information about each toy construction element (including attributes like ID and color) as well as their spatial position (translation and / or rotation) in 3D space.

[0162] In step S31 , the process translates the received digital representation 71 into a graph representation 72 or another structured representation of the known toy construction model, where the structured representation includes information about the toy construction elements of the model and their mutual interconnections. An example of a graph representation with be described below with reference to FIG. 8. An example of a process for translating a received digital representation into a graph representation will be described below with reference to FIG. 9.

[0163] The graph representation may be an undirected heterogeneous graph representation or another suitable type of graph representation. A graph represents a toy construction model and includes nodes and edges.

[0164] Generally, in a graph representation of a toy construction model, each toy construction element of the toy construction model represented by the graph is represented as a node in the graph. A node can contain attributes such as (but not limited to):

[0165] - Element ID

[0166] - Color

[0167] - Type of building element (e.g. which of a set of subsystems of a toy construction system the element relates to)

[0168] - Element group (e.g., brick, beam, plate, botanic, figure, tire, decoration, connector, tube, and / or the like

[0169] - Material group (e.g., moulded, decorated, foil, textile, and / or the like)

[0170] - Absolute positional information (3D coordinates and / or rotation and translation from the origin)

[0171] The mutual interconnections between respective ones of the toy construction elements of the toy construction models are represented by edges of the graph. And edge connects two nodes if the toy construction elements represented by the nodes are physically connected in the toy construction model represented by the graph. Edges can contain attributes such as (but not limited to):

[0172] - Relative positional information (orientation, number of connection points, e.g., number of coupling members that provide the connection, e.g. the number of studs, and / or the like)

[0173] - Type of connection (e.g., Knob and Tube, Bush and Hole, Shaft and Holder, Ball joint, and / or the like.)

[0174] In step S32, the process feeds the graph representation 72 into a generative machine-learning model trained to create a new graph 73 from a known graph 72. The generative machine-learning model may e.g. comprise a graph neural network. The machine-learning model may implement a discrete diffusion model, a deep generative method, a discrete flow method, an energy-based model, a Markov chain, a recurrent neural network or other probabilistic model for creating a sequence of graphs from an input graph. The new graph 73 represents a new toy construction model. In step S33, the process may perform optional post-processing, such as a stability and / or collision analysis.

[0175] Finally, in step S34, the process may translate the created new graph 73 into another digital representation 74 of the new toy construction model, e.g. an XML-based format or another suitable format.

[0176] FIG. 8 schematically illustrates an example of a graph representation of a toy construction model. In particular, FIG. 8 shows a toy construction model 132 constructed from a plurality of interconnected toy construction elements. FIG. 8 further shows a graph representation 232 of the toy construction model 132. The graph representation comprises a plurality of nodes, represented by circles in FIG. 8, and a plurality of edges, represented by straight lines each connecting a pair of nodes.

[0177] Each node represents one of the toy construction elements of the toy construction model 132. Each node may be represented by a data structure representing the type of toy construction element and one or more attributes of the toy construction element, e.g. its color etc.

[0178] Each edge connects two nodes and represents how the toy construction elements represented by the two nodes are interconnected with each other within the toy construction models 132. In this example, solid lines represent a connection where coupling members of the two toy construction elements interact so as to cause a physical connection between the toy construction elements. Dotted lines represent toy construction elements that touch each other without forming a physical connection that would require a force to disengage.

[0179] Each edge, or at least each edge representing an actual connection by means of coupling members, may have associated with it one or more attributes, e.g. attributes representing a strength of the connection and / or the type and / or number of connection elements of the toy construction elements that interact with each other so as to form the connection between the two toy construction elements.

[0180] As will readily be appreciated from FIG. 8, the digital representation of the visual appearance of the toy construction model 132 shown on one side of the drawing allows the viewer to readily perceive and appreciate how the toy construction will look when constructed from the toy construction elements. However, the graph representation 232, while particularly suitable for being processed by a machine-learning model, is not particularly suitable for presenting the visual appearance of the toy construction model.

[0181] FIG. 9 illustrates an example of a process for translating a received digital representation into a graph representation.

[0182] In initial step S311 , the process receives a digital representation of a toy construction model, e.g. a toy construction model to be used as a part of a training set for training a machine-learning model, or a toy construction model used as an input to an already trained generative machine-learning model for creating a new toy construction model. The received digital representation may e.g. be a textual representation, e.g. in the form of an LXFML model format file, or other way to represent a toy construction model, e.g. as a list of toy construction elements including information about the properties of the toy construction elements and about their position in 3D space.

[0183] In step S312, the process detects connections between the toy construction elements within the toy construction model. For example, the detection may purely or partly be based on spatial coordinate information, e.g. based on detected proximity between toy construction elements. Alternatively, or additionally, the detection may use available data representing the coupling members of the toy construction elements. Yet alternatively or additionally, the process may calculate connections from the mesh geometry of the toy construction elements. In step S313, the process creates a graph representation from the listed toy construction elements and from the detected connections. The resulting graph may be uniquely specified, with all necessary information for unique connection (angle, connection points, etc.). Alternatively, the resulting graph may be partially specified, e.g. by specifying that two toy construction elements are connected with each other but without specifying details of the connection, such as which coupling members (studs / connectors) provide the connection, or at which angles the toy construction elements are connected with each other.

[0184] FIG. 10 illustrates a process for training a generative machine-learning model to create graph representations of toy construction models.

[0185] The process comprises a training phase 91 and an operational phase 92. During the training phase, a trained generative model 97 is created. The trained generative model 97 is then used during the operational phase 92 for creating graph representations 98 of new toy construction models.

[0186] During the training phase 91 , the process receives graph representations 93 of a training set of known toy construction models. The machine-learning model 94 then creates a plurality of new graph representations 95. The process provides feedback 96 on which of the created graphs correspond to desirable toy construction models. The feedback may be performed based on a manual inspection of the resulting toy construction models represented by the created graphs and / or by an automatic analysis of the created graphs. Examples of automatic analysis include a stability analysis or another analysis for determining whether the created graphs or their corresponding toy construction models fulfil one or more predetermined criteria, e.g. in terms of the size of the model, complexity, etc. Another example of an automatic analysis includes the use of an adversarial network that is trained to classify graphs or other representations of toy construction models into certain criteria, e.g. complexity level, age range, object classes (e.g. “car” vs. “animal”), etc.

[0187] Based on the feedback 96, the model parameters of the generative machinelearning model 94 are adapted, e.g. by a back-propagation algorithm by a reinforcement training algorithm or another suitable type of training algorithm, thereby training the model to create graphs representing toy construction models of a desired type or having desired properties.

[0188] Once trained, the trained machine-learning model 97 is capable of creating new graphs 98, e.g. based on user prompts and / or other constraints as described herein. Hence, in FIG. 10, block 97 represents the machinelearning model resulting from the training of machine-learning model 94, i.e. block 97 represents the trained version of model 94. In some embodiments, feedback 99 - e.g. user feedback - may be collected for adaptively improving the generative model.

[0189] FIG. 11 illustrates a process for training a generative machine-learning model to create graph representations of toy construction models using a diffusion model.

[0190] The input to the process is a training set 93 of valid graphs, each representing a toy construction model as described herein, e.g. as described in connection with FIG. 8 above or otherwise.

[0191] For each valid graph Gorepresenting a toy construction model of the training set 93, the process performs a forward pass as represented by solid arrows and a reverse pass, as represented by dashed arrows.

[0192] During the forward pass, the process gradually adds noise to the graph structure. To this end, the process may use different types of noise, e.g. one or more of the following:

[0193] - Discrete noise over node or edge attributes Gaussian noise on node latent variables

[0194] Gaussian noise on the adjacency matrix of the graph and / or or on node features.

[0195] After T steps, the final graph GT becomes partially or fully randomized, as illustrated by the randomized versions 931 of the graphs of the training set.

[0196] During the reverse reverse pass, the machine-learning model learns the reverse (denoising) process from GT back to Go. During this training process, the model is optimized to predict the original graph from a partially or fully noisy one.

[0197] In one example, the process may be implemented as follows: At every timestep t (0 < t < T), where the time increment indicates the current noise level) the machine-learning model takes in a noisy graph Gt, and optional conditioning inputs (e.g., desired color, structure hints, etc); the process then outputs either a denoised version of the graph or a noise estimate to be subtracted from Gt to recover a version of Gt-i.

[0198] Training may be supervised using loss functions such as Mean Squared Error (MSE), Cross-entropy, or more specific graph-based losses (considering adjacency matrices and node / edge sets, for instance). For instance, an MSE loss can be applied to the perturbed node and edge attributes, measuring the difference between the model’s predicted values and the ground truth from Go (or an intermediate Gt).

[0199] Models used in the reverse pass include Graph Neural Networks (GN Ns), Graph Attention Networks (GATs), or other suitable ML architectures designed for graph data. In particular, examples of machine-learning models that may be used in the reverse pass include:

[0200] - Graph Neural Networks (GNNs): For example, a 7-layer GNN (e.g. as described in [8] in the context of molecular generation for drug discovery) may encode graphs into a latent space, introduce noise, and learn to remove that noise to reconstruct the original structure.

[0201] - Graph Attention Network (GAT): A 7-layer GAT (e.g. as described in [9]) or other attention-based architectures can denoise graphs by predicting the node type and its connections to previously denoised nodes, using attentive message passing.

[0202] - Message Passing Neural Networks (MPNNs), e.g. as described in

[0010] in the context of drug discovery and circuit design, with the number of layers searched from {3, 5, 8}, MPNNs iteratively update node representations using local neighborhood information to reconstruct clean structures.

[0203] - Specialized Graph Transformers: For example, e.g. as described in

[0011] in the context of molecule generation, a transformer model with six encoding functions, four attention biases, and two attention channels may be used. The model takes a noisy graph as input and outputs a denoised graph. The attention mechanism enables joint reasoning over 2D connectivity and 3D geometry, allowing simultaneous recovery of graph structure and spatial arrangements.

[0204] Accordingly, at each timestep t, the machine-learning model is trained to predict either the original graph Go, or the noise that was added at step t, enabling it to iteratively denoise during generation.

[0205] Once this reverse process is learned, the thus trained machine-learning model can generate new graphs, e.g. starting from random noise or otherwise.

[0206] The graph creation performed by a thus trained machine-learning model can be iterative and also guided by user input such as: desired properties (e.g., color, specific elements), descriptive text prompts, sketches or structural hints, or any other guiding modality. To generate toy construction models that adhere to specific user-defined constraints or prompts, the reverse pass can be guided using various techniques. One approach is to train the denoising network (i.e. the reverse pass) to predict the original graph structure conditioned on specific target properties, as the desired attributes, e.g. using techniques described in

[0012] , This conditioning helps steer the denoising process toward configurations that satisfy the specified constraints.

[0207] References:

[0208] [1] Liu, Q., Allamanis, M., Brockschmidt, M., & Gaunt, A. (2018). Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31.

[0209] [2] Mohammadi, S., O’Dowd, B., Paulitz-Erdmann, C. & Goerlitz, L. (2019). Penalized Variational Autoencoder for Molecular Design. ChemRxiv.

[0210] [3] Simonovsky & M., Komodakis, N. (2018). GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders. 27th International Conference on Artificial Neural Networks (ICANN).

[0211] [4] Wang, H., Wang, J., Wang, J., Zhao, M., Zhang, W., Zhang, F., et al. (2018). GraphGAN: Graph representation learning with generative adversarial nets. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).

[0212] [5] Liu, J., Kumar, A., Ba, J., Kiros, J., & Swersky, K. (2019). Graph normalizing flows. Advances in Neural Information Processing Systems, 32.

[0213] [6] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851 .

[0214] [7] Liu, C., Fan, W., Liu, Y., Li, J., Li, H., Liu, H., Tang, J., & Li, Q. (2023). Generative Diffusion Models on Graphs: Methods and Applications. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 6702-6711. [8] Pombala, P., Grossmann, G., & Wolf, V. (2025). Exploring Molecule Generation Using Latent Space Graph Diffusion. arXiv preprint arXiv:2501.03696.

[0215] [9] Kong, L., Cui, J., Sun, H., Zhuang, Y., Prakash, B. A., & Zhang, C. (2023). Autoregressive diffusion model for graph generation. In International conference on machine learning (pp. 17391-17408).

[0216]

[0010] Xu, Z., Qiu, R., Chen, Y., Chen, H., Fan, X., Pan, M., ... & Tong, H. (2024). Discrete-state Continuous-time Diffusion for Graph Generation. arXiv preprint arXiv:2405.11416.

[0217]

[0011] Hua, C., Luan, S., Xu, M., Ying, Z., Fu, J., Ermon, S., & Precup, D. (2024, April). Mudiff: Unified diffusion for complete molecule generation. In Learning on Graphs Conference (pp. 33-1 ).

[0218]

[0012] Hoogeboom, E., Satorras, V. G., Vignac, C., & Welling, M. (2022). Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, 8867-8887.

[0219]

[0013] Mazeika, Stella: Hierarchical Style Modeling: A generative framework for Style-Centric Generation of 3D Models, 2019, dissertation Univ, of California, Santa Cruz, Advisor: Whitehead, Jim, available at

[0220]

[0014] Radford, A. et al., “Learning Transferable Visual Models From Natural Language Supervision”, arXiv e-prints, Art. no. arXiv:2103.00020, 2021. doi: 10.48550 / arXiv.2103.00020

Claims

Claims:

1. A computer-implemented method of creating a digital representation of a toy construction model, the method comprising:- receiving a user prompt to create a digital representation of a toy construction model, the user prompt including one or more desired attributes of the toy construction model,- using a generative machine-learning model to create, based on the user prompt, a structured representation of a toy construction model, the toy construction model being constructed from a set of mutually interconnected toy construction elements, the structured representation being indicative of the toy construction elements of said set and of the mutual interconnections between respective ones of said toy construction elements,- translating the created structured representation into a digital 2D or 3D representation of a visual appearance of the toy construction model and / or into a set of building instructions for creating the toy construction model.

2. The method according to claim 1 , wherein the one or more attributes include one or more attributes selected from:- an identity of one or more toy construction elements to be included in the toy construction model,- a target size or a target range of sizes of the toy construction model,- one or more colors or a color scheme of the toy construction model,- a target age or age range, or a target proficiency level or range of proficiency levels, of a user constructing the toy construction model,- an indication of a type of object represented by the toy construction model,- a target pool of toy construction elements from which the toy construction elements making up the toy construction model are to be selected,3. The method according to any one of the preceding claims, wherein using the generative machine-learning model to create the structured representation comprises using the generative machine-learning model to create the structured representation based on one or more constraints to be fulfilled by the toy construction model represented by the structured representation.

4. The method according to claim 3, wherein at least one of the one or more attributes is indicative of at least one of said constraints.

5. The method according to claim 3 or 4, wherein at least one of the one or more constraints is chosen from the group of constraints consisting of: a constraint on the size of the toy construction model, a constraint of the shape of the toy construction model, a constraint on the color of the toy construction model, a constraint on the number of toy construction elements making up the toy construction model, a constraint on the pool or set of toy construction elements that may be included in the toy construction model.

6. The method according to any one of the preceding claims, comprising:- receiving a user input indicative of a user-initiated modification to the created toy construction model,- using the generative machine-learning model to create, based at least on the structured representation of the created toy construction model and on the user input, a structured representation of a modified toy construction model,- translating the created structured representation of the modified toy construction model into a digital 2D or 3D visual representation of the modified toy construction model and / or into a set of building instructions for creating the modified toy construction model.

7. The method according to claim 6, further comprising iteratively training the generative machine-learning model based on the user-initiated modifications.

8. The method according to any one of the preceding claims, comprising: presenting a user with a plurality of alternative initial partial models, each initial partial model including one or more toy construction elements, receiving a user-input indicative of a user-selected one of the alternative initial partial models as a current selected partial model.

9. The method according to claim 8, wherein the method further comprises, repeatedly:- using the generative machine-learning model to create digital representations of a plurality of alternative additional model parts to be added to the selected partial model,- presenting the created digital representations of said plurality of alternative additional model parts,- receiving a user input indicative of a user-selected one of the alternative model parts to be added to the current partial model,- creating a structured representation of a new current partial model from the current partial model and the user-selected one of the alternative model parts.

10. The method according to any one of the preceding claims wherein translating the created structured representation comprises automatically translating the created structured representation into a digital 2D or 3D representation of a visual appearance of the toy construction model and / or into a set of building instructions for creating the toy construction model, or performing the translation as a semi-automatic, user-assisted translation process.

11. A computer-implemented method of training a generative machinelearning model for creating digital representations of toy construction models, the method comprising:- providing digital representations of a plurality of training toy construction models, each training toy construction model being constructed from a respective set of mutually interconnected toy construction elements,- creating structured representations of the training toy construction models from the digital representations, each of the structured representations being a representation of a respective one of the plurality of training toy construction model, the structured representations including a first structured representation of a first training toy construction model constructed from a first set of toy construction elements, the first structured representation being indicative of the toy construction elements of said first set and of the mutual interconnections between respective ones of said toy construction elements of said first set making up said first training toy construction model,- using the created graph representations as a training set for creating a generative machine-learning model trained to create new toy construction models.

12. The method according to claim 11 , comprising using a diffusion model to train the machine-learning model.

13. The method according to any one of the preceding claims, wherein the toy construction elements include coupling members for releasably connecting the toy construction elements with each other so as to form a modular spatial structure, and wherein the structured representation is indicative of which toy construction elements of said set are interconnectedwith each other by their respective coupling members when assembled to form the toy construction model.

14. The method according to any one of the preceding claims, wherein the structured representation is indicative of one or more element attributes of each toy construction element of the set of toy construction elements.

15. The method according to claim 14, wherein each element attribute includes at least a type of the respective toy construction element and / or a color of the toy construction element.

16. The method according to any one of the preceding claims, wherein the structured representation includes connection attributes of some or all of the represented interconnections.

17. The method according to claim 16, wherein each connection attribute includes at least a type of connection and / or a number and / or type of coupling members providing said connection.

18. The method according to any one of the preceding claims, where the structured representation comprises a graph representation, the graph representation being indicative of a graph comprising nodes and edges, each edge connecting respective nodes, each node representing a toy construction element and each edge representing a connection between two of said toy construction elements.

19. The method according to claim 18, when dependent on claim 14 or 15, wherein each node has associated with it one or more of said element attributes.

20. The method according to claim 18 or 19, when dependent on claim 16 or 17, wherein each edge has associated with it one or more of said connection attributes.

21. A computer-implemented method of creating a digital representation of a toy construction model, the method comprising, repeatedly:- presenting a visual representation of an intermediate toy construction model,- receiving a user input indicative of a user-selected modification to said intermediate toy construction model,- using a generative machine-learning model to create, based at least on the user input, a visual representation of at least a part of a modified intermediate toy construction model for presentation to the user as a modified intermediate toy construction model.

22. A data processing system comprising a processing unit and a display, wherein the data processing system is adapted to perform the steps of the method according to any one of the preceding claims.

23. A computer program product configured to be executed by a data processing system and, when executed by the data processing system, to cause the data processing system to perform the steps of the method according to any one of claims 1 through 21 .

24. A toy construction system, comprising a plurality of toy construction elements with coupling members for releasably interconnecting the toy construction elements with each other, wherein the toy construction system further comprises the computer program product of claim 23 and / or the data processing system of claim 22.

Images