Design code generation method, knowledge graph construction method and related device

Abstract

Disclosed in the embodiments of the present application is a cloud service-based design code generation method, the method being applied to a cloud management platform. The method can not only acquire from a configuration interface an explicit design intention input by a user, but also query a knowledge graph for target design knowledge associated with the current explicit design intention of the user so as to capture an implicit design intention of a target design object. Thus, the explicit design intention and the target design knowledge from the knowledge graph can be combined to guide a large language model in efficiently and accurately generating target design code that meets a design requirement of the user, thereby intelligently generating the target design code so as to accurately and efficiently implement a design task on the basis of the target design code to obtain the target design object.

Description

Design code generation methods, knowledge graph construction methods, and related equipment.

[0001] This application claims priority to Chinese patent application filed on October 21, 2024, with application number 202411471277.2, entitled "Design Code Generation Method, Knowledge Graph Construction Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of industrial design technology, specifically to design code generation methods, knowledge graph construction methods, and related equipment. Background Technology

[0003] In the industrial sector, traditional design work (such as the design and assembly of 3D models, and the design of user interfaces (UI)) has a high learning cost, requiring designers to have a certain level of computer skills and sufficient professional knowledge (for example, if the designed 3D model is applied to the engineering field, then professional knowledge in the engineering field is required). Therefore, the labor cost is high, the production cycle is long, and it is difficult to adjust and modify.

[0004] Currently, the introduction of large language models (LLMs) in the field of artificial intelligence (AI) has made intelligent design possible.

[0005] However, when using large language models to assist industrial design, since design involves higher-dimensional reasoning from natural language to 3D models, large language models have difficulty perceiving and understanding the user's design intent from the text prompts input by the user. Moreover, as a general model, large language models usually lack specific domain knowledge in the industrial field, which means that industrial design based on large language models can usually only generate very simple structures, and cannot realize the design and / or assembly tasks of complex structures.

[0006] It is evident that there is an urgent need for a method that can efficiently and accurately perform intelligent industrial design. Summary of the Invention

[0007] This application provides a cloud-based design code generation method that can conveniently and efficiently generate design code for users, enabling efficient implementation of design tasks such as 3D models and user interfaces. This application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0008] The first aspect of this application provides a design code generation method based on cloud services. This method is applied to a cloud management platform, which manages the infrastructure providing cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center. The cloud services run on at least one server located in at least one cloud data center across the multiple regions. The method includes: providing a configuration interface for obtaining user-inputted design intent information, the design intent indicating the generation of a target design object; retrieving target design knowledge corresponding to the design intent from a knowledge graph, the knowledge graph including one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, the target design knowledge describing the implicit design intent for the target design object; and processing the design intent and target design knowledge through a large language model to obtain target design code, the target design code being used to generate the target design object.

[0009] In the first aspect, not only can the explicit design intent of the user input be obtained from the configuration interface, but also the target design knowledge associated with the user's current explicit design intent can be queried from the knowledge graph in order to capture the implicit design intent of the target design object.

[0010] In this way, explicit design intent and target design knowledge from knowledge graphs can be combined to guide the large language model to generate target design code that meets the user's design needs efficiently and accurately. This achieves intelligent generation of target design code, enabling the design task to be completed accurately and efficiently based on the target design code, and the target design object to be obtained.

[0011] In one possible implementation of the first aspect, before retrieving design knowledge corresponding to the design intent from the knowledge graph, the method further includes: acquiring multimodal design materials, the multimodal design materials including a first design object, and the multimodal design materials further including one or more of the following: first text and a first image; obtaining, based on the first design object, a first geometric feature of a first component among one or more components included in the first design object, the first geometric feature being used to describe features related to geometric elements in the first component; obtaining design knowledge corresponding to the first design object based on the first geometric feature; after obtaining the design knowledge corresponding to the design materials of each modality, jointly representing the design knowledge corresponding to the design materials of each modality to obtain a target joint representation; and constructing a knowledge graph based on the target joint representation.

[0012] In traditional knowledge graph construction, knowledge is usually extracted only from text, and it is difficult to extract knowledge from other forms of materials.

[0013] In this possible implementation, design knowledge can be extracted from multimodal design materials. These multimodal design materials include a first design object, which can be considered a historical design object. After obtaining the first design object, design knowledge implicit in it can be mined based on geometric features and jointly represented with design knowledge mined from other modal design materials. For example, a joint representation of targets in vector form can be used to achieve the fusion of design knowledge from the multimodal design materials, thereby constructing a knowledge graph. Furthermore, since this knowledge graph can include design knowledge mined from the multimodal design materials, it contains relatively rich design knowledge, providing a good data foundation for related applications of knowledge graphs.

[0014] In one possible implementation of the first aspect, the target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object, and the target triple information includes the operation object, operation instructions, and operation parameters corresponding to the at least one design code module.

[0015] In this possible implementation, the at least one design code module can also be considered as a function. Each design code module can provide an interface for invocation. When invoking, the information of the triple of the operation object, operation instruction, and operation parameter corresponding to the design code module can be used as the input information of the interface, thereby instructing the design code module to implement the corresponding design based on the information of the operation object, operation instruction, and operation parameter corresponding to the design code module.

[0016] In one possible implementation of the first aspect, the design intent and target design knowledge are processed through a large language model to obtain the target design code, including: processing the design intent and target design knowledge through a large language model to obtain target module identifiers and target triple information, wherein the target module identifier includes a module identifier corresponding to at least one design code module; and obtaining the target design code through a large language model based on the target module identifier and the target triple information.

[0017] In this possible implementation, the design code generation task is executed hierarchically through at least two calls to the large language model, resulting in target design code that can make interface calls to at least one design code module. This makes the logic for generating the design code clearer, thereby ensuring the accuracy of the target design code.

[0018] In one possible implementation of the first aspect, the design intent includes an intent derived from a second design object, and further includes: obtaining a second geometric feature of a second component associated with a target design object among one or more components included in the second design object, the second geometric feature being used to describe features related to geometric elements in the second component; and obtaining the intent derived from the second design object based on the second geometric feature.

[0019] In this possible implementation, for example, the geometric elements may include primitives in a two-dimensional image or geometric elements in a three-dimensional image. The primitives in the two-dimensional image may include one or more of points, lines, surfaces, etc., while the geometric elements in the three-dimensional image may include spheres or basic geometric solids, etc. The second geometric feature may include the features of the geometric elements themselves in the second component, or it may include structural features related to the structure formed by the geometric elements in the second component; furthermore, the second geometric feature may be described through relevant parameter data of the geometric elements, or it may be described through semantic form.

[0020] In one possible implementation of the first aspect, obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object includes: decomposing the second design object to obtain one or more levels of components in the second design object; obtaining the component of the target level from the one or more levels of components in the second design object as the second component; and obtaining the second geometric feature of the second component.

[0021] In this possible implementation, after decomposing the first design object, the topological hierarchy of each level of the decomposed components can be described using a directed acyclic graph (DAG). After obtaining components at one or more levels, the components corresponding to the lowest level (i.e., the leaf nodes in the DAG) can be assembled from bottom to top to obtain the first design object. Therefore, the components corresponding to the leaf nodes can describe the characteristics of the first design object, and the components corresponding to the leaf nodes are usually in the form of relatively common parts in the design field. Therefore, the components corresponding to the leaf nodes can be used as the first components.

[0022] In one possible implementation of the first aspect, the design intent further includes an intent derived from the second text. Obtaining an intent derived from the second design object based on the second geometric feature includes: jointly representing the design information in the second geometric feature and the second text to obtain a first joint representation; and obtaining a first joint representation intent based on the first joint representation, wherein the first joint representation intent includes the intent derived from the second design object and the intent derived from the second text.

[0023] In this possible implementation, during joint representation, the design information corresponding to the second design object (e.g., the second geometric features of the second design object) can be feature-encoded (embedding) to convert it into vector form. Similarly, the design information corresponding to the second text (e.g., entities extracted from the second text, relationships between entities, entity attributes, etc.) can also be feature-encoded and converted into vector form. This ensures that the design information corresponding to the second design object and the design information corresponding to the second text are mapped to the same feature space and described using the same format. In this way, the design information corresponding to the second design object and the design information corresponding to the second text can be represented in the same feature space, thus achieving joint representation of the design information corresponding to the second design object and the design information corresponding to the second text.

[0024] In one possible implementation of the first aspect, the design intent further includes an intent derived from the second image. The method further includes: identifying design information in the second image through object detection and / or semantic segmentation, the design information in the second image including one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; obtaining an intent derived from the second design object based on second geometric features, including: jointly representing the second geometric features and the design information in the second image to obtain a second joint representation; and obtaining a second joint representation intent based on the second joint representation, the second joint representation intent including the intent derived from the second design object and the intent derived from the second image.

[0025] In this possible implementation, during joint representation, the design information corresponding to the second design object (e.g., the second geometric features of the second design object) can be feature-encoded (embedding) to convert it into vector form. Similarly, the design information corresponding to the second image (e.g., entities extracted from the second image, relationships between entities, entity attributes, etc.) can also be feature-encoded and converted into vector form. This ensures that the design information corresponding to the second design object and the design information corresponding to the second image are mapped to the same feature space and described using the same form. In this way, the design information corresponding to the second design object and the design information corresponding to the second image can be represented in the same feature space, thus achieving joint representation of the design information corresponding to the second design object and the design information corresponding to the second image.

[0026] In one possible implementation of the first aspect, the second geometric feature includes one or more of the following: spatial information of geometric elements in the second component, semantic information for describing the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0027] In one possible implementation of the first aspect, the design intent includes an intent derived from a second image, and the method further includes: identifying design information in the second image through object detection and / or semantic segmentation, the design information in the second image including one or more of the following: entities in the second image, relationships between entities in the second image, attributes of entities in the second image; and obtaining the first image intent based on the design information in the second image.

[0028] In this possible implementation, computer vision and other methods can be used to extract the user's design intent from the second image, so as to better instruct the large language model to obtain the target design code that meets the user's design needs.

[0029] In one possible implementation of the first aspect, the design intent further includes the intent obtained based on the second text. Obtaining the first image intent based on the design information in the second image includes: performing a joint representation on the design information in the second text and the design information in the second image to obtain a third joint representation; obtaining a third joint representation intent based on the third joint representation, wherein the third joint representation intent includes the intent obtained based on the second text and the intent obtained based on the second image.

[0030] In one possible implementation of the first aspect, the method further includes: when the target design code is invoked to design the first object, if the target design code instructs to operate on a target component in the first object that does not belong to the preset component database, then based on the component parameters of the target component in the target design code, a target preset component whose similarity to the target component meets the specified condition is retrieved from multiple preset components in the preset component database; shape fitting is performed according to the target preset component and the component parameters of the target component to generate a target component, so as to operate on the target component according to the target design code.

[0031] In one possible implementation, for example, the target preset component can be used as a reference for the target component. If the target preset component and the target component have the same topological structure, the difference in dimensions between the target component and the target preset component can be obtained based on the topological structure. Based on the difference and related triplet information, shape fitting is performed to obtain the target component that meets the shape requirements in the target design code as the component required to generate the target design object.

[0032] In one possible implementation of the first aspect, after obtaining the target design code, the method further includes: obtaining the target design object obtained by calling the target design code; querying the global constraint information corresponding to the target design object from the knowledge graph; and optimizing the target design object based on the global constraint information.

[0033] In this possible implementation, the target design code typically instructs a bottom-up, local-to-global design process, resulting in a target design object usually composed of multiple components. Therefore, to ensure the rationality of the target design object, a large language model can query the knowledge graph for the global constraint information corresponding to the target design object. Then, the large language model can be used to check whether the target design object satisfies the queried global constraints. If not, the target design object can be adjusted based on the global constraints to obtain an optimized target design object.

[0034] A second aspect of this application provides a knowledge graph construction method applied to a cloud management platform. The cloud management platform manages infrastructure providing cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center. The cloud services run on at least one server located in at least one cloud data center across the multiple regions. The method includes: acquiring multimodal design materials, which include a first design object and further include one or more of the following: first text and a first image; obtaining a first geometric feature of a first component among one or more components included in the first design object, the first geometric feature describing features related to geometric elements in the first component; obtaining design knowledge corresponding to the first design object based on the first geometric feature; after obtaining the design knowledge corresponding to each modality's design materials, jointly representing the design knowledge corresponding to each modality's design materials to obtain a target joint representation; and constructing a knowledge graph based on the target joint representation.

[0035] In traditional knowledge graph construction, knowledge is usually extracted only from text, and it is difficult to extract knowledge from other forms of materials.

[0036] In the second aspect, design knowledge can be extracted from multimodal design materials. These multimodal design materials include a first design object, which can be considered a historical design object. After obtaining the first design object, design knowledge implicit in it can be mined based on geometric features and jointly represented with design knowledge mined from other modal design materials. For example, a joint representation of targets in vector form can be used to achieve the fusion of design knowledge from multimodal design materials, thereby constructing a knowledge graph. Furthermore, since this knowledge graph can include design knowledge mined from multimodal design materials, it contains relatively rich design knowledge, providing a good data foundation for related applications of knowledge graphs.

[0037] In one possible implementation of the second aspect, the method further includes: providing a configuration interface for obtaining user-inputted design intent information, the design intent being used to instruct the generation of a target design object; retrieving target design knowledge corresponding to the design intent from a knowledge graph, the target design knowledge being used to describe the implicit design intent for the target design object; and processing the design intent and target design knowledge through a large language model to obtain target design code, the target design code being used to generate the target design object.

[0038] In one possible implementation of the second aspect, there are multiple first design objects. The design knowledge corresponding to the first design objects is obtained based on the first geometric features, including: determining the target first geometric feature from the first geometric features corresponding to the multiple first design objects based on the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects; and obtaining the design knowledge corresponding to the first design objects based on the target first geometric feature.

[0039] In this possible implementation, since a single first design object is an independent design object, it may contain some personalized or inappropriate designs. Therefore, based on the co-occurrence relationship, the more commonly used target first geometric features can be selected from multiple first design objects to serve as design knowledge in the knowledge graph, thereby ensuring the reliability of the design knowledge in the knowledge graph.

[0040] In one possible implementation of the second aspect, the weight of the design knowledge corresponding to the first geometric feature of the target in the knowledge graph is determined based on the co-occurrence probability of the first geometric feature of the target in the co-occurrence relation.

[0041] In this possible implementation, weights can be assigned to the design knowledge from the first design object (i.e., the design knowledge corresponding to the first geometric feature of the target) within the knowledge graph. The weights of the design knowledge corresponding to the first geometric feature of the target can also be considered as the confidence level of that design knowledge, reflecting its credibility.

[0042] In one possible implementation of the second aspect, the knowledge graph includes a first subgraph and at least one second subgraph. The first subgraph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in the multiple first design objects satisfies a specified condition. Any second subgraph is obtained based on one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

[0043] In this possible implementation, the knowledge graph can take the form of 1+n. Here, "1" indicates that the knowledge graph includes a subgraph describing common design knowledge (i.e., the first subgraph), while "n" indicates that the knowledge graph includes n subgraphs containing design knowledge extracted from independent historical design objects (i.e., n second subgraphs), where n can be a positive integer, that is, the number of second subgraphs can be at least one.

[0044] It is evident that the first and second sub-graphs enable the separate storage, application, updating, and management of design knowledge from different sources and with different levels of confidence.

[0045] In one possible implementation of the second aspect, the first geometric feature includes one or more of the following: spatial information of geometric elements in the first component, semantic information for describing the first component and / or geometric elements in the first component, structural features of the first component, and gradient features of the first component.

[0046] In one possible implementation of the second aspect, obtaining the first geometric feature of a first component among one or more components included in the first design object, based on the first design object, includes: decomposing the first design object to obtain one or more levels of components in the first design object; obtaining a component of a target level from the one or more levels of components in the first design object as the first component; and obtaining the first geometric feature of the first component.

[0047] A third aspect of this application provides a cloud-based design code generation apparatus, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, such as an interface module and a processing module.

[0048] A fourth aspect of this application provides a computing device cluster including at least one computing device, the at least one computing device including a processor and a memory, the memory of the at least one computing device storing computer-executable instructions that can run on the processor, and when the computer-executable instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0049] The fifth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0050] The sixth aspect of this application provides a computer program product that stores one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0051] A seventh aspect of this application provides a chip system including a processor for supporting the processor in implementing the functions involved in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0052] An eighth aspect of this application provides a knowledge graph construction apparatus, which has the function of implementing the method of the second aspect or any possible implementation of the second aspect described above. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, such as an interface module and a processing module.

[0053] The ninth aspect of this application provides a computing device cluster including at least one computing device, the at least one computing device including a processor and a memory, the memory of the at least one computing device storing computer-executable instructions that can run on the processor, and when the computer-executable instructions are executed by the processor, the processor executes a method as described in the second aspect above or any possible implementation of the second aspect.

[0054] The tenth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the second aspect above or any possible implementation thereof.

[0055] The eleventh aspect of this application provides a computer program product that stores one or more computer execution instructions, wherein when the computer execution instructions are executed by a processor, the processor executes a method as described in the second aspect above or any possible implementation thereof.

[0056] The twelfth aspect of this application provides a chip system including a processor for supporting the processor in implementing the functions involved in the second aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0057] The technical effects of the third to seventh aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and the technical effects of the eighth to twelfth aspects or any of their possible implementations can be found in the second aspect or the related possible implementations of the second aspect, and will not be repeated here. Attached Figure Description

[0058] Figure 1 is a schematic diagram of an exemplary framework of a cloud data center provided in an embodiment of this application;

[0059] Figure 2 is an exemplary schematic diagram of the cloud service provided in an embodiment of this application;

[0060] Figure 3 is an exemplary flowchart provided in an embodiment of this application;

[0061] Figure 4 is an exemplary schematic diagram of knowledge graph construction provided in an embodiment of this application;

[0062] Figure 5 is an exemplary schematic diagram of a knowledge graph construction method provided in an embodiment of this application;

[0063] Figure 6 is an exemplary structural diagram of a multi-agent framework provided in an embodiment of this application;

[0064] Figure 7 is an exemplary schematic diagram of the design code generation method provided in an embodiment of this application;

[0065] Figure 8 is an exemplary flowchart provided in an embodiment of this application;

[0066] Figure 9 is an exemplary information interaction diagram provided in an embodiment of this application;

[0067] Figure 10 is another exemplary information interaction diagram provided in an embodiment of this application;

[0068] Figure 11 is an exemplary schematic diagram of a cloud service-based design code generation device provided in an embodiment of this application;

[0069] Figure 12 is an exemplary schematic diagram of a knowledge graph construction apparatus provided in an embodiment of this application;

[0070] Figure 13 is a structural schematic diagram of a computing device provided in an embodiment of this application;

[0071] Figure 14 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0072] Figure 15 is a schematic diagram of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0073] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0074] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0075] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatus.

[0076] 1. Large Language Model (LLM)

[0077] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0078] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0079] 2. Graph model

[0080] A graph model is a graph consisting of nodes and edges used to describe a system.

[0081] In this embodiment of the application, the graph model may specifically include a directed acyclic graph (DAG) and an undirected acyclic graph.

[0082] 3. Decomposition of Directed Acyclic Graph (DAG)

[0083] Directed Acyclic Graph (DAG) decomposition is a task decomposition method based on the concept of a directed acyclic graph. A DAG is a directed graph without cycles, meaning that starting from a vertex, it is impossible to return to that vertex via a series of edges. This graph structure is well-suited for representing and solving problems with sequential or conditional dependencies, such as task scheduling and the shortest path problem in algorithms.

[0084] In Directed Acyclic Graph (DAG) decomposition, a complex task is broken down into multiple independent subtasks, each of which can be executed independently. This decomposition process first requires determining the in-degree (the number of directed edges entering a vertex) of each vertex in the graph. Then, vertices with an in-degree of 0 are assigned to the top layer or the first layer. Next, vertices with an in-degree of 0 are found in the subgraph after removing vertices from the previous layer and assigned to the next layer, and so on, until all vertices are assigned to different layers. Each layer can be executed independently, thus achieving parallel processing of the task.

[0085] 4. Knowledge Graph

[0086] A knowledge graph is a graph-based data structure and a form of semantic network used to display entities and the relationships between them. A knowledge graph consists of nodes and edges. Nodes represent entities, such as people, place names, and companies, and may also correspond to entity attributes. Edges represent relationships between entities. Data in a knowledge graph can be organized using tuples, which can include triples, quadruples, or quintuples. Triples are represented in formats such as "node-edge-node" and "node-attribute name-attribute value."

[0087] This application provides a cloud service-based design code generation method that can conveniently and efficiently generate design code for users, enabling them to efficiently complete design tasks such as 3D models and user interfaces, thereby achieving intelligent industrial design.

[0088] The embodiments of this application can be applied to cloud management platforms.

[0089] The cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center, and the cloud services run on at least one server located in at least one cloud data center in the multiple regions.

[0090] The following is an exemplary description of a cloud data center, illustrated in Figure 1.

[0091] In Figure 1, the cloud management platform interacts with one or more servers (Server 1 and Server 2 in Figure 1) through the data center's internal network. The servers consist of a hardware layer and a software layer. The hardware layer includes the server's hardware configuration, such as PCI devices like network interface cards (NICs), graphics processing units (GPUs), and offloading cards, which can be plugged into peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) slots. The software layer includes the operating system installed and running on the server (the operating system relative to the virtual machine can be called the host operating system). The host operating system contains a virtual machine manager (also called a hypervisor), whose role is to implement compute virtualization, network virtualization, and storage virtualization of the virtual machines and to manage them. A virtual machine (VM) refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. In the system architecture shown in Figure 1, the infrastructure includes multiple servers, which can be used to run virtual machines. The specifications of the virtual machines can be the same or different. Virtual machines can also be called cloud servers (elastic compute service, ECS), elastic instances, etc., and different cloud service providers may have different names for them.

[0092] In one example of an embodiment of this application, the cloud management platform can be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities can provide cloud services to users. Users obtain cloud services through the Internet but do not own cloud computing resources. In other embodiments of this application, the cloud management platform can be a private cloud platform or a hybrid cloud platform, and this application does not impose any restrictions on this.

[0093] Specifically, in the example shown in Figure 1, the cloud management platform can provide access interfaces (such as user interfaces or application programming interfaces, APIs). Users of the cloud management platform and cloud service providers can use these interfaces to remotely register cloud accounts and passwords on the platform. After successful authentication of the cloud account and password, they can log in to the cloud management platform to create, manage, log in to, and operate virtual machines in the cloud data center. Therefore, enterprises, organizations, or individuals can purchase or rent cloud services through the cloud management platform, and thus perform specified tasks using the cloud services provided by the platform's cloud resources.

[0094] For example, as shown in Figure 2, the cloud services provided by the cloud management platform may include design code generation services. In addition, in some examples, design scheme generation services may also be included.

[0095] For example, in some examples shown in Figure 2, users can send design intent information to the cloud management platform to instruct the platform to perform design code generation tasks. The platform then intelligently generates the target design code required for the design task based on knowledge graphs and large language models and feeds it back to the user. This allows the user to perform design tasks on the target design object (such as styling tasks for 3D models, assembly tasks, or user interface design tasks) based on the target design code and obtain the target design object as the design result.

[0096] The target design object, also known as the target design instance, is a design object generated during the design task. A single target design object can be called a single target design instance. For example, it could be a 3D model completing the styling design, a 3D model completing the assembly design, or a user interface completing the 2D design. Therefore, the specific types of target design objects can be varied and can be determined based on the actual application scenario.

[0097] Alternatively, in some examples shown in Figure 2, a user can send design intent information to the cloud management platform, instructing it to execute a design code generation task. This task intelligently generates the target design code required for the design task based on a knowledge graph and a large language model. The design code generation service can then pass this target design code to the cloud management platform's design solution generation service. Next, the design solution generation service can invoke the target design code to perform a design task on the target design object (e.g., a 3D model design task, an assembly task, or a user interface design task) to obtain the target design object as the design result (e.g., the designed 3D model or user interface), and then output the target design object to the user from the cloud management platform.

[0098] It should be noted that the services shown in Figure 2 are only one example of the services provided by the cloud management platform, and not a limitation.

[0099] In other examples provided in this application, the aforementioned design code generation service and design scheme generation service can be deployed as a whole on a cloud management platform; furthermore, the functional division of each service may differ from that shown in Figure 2, and the deployment methods of each service may also differ. Each service can be provided independently, embedded in other services, or multiple services can be combined for deployment; this application does not impose any restrictions on this.

[0100] Based on the cloud management platform described above, and referring to the system architecture shown in Figure 2, as shown in Figure 3, the method of this application embodiment may include one or more of the following aspects:

[0101] Knowledge graph construction, enhancement and constraint of large language model for design code generation tasks, design code generation based on knowledge graph and large language model, calling the generated design code for design, and optimization of design results.

[0102] The above aspects will be described in detail below.

[0103] I. Knowledge Graph Construction

[0104] The knowledge graph construction in this application embodiment may include one or more of the following aspects:

[0105] The construction of the knowledge layer (schema) of the knowledge graph, the mining of knowledge about design materials, the joint representation of design knowledge from multimodal design materials, and knowledge fusion.

[0106] The following provides an illustrative overview of several aspects that may be involved in the construction of knowledge graphs.

[0107] 1. Construction of the data model (schema) of knowledge graph

[0108] A knowledge graph can be considered a structured semantic knowledge base, which may include a data model schema and an instance layer. The schema, also known as a data pattern, can be considered a data model for a specific domain (e.g., the design domain in this embodiment), containing meaningful concept types and their attributes within that domain. The schema for this specific domain is primarily expressed by types and properties. The instance layer includes instance data; for example, the instance data corresponding to the entity "name" in the knowledge graph includes Zhang San, Li Si, etc.

[0109] In this embodiment of the application, the specific method of constructing the schema is not limited, and existing or future schema construction methods can be used.

[0110] For example, a schema can be constructed based on expert experience. For instance, in the example shown in Figure 4, for the relevant application domain (i.e., the design domain in this embodiment), experts predict the schema from top to bottom based on the design objects within the design domain, then summarize the schema from bottom to top, and finally fuse information to construct the schema. For example, pre-defined texts within the design domain can be collected and organized. These pre-defined texts typically have a small data volume to facilitate expert summarization and information extraction. Then, the characteristics of entities, relationships, and attributes in the pre-defined texts are summarized and generalized to extract the relevant data framework for the design domain, forming a definite schema.

[0111] After the schema is constructed, information can be extracted from the design materials in a targeted manner based on the constructed schema (for example, extracting information such as entities, relationships, and attributes that meet the requirements of the schema), thereby constructing a knowledge graph.

[0112] 2. Knowledge mining of design materials

[0113] In this embodiment of the application, the design material can be one or more types; in other words, the design material can be a single-modal design material or a multi-modal design material.

[0114] For example, design materials may include one or more of the following: first text, first design object, and first image.

[0115] The following provides an exemplary introduction to knowledge mining of design materials for any of the above modalities, and also provides an exemplary introduction to possible joint characterization schemes among design materials of multiple modalities.

[0116] (1) Mining explicit knowledge carried by documents

[0117] In this embodiment of the application, design knowledge carried by documents can be considered as explicit knowledge.

[0118] Explicit knowledge that is documented can include unstructured documents, semi-structured documents, and structured documents.

[0119] In one example, a structured document may include diagrams, flowcharts, and other graphical structures, as well as / or tables and other structured formats. Because the data structure in a structured document is clear, it is easy to extract structured information such as entities, relationships, and attributes from it, which can then be used as design knowledge.

[0120] The following sections provide exemplary descriptions of knowledge mining of the first image and the first text that may be contained in unstructured or semi-structured documents.

[0121] In one example, an unstructured or semi-structured document may include, but is not limited to, a first image.

[0122] The first image may include one or more images, and the data type of the first image can be varied and is not limited here. For example, the first image may be in bitmap format (e.g., bitmap (BMP), portable network graphics (PNG), joint photographic experts group (JPEG), etc.), vector graphics format, etc.

[0123] The following describes an exemplary method for obtaining design knowledge from the first image.

[0124] In some embodiments, the design material includes a first image, and the specific methods for obtaining the design knowledge corresponding to the first image include:

[0125] Design knowledge in a first image is identified through object detection and / or semantic segmentation. The design knowledge in the first image includes one or more of the following: entities in the first image, relationships between entities in the first image, and attributes of entities in the first image.

[0126] In this embodiment of the application, referring to the example shown in FIG4, design information recognition can be performed on the first image through object detection and / or semantic segmentation in computer vision (CV). For example, entities in the first image may include one or more targets detected in the first image; and the relationships between entities can be determined based on their relative positions, etc. Furthermore, the attributes of entities in the first image may include one or more of the following information about the entities in the first image: position, size (e.g., described by a bounding box such as the entity's minimum bounding rectangle), and orientation.

[0127] For example, object detection can be used to identify information such as the category and location of entities in the first image. There are various specific methods for object detection, which are not limited here. For instance, object detection can be achieved using machine learning models capable of object detection (such as convolutional neural networks (CNN), support vector machines (SVM), etc.) or traditional object detection methods (such as scale-invariant feature transform (SIFT) and random sample consensus (RANSAC)).

[0128] Furthermore, semantic segmentation can identify the category to which each pixel in the first image belongs. In other words, semantic segmentation can divide the first image into one or more regions and obtain the semantic category to which each of these regions belongs. Therefore, based on the semantic categories identified by semantic segmentation and the segmented regions, entities contained in the first image can be identified. Furthermore, relationships between entities can be determined based on their relative positions, and entity attributes can be determined based on their positions and region information.

[0129] As can be seen, in the embodiments of this application, design knowledge such as entities in the first image, relationships between entities, and / or attributes of entities can be identified through object detection and / or semantic segmentation.

[0130] In one example, an unstructured or semi-structured document may contain, but is not limited to, the first text.

[0131] The following describes an exemplary way to obtain design knowledge from the first text.

[0132] In this embodiment, named entity recognition can be performed on the first text to identify the named entities in the first text. Then, information such as the relationships between named entities and the attributes of named entities can also be identified from the first text to obtain the entities, the relationships between entities, and / or the attributes of entities in the first text as design knowledge of the first text.

[0133] There are various methods for named entity recognition, as well as ways to identify the relationships between named entities and the attributes of named entities.

[0134] For example, in some cases, the large language model to be fine-tuned can be fine-tuned so that the fine-tuned large language model can recognize the design information of the first text.

[0135] Specifically, in some embodiments, the method further includes:

[0136] Based on the preset text and its labels, the large language model to be fine-tuned is fine-tuned to obtain a large language model for text design information recognition after fine-tuning. The labels of the preset text include one or more of the following information: entities in the preset text, relationships between entities in the preset text, and attributes of entities in the preset text.

[0137] By using a large language model for identifying design information in text, information is extracted from the first text to obtain the design knowledge corresponding to the first text.

[0138] Specifically, in the example shown in Figure 4, the preset text may include small sample data from the design domain for fully supervised learning. This small sample data may have corresponding labels, which are used to describe the entities in the small sample data.

[0139] In this way, the first fine-tuning of the large language model to be fine-tuned can be achieved based on the small sample data and its labels, so that the large language model after the first fine-tuning can recognize named entities in the text of the design domain.

[0140] Then, weakly supervised data can be obtained. Some data in this weakly supervised data may have corresponding labels, which describe the relationships and / or attributes of named entities within that portion of the data. Based on this weakly supervised data and the labels corresponding to some of the data, the large language model, after its first fine-tuning, can be fine-tuned a second time. This second-fine-tuned large language model can then extract information such as entities, entity relationships, and entity attributes from the text. This second-fine-tuned large language model can be considered a large language model for identifying design information in text. Therefore, when constructing a knowledge graph, this large language model for identifying design information in text can be used to identify design information in the first text, thereby obtaining the design knowledge within the first text.

[0141] It is understood that in this embodiment, the large language model used for identifying design information in text differs from the fine-tuning operation involved in the large language model used in generating design code. Therefore, the large language model used for identifying design information in text may differ from the large language model used in generating design code. For example, the large language model used in generating design code may be obtained by further fine-tuning the large language model used for identifying design information in text, or it may be obtained by fine-tuning a large language model to be fine-tuned in other cases.

[0142] Alternatively, in other examples, other existing or future recognition methods can be used to identify design information in the first text.

[0143] (2) Mining tacit knowledge without document support

[0144] In this embodiment of the application, design knowledge that is not documented can be considered as tacit knowledge.

[0145] In real-world design scenarios, there are often a large number of historical design objects. These historical design objects can also be considered historical design instances, that is, design objects obtained from historical design tasks. For example, in a 3D design scenario, this historical design object (i.e., a historical design instance) can be a 3D model obtained from a historical design, while in a user interface design scenario, this historical design object can be considered a user interface obtained from a historical design.

[0146] A large number of historical design objects typically contain a wealth of design knowledge. This knowledge can include personalized design knowledge within specific design scenarios, as well as common design knowledge derived from shared design specifications and / or constraints. However, traditional knowledge graph construction often relies solely on textual extraction, making it difficult to extract knowledge from other forms of material.

[0147] In this embodiment of the application, implicit knowledge can be mined from historical design objects to enrich the knowledge in the knowledge graph and fully explore the design knowledge in the design field.

[0148] The following section uses the first design object in the historical design objects as an example to introduce an exemplary method for mining tacit knowledge from historical design objects.

[0149] Specifically, in some embodiments, the design material includes a first design object, and the method includes:

[0150] Based on the first design object, obtain the first geometric feature of the first component among one or more components included in the first design object, the first geometric feature being used to describe the features related to the geometric elements in the first component;

[0151] Based on the first geometric feature, obtain the design knowledge corresponding to the first design object.

[0152] The first design object, also known as the first design instance, can be considered a historical design object. It is a design object generated in a historical design task, such as a 3D model that completes the shape design, a 3D model that completes the assembly design, or a user interface that completes the graphic design.

[0153] In this embodiment of the application, the first geometric feature of the first component in the first design object can be defined.

[0154] The first geometric feature describes features related to geometric elements in the first component. This geometric element can refer to a fundamental, indivisible object or entity existing in space. For example, the geometric element can include primitives in a two-dimensional image or geometric elements in a three-dimensional image. Primitives in a two-dimensional image can include one or more of points, lines, surfaces, etc., while geometric elements in a three-dimensional image can include spheres or basic geometric solids, etc.

[0155] The first geometric feature may include the features of the geometric elements themselves in the first component, or the features of the structure composed of the geometric elements in the first component; and the first geometric feature may be described by the relevant parameter data of the geometric elements, or it may be described by semantic form.

[0156] For example, the first geometric feature includes one or more of the following:

[0157] Spatial information of geometric elements in the first component, semantic information used to describe the first component and / or geometric elements in the first component, structural features of the first component, and gradient features of the first component.

[0158] The spatial information of the geometric elements in the first component may include one or more of the following: position, orientation, size, scaling, etc. Semantic information describing the first component and / or its geometric elements may describe the semantics of the first component and / or its geometric elements, such as the name, shape, and meaning of the first component and / or its geometric elements. The structural features of the first component may include its outline and / or the characteristics of its internal structure, such as shape information like symmetry, and relationships between geometric elements (e.g., parallelism, perpendicularity, intersection). The gradient information of the first component, also known as its sharpness information, is described based on the magnitude of its normal gradient.

[0159] As can be seen, the first geometric features of the first component can describe the first component from one or more levels, from the most basic geometric elements of the first component to the structure composed of geometric elements.

[0160] There are several ways to identify the first geometric features of the first component. For example, the first geometric features of the first component can be identified by machine learning models such as convolutional neural networks (CNNs), or by algorithms such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), and features from accelerated segment test (FAST).

[0161] The first component can also have multiple forms.

[0162] For example, when the first design object is relatively simple, such as including only one component or several components that are simply assembled, the one component or several components that are simply assembled can be used as the first component.

[0163] Alternatively, the first design object can be decomposed to obtain some or all of the components in the first design object as the first component.

[0164] Specifically, in some embodiments, obtaining a first geometric feature of a first component among one or more components included in the first design object, based on the first design object, includes:

[0165] Decompose the first design object to obtain one or more levels of components within the first design object;

[0166] Obtain the component of the target level from one or more levels of components in the first design object as the first component;

[0167] Obtain the first geometric feature of the first component.

[0168] In this embodiment of the application, referring to the example shown in FIG4, the first design object can be decomposed from top to bottom to obtain one or more levels of components of the first design object. The lowest level of the components in the one or more levels can usually be parts, such as bolts, nuts and other structures, so as to decompose the first design object into components that are more common in the design field, have common features and are easy to describe using design knowledge in the design field.

[0169] In this process, after decomposing the first design object, the topological hierarchy of each level of the decomposed components can be described by DAG or similar methods. Furthermore, the DAG can be optimized by graph neural networks (GNNs) to obtain one or more levels of components in the first design object after optimization.

[0170] After obtaining components at one or more levels, the component at the lowest level can be used as the component at the target level, thus obtaining the component at the lowest level (i.e., the leaf node in the DAG) as the first component. Specifically, referring to the example shown in Figure 4, the component corresponding to the leaf node in the DAG can be obtained as the first component. It can be understood that the component corresponding to the leaf node can be assembled from bottom to top to obtain the first design object. Therefore, the component corresponding to the leaf node can describe the characteristics of the first design object. Furthermore, the component corresponding to the leaf node is usually in the form of a relatively common component in the design field. Therefore, the component corresponding to the leaf node can be used as the first component to obtain the first geometric features of the component corresponding to the leaf node to obtain the design knowledge in the first design object.

[0171] After obtaining the first geometric features of the first component, entities can be obtained based on the first component, and attributes of the first component can be obtained based on the first geometric features of the first component. Information such as the relationships between the first components and / or the relationships between geometric elements or geometric bodies in the first component can also be obtained to construct a knowledge graph.

[0172] Furthermore, in some embodiments, since a single first design object is an independent design object, it may contain some personalized or inappropriate designs. Therefore, commonly used target first geometric features can be selected from multiple first design objects to serve as design knowledge in the knowledge graph, thereby ensuring the reliability of the design knowledge in the knowledge graph.

[0173] Specifically, in some embodiments, the number of first design objects is multiple, and design knowledge corresponding to the first design objects is obtained based on first geometric features, including:

[0174] Based on the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects, the target first geometric feature is determined from the first geometric features corresponding to the multiple first design objects.

[0175] Based on the first geometric features of the target, obtain the design knowledge corresponding to the first design object.

[0176] Co-occurrence refers to the frequency or probability of two things appearing in the same context.

[0177] In this embodiment, the co-occurrence relationship of a certain first geometric feature can be the frequency or probability of the first geometric feature and the first design object appearing simultaneously in multiple first design objects. In some examples, the co-occurrence relationship of a certain first geometric feature can also be the frequency or probability of the first geometric feature and the first design object appearing simultaneously in multiple first design objects of the same type. It can be seen that the strength of the co-occurrence relationship can be reflected by the co-occurrence probability. If the co-occurrence relationship indicates that the probability (i.e., the co-occurrence probability) of a certain first geometric feature and the first design object appearing simultaneously is high (e.g., higher than a specified probability threshold), then it can be considered that the first design object is more likely to contain the first geometric feature. The first geometric feature can be considered to be a commonly used geometric feature in the first design object. It can be determined that the first geometric feature is the target first geometric feature among the first geometric features contained in the corresponding first design object. Therefore, the design knowledge corresponding to the target first geometric feature in the first design object can be regarded as the more reliable design knowledge corresponding to the first design object.

[0178] In some embodiments, in a knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined based on the co-occurrence probability of the first geometric feature of the target in the co-occurrence relationship.

[0179] In this embodiment, the weight can be determined based on the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship. For example, the weight can be equal to the corresponding co-occurrence probability, or it can be calculated by calculating the corresponding co-occurrence probability according to a specified calculation method. The co-occurrence probability corresponding to the target first geometric feature refers to the probability among multiple first design objects that contain the target first geometric feature.

[0180] As can be seen, in a knowledge graph, weights can be assigned to the design knowledge from the first design object (i.e., the design knowledge corresponding to the first geometric feature of the target). The weight of the design knowledge corresponding to the first geometric feature of the target can also be regarded as the confidence level of the design knowledge corresponding to the first geometric feature of the target, which can reflect the credibility of the design knowledge corresponding to the first geometric feature of the target.

[0181] Furthermore, as design tasks are continuously executed, new first design objects can be continuously generated. Based on these new first design objects, the design knowledge and its weights corresponding to the target first geometric features in the knowledge graph can be continuously updated, thus improving the design knowledge in the knowledge graph and enhancing its reliability. This provides a better data foundation for users to execute design tasks using the design knowledge from the knowledge graph.

[0182] 3. Joint representation of design knowledge from multimodal design materials

[0183] In this application embodiment, one or more modal design materials can be obtained. Based on any of the above-mentioned knowledge mining embodiments, after obtaining the design knowledge corresponding to each design material, a knowledge graph can be constructed according to the design knowledge corresponding to each design material.

[0184] In some examples, the design material can be a multimodal design material, in which case a knowledge graph can be constructed through the joint representation of design knowledge from the multimodal design material.

[0185] Specifically, referring to the example shown in Figure 5, a knowledge graph construction method in an embodiment of this application may include steps 501-504.

[0186] Step 501: Obtain multimodal design materials.

[0187] Step 502: Obtain the design knowledge corresponding to the design material for each modality.

[0188] For details on how to acquire the design knowledge corresponding to the design materials for each modality, please refer to the above-mentioned examples of knowledge mining of design materials. These will not be repeated here.

[0189] Step 503: After obtaining the design knowledge corresponding to the design material of each modality, perform joint characterization on the design knowledge corresponding to the design material of each modality to obtain the target joint characterization.

[0190] Step 504: Construct a knowledge graph based on the joint representation of the target.

[0191] This joint characterization can involve design materials with at least two modalities. The following are exemplary descriptions of different cases of design materials with at least two modalities.

[0192] Joint Representation 1: Joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text.

[0193] Specifically, in some embodiments, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a target joint characterization, including:

[0194] The design knowledge corresponding to the first design object and the design knowledge corresponding to the first text are jointly represented to obtain a joint representation of the target.

[0195] For example, the first text may include text describing the design information of the first design object, such as a description document of the first design object, used to describe design information such as constraints related to the first design object, semantic information of the first design object, and relationships between components in the first design object. Furthermore, the first text may also include text describing other design information; this embodiment of the application does not impose limitations on this.

[0196] When performing joint representation, the design knowledge corresponding to the first design object (such as the first geometric features of the first design object) can be feature-encoded (embedding) to convert it into vector form. Similarly, the design knowledge corresponding to the first text (such as entities extracted from the first text, relationships between entities, and entity attributes) can also be feature-encoded and converted into vector form. This allows the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text to be mapped to the same feature space and described using the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text can be represented in the same feature space, thus achieving joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text.

[0197] Joint Representation 2: Joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image.

[0198] Specifically, in some embodiments, the first image includes a first sub-image related to the first design object. Then, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a joint representation of the target, including:

[0199] The design knowledge corresponding to the first design object and the design knowledge corresponding to the first sub-image are jointly represented to obtain the joint representation of the target.

[0200] For example, the first image may include an image related to the first design object, such as a photograph of the first design object, an engineering drawing describing the first design object (e.g., a two-dimensional computer-aided design (CAD) drawing), etc.

[0201] During joint representation, the design knowledge corresponding to the first design object can be feature-encoded and converted into vector form. Similarly, the design knowledge corresponding to the first image (e.g., entities extracted from the first image, relationships between entities, entity attributes, etc.) can also be feature-encoded and converted into vector form. This ensures that the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image are mapped to the same feature space and described using the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image can be represented and fused in the same feature space, thus achieving joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image.

[0202] Joint representation 3: Joint representation between the design knowledge corresponding to the first image and the design knowledge corresponding to the first text.

[0203] Specifically, in some embodiments, the first image includes a first sub-image related to the first design object. Then, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a joint representation of the target, including:

[0204] The design knowledge corresponding to the first image and the design knowledge corresponding to the first text are jointly represented to obtain a joint representation of the target.

[0205] For example, the first text may include second subtext related to the first image. For instance, if one or more first images and certain second subtexts are contained in the same document, then the design knowledge of the one or more first images and second subtexts is jointly represented, thereby allowing the design knowledge of the one or more first images and second subtexts to be fused together with the context in the document.

[0206] It is understood that, in the embodiments of this application, the joint characterization of the design knowledge corresponding to the design materials of each modality may include one or more of the above three joint characterization methods.

[0207] For example, when the multimodal design materials include a first design object, a first text, and a first image, the design knowledge corresponding to the three modal design materials can be feature-encoded separately so that the design knowledge corresponding to the three modal design materials can be converted into vector form. The joint representation of the design knowledge corresponding to the three modal design materials can then achieve the fusion of the design knowledge corresponding to the three modal design materials.

[0208] In the embodiments of this application, there are various specific times for joint characterization.

[0209] For example, in some examples, after mining explicit knowledge (e.g., design knowledge corresponding to the first image and / or design knowledge of the first text unrelated to the first design object), joint representation can be performed to obtain a preliminary subgraph recording explicit knowledge. Similarly, after mining tacit knowledge (e.g., design knowledge corresponding to the first design object and / or design knowledge of the first text related to the first design object), joint representation can be performed to obtain a preliminary subgraph recording tacit knowledge. Then, a knowledge graph can be constructed based on the preliminary subgraphs recording explicit and tacit knowledge.

[0210] Alternatively, after obtaining the design knowledge corresponding to all modal design materials, the design knowledge corresponding to all modal design materials can be jointly represented to construct a knowledge graph.

[0211] In this embodiment, design knowledge can be obtained from multimodal design materials. These multimodal design materials include a first design object, which can be considered a historical design object. After obtaining the first design object, design knowledge implicit in it can be mined based on geometric features and jointly represented with design knowledge mined from other modal design materials. For example, a target joint representation in vector form can be used to achieve the fusion of design knowledge from the multimodal design materials, thereby constructing a knowledge graph. Furthermore, since this knowledge graph can include design knowledge mined from the multimodal design materials, the constructed knowledge graph contains relatively rich design knowledge, providing a good data foundation for related applications of the knowledge graph.

[0212] 4. Knowledge Integration

[0213] In this embodiment of the application, referring to the example shown in Figure 4, after joint characterization, the fusion of design knowledge corresponding to multimodal design materials can be realized. For example, semantic fusion and / or structural fusion can be performed to obtain the fusion result.

[0214] For example, semantic fusion can include the fusion of identical descriptions in entities, relationships, attributes, etc., and the unification of synonyms, to achieve semantic alignment and fusion of design knowledge corresponding to multimodal design materials. Structural fusion can include the alignment and fusion of information such as relationships between entities and entity attributes, to achieve structural alignment and fusion of design knowledge corresponding to multimodal design materials.

[0215] In this way, a knowledge graph can be constructed in a specified structured form based on the constructed schema and the fusion results, and can be stored in a graph database.

[0216] Furthermore, in the embodiments of this application, the knowledge graph can take many different forms.

[0217] For example, in some examples, the knowledge graph includes a first subgraph and at least one second subgraph. The first subgraph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in the multiple first design objects satisfies a specified condition. Any second subgraph is obtained based on one or more of the following: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

[0218] As can be seen, in this example, the knowledge graph can take the form of 1+n. Here, "1" indicates that the knowledge graph includes a subgraph describing common design knowledge (i.e., the first subgraph), while "n" indicates that the knowledge graph includes n subgraphs of design knowledge extracted from independent historical design objects (i.e., n second subgraphs), where n can be a positive integer, that is, the number of second subgraphs can be at least one.

[0219] The sources of common design knowledge in the first sub-map may include one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in the multiple first design objects satisfies a specified condition.

[0220] Common design knowledge can be considered as relatively universal design knowledge in the design field, such as the fit between standard bolts and standard nuts. Therefore, at least one first text used to obtain common design knowledge may include text describing relatively standardized design knowledge in the design field, while in many scenarios it may not include text describing personalized designs. For example, in some scenarios, at least one first text may not include text related to the first design object, such as a description document. Furthermore, at least one first image used to obtain common design knowledge may also be an image describing relatively universal and standardized design knowledge. For example, at least one first image used to obtain common design knowledge may include a photograph of a standard part or a two-dimensional CAD drawing of a standard part; additionally, at least one first image used to obtain common design knowledge may also be an image located in the same document as the text describing relatively standardized design knowledge in the design field, so as to combine the contextual information of the document to extract accurate design knowledge from the text and image in the document as common design knowledge in the first sub-map.

[0221] Furthermore, in some examples, common design knowledge may include design knowledge from multiple first design objects whose co-occurrence probabilities among the multiple first design objects satisfy a specified condition.

[0222] In this embodiment, the co-occurrence probability can be higher than a high probability threshold to ensure that the design knowledge is highly generalized across multiple first design objects. Furthermore, in some examples, the number of multiple first design objects can also reach a specified threshold to ensure that the design knowledge from multiple design objects is extracted from a sufficient number of first design objects, thus guaranteeing the generality of the design knowledge. In this case, highly generalized design knowledge from multiple first design objects can also be considered as common design knowledge in the first sub-graph.

[0223] The second sub-map can be design knowledge derived from one or more independent first design objects in history.

[0224] In traditional knowledge graphs, it is difficult to extract effective design knowledge from the first design object. However, in this embodiment, based on geometric features, co-occurrence relationships, etc., highly reliable design knowledge can be extracted from one or more first design objects and other design materials associated with one or more first design objects (such as first text associated with one or more first design objects, first images associated with one or more first design objects, etc.) to serve as design knowledge in the knowledge graph. Furthermore, design knowledge from the first design object and related design materials can be stored through one or more second sub-graphs to distinguish it from the common design knowledge in the first sub-graph. In this way, as the design task is continuously executed, the design knowledge in the second sub-graph can be easily updated based on newly generated first design objects. Moreover, it is convenient for users to select design knowledge that meets their needs from the first and second sub-graphs to execute the design task.

[0225] When multiple second sub-graphs exist, there can be several ways to divide them, which are not limited here. For example, different second sub-graphs can record design knowledge of different types of first design objects.

[0226] It is evident that the first and second sub-graphs enable the separate storage, application, updating, and management of design knowledge from different sources and with different levels of confidence.

[0227] In other examples, knowledge graphs can take different forms. For instance, a knowledge graph can include sub-graphs corresponding to different design types; or, design knowledge corresponding to various design materials can be integrated into the same knowledge graph. Based on the above methods of constructing knowledge graphs, a knowledge graph can be built for subsequent design code generation tasks. Furthermore, as shown in the example in Figure 4, this knowledge graph can be updated and iterated based on expert knowledge or the execution results of subsequent design tasks based on the generated design code. For example, the weights of corresponding design knowledge in the knowledge graph can be updated according to the specific circumstances of the target design object obtained by the design task.

[0228] In traditional knowledge graph construction, knowledge is usually extracted only from text, and it is difficult to extract knowledge from other forms of materials.

[0229] It is understood that in this embodiment of the application, the knowledge graph may be constructed through a cloud management platform and deployed in the infrastructure managed by the cloud management platform. Alternatively, the knowledge graph may be constructed by other devices and then transmitted and deployed to the infrastructure managed by the cloud management platform.

[0230] II. Enhancement and Constraints of Large Language Models for Design Code Generation Tasks

[0231] In some embodiments, in order to perform subsequent design code generation tasks, the large language model can be enhanced and constrained in advance so that the enhanced and constrained large language model can perform design code generation tasks more accurately.

[0232] Based on the different scenario requirements of subsequent design code generation tasks, the reasoning methods of the enhanced and constrained large language model in design code generation tasks can be varied. Correspondingly, the specific forms of training data and their labels used to enhance and constrain the large language model may also vary. Examples will be provided below.

[0233] The specific form of training data and its labels 1: Training data includes preset design intent and preset design knowledge, and the labels of training data include preset design code.

[0234] In this example, the specific generation method of the preset design intent for any training data can refer to the relevant embodiments of the design code generation part based on knowledge graphs and large language models, where the design intent is generated; or, the preset design intent can be from a third party; or, the preset design intent can be manually configured by the user in advance.

[0235] The preset design knowledge in the training data can be design knowledge related to the preset design intent retrieved from a knowledge graph; or, the preset design knowledge can be manually configured by the user in advance.

[0236] The labels of the training data may include preset design code, which can be used as the execution script of the design software, but does not include information about the preset design code modules in the subsequent examples. That is to say, the preset design code does not perform the design by calling at least one design code module, but includes the execution script of specific execution instructions.

[0237] The specific type of the pre-defined design code is not limited here and can be determined based on the needs of the design software performing the design task. For example, the pre-defined design code can be a domain-specific language (DSL). Specifically, in some examples, since Python has become a common secondary development script for many design software programs, the pre-defined design code can be a Python script.

[0238] The specific form of training data and its labels 2: The training data includes the preset design intent and preset design knowledge. The labels of the training data include preset triplet information and the module identifier of at least one preset design code module. The preset triplet information includes the operation object, operation instruction and operation parameter corresponding to at least one preset design code module.

[0239] In this example, you can pre-write the module identifier, specific content, and corresponding calling interface of the design code module.

[0240] For example, developers can deconstruct the historical design tasks that implement the historical design object, obtaining multiple design subtasks for designing the historical design object. They can then write design code modules for one or more of these subtasks, enabling any design code module to execute the corresponding subtask and implement the specified design function. These design code modules can be understood as functions.

[0241] Furthermore, in some examples, the design code modules can be atomic modules, meaning that any design code module can be decomposed into the smallest, indivisible code modules. This facilitates the flexible use of each design code module during subsequent design code generation, enabling flexible combinations of these modules.

[0242] After writing the design code modules, corresponding calling interfaces can be created to facilitate invoking the design code modules through these interfaces. Furthermore, a module identifier (e.g., the name of the design code module) can be determined for each design code module to uniquely identify it. The calling information of the design code modules can be standardized into triple information. Specifically, this triple information can include the operation object, operation instruction, and operation parameters. In this way, the structure of the triple information in the label—object, predicate, and object complement—can instruct the large language model to learn from natural language to code generation. This enables the augmented and constrained large language model to identify the module identifier of at least one pre-defined design code module required for the corresponding design task based on training data, and to extract pre-defined triple information from the training data to facilitate interface calls to at least one pre-defined design code module.

[0243] In this example, the labels of the training data may include preset triplet information and module identifiers of at least one preset design code module, but do not include execution statements, so they cannot be directly used as an execution script.

[0244] The specific form of training data and its labels 3: The training data includes preset design intent and preset design knowledge. The labels of the training data include preset design code that can call at least one design code module through preset triplet information and module identifier of at least one preset design code module.

[0245] In this embodiment, the labels of the training data include preset design code. This preset design code includes not only preset triplet information and module identifiers of at least one preset design code module, but also execution statements; therefore, it can be used as an execution script. Thus, this preset design code can invoke at least one preset design code module based on the preset triplet information and the module identifiers of at least one preset design code module.

[0246] The reasoning methods of the large language models enhanced and constrained by the above training data and their different forms of labels may differ in the design code generation task. For specific reasoning methods, please refer to the introduction of the relevant embodiments of the design code generation task in the following sections, which will not be repeated here.

[0247] In this embodiment of the application, after obtaining multiple sets of training data and their labels, the large language model to be enhanced can be enhanced and constrained by means of backpropagation based on the multiple sets of training data and their labels, so that the large language model obtained after enhancement and constraint can generate corresponding design code according to the input design intent.

[0248] In one example, a multi-agent framework can be used to augment and constrain the large language model to be augmented.

[0249] For example, Figure 6 shows an exemplary schematic diagram of this multi-agent framework.

[0250] The multi-agent framework can include one or more of the following: intent understanding, question deconstruction, large language model, knowledge graph, and feedback model. The process of enhancing and constraining the large language model to be enhanced includes at least one iteration. The following description uses the i-th iteration in the at least one iteration process as an example.

[0251] For example, the i-th iteration process in at least one iteration process includes: reasoning on the training data using the large language model to be enhanced in the i-th iteration process to obtain output data; evaluating whether the output data meets the specifications and / or constraints using a knowledge graph to obtain an evaluation result; and updating the large language model to be enhanced in the i-th iteration process based on the difference between the output data and the labels, and the evaluation result.

[0252] During the enhancement and constraint process, the user's input of the preset design intent can be understood, and the corresponding preset design knowledge can be queried from the knowledge graph to obtain the preset design intent and preset design knowledge. In some examples, the user's design intent can also be deconstructed to obtain one or more design subtasks, each of which can be implemented by a design code module. In this way, the preset design intent and preset design knowledge corresponding to each design subtask can be obtained as training data. Then, the training data containing the preset design intent and preset design knowledge corresponding to the design subtasks obtained from the problem deconstruction can be used as input data and fed into the large language model to be enhanced to obtain the output data of the large language model to be enhanced.

[0253] During reasoning or after obtaining output data, large language models can query the knowledge graph. For example, during reasoning, a large language model can query the knowledge graph for specifications and / or constraints of a specified design; furthermore, after obtaining output data, the large language model can query the knowledge graph to determine whether the output data meets the specifications and / or constraints, thus obtaining an evaluation result assessing whether the output data satisfies the specifications and / or constraints. Specifications can include design standards stored in the knowledge graph, or can be considered guiding conditions, which are usually relatively objective and uniform. Constraints refer to restrictive conditions, which can be described using functions or inequalities.

[0254] Thus, based on the difference between the output data and the labels, and the evaluation results, backpropagation is performed on the large language model to be enhanced in the i-th iteration process to update the weights of the large language model to be enhanced in the i-th iteration process.

[0255] When the number of iterations reaches a specified number, or when the loss value of the output data of the large language model to be enhanced converges to the desired state, the enhancement and constraint of the large language model can be completed, and the large language model can be used for subsequent design code generation tasks.

[0256] In some examples, after applying the large language model to perform design code generation tasks in real-world application scenarios to obtain and output the target design object, the large language model can be further enhanced and constrained based on expert feedback on the target design object, through the feedback model shown in the example in Figure 6.

[0257] Specifically, experts can assess the accuracy of the target design object and obtain feedback information. The feedback model can then query the knowledge graph to determine if this feedback information conforms to specifications and / or constraints, and obtain the target feedback information based on the query results. Alternatively, in some examples, the expert's input feedback information on the target design object can be directly used as the target feedback information. After obtaining the target feedback information, the feedback model can use it to perform reinforcement learning on the large language model, further enhancing and constraining the large language model during application, and continuously improving its performance.

[0258] It is understood that, in the embodiments of this application, the enhancement and constraint of the large language model can be implemented through the infrastructure managed by the cloud management platform of the embodiments of this application, or it can be implemented in other devices and then deployed to at least one server of the infrastructure.

[0259] III. Design Code Generation Based on Knowledge Graphs and Large Language Models

[0260] After constructing a knowledge graph and obtaining a fine-tuned large language model, design code generation tasks can be performed using the knowledge graph and the large language model. For example, the design code generation task can be performed using the design code generation service of the cloud management platform, as shown in Figure 2, to implement the design code generation method.

[0261] Specifically, as shown in Figure 7, the cloud service-based design code generation method may include steps 701-703.

[0262] Step 701: Provide the configuration interface.

[0263] The configuration interface is used to obtain information about the design intent input by the user, which is used to instruct the generation of the target design object.

[0264] In this embodiment of the application, the cloud management platform can provide a configuration interface to the user's client, so that the user can input information of design intent through the configuration interface displayed on the client.

[0265] For example, the design intent information may include one or more of the following: second text, second design object, and second image. Accordingly, the design intent may include one or more of the following: intent derived from second text, intent derived from second design object, and intent derived from second image.

[0266] Thus, after obtaining one or more of the second text, the second design object, and the second image through the configuration interface, the design intent can be extracted from these elements. Since this design intent is derived from user input, it can be considered to describe the user's explicit design intent regarding the target design object.

[0267] Since the specific content of design intent information can vary, there can also be various ways to obtain design intent.

[0268] For example, in some examples, the design intent includes intent derived from a second design object, and the method also includes:

[0269] Obtain the second geometric feature of the second component associated with the target design object in one or more components included in the second design object. The second geometric feature is used to describe the features related to the geometric elements in the second component.

[0270] Based on the second geometric feature, the intent derived from the second design object is obtained.

[0271] The second design object input by the user can be of various types. For example, the second design object can be a design object indicated by the user as a reference; or, the second design object can be a design object targeted by the design task. For example, if the design task to be performed is an assembly design task, then the second design object can be the object to be assembled. Furthermore, there can be one or more second design objects, which is not limited here.

[0272] In some embodiments, obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object includes:

[0273] Decompose the second design object to obtain one or more levels of components within the second design object;

[0274] Obtain the component of the target level from one or more levels of components in the second design object as the second component;

[0275] Obtain the second geometric features of the second component.

[0276] In some embodiments, the second geometric feature includes one or more of the following:

[0277] Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0278] In this embodiment, the specific content of the second geometric feature and the specific method of obtaining the second geometric feature can refer to the specific content of the first geometric feature and the specific method of obtaining the first geometric feature in any of the above embodiments, and will not be repeated here. Furthermore, in this embodiment, the process of decomposing the second design object to obtain the second geometric feature of the second component can also be understood as problem deconstruction.

[0279] After obtaining information about the second geometric features of the second design object, the second design object input by the user can be described using the information about the second geometric features, thereby obtaining the intent based on the second design object.

[0280] In some embodiments, the design material includes a second image, and the method further includes:

[0281] Design information in the second image is identified through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image.

[0282] Based on the design information in the second image, obtain the intent derived from the second image.

[0283] In this embodiment, any second image can be an independent image; or it can be an image related to the second design object, such as a photograph or 2D CAD drawing of the second design object; or it can be an image associated with the second text, such as being located in the same document as the second text. The method for obtaining the design information of the second image can refer to the method for obtaining the design knowledge of the first image in the above embodiments, and will not be repeated here.

[0284] In some embodiments, the design material includes a second text.

[0285] The second text can be descriptive text entered by the user, used to describe design requirements, such as information about the target design object to be designed.

[0286] In some examples, machine learning models or other algorithms can also be used to understand the intent of the second text in order to uncover the design intent contained within it.

[0287] Specifically, in some embodiments, the design material includes a second text, and the method further includes:

[0288] Information is extracted from the second text using a large language model or other information extraction algorithms to obtain the design information corresponding to the second text.

[0289] For example, a complex design task can be decomposed into multiple design sub-tasks based on a large language model or other information extraction algorithm for information extraction, using a tree of thoughts (ToT) and / or a chain of thoughts (CoT), thereby achieving problem deconstruction and obtaining the intent based on the second text. The intent can also be understood by utilizing the context of the second text.

[0290] In other examples, the second text can be directly taken as the intent derived from the second text.

[0291] Furthermore, in some embodiments, when the information input by the user includes data from at least two modalities (e.g., at least two of second text, second design object, and second image), joint representation can be performed in order to achieve the fusion of the intents corresponding to the two modalities.

[0292] For example, in some embodiments, the design intent includes an intent derived from a second design object and an intent derived from a second text; obtaining the design intent may include:

[0293] A first joint representation is obtained by jointly representing the second geometric features of the second component in the second design object and the design information in the second text.

[0294] Based on the first joint representation, the first joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second text.

[0295] In some embodiments, the design intent includes an intent derived from a second design object and an intent derived from a second image. Obtaining the design intent may include:

[0296] A second joint representation is obtained by jointly representing the second geometric features of the second component in the second design object and the design information in the second image;

[0297] Based on the second joint representation, the second joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second image.

[0298] In some embodiments, the design intent includes an intent derived from the second text and an intent derived from the second image. Obtaining the design intent may include:

[0299] A third joint representation is obtained by jointly representing the design information in the second text and the design information in the second image.

[0300] Based on the third joint representation, the third joint representation intent is obtained, which includes the intent obtained based on the second text and the intent obtained based on the second image.

[0301] In this embodiment, the first joint representation, the second joint representation, and the third joint representation can be in vector form to unify the intent information of data from different modalities in the same feature space, thereby facilitating subsequent unified processing.

[0302] The specific generation methods of the first joint characterization, the second joint characterization, and the third joint characterization can be referred to the relevant descriptions in the above embodiments of the joint characterization of the design knowledge corresponding to the design materials of each modality, and will not be repeated here.

[0303] As can be seen, in the embodiments of this application, the specific content of the design intent can vary depending on the different information input by the user.

[0304] Furthermore, the specific form of design intent can vary. For example, when the design intent only includes intent derived from a second text, the design intent can be in text form. However, when the design intent includes intent derived from data from multiple modalities, the design intent can be in vector form.

[0305] Furthermore, in some examples, the design intent can also be in the form of a triple, which includes the operation object, the operation instruction, and the operation parameters. The information about the operation object, operation instruction, and operation parameters in the design intent can be described in text or vector form. Therefore, in this example, the design intent includes triple information obtained based on one or more of the second text, the second design object, or the second image.

[0306] Step 702: Retrieve the target design knowledge corresponding to the design intent from the knowledge graph.

[0307] The knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, design knowledge from the first image, and target design knowledge used to describe the implicit design intent for the target design object.

[0308] After obtaining the design intent, the target design knowledge corresponding to the design intent can be retrieved from the knowledge graph.

[0309] The knowledge graph can be constructed using any of the above-described embodiments of knowledge graph construction. Therefore, the knowledge graph can include one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. In this way, a relatively comprehensive set of target design knowledge matching one or more of the intentions obtained from the second text, the second design object, and the second image can be retrieved from the knowledge graph.

[0310] For example, in some examples, traditional knowledge graph construction schemes struggle to extract and store design knowledge from historical design objects (i.e., the first design object), and in practical applications, they also cannot extract design intent from the second design object.

[0311] In the knowledge graph of any embodiment of this application, design knowledge can be extracted and stored from historical design objects. Thus, a user can input a second design object through a configuration interface, and the cloud management platform can extract the intent derived from the second design object input by the user and query relevant target design knowledge from the knowledge graph.

[0312] Target design knowledge is used to describe the implicit design intent of the target design object. In other words, through knowledge graphs, based on the linear design intent information input by the user, more comprehensive design intents that cannot be directly extracted from the user's input information can be further mined, so as to guide the large language model to generate reasonable target design code in a more comprehensive way.

[0313] For example, in one instance, if the user's input design intent instructs the user to design a car, then design knowledge such as design specifications and constraints related to car design can be mined from the knowledge graph as target design knowledge. This target design knowledge is not included in the user's input information, but can be retrieved through the knowledge graph. This knowledge can then guide the large language model to generate target design code that can more reasonably design the car in subsequent steps.

[0314] Step 703: Process the design intent and target design knowledge through a large language model to obtain the target design code.

[0315] The target design code is used to generate the target design object.

[0316] In this embodiment, design intent and target design knowledge can be input into a large language model, which can then be invoked to process the design intent and target design knowledge and output the target design code.

[0317] The specific type of the target design code is not limited here and can be determined based on the needs of the design software that performs the design task. For example, the target design code can be a domain-specific language (DSL). Specifically, in some examples, since Python has become a common secondary development script for many design software programs, the target design code can be a Python script.

[0318] The target design code can be called by the design software that performs the design task to generate executable code to perform the corresponding design task and obtain the target design object as the design result.

[0319] In this application embodiment, there are several possible scenarios for obtaining the target design code through a large language model, which will be described by example below.

[0320] Reasoning Method 1: Input the design intent and target design knowledge into the large language model for processing, so that the large language model outputs non-modular target design code.

[0321] In this example, the large language model can output the target design code, which serves as the execution script for the design software, through a single call. Furthermore, in this example, the design code module for subsequent inference method 2 is not obtained in advance. Therefore, in this embodiment, the target design code does not include information about at least one design code module; that is, the design is not performed by calling at least one design code module, but rather a specific execution script containing specific execution instructions can be directly generated as the target design code.

[0322] The large language model in this example can be obtained by fine-tuning the first type of training data and its labels in the relevant embodiments of fine-tuning the large language model described above.

[0323] Reasoning Method 2: Input the design intent and target design knowledge into the large language model for processing, so that the large language model outputs target design code containing target triple information and information of at least one design code module.

[0324] Specifically, in this example, the target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object, and the target triple information includes the operation object, operation instructions, and operation parameters corresponding to the at least one design code module.

[0325] In this embodiment of the application, the design code module can be pre-configured. For details, please refer to the relevant embodiments of large language model fine-tuning, which will not be repeated here.

[0326] The at least one design code module can also be considered as a function. Each design code module can provide an interface for calling. When calling, the information of the triple of operation object, operation instruction and operation parameter corresponding to the design code module can be used as the input information of the interface, thereby instructing the design code module to implement the corresponding design according to the information of operation object, operation instruction and operation parameter corresponding to the design code module.

[0327] The target design code includes target triple information and information of at least one design code module, which can take various forms. The specific forms of the target design code and the corresponding reasoning methods are introduced below.

[0328] Reasoning Method 2.1: By calling the large language model twice, obtain the target design code that can make interface calls to at least one design code module.

[0329] Specifically, in some embodiments, step 703 includes:

[0330] By processing design intent and target design knowledge through a large language model, target module identifiers and target triple information are obtained. The target module identifier includes the module identifier corresponding to at least one design code module.

[0331] Based on the target module identifier and target triple information, the target design code is obtained through a large language model.

[0332] In this embodiment, a large language model can be used to process design intent and target design knowledge to obtain target module identifiers and target triple information. Thus, the large language model can determine, from multiple pre-configured design code modules, at least one design code module required for the current design task (i.e., the target module identifier of that at least one design code module) based on the design intent and target design knowledge, and can determine the target triple information required to execute that at least one design code module. The module identifier corresponding to a specific design code module can uniquely identify that design code module; for example, the module identifier can be the name or number of that design code module.

[0333] Then, the large language model can be called a second time to process the target module identifier and target triple information, generating target design code. This target design code can be used to call the interface of at least one design code module based on the target triple information to achieve the design task. It is evident that this target design code can indicate the execution order of at least one design code module.

[0334] For example, in a typical scenario, a user's design intent for the car design can be obtained through a configuration interface, and the corresponding target design knowledge for the car design can be queried in a knowledge graph. Then, the design intent and target design knowledge can be input into a large language model, causing the large language model to output the target module identifier and target triple information of at least one design code module required for the car design. For example, at least one design code module may include a design code module for designing the car's engine, a design code module for designing the car's chassis, and a design code module for designing the car's exterior. Next, the target module identifier and target triple information can be processed by the large language model to generate target design code. This target design code can instruct the interface of the design code module for designing the car's engine to be called first, then the interface of the design code module for designing the car's chassis to be called first, and then the interface of the design code module for designing the car's exterior to be called first, thus designing the car rationally from the inside out, avoiding design conflicts that could lead to unreasonable design results.

[0335] In this embodiment, the design code generation task can be executed hierarchically through two calls to the large language model, making the logic for generating the design code clearer and thus ensuring the accuracy of the target design code.

[0336] Reasoning Method 2.2: By calling a large language model once, the target design code, which includes target triple information and information of at least one design code module, is obtained. However, the target design code does not include execution statements, so it cannot be directly used as an execution script.

[0337] Specifically, in some embodiments, step 703 includes:

[0338] By processing design intent and target design knowledge through a large language model, target module identifiers and target triple information are obtained as target design codes.

[0339] In this example, the design software can obtain the target triplet information and the information of at least one design code module in the target design code to generate an execution script. The executable code is then obtained based on the generated execution script to perform the design task and obtain the target design object as the design result.

[0340] In the examples of reasoning method 2.1 and reasoning method 2.2, the large language model used can be obtained by fine-tuning the second type of training data and its labels in the relevant embodiments of fine-tuning the large language model described above.

[0341] Reasoning Method 2.3: Obtain the target design code that can make interface calls to at least one design code module through a single call to the large language model.

[0342] In this example, inputting the design intent and target design knowledge into a large language model allows the model to directly output interfaces that can call at least one design code module to achieve the target design code for the corresponding design task. It is evident that this target design code can indicate the execution order of at least one design code module. Furthermore, this target design code can be executed as an executable script for the design software.

[0343] In this example, the large language model used can be obtained by fine-tuning the third type of training data and its labels in the relevant embodiments of fine-tuning the large language model described above.

[0344] As can be seen, in this embodiment of the application, not only can the explicit design intent input by the user be obtained from the configuration interface, but also the target design knowledge associated with the user's current explicit design intent can be queried from the knowledge graph in order to capture the implicit design intent of the target design object.

[0345] In this way, explicit design intent and target design knowledge from knowledge graphs can be combined to guide the large language model to generate target design code that meets the user's design needs efficiently and accurately. This achieves intelligent generation of target design code, enabling the design task to be completed accurately and efficiently based on the target design code, and the target design object to be obtained.

[0346] IV. Using the generated design code for design.

[0347] In this embodiment of the application, after obtaining the target design code, the generated target design code can be called by the corresponding design software to perform the design and obtain the target design object.

[0348] It should be noted that, in the embodiments of this application, the target design code generated can be called by the corresponding design software in the cloud service provided by the cloud management platform (such as the design scheme generation service shown in Figure 2) to perform the design, so as to obtain the target design object and then output it to the user; or, the cloud management platform can obtain the target design code and output it to the user, so that the user can call the generated target design code through the client's design software to perform the design and obtain the target design object.

[0349] During the execution of a specific design task, the design software may call the generated target design code to determine the components required to generate the target design object.

[0350] For example, in some examples, when calling target design code to design a target design object, if the target design code instructs to operate on a certain component of the target design object, then that component can be identified as the component required to generate the target design object. The required component can be retrieved from a preset component database through methods such as nearest neighbor component retrieval. This preset component database can store multiple preset components, any of which can be in the form of general-purpose parts to facilitate general use in various design tasks. In this case, the preset component can be considered a standard part. Of course, in some examples, the preset component database may also include components other than standard parts; this is not limited here.

[0351] For example, preset components that match the components required to generate the target design object can be retrieved from a preset component database by means of structural topology or other methods.

[0352] If a matching preset component is found in the preset component database, that matching preset component can be obtained as the component required to generate the target design object.

[0353] Alternatively, in some examples, when the target design code is invoked to design the target design object, if the target design code instructs to operate on a target component in the first object that does not belong to the preset component database, that is, if no preset component matching the target component is found in the preset component database, then based on the component parameters of the target component in the target design code, a target preset component whose similarity to the target component meets a specified condition is retrieved from multiple preset components in the preset component database. For example, meeting the specified similarity condition could mean that among multiple preset components in the preset component database, the component with the same topological structure as the target component has the highest similarity to the target component, or the similarity is higher than a specified similarity threshold, etc.

[0354] Then, shape fitting can be performed based on the target preset component and the component parameters of the target component to generate the target component, so that the target component can be operated according to the target design code.

[0355] Specifically, the target preset component can be used as the reference for the target component. If the target preset component and the target component have the same topology, the difference in dimensions between the target component and the target preset component can be obtained based on the topology. Based on the difference and related triplet information, shape fitting is performed to obtain the target component that meets the shape requirements in the target design code as the component required to generate the target design object.

[0356] Once the components required to generate the target design object are obtained, the design software can call the generated target design code and, based on the topological hierarchy (e.g., DAG) of the components required to generate the target design object, execute design instructions from bottom to top (e.g., from parts to higher-level parts and then to the target design object) and based on the design parameters in the target triplet information, to generate the target design object.

[0357] V. Design Result Optimization

[0358] After generating the target design object, the target design object can be optimized, and then the optimized target design object can be output to the user.

[0359] For example, in some embodiments, after obtaining the target design code, the process further includes:

[0360] Retrieve the target design object obtained by calling the target design code for design;

[0361] Retrieve global constraint information corresponding to the target design object from the knowledge graph;

[0362] Optimize the target design object based on global constraint information.

[0363] In the embodiments of this application, during the design process, the target design code usually indicates a bottom-up, local-to-global design, and the resulting target design object is usually composed of multiple components.

[0364] Therefore, to ensure the rationality of the target design object, the large language model can query the knowledge graph for the global constraint information corresponding to the target design object. Then, the large language model can be used to check whether the target design object meets the queried global constraint information. If it does not meet the requirement, the target design object can be adjusted based on the global constraint information to obtain an optimized target design object.

[0365] In addition, in some examples, the cloud management platform can also receive user feedback on the generated target design object, and determine optimization strategies for the large language model and / or knowledge graph based on the feedback, thereby optimizing the large language model and / or knowledge graph.

[0366] For example, reinforcement learning can be used to optimize large language models, while the knowledge graph can be optimized by referencing relevant knowledge graph construction schemes for the target design object and its related information. For instance, the weights of the corresponding design knowledge in the knowledge graph can be updated based on the design knowledge used by the target design object. For example, if user feedback indicates that the target design object is accurate, the weights of the design knowledge used by the target design object in the knowledge graph can be increased; conversely, if user feedback indicates that the design of a certain component in the target design object is inaccurate, the weights of the design knowledge associated with that component in the knowledge graph can be decreased.

[0367] Figure 8 shows an exemplary flowchart of an embodiment of this application.

[0368] In the example shown in Figure 8, the design task can be an assembly design task.

[0369] In this exemplary scenario, the intent understanding module can obtain the design intent, which may include one or more of the following: intent obtained based on the second text, intent obtained based on the second design object, and intent obtained based on the second image.

[0370] Then, the intent understanding module can query the target design knowledge corresponding to the design intent from the knowledge graph, thereby inputting the target design knowledge and design intent into the large language model.

[0371] Large language models process target design knowledge and design intent, which can generate target design code and output it to design software.

[0372] Design software can call the target design code and perform one or more steps such as component matching, shape fitting, assembly parameter determination, and assembly to obtain an assembly scheme as the target design object.

[0373] Then, the assembly scheme can be globally optimized. If the user determines through the specified configuration interface that the optimized assembly scheme meets the requirements, the optimized assembly scheme can be output; if the user determines through the specified configuration interface that the optimized assembly scheme does not meet the requirements, the steps of obtaining the design intent and subsequent steps can be re-executed to regenerate the design code and obtain a new assembly scheme.

[0374] The following is an exemplary information interaction diagram of an embodiment of this application, using a vehicle styling design scenario as an example.

[0375] In one example, as shown in Figure 9, a vehicle can be styled. The styling design system may include styling design software, and, exemplary, the styling design system may be located on a cloud management platform.

[0376] This example may specifically include the following steps:

[0377] 1. Users input data such as text, design objects, and / or images into the styling design system;

[0378] 2. The styling design system preprocesses user-input data, such as data decomposition and feature recognition.

[0379] 3. The large language model performs intent understanding on the preprocessed user input data to obtain the design intent;

[0380] 4. Retrieve design intent-related design knowledge from knowledge graphs using large language models;

[0381] 5. The knowledge graph returns target design knowledge related to design intent to the large language model;

[0382] 6. The large language model generates target design code based on design intent and target design knowledge;

[0383] 7. The large language model returns the target design code to the styling design software of the styling design system;

[0384] 8. The styling design software calls the target design code to generate a preliminary design scheme as the target design object;

[0385] 9. The large language model queries the knowledge graph for global constraint information of the preliminary design scheme;

[0386] 10. Knowledge graphs return global constraint information to large language models;

[0387] 11. The large language model performs global optimization on the preliminary design scheme based on global constraint information to obtain the optimized design scheme;

[0388] 12. The large language model returns the optimized design scheme to the styling design system;

[0389] 13. The styling design system outputs optimized design solutions to the user;

[0390] 14. Users send feedback information on the optimized design scheme to the styling design system;

[0391] 15. Use feedback information from the styling design system to formulate optimization strategies for the large language model and knowledge graph;

[0392] 16. Optimize the large language model;

[0393] 17. Optimize the knowledge graph.

[0394] In another example, the vehicle's styling can be designed first, followed by assembly design based on the styling design results. The styling design system may include styling design software, and the assembly design system may include assembly design software. Exemplarily, both the styling design system and the assembly design system can reside on a cloud management platform. As shown in Figure 10, this example may specifically include the following steps:

[0395] 1. User 1 (e.g., a stylist) inputs initial input data into the styling design system, such as text, design objects, and / or images;

[0396] 2. The styling design system preprocesses user-input data, such as data decomposition and feature recognition.

[0397] 3. The large language model used for styling design performs intent understanding on the preprocessed user input data to obtain the design intent;

[0398] 4. Use a large language model for styling design to retrieve design knowledge related to design intent from a knowledge graph;

[0399] 5. The knowledge graph used for styling design returns the primary target design knowledge related to the design intent to the large language model;

[0400] 6. The large language model used for styling design generates the first-target design code based on the design intent and the first-target design knowledge;

[0401] 7. The large language model used for styling design returns the first target design code to the styling design software of the styling design system;

[0402] 8. The styling design software calls the first target design code to generate a preliminary styling design scheme;

[0403] 9. The large language model used for styling design queries the knowledge graph used for styling design for global constraint information of the preliminary styling design scheme;

[0404] 10. The knowledge graph used for styling design returns global constraint information to the large language model used for styling design;

[0405] 11. The large language model used for styling design performs global optimization on the preliminary styling design scheme based on global constraint information to obtain the optimized styling design scheme;

[0406] 12. The large language model used for styling design returns the optimized styling design scheme to the styling design system;

[0407] 13. The styling design system outputs the optimized styling design scheme to user 2 (e.g., an assembly designer);

[0408] 14. User 2 inputs secondary input data such as text, design objects, and / or images into the assembly design system;

[0409] 15. The assembly design system preprocesses user-input data, such as data decomposition and feature recognition;

[0410] 16. The large language model used for assembly design performs intent understanding on preprocessed user input data to obtain design intent;

[0411] 17. Use a large language model for assembly design to retrieve design intent-related design knowledge from a knowledge graph;

[0412] 18. The knowledge graph used for assembly design returns second-objective design knowledge related to the design intent to the large language model;

[0413] 19. The large language model used for assembly design generates second-target design code based on design intent and second-target design knowledge;

[0414] 20. The large language model used for assembly design returns the second target design code to the assembly design software of the assembly design system;

[0415] 21. The assembly design software calls the second target design code to generate a preliminary assembly design scheme;

[0416] 22. The large language model used for assembly design queries the knowledge graph used for assembly design for global constraint information of the preliminary assembly design scheme;

[0417] 23. The knowledge graph used for assembly design returns global constraint information to the large language model used for assembly design;

[0418] 24. The large language model used for assembly design performs global optimization on the preliminary assembly design scheme based on global constraint information to obtain the optimized assembly design scheme;

[0419] 25. The large language model used for assembly design returns the optimized assembly design scheme to the assembly design system;

[0420] 26. User 2 sends feedback information on the optimized assembly design scheme to the assembly design system;

[0421] 27. The assembly design system formulates optimization strategies for the large language model and knowledge graph used for assembly design based on feedback information.

[0422] 28. Optimize the large language model used for assembly design;

[0423] 29. Optimize the knowledge graph used for assembly design.

[0424] The above describes the design code generation method provided by the embodiments of this application from multiple aspects. The following, with reference to the accompanying drawings, describes the cloud service-based design code generation device 11 and the knowledge graph construction device 12 provided by the embodiments of this application.

[0425] As shown in Figure 11, this application embodiment provides a cloud service-based design code generation device 11, applied to a cloud management platform. The cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center. The cloud services run on at least one server located in at least one cloud data center in the multiple regions. The device 11 includes:

[0426] Interface module 1101 is used to provide a configuration interface. The configuration interface is used to obtain the design intent information input by the user. The design intent is used to instruct the generation of the target design object.

[0427] Processing module 1102 is used for:

[0428] Retrieve target design knowledge corresponding to design intent from the knowledge graph. The knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. The target design knowledge is used to describe the implicit design intent for the target design object.

[0429] By processing design intent and target design knowledge through a large language model, target design code is obtained, which is then used to generate target design objects.

[0430] Optionally, interface module 1101 is used for:

[0431] Acquire multimodal design materials, which include a first design object and further include one or more of the following: first text and a first image;

[0432] Processing module 1102 is used for:

[0433] Based on the first design object, obtain the first geometric feature of the first component among one or more components included in the first design object, the first geometric feature being used to describe the features related to the geometric elements in the first component;

[0434] Based on the first geometric feature, obtain the design knowledge corresponding to the first design object;

[0435] After obtaining the design knowledge corresponding to the design material of each modality, the design knowledge corresponding to the design material of each modality is jointly characterized to obtain the target joint representation;

[0436] Construct a knowledge graph based on the joint representation of the objectives.

[0437] Optionally, the target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object, and the target triple information includes the operation object, operation instructions, and operation parameters corresponding to the at least one design code module.

[0438] Optionally, the processing module 1102 is used for:

[0439] By processing design intent and target design knowledge through a large language model, target module identifiers and target triple information are obtained. The target module identifier includes the module identifier corresponding to at least one design code module.

[0440] Based on the target module identifier and target triple information, the target design code is obtained through a large language model.

[0441] Optionally, the design intent includes the intent derived from the second design object.

[0442] Interface module 1101 is used to: obtain a second geometric feature of a second component associated with a target design object among one or more components included in the second design object, wherein the second geometric feature is used to describe features related to geometric elements in the second component;

[0443] The processing module 1102 is used to: obtain the intent based on the second design object according to the second geometric features.

[0444] Optionally, the processing module 1102 is used for:

[0445] Decompose the second design object to obtain one or more levels of components within the second design object;

[0446] Obtain the component of the target level from one or more levels of components in the second design object as the second component;

[0447] Obtain the second geometric features of the second component.

[0448] Optionally, the design intent also includes an intent derived from the second text, for which the processing module 1102 is used to:

[0449] The design information in the second geometric feature and the second text is jointly represented to obtain the first joint representation;

[0450] Based on the first joint representation, the first joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second text.

[0451] Optionally, the design intent also includes an intent derived from the second image, which the processing module 1102 is used to:

[0452] Design information in the second image is identified through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image.

[0453] The design information in the second geometric feature and the second image are jointly represented to obtain the second joint representation;

[0454] Based on the second joint representation, the second joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second image.

[0455] Optionally, the second geometric feature includes one or more of the following:

[0456] Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0457] As shown in Figure 12, this application embodiment provides a knowledge graph construction device 12, applied to a cloud management platform. The cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center, and cloud services running on at least one server located in at least one cloud data center in the multiple regions. The device 12 includes:

[0458] Interface module 1201 is used to acquire multimodal design materials, which include a first design object and further include one or more of the following: first text and a first image;

[0459] Processing module 1202 is used for:

[0460] Based on the first design object, obtain the first geometric feature of the first component among one or more components included in the first design object, the first geometric feature being used to describe the features related to the geometric elements in the first component;

[0461] Based on the first geometric feature, obtain the design knowledge corresponding to the first design object;

[0462] After obtaining the design knowledge corresponding to the design material of each modality, the design knowledge corresponding to the design material of each modality is jointly characterized to obtain the target joint representation;

[0463] Construct a knowledge graph based on the joint representation of the objectives.

[0464] Optionally, the interface module 1201 is used to: provide a configuration interface, which is used to obtain information about the design intent input by the user, and the design intent is used to instruct the generation of the target design object;

[0465] Processing module 1202 is used for:

[0466] Retrieve target design knowledge corresponding to design intent from the knowledge graph. Target design knowledge is used to describe the implicit design intent for the target design object.

[0467] By processing design intent and target design knowledge through a large language model, target design code is obtained, which is then used to generate target design objects.

[0468] Optionally, the number of first design objects is multiple, and the processing module 1202 is used for:

[0469] Based on the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects, the target first geometric feature is determined from the first geometric features corresponding to the multiple first design objects.

[0470] Based on the first geometric features of the target, obtain the design knowledge corresponding to the first design object.

[0471] Optionally, in the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined based on the co-occurrence probability of the first geometric feature of the target in the co-occurrence relationship.

[0472] Optionally, the knowledge graph includes a first subgraph and at least one second subgraph. The first subgraph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in multiple first design objects satisfies a specified condition. Any second subgraph is obtained based on one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

[0473] Both the processing module and the interface module can be implemented in software or hardware. For example, the implementation of the processing module will be described below. Similarly, the implementation of the interface module can be referenced from that of the processing module.

[0474] As an example of a software functional unit, a processing module may include code running on a computing instance. A computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance may be one or more. For example, a processing module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0475] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0476] As an example of a hardware functional unit, a processing module may include at least one computing device, such as a server. Alternatively, a processing module may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0477] The processing module comprises multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0478] It should be noted that, in other embodiments, the processing module can be used to execute any step in the cloud service-based design code generation method or knowledge graph construction method, and the interface module can be used to execute any step in the cloud service-based design code generation method or knowledge graph construction method. The steps implemented by the processing module and the interface module can be specified as needed. By implementing different steps in the cloud service-based design code generation method or knowledge graph construction method through the processing module and the interface module, all functions of the cloud service-based design code generation device or knowledge graph construction device can be realized.

[0479] This application also provides a computing device 130. As shown in FIG13, the computing device 130 includes a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other via the bus 132. The computing device 130 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 130.

[0480] Bus 132 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in Figure 13, but this does not imply that there is only one bus or one type of bus. Bus 132 can include pathways for transmitting information between various components of computing device 130 (e.g., memory 136, processor 134, communication interface 138).

[0481] The processor 134 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0482] Memory 136 may include volatile memory, such as random access memory (RAM). Memory 136 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD) or one or more of these. Furthermore, memory 136 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0483] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 136, or two or more types of storage media can be configured to realize the function of memory 136. This application does not limit this.

[0484] The memory 136 stores executable program code, and the processor 134 executes the executable program code to implement the functions of the aforementioned interface module and processing module, thereby realizing the cloud service-based design code generation method or knowledge graph construction method applied to the cloud management platform in the above embodiments. That is, the memory 136 stores instructions for executing the cloud service-based design code generation method or knowledge graph construction method applied to the cloud management platform in the above embodiments.

[0485] The communication interface 138 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 130 and other devices or communication networks.

[0486] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0487] As shown in Figure 14, the computing device cluster includes at least one computing device 130. The memory 136 of one or more computing devices 130 in the computing device cluster may store the same instructions for executing design code generation methods or knowledge graph construction methods.

[0488] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing cloud-based design code generation methods or knowledge graph construction methods. In other words, a combination of one or more computing devices 130 can jointly execute instructions for executing cloud-based design code generation methods or knowledge graph construction methods.

[0489] It should be noted that the memory 136 in different computing devices 130 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the cloud service-based design code generation method or the knowledge graph construction method. That is, the instructions stored in the memory 136 of different computing devices 130 can implement the functions of one or more modules in the interface module and processing module.

[0490] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 illustrates one possible implementation. As shown in Figure 15, computing devices 130A and 130B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 136 in computing device 130A can store instructions for executing the functions of the interface module. Simultaneously, the memory 136 in computing device 130B can store instructions for executing the functions of the processing module. Alternatively, in other examples, the memory 136 in computing device 130A can store instructions for executing some functions of the processing module. Simultaneously, the memory 136 in computing device 130B can store instructions for executing another part of the functions of the processing module.

[0491] It should be understood that the functions of computing device 130A shown in Figure 15 can also be performed by multiple computing devices 130. Similarly, the functions of computing device 130B can also be performed by multiple computing devices 130.

[0492] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster in Figures 14 and 15. The difference is that the memory 136 in one or more computing devices 130 in this computing device cluster can store the same instructions for executing cloud service-based design code generation methods or knowledge graph construction methods.

[0493] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing cloud-based design code generation methods or knowledge graph construction methods. In other words, a combination of one or more computing devices 130 can jointly execute instructions for executing cloud-based design code generation methods or knowledge graph construction methods.

[0494] It should be noted that the memory 136 in different computing devices 130 within the computing device cluster can store different instructions for executing parts of the functions of cloud service-based design code generation methods or knowledge graph construction methods. That is, the instructions stored in the memory 136 of different computing devices 130 can implement the functions of one or more modules in the interface module and processing module.

[0495] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute a cloud-based design code generation method or a knowledge graph construction method.

[0496] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a cloud-based design code generation method or a knowledge graph construction method.

[0497] This application also provides a chip system including a processor for implementing the steps performed by the aforementioned computing device cluster. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0498] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0499] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0500] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0501] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0502] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A design code generation method based on cloud services, characterized in that, The method is applied to a cloud management platform for managing infrastructure that provides cloud services. The infrastructure includes facilities located in multiple regions, each region including at least one cloud data center. The cloud services run on at least one server located in at least one cloud data center across the multiple regions. The method includes: A configuration interface is provided, which is used to obtain information about the design intent input by the user, and the design intent is used to instruct the generation of the target design object; Retrieve target design knowledge corresponding to the design intent from a knowledge graph, wherein the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain target design code, which is used to generate the target design object.

2. The method according to claim 1, characterized in that, Before retrieving the design knowledge corresponding to the design intent from the knowledge graph, the method further includes: Obtain multimodal design materials, the multimodal design materials including the first design object, and the multimodal design materials further including one or more of the following: the first text and the first image; Based on the first design object, a first geometric feature of a first component among one or more components included in the first design object is obtained, wherein the first geometric feature is used to describe features related to geometric elements in the first component; Based on the first geometric feature, the design knowledge corresponding to the first design object is obtained; After obtaining the design knowledge corresponding to the design material for each modality, the design knowledge corresponding to the design material for each modality is jointly characterized to obtain the target joint representation. The knowledge graph is constructed based on the joint representation of the target.

3. The method according to claim 1 or 2, characterized in that, The target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object. The target triple information includes the operation object, operation instructions, and operation parameters corresponding to the at least one design code module.

4. The method according to claim 3, characterized in that, The process of processing the design intent and the target design knowledge using a large language model to obtain the target design code includes: The design intent and the target design knowledge are processed by the large language model to obtain the target module identifier and the target triple information. The target module identifier includes the module identifier corresponding to the at least one design code module. Based on the target module identifier and the target triple information, the target design code is obtained through the large language model.

5. The method according to any one of claims 1-4, characterized in that, The design intent includes intent derived from a second design object, and the method further includes: Obtain a second geometric feature of a second component associated with the target design object from one or more components included in the second design object, the second geometric feature being used to describe features related to geometric elements in the second component; The intent derived from the second design object is obtained based on the second geometric feature.

6. The method according to claim 5, characterized in that, The step of obtaining the second geometric feature of the second component associated with the target design object from one or more components included in the second design object includes: The second design object is decomposed to obtain one or more levels of components within the second design object; Obtain the component of the target level from one or more levels of components in the second design object as the second component; Obtain the second geometric feature of the second component.

7. The method according to claim 5 or 6, characterized in that, The design intent also includes intent derived from the second text, and obtaining the intent derived from the second design object based on the second geometric features includes: The second geometric feature and the design information in the second text are jointly represented to obtain a first joint representation; Based on the first joint representation, a first joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second text.

8. The method according to claim 5 or 6, characterized in that, The design intent also includes intent derived from the second image, and the method further includes: Design information in the second image is identified through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image. Obtaining the intent based on the second design object according to the second geometric feature includes: The second geometric feature and the design information in the second image are jointly characterized to obtain a second joint representation; Based on the second joint representation, a second joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second image.

9. The method according to any one of claims 5-8, characterized in that, The second geometric feature includes one or more of the following: The spatial information of the geometric elements in the second component, the semantic information used to describe the second component and / or the geometric elements in the second component, the structural features of the second component, and the gradient features of the second component.

10. A method for constructing a knowledge graph, characterized in that, The method is applied to a cloud management platform for managing infrastructure that provides cloud services. The infrastructure includes facilities located in multiple regions, each region including at least one cloud data center. The cloud services run on at least one server located in at least one cloud data center across the multiple regions. The method includes: Obtain multimodal design materials, the multimodal design materials including a first design object, and the multimodal design materials further including one or more of the following: first text, first image; Based on the first design object, a first geometric feature of a first component among one or more components included in the first design object is obtained, wherein the first geometric feature is used to describe features related to geometric elements in the first component; Based on the first geometric feature, the design knowledge corresponding to the first design object is obtained; After obtaining the design knowledge corresponding to the design material for each modality, the design knowledge corresponding to the design material for each modality is jointly characterized to obtain the target joint representation. A knowledge graph is constructed based on the joint representation of the target.

11. The method according to claim 10, characterized in that, The method further includes: A configuration interface is provided, which is used to obtain information about the design intent input by the user, and the design intent is used to instruct the generation of the target design object; Retrieve target design knowledge corresponding to the design intent from the knowledge graph, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain target design code, which is used to generate the target design object.

12. The method according to claim 10 or 11, characterized in that, The number of the first design objects is multiple, and the step of obtaining the design knowledge corresponding to the first design objects based on the first geometric features includes: Based on the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects, the target first geometric feature is determined from the first geometric features corresponding to the multiple first design objects. Based on the first geometric features of the target, the design knowledge corresponding to the first design object is obtained.

13. The method according to claim 12, characterized in that, In the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined based on the co-occurrence probability of the first geometric feature of the target in the co-occurrence relationship.

14. The method according to any one of claims 10-13, characterized in that, The knowledge graph includes a first subgraph and at least one second subgraph. The first subgraph is used to describe common design knowledge in the design knowledge corresponding to the multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in multiple first design objects satisfies a specified condition. Any second subgraph is obtained based on one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

15. A design code generation device based on cloud services, characterized in that, An apparatus for use in a cloud management platform for managing infrastructure providing cloud services, the infrastructure comprising multiple regions, each region including at least one cloud data center, the cloud services running on at least one server located in at least one cloud data center across the multiple regions, the apparatus comprising: The interface module is used to provide a configuration interface, which is used to obtain information about the design intent input by the user, and the design intent is used to instruct the generation of the target design object. Processing module, used for: Retrieve target design knowledge corresponding to the design intent from a knowledge graph, wherein the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain target design code, which is used to generate the target design object.

16. The apparatus according to claim 15, characterized in that, The interface module is used for: Obtain multimodal design materials, the multimodal design materials including the first design object, and the multimodal design materials further including one or more of the following: the first text and the first image; The processing module is used for: Based on the first design object, a first geometric feature of a first component among one or more components included in the first design object is obtained, wherein the first geometric feature is used to describe features related to geometric elements in the first component; Based on the first geometric feature, the design knowledge corresponding to the first design object is obtained; After obtaining the design knowledge corresponding to the design material for each modality, the design knowledge corresponding to the design material for each modality is jointly characterized to obtain the target joint representation. The knowledge graph is constructed based on the joint representation of the target.

17. The apparatus according to claim 15 or 16, characterized in that, The target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object. The target triple information includes the operation object, operation instructions, and operation parameters corresponding to the at least one design code module.

18. The apparatus according to claim 17, characterized in that, The processing module is used for: The design intent and the target design knowledge are processed by the large language model to obtain the target module identifier and the target triple information. The target module identifier includes the module identifier corresponding to the at least one design code module. Based on the target module identifier and the target triple information, the target design code is obtained through the large language model.

19. The apparatus according to any one of claims 15-18, characterized in that, The design intent includes intent derived from the second design object. The interface module is used to: obtain a second geometric feature of a second component associated with the target design object from one or more components included in the second design object, wherein the second geometric feature is used to describe features related to geometric elements in the second component; The processing module is used to: obtain the intent based on the second design object according to the second geometric feature.

20. The apparatus according to claim 19, characterized in that, The processing module is used for: The second design object is decomposed to obtain one or more levels of components within the second design object; Obtain the component of the target level from one or more levels of components in the second design object as the second component; Obtain the second geometric feature of the second component.

21. The apparatus according to claim 19 or 20, characterized in that, The design intent also includes intent derived from the second text, and the processing module is used to: The second geometric feature and the design information in the second text are jointly represented to obtain a first joint representation; Based on the first joint representation, a first joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second text.

22. The apparatus according to claim 19 or 20, characterized in that, The design intent also includes an intent derived from the second image, and the processing module is used to: Design information in the second image is identified through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image. The second geometric feature and the design information in the second image are jointly characterized to obtain a second joint representation; Based on the second joint representation, a second joint representation intent is obtained, which includes the intent obtained based on the second design object and the intent obtained based on the second image.

23. The apparatus according to any one of claims 19-22, characterized in that, The second geometric feature includes one or more of the following: The spatial information of the geometric elements in the second component, the semantic information used to describe the second component and / or the geometric elements in the second component, the structural features of the second component, and the gradient features of the second component.

24. A knowledge graph construction device, characterized in that, An apparatus for use in a cloud management platform for managing infrastructure providing cloud services, the infrastructure comprising multiple regions, each region including at least one cloud data center, the cloud services running on at least one server located in at least one cloud data center across the multiple regions, the apparatus comprising: An interface module is used to acquire multimodal design materials, wherein the multimodal design materials include a first design object, and further include one or more of the following: first text, first image; Processing module, used for: Based on the first design object, a first geometric feature of a first component among one or more components included in the first design object is obtained, wherein the first geometric feature is used to describe features related to geometric elements in the first component; Based on the first geometric feature, the design knowledge corresponding to the first design object is obtained; After obtaining the design knowledge corresponding to the design material for each modality, the design knowledge corresponding to the design material for each modality is jointly characterized to obtain the target joint representation. A knowledge graph is constructed based on the joint representation of the target.

25. The apparatus according to claim 24, characterized in that, The interface module is used to: provide a configuration interface, which is used to obtain information about the design intent input by the user, and the design intent is used to instruct the generation of a target design object; The processing module is used for: Retrieve target design knowledge corresponding to the design intent from the knowledge graph, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain target design code, which is used to generate the target design object.

26. The apparatus according to claim 24 or 25, characterized in that, The number of the first design objects is multiple, and the processing module is used for: Based on the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects, the target first geometric feature is determined from the first geometric features corresponding to the multiple first design objects. Based on the first geometric features of the target, the design knowledge corresponding to the first design object is obtained.

27. The apparatus according to claim 26, characterized in that, In the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined based on the co-occurrence probability of the first geometric feature of the target in the co-occurrence relationship.

28. The apparatus according to any one of claims 24-27, characterized in that, The knowledge graph includes a first subgraph and at least one second subgraph. The first subgraph is used to describe common design knowledge in the design knowledge corresponding to the multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects whose co-occurrence probability in multiple first design objects satisfies a specified condition. Any second subgraph is obtained based on one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

29. A computing device cluster, characterized in that, It includes at least one computing device, said at least one computing device including a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device cluster to perform the method as described in any one of claims 1-9, or to perform the method as described in any one of claims 10-14.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a processor, causes the processor to perform the method as described in any one of claims 1-14.

31. A computer program product containing instructions, characterized in that, When the instructions are executed by the processor, the method described in any one of claims 1-14 is implemented.

Images