Design code generation method, knowledge graph construction method and related equipment

Abstract

The embodiment of the invention discloses a design code generation method based on cloud service. The method is applied to a cloud management platform. According to the method, the dominant design intention input by the user can be obtained from the configuration interface, and the target design knowledge associated with the current dominant design intention of the user can be inquired from the knowledge graph so as to capture the implicit design intention implicit to the target design object. Thus, the dominant design intention and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate the target design code meeting the design requirement of the user, intelligent generation of the target design code is achieved, the design task is accurately and efficiently achieved according to the target design code, and the user experience is improved. And obtaining a target design object.

Description

Technical Field

[0001] The present application relates to the field of industrial design technology, and specifically to a design code generation method, a knowledge graph construction method, and related equipment. Background Art

[0002] In the industrial field, traditional design work (such as the design and assembly of three-dimensional models, user interface (UI) design, etc.) has a high learning cost and requires designers to have a certain degree of computer skills and sufficient professional knowledge (for example, if the designed three-dimensional model is used in the engineering field, professional knowledge in the engineering field is required). Therefore, the labor cost is high and the production cycle is long, and it is difficult to adjust and modify.

[0003] At present, the introduction of large language model (LLM) in the field of artificial intelligence (AI) has made intelligent design possible.

[0004] However, when using large language models to assist industrial design, since the design from natural language to three-dimensional models involves dimensional reasoning, it is difficult for large language models to perceive and understand the user's design intent from the text prompts input by the user. Moreover, as a general model, large language models usually lack specific domain knowledge in the industrial field, resulting in industrial design based on large language models usually only able to generate very simple structures, and unable to achieve the design and / or assembly tasks of complex structures.

[0005] It can be seen that there is an urgent need for a method that can perform intelligent industrial design efficiently and accurately. Summary of the invention

[0006] The embodiment of the present application provides a design code generation method based on cloud services, which can conveniently and efficiently generate design codes for users intelligently to efficiently implement design tasks such as three-dimensional models and user interfaces. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0007] The first aspect of the present application provides a design code generation method based on cloud services, which is applied to a cloud management platform, and the cloud management platform is used to manage the infrastructure for providing cloud services, the infrastructure includes being set up in multiple regions, each region includes at least one cloud data center, and the cloud service runs on at least one server located in at least one cloud data center in multiple regions. The method includes: providing a configuration interface, the configuration interface is used to obtain information about the design intent input by the user, and the design intent is used to indicate the generation of a target design object; retrieving target design knowledge corresponding to the design intent from a knowledge graph, and the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, and the target design knowledge is used to describe the implicit design intent of the target design object; processing the design intent and the target design knowledge through a large language model to obtain a target design code, and the target design code is used to generate a target design object.

[0008] In the first aspect, not only can the explicit design intent of the user input be obtained from the configuration interface, but the target design knowledge associated with the user's current explicit design intent can also be queried from the knowledge graph to capture the implicit design intent for the target design object.

[0009] In this way, the explicit design intent and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate target design codes that meet the user's design needs, thereby realizing the intelligent generation of target design codes, so as to accurately and efficiently implement design tasks according to the target design codes and obtain the target design objects.

[0010] In a possible implementation of the first aspect, before retrieving design knowledge corresponding to the design intent from the knowledge graph, the method also includes: acquiring multimodal design materials, the multimodal design materials including a first design object, and the multimodal design materials also include one or more of the following: a first text and a first image; according to the first design object, obtaining a first geometric feature of a first component among the one or more components included in the first design object, the first geometric feature being used to describe features related to geometric elements in the first component; according to the first geometric feature, obtaining design knowledge corresponding to the first design object; after obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design materials of each modality to obtain a target joint representation; and constructing a knowledge graph according to the target joint representation.

[0011] In traditional knowledge graph construction, knowledge is usually extracted only from text, while it is difficult to extract knowledge from other forms of materials.

[0012] In this possible implementation, design knowledge can be obtained from multimodal design materials. The multimodal design materials include a first design object, which can be considered as a historical design object. After obtaining the first design object, the design knowledge implicit in the first design object can be mined from the first design object based on geometric features, and the design knowledge mined from design materials of other modalities can be jointly represented, for example, by using target joint representation in the form of vectors, so as to achieve the fusion of design knowledge of multimodal design materials and realize the construction of a knowledge graph. In addition, since the knowledge graph can include design knowledge mined from multimodal design materials, the design knowledge contained in the knowledge graph is relatively rich, providing a good data foundation for related applications of the knowledge graph.

[0013] In a possible implementation of the first aspect, the target design code includes target triplet information and information of at least one design code module, the at least one design code module is used to generate a target design object, and the target triplet information includes operation objects, operation instructions and operation parameters corresponding to the at least one design code module.

[0014] In this possible implementation, the at least one design code module can also be considered as a function. Each design code module can provide an interface for calling, and when calling, the triplet of information of the operation object, operation instruction and operation parameter corresponding to the design code module can be used as the input information of the interface, thereby instructing the design code module to implement the corresponding design according to the information of the operation object, operation instruction and operation parameter corresponding to the design code module.

[0015] In a possible implementation of the first aspect, the design intent and target design knowledge are processed through a large language model to obtain a target design code, including: processing the design intent and target design knowledge through a large language model to obtain a target module identifier and target triplet information, the target module identifier including a module identifier corresponding to at least one design code module; obtaining the target design code through a large language model according to the target module identifier and the target triplet information.

[0016] In this possible implementation, the design code generation task is performed hierarchically by calling the large language model at least twice, and the target design code that can make an interface call to at least one design code module is obtained, so that the generation logic of the execution design code is clearer, thereby ensuring the accuracy of the target design code.

[0017] In a possible implementation of the first aspect, the design intent includes an intent obtained based on a second design object, and also includes: obtaining a second geometric feature of a second component associated with a target design object in one or more components included in the second design object, the second geometric feature being used to describe features related to geometric elements in the second component; and obtaining the intent obtained based on the second design object based on the second geometric feature.

[0018] In this possible implementation, the geometric elements may include primitives in a two-dimensional image or geometric elements in a three-dimensional image, for example. The primitives in the two-dimensional image may include one or more of points, lines, and surfaces, and the geometric elements in the three-dimensional image may include spheres or basic geometric bodies. The second geometric features may include features of the geometric elements themselves in the second component, or features related to the structure composed of the geometric elements in the second component; and the second geometric features may be described by related parameter data of the geometric elements or by semantic forms.

[0019] In a possible implementation of the first aspect, obtaining a second geometric feature of a second component associated with a target design object among one or more components included in the second design object includes: decomposing the second design object to obtain components of one or more levels in the second design object; obtaining a component of a target level from the components of one or more levels in the second design object as the second component; and obtaining the second geometric feature of the second component.

[0020] In this possible implementation, after the first design object is decomposed, the topological hierarchy of the components at each level after decomposition can be described by a directed acyclic graph (DAG), etc. After obtaining components of one or more levels, the components corresponding to the lowest level of the one or more levels (i.e., the leaf nodes in the DAG) can be assembled from the bottom up to obtain the first design object. Therefore, the components corresponding to the leaf nodes can describe the characteristics of the first design object, and the components corresponding to the leaf nodes are usually in the form of relatively common parts in the design field. Therefore, the components corresponding to the leaf nodes can be used as the first components.

[0021] In a possible implementation of the first aspect, the design intent also includes an intent obtained based on the second text, and according to the second geometric feature, an intent obtained based on the second design object is obtained, including: jointly representing the second geometric feature and the design information in the second text to obtain a first joint representation; according to the first joint representation, obtaining a first joint representation intent, the first joint representation intent including the intent obtained based on the second design object and the intent obtained based on the second text.

[0022] In this possible implementation, when performing joint representation, the design information corresponding to the second design object (such as the second geometric features of the second design object and other information) can be feature encoded (embedding) to convert it into a vector form, and the design information corresponding to the second text (such as entities extracted from the second text, relationships between entities, attributes of entities and other information) can also be feature encoded and converted into a vector form, so that the design information corresponding to the second design object and the design information corresponding to the second text are mapped to the same feature space and described in the same form. In this way, the design information corresponding to the second design object and the design information corresponding to the second text can be represented in the same feature space, that is, the joint representation of the design information corresponding to the second design object and the design information corresponding to the second text is achieved.

[0023] In a possible implementation manner of the first aspect, the design intent also includes an intent obtained based on the second image, and the method also includes: identifying design information in the second image through target detection and / or semantic segmentation, the design information in the second image including one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; obtaining an intent based on the second design object according to the second geometric feature, including: jointly characterizing the second geometric feature and the design information in the second image to obtain a second joint representation; obtaining a second joint representation intent according to the second joint representation, the second joint representation intent including the intent obtained based on the second design object and the intent obtained based on the second image.

[0024] In this possible implementation, when performing joint representation, the design information corresponding to the second design object (such as the second geometric features of the second design object and other information) can be feature encoded (embedding) to be converted into a vector form, and the design information corresponding to the second image (such as entities extracted from the second image, relationships between entities, attributes of entities and other information) can also be feature encoded and converted into a vector form, so that the design information corresponding to the second design object and the design information corresponding to the second image are mapped to the same feature space and described in the same form. In this way, the design information corresponding to the second design object and the design information corresponding to the second image can be represented in the same feature space, that is, the joint representation of the design information corresponding to the second design object and the design information corresponding to the second image is achieved.

[0025] In a possible implementation manner of the first aspect, the second geometric feature includes one or more of the following: spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0026] In a possible implementation manner of the first aspect, the design intent includes an intent obtained based on the second image, and the method also includes: identifying design information in the second image through target detection and / or semantic segmentation, the design information in the second image including one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; and obtaining the intent of the first image based on the design information in the second image.

[0027] In this possible implementation, computer vision or other methods may be used to extract the user's design intent from the second image to better instruct the large language model to obtain a target design code that meets the user's design needs.

[0028] In a possible implementation of the first aspect, the design intent also includes an intent obtained based on the second text, and the first image intent is obtained based on the design information in the second image, including: jointly characterizing the design information in the second text and the design information in the second image to obtain a third joint representation; obtaining a third joint representation intent based on the third joint representation, the third joint representation intent including the intent obtained based on the second text and the intent obtained based on the second image.

[0029] In a possible implementation manner of the first aspect, it also includes: when calling the target design code to design the first object, if the target design code indicates to operate on the target component in the first object that does not belong to the preset component database, then based on the component parameters of the target component in the target design code, retrieve the target preset component whose similarity with the target component meets the specified conditions from multiple preset components in the preset component database; perform shape fitting according to the target preset component and the component parameters of the target component to generate the target component, so as to operate the target component according to the target design code.

[0030] In this possible implementation, illustratively, the target preset component can be used as a benchmark for the target component. When the topological structures of the target preset component and the target component are consistent, the difference in dimension between the target component and the target preset component can be obtained based on the topological structure, and shape fitting can be performed based on the difference and related triplet information, so as to obtain the target component that meets the shape requirements in the target design code as the component required to generate the target design object.

[0031] In a possible implementation of the first aspect, after obtaining the target design code, it also includes: obtaining a target design object obtained by calling the target design code for design; querying global constraint information corresponding to the target design object from the knowledge graph; and optimizing the target design object according to the global constraint information.

[0032] In this possible implementation, during the design process, the target design code usually indicates that the design should be carried out from bottom to top and from local to global, and the target design object obtained by design is usually composed of multiple components. Therefore, in order to ensure the rationality of the target design object, the large language model can query the knowledge graph for the global constraint information corresponding to the target design object. Then, the large language model can be used to detect whether the target design object satisfies the queried global constraint information. If not, the target design object can be adjusted according to the global constraint information to obtain the optimized target design object.

[0033] A second aspect of the present application provides a knowledge graph construction method, which is applied to a cloud management platform, where the cloud management platform is used to manage infrastructure for providing cloud services, where the infrastructure includes being set up in multiple regions, each region including at least one cloud data center, and the cloud service runs on at least one server of at least one cloud data center located in multiple regions, the method including: obtaining multimodal design materials, where the multimodal design materials include a first design object, and the multimodal design materials also include one or more of the following: a first text, a first image; based on the first design object, obtaining a first geometric feature of a first component among one or more components included in the first design object, where the first geometric feature is used to describe features related to geometric elements in the first component; based on the first geometric feature, obtaining design knowledge corresponding to the first design object; after obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design material of each modality to obtain a target joint representation; and constructing a knowledge graph based on the target joint representation.

[0034] In traditional knowledge graph construction, knowledge is usually extracted only from text, while it is difficult to extract knowledge from other forms of materials.

[0035] In the second aspect, design knowledge can be obtained from multimodal design materials. Among them, the multimodal design materials include a first design object, which can be considered as a historical design object. After obtaining the first design object, the design knowledge implied in the first design object can be mined from the first design object based on geometric features, and jointly represented with the design knowledge mined from design materials of other modalities, for example, using target joint representation such as vector form, so as to achieve the fusion of design knowledge of multimodal design materials and realize the construction of knowledge graph. In addition, since the knowledge graph can include the design knowledge mined from multimodal design materials, the design knowledge contained in the knowledge graph is relatively rich, providing a good data foundation for related applications of the knowledge graph.

[0036] In a possible implementation of the second aspect, the method also includes: providing a configuration interface, the configuration interface is used to obtain information about design intent input by a user, the design intent is used to indicate the generation of a target design object; retrieving target design knowledge corresponding to the design intent from a knowledge graph, the target design knowledge is used to describe the implicit design intent for the target design object; processing the design intent and the target design knowledge through a large language model to obtain a target design code, and the target design code is used to generate a target design object.

[0037] In a possible implementation manner of the second aspect, there are multiple first design objects, and design knowledge corresponding to the first design objects is obtained according to the first geometric features, including: determining a target first geometric feature from the first geometric features corresponding to the multiple first design objects according to a co-occurrence relationship of the first geometric features corresponding to the multiple first design objects; and obtaining the design knowledge corresponding to the first design object according to the target first geometric feature.

[0038] In this possible implementation, since a single first design object is an independent design object and may contain some personalized or inappropriate designs, the more commonly used target first geometric features can be screened out from multiple first design objects based on the co-occurrence relationship as the design knowledge of the knowledge graph, thereby ensuring the reliability of the design knowledge in the knowledge graph.

[0039] In a possible implementation manner of the second aspect, in the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship.

[0040] In this possible implementation, in the knowledge graph, a weight can be set for the design knowledge from the first design object (that is, the design knowledge corresponding to the first geometric feature of the target). The weight of the design knowledge corresponding to the first geometric feature of the target can also be considered as the confidence of the design knowledge corresponding to the first geometric feature of the target, which can reflect the credibility of the design knowledge corresponding to the first geometric feature of the target.

[0041] In a possible implementation of the second aspect, the knowledge graph includes a first sub-graph and at least one second sub-graph, the first sub-graph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials, the common design knowledge including one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets specified conditions, and any second sub-graph is obtained according to one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to one or more first texts associated with one or more first design objects, and design knowledge corresponding to one or more first images associated with one or more first design objects.

[0042] In this possible implementation, the knowledge graph may be in the form of 1+n, where "1" indicates that the knowledge graph includes a subgraph describing common design knowledge (i.e., the first subgraph), and "n" indicates that the knowledge graph includes n subgraphs of design knowledge extracted from independent historical design objects (i.e., n second subgraphs), where n may be a positive integer, that is, the number of second subgraphs may be at least one.

[0043] It can be seen that through the first sub-graph and the second sub-graph, the design knowledge from different sources and with different confidence levels can be stored, applied, updated and managed separately.

[0044] In a possible implementation of the second aspect, the first geometric feature includes one or more of the following: spatial information of geometric elements in the first component, semantic information used to describe the first component and / or geometric elements in the first component, structural features of the first component, and gradient features of the first component.

[0045] In a possible implementation of the second aspect, according to the first design object, obtaining a first geometric feature of a first component among one or more components included in the first design object includes: decomposing the first design object to obtain components of one or more levels in the first design object; obtaining a component of a target level as the first component from the components of one or more levels in the first design object; and obtaining a first geometric feature of the first component.

[0046] In a third aspect, the present application provides a design code generation device based on a cloud service, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an interface module and a processing module.

[0047] In a fourth aspect, the present application provides a computing device cluster, which includes at least one computing device, and the at least one computing device includes a processor and a memory. The memory of at least one computing device stores computer execution instructions that can be run on the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation method of the first aspect.

[0048] The fifth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0049] The sixth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0050] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting the processor to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0051] In an eighth aspect, the present application provides a knowledge graph construction device, which has the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an interface module and a processing module.

[0052] In a ninth aspect of the present application, a computing device cluster is provided, which includes at least one computing device, and the at least one computing device includes a processor and a memory. The memory of at least one computing device stores computer execution instructions that can be run on the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the second aspect above or any possible implementation method of the second aspect.

[0053] The tenth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.

[0054] In an eleventh aspect of the present application, there is provided a computer program product storing one or more computer executable instructions. When the computer executable instructions are executed by a processor, the processor executes a method as in the second aspect or any possible implementation of the second aspect.

[0055] In a twelfth aspect of the present application, a chip system is provided, which includes a processor for supporting the processor to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0056] Among them, the technical effects brought about by the third to seventh aspects or any possible implementation methods thereof may refer to the technical effects brought about by the first aspect or the related possible implementation methods of the first aspect, and the technical effects brought about by the eighth to twelfth aspects or any possible implementation methods thereof may refer to the technical effects brought about by the second aspect or the related possible implementation methods of the second aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an exemplary framework diagram of a cloud data center provided in an embodiment of the present application;

[0058] Figure 2 is an exemplary schematic diagram of a cloud service provided by an embodiment of the present application;

[0059] Figure 3 It is a schematic diagram of an exemplary process provided in an embodiment of the present application;

[0060] Figure 4 It is an exemplary schematic diagram of the knowledge graph construction provided in the embodiment of the present application;

[0061] Figure 5 It is an exemplary schematic diagram of the knowledge graph construction method provided in the embodiment of the present application;

[0062] Figure 6 is an exemplary structural diagram of a multi-agent framework provided in an embodiment of the present application;

[0063] Figure 7 is an exemplary schematic diagram of the design code generation method provided in an embodiment of the present application;

[0064] Figure 8 It is a schematic diagram of an exemplary process provided in an embodiment of the present application;

[0065] Fig. 9 is an exemplary information interaction diagram provided in an embodiment of the present application;

[0066] Fig.10 is another exemplary information interaction diagram provided in an embodiment of the present application;

[0067] Fig.11 is an exemplary schematic diagram of a design code generation device based on cloud service provided in an embodiment of the present application;

[0068] Fig.12 is an exemplary schematic diagram of a knowledge graph construction device provided in an embodiment of the present application;

[0069] Fig.13 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0070] Fig.14 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0071] Fig.15 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0073] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0074] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0075] 1. Large language model (LLM)

[0076] A large language model is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0077] Specifically, the big language model is a technology that has emerged in recent years. Because the big language model has undergone sophisticated data engineering and training processes, its parameters have learned a lot of existing natural language processing knowledge. This knowledge can already replace people in many language-related tasks, such as letting the big language model write code or let the big language model do text summarization and other tasks.

[0078] 2. Graph Model

[0079] A graphical model refers to a graph consisting of nodes and edges used to describe a system.

[0080] In the embodiment of the present application, the graph model may specifically include a directed acyclic graph (DAG) and an undirected acyclic graph.

[0081] 3. Directed acyclic graph (DAG) decomposition

[0082] DAG decomposition is a task decomposition method that decomposes tasks based on the concept of directed acyclic graphs. DAG is a directed graph in which there are no cycles, that is, starting from a vertex, it is impossible to return to the vertex through a series of edges. The DAG graph structure is very suitable for representing and solving some problems with sequence or conditional dependencies, such as task scheduling and the shortest path problem in algorithms.

[0083] In DAG decomposition, a complex task is decomposed into multiple independent subtasks, each of which can be executed independently. This decomposition process first requires clarifying the in-degree of each point in the graph (the number of directed edges entering the vertex), then classifying the vertices with in-degree 0 as the top layer or the first layer, and then searching for vertices with in-degree 0 in the subgraph without the vertices of the previous layer to classify them into the next layer, and so on, until all vertices are assigned to different layers. Each layer can be executed independently, thus realizing parallel processing of tasks.

[0084] 4. Knowledge Graph

[0085] The knowledge graph is a graph-based data structure and a representation of a semantic network, used to display entities and the relationships between entities. The knowledge graph includes nodes and edges, where nodes are used to represent entities, such as people, place names, companies, etc. The knowledge graph can also correspond to the attributes of entities, and edges represent the relationships between entities. In the knowledge graph, data can be organized through multi-tuple data. The multi-tuple data can include triples, quadruples, or quintuples, etc. Among them, the representation of triple data includes: "node-edge-node" and "node-attribute name-attribute value", etc.

[0086] The embodiment of the present application provides a design code generation method based on cloud services, which can conveniently and efficiently generate design codes for users intelligently, so as to efficiently implement design tasks such as three-dimensional models and user interfaces, and realize intelligent industrial design.

[0087] The embodiments of the present application can be applied to a cloud management platform.

[0088] The cloud management platform is used to manage the infrastructure for providing cloud services. The infrastructure includes being set up in multiple regions, each region including at least one cloud data center, and the cloud service runs on at least one server located in at least one cloud data center in the multiple regions.

[0089] Combine the following Figure 1 An architectural diagram shown provides an exemplary introduction to a cloud data center.

[0090] Figure 1 In the cloud management platform, the cloud management platform communicates with one or more servers (such as Figure 1The server 1 and server 2 in the server exchange information. The server includes a hardware layer and a software layer. The hardware layer includes the hardware configured by the server, wherein the PCI device may be, for example, a network card, a graphics processing unit (GPU), an offload card, etc., which can be inserted into the peripheral component interconnect standard (PCI) or peripheral component interconnect express (PCIe) slot of the server. The software layer includes an operating system installed and running on the server (relative to the operating system of the virtual machine, it can be called the host operating system). The host operating system is provided with a virtual machine manager (also called a hypervisor). The role of the virtual machine manager is to realize the computing virtualization, network virtualization and storage virtualization of the virtual machine, and is responsible for managing the virtual machine. Among them, a virtual machine (virtual machine) refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. Figure 1 In the system architecture shown, multiple servers are set up in the infrastructure, and the servers can be used to run virtual machines. The specifications of each virtual machine can be the same or different. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS), elastic instance, etc. Different cloud service providers may have different names.

[0091] In one example of the embodiment of the present application, the cloud management platform can be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities can provide cloud services to users. Users obtain cloud services through the Internet but do not own cloud computing resources. In other embodiments of the present application, the cloud management platform can be a private cloud platform or a hybrid cloud platform, and the present application does not limit this.

[0092] Specifically, in Figure 1 In the example shown, the cloud management platform can provide an access interface (such as an interface or an application programming interface (API)), and users of the cloud management platform and cloud service providers can operate the client remote access access interface to register a cloud account and password on the cloud management platform, and after the cloud management platform successfully authenticates the cloud account and password, log in to the cloud management platform, thereby creating, managing, logging in and operating virtual machines in the cloud data center. It can be seen that enterprises, organizations or individuals can purchase or rent cloud services through the cloud management platform, so that they can perform designated tasks through the cloud services provided by the cloud resources of the cloud management platform.

[0093] For example, if Figure 2 As shown, the cloud services provided by the cloud management platform may include design code generation services, and in some examples may also include design solution generation services.

[0094] For example, in Figure 2 In some of the examples shown, the user can send information about the design intent to the cloud management platform to instruct the cloud management platform to perform a design code generation task, thereby intelligently generating the target design code required for the design task based on the knowledge graph and the large language model and feeding it back to the user, so that the user can perform the design task of the target design object (such as a modeling design task, an assembly task, or a design task for a user interface, etc.) according to the target design code and obtain the target design object as a design result.

[0095] The target design object can also be called a target design instance, which is a design object generated in a design task. A target design object can also be called a target design instance, for example, a 3D model that completes a shape design, a 3D model that completes an assembly design, or a user interface that completes a graphic design. It can be seen that there can be multiple specific types of target design objects, which can be determined according to actual application scenarios.

[0096] Or, in Figure 2 In some of the examples shown, the user can send information about the design intent to the cloud management platform to instruct the cloud management platform to perform the design code generation task, thereby intelligently generating the target design code required for the design task based on the knowledge graph and the large language model. The design code generation service can then pass the target design code to the design solution generation service of the cloud management platform. Next, the design solution generation service can call the target design code to perform a design task on the target design object (such as a modeling design task, an assembly task, or a design task for a user interface for a three-dimensional model, etc.) to obtain the target design object as a design result (such as a designed three-dimensional model or user interface, etc.), and then output the target design object to the user from the cloud management platform.

[0097] It should be noted that Figure 2 The services mentioned are only examples of services provided by the cloud management platform and are not limiting.

[0098] In other examples provided in this application, the above-mentioned design code generation service and design solution generation service can be deployed as a whole on the cloud management platform; and the functional division method of each service can be the same as Figure 2 The services shown are different, and the deployment methods of each service may also be different. Each service can be provided independently, or embedded in other services, or multiple services can be combined and deployed, and this application does not limit this.

[0099] Based on the above cloud management platform, refer to Figure 2 The system architecture shown in Figure 3 As shown, the method of the embodiment of the present application may include one or more of the following aspects:

[0100] Knowledge graph construction, large language model enhancement and constraints for design code generation tasks, design code generation based on knowledge graph and large language model, calling the generated design code for design, and optimization of design results.

[0101] The above-mentioned aspects are introduced exemplarily below.

[0102] 1. Knowledge Graph Construction

[0103] The knowledge graph construction in the embodiment of the present application may include one or more of the following aspects:

[0104] Construction of the knowledge layer (schema) of the knowledge graph, knowledge mining of design materials, joint representation of design knowledge from multimodal design materials, and knowledge fusion.

[0105] The following is an illustrative introduction to the various aspects that may be involved in building a knowledge graph.

[0106] 1. Construction of the data model (schema) of the knowledge graph

[0107] The knowledge graph can be considered as a structured semantic knowledge base, which can include a data model schema and an instance layer. Among them, the schema can also be called a data model, etc., which can be considered as a data model of a specified field (for example, the design field in the embodiment of the present application), which contains meaningful concept types in the specified field and the attributes of these concept types. The schema of the specified field is mainly expressed by type and property. The instance layer includes instance data. For example, the instance data corresponding to the entity "name" in the knowledge graph includes Zhang San, Li Si, etc.

[0108] In the embodiments of the present application, the specific method of constructing the schema is not limited here, and an existing or subsequently developed schema construction method can be adopted.

[0109] For example, a schema can be constructed based on expert experience. Figure 4In the example shown, for the application field involved (that is, the design field in the embodiment of the present application), the expert predicts the schema from top to bottom according to the design object of the design field, summarizes the schema from bottom to top, and then performs information fusion to construct the schema. For example, the preset text in the design field can be collected and sorted. The preset text usually has a small amount of data to facilitate the expert to summarize and extract information; then summarize and summarize the characteristics of entities, relationships and attributes in the preset text, so as to extract the relevant data framework of the design field and form a certain schema.

[0110] After constructing the schema, information can be extracted from the design materials in a targeted manner based on the constructed schema (for example, extracting information such as entities, relationships, and attributes that meet the requirements of the schema) to construct a knowledge graph.

[0111] 2. Knowledge mining of design materials

[0112] In the embodiment of the present application, the design material may be of one or more types. In other words, the design material may be a single-mode design material or a multi-mode design material.

[0113] Exemplarily, the design material may include one or more of the following: a first text, a first design object, and a first image.

[0114] The following is an exemplary introduction to the knowledge mining of design materials of any of the above-mentioned modalities, and an exemplary introduction to possible joint characterization schemes between design materials of multiple modalities.

[0115] (1) Mining explicit knowledge contained in documents

[0116] In the embodiment of the present application, the design knowledge carried by the document can be regarded as explicit knowledge.

[0117] Documented explicit knowledge may include unstructured documents, semi-structured documents, and structured documents.

[0118] In one example, the structured document may include a structure diagram, a flow chart, or other diagram structure and / or a structured form such as a table. Since the data structure in the structured document is clear, it is convenient to extract structured information such as entities, relationships, and attributes from the structured document as design knowledge extracted from the structured document.

[0119] The following is an exemplary introduction to the knowledge mining of the first image and the first text that may be contained in the unstructured document or the semi-structured document.

[0120] In an example, the unstructured document or the semi-structured document may include but is not limited to the first image.

[0121] The first image may include one or more images, and the data type of the first image may be in various situations, which are not limited herein. Exemplarily, the first image may be in a bitmap format (e.g., a bitmap (BMP), portable network graphics (PNG), joint photographic experts group (JPEG), etc.), a vector graphics format, etc.

[0122] An exemplary manner of obtaining design knowledge from the first image is described below.

[0123] In some embodiments, the design material includes a first image, and the method of obtaining the design knowledge corresponding to the first image specifically includes:

[0124] Through object detection and / or semantic segmentation, design knowledge in the first image is identified, where the design knowledge in the first image includes one or more of the following: entities in the first image, relationships between entities in the first image, and attributes of entities in the first image.

[0125] In the present application examples, reference Figure 4 In the example shown, the design information of the first image can be identified by means of target detection and / or semantic segmentation in computer vision (CV). Exemplarily, the entity in the first image may include one or more targets detected in the first image; and the relationship between the entities may be determined according to the relative positions between the entities; in addition, the attributes of the entity in the first image may include one or more of the following information of the entity in the first image: position, size (for example, it can be described by a bounding box such as a minimum circumscribed rectangular box of the entity), and direction.

[0126] For example, target detection can be used to identify information such as the category and position of the entity in the first image. There can be many specific ways of target detection, which are not limited here. Exemplarily, target detection can be achieved through machine learning models that can achieve target detection (such as convolutional neural networks (CNN), support vector machines (SVM), etc.) or traditional target detection methods (such as scale-invariant feature transform (SIFT) and random sample consensus (RANSAC)).

[0127] In addition, semantic segmentation can also be used to identify the category to which each pixel in the first image belongs. In other words, semantic segmentation can segment the first image into one or more regions and obtain the semantic category to which each of the one or more regions belongs. It can be seen that based on the semantic categories and segmented regions identified by semantic segmentation, the entities contained in the first image can be identified, the relationship between the entities can be determined based on the relative positions between the entities, and the attributes of the entities can be determined based on the positions of the entities and the information of the regions.

[0128] It can be seen that in the embodiments of the present application, design knowledge such as entities, relationships between entities, and / or attributes of entities in the first image can be identified through target detection and / or semantic segmentation.

[0129] In an example, the unstructured document or the semi-structured document may include but is not limited to the first text.

[0130] An exemplary manner of obtaining design knowledge from the first text is described below.

[0131] In the embodiment of the present application, named entity recognition can be performed on the first text to identify the named entities in the first text. Then, information such as the relationship between the named entities and the attributes of the named entities can be identified from the first text to obtain the entities in the first text, the relationship between the entities and / or the attributes of the entities as the design knowledge of the first text.

[0132] There are many named entity recognition methods, as well as ways of identifying the relationships between named entities and the attributes of named entities.

[0133] For example, in some examples, the large language model to be fine-tuned may be fine-tuned so that the fine-tuned large language model can recognize the design information of the first text.

[0134] Specifically, in some embodiments, the method further includes:

[0135] Fine-tuning the large language model to be fine-tuned according to the preset text and the label of the preset text, so as to obtain the large language model for design information recognition of the text after fine-tuning, wherein the label of the preset text includes one or more of the following information: entities in the preset text, relations between entities in the preset text, and attributes of entities in the preset text;

[0136] Information is extracted from the first text through a large language model used for identifying design information of the text to obtain design knowledge corresponding to the first text.

[0137] Specifically, Figure 4 In the example shown, the preset text may include small sample data for fully supervised learning in the design field, and the small sample data may correspond to a label, which is used to describe the entity of the small sample data.

[0138] In this way, the first fine-tuning of the large language model to be fine-tuned can be achieved based on the small sample data and its labels, so that the large language model after the first fine-tuning can recognize named entities in the text of the design field.

[0139] Then, weakly supervised data can be obtained, in which some data may correspond to labels, and the labels are used to describe the relationship and / or attribute information of the named entities in the part of the data. In this way, the large language model after the first fine-tuning can be fine-tuned for the second time according to the weakly supervised data and the labels corresponding to some of the data therein. In this way, the large language model after the second fine-tuning can realize the extraction of information such as entities in the text, relationships between entities, attributes of entities, etc. The large language model after the second fine-tuning can be considered as a large language model for design information recognition of text. In this way, when constructing a knowledge graph, the large language model used for design information recognition of text can be used to perform design information recognition on the first text to obtain the design knowledge in the first text.

[0140] It can be understood that in the embodiment of the present application, the large language model used for text design information recognition is different from the fine-tuning operation involved in the large language model used in the process of generating design codes. Therefore, the large language model used for text design information recognition can be different from the large language model used in the process of generating design codes. For example, the large language model used in the process of generating design codes can be obtained by further fine-tuning the large language model used for text design information recognition, or it can also be obtained by fine-tuning the large language model to be fine-tuned in other cases.

[0141] Alternatively, in some other examples, other existing or subsequently developed identification methods may be used to identify the design information in the first text.

[0142] (2) Mining implicit knowledge without documentation

[0143] In the embodiment of the present application, design knowledge that is not carried in documents can be considered as implicit knowledge.

[0144] In actual design scenarios, there are often a large number of historical design objects. The historical design objects can also be considered as historical design instances, that is, design objects obtained by performing historical design tasks. For example, in a three-dimensional design scenario, the historical design object (that is, the historical design instance) can be a three-dimensional model obtained by historical design, and in a user interface design scenario, the historical design object can be considered as a user interface obtained by historical design.

[0145] A large number of historical design objects usually contain a large amount of design knowledge, which can include personalized design knowledge in personalized design scenarios, and can also include common design knowledge obtained based on common design specifications and / or design constraints. In traditional knowledge graph construction, knowledge is usually extracted only from text, and it is difficult to extract knowledge from other forms of materials.

[0146] In the embodiment of the present application, implicit knowledge can be mined from historical design objects to enrich the knowledge in the knowledge graph and fully mine the design knowledge in the design field.

[0147] The following takes the first design object in the historical design objects as an example to introduce an exemplary method for mining implicit knowledge from historical design objects.

[0148] Specifically, in some embodiments, the design material includes a first design object, and the method includes:

[0149] According to the first design object, a first geometric feature of a first component among the one or more components included in the first design object is obtained, where the first geometric feature is used to describe features related to geometric elements in the first component;

[0150] Design knowledge corresponding to the first design object is obtained according to the first geometric feature.

[0151] The first design object may also be referred to as the first design instance, which may be considered as a historical design object, being a design object generated in a historical design task, for example, a three-dimensional model that completes a shape design, a three-dimensional model that completes an assembly design, or a user interface that completes a graphic design.

[0152] In the embodiment of the present application, the first geometric feature of the first component in the first design object can be designed.

[0153] The first geometric feature is used to describe features related to geometric elements in the first component. The geometric element may refer to a basic, indivisible object or entity existing in space. Exemplarily, the geometric element may include a primitive in a two-dimensional image or a geometric element in a three-dimensional image. The primitive in a two-dimensional image may include one or more of a point, a line, a surface, etc., and the geometric element in a three-dimensional image may include a sphere or a basic geometric body, etc.

[0154] The first geometric feature may include features of the geometric elements themselves in the first component, and may also include features of the structure composed of the geometric elements in the first component; and the first geometric feature may be described by relevant parameter data of the geometric elements, or may be described in a semantic form.

[0155] Exemplarily, the first geometric feature includes one or more of the following:

[0156] Spatial information of geometric elements in the first component, semantic information used to describe the first component and / or geometric elements in the first component, structural features of the first component, and gradient features of the first component.

[0157] The spatial information of the geometric elements in the first component may include one or more of the following: the position, direction, size, scaling, etc. of the geometric elements. The semantic information used to describe the first component and / or the geometric elements in the first component may describe the semantics of the first component and / or the geometric elements in the first component, for example, describing the name, form, meaning, etc. of the first component and / or the geometric elements in the first component. The structural features of the first component may include the contour of the first component and / or the characteristics of the internal structure of the first component, for example, may include shape information such as the symmetry of the first component, the relationship between the geometric elements in the first component (for example, parallel, vertical, intersection), etc. The gradient information of the first component may also be referred to as the sharp information of the first component, which is described according to the size of the normal gradient of the first component.

[0158] It can be seen that the first geometric feature of the first component can describe the first component at one or more levels, from the most basic geometric element of the first component to the structure composed of the geometric elements.

[0159] There may be a variety of ways to identify the first geometric feature of the first component. For example, the first geometric feature of the first component may be identified by a machine learning model such as a convolutional neural network (CNN), or by an algorithm such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), or features from accelerated segment test (FAST).

[0160] The first component may also have multiple instances.

[0161] Exemplarily, when the first design object is relatively simple, for example, including only one component or several components simply spliced ​​together, the one component or the several components simply spliced ​​together can be used as the first component.

[0162] Alternatively, the first design object may be decomposed to obtain part or all of the components in the first design object as the first components.

[0163] Specifically, in some embodiments, obtaining, according to the first design object, a first geometric feature of a first component among one or more components included in the first design object includes:

[0164] Decomposing the first design object to obtain one or more hierarchical components in the first design object;

[0165] From the components of one or more levels in the first design object, obtain a component of a target level as a first component;

[0166] A first geometric feature of a first component is obtained.

[0167] In the present application examples, reference Figure 4 In the example shown, the first design object can be decomposed from top to bottom to obtain one or more levels of components of the first design object, wherein the lowest level of the one or more levels of components can generally be spare parts, such as bolts, nuts and other structures, so as to decompose the first design object into components that are relatively common in the design field and are easy to describe using design knowledge in the design field.

[0168] Among them, after the first design object is decomposed, the topological hierarchy of each level of components after decomposition can be described by DAG, etc., and the DAG can be optimized by graph neural network (GNN), etc., and one or more levels of components in the first design object can be obtained after optimization.

[0169] After obtaining components of one or more levels, the component at the lowest level in the one or more levels can be used as the component of the target level, thereby obtaining the component at the lowest level (that is, the leaf node in the DAG) as the first component. Figure 4 In the example shown, in the DAG, the component corresponding to the leaf node can be obtained as the first component. It can be understood that the components corresponding to the leaf node can be assembled from the bottom up to obtain the first design object, so the component corresponding to the leaf node can describe the characteristics of the first design object, and the component corresponding to the leaf node is usually in the form of a more common component in the design field, so the component corresponding to the leaf node can be used as the first component to obtain the first geometric feature of the component corresponding to the leaf node to obtain the design knowledge in the first design object.

[0170] After obtaining the first geometric feature of the first component, an entity can be obtained based on the first component, and based on the first geometric feature of the first component, an attribute of the first component can be obtained. Information such as the relationship between the first components and / or the relationship between geometric elements in the first component or geometric bodies in the first component can also be obtained to construct a knowledge graph.

[0171] In addition, in some embodiments, since a single first design object is an independent design object and may contain some personalized or inappropriate designs, more commonly used target first geometric features can be screened out from multiple first design objects as design knowledge of the knowledge graph, thereby ensuring the reliability of the design knowledge in the knowledge graph.

[0172] Specifically, in some embodiments, the number of the first design objects is multiple, and obtaining design knowledge corresponding to the first design objects according to the first geometric feature includes:

[0173] Determine a target first geometric feature from among the first geometric features corresponding to the plurality of first design objects according to a co-occurrence relationship of the first geometric features corresponding to the plurality of first design objects;

[0174] According to the target first geometric feature, design knowledge corresponding to the first design object is obtained.

[0175] Co-occurrence refers to the frequency or probability of two things appearing in the same context.

[0176] In an embodiment of the present application, the co-occurrence relationship of a first geometric feature may be the frequency or probability of the first geometric feature and the first design object appearing at the same time in a plurality of first design objects. In some examples, the co-occurrence relationship of a first geometric feature may also be the frequency or probability of the first geometric feature and the first design object appearing at the same time in a plurality of first design objects of the same type. It can be seen that the strength of the co-occurrence relationship can be reflected by the co-occurrence probability. If the co-occurrence relationship indicates that the probability (i.e., the co-occurrence probability) of a first geometric feature and the first design object appearing at the same time is high (e.g., higher than a specified probability threshold), it can be considered that the first design object has a high probability of containing the first geometric feature, and the first geometric feature can be considered to be a more commonly used geometric feature in the first design object, and the first geometric feature can be determined to be the target first geometric feature among the first geometric features contained in the corresponding first design object, so that the design knowledge corresponding to the target first geometric feature in the first design object can be used as the more credible design knowledge corresponding to the first design object.

[0177] In some embodiments, in the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship.

[0178] In the embodiment of the present application, the weight can be determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship. For example, the weight can be equal to the corresponding co-occurrence probability, or can be obtained by calculating the corresponding co-occurrence probability according to a specified calculation method. Among them, the co-occurrence probability corresponding to the first geometric feature of the target refers to the probability of the first design object containing the first geometric feature of the target among multiple first design objects.

[0179] It can be seen that in the knowledge graph, a weight can be set for the design knowledge from the first design object (that is, the design knowledge corresponding to the first geometric feature of the target). The weight of the design knowledge corresponding to the first geometric feature of the target can also be considered as the confidence of the design knowledge corresponding to the first geometric feature of the target, which can reflect the credibility of the design knowledge corresponding to the first geometric feature of the target.

[0180] Moreover, as the design task is continuously executed, new first design objects can be continuously generated. Based on the new first design objects, the design knowledge and its weight corresponding to the target first geometric feature in the knowledge graph can be continuously updated, so that the design knowledge in the knowledge graph is continuously improved and more reliable, providing a better data foundation for users to use the design knowledge in the knowledge graph to perform design tasks.

[0181] 3. Joint representation of design knowledge from multimodal design materials

[0182] In an embodiment of the present application, design materials of one or more modalities can be obtained. Based on any of the above-mentioned knowledge mining embodiments, after the design knowledge corresponding to each design material is obtained, a knowledge graph can be constructed according to the design knowledge corresponding to each design material.

[0183] In some examples, the design material may be a multimodal design material, and the construction of a knowledge graph may be achieved through the joint representation of design knowledge from the multimodal design material.

[0184] Specifically, refer to Figure 5 In the example shown, a knowledge graph construction method in an embodiment of the present application may include steps 501-504.

[0185] Step 501, obtaining multi-modal design materials.

[0186] Step 502: Obtain design knowledge corresponding to the design material of each modality.

[0187] The specific method of acquiring the design knowledge corresponding to the design material of each modality can refer to the above-mentioned related embodiments of knowledge mining of design materials, which will not be repeated here.

[0188] Step 503, after obtaining the design knowledge corresponding to the design material of each modality, the design knowledge corresponding to the design material of each modality is jointly represented to obtain a target joint representation.

[0189] Step 504, construct a knowledge graph based on the target joint representation.

[0190] The joint characterization may involve design materials of at least two modalities. Different situations of design materials of at least two modalities are exemplarily introduced below.

[0191] Joint representation 1. Joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text.

[0192] Specifically, in some embodiments, the design knowledge corresponding to the design materials of each modality is jointly represented to obtain a target joint representation, including:

[0193] The design knowledge corresponding to the first design object and the design knowledge corresponding to the first text are jointly represented to obtain a target joint representation.

[0194] Exemplarily, the first text may include a description document of the first design object, or other text describing design information of the first design object, for describing constraints related to the first design object, semantic information of the first design object, relationships between components in the first design object, and other design information. In addition, the first text may also include text describing other design information, which is not limited in the embodiments of the present application.

[0195] When performing joint representation, the design knowledge corresponding to the first design object (such as the first geometric feature of the first design object and other information) can be feature encoded (embedding) to convert it into a vector form, and the design knowledge corresponding to the first text (such as entities extracted from the first text, relationships between entities, attributes of entities and other information) can also be feature encoded and converted into a vector form, so that the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text are mapped to the same feature space and described in the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text can be represented in the same feature space, that is, the joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text is achieved.

[0196] Joint representation 2: Joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image.

[0197] Specifically, in some embodiments, the first image includes a first sub-image related to the first design object, and the design knowledge corresponding to the design materials of each modality is jointly represented to obtain the target joint representation, including:

[0198] The design knowledge corresponding to the first design object and the design knowledge corresponding to the first sub-image are jointly represented to obtain a target joint representation.

[0199] Exemplarily, the first image may include images related to the first design object, such as photographs corresponding to the first design object, engineering drawings describing the first design object (such as two-dimensional computer aided design (CAD) drawings), and the like.

[0200] When performing joint representation, the design knowledge corresponding to the first design object can be feature encoded to be converted into a vector form, and the design knowledge corresponding to the first image (such as entities extracted from the first image, relationships between entities, attributes of entities, and other information) can also be feature encoded and converted into a vector form, so that the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image are mapped to the same feature space and described in the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image can be represented and fused in the same feature space, that is, the joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image is achieved.

[0201] Joint representation 3: Joint representation between the design knowledge corresponding to the first image and the design knowledge corresponding to the first text.

[0202] Specifically, in some embodiments, the first image includes a first sub-image related to the first design object, and the design knowledge corresponding to the design materials of each modality is jointly represented to obtain the target joint representation, including:

[0203] The design knowledge corresponding to the first image and the design knowledge corresponding to the first text are jointly represented to obtain a target joint representation.

[0204] Exemplarily, the first text may include a second sub-text related to the first image. For example, if one or more first images and certain second sub-texts are included in the same document, the design knowledge of the one or more first images and the second sub-texts is jointly represented, so that the design knowledge of the one or more first images and the second sub-texts can be integrated with the context in the document.

[0205] It can be understood that in the embodiment of the present application, the joint characterization of the design knowledge corresponding to the design materials of each modality may include one or more of the above three joint characterization methods.

[0206] For example, when multimodal design materials include a first design object, a first text, and a first image, the design knowledge corresponding to the design materials of the three modalities can be feature encoded respectively, so that the design knowledge corresponding to the design materials of the three modalities are converted into vector form, and the design knowledge corresponding to the design materials of the three modalities are jointly represented, thereby realizing the fusion of the design knowledge corresponding to the design materials of the three modalities.

[0207] In the embodiment of the present application, there may be various specific times for performing joint characterization.

[0208] For example, in some examples, after mining explicit knowledge (e.g., including design knowledge corresponding to the first image and / or design knowledge of the first text that is not related to the first design object, etc.), joint representation can be performed to obtain a preliminary sub-graph that records explicit knowledge, and after mining implicit knowledge (e.g., including design knowledge corresponding to the first design object and / or design knowledge of the first text that is related to the first design object, etc.), joint representation can be performed to obtain a preliminary sub-graph that records implicit knowledge. Then, a knowledge graph can be constructed based on the preliminary sub-graph that records explicit knowledge and the preliminary sub-graph that records implicit knowledge.

[0209] Alternatively, after obtaining the design knowledge corresponding to the design materials of all modalities, the design knowledge corresponding to the design materials of all modalities may be jointly represented to construct a knowledge graph.

[0210] In an embodiment of the present application, design knowledge can be acquired from multimodal design materials. The multimodal design materials include a first design object, which can be considered as a historical design object. After obtaining the first design object, the design knowledge implied in the first design object can be mined from the first design object based on geometric features, and the design knowledge mined from design materials of other modalities can be jointly represented, for example, by using target joint representations such as vector forms, so as to achieve the fusion of design knowledge of multimodal design materials and realize the construction of a knowledge graph. Moreover, since the knowledge graph can include the design knowledge mined from multimodal design materials, the constructed knowledge graph contains relatively rich design knowledge, providing a good data foundation for related applications of the knowledge graph.

[0211] 4. Knowledge Fusion

[0212] In the present application examples, reference Figure 4 In the example shown, after joint representation, the fusion of design knowledge corresponding to multimodal design materials can be achieved. For example, semantic fusion and / or structural fusion can be performed to obtain a fusion result.

[0213] Exemplarily, semantic fusion may include the fusion of identical descriptions in descriptions of entities, relationships, attributes, etc., unification of synonyms, etc., to achieve semantic alignment and fusion of design knowledge corresponding to multimodal design materials. Structural fusion may include the alignment and fusion of information such as relationships between entities and attributes of entities, etc., to achieve structural alignment and fusion of design knowledge corresponding to multimodal design materials.

[0214] In this way, a knowledge graph can be constructed in a specified structured form based on the constructed schema and fusion results, and can be stored in a graph database.

[0215] In addition, in the embodiments of the present application, the specific form of the knowledge graph can be in various situations.

[0216] For example, in some examples, the knowledge graph includes a first sub-graph and at least one second sub-graph, the first sub-graph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials, the common design knowledge including one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets specified conditions, and any second sub-graph is obtained according to one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

[0217] It can be seen that in this example, the knowledge graph can be in the form of 1+n. "1" indicates that the knowledge graph includes a subgraph describing common design knowledge (i.e., the first subgraph), and "n" indicates that the knowledge graph includes n subgraphs of design knowledge extracted from independent historical design objects (i.e., n second subgraphs), where n can be a positive integer, that is, the number of second subgraphs can be at least one.

[0218] Among them, the sources of common design knowledge in the first sub-graph may include one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects and whose co-occurrence probability in multiple first design objects meets specified conditions.

[0219] Common design knowledge can be considered as relatively common design knowledge in the design field, such as the matching relationship between bolt standard parts and nut standard parts. Therefore, at least one first text used to obtain common design knowledge may include text describing relatively standardized design knowledge in the design field, but does not include text describing personalized design in many scenarios. For example, in some scenarios, at least one first text may not include text related to the first design object, such as a description document of the first design object. In addition, at least one first image used to obtain common design knowledge may also be an image describing relatively common and standardized design knowledge. For example, at least one first image used to obtain common design knowledge may include a photograph of a standard part or a two-dimensional CAD drawing of a standard part; in addition, at least one first image used to obtain common design knowledge may also be an image located in the same document as the text describing relatively standardized design knowledge in the design field, so as to extract accurate design knowledge from the text and image in the document in combination with the contextual information of the document as the common design knowledge in the first sub-atlas.

[0220] Furthermore, in some examples, the common design knowledge may include design knowledge from a plurality of first design objects and whose co-occurrence probability in the plurality of first design objects satisfies a specified condition.

[0221] In the embodiment of the present application, the co-occurrence probability may be higher than a higher probability threshold to ensure that the design knowledge is very common design knowledge among the multiple first design objects. In addition, in some examples, the number of the multiple first design objects may also reach a specified number threshold to ensure that the design knowledge from the multiple design objects is extracted from a sufficient number of first design objects to ensure the universality of the design knowledge. At this time, the very common design knowledge from the multiple first design objects may also be used as the common design knowledge in the first sub-graph.

[0222] The second sub-graph may be design knowledge acquired from one or more historical independent first design objects.

[0223] In traditional knowledge graphs, it is difficult to extract effective design knowledge from the first design object. In the embodiments of the present application, design knowledge with a high degree of credibility can be extracted from one or more first design objects, and other design materials associated with one or more first design objects (for example, first texts associated with one or more first design objects, first images associated with one or more first design objects, etc.) based on geometric features, co-occurrence relationships, etc., as the design knowledge in the knowledge graph. In addition, the design knowledge from the first design object and related design materials can be stored through one or more second sub-graphs to distinguish it from the common design knowledge in the first sub-graph. In this way, as the design task is continuously executed, the design knowledge in the second sub-graph can be conveniently updated according to the newly generated first design object, and the user can select design knowledge that meets the needs from the first sub-graph and the second sub-graph according to actual needs to perform the design task.

[0224] When there are multiple second sub-graphs, the division methods of different second sub-graphs can be various, which are not limited here. Exemplarily, different second sub-graphs can record design knowledge of different types of first design objects respectively.

[0225] It can be seen that through the first sub-graph and the second sub-graph, the design knowledge from different sources and with different confidence levels can be stored, applied, updated and managed separately.

[0226] In other examples, the knowledge graph can also have other forms. For example, the knowledge graph can include sub-graphs corresponding to different design types; or the design knowledge corresponding to each design material can also be integrated into the same knowledge graph. Based on the above method of constructing the knowledge graph, a knowledge graph can be constructed for subsequent design code generation tasks. In addition, Figure 4 In the example shown, the knowledge graph can also be updated and iterated based on expert knowledge or the execution results of subsequent design tasks based on the generated design code, for example, the weight of the corresponding design knowledge in the knowledge graph can be updated according to the specific circumstances of the target design object obtained by the design task.

[0227] In traditional knowledge graph construction, knowledge is usually extracted only from text, while it is difficult to extract knowledge from other forms of materials.

[0228] It is understood that in the embodiment of the present application, the knowledge graph can be constructed by the cloud management platform and deployed in the infrastructure managed by the cloud management platform. Alternatively, the knowledge graph can also be constructed by other devices, and then transmitted and deployed to the infrastructure managed by the cloud management platform.

[0229] 2. Large Language Model Enhancement and Constraints for Design Code Generation Tasks

[0230] In some embodiments, in order to perform subsequent design code generation tasks, the large language model may be enhanced and constrained in advance, so that the enhanced and constrained large language model can perform the design code generation task more accurately.

[0231] Based on the different scenario requirements of the subsequent design code generation task, there may be many ways to reason about the enhanced and constrained large language model in the design code generation task. Accordingly, there may be many specific forms of the training data and its labels used to enhance and constrain the large language model, which are exemplarily introduced below.

[0232] Specific form of training data and its labels 1: The training data includes preset design intent and preset design knowledge, and the labels of the training data include preset design codes.

[0233] In this example, the specific method of generating the preset design intent of any training data can refer to the method of generating the design intent in the relevant embodiments of the design code generation part based on the knowledge graph and the large language model; alternatively, the preset design intent may come from a third party; alternatively, the preset design intent may also be manually configured by the user in advance.

[0234] The preset design knowledge in the training data may be design knowledge related to the preset design intent retrieved from the knowledge graph; or, the preset design knowledge may be manually configured by the user in advance.

[0235] The label of the training data may include a preset design code, and the preset design code can be used as an execution script of the design software, but does not include information of the preset design code module in subsequent examples. That is to say, the preset design code does not perform design by calling at least one design code module, but includes an execution script of specific execution instructions.

[0236] The specific type of the preset design code is not limited here, and can be determined based on the needs of the design software that performs the design task. Exemplarily, the preset design code can be a domain specific language (DSL). Specifically, in some examples, since python has become a common secondary development script for many design software, the preset design code can be a python script.

[0237] Specific form 2 of training data and its labels: The training data includes preset design intent and preset design knowledge, the label of the training data includes preset triple information and the module identifier of at least one preset design code module, and the preset triple information includes the operation object, operation instruction and operation parameter corresponding to at least one preset design code module.

[0238] In this example, the module identification and specific content of the design code module can be written in advance, and the calling interface corresponding to the design code module can be created.

[0239] For example, a developer can deconstruct the historical design task of implementing the design of the historical design object, obtain multiple design subtasks for designing the historical design object, and then write a design code module for one or more of the design subtasks, so that any design code module can execute the corresponding design subtask and realize the specified design function. It can be understood that the design code module can also be considered as a function.

[0240] In addition, in some examples, the design code module may be an atomic module, that is, any design code module may be decomposed into the smallest, indivisible code module. In this way, when subsequently generating the design code, it is convenient to flexibly use each design code module and realize flexible combination of each design code module.

[0241] After writing the design code module, a calling interface corresponding to the design code module can be created to facilitate calling the design code module through the calling interface. In addition, the module identifier of each design code module (such as the name of the design code module, etc.) can also be determined to uniquely identify the corresponding design code module. Moreover, the calling information of the design code module can be normalized into the form of triple information. Specifically, the triple information may include an operation object, an operation instruction, and an operation parameter. In this way, the large language model can be instructed to implement learning from natural language to code generation through the structure of the object, predicate, and object complement of the triple information in the label, so that the large language model after enhancement and constraint has the ability to identify the module identifier of at least one preset design code module required for the corresponding design task according to the training data, and can extract the preset triple information according to the training data, so as to implement the interface call of at least one preset design code module.

[0242] In this example, the label of the training data may include preset triple information and a module identifier of at least one preset design code module, but does not include an execution statement, so it cannot be directly used as an execution script.

[0243] Specific form 3 of the training data and its labels: the training data includes preset design intent and preset design knowledge, and the labels of the training data include a preset design code that can call at least one design code module through preset triple information and a module identifier of at least one preset design code module.

[0244] In the embodiment of the present application, the label of the training data includes a preset design code, and the preset design code includes not only preset triple information and the module identifier of at least one preset design code module, but also an execution statement, and therefore, can be used as an execution script. It can be seen that in the preset design code, at least one preset design code module can be called according to the preset triple information and the module identifier of at least one preset design code module.

[0245] The large language model enhanced and constrained by different forms of the above-mentioned training data and its labels may have different reasoning methods in the design code generation task. The specific reasoning method can be referred to the introduction of the subsequent design code generation task related embodiments, which will not be repeated here.

[0246] In an embodiment of the present application, after obtaining multiple sets of training data and their labels, the large language model to be enhanced can be enhanced and constrained by back propagation or the like based on the multiple sets of training data and their labels, so that the large language model obtained after enhancement and constraint can generate corresponding design code according to the input design intent.

[0247] In one example, a multi-agent framework can be used to enhance and constrain the large language model to be enhanced.

[0248] For example, Figure 6 The example shown is an exemplary schematic diagram of the multi-agent framework.

[0249] In the multi-agent framework, one or more of intent understanding, problem deconstruction, large language model, knowledge graph, feedback model, etc. may be included. The process of enhancing and constraining the large language model to be enhanced includes at least one iterative process, and the i-th iterative process in at least one iterative process is taken as an example for introduction below.

[0250] Exemplarily, the i-th iteration process in at least one iteration process includes: reasoning on the training data through the large language model to be enhanced in the i-th iteration process to obtain output data; evaluating whether the output data meets the specifications and / or constraints through the knowledge graph to obtain an evaluation result; and updating the large language model to be enhanced in the i-th iteration process according to the difference between the output data and the label, and the evaluation result.

[0251] During the enhancement and constraint process, the information of the preset design intent input by the user can be understood, and the preset design knowledge corresponding to the preset design intent can be queried from the knowledge graph to obtain the preset design intent and the preset design knowledge. In some examples, in addition, the user's design intent can be deconstructed to obtain one or more design subtasks, each of which can be implemented by a design code module. In this way, the preset design intent and preset design knowledge corresponding to each design subtask can be obtained as training data. Then, the training data containing the preset design intent and preset design knowledge corresponding to the design subtask obtained by problem deconstruction can be used as input data and input into the large language model to be enhanced to obtain the output data of the large language model to be enhanced.

[0252] The large language model can query the knowledge graph during the reasoning process or after obtaining the output data. For example, the large language model can query the knowledge graph for the specifications and / or constraints of the specified design during the reasoning process; in addition, after obtaining the output data, the large language model can query the knowledge graph whether the output data meets the specifications and / or constraints to obtain an evaluation result of whether the output data meets the specifications and / or constraints. Among them, the specifications can include the design standards stored in the knowledge graph, which can also be considered as guiding conditions, which are usually more objective and unified. Constraints refer to restrictive conditions, which can be described in the form of functions or inequalities.

[0253] In this way, according to the difference between the output data and the label, as well as the evaluation result, the large language model to be enhanced in the i-th iteration process is back-propagated to update the weight of the large language model to be enhanced in the i-th iteration process.

[0254] When the number of iterations reaches the specified number, or when the loss value of the output data of the large language model to be enhanced converges to the expected state, the enhancement and constraint of the large language model can be completed, and the large language model can be obtained for subsequent design code generation tasks.

[0255] In some examples, after applying the large language model to the actual application scenario to perform the design code generation task to obtain the target design object and output it, it can also be based on the feedback information of the expert on the target design object, etc. Figure 6 The feedback model in the example shown further enhances and constrains the large language model.

[0256] Specifically, the expert can evaluate the accuracy of the target design object and obtain feedback information of the target design object. Then, the feedback model can query the knowledge graph whether the feedback information meets the specifications and / or constraints, and obtain the target feedback information based on the corresponding query results. Of course, in some examples, the feedback information on the target design object input by the expert can also be directly used as the target feedback information. After obtaining the target feedback information, the feedback model can use the target feedback information to perform reinforcement learning on the large language model to further enhance and constrain the large language model in the application process, and continuously improve the performance of the large language model.

[0257] It can be understood that in the embodiments of the present application, the enhancement and constraint of the large language model can be implemented by executing the infrastructure managed by the cloud management platform of the embodiments of the present application, or it can be implemented in other devices and then deployed to at least one server of the infrastructure.

[0258] 3. Design code generation based on knowledge graph and large language model

[0259] After building the knowledge graph and obtaining the fine-tuned large language model, the design code generation task can be performed through the knowledge graph and the large language model. For example, Figure 2 In the example shown, the design code generation service of the cloud management platform performs the design code generation task to implement the design code generation method.

[0260] Specifically, Figure 7 As shown, the cloud service-based design code generation method may include steps 701-703.

[0261] Step 701, providing a configuration interface.

[0262] The configuration interface is used to obtain the design intent information input by the user, and the design intent is used to instruct the generation of a target design object.

[0263] In an embodiment of the present application, the cloud management platform may provide a configuration interface to the user's client, so that the user may input information of the design intent through the configuration interface displayed by the client.

[0264] Exemplarily, the information of the design intent may include one or more of the following: a second text, a second design object, and a second image. Accordingly, the design intent may include one or more of the following: an intent obtained based on the second text, an intent obtained based on the second design object, and an intent obtained based on the second image.

[0265] In this way, after obtaining one or more of the second text, the second design object, and the second image through the configuration interface, the design intent can be obtained from one or more of the second text, the second design object, and the second image. Since the design intent is obtained based on the information input by the user, it can be considered that the design intent is used to describe the user's explicit design intent for the target design object.

[0266] Since the specific content of the information on the design intent may be in many situations, there may also be many ways to obtain the design intent.

[0267] For example, in some examples, the design intent includes an intent derived based on the second design object, and the method further includes:

[0268] Acquire a second geometric feature of a second component associated with a target design object in one or more components included in the second design object, where the second geometric feature is used to describe features related to geometric elements in the second component;

[0269] According to the second geometric feature, an intention based on the second design object is obtained.

[0270] The second design object input by the user may be in various situations. For example, the second design object may be a design object indicated by the user as a reference; or, the second design object may be a design object for a design task, for example, if the design task to be performed is an assembly design task, the second design object may be an object to be assembled. In addition, the number of the second design objects may be one or more, which is not limited here.

[0271] In some embodiments, obtaining a second geometric feature of a second component associated with a target design object among one or more components included in the second design object includes:

[0272] Decomposing the second design object to obtain one or more hierarchical components in the second design object;

[0273] From the components of one or more levels in the second design object, obtain a component of a target level as a second component;

[0274] A second geometric feature of the second component is obtained.

[0275] In some embodiments, the second geometric feature includes one or more of the following:

[0276] Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0277] In the embodiment of the present application, the specific content of the second geometric feature and the specific method of obtaining the second geometric feature can refer to the specific content of the first geometric feature and the specific method of obtaining the first geometric feature in any of the above embodiments, and will not be repeated here. In addition, in the embodiment of the present application, the process of decomposing the second design object to obtain the second geometric feature of the second component can also be understood as problem deconstruction.

[0278] After obtaining the information of the second geometric feature of the second design object, the second design object input by the user may be described by the information of the second geometric feature, thereby obtaining the intention based on the second design object.

[0279] In some embodiments, the design material includes a second image, the method further comprising:

[0280] Identify design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image;

[0281] According to the design information in the second image, the intention obtained based on the second image is obtained.

[0282] In the embodiment of the present application, any second image may be an independent image; or, it may be an image related to the second design object, for example, it may be a photo of the second design object, a two-dimensional CAD drawing, etc.; or, it may be an image associated with the second text, for example, it may be located in the same document as the second text. The method for obtaining the design information of the second image may refer to the method for obtaining the design knowledge of the first image in the above embodiment, and will not be repeated here.

[0283] In some embodiments, the design material includes a second text.

[0284] The second text may be a description text input by a user, and is used to describe design requirements. For example, it may describe information such as a target design object to be designed.

[0285] In some examples, the intent of the second text may also be understood through a machine learning model or other algorithms to mine the design intent contained in the second text.

[0286] Specifically, in some embodiments, the design material includes a second text, and the method further includes:

[0287] Information extraction is performed on the second text using a large language model for information extraction or other information extraction algorithms to obtain design information corresponding to the second text.

[0288] Exemplarily, a large language model for information extraction or other information extraction algorithms can be used to decompose a complex design task into multiple design sub-tasks based on a tree of thoughts (ToT) and / or a chain of thought (CoT), thereby achieving problem deconstruction to obtain the intent based on the second text. The context of the second text can also be used to understand the intent to obtain the intent based on the second text.

[0289] In other examples, the second text may be directly used as the intent obtained based on the second text.

[0290] In addition, in some embodiments, when the information input by the user includes data of at least two modalities (for example, at least two of a second text, a second design object, and a second image), a joint representation may be performed in order to achieve a fusion of the intentions corresponding to the two modalities.

[0291] For example, in some embodiments, the design intent includes an intent obtained based on the second design object and an intent obtained based on the second text, and obtaining the design intent may include:

[0292] Performing joint representation on the second geometric feature of the second component in the second design object and the design information in the second text to obtain a first joint representation;

[0293] According to the first joint representation, a first joint representation intention is obtained, where the first joint representation intention includes an intention obtained based on the second design object and an intention obtained based on the second text.

[0294] In some embodiments, the design intent includes an intent obtained based on the second design object and an intent obtained based on the second image, and obtaining the design intent may include:

[0295] performing joint characterization on a second geometric feature of a second component in a second design object and design information in the second image to obtain a second joint characterization;

[0296] According to the second joint representation, a second joint representation intention is obtained, where the second joint representation intention includes an intention obtained based on the second design object and an intention obtained based on the second image.

[0297] In some embodiments, the design intent includes an intent obtained based on the second text and an intent obtained based on the second image, and obtaining the design intent may include:

[0298] Performing joint representation on the design information in the second text and the design information in the second image to obtain a third joint representation;

[0299] According to the third joint representation, a third joint representation intent is obtained, where the third joint representation intent includes an intent obtained based on the second text and an intent obtained based on the second image.

[0300] In an embodiment of the present application, the first joint representation, the second joint representation and the third joint representation may be in vector form to describe the intent information of data from different modalities in a unified manner in the same feature space, thereby facilitating subsequent unified processing.

[0301] The specific generation method of the above-mentioned first joint representation, second joint representation and third joint representation can refer to the relevant description in the above-mentioned related embodiments of jointly representing the design knowledge corresponding to the design materials of each modality, and will not be repeated here.

[0302] It can be seen that in the embodiments of the present application, based on different situations of information input by the user, the specific content of the design intent may have multiple situations.

[0303] In addition, the specific form of the design intent may also be in various situations. For example, when the design intent only includes the intent obtained based on the second text, the design intent may be in text form. When the design intent includes the intent obtained from data of multiple modalities, the design intent may be in vector form.

[0304] Furthermore, in some examples, the design intent may also be in the form of a triple, which includes an operation object, an operation instruction, and an operation parameter. The information of the operation object, the operation instruction, and the operation parameter in the design intent may be described in the form of text or the like, or in the form of a vector. It can be seen that in this example, the design intent includes triple information obtained based on one or more of the second text, the second design object, or the second image.

[0305] Step 702, retrieve the target design knowledge corresponding to the design intent from the knowledge graph.

[0306] The knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image. The target design knowledge is used to describe the implicit design intention for the target design object.

[0307] After obtaining the design intent, the target design knowledge corresponding to the design intent can be retrieved from the knowledge graph.

[0308] The knowledge graph may be a knowledge graph constructed by any of the above-mentioned embodiments of knowledge graph construction, and therefore, the knowledge graph may include one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. In this way, relatively comprehensive target design knowledge matching one or more of the intents obtained based on the second text, the intent obtained based on the second design object, and the intent obtained based on the second image can be retrieved from the knowledge graph.

[0309] For example, in some examples, in traditional knowledge graph construction solutions, it is difficult to extract and store design knowledge from historical design objects (i.e., the first design object), and in actual application processes, it is also impossible to extract design intent from the second design object.

[0310] In the knowledge graph of any embodiment of the present application, design knowledge can be extracted and stored from the historical design objects. In this way, the user can input the second design object through the configuration interface, and the cloud management platform can extract the intent based on the second design object from the second design object input by the user, and query the relevant target design knowledge from the knowledge graph.

[0311] Target design knowledge is used to describe the implicit design intentions of the target design object. In other words, the knowledge graph can be used to further explore more comprehensive design intentions that cannot be directly extracted from the user's input information based on the linear design intention information input by the user, so as to guide the large language model to generate reasonable target design codes in a more comprehensive manner in the future.

[0312] For instance, in one example, the design intent information input by the user indicates that the user is designing a car. Then, design knowledge such as design specifications and constraints related to car design can be mined from the knowledge graph as target design knowledge. These target design knowledge are not included in the user's input information, but can be retrieved through the knowledge graph, so that in subsequent steps, the large language model can be guided to generate target design codes that can more reasonably design the car.

[0313] Step 703: Process the design intent and target design knowledge through the large language model to obtain the target design code.

[0314] The target design code is used to generate the target design object.

[0315] In an embodiment of the present application, the design intent and target design knowledge may be input into a large language model, thereby invoking the large language model to process the design intent and target design knowledge and output a target design code.

[0316] The specific type of the target design code is not limited here and can be determined based on the needs of the design software that performs the design task. Exemplarily, the target design code can be a domain specific language (DSL). Specifically, in some examples, since python has become a common secondary development script for many design software, the target design code can be a python script.

[0317] The target design code can be called by the design software that executes the design task to generate executable code to execute the corresponding design task and obtain the target design object as the design result.

[0318] In the embodiments of the present application, there may be many possible situations for the specific reasoning process of obtaining the target design code through the large language model, which are exemplarily introduced below.

[0319] Reasoning method 1: Design intent and target design knowledge are input into the large language model for processing, so that the large language model outputs non-modularized target design code.

[0320] In this example, the large language model can output the target design code as the execution script of the design software through a single call. In addition, in this example, the design code module in the subsequent reasoning method 2 is not obtained in advance. Therefore, in the embodiment of the present application, the target design code does not include information of at least one design code module, that is, the design is not performed by calling at least one design code module, but a specific execution script including the content of specific execution instructions can be directly generated as the target design code.

[0321] The large language model in this example can be obtained by fine-tuning using the first type of training data and its labels in the related embodiments of the large language model fine-tuning described above.

[0322] Reasoning method 2: Design intent and target design knowledge are input into a large language model for processing, so that the large language model outputs a target design code containing target triple information and information of at least one design code module.

[0323] Specifically, in this example, the target design code includes target triplet information and information of at least one design code module, the at least one design code module is used to generate a target design object, and the target triplet information includes operation objects, operation instructions and operation parameters corresponding to the at least one design code module.

[0324] In an embodiment of the present application, the design code module may be pre-configured. For details, reference may be made to the relevant embodiments of large language model fine-tuning, which will not be described in detail here.

[0325] The at least one design code module can also be considered as a function. Each design code module can provide an interface for calling, and when calling, the triplet of information of the operation object, operation instruction and operation parameter corresponding to the design code module can be used as the input information of the interface, thereby instructing the design code module to implement the corresponding design according to the information of the operation object, operation instruction and operation parameter corresponding to the design code module.

[0326] The target design code may include target triple information and information of at least one design code module in a variety of specific forms. The specific forms of the target design code and the corresponding reasoning methods are introduced below.

[0327] Reasoning method 2.1: By calling the large language model twice, a target design code capable of making an interface call to at least one design code module is obtained.

[0328] Specifically, in some embodiments, step 703 includes:

[0329] Processing the design intent and target design knowledge through a large language model to obtain a target module identifier and target triple information, wherein the target module identifier includes a module identifier corresponding to at least one design code module;

[0330] According to the target module identifier and target triple information, the target design code is obtained through the large language model.

[0331] In an embodiment of the present application, the design intent and target design knowledge can be processed by a large language model to obtain a target module identifier and target triplet information. In this way, the large language model can determine at least one design code module required for the current design task from a plurality of pre-configured design code modules based on the design intent and target design knowledge (that is, determine the target module identifier of the at least one design code module), and can determine the target triplet information required to execute the at least one design code module. Among them, the module identifier corresponding to a certain design code module can uniquely identify the corresponding design code module. Exemplarily, the module identifier corresponding to a certain design code module can be the name or number of the design code module, etc.

[0332] Then, the large language model can be called for a second time to process the target module identifier and the target triple information through the large language model to generate a target design code, which can be used to call the interface of at least one design code module according to the target triple information to achieve the design task. It can be seen that the target design code can indicate the execution order of at least one design code module.

[0333] For example, in an exemplary scenario, the user's design intent for automobile design can be obtained through the configuration interface, and the target design knowledge for automobile design corresponding to the design intent can be queried in the knowledge graph. Then, the design intent and the target design knowledge can be input into the large language model so that the large language model outputs the target module identifier and target triple information of at least one design code module required for automobile design. For example, at least one design code module can include a design code module for designing the engine of the car, a design code module for designing the chassis of the car, and a design code module for designing the appearance of the car. Then, the target module identifier and the target triple information can be processed by the large language model to generate a target design code. In the target design code, it can be indicated that the interface of the design code module for designing the engine of the car is first called to design the engine of the car, and then the interface of the design code module for designing the chassis of the car is called to design the chassis of the car, and then the interface of the design code module for designing the appearance of the car is called to design the appearance of the car, so as to reasonably design the car from the inside out, avoid design conflicts, and cause unreasonable design results.

[0334] In the embodiment of the present application, the design code generation task can be executed hierarchically through two calls of the large language model, so that the generation logic of the execution design code is clearer, thereby ensuring the accuracy of the target design code.

[0335] Reasoning method 2.2: Through a call to a large language model, a target design code including target triple information and information of at least one design code module is obtained, but the target design code does not include execution statements and therefore cannot be directly used as an execution script.

[0336] Specifically, in some embodiments, step 703 includes:

[0337] The design intent and target design knowledge are processed through a large language model to obtain the target module identifier and target triplet information as the target design code.

[0338] In this example, the design software can obtain the target triplet information and information of at least one design code module in the target design code to generate an execution script, thereby obtaining executable code according to the generated execution script to perform the design task and obtain the target design object as a design result.

[0339] In the examples of reasoning method 2.1 and reasoning method 2.2, the large language model used can be obtained by fine-tuning using the second training data and its labels in the related embodiments of the above-mentioned large language model fine-tuning.

[0340] Reasoning method 2.3: By calling a large language model once, a target design code capable of making an interface call to at least one design code module is obtained.

[0341] In this example, the design intent and target design knowledge are input into the large language model, so that the large language model can directly output the interface that can call at least one design code module to achieve the target design code of the corresponding design task. It can be seen that the target design code can indicate the execution order of at least one design code module. In addition, the target design code can be executed as an execution script of the design software.

[0342] In this example, the large language model used can be obtained by fine-tuning using the third type of training data and its labels in the related embodiments of the large language model fine-tuning described above.

[0343] It can be seen that in the embodiment of the present application, not only can the explicit design intent input by the user be obtained from the configuration interface, but the target design knowledge associated with the user's current explicit design intent can also be queried from the knowledge graph to capture the implicit design intent for the target design object.

[0344] In this way, the explicit design intent and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate target design codes that meet the user's design needs, thereby realizing the intelligent generation of target design codes, so as to accurately and efficiently implement design tasks according to the target design codes and obtain the target design objects.

[0345] 4. Call the generated design code for design

[0346] In the embodiment of the present application, after obtaining the target design code, the generated target design code can be called by corresponding design software to perform design to obtain the target design object.

[0347] It should be noted that in the embodiment of the present application, it may be a cloud service provided by a cloud management platform (for example Figure 2 In the design scheme generation service shown in the figure, the target design code generated by calling the corresponding design software is designed to obtain the target design object and then output it to the user; or, the cloud management platform may obtain the target design code and then output it to the user, so that the user can design by calling the generated target design code through the design software on the client to obtain the target design object.

[0348] During the execution of a specific design task, the design software may call the generated target design code to determine the components required to generate the target design object.

[0349] For example, in some examples, when the target design code is called to design the target design object, if the target design code indicates to operate a certain component in the target design object, the component can be determined as the component required to generate the target design object, and the component required to generate the target design object can be retrieved from the preset component database by nearest neighbor component retrieval and other methods. The preset component database can store multiple preset components, and any preset component can be in the form of a common component, etc., so that it can be used in various design tasks. At this time, the preset component can be considered as a standard part. Of course, in some examples, the preset component database can also include other components besides standard parts, which is not limited here.

[0350] Exemplarily, preset components matching the components required to generate the target design object may be retrieved from the preset component database by means of structural topology or the like.

[0351] If a matching preset component is retrieved from the preset component database, the matching preset component can be obtained as a component required for generating the target design object.

[0352] Alternatively, in some examples, when the target design code is called to design the target design object, if the target design code indicates to operate on the target component in the first object that does not belong to the preset component database, that is, the preset component matching the target component is not retrieved in the preset component database, then based on the component parameters of the target component in the target design code, a target preset component whose similarity with the target component meets a specified condition is retrieved from multiple preset components in the preset component database. Exemplarily, the similarity meeting the specified condition can be that among multiple preset components in the preset component database, the topological structure is consistent with the target component and the similarity with the target component is the highest, or the similarity is higher than a specified similarity threshold, etc.

[0353] Then, shape fitting can be performed according to the target preset component and component parameters of the target component to generate the target component so as to operate the target component according to the target design code.

[0354] Specifically, the target preset component can be used as a benchmark for the target component. When the topological structure of the target preset component is consistent with that of the target component, the difference in dimension between the target component and the target preset component can be obtained based on the topological structure, and shape fitting can be performed based on the difference and related triplet information, so as to obtain a target component that meets the shape requirements in the target design code as the component required to generate the target design object.

[0355] After obtaining the components required to generate the target design object, the design software can call the generated target design code, and execute the design instructions according to the topological hierarchy (such as DAG) of the components required to generate the target design object, from bottom to top (such as from components to higher-level components and then to the target design object), according to the design parameters in the target triplet information, etc., to generate the target design object.

[0356] After the target design object is generated, the target design object may be optimized and then the optimized target design object may be output to the user.

[0357] For example, in some embodiments, after obtaining the target design code, the method further includes:

[0358] Obtaining a target design object obtained by calling the target design code for design;

[0359] From the knowledge graph, query the global constraint information corresponding to the target design object;

[0360] Optimize the target design object based on the global constraint information.

[0361] In the embodiment of the present application, during the design process, the target design code usually indicates to design from bottom to top and from local to global, and the target design object obtained by design is usually composed of a combination of multiple components.

[0362] Therefore, in order to ensure the rationality of the target design object, the large language model can query the knowledge graph for the global constraint information corresponding to the target design object. Then, the large language model can be used to detect whether the target design object satisfies the queried global constraint information. If not, the target design object can be adjusted according to the global constraint information to obtain an optimized target design object.

[0363] In addition, in some examples, the cloud management platform may also receive user feedback information on the generated target design object to determine an optimization strategy for the large language model and / or knowledge graph based on the feedback information, thereby optimizing the large language model and / or knowledge graph.

[0364] Exemplarily, reinforcement learning can be used to optimize the large language model, and the target design object and its related information can be referenced to the relevant schemes for constructing the knowledge graph to optimize the knowledge graph. For example, the weight corresponding to the corresponding design knowledge in the knowledge graph can be updated according to the design knowledge adopted by the target design object. For example, if the user's feedback information indicates that the target design object is an accurate design object, the weight corresponding to the design knowledge adopted by the target design object in the knowledge graph can be increased; and if the user's feedback information indicates that the design of a certain component in the target design object is inaccurate, the weight corresponding to the design knowledge associated with the design of the component in the knowledge graph can be reduced.

[0365] like Figure 8 The figure shows an exemplary flow chart of an embodiment of the present application.

[0366] Among them, Figure 8 In the example shown, the design task may be an assembly design task.

[0367] In this exemplary scenario, the intent understanding module may obtain the design intent, which may include one or more of the following: intent obtained based on the second text, intent obtained based on the second design object, and intent obtained based on the second image.

[0368] Then, the intent understanding module can query the target design knowledge corresponding to the design intent from the knowledge graph, thereby inputting the target design knowledge and design intent into the large language model.

[0369] The large language model processes the target design knowledge and design intent, obtains the target design code and outputs it to the design software.

[0370] The design software may call the target design code to perform one or more steps such as component matching, shape fitting, assembly parameter determination, and assembly to obtain an assembly solution as a target design object.

[0371] Then, the assembly scheme can be globally optimized. If the user determines that the optimized assembly scheme meets the requirements through the specified configuration interface, the optimized assembly scheme can be output; if the user determines that the optimized assembly scheme does not meet the requirements through the specified configuration interface, the acquisition of design intent and subsequent steps can be re-executed to regenerate the design code to obtain a new assembly scheme.

[0372] Below, taking the vehicle styling design scenario as an example, an exemplary information interaction diagram of an embodiment of the present application is introduced.

[0373] In one example, if Fig. 9As shown, the vehicle can be styled, and the styling design system may include styling design software. Exemplarily, the styling design system may be located on a cloud management platform.

[0374] The example may specifically include the following steps:

[0375] 1. The user inputs data such as text, design objects and / or images into the modeling design system;

[0376] 2. The modeling design system pre-processes the data input by the user, such as data decomposition and feature recognition.

[0377] 3. The large language model understands the intent of the pre-processed user input data and obtains the design intent;

[0378] 4. The large language model retrieves design knowledge related to design intent from the knowledge graph;

[0379] 5. The knowledge graph returns the target design knowledge related to the design intent to the large language model;

[0380] 6. The large language model generates target design code based on design intent and target design knowledge;

[0381] 7. The large language model returns the target design code to the modeling design software of the modeling design system;

[0382] 6. The modeling design software calls the target design code and generates a preliminary design plan as the target design object;

[0383] 7. The large language model queries the knowledge graph for global constraint information of the preliminary design plan;

[0384] 8. The knowledge graph returns global constraint information to the large language model;

[0385] 9. The large language model globally optimizes the preliminary design scheme according to the global constraint information to obtain the optimized design scheme;

[0386] 10. The large language model returns the optimized design solution to the modeling design system;

[0387] 11. The modeling design system outputs the optimized design plan to the user;

[0388] 12. The user sends feedback information on the optimized design plan to the modeling design system;

[0389] 13. The modeling design system uses feedback information to formulate optimization strategies for large language models and knowledge graphs;

[0390] 14. Optimize large language models;

[0391] 15. Optimize the knowledge graph.

[0392] In another example, the vehicle may be designed first, and then assembly design may be performed based on the design results. The design system may include design software, and the assembly design system may include assembly design software. For example, the design system and assembly design system may be located on a cloud management platform. Fig.10 As shown, the example may specifically include the following steps:

[0393] 1. User 1 (e.g., a styling designer) inputs first input data such as text, design objects, and / or images into the styling design system;

[0394] 2. The modeling design system pre-processes the data input by the user, such as data decomposition and feature recognition.

[0395] 3. The large language model used for modeling design understands the intent of the pre-processed user input data and obtains the design intent;

[0396] 4. The large language model used for modeling design retrieves design knowledge related to design intent from the knowledge graph;

[0397] 5. The knowledge graph used for modeling design returns the first target design knowledge related to the design intent to the large language model;

[0398] 6. The large language model for modeling design generates the first target design code according to the design intention and the first target design knowledge;

[0399] 7. The large language model for modeling design returns the first target design code to the modeling design software of the modeling design system;

[0400] 6. The modeling design software calls the first target design code to generate a preliminary modeling design plan;

[0401] 7. The large language model for modeling design queries the knowledge graph for modeling design for global constraint information of the preliminary modeling design solution;

[0402] 8. The knowledge graph for modeling design returns global constraint information to the large language model for modeling design;

[0403] 9. The large language model used for modeling design performs global optimization on the preliminary modeling design scheme according to the global constraint information to obtain the optimized modeling design scheme;

[0404] 10. The large language model used for modeling design returns the optimized modeling design solution to the modeling design system;

[0405] 11. The modeling design system outputs the optimized modeling design solution to the user 2 (e.g., an assembly designer);

[0406] 12. User 2 inputs second input data such as text, design objects and / or images into the assembly design system;

[0407] 13. The assembly design system pre-processes the user input data such as data decomposition and feature recognition;

[0408] 14. The large language model for assembly design understands the intent of pre-processed user input data and obtains the design intent;

[0409] 15. A large language model for assembly design retrieves design knowledge related to design intent from the knowledge graph;

[0410] 16. The knowledge graph for assembly design returns the second target design knowledge related to the design intent to the large language model;

[0411] 17. A large language model for assembly design generates second-target design code based on design intent and second-target design knowledge;

[0412] 18. The large language model for assembly design returns the second target design code to the assembly design software of the assembly design system;

[0413] 19. The assembly design software calls the second target design code to generate a preliminary assembly design plan;

[0414] 20. The large language model for assembly design queries the knowledge graph for assembly design for global constraint information of the preliminary assembly design solution;

[0415] 21. The knowledge graph for assembly design returns global constraint information to the large language model for assembly design;

[0416] 22. The large language model for assembly design performs global optimization on the preliminary assembly design scheme according to the global constraint information to obtain the optimized assembly design scheme;

[0417] 23. The large language model for assembly design returns the optimized assembly design solution to the assembly design system;

[0418] 24. User 2 sends feedback information on the optimized assembly design solution to the assembly design system;

[0419] 25. The assembly design system formulates an optimization strategy for the large language model used for assembly design and the knowledge graph used for assembly design based on the feedback information;

[0420] 26. Optimize the large language model used for assembly design;

[0421] 27. Optimize the knowledge graph used for assembly design.

[0422] The above describes the design code generation method provided in the embodiment of the present application from multiple aspects. The following, in conjunction with the accompanying drawings, describes the cloud service-based design code generation device 11 and the knowledge graph construction device 12 provided in the embodiment of the present application.

[0423] like Fig.11 As shown, the embodiment of the present application provides a design code generation device 11 based on cloud service, which is applied to a cloud management platform. The cloud management platform is used to manage the infrastructure for providing cloud services. The infrastructure includes being set in multiple regions, each region including at least one cloud data center, and the cloud service runs on at least one server located in at least one cloud data center in the multiple regions. The device 11 includes:

[0424] The interface module 1101 is used to provide a configuration interface, the configuration interface is used to obtain the design intent information input by the user, and the design intent is used to indicate the generation of a target design object;

[0425] The processing module 1102 is used to:

[0426] Retrieving target design knowledge corresponding to the design intent from the knowledge graph, where the knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image, where the target design knowledge is used to describe the implicit design intent for the target design object;

[0427] The design intent and target design knowledge are processed through a large language model to obtain a target design code, which is used to generate a target design object.

[0428] Optionally, the interface module 1101 is used for:

[0429] Acquire a multimodal design material, wherein the multimodal design material includes a first design object, and the multimodal design material further includes one or more of the following: a first text and a first image;

[0430] The processing module 1102 is used for:

[0431] According to the first design object, a first geometric feature of a first component among the one or more components included in the first design object is obtained, where the first geometric feature is used to describe features related to geometric elements in the first component;

[0432] According to the first geometric feature, obtaining design knowledge corresponding to the first design object;

[0433] After obtaining the design knowledge corresponding to the design materials of each modality, the design knowledge corresponding to the design materials of each modality is jointly represented to obtain a target joint representation;

[0434] Construct a knowledge graph based on the joint representation of the target.

[0435] Optionally, the target design code includes target triplet information and information of at least one design code module, the at least one design code module is used to generate a target design object, and the target triplet information includes an operation object, an operation instruction and an operation parameter corresponding to the at least one design code module.

[0436] Optionally, the processing module 1102 is used to:

[0437] Processing the design intent and target design knowledge through a large language model to obtain a target module identifier and target triple information, wherein the target module identifier includes a module identifier corresponding to at least one design code module;

[0438] According to the target module identifier and target triple information, the target design code is obtained through the large language model.

[0439] Optionally, the design intent includes an intent derived based on a second design object,

[0440] The interface module 1101 is used to: obtain a second geometric feature of a second component associated with a target design object in one or more components included in the second design object, where the second geometric feature is used to describe features related to geometric elements in the second component;

[0441] The processing module 1102 is used to obtain, according to the second geometric feature, an intention based on the second design object.

[0442] Optionally, the processing module 1102 is used to:

[0443] Decomposing the second design object to obtain one or more hierarchical components in the second design object;

[0444] From the components of one or more levels in the second design object, obtain a component of a target level as a second component;

[0445] A second geometric feature of the second component is obtained.

[0446] Optionally, the design intent further includes an intent obtained based on the second text, and the processing module 1102 is used to:

[0447] Performing joint representation on the second geometric feature and the design information in the second text to obtain a first joint representation;

[0448] According to the first joint representation, a first joint representation intention is obtained, where the first joint representation intention includes an intention obtained based on the second design object and an intention obtained based on the second text.

[0449] Optionally, the design intent further includes an intent obtained based on the second image, and the processing module 1102 is used to:

[0450] Identify design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image;

[0451] performing joint characterization on the second geometric feature and the design information in the second image to obtain a second joint characterization;

[0452] According to the second joint representation, a second joint representation intention is obtained, where the second joint representation intention includes an intention obtained based on the second design object and an intention obtained based on the second image.

[0453] Optionally, the second geometric feature includes one or more of the following:

[0454] Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

[0455] like Fig.12 As shown, the embodiment of the present application provides a knowledge graph construction device 12, which is applied to a cloud management platform. The cloud management platform is used to manage the infrastructure for providing cloud services. The infrastructure includes multiple regions, each region includes at least one cloud data center, and the cloud service runs on at least one server located in at least one cloud data center in the multiple regions. The device 12 includes:

[0456] The interface module 1201 is used to obtain multimodal design materials, where the multimodal design materials include a first design object, and the multimodal design materials further include one or more of the following: a first text, a first image;

[0457] The processing module 1202 is used to:

[0458] According to the first design object, a first geometric feature of a first component among the one or more components included in the first design object is obtained, where the first geometric feature is used to describe features related to geometric elements in the first component;

[0459] According to the first geometric feature, obtaining design knowledge corresponding to the first design object;

[0460] After obtaining the design knowledge corresponding to the design materials of each modality, the design knowledge corresponding to the design materials of each modality is jointly represented to obtain a target joint representation;

[0461] Construct a knowledge graph based on the joint representation of the target.

[0462] Optionally, the interface module 1201 is used to: provide a configuration interface, the configuration interface is used to obtain information of a design intent input by a user, the design intent is used to indicate the generation of a target design object;

[0463] The processing module 1202 is used to:

[0464] Retrieve the target design knowledge corresponding to the design intent from the knowledge graph. The target design knowledge is used to describe the implicit design intent of the target design object.

[0465] The design intent and target design knowledge are processed through a large language model to obtain a target design code, which is used to generate a target design object.

[0466] Optionally, the number of the first design objects is multiple, and the processing module 1202 is used to:

[0467] Determine a target first geometric feature from among the first geometric features corresponding to the plurality of first design objects according to a co-occurrence relationship of the first geometric features corresponding to the plurality of first design objects;

[0468] According to the target first geometric feature, design knowledge corresponding to the first design object is obtained.

[0469] Optionally, in the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship.

[0470] Optionally, the knowledge graph includes a first sub-graph and at least one second sub-graph, the first sub-graph is used to describe common design knowledge in the design knowledge corresponding to multimodal design materials, the common design knowledge including one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets specified conditions, and any second sub-graph is obtained according to one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first texts associated with one or more first design objects, and design knowledge corresponding to the first images associated with one or more first design objects.

[0471] Among them, the processing module and the interface module can be implemented by software or hardware. Exemplarily, the implementation of the processing module is introduced below by taking the processing module as an example. Similarly, the implementation of the interface module can refer to the implementation of the processing module.

[0472] As an example of a software functional unit, a processing module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the processing module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0473] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0474] As an example of a hardware functional unit, the processing module may include at least one computing device, such as a server, etc. Alternatively, the processing module may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0475] The multiple computing devices included in the processing module can be distributed in the same region or in different regions. The multiple computing devices included in the processing module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and acceleration cards.

[0476] It should be noted that, in other embodiments, the processing module can be used to execute any step in the design code generation method based on cloud service or the knowledge graph construction method, and the interface module can be used to execute any step in the design code generation method based on cloud service or the knowledge graph construction method. The steps that the processing module and the interface module are responsible for implementing can be specified as needed. The processing module and the interface module respectively implement different steps in the design code generation method based on cloud service or the knowledge graph construction method to realize all the functions of the design code generation device based on cloud service or the knowledge graph construction device.

[0477] The present application embodiment also provides a computing device 130. Fig.13As shown, the computing device 130 includes: a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other through the bus 132. The computing device 130 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 130.

[0478] The bus 132 may be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Among them, the unified bus is also called a Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 The bus 134 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 134 may include a path for transmitting information between various components of the computing device 130 (eg, the memory 136, the processor 134, and the communication interface 138).

[0479] The processor 134 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, etc.

[0480] The memory 136 may include a volatile memory, such as a random access memory (RAM). The memory 136 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). In addition, the memory 136 may also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0481] It is worth noting that the same type of storage medium can be configured in the same computing device to implement the function of the memory 136, or two or more types of storage media can be configured to implement the function of the memory 136, and this application does not limit this.

[0482] The memory 136 stores executable program codes, and the processor 134 executes the executable program codes to respectively implement the functions of the aforementioned interface module and the processing module, thereby realizing the cloud service-based design code generation method or the knowledge graph construction method applied to the cloud management platform in the above-mentioned embodiments. That is, the memory 136 stores instructions for executing the cloud service-based design code generation method or the knowledge graph construction method applied to the cloud management platform in the above-mentioned embodiments.

[0483] The communication interface 138 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 130 and other devices or a communication network.

[0484] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0485] like Fig.14 As shown, the computing device cluster includes at least one computing device 130. The memory 136 in one or more computing devices 130 in the computing device cluster may store the same instructions for executing the design code generation method or the knowledge graph construction method.

[0486] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the cloud service-based design code generation method or the knowledge graph construction method. In other words, the combination of one or more computing devices 130 may jointly execute instructions for executing the cloud service-based design code generation method or the knowledge graph construction method.

[0487] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the cloud service-based design code generation method or the knowledge graph construction method. That is, the instructions stored in the memory 136 in different computing devices 130 can implement the functions of one or more modules in the interface module and the processing module.

[0488] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.15 A possible implementation is shown. Fig.15 As shown, the computing device 130A and the computing device 130B are connected via a network. Specifically, the network is connected via the communication interface in each computing device. In this type of possible implementation, the memory 136 in the computing device 130A may store instructions for executing the functions of the interface module. At the same time, the memory 136 in the computing device 130B may store instructions for executing the functions of the processing module. Alternatively, in other examples, the memory 136 in the computing device 130A may store instructions for executing part of the functions of the processing module. At the same time, the memory 136 in the computing device 130B may store instructions for executing another part of the functions of the processing module.

[0489] It should be understood that Fig.15 The functionality of computing device 130A shown in FIG. 1 may also be performed by multiple computing devices 130. Similarly, the functionality of computing device 130B may also be performed by multiple computing devices 130.

[0490] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.14 and Fig.15 The difference is that the memory 136 in one or more computing devices 130 in the computing device cluster may store the same instructions for executing the cloud service-based design code generation method or knowledge graph construction method.

[0491] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the cloud service-based design code generation method or the knowledge graph construction method. In other words, the combination of one or more computing devices 130 may jointly execute instructions for executing the cloud service-based design code generation method or the knowledge graph construction method.

[0492] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster can store different instructions for executing part of the functions of the cloud service-based design code generation method or the knowledge graph construction method. That is, the instructions stored in the memory 136 in different computing devices 130 can implement the functions of one or more modules in the interface module and the processing module.

[0493] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a design code generation method or a knowledge graph construction method based on a cloud service.

[0494] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute a cloud-based design code generation method or a knowledge graph construction method.

[0495] The embodiment of the present application also provides a chip system, which includes a processor, and the processor is used to implement the steps performed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0496] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0497] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0498] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0499] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0500] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A design code generation method based on cloud service, characterized in that: The method is applied to a cloud management platform, the cloud management platform is used to manage infrastructure for providing cloud services, the infrastructure includes being set up in multiple regions, each region includes at least one cloud data center, the cloud service runs on at least one server located in at least one cloud data center in the multiple regions, the method includes: Providing a configuration interface, the configuration interface is used to obtain information of a design intent input by a user, the design intent is used to instruct generation of a target design object; Retrieving target design knowledge corresponding to the design intent from a knowledge graph, wherein the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, wherein the target design knowledge is used to describe an implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain a target design code, and the target design code is used to generate the target design object.

2. The method according to claim 1, characterized in that Before retrieving the design knowledge corresponding to the design intent from the knowledge graph, the method further includes: Acquire a multimodal design material, wherein the multimodal design material includes the first design object, and the multimodal design material further includes one or more of the following: the first text and the first image; According to the first design object, obtaining a first geometric feature of a first component among the one or more components included in the first design object, where the first geometric feature is used to describe features related to geometric elements in the first component; Obtaining design knowledge corresponding to the first design object according to the first geometric feature; After obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design material of each modality to obtain a target joint representation; The knowledge graph is constructed based on the joint representation of the target.

3. The method according to claim 1 or 2, characterized in that: The target design code includes target triplet information and information of at least one design code module, the at least one design code module is used to generate the target design object, and the target triplet information includes operation objects, operation instructions and operation parameters corresponding to the at least one design code module.

4. The method according to claim 3, characterized in that The step of processing the design intent and the target design knowledge by using a large language model to obtain a target design code includes: Processing the design intent and the target design knowledge through the large language model to obtain a target module identifier and the target triplet information, wherein the target module identifier includes a module identifier corresponding to the at least one design code module; The target design code is obtained according to the target module identifier and the target triplet information through the large language model.

5. The method according to any one of claims 1 to 4, characterized in that: The design intent includes an intent obtained based on a second design object, and the method further includes: Acquire a second geometric feature of a second component associated with the target design object among the one or more components included in the second design object, where the second geometric feature is used to describe features related to geometric elements in the second component; The intention based on the second design object is obtained according to the second geometric feature.

6. The method according to claim 5, characterized in that The acquiring a second geometric feature of a second component associated with the target design object among the one or more components included in the second design object comprises: Decomposing the second design object to obtain one or more hierarchical components in the second design object; Acquire a component at a target level from components at one or more levels in the second design object as the second component; The second geometric feature of the second component is obtained.

7. The method according to claim 5 or 6, characterized in that: The design intent also includes an intent obtained based on the second text, and obtaining the intent obtained based on the second design object according to the second geometric feature includes: Performing joint characterization on the second geometric feature and the design information in the second text to obtain a first joint characterization; A first joint representation intention is obtained according to the first joint representation, where the first joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second text.

8. The method according to claim 5 or 6, characterized in that: The design intent also includes an intent obtained based on the second image, and the method further includes: Identify design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relations between entities in the second image, and attributes of entities in the second image; The obtaining, according to the second geometric feature, the intention based on the second design object comprises: performing joint characterization on the second geometric feature and the design information in the second image to obtain a second joint characterization; A second joint representation intention is obtained according to the second joint representation, where the second joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second image.

9. The method according to any one of claims 5 to 8, characterized in that: The second geometric feature includes one or more of the following: Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

10. A knowledge graph construction method, characterized in that: The method is applied to a cloud management platform, the cloud management platform is used to manage infrastructure for providing cloud services, the infrastructure includes being set up in multiple regions, each region includes at least one cloud data center, the cloud service runs on at least one server located in at least one cloud data center in the multiple regions, the method includes: Acquire a multimodal design material, wherein the multimodal design material includes a first design object, and the multimodal design material further includes one or more of the following: a first text, a first image; According to the first design object, obtaining a first geometric feature of a first component among the one or more components included in the first design object, where the first geometric feature is used to describe features related to geometric elements in the first component; Obtaining design knowledge corresponding to the first design object according to the first geometric feature; After obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design material of each modality to obtain a target joint representation; Based on the joint representation of the objectives, a knowledge graph is constructed.

11. The method according to claim 10, characterized in that The method further comprises: Providing a configuration interface, the configuration interface is used to obtain information of a design intent input by a user, the design intent is used to instruct generation of a target design object; Retrieving target design knowledge corresponding to the design intent from the knowledge graph, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain a target design code, and the target design code is used to generate the target design object.

12. The method according to claim 10 or 11, characterized in that: There are multiple first design objects, and obtaining design knowledge corresponding to the first design objects according to the first geometric features includes: Determine a target first geometric feature from among the first geometric features respectively corresponding to the plurality of first design objects according to a co-occurrence relationship of the first geometric features respectively corresponding to the plurality of first design objects; According to the target first geometric feature, design knowledge corresponding to the first design object is obtained.

13. The method according to claim 12, characterized in that In the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship.

14. The method according to any one of claims 10 to 13, characterized in that: The knowledge graph includes a first sub-graph and at least one second sub-graph, the first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to the multimodal design material, the common design knowledge includes one or more of the following: design knowledge from at least one of the first texts, design knowledge from at least one of the first images, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets the specified conditions, any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more of the first design objects, design knowledge corresponding to the first texts associated with one or more of the first design objects, and design knowledge corresponding to the first images associated with one or more of the first design objects.

15. A design code generation device based on cloud service, characterized in that: Applied to a cloud management platform, the cloud management platform is used to manage infrastructure for providing cloud services, the infrastructure includes being arranged in multiple regions, each region includes at least one cloud data center, the cloud service runs on at least one server located in at least one cloud data center in the multiple regions, the device includes: An interface module, used for providing a configuration interface, wherein the configuration interface is used for obtaining information of a design intent input by a user, wherein the design intent is used for indicating generation of a target design object; Processing modules for: Retrieving target design knowledge corresponding to the design intent from a knowledge graph, wherein the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image, wherein the target design knowledge is used to describe an implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain a target design code, and the target design code is used to generate the target design object.

16. The device according to claim 15, characterized in that The interface module is used for: Acquire a multimodal design material, wherein the multimodal design material includes the first design object, and the multimodal design material further includes one or more of the following: the first text and the first image; The processing module is used for: According to the first design object, obtaining a first geometric feature of a first component among the one or more components included in the first design object, where the first geometric feature is used to describe features related to geometric elements in the first component; Obtaining design knowledge corresponding to the first design object according to the first geometric feature; After obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design material of each modality to obtain a target joint representation; The knowledge graph is constructed based on the joint representation of the target.

17. The device according to claim 15 or 16, characterized in that The target design code includes target triplet information and information of at least one design code module, the at least one design code module is used to generate the target design object, and the target triplet information includes operation objects, operation instructions and operation parameters corresponding to the at least one design code module.

18. The device according to claim 17, characterized in that The processing module is used for: Processing the design intent and the target design knowledge through the large language model to obtain a target module identifier and the target triplet information, wherein the target module identifier includes a module identifier corresponding to the at least one design code module; The target design code is obtained according to the target module identifier and the target triplet information through the large language model.

19. The device according to any one of claims 15 to 18, characterized in that: The design intent includes an intent obtained based on a second design object, The interface module is used to: obtain a second geometric feature of a second component associated with the target design object among one or more components included in the second design object, wherein the second geometric feature is used to describe features related to geometric elements in the second component; The processing module is used to obtain the intention based on the second design object according to the second geometric feature.

20. The device according to claim 19, characterized in that The processing module is used for: Decomposing the second design object to obtain one or more hierarchical components in the second design object; Acquire a component at a target level from components at one or more levels in the second design object as the second component; The second geometric feature of the second component is obtained.

21. The device according to claim 19 or 20, characterized in that The design intent also includes an intent obtained based on the second text, and the processing module is used for: Performing joint characterization on the second geometric feature and the design information in the second text to obtain a first joint characterization; A first joint representation intention is obtained according to the first joint representation, where the first joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second text.

22. The device according to claim 19 or 20, characterized in that The design intent also includes an intent obtained based on the second image, and the processing module is used to: Identify design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; performing joint characterization on the second geometric feature and the design information in the second image to obtain a second joint characterization; A second joint representation intention is obtained according to the second joint representation, where the second joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second image.

23. The device according to any one of claims 19 to 22, characterized in that The second geometric feature includes one or more of the following: Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

24. A knowledge graph construction device, characterized in that: Applied to a cloud management platform, the cloud management platform is used to manage infrastructure for providing cloud services, the infrastructure includes being arranged in multiple regions, each region includes at least one cloud data center, the cloud service runs on at least one server located in at least one cloud data center in the multiple regions, the device includes: An interface module, configured to obtain multimodal design materials, wherein the multimodal design materials include a first design object, and the multimodal design materials further include one or more of the following: a first text, a first image; Processing modules for: According to the first design object, obtaining a first geometric feature of a first component among the one or more components included in the first design object, where the first geometric feature is used to describe features related to geometric elements in the first component; Obtaining design knowledge corresponding to the first design object according to the first geometric feature; After obtaining the design knowledge corresponding to the design material of each modality, jointly representing the design knowledge corresponding to the design material of each modality to obtain a target joint representation; Based on the joint representation of the objectives, a knowledge graph is constructed.

25. The device according to claim 24, characterized in that The interface module is used to: provide a configuration interface, the configuration interface is used to obtain information of design intent input by a user, and the design intent is used to indicate the generation of a target design object; The processing module is used for: Retrieving target design knowledge corresponding to the design intent from the knowledge graph, wherein the target design knowledge is used to describe the implicit design intent for the target design object; The design intent and the target design knowledge are processed by a large language model to obtain a target design code, and the target design code is used to generate the target design object.

26. The device according to claim 24 or 25, characterized in that The number of the first design objects is multiple, and the processing module is used for: Determine a target first geometric feature from among the first geometric features respectively corresponding to the plurality of first design objects according to a co-occurrence relationship of the first geometric features respectively corresponding to the plurality of first design objects; According to the target first geometric feature, design knowledge corresponding to the first design object is obtained.

27. The device according to claim 26, characterized in that In the knowledge graph, the weight of the design knowledge corresponding to the first geometric feature of the target is determined according to the co-occurrence probability corresponding to the first geometric feature of the target in the co-occurrence relationship.

28. The device according to any one of claims 24 to 27, characterized in that The knowledge graph includes a first sub-graph and at least one second sub-graph, the first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to the multimodal design material, the common design knowledge includes one or more of the following: design knowledge from at least one of the first texts, design knowledge from at least one of the first images, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets the specified conditions, any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more of the first design objects, design knowledge corresponding to the first texts associated with one or more of the first design objects, and design knowledge corresponding to the first images associated with one or more of the first design objects.

29. A computing device cluster, characterized in that: comprising at least one computing device, the at least one computing device comprising a processor and a memory; The processor is used to execute the instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 1 to 9, or executes the method according to any one of claims 10 to 14.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor is enabled to execute the method according to any one of claims 1 to 14.

31. A computer program product comprising instructions, characterized in that When the instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images