Design code generation method, knowledge graph construction method and related devices

Abstract

An embodiment of the present application discloses a design code generation method based on cloud services, and this method is applied to a cloud management platform. In this method, not only can explicit design intentions input by users be obtained from the configuration interface, but also target design knowledge associated with the current explicit design intentions of users can be queried from the knowledge graph to capture implicit design intentions for target design objects. In this way, the explicit design intentions and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate target design codes that meet the design requirements of users, realizing the intelligent generation of target design codes, and accurately and efficiently implementing design tasks according to the target design codes to obtain target design objects.

Description

Technical Field

[0001] This application relates to the technical field of industrial design, and specifically relates to a design code generation method, a knowledge graph construction method, and related devices. Background Art

[0002] In the industrial field, the learning cost of traditional design work (such as the design and assembly of 3D models, the design of user interfaces (UIs), etc.) is relatively high. Designers need to have a certain degree of computer skills and sufficient professional knowledge (for example, if the designed 3D model is applied to the engineering field, professional knowledge in the engineering field is required). Therefore, the labor cost is high and the production cycle is long, making it difficult to adjust and modify.

[0003] Currently, the introduction of large language models (LLMs) in the field of artificial intelligence (AI) has made intelligent design possible.

[0004] However, when using large language models to assist industrial design, since the design from natural language to 3D models involves dimensionality - raising reasoning, it is difficult for large language models to perceive and understand the design intent from the text prompts input by users. Moreover, as general models, large language models usually lack specific domain knowledge in the industrial field, resulting in industrial designs based on large language models usually being able to generate only very simple structures and unable to achieve complex structure design and / or assembly tasks.

[0005] It can be seen that there is an urgent need for a method that can perform intelligent industrial design efficiently and accurately. Summary of the Invention

[0006] The embodiments of this application provide a design code generation method based on cloud services, which can conveniently and efficiently generate design codes for users intelligently to efficiently implement design tasks such as 3D models and user interfaces. This application also provides corresponding devices, equipment, computer - readable storage media, computer program products, etc.

[0007] The first aspect of the present application provides a method for generating design code based on cloud services. This method is applied to a cloud management platform, which is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple regions, and each region includes at least one cloud data center. The cloud services run on at least one server in at least one cloud data center located in multiple regions. The method includes: providing a configuration interface, which is used to obtain information about the design intent input by the user, and the design intent is used to indicate the generation of a target design object; retrieving the target design knowledge corresponding to the design intent from a knowledge graph, and the knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, design knowledge from a first image, and the target design knowledge is used to describe the implicit design intent for the target design object; processing the design intent and the target design knowledge through a large language model to obtain a target design code, and the target design code is used to generate the target design object.

[0008] In the first aspect, not only can the explicit design intent input by the user be obtained from the configuration interface, but also the target design knowledge associated with the user's current explicit design intent can be queried from the knowledge graph to capture the implicit design intent for the target design object.

[0009] In this way, the explicit design intent and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate the target design code that meets the user's design requirements, realizing the intelligent generation of the target design code, and accurately and efficiently implementing the design task according to the target design code to obtain the target design object.

[0010] In a possible implementation manner of the first aspect, before retrieving the design knowledge corresponding to the design intent from the knowledge graph, the method further includes: obtaining multimodal design materials, the multimodal design materials include a first design object, and the multimodal design materials further include one or more of the following: a first text and a first image; obtaining the first geometric feature of the first component in one or more components included in the first design object according to the first design object, and the first geometric feature is used to describe the features related to geometric elements in the first component; obtaining the design knowledge corresponding to the first design object according to the first geometric feature; after obtaining the design knowledge corresponding to each modality of the design materials, jointly representing the design knowledge corresponding to the design materials of each modality to obtain a target joint representation; constructing a knowledge graph according to the target joint representation.

[0011] In traditional knowledge graph construction, knowledge extraction is usually only performed from text, and it is difficult to perform knowledge extraction from other forms of materials.

[0012] In this possible implementation, design knowledge can be obtained from multi-modal design materials. Among them, the multi-modal design materials include a first design object, which can be regarded as a historical design object. After obtaining the first design object, based on geometric features, the design knowledge implicit in the first design object can be mined from the first design object, and jointly represented with the design knowledge mined from other modal design materials. For example, an object joint representation in the form of a vector can be used to achieve the fusion of the design knowledge of the multi-modal design materials, so as to realize the construction of a knowledge graph. Moreover, since the knowledge graph can include the design knowledge mined from multi-modal design materials, the design knowledge contained in the knowledge graph is relatively rich, providing a good data basis for related applications of the knowledge graph.

[0013] In a possible implementation of the first aspect, the target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate a target design object, and the target triple information includes an operation object, an operation instruction, and operation parameters corresponding to at least one design code module.

[0014] In this possible implementation, the at least one design code module can also be regarded as a function. Each design code module can provide an interface for calling. Moreover, when calling, the information of the triple of the operation object, operation instruction, and operation parameters corresponding to the design code module can be used as the incoming information of the interface, so as to instruct the design code module to implement the corresponding design according to the information of the operation object, operation instruction, and operation parameters corresponding to the design code module.

[0015] In a possible implementation of the first aspect, the target design code is obtained by processing the design intention and target design knowledge through a large language model, including: processing the design intention and target design knowledge through a large language model to obtain a target module identifier and target triple information. The target module identifier includes the module identifiers corresponding to at least one design code module; according to the target module identifier and the target triple information, the target design code is obtained through the large language model.

[0016] In this possible implementation, through at least two calls of the large language model, the design code generation task is executed hierarchically to obtain the target design code that can call the interface of at least one design code module, making the generation logic of the design code clearer, so as to ensure the accuracy of the target design code.

[0017] In a possible implementation of the first aspect, the design intention includes the intention obtained based on the second design object, and further includes: obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object, where the second geometric feature is used to describe the feature related to the geometric elements in the second component; and obtaining the intention obtained based on the second design object according to the second geometric feature.

[0018] In this possible implementation, exemplarily, the geometric elements may include primitive elements in a two-dimensional image or geometric elements in a three-dimensional image. Among them, the primitive elements in the two-dimensional image may include one or more of points, lines, planes, etc., and the geometric elements in the three-dimensional image may include spheres or basic geometric bodies, etc. And the second geometric feature may include the feature of the geometric elements themselves in the second component, or may include the feature related to the structure formed by the geometric elements in the second component; and the second geometric feature may be described by the relevant parameter data of the geometric elements or may be described in a semantic form.

[0019] In a possible implementation of the first aspect, obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object includes: decomposing the second design object to obtain one or more levels of components in the second design object; obtaining the component at the target level from the one or more levels of components in the second design object as the second component; and obtaining the second geometric feature of the second component.

[0020] In this possible implementation, after decomposing the first design object, the topological levels of the decomposed components at all levels can be described by a directed acyclic graph (DAG), etc. After obtaining one or more levels of components, the components corresponding to the lowest level (i.e., the leaf nodes in the DAG) among the one or more levels can be assembled from bottom to top to obtain the first design object. Therefore, the components corresponding to the leaf nodes can describe the features of the first design object, and the components corresponding to the leaf nodes are usually in the form of relatively general components in the design field. Therefore, the components corresponding to the leaf nodes can be used as the first components.

[0021] In a possible implementation of the first aspect, the design intention further includes the intention obtained based on the second text. Obtaining the intention obtained based on the second design object according to the second geometric feature includes: jointly representing the design information in the second geometric feature and the second text to obtain a first joint representation; and obtaining a first joint representation intention according to the first joint representation, where the first joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second text.

[0022] In this possible implementation, when performing joint representation, the design information corresponding to the second design object (such as the second geometric feature of the second design object, etc.) can be feature-encoded (embedded) to be converted into a vector form. Moreover, the design information corresponding to the second text (such as the entities extracted from the second text, the relationships between the entities, the attributes of the entities, etc.) can also be feature-encoded and converted into a vector form, so that the design information corresponding to the second design object and the design information corresponding to the second text are mapped to the same feature space and described in the same form. In this way, the design information corresponding to the second design object and the design information corresponding to the second text can be represented in the same feature space, that is, the joint representation of the design information corresponding to the second design object and the design information corresponding to the second text is achieved.

[0023] In a possible implementation of the first aspect, the design intention further includes the intention obtained based on the second image. The method further includes: identifying the design information in the second image through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, attributes of entities in the second image; obtaining the intention obtained based on the second design object according to the second geometric feature, including: jointly representing the second geometric feature and the design information in the second image to obtain a second joint representation; obtaining a second joint representation intention according to the second joint representation, and the second joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second image.

[0024] In this possible implementation, when performing joint representation, the design information corresponding to the second design object (such as the second geometric feature of the second design object, etc.) can be feature-encoded (embedded) to be converted into a vector form. Moreover, the design information corresponding to the second image (such as the entities extracted from the second image, the relationships between the entities, the attributes of the entities, etc.) can also be feature-encoded and converted into a vector form, so that the design information corresponding to the second design object and the design information corresponding to the second image are mapped to the same feature space and described in the same form. In this way, the design information corresponding to the second design object and the design information corresponding to the second image can be represented in the same feature space, that is, the joint representation of the design information corresponding to the second design object and the design information corresponding to the second image is achieved.

[0025] In a possible implementation of the first aspect, the second geometric feature includes one or more of the following: spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, gradient features of the second component.

[0026] In a possible implementation of the first aspect, the design intention includes an intention obtained based on a second image. The method further includes: identifying design information in the second image through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; obtaining a first image intention based on the design information in the second image.

[0027] In this possible implementation, computer vision or other means can be used to extract the user's design intention from the second image, so as to better instruct the large language model to obtain a target design code that meets the user's design requirements.

[0028] In a possible implementation of the first aspect, the design intention further includes an intention obtained based on a second text. Obtaining a first image intention based on the design information in the second image includes: jointly representing the design information in the second text and the design information in the second image to obtain a third joint representation; obtaining a third joint representation intention based on the third joint representation. The third joint representation intention includes an intention obtained based on the second text and an intention obtained based on the second image.

[0029] In a possible implementation of the first aspect, it further includes: when calling a target design code to design a first object, if the target design code instructs an operation on a target component in the first object that does not belong to a preset component database, then based on the component parameters of the target component in the target design code, retrieve a target preset component in a plurality of preset components in the preset component database whose similarity to the target component meets a specified condition; perform shape fitting according to the component parameters of the target preset component and the target component to generate a target component, so as to operate on the target component according to the target design code.

[0030] In this possible implementation, exemplarily, the target preset component can be used as a benchmark for the target component. When the topological structure of the target preset component is consistent with that of the target component, the dimensional difference between the target component and the target preset component can be obtained based on this topological structure, and shape fitting can be performed based on this difference and relevant triple information, etc., to obtain a target component that meets the shape requirements in the target design code as the component required for generating the target design object.

[0031] In a possible implementation of the first aspect, after obtaining the target design code, it further includes: obtaining a target design object obtained by designing by calling the target design code; querying global constraint information corresponding to the target design object from a knowledge graph; optimizing the target design object according to the global constraint information.

[0032] In this possible implementation method, during the design process, the target design code usually indicates a bottom-up and local-to-global design. The obtained target design object is usually composed of multiple components. Therefore, to ensure the rationality of the target design object, the large language model can query the knowledge graph for the global constraint information corresponding to the target design object. Then, the large language model or other means can be used to detect whether the target design object meets the queried global constraint information. If not, the target design object can be adjusted according to the global constraint information to obtain an optimized target design object.

[0033] The second aspect of this application provides a method for constructing a knowledge graph. This method is applied to a cloud management platform, which is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple regions, each region includes at least one cloud data center, and cloud services run on at least one server in at least one cloud data center located in multiple regions. This method includes: obtaining multi-modal design materials, where the multi-modal design materials include a first design object, and the multi-modal design materials also include one or more of the following: a first text, a first image; according to the first design object, obtaining the first geometric feature of the first component in one or more components included in the first design object, where the first geometric feature is used to describe the features related to geometric elements in the first component; according to the first geometric feature, obtaining the design knowledge corresponding to the first design object; after obtaining the design knowledge corresponding to each modal of design materials, jointly representing the design knowledge corresponding to each modal of design materials to obtain a target joint representation; constructing a knowledge graph according to the target joint representation.

[0034] In traditional knowledge graph construction, knowledge extraction is usually only carried out from text, and it is difficult to extract knowledge from other forms of materials.

[0035] In the second aspect, design knowledge can be obtained from multi-modal design materials. Among them, the multi-modal design materials include a first design object, and this first design object can be regarded as a historical design object. After obtaining the first design object, based on geometric features, the design knowledge hidden in the first design object can be mined from the first design object, and jointly represented with the design knowledge mined from other modal design materials, such as using a target joint representation in the form of a vector, so as to realize the fusion of the design knowledge of multi-modal design materials and achieve the construction of the knowledge graph. And because the knowledge graph can include the design knowledge mined from multi-modal design materials, the design knowledge contained in this knowledge graph is relatively rich, providing a good data basis for related applications of the knowledge graph.

[0036] In a possible implementation of the second aspect, the method further includes: providing a configuration interface for obtaining information on the design intent input by the user, where the design intent is used to indicate the generation of a target design object; retrieving, from the knowledge graph, the target design knowledge corresponding to the design intent, where the target design knowledge is used to describe the implicit design intent for the target design object; and processing the design intent and the target design knowledge through a large language model to obtain a target design code for generating the target design object.

[0037] In a possible implementation of the second aspect, the number of the first design objects is multiple. Obtaining the design knowledge corresponding to the first design object according to the first geometric feature includes: determining a target first geometric feature from the first geometric features corresponding to the multiple first design objects according to the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects; and obtaining the design knowledge corresponding to the first design object according to the target first geometric feature.

[0038] In this possible implementation, since a single first design object is an independent design object and may contain some personalized or inappropriate designs, it is possible to screen out the relatively common target first geometric features from the multiple first design objects based on the co-occurrence relationship to serve as the design knowledge of the knowledge graph, thereby ensuring the reliability of the design knowledge in the knowledge graph.

[0039] In a possible implementation of the second aspect, in the knowledge graph, the weight of the design knowledge corresponding to the target first geometric feature is determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship.

[0040] In this possible implementation, in the knowledge graph, a weight can be set for the design knowledge from the first design object (i.e., the design knowledge corresponding to the target first geometric feature). The weight of the design knowledge corresponding to the target first geometric feature can also be regarded as the confidence level of the design knowledge corresponding to the target first geometric feature, which can reflect the credibility of the design knowledge corresponding to the target first geometric feature.

[0041] In a possible implementation of the second aspect, the knowledge graph includes a first sub-graph and at least one second sub-graph. The first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to multi-modal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects with a co-occurrence probability in the multiple first design objects satisfying a specified condition. Any one of the second sub-graphs is obtained based on one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, and design knowledge corresponding to the first image associated with one or more first design objects.

[0042] In this possible implementation, the knowledge graph can adopt the form of 1 + n. The "1" indicates that the knowledge graph includes a sub-graph (i.e., the first sub-graph) that describes the common design knowledge, and the "n" indicates that the knowledge graph includes n sub-graphs (i.e., n second sub-graphs) of design knowledge extracted from independent historical design objects. Here, n can be a positive integer, that is to say, the number of second sub-graphs can be at least one.

[0043] It can be seen that through the first sub-graph and the second sub-graph, the separate storage, application, update, and management of design knowledge from different sources and with different confidence levels can be achieved.

[0044] In a possible implementation of the second aspect, the first geometric feature includes one or more of the following: the spatial information of geometric elements in the first component, the semantic information used to describe the first component and / or the geometric elements in the first component, the structural feature of the first component, and the gradient feature of the first component.

[0045] In a possible implementation of the second aspect, obtaining the first geometric feature of the first component included in one or more components included in the first design object according to the first design object includes: decomposing the first design object to obtain one or more levels of components in the first design object; obtaining the component at the target level from one or more levels of components in the first design object as the first component; and obtaining the first geometric feature of the first component.

[0046] The third aspect of the present application provides a design code generation device based on cloud services. The device has the function of implementing the method in the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an interface module and a processing module.

[0047] The fourth aspect of the present application provides a computing device cluster, which includes at least one computing device. The at least one computing device includes a processor and a memory. Computer-executable instructions that can run on the processor are stored in the memory of the at least one computing device. When the computer-executable instructions are executed by the processor, the processor executes the method according to the first aspect or any possible implementation manner of the first aspect as described above.

[0048] The fifth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method according to the first aspect or any possible implementation manner of the first aspect as described above.

[0049] The sixth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method according to the first aspect or any possible implementation manner of the first aspect as described above.

[0050] The seventh aspect of the present application provides a chip system, which includes a processor for supporting the processor to implement the functions involved in the first aspect or any possible implementation manner of the first aspect as described above. In a possible design, the chip system may further include a memory for storing necessary program instructions and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0051] The eighth aspect of the present application provides a knowledge graph construction device, which has the function of implementing the method according to the second aspect or any possible implementation manner of the second aspect as described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an interface module and a processing module.

[0052] The ninth aspect of the present application provides a computing device cluster, which includes at least one computing device. The at least one computing device includes a processor and a memory. Computer-executable instructions that can run on the processor are stored in the memory of the at least one computing device. When the computer-executable instructions are executed by the processor, the processor executes the method according to the second aspect or any possible implementation manner of the second aspect as described above.

[0053] The tenth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method according to the second aspect or any possible implementation manner of the second aspect as described above.

[0054] The eleventh aspect of this application provides a computer program product for storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the second aspect or any possible implementation manner of the second aspect.

[0055] The twelfth aspect of this application provides a chip system. The chip system includes a processor for supporting the processor to implement the functions involved in the second aspect or any possible implementation manner of the second aspect. In a possible design, the chip system may further include a memory for storing necessary program instructions and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0056] Among them, the technical effects brought by the third aspect to the seventh aspect or any possible implementation manner thereof can be referred to the technical effects brought by the first aspect or the relevant possible implementation manners of the first aspect. The technical effects brought by the eighth aspect to the twelfth aspect or any possible implementation manner thereof can be referred to the technical effects brought by the second aspect or the relevant possible implementation manners of the second aspect, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an exemplary framework schematic diagram of a cloud data center provided by an embodiment of this application;

[0058] Figure 2 is an exemplary schematic diagram of a cloud service provided by an embodiment of this application;

[0059] Figure 3 is an exemplary flowchart provided by an embodiment of this application;

[0060] Figure 4 is an exemplary schematic diagram of knowledge graph construction provided by an embodiment of this application;

[0061] Figure 5 is an exemplary schematic diagram of a knowledge graph construction method provided by an embodiment of this application;

[0062] Figure 6 is an exemplary structural schematic diagram of a multi-agent framework provided by an embodiment of this application;

[0063] Figure 7 is an exemplary schematic diagram of a design code generation method provided by an embodiment of this application;

[0064] Figure 8 is an exemplary flowchart provided by an embodiment of this application;

[0065] Figure 9 is an exemplary information interaction schematic diagram provided by an embodiment of this application;

[0066] Figure 10 is another exemplary information interaction diagram provided by the embodiments of the present application;

[0067] Figure 11 is an exemplary diagram of a design code generation device based on cloud services provided by the embodiments of the present application;

[0068] Figure 12 is an exemplary diagram of a knowledge graph construction device provided by the embodiments of the present application;

[0069] Figure 13 is a structural diagram of a computing device provided by the embodiments of the present application;

[0070] Figure 14 is a structural diagram of a computing device cluster provided by the embodiments of the present application;

[0071] Figure 15 is a structural diagram of a computing device cluster provided by the embodiments of the present application. Detailed implementation manners

[0072] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only for explaining the specific embodiments of the present application and are not intended to limit the present application.

[0073] As can be known to those of ordinary skill in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0074] In this application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single items or plural items. The terms "first", "second", etc. in the description, claims, and the above-mentioned drawings of this application are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of this application. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products, or devices.

[0075] 1. Large Language Model (LLM)

[0076] A large language model refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. A large language model can handle various natural language tasks, such as text classification, question answering, dialogue, etc., and is an important approach to artificial intelligence.

[0077] Specifically, the large language model is a technology that has emerged in recent years. Because the large language model has undergone a refined data engineering and training process, a large amount of existing natural language processing knowledge has been learned inside its parameters. This knowledge can already replace people to do many language-related jobs, such as asking the large language model to write code or asking the large language model to do text summarization and other jobs.

[0078] 2. Graph Model

[0079] A graph model refers to a graph composed of nodes and edges used to describe a system.

[0080] In the embodiments of this application, the graph model may specifically include a directed acyclic graph (DAG) and an undirected acyclic graph.

[0081] 3. Directed Acyclic Graph (DAG) Decomposition

[0082] DAG decomposition is a task decomposition method that decomposes tasks based on the concept of a directed acyclic graph. A DAG is a directed graph in which there are no cycles, that is, starting from a vertex, it is impossible to return to that vertex through a series of edges. This graph structure of DAG is very suitable for representing and solving some problems with sequential or conditional dependencies, such as task scheduling, the shortest path problem in algorithms, etc.

[0083] [[ID=z3]]In DAG decomposition, a complex task is decomposed into multiple independent subtasks, and each subtask can be executed independently. This decomposition process first needs to clarify the in-degree (the number of directed edges entering a vertex) of each point in the graph, then classify the vertices with an in-degree of 0 as the top layer or the first layer, and then find the vertices with an in-degree of 0 in the subgraph after removing the vertices of the previous layer and classify them into the next layer, and so on until all vertices are assigned to different layers. Each layer can be executed independently, thus achieving parallel processing of tasks.

[0084] 4. Knowledge Graph

[0085] A knowledge graph is a graph-based data structure and a form of semantic network used to display entities and the relationships between entities. A knowledge graph includes nodes and edges. Among them, nodes are used to represent entities, such as people, place names, companies, etc. Attributes corresponding to entities can also exist in the knowledge graph, and edges represent the relationships between entities. In a knowledge graph, data can be organized through multi-tuple data. This multi-tuple data can include triple data, quadruple, quintuple, etc. Among them, the representation forms of triple data include: "node - edge - node" and "node - attribute name - attribute value", etc.

[0086] The embodiments of this application provide a design code generation method based on cloud services, which can conveniently and efficiently generate design codes intelligently for users to efficiently implement design tasks such as 3D models and user interfaces, and achieve intelligent industrial design.

[0087] The embodiments of this application can be applied to a cloud management platform.

[0088] A cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes being set in multiple regions, and each region includes at least one cloud data center. Cloud services run on at least one server in at least one cloud data center located in multiple regions.

[0089] The following combines Figure 1 ]>[[ID=z3]]the following schematic architecture diagram to give an exemplary introduction to the cloud data center.

[0090] Figure 1 In [diagram], the cloud management platform is connected to one or more servers (such as Figure 1The servers 1 and 2 in it) perform information interaction. A server includes a hardware layer and a software layer. The hardware layer includes the hardware configured for the server. Among them, the PCI device can be, for example, a network card, a graphics processing unit (GPU), an offload card, or other devices that can be inserted into the peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) slots of the server. The software layer includes the operating system installed and running on the server (the operating system of the host computer can be called the host operating system relative to the operating system of the virtual machine). A virtual machine manager (also called a hypervisor) is set in the host operating system. The role of the virtual machine manager is to implement the computing virtualization, network virtualization, and storage virtualization of the virtual machine and is responsible for managing the virtual machine. Among them, a virtual machine refers to a complete computer system with the functions of a complete hardware system simulated by software and running in a completely isolated environment. Figure 1 In the system architecture shown, multiple servers are set in the infrastructure. The servers can be used to run virtual machines. The specifications of each virtual machine can be the same or different. Among them, the virtual machine can also be called an elastic compute service (ECS), an elastic instance, etc. Different cloud service providers may have different names.

[0091] In an example of the embodiment of the present application, the cloud management platform can be a public cloud platform. At this time, cloud service providers such as individuals or software developers with cloud resource development capabilities can provide cloud services for users. Users can obtain cloud services through the Internet but do not own cloud computing resources. In other embodiments of the present application, the cloud management platform can be a private cloud platform or a hybrid cloud platform. The present application does not limit this.

[0092] Specifically, in Figure 1 In the example shown, the cloud management platform can provide access interfaces (such as an interface or an application programming interface (API)). The users of the cloud management platform and cloud service providers can operate the client to remotely access the access interface to register cloud accounts and passwords on the cloud management platform. After the cloud management platform authenticates the cloud accounts and passwords successfully, they can log in to the cloud management platform, so as to create, manage, log in to, and operate virtual machines in the cloud data center. It can be seen that enterprises, organizations, or individuals can purchase or rent cloud services through the cloud management platform, and thus can execute specified tasks through the cloud services provided by the cloud resources of the cloud management platform.

[0093] For example, as Figure 2 shown, the cloud services provided by the cloud management platform may include a design code generation service. In addition, in some examples, it may also include a design solution generation service.

[0094] For example, in Figure 2 some of the examples shown, the user can send information about the design intent to the cloud management platform to instruct the cloud management platform to execute the design code generation task, so as to intelligently generate the target design code required for the design task based on the knowledge graph and the large language model and feedback it to the user, so that the user can execute the design task of the target design object (such as the modeling design task, assembly task of the three-dimensional model or the design task of the user interface, etc.) according to the target design code and obtain the target design object as the design result.

[0095] Among them, the target design object can also be called the target design instance, which is the design object generated in the design task. A target design object can also be called a target design instance. For example, it can be a three-dimensional model that has completed the modeling design, a three-dimensional model that has completed the assembly design, or a user interface that has completed the graphic design, etc. It can be seen that there can be multiple specific types of target design objects, which can be determined according to the actual application scenario.

[0096] Or, in Figure 2 some of the examples shown, the user can send information about the design intent to the cloud management platform to instruct the cloud management platform to execute the design code generation task, so as to intelligently generate the target design code required for the design task based on the knowledge graph and the large language model. Then, the design code generation service can pass the target design code to the design solution generation service of the cloud management platform. Next, the design solution generation service can call the target design code to execute the design task of the target design object (such as the modeling design task, assembly task of the three-dimensional model or the design task of the user interface, etc.) to obtain the target design object as the design result (such as the designed three-dimensional model or user interface, etc.), and then output the target design object from the cloud management platform to the user.

[0097] It should be noted that Figure 2 the services in

[0098] are only an example of the services provided by the cloud management platform, rather than a limitation. Figure 2 In other examples provided in this application, the above-mentioned design code generation service and design solution generation service can be integrally deployed on the cloud management platform; and the functional division methods of each service can be different from those of the Figure 2 services shown, and the deployment methods of each service can also be different. Each service can be provided independently, or embedded in other services, or multiple services can be combined and then deployed. This application does not make any restrictions on this.

[0099] Based on the above cloud management platform, referring to Figure 2 the system architecture shown in Figure 3 as shown, the method of the embodiment of the present application may include one or more of the following aspects:

[0100] Knowledge graph construction, large language model enhancement and constraints for design code generation tasks, design code generation based on knowledge graph and large language model, invoking the generated design code for design, and design result optimization.

[0101] The following will give an exemplary introduction to each of the above aspects respectively.

[0102] I. Knowledge graph construction

[0103] The knowledge graph construction in the embodiment of the present application may include one or more of the following aspects:

[0104] Construction of the knowledge layer (schema) of the knowledge graph, knowledge mining of design materials, joint representation of design knowledge from multi-modal design materials, and knowledge fusion.

[0105] The following will give an exemplary introduction to the multiple aspects that the knowledge graph construction may involve.

[0106] 1. Construction of the data model (schema) of the knowledge graph

[0107] The knowledge graph can be regarded as a structured semantic knowledge base, which can include a data model schema and an instance layer. Among them, the schema can also be called a data schema, etc., and can be regarded as a data model in a specified domain (for example, the design domain in the embodiment of the present application), which contains meaningful concept types in the specified domain and the attributes of these concept types. The schema of the specified domain is mainly expressed by types and properties. The instance layer includes instance data. For example, the instance data corresponding to the entity "name" in the knowledge graph includes Zhang San, Li Si, etc.

[0108] In the embodiment of the present application, the specific method of constructing the schema is not limited herein, and existing or subsequent-developed schema construction methods can be adopted.

[0109] Exemplarily, the schema can be constructed based on expert experience. For example, in Figure 4In the illustrated example, for the application field involved (i.e., the design field in the embodiments of the present application), experts predict the schema from top to bottom based on the design objects in the design field, then summarize the schema from bottom to top, and then perform information fusion to construct the schema. For example, the preset texts in the design field can be collected and sorted out. The amount of data in the preset texts is usually small to facilitate experts' summarization and information extraction. Then, the characteristics of entities, relationships, and attributes in the preset texts are summarized and inducted, so as to extract the relevant data framework in the design field and form a definite schema.

[0110] After the schema is constructed, information extraction can be carried out from the design materials targeted according to the constructed schema (for example, extracting information such as entities, relationships, and attributes that meet the requirements of the schema), so as to construct a knowledge graph.

[0111] 2. Knowledge mining of design materials

[0112] In the embodiments of the present application, the type of the design materials can be one or more. In other words, the design materials can be single-modal design materials or multi-modal design materials.

[0113] Exemplarily, the design materials can include one or more of the following: the first text, the first design object, and the first image.

[0114] Next, an exemplary introduction to the knowledge mining of any one of the above-mentioned modal design materials will be given, and an exemplary introduction to the possible joint representation schemes between multiple modal design materials can also be given.

[0115] (1) Mining of explicit knowledge carried by documents

[0116] In the embodiments of the present application, the design knowledge carried by documents can be regarded as explicit knowledge.

[0117] Among the explicit knowledge in the documents, there can be unstructured documents, semi-structured documents, and structured documents.

[0118] In one example, the structured documents can include graphic structures such as structure diagrams and flowcharts and / or structured forms such as tables. Since the data structure in the structured documents is clear, it is convenient to extract structured information such as entities, relationships, and attributes from the structured documents as the design knowledge extracted from the structured documents.

[0119] Next, an exemplary introduction to the knowledge mining of the first image and the first text that may be included in the unstructured documents or semi-structured documents will be given respectively.

[0120] In one example, an unstructured document or a semi-structured document may include, but is not limited to, a first image.

[0121] The first image may include one or more images, and there may be various cases for the data type of the first image, which is not limited herein. Exemplarily, the first image may be in a bitmap format (such as bitmap (BMP), portable network graphics (PNG), joint photographic experts group (JPEG), etc.), a vector graphics format, etc.

[0122] The following introduces an exemplary way to obtain design knowledge from the first image.

[0123] In some embodiments, when the design material includes the first image, the way to obtain the design knowledge corresponding to the first image specifically includes:

[0124] By object detection and / or semantic segmentation, identify the design knowledge in the first image. The design knowledge in the first image includes one or more of the following: entities in the first image, relationships between entities in the first image, and attributes of entities in the first image.

[0125] In the embodiments of the present application, referring to Figure 4 the example shown, the design information of the first image can be identified by means of object detection and / or semantic segmentation in computer vision (CV). Among them, exemplarily, the entities in the first image may include one or more objects detected in the first image; and the relationships between entities can be determined according to the relative positions between the entities, etc.; in addition, the attributes of the entities in the first image may include one or more of the following information of the entities in the first image: position, size (which can be described by a bounding box such as the minimum bounding rectangle of the entity), and orientation.

[0126] For example, through object detection, information such as the category and location of entities in the first image can be identified. There can be various specific ways of this object detection, which are not limited here. Exemplarily, it can be achieved through a machine learning model capable of object detection (such as convolutional neural networks (CNN), support vector machine (SVM), etc.) or traditional object detection methods (such as scale-invariant feature transform (SIFT) and random sample consensus (RANSAC)).

[0127] In addition, through semantic segmentation, the category to which each pixel in the first image belongs can be identified. In other words, semantic segmentation can divide the first image into one or more regions and obtain the semantic categories to which the one or more regions belong respectively. It can be seen that based on the semantic categories identified by semantic segmentation and the segmented regions, the entities included in the first image can be identified, and the relationships between entities can be determined according to the relative positions between entities, etc. The attributes of entities can be determined according to the positions of entities and information of regions, etc.

[0128] It can be seen that in the embodiments of the present application, design knowledge such as entities in the first image, relationships between entities, and / or attributes of entities can be identified through object detection and / or semantic segmentation.

[0129] In one example, the unstructured document or semi-structured document may include, but is not limited to, the first text.

[0130] The following introduces an exemplary way to obtain design knowledge from the first text.

[0131] In the embodiments of the present application, named entity recognition can be performed on the first text to identify the named entities in the first text. Then, information such as the relationships between the named entities and the attributes of the named entities can also be identified from the first text, and entities, relationships between entities, and / or attributes of entities in the first text are obtained as the design knowledge of the first text.

[0132] Among them, there can be various named entity recognition methods and ways to identify the relationships between named entities and the attributes of named entities.

[0133] For example, in some examples, the large language model to be fine-tuned can be fine-tuned so that the fine-tuned large language model can identify the design information of the first text.

[0134] Specifically, in some embodiments, the method further includes:

[0135] Fine-tune the large language model to be fine-tuned according to the preset text and the tags of the preset text, so as to obtain a large language model for identifying design information of the text after fine-tuning. The tags of the preset text include one or more of the following information: entities in the preset text, relationships between entities in the preset text, attributes of entities in the preset text;

[0136] Extract information from the first text through the large language model for identifying design information of the text, and obtain the design knowledge corresponding to the first text.

[0137] Specifically, as Figure 4 In the example shown, the preset text may include small sample data for full supervision learning in the design field. The small sample data may have corresponding labels, and the labels are used to describe the entities of the small sample data.

[0138] In this way, according to the small sample data and its labels, the first fine-tuning of the large language model to be fine-tuned can be realized, and the large language model after the first fine-tuning can identify the named entities in the text in the design field.

[0139] Then, weak supervision data can be obtained. In the weak supervision data, some data may have corresponding labels, and the labels are used to describe the relationships and / or attribute information of the named entities in the part of the data. In this way, according to the weak supervision data and the labels corresponding to the part of the data, the large language model after the first fine-tuning can be secondarily fine-tuned. In this way, the large language model after the second fine-tuning can extract information such as entities, relationships between entities, and attributes of entities in the text. The large language model after the second fine-tuning can be regarded as a large language model for identifying design information of the text. In this way, when constructing a knowledge graph, the design information of the first text can be identified through the large language model for identifying design information of the text to obtain the design knowledge in the first text.

[0140] It can be understood that in the embodiments of the present application, the fine-tuning operations involved in the large language model for identifying design information of the text are different from those of the large language model used in the process of generating design codes. Therefore, the large language model for identifying design information of the text may be different from the large language model used in the process of generating design codes. For example, the large language model used in the process of generating design codes may be obtained by further fine-tuning the large language model for identifying design information of the text, or may also be obtained by fine-tuning the large language model to be fine-tuned in other cases.

[0141] Alternatively, in some other examples, other existing or subsequent developed recognition methods can be used to recognize the design information in the first text.

[0142] (2) Mining of Tacit Knowledge without Document Carriage

[0143] In the embodiments of the present application, the design knowledge without document carriage can be regarded as tacit knowledge.

[0144] In actual design scenarios, there are often a large number of historical design objects. The historical design object can also be regarded as a historical design instance, that is, a design object obtained by performing a design task historically. For example, in a 3D design scenario, the historical design object (i.e., the historical design instance) can be a 3D model obtained by historical design, and in a user interface design scenario, the historical design object can be regarded as a user interface obtained by historical design.

[0145] Among a large number of historical design objects, there usually also contains a large amount of design knowledge, which can include personalized design knowledge in personalized design scenarios, and can also include common design knowledge obtained by designing based on common design specifications and / or design constraints, etc. In traditional knowledge graph construction, knowledge extraction is usually only performed from texts, and it is difficult to perform knowledge extraction from other forms of materials.

[0146] In the embodiments of the present application, tacit knowledge can be mined from historical design objects to enrich the knowledge in the knowledge graph and achieve full mining of design knowledge in the design field.

[0147] Taking the first design object in the historical design object as an example below, an exemplary method for mining tacit knowledge from the historical design object is introduced.

[0148] Specifically, in some embodiments, the design material includes the first design object, and the method includes:

[0149] According to the first design object, obtain the first geometric feature of the first component included in the first design object, where the first geometric feature is used to describe the features related to geometric elements in the first component;

[0150] According to the first geometric feature, obtain the design knowledge corresponding to the first design object.

[0151] The first design object can also be called the first design instance, and can be regarded as a historical design object, which is a design object generated in a historical design task. For example, it can be a 3D model that completes a styling design, a 3D model that completes an assembly design, or a user interface that completes a graphic design, etc.

[0152] In the embodiments of the present application, the first geometric feature of the first component in the first design object can be.

[0153] Among them, the first geometric feature is used to describe the features related to geometric elements in the first component. The geometric element can refer to an object or entity that exists in space, is fundamental, and is indivisible. Exemplarily, the geometric element can include primitive elements in a two-dimensional image or geometric elements in a three-dimensional image. Among them, the primitive elements in a two-dimensional image can include one or more of points, lines, planes, etc., and the geometric elements in a three-dimensional image can include spheres or basic geometric bodies, etc.

[0154] The first geometric feature can include the features of the geometric elements themselves in the first component or the features of the structure formed by the geometric elements in the first component; moreover, the first geometric feature can be described by relevant parameter data of the geometric elements or by semantic forms.

[0155] Exemplarily, the first geometric feature includes one or more of the following:

[0156] The spatial information of the geometric elements in the first component, the semantic information used to describe the first component and / or the geometric elements in the first component, the structural features of the first component, the gradient features of the first component.

[0157] Among them, the spatial information of the geometric elements in the first component can include one or more of the following: the position, direction, size, scaling ratio, etc. of the geometric elements. The semantic information used to describe the first component and / or the geometric elements in the first component can describe the semantics of the first component and / or the geometric elements in the first component. For example, it can describe the name, form, meaning, etc. of the first component and / or the geometric elements in the first component. The structural features of the first component can include the characteristics of the outline and / or the internal structure of the first component. For example, it can include shape information such as the symmetry of the first component, the relationships (such as parallel, perpendicular, intersecting) between the geometric elements in the first component, etc. The gradient information of the first component can also be called the sharpness information of the first component and is described according to the magnitude of the normal gradient of the first component.

[0158] It can be seen that the first geometric feature of the first component can describe the first component from one or more levels, from the most basic geometric elements of the first component to the structure formed by the geometric elements.

[0159] There can be multiple ways to identify the first geometric feature of the first component. For example, the first geometric feature of the first component can be identified by a machine learning model such as a convolutional neural network (CNN), or by algorithms such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), features from accelerated segment test (FAST), etc.

[0160] There can also be multiple situations for the first component.

[0161] Exemplarily, when the first design object is relatively simple, for example, it only includes one component or several components simply spliced together, the one component or several components simply spliced together can be used as the first component.

[0162] Alternatively, the first design object can also be decomposed to obtain some or all of the components in the first design object as the first component.

[0163] Specifically, in some embodiments, obtaining the first geometric feature of the first component from one or more components included in the first design object includes:

[0164] Decompose the first design object to obtain one or more levels of components in the first design object;

[0165] Obtain the components at the target level from one or more levels of components in the first design object as the first component;

[0166] Obtain the first geometric feature of the first component.

[0167] In the embodiments of the present application, with reference to Figure 4 the example shown, the first design object can be decomposed from top to bottom to obtain one or more levels of components in the first design object. Among them, the components at the lowest level in the one or more levels of components are usually parts, such as structures like bolts and nuts, so as to decompose the first design object into components that are more common in the design field, have commonalities, and are convenient to describe using the design knowledge in the design field.

[0168] Among them, after decomposing the first design object, the topological hierarchy of the components at all levels after decomposition can be described by a DAG or the like, and the DAG can also be optimized by a graph neural network (GNN) or the like, and one or more levels of components in the first design object can be obtained after optimization.

[0169] After obtaining one or more levels of components, the components at the lowest level in one or more levels can be used as the components at the target level, so as to obtain the components at the lowest level (that is, the leaf nodes in the DAG) as the first components. Specifically, referring to Figure 4 the example shown, in the DAG, the components corresponding to the leaf nodes can be obtained as the first components. It can be understood that the components corresponding to the leaf nodes can be assembled from bottom to top to obtain the first design object. Therefore, the components corresponding to the leaf nodes can describe the characteristics of the first design object, and the components corresponding to the leaf nodes are usually in the form of relatively common components in the design field. Therefore, the components corresponding to the leaf nodes can be used as the first components to obtain the first geometric features of the components corresponding to the leaf nodes to obtain the design knowledge in the first design object.

[0170] After obtaining the first geometric features of the first components, entities can be obtained according to the first components, and the attributes of the first components can be obtained according to the first geometric features of the first components. Information such as the relationships between the first components and / or the geometric elements in the first components or the relationships between the geometric bodies in the first components can also be obtained to construct a knowledge graph.

[0171] In addition, in some embodiments, since a single first design object is an independent design object and may contain some personalized or inappropriate designs, relatively common target first geometric features can be selected from multiple first design objects as the design knowledge of the knowledge graph, so as to ensure the reliability of the design knowledge in the knowledge graph.

[0172] Specifically, in some embodiments, the number of first design objects is multiple, and obtaining the design knowledge corresponding to the first design object according to the first geometric features includes:

[0173] Determining target first geometric features from the first geometric features corresponding to each of the multiple first design objects according to the co-occurrence relationship of the first geometric features corresponding to each of the multiple first design objects;

[0174] Obtaining the design knowledge corresponding to the first design object according to the target first geometric features.

[0175] The co-occurrence relationship refers to the frequency or probability of two things appearing in the same context.

[0176] In the embodiments of the present application, the co-occurrence relationship of a certain first geometric feature can be the frequency or probability of the simultaneous occurrence of the first geometric feature and the first design object among multiple first design objects. In some examples, the co-occurrence relationship of a certain first geometric feature can specifically be the frequency or probability of the simultaneous occurrence of the first geometric feature and the first design object among multiple first design objects of the same type. It can be seen that the strength of the co-occurrence relationship can be reflected by the co-occurrence probability. If the co-occurrence relationship indicates that the probability of the simultaneous occurrence of a certain first geometric feature and the first design object (i.e., the co-occurrence probability) is relatively high (for example, higher than a specified probability threshold), it can be considered that the possibility of the first design object containing the first geometric feature is relatively high, and the first geometric feature can be considered as a relatively commonly used geometric feature in the first design object. It can be determined that the first geometric feature is the target first geometric feature among the first geometric features included in the corresponding first design object, so that the design knowledge corresponding to the target first geometric feature in the first design object can be used as relatively reliable design knowledge corresponding to the first design object.

[0177] In some embodiments, in the knowledge graph, the weight of the design knowledge corresponding to the target first geometric feature is determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship.

[0178] In the embodiments of the present application, the weight can be determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship. For example, the weight can be equal to the corresponding co-occurrence probability, or can be obtained by calculating the corresponding co-occurrence probability according to a specified calculation method. Among them, the co-occurrence probability corresponding to the target first geometric feature refers to the probability that the first design object containing the target first geometric feature accounts for among multiple first design objects.

[0179] It can be seen that in the knowledge graph, weights can be set for the design knowledge from the first design object (that is, the design knowledge corresponding to the target first geometric feature). The weight of the design knowledge corresponding to the target first geometric feature can also be regarded as the confidence of the design knowledge corresponding to the target first geometric feature, and can reflect the credibility of the design knowledge corresponding to the target first geometric feature.

[0180] Moreover, since new first design objects can be continuously generated as the design task is continuously executed. And based on the new first design objects, the design knowledge corresponding to the target first geometric feature in the knowledge graph and its weight can be continuously updated, so that the design knowledge in the knowledge graph is continuously improved and has better reliability, providing a good data basis for users to use the design knowledge in the knowledge graph to execute design tasks.

[0181] 3. Joint Representation of Design Knowledge from Multimodal Design Materials

[0182] In the embodiments of the present application, one or more modalities of design materials can be obtained. Based on any of the above embodiments of knowledge mining, after obtaining the design knowledge corresponding to each design material, a knowledge graph can be constructed according to the design knowledge corresponding to each design material.

[0183] Among them, in some examples, the design material can be a multi-modal design material, and the construction of the knowledge graph can be realized through the joint representation of the design knowledge from the multi-modal design materials.

[0184] Specifically, referring to Figure 5 the example shown, a method for constructing a knowledge graph in the embodiments of the present application can include steps 501-504.

[0185] Step 501, obtain multi-modal design materials.

[0186] Step 502, obtain the design knowledge corresponding to the design materials of each modality.

[0187] The specific way to obtain the design knowledge corresponding to the design materials of each modality can refer to the relevant embodiments of the knowledge mining of the design materials above, and will not be elaborated here.

[0188] Step 503, after obtaining the design knowledge corresponding to the design materials of each modality, perform a joint representation on the design knowledge corresponding to the design materials of each modality to obtain a target joint representation.

[0189] Step 504, construct a knowledge graph according to the target joint representation.

[0190] This joint representation can involve at least two modalities of design materials. The following will respectively give an exemplary introduction to different situations of at least two modalities of design materials.

[0191] Joint representation 1: The joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text.

[0192] Specifically, in some embodiments, performing a joint representation on the design knowledge corresponding to the design materials of each modality to obtain a target joint representation includes:

[0193] Performing a joint representation on the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text to obtain a target joint representation.

[0194] Exemplarily, the first text can include a description document of the first design object and other texts describing the design information of the first design object, such as texts for describing the constraints related to the first design object, the semantic information of the first design object, the relationships between components in the first design object, etc. In addition, the first text can also include texts describing other design information, and the embodiments of the present application do not limit this.

[0195] When performing joint representation, the design knowledge corresponding to the first design object (such as information on the first geometric features of the first design object, etc.) can be feature-encoded (embedded) to be converted into a vector form. Moreover, the design knowledge corresponding to the first text (such as information on entities extracted from the first text, relationships between entities, attributes of entities, etc.) can also be feature-encoded and converted into a vector form so that the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text are mapped to the same feature space and described in the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text can be represented in the same feature space, that is, to achieve the joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first text.

[0196] Joint representation 2, the joint representation between the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image.

[0197] Specifically, in some embodiments, if the first image includes a first sub-image related to the first design object, then the design knowledge corresponding to the design materials of each modality is jointly represented to obtain a target joint representation, including:

[0198] Jointly represent the design knowledge corresponding to the first design object and the design knowledge corresponding to the first sub-image to obtain the target joint representation.

[0199] Exemplarily, the first image may include an image related to the first design object, which may include a photographed picture corresponding to the first design object, an engineering drawing (such as a two-dimensional computer-aided design (CAD) drawing) describing the first design object, etc.

[0200] When performing joint representation, the design knowledge corresponding to the first design object can be feature-encoded to be converted into a vector form. Moreover, the design knowledge corresponding to the first image (such as information on entities extracted from the first image, relationships between entities, attributes of entities, etc.) can also be feature-encoded and converted into a vector form so that the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image are mapped to the same feature space and described in the same form. In this way, the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image can be represented and fused in the same feature space, that is, to achieve the joint representation of the design knowledge corresponding to the first design object and the design knowledge corresponding to the first image.

[0201] Joint representation 3, the joint representation between the design knowledge corresponding to the first image and the design knowledge corresponding to the first text.

[0202] Specifically, in some embodiments, if the first image includes a first sub-image related to the first design object, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a target joint characterization, including:

[0203] Jointly characterize the design knowledge corresponding to the first image and the design knowledge corresponding to the first text to obtain a target joint characterization.

[0204] Exemplarily, the first text may include a second sub-text related to the first image. For example, if one or more first images and certain second sub-texts are included in the same document, jointly characterize the design knowledge of the one or more first images and the second sub-text, so as to integrate the design knowledge of the one or more first images and the second sub-text in combination with the context in the document.

[0205] It can be understood that in the embodiments of the present application, jointly characterizing the design knowledge corresponding to the design materials of each modality may include one or more of the above three joint characterization methods.

[0206] For example, when the multi-modal design materials include a first design object, a first text, and a first image, the design knowledge corresponding to the design materials of the three modalities can be respectively feature-encoded so that the design knowledge corresponding to the design materials of the three modalities is converted into a vector form, and the joint characterization of the design knowledge corresponding to the design materials of the three modalities is realized, so as to achieve the fusion of the design knowledge corresponding to the design materials of the three modalities.

[0207] In the embodiments of the present application, there are various situations for the specific timing of joint characterization.

[0208] For example, in some examples, after the excavation of explicit knowledge (such as the design knowledge corresponding to the first image and / or the design knowledge of the first text unrelated to the first design object, etc.) is realized, joint characterization can be carried out to obtain a preliminary sub-graph recording the explicit knowledge. And after the excavation of implicit knowledge (such as the design knowledge corresponding to the first design object and / or the design knowledge of the first text related to the first design object, etc.) is realized, joint characterization can be carried out to obtain a preliminary sub-graph recording the implicit knowledge. Then, a knowledge graph can be constructed according to the preliminary sub-graph recording the explicit knowledge and the preliminary sub-graph recording the implicit knowledge.

[0209] Alternatively, after obtaining the design knowledge corresponding to the design materials of all modalities, the design knowledge corresponding to the design materials of all modalities can be jointly characterized to construct a knowledge graph.

[0210] In the embodiments of the present application, design knowledge can be obtained from multimodal design materials. Among them, the multimodal design materials include a first design object, which can be regarded as a historical design object. After obtaining the first design object, based on geometric features, the design knowledge hidden in the first design object can be mined from the first design object, and jointly represented with the design knowledge mined from other modal design materials, such as using an object joint representation in the form of a vector, so as to realize the fusion of the design knowledge of multimodal design materials and construct a knowledge graph. Moreover, since the knowledge graph can include the design knowledge mined from multimodal design materials, the constructed knowledge graph contains relatively rich design knowledge, providing a good data basis for related applications of the knowledge graph.

[0211] 4. Knowledge Fusion

[0212] In the embodiments of the present application, referring to Figure 4 the example shown, after joint representation, the fusion of the design knowledge corresponding to multimodal design materials can be realized. For example, semantic fusion and / or structural fusion can be performed to obtain a fusion result.

[0213] Exemplarily, semantic fusion can include the fusion of the same descriptions in the descriptions of entities, relationships, attributes, etc., and the unification of synonyms, etc., to achieve semantic alignment and fusion of the design knowledge corresponding to multimodal design materials. And structural fusion can include the alignment and fusion of information such as the relationships between entities and the attributes of entities, to achieve structural alignment and fusion of the design knowledge corresponding to multimodal design materials.

[0214] In this way, according to the constructed schema and the fusion result, a knowledge graph can be constructed in a specified structured form and stored in a graph database.

[0215] In addition, in the embodiments of the present application, there can be various specific forms of the knowledge graph.

[0216] For example, in some examples, the knowledge graph includes a first sub-graph and at least one second sub-graph. The first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to multimodal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, design knowledge from multiple first design objects and the co-occurrence probability in multiple first design objects meets the specified conditions. Any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, design knowledge corresponding to the first image associated with one or more first design objects.

[0217] It can be seen that in this example, the knowledge graph can adopt the form of 1 + n. The "1" therein indicates that the knowledge graph includes a sub-graph (i.e., the first sub-graph) that describes common design knowledge, while the "n" indicates that the knowledge graph includes n sub-graphs (i.e., n second sub-graphs) of design knowledge extracted from independent historical design objects. Here, n can be a positive integer, that is to say, the number of the second sub-graphs can be at least one.

[0218] Among them, the source of the common design knowledge in the first sub-graph can include one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, and design knowledge from multiple first design objects with a co-occurrence probability in the multiple first design objects meeting specified conditions.

[0219] The common design knowledge can be regarded as relatively general design knowledge in the design field. For example, the mating relationship between a bolt standard part and a nut standard part, etc. Therefore, at least one first text for obtaining the common design knowledge can include texts that describe relatively standardized design knowledge in the design field, and in many scenarios, do not include texts that describe personalized designs. For example, in some scenarios, at least one first text may not include texts related to the first design object, such as the description document of the first design object. And at least one first image for obtaining the common design knowledge can also be an image that describes relatively general and standardized design knowledge. For example, at least one first image for obtaining the common design knowledge can include a photographed picture of a standard part or a 2D CAD drawing of a standard part; in addition, at least one first image for obtaining the common design knowledge can also be an image located in the same document as the text that describes relatively standardized design knowledge in the design field, so as to combine the context information of the document and extract accurate design knowledge from the text and image in the document as the common design knowledge in the first sub-graph.

[0220] In addition, in some examples, the common design knowledge can include design knowledge from multiple first design objects with a co-occurrence probability in the multiple first design objects meeting specified conditions.

[0221] In the embodiments of the present application, this co-occurrence probability can be higher than a relatively high probability threshold to ensure that this design knowledge is very general design knowledge among multiple first design objects. In addition, in some examples, the number of the multiple first design objects can also reach a specified number threshold to ensure that the design knowledge from multiple design objects is extracted from a sufficient number of first design objects to ensure the generality of this design knowledge. At this time, the very general design knowledge from multiple first design objects can also be used as the common design knowledge in the first sub-graph.

[0222] The second sub-graph can be design knowledge obtained from one or more independent first design objects in history.

[0223] In a traditional knowledge graph, it is difficult to extract effective design knowledge from the first design object. However, in the embodiments of the present application, according to geometric features, co-occurrence relationships, etc., from one or more first design objects and other design materials associated with one or more first design objects (such as the first text associated with one or more first design objects, the first image associated with one or more first design objects, etc.), design knowledge with a relatively high degree of credibility can be extracted as the design knowledge in the knowledge graph. Moreover, through one or more second sub-graphs, the design knowledge from the first design object and related design materials can be stored to distinguish it from the common design knowledge in the first sub-graph. In this way, as the design task is continuously executed, according to the newly generated first design object, the design knowledge in the second sub-graph can be conveniently updated, and it can facilitate the user to select the design knowledge that meets the requirements from the first sub-graph and the second sub-graph to execute the design task.

[0224] When there are multiple second sub-graphs, there can be multiple ways to divide different second sub-graphs, which are not limited here. Exemplarily, different second sub-graphs can respectively record the design knowledge of different types of first design objects.

[0225] It can be seen that through the first sub-graph and the second sub-graph, the separate storage, application, update, and management of design knowledge from different sources and with different confidence levels can be achieved.

[0226] In some other examples, the knowledge graph can also have other forms. For example, the knowledge graph can include sub-graphs corresponding to different design types; or, the design knowledge corresponding to each design material can also be integrated into the same knowledge graph. Based on the above method of constructing the knowledge graph, a knowledge graph can be constructed for subsequent design code generation tasks. In addition, as shown in Figure 4 the example, the knowledge graph can also be updated and iterated according to expert knowledge or the execution result of the design task executed based on the generated design code, etc. For example, according to the specific situation of the target design object obtained from the design task, the weight of the corresponding design knowledge in the knowledge graph can be updated.

[0227] In the construction of traditional knowledge graphs, knowledge extraction is usually only carried out from text, and it is difficult to achieve knowledge extraction from other forms of materials.

[0228] It can be understood that in the embodiments of the present application, the knowledge graph can be constructed through a cloud management platform and deployed in the infrastructure managed by the cloud management platform. Alternatively, the knowledge graph can also be constructed by other devices and then transmitted and deployed to the infrastructure managed by the cloud management platform.

[0229] II. Enhancement and Constraint of Large Language Models for Design Code Generation Tasks

[0230] In some embodiments, in order to perform subsequent design code generation tasks, large language models can be enhanced and constrained in advance so that the enhanced and constrained large language models can perform design code generation tasks more accurately.

[0231] Based on the different scenario requirements of subsequent design code generation tasks, there can be multiple cases for the inference methods of the enhanced and constrained large language models in design code generation tasks. Correspondingly, there can be multiple cases for the specific forms of the training data and their labels used to enhance and constrain the large language models. Exemplary introductions are given below.

[0232] Specific form 1 of training data and its labels: The training data includes preset design intents and preset design knowledge, and the labels of the training data include preset design codes.

[0233] In this example, the specific generation method of the preset design intent of any training data can refer to the generation method of the design intent in the relevant embodiments of the design code generation part based on the knowledge graph and large language models; or, the preset design intent can come from a third party; or, the preset design intent can also be pre-manually configured by the user.

[0234] The preset design knowledge in the training data can be the design knowledge retrieved from the knowledge graph related to the preset design intent; or, the preset design knowledge can also be pre-manually configured by the user.

[0235] The labels of the training data can include preset design codes, and the preset design codes can be used as execution scripts of the design software, but do not include the information of the preset design code modules in the subsequent examples. That is to say, the preset design codes are not designed by calling at least one design code module, but are execution scripts including specific execution instructions.

[0236] The specific type of the preset design code is not limited here and can be determined based on the needs of the design software for performing design tasks. Exemplarily, the preset design code can be a domain specific language (DSL). Specifically, in some examples, since python has become a common secondary development script for many design software, the preset design code can be a python script.

[0237] Specific form 2 of training data and its labels: The training data includes preset design intents and preset design knowledge. The labels of the training data include preset triple information and the module identifiers of at least one preset design code module. The preset triple information includes the operation object, operation instruction, and operation parameter corresponding to at least one preset design code module.

[0238] In this example, the module identifiers, specific contents of the design code modules, and the call interfaces corresponding to the creation of the design code modules can be pre-written.

[0239] For example, developers can deconstruct the historical design tasks for implementing the design of historical design objects to obtain multiple design subtasks for designing the historical design objects, and then write design code modules for one or more of the design subtasks respectively, so that any design code module can execute the corresponding design subtask and achieve the specified design function. It can be understood that the design code module can also be considered as a function.

[0240] In addition, in some examples, the design code module can be an atomic module, that is to say, any design code module can be decomposed into the smallest and indivisible code modules. In this way, it is convenient to flexibly use each design code module when generating design code later and achieve the flexible combination of each design code module.

[0241] After writing the design code module, the call interface corresponding to the design code module can be created to facilitate the invocation of the design code module through the call interface. In addition, the module identifier of each design code module (such as the name of the design code module, etc.) can be determined to uniquely identify the corresponding design code module. And the call information of the design code module can be standardized into the form of triple information. Specifically, the triple information can include the operation object, operation instruction, and operation parameter. In this way, through the triple information in the label in the structure of object, predicate, and object complement, it can be indicated that the large language model realizes the learning from natural language to code generation, so that the enhanced and constrained large language model has the ability to identify the module identifier of at least one preset design code module required for the corresponding design task according to the training data, and can extract the preset triple information according to the training data to facilitate the interface call of at least one preset design code module.

[0242] In this example, the label of the training data can include preset triple information and the module identifiers of at least one preset design code module, but does not include execution statements, so it cannot be directly used as an execution script.

[0243] Specific form 3 of training data and its labels: The training data includes a preset design intention and preset design knowledge, and the labels of the training data include a preset design code that can call at least one design code module through the preset triple information and the module identifier of at least one preset design code module.

[0244] In the embodiments of the present application, the labels of the training data include a preset design code, and this preset design code not only includes the preset triple information and the module identifier of at least one preset design code module, but also includes execution statements. Therefore, it can be used as an execution script. It can be seen that in this preset design code, at least one preset design code module can be called according to the preset triple information and the module identifier of at least one preset design code module.

[0245] The inference methods of the large language model enhanced and constrained by using the above different forms of training data and its labels in the design code generation task may be different. The specific inference methods can refer to the introduction of the relevant embodiments of the subsequent design code generation task, which will not be elaborated here.

[0246] In the embodiments of the present application, after obtaining multiple sets of training data and its labels, the large language model to be enhanced can be enhanced and constrained by using methods such as backpropagation according to the multiple sets of training data and its labels, so that the large language model obtained after enhancement and constraint can generate corresponding design codes according to the input design intention.

[0247] Among them, in one example, a multi-agent framework can be used to enhance and constrain the large language model to be enhanced.

[0248] For example, as Figure 6 shown in the example, it is an exemplary schematic diagram of the multi-agent framework.

[0249] In the multi-agent framework, it can include one or more of intention understanding, problem decomposition, large language model, knowledge graph, and feedback model, etc. The process of enhancing and constraining the large language model to be enhanced includes at least one iteration process. Hereinafter, the i-th iteration process in at least one iteration process will be taken as an example for introduction.

[0250] Exemplarily, the i-th iteration process in at least one iteration process includes: reasoning on the training data through the large language model to be enhanced in the i-th iteration process to obtain output data; evaluating whether the output data meets the specifications and / or constraints through the knowledge graph to obtain an evaluation result; updating the large language model to be enhanced in the i-th iteration process according to the difference between the output data and the label and the evaluation result.

[0251] During the enhancement and constraint process, the information of the preset design intent input by the user can be understood, and the preset design knowledge corresponding to the preset design intent can be queried from the knowledge graph to obtain the preset design intent and the preset design knowledge. In some examples, in addition, the user's design intent can be deconstructed into problems to obtain one or more design subtasks, and each design subtask can be implemented by a design code module. In this way, the preset design intent and the preset design knowledge corresponding to each design subtask can be obtained as training data. Then, the training data containing the preset design intent and the preset design knowledge corresponding to the design subtasks obtained by the problem deconstruction can be used as input data and input into the large language model to be enhanced to obtain the output data of the large language model to be enhanced.

[0252] During the inference process or after obtaining the output data, the large language model can query the knowledge graph. For example, during the inference process, the large language model can query the specifications and / or constraints of the specified design from the knowledge graph; in addition, after obtaining the output data, the large language model can query the knowledge graph to determine whether the output data meets the specifications and / or constraints to obtain an evaluation result on whether the output data meets the specifications and / or constraints. Among them, the specifications can include the design standards stored in the knowledge graph, and can also be regarded as guiding conditions, which are usually relatively objective and unified. The constraints refer to restrictive conditions and can be described in the form of functions or inequalities.

[0253] In this way, according to the difference between the output data and the label, and the evaluation result, backpropagation is performed on the large language model to be enhanced in the i-th iteration process to update the weights of the large language model to be enhanced in the i-th iteration process.

[0254] When the number of iterations reaches the specified number, or when the loss value of the output data of the large language model to be enhanced converges to the desired state, the enhancement and constraint of the large language model can be completed, and the large language model can be obtained for subsequent design code generation tasks.

[0255] In some examples, after the large language model is applied to the actual application scenario to execute the design code generation task to obtain the target design object and output it, it can also be further enhanced and constrained according to the feedback information of the expert on the target design object, etc., through Figure 6 the feedback model in the example shown, for the large language model.

[0256] Specifically, an expert can evaluate the accuracy of a target design object to obtain feedback information of the target design object. Then, the feedback model can query the knowledge graph to check whether the feedback information conforms to specifications and / or constraints, and based on the corresponding query results, obtain target feedback information. Of course, in some examples, the feedback information input by the expert for the target design object can also be directly used as the target feedback information. After obtaining the target feedback information, the feedback model can perform reinforcement learning on the large language model through the target feedback information to further enhance and constrain the large language model during the application process, continuously improving the performance of the large language model.

[0257] It can be understood that in the embodiments of the present application, the enhancement and constraint of the large language model can be achieved by executing the infrastructure managed by the cloud management platform of the embodiments of the present application, or can be implemented in other devices and then deployed to at least one server of the infrastructure.

[0258] III. Design code generation based on knowledge graph and large language model

[0259] After constructing the knowledge graph and obtaining the fine-tuned large language model, a design code generation task can be performed through the knowledge graph and the large language model. For example, Figure 2 In the example shown, the design code generation service of the cloud management platform performs the design code generation task to implement the design code generation method.

[0260] Specifically, as Figure 7 shown, the cloud service-based design code generation method may include steps 701-703.

[0261] Step 701, provide a configuration interface.

[0262] The configuration interface is used to obtain information on the design intent input by the user, and the design intent is used to indicate the generation of a target design object.

[0263] In the embodiments of the present application, the cloud management platform can provide a configuration interface to the user's client, so that the user can input information on the design intent through the configuration interface displayed on the client.

[0264] Exemplarily, the information on the design intent may include one or more of the following: a second text, a second design object, a second image. Correspondingly, the design intent may include one or more of the following: an intent based on the second text, an intent based on the second design object, an intent based on the second image.

[0265] In this way, after obtaining one or more of the second text, the second design object, and the second image through the configuration interface, the design intention can be obtained from one or more of the second text, the second design object, and the second image. Since this design intention is obtained based on the information input by the user, it can be considered that this design intention is used to describe the explicit design intention of the user for the target design object.

[0266] And since there can be various situations for the specific content of the information of the design intention, there can also be various situations for the way to obtain the design intention.

[0267] For example, in some examples, the design intention includes the intention obtained based on the second design object, and the method further includes:

[0268] Obtain the second geometric feature of the second component associated with the target design object among one or more components included in the second design object, where the second geometric feature is used to describe the features related to the geometric elements in the second component;

[0269] Obtain the intention obtained based on the second design object according to the second geometric feature.

[0270] There can be various situations for the second design object input by the user. For example, the second design object can be the design object indicated by the user as a reference; or, the second design object can be the design object targeted by the design task. For example, if the design task to be executed is an assembly design task, then the second design object can be the object to be assembled. In addition, the number of the second design objects can be one or more, and this is not limited here.

[0271] Among them, in some embodiments, obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object includes:

[0272] Decompose the second design object to obtain one or more levels of components in the second design object;

[0273] Obtain the component at the target level from one or more levels of components in the second design object as the second component;

[0274] Obtain the second geometric feature of the second component.

[0275] In some embodiments, the second geometric feature includes one or more of the following:

[0276] The spatial information of the geometric elements in the second component, the semantic information used to describe the second component and / or the geometric elements in the second component, the structural feature of the second component, the gradient feature of the second component.

[0277] In the embodiments of the present application, the specific content of the second geometric feature and the specific manner of obtaining the second geometric feature may refer to the specific content of the first geometric feature and the specific manner of obtaining the first geometric feature in any of the above embodiments, which will not be elaborated herein. Moreover, in the embodiments of the present application, the process of decomposing the second design object to obtain the second geometric feature of the second component can also be understood as problem deconstruction.

[0278] After obtaining the information of the second geometric feature of the second design object, the second design object input by the user can be described through the information of the second geometric feature, so as to obtain the intention based on the second design object.

[0279] In some embodiments, the design material includes a second image, and the method further includes:

[0280] Identifying the design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image;

[0281] Obtaining the intention based on the second image according to the design information in the second image.

[0282] In the embodiments of the present application, any second image can be an independent image; or, it can also be an image related to the second design object. For example, it can be a photo of the second design object, a 2D CAD drawing, etc.; or, it can also be an image associated with the second text. For example, it is in the same document as the second text. The manner of obtaining the design information of the second image can refer to the manner of obtaining the design knowledge of the first image in the above embodiments, which will not be elaborated herein.

[0283] In some embodiments, the design material includes a second text.

[0284] The second text can be a descriptive text input by the user to describe the design requirements. For example, it can describe information such as the target design object to be designed.

[0285] In some examples, the intention of the second text can also be understood through a machine learning model or other algorithms, etc., to mine the design intention contained in the second text.

[0286] Specifically, in some embodiments, the design material includes a second text, and the method further includes:

[0287] Performing information extraction on the second text through a large language model for information extraction or other information extraction algorithms to obtain the design information corresponding to the second text.

[0288] Exemplarily, a large language model for information extraction or other information extraction algorithms can decompose complex design tasks into multiple design subtasks based on the tree of thoughts (ToT) and / or the chain of thought (CoT), so as to achieve problem deconstruction to obtain the intention based on the second text. The context of the second text can also be used for intention understanding to obtain the intention based on the second text.

[0289] In other examples, the second text can be directly used as the intention obtained based on the second text.

[0290] In addition, in some embodiments, when the information input by the user includes at least two modalities of data (for example, including at least two of the second text, the second design object, and the second image), in order to achieve the fusion of the intentions corresponding to the two modalities, joint representation can be performed.

[0291] For example, in some embodiments, the design intention includes the intention obtained based on the second design object and the intention obtained based on the second text. Obtaining the design intention may include:

[0292] Jointly represent the second geometric features of the second component in the second design object and the design information in the second text to obtain a first joint representation;

[0293] According to the first joint representation, obtain a first joint representation intention, which includes the intention obtained based on the second design object and the intention obtained based on the second text.

[0294] In some embodiments, the design intention includes the intention obtained based on the second design object and the intention obtained based on the second image. Obtaining the design intention may include:

[0295] Jointly represent the second geometric features of the second component in the second design object and the design information in the second image to obtain a second joint representation;

[0296] According to the second joint representation, obtain a second joint representation intention, which includes the intention obtained based on the second design object and the intention obtained based on the second image.

[0297] In some embodiments, the design intention includes the intention obtained based on the second text and the intention obtained based on the second image. Obtaining the design intention may include:

[0298] Jointly represent the design information in the second text and the design information in the second image to obtain a third joint representation;

[0299] According to the third combined representation, a third combined representation intention is obtained. The third combined representation intention includes an intention obtained based on the second text and an intention obtained based on the second image.

[0300] In the embodiments of the present application, the first combined representation, the second combined representation, and the third combined representation may be in vector form, so as to unify the intention information of data from different modalities in the same feature space for description, thereby facilitating subsequent unified processing.

[0301] For the specific generation methods of the above-mentioned first combined representation, second combined representation, and third combined representation, reference may be made to the relevant descriptions in the relevant embodiments of the combined representation of the design knowledge corresponding to the design materials of each modality above, which will not be elaborated here.

[0302] It can be seen that in the embodiments of the present application, depending on different situations of the information input by the user, there can be multiple situations for the specific content of the design intention.

[0303] In addition, there can also be multiple situations for the specific form of the design intention. Exemplarily, when the design intention only includes the intention obtained based on the second text, the design intention can be in text form. When the design intention includes the intention obtained from data of multiple modalities, the design intention can be in vector form.

[0304] Moreover, in some examples, the design intention can also be in the form of a triple, and the triple includes an operation object, an operation instruction, and an operation parameter. Among them, the information of the operation object, operation instruction, and operation parameter in the design intention can be described in forms such as text or in vector form. It can be seen that in this example, the design intention includes triple information obtained based on one or more of the second text, the second design object, or the second image.

[0305] Step 702, retrieve the target design knowledge corresponding to the design intention from the knowledge graph.

[0306] The knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. The target design knowledge is used to describe the implicit design intention for the target design object.

[0307] After obtaining the design intention, the target design knowledge corresponding to the design intention can be retrieved from the knowledge graph.

[0308] Among them, the knowledge graph can be the knowledge graph constructed by any of the above embodiments of constructing the knowledge graph. Therefore, the knowledge graph can include one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. In this way, relatively comprehensive target design knowledge that matches one or more of the intents obtained based on the second text, the intents obtained based on the second design object, and the intents obtained based on the second image can be retrieved from the knowledge graph.

[0309] For example, in some examples, in traditional knowledge graph construction schemes, it is difficult to extract and store design knowledge from historical design objects (i.e., the first design object), and in actual application processes, it is also impossible to extract design intents from the second design object.

[0310] However, in the knowledge graph of any embodiment of the present application, design knowledge can be extracted from historical design objects and stored. In this way, the user can input the second design object through the configuration interface, and the cloud management platform can extract the intent obtained based on the second design object from the second design object input by the user and query relevant target design knowledge from the knowledge graph.

[0311] The target design knowledge is used to describe the implicit design intent of the target design object. In other words, through the knowledge graph, according to the information of the linear design intent input by the user, more comprehensive design intents that cannot be directly extracted from the user's input information can be further mined to more comprehensively guide the large language model to generate reasonable target design code in the subsequent steps.

[0312] For example, in one example, if the information of the design intent input by the user indicates that the user designs a car, then design knowledge such as design specifications and constraints related to car design can be mined from the knowledge graph as the target design knowledge. These target design knowledge are not included in the user's input information but can be retrieved through the knowledge graph, so that in subsequent steps, the large language model can be guided to generate target design code that can more reasonably design the car.

[0313] Step 703, process the design intent and the target design knowledge through a large language model to obtain the target design code.

[0314] The target design code is used to generate the target design object.

[0315] In the embodiments of the present application, the design intent and the target design knowledge can be input into the large language model, so as to call the large language model to process the design intent and the target design knowledge and output the target design code.

[0316] The specific type of the target design code is not limited herein and can be determined based on the needs of the design software for executing design tasks. Exemplarily, the target design code can be a domain specific language (DSL). Specifically, in some examples, since Python has become a common secondary development script for many design softwares, the target design code can be a Python script.

[0317] The target design code can be called by the design software for executing design tasks to generate executable code to execute the corresponding design tasks, and obtain a target design object as the design result.

[0318] In the embodiments of the present application, there can be various possible cases for the specific inference process of obtaining the target design code through the large language model, and the following will be introduced exemplarily respectively.

[0319] Inference method 1: The design intention and the target design knowledge are input into the large language model for processing, so that the large language model outputs an unmodularized target design code.

[0320] In this example, through one call, the large language model can output the target design code that is the execution script of the design software. And in this example, the design code modules in the subsequent Inference method 2 are not obtained in advance. Therefore, in the embodiments of the present application, the target design code does not include the information of at least one design code module, that is, the design is not carried out by calling at least one design code module, but a specific execution script including the content of specific execution instructions can be directly generated as the target design code.

[0321] The large language model in this example can be obtained by fine-tuning in the manner of the first training data and its labels in the related embodiments of fine-tuning the above large language model.

[0322] Inference method 2: The design intention and the target design knowledge are input into the large language model for processing, so that the large language model outputs a target design code including target triple information and the information of at least one design code module.

[0323] Specifically, in this example, the target design code includes target triple information and the information of at least one design code module. The at least one design code module is used to generate a target design object, and the target triple information includes an operation object, an operation instruction, and an operation parameter corresponding to the at least one design code module.

[0324] In the embodiments of the present application, the design code modules can be pre-configured. Specifically, reference can be made to the related embodiments of fine-tuning the large language model, which will not be elaborated herein.

[0325] The at least one design code module can also be regarded as a function. Each design code module can provide an interface for calling. And when called, the information of the triple of the operation object, operation instruction, and operation parameter corresponding to the design code module can be used as the input information of the interface, so as to instruct the design code module to implement the corresponding design according to the information of the operation object, operation instruction, and operation parameter corresponding to the design code module.

[0326] In the target design code, there can be various specific forms of the included target triple information and the information of at least one design code module. The specific forms of the target design code and the corresponding reasoning methods will be introduced below.

[0327] Reasoning method 2.1: Obtain the target design code that can call the interface of at least one design code module through two calls to the large language model.

[0328] Specifically, in some embodiments, step 703 includes:

[0329] Process the design intention and target design knowledge through the large language model to obtain the target module identifier and target triple information. The target module identifier includes the module identifiers corresponding to at least one design code module;

[0330] According to the target module identifier and the target triple information, obtain the target design code through the large language model.

[0331] In the embodiments of the present application, the design intention and target design knowledge can be processed through the large language model to obtain the target module identifier and target triple information. In this way, the large language model can, according to the design intention and target design knowledge, determine at least one design code module required for the current design task (that is, determine the target module identifier of the at least one design code module) from multiple pre-configured design code modules, and can determine the target triple information required to execute the at least one design code module. Among them, the module identifier corresponding to a certain design code module can uniquely identify the corresponding design code module. Exemplarily, the module identifier corresponding to a certain design code module can be the name or number of the design code module, etc.

[0332] Then, a second call can be made to the large language model to process the target module identifier and the target triple information through the large language model to generate the target design code. The target design code can be used to call the interface of at least one design code module according to the target triple information to implement the design task. It can be seen that in the target design code, the execution order of at least one design code module can be indicated, etc.

[0333] For example, in an exemplary scenario, the design intention of the user for the vehicle design can be obtained through a configuration interface, and in the knowledge graph, the target design knowledge for the vehicle design corresponding to the design intention can be queried. Then, the design intention and the target design knowledge can be input into a large language model so that the large language model outputs the target module identifier and the target triple information of at least one design code module required for the vehicle design. For example, the at least one design code module can include a design code module for designing the engine of the vehicle, a design code module for designing the chassis of the vehicle, a design code module for designing the exterior of the vehicle, and the like. Next, the target design code can be generated by processing the target module identifier and the target triple information through the large language model. In the target design code, it can be indicated to first call the interface of the design code module for designing the engine of the vehicle to design the engine of the vehicle, then call the interface of the design code module for designing the chassis of the vehicle to design the chassis of the vehicle, and then call the interface of the design code module for designing the exterior of the vehicle to design the exterior of the vehicle, so as to reasonably design the vehicle from the inside out and avoid design conflicts resulting in unreasonable design results.

[0334] In the embodiments of the present application, the design code generation task can be hierarchically executed through two calls of the large language model, so that the generation logic of the design code is relatively clear, thereby ensuring the accuracy of the target design code.

[0335] Inference method 2.2: By calling the large language model once, a target design code including the target triple information and the information of at least one design code module is obtained, but the target design code does not include execution statements, so it cannot be directly used as an execution script.

[0336] Specifically, in some embodiments, step 703 includes:

[0337] The large language model processes the design intention and the target design knowledge to obtain the target module identifier and the target triple information as the target design code.

[0338] In this example, the design software can obtain the target triple information and the information of at least one design code module in the target design code to generate an execution script, and thus obtain executable code according to the generated execution script to execute the design task and obtain the target design object as the design result.

[0339] In the examples of Inference method 2.1 and Inference method 2.2, the large language model adopted can be obtained by fine-tuning in the second way of the second training data and its labels in the related embodiments of fine-tuning the above large language model.

[0340] Inference method 2.3: Through one call of the large language model, obtain the target design code that can make interface calls to at least one design code module.

[0341] In this example, inputting the design intention and target design knowledge into the large language model can enable the large language model to directly output the interface that can call at least one design code module to achieve the target design code for the corresponding design task. It can be seen that in this target design code, the execution order of at least one design code module can be indicated. And this target design code can be executed as an execution script of the design software.

[0342] In this example, the large language model adopted can be obtained by fine-tuning in the third way of training data and its labels in the relevant embodiments of fine-tuning the above large language model.

[0343] It can be seen that in the embodiments of the present application, not only can the explicit design intention input by the user be obtained from the configuration interface, but also the target design knowledge associated with the user's current explicit design intention can be queried from the knowledge graph to capture the implicit design intention of the target design object.

[0344] In this way, the explicit design intention and the target design knowledge from the knowledge graph can be combined to guide the large language model to efficiently and accurately generate the target design code that meets the user's design requirements, realizing the intelligent generation of the target design code, and accurately and efficiently implementing the design task according to the target design code to obtain the target design object.

[0345] IV. Design by Invoking the Generated Design Code

[0346] In the embodiments of the present application, after obtaining the target design code, the generated target design code can be called through the corresponding design software for design to obtain the target design object.

[0347] It should be noted that in the embodiments of the present application, it can be in the cloud service provided by the cloud management platform (such as Figure 2 the design scheme generation service shown) that the generated target design code is called through the corresponding design software for design to obtain the target design object and then output to the user; or, it can also be that after the cloud management platform obtains the target design code, it is output to the user, so that the user can call the generated target design code through the design software of the client for design to obtain the target design object.

[0348] In the specific design task execution process, the design software can call the generated target design code to determine the components required to generate the target design object.

[0349] For example, in some examples, when invoking the target design code to design the target design object, if the target design code indicates an operation on a certain component in the target design object, then this component can be determined as the component required to generate the target design object, and from the preset component database, through methods such as nearest neighbor component retrieval, the components required to generate the target design object can be retrieved. Multiple preset components can be stored in the preset component database, and any preset component can be in the form of general components and parts, etc., to be common in various design tasks. At this time, this preset component can be considered a standard part. Of course, in some examples, the preset component database can also include other components besides standard parts, which are not limited here.

[0350] Exemplarily, through methods such as structural topology, the preset components matching the components required to generate the target design object can be retrieved in the preset component database.

[0351] If a matching preset component is retrieved from the preset component database, then this matching preset component can be obtained as the component required to generate the target design object.

[0352] Or, in some examples, when invoking the target design code to design the target design object, if the target design code indicates an operation on a target component in the first object that does not belong to the preset component database, that is, no matching preset component for this target component is retrieved in the preset component database, then based on the component parameters of the target component in the target design code, among the multiple preset components in the preset component database, the target preset component whose similarity to the target component meets the specified conditions is retrieved. Exemplarily, the similarity meeting the specified conditions can be that among the multiple preset components in the preset component database, the topological structure is the same as that of the target component and the similarity to the target component is the highest, or it can also be that the similarity is higher than the specified similarity threshold, etc.

[0353] Then, shape fitting can be performed according to the component parameters of the target preset component and the target component to generate the target component, so as to operate on the target component according to the target design code.

[0354] Specifically, this target preset component can be used as the benchmark of the target component. When the topological structure of the target preset component is the same as that of the target component, based on this topological structure, the dimensional difference between the target component and the target preset component can be obtained, and shape fitting can be performed based on this difference and relevant triple information, etc., to obtain the target component that meets the shape requirements in the target design code as the component required to generate the target design object.

[0355] After obtaining the components required to generate the target design object, the design software can call the generated target design code and, based on the topological hierarchy (such as a DAG) of the components required to generate the target design object, etc., from bottom to top (for example, in the manner from component parts to higher-level components and then to the target design object), execute design instructions according to the design parameters in the target triple information, etc., to generate the target design object.

[0356] After generating the target design object, the target design object can also be optimized and then the optimized target design object can be output to the user.

[0357] For example, in some embodiments, after obtaining the target design code, it further includes:

[0358] Obtain the target design object obtained by designing by calling the target design code;

[0359] Query the global constraint information corresponding to the target design object from the knowledge graph;

[0360] Optimize the target design object according to the global constraint information.

[0361] In the embodiments of the present application, during the design process, the target design code usually instructs to design from bottom to top and from local to global, and the obtained target design object is usually composed of multiple components.

[0362] Therefore, in order to ensure the rationality of the target design object, the large language model can query the global constraint information corresponding to the target design object from the knowledge graph. Then, it can be detected whether the target design object meets the queried global constraint information through the large language model, etc. If not, the target design object can be adjusted according to the global constraint information to obtain the optimized target design object.

[0363] In addition, in some examples, the cloud management platform can also receive the feedback information of the user on the generated target design object, so as to determine the optimization strategy for the large language model and / or the knowledge graph according to the feedback information, and thus optimize the large language model and / or the knowledge graph.

[0364] Exemplarily, reinforcement learning can be adopted to optimize the large language model, and the target design object and its related information can be used to optimize the knowledge graph with reference to the relevant solutions for constructing the knowledge graph. For example, the weights corresponding to the corresponding design knowledge in the knowledge graph can be updated according to the design knowledge adopted by the target design object. For example, if the feedback information of the user indicates that the target design object is an accurate design object, the weight corresponding to the design knowledge adopted by the target design object in the knowledge graph can be increased; if the feedback information of the user indicates that the design of a certain component in the target design object is inaccurate, the weight corresponding to the design knowledge associated with the design of this component in the knowledge graph can be decreased.

[0365] As Figure 8 shown, it is an exemplary flowchart of an embodiment of the present application.

[0366] Among them, in Figure 8 the example shown, the design task can be an assembly design task.

[0367] In this exemplary scenario, the intent understanding module can obtain the design intent, which can include one or more of the following: the intent obtained based on the second text, the intent obtained based on the second design object, and the intent obtained based on the second image.

[0368] Then, the intent understanding module can query the target design knowledge corresponding to the design intent from the knowledge graph, and thus input the target design knowledge and the design intent into the large language model.

[0369] The large language model processes the target design knowledge and the design intent, and can obtain the target design code and output it to the design software.

[0370] The design software can call the target design code and execute one or more steps such as component matching, shape fitting, assembly parameter determination, and assembly to obtain the assembly plan as the target design object.

[0371] Then, the assembly plan can be globally optimized. If the user determines through the specified configuration interface that the optimized assembly plan meets the requirements, the optimized assembly plan can be output; if the user determines through the specified configuration interface that the optimized assembly plan does not meet the requirements, the design intent and subsequent steps can be re-executed to regenerate the design code to obtain a new assembly plan.

[0372] Next, taking the vehicle styling design scenario as an example, an exemplary information interaction diagram of an embodiment of the present application is introduced.

[0373] In one example, as Figure 9As shown, the vehicle can be modeled, and the modeling design system can include modeling design software. Exemplarily, the modeling design system can be located on the cloud management platform.

[0374] Specifically, this example can include the following steps:

[0375] 1. The user inputs data such as text, design objects, and / or images into the modeling design system;

[0376] 2. The modeling design system pre-processes the data input by the user, such as data decomposition and feature recognition.

[0377] 3. The large language model understands the intention of the pre-processed user input data to obtain the design intention;

[0378] 4. The large language model retrieves design knowledge related to the design intention from the knowledge graph;

[0379] 5. The knowledge graph returns the target design knowledge related to the design intention to the large language model;

[0380] 6. The large language model generates target design code based on the design intention and the target design knowledge;

[0381] 7. The large language model returns the target design code to the modeling design software of the modeling design system;

[0382] 6. The modeling design software calls the target design code to generate a preliminary design scheme as the target design object;

[0383] 7. The large language model queries the global constraint information of the preliminary design scheme from the knowledge graph;

[0384] 8. The knowledge graph returns the global constraint information to the large language model;

[0385] 9. The large language model globally optimizes the preliminary design scheme according to the global constraint information to obtain an optimized design scheme;

[0386] 10. The large language model returns the optimized design scheme to the modeling design system;

[0387] 11. The modeling design system outputs the optimized design scheme to the user;

[0388] 12. The user sends feedback information on the optimized design scheme to the modeling design system;

[0389] 13. The modeling design system formulates an optimization strategy for the large language model and the knowledge graph based on the feedback information;

[0390] 14. Optimize the large language model;

[0391] 15. Optimize the knowledge graph.

[0392] In another example, the vehicle can be first styled, and then the assembly design can be carried out according to the styling design result. The styling design system can include styling design software, and the assembly design system can include assembly design software. Exemplarily, the styling design system and the assembly design system can be located on a cloud management platform. As Figure 10 shown, this example can specifically include the following steps:

[0393] 1. User 1 (such as a styling designer) inputs first input data such as text, design objects, and / or images into the styling design system;

[0394] 2. The styling design system preprocesses the user input data such as data decomposition and feature recognition.

[0395] 3. The large language model for styling design understands the intention of the preprocessed user input data to obtain the design intention;

[0396] 4. The large language model for styling design retrieves design knowledge related to the design intention from the knowledge graph;

[0397] 5. The knowledge graph for styling design returns the first target design knowledge related to the design intention to the large language model;

[0398] 6. The large language model for styling design generates the first target design code based on the design intention and the first target design knowledge;

[0399] 7. The large language model for styling design returns the first target design code to the styling design software of the styling design system;

[0400] 6. The styling design software calls the first target design code to generate a preliminary styling design scheme;

[0401] 7. The large language model for styling design queries the global constraint information of the preliminary styling design scheme from the knowledge graph for styling design;

[0402] 8. The knowledge graph for styling design returns the global constraint information to the large language model for styling design;

[0403] 9. The large language model for styling design globally optimizes the preliminary styling design scheme according to the global constraint information to obtain an optimized styling design scheme;

[0404] 10. The large language model for styling design returns the optimized styling design scheme to the styling design system;

[0405] 11. The styling design system outputs the optimized styling design scheme to User 2 (such as an assembly designer);

[0406] 12. User 2 inputs second input data such as text, design objects, and / or images into the assembly design system;

[0407] 13. The assembly design system preprocesses the data input by the user, such as data decomposition, feature recognition, etc.;

[0408] 14. The large language model for assembly design understands the intent of the preprocessed user input data to obtain the design intent;

[0409] 15. The large language model for assembly design retrieves design knowledge related to the design intent from the knowledge graph;

[0410] 16. The knowledge graph for assembly design returns the second target design knowledge related to the design intent to the large language model;

[0411] 17. The large language model for assembly design generates the second target design code based on the design intent and the second target design knowledge;

[0412] 18. The large language model for assembly design returns the second target design code to the assembly design software of the assembly design system;

[0413] 19. The assembly design software calls the second target design code to generate a preliminary assembly design scheme;

[0414] 20. The large language model for assembly design queries the knowledge graph for assembly design for the global constraint information of the preliminary assembly design scheme;

[0415] 21. The knowledge graph for assembly design returns the global constraint information to the large language model for assembly design;

[0416] 22. The large language model for assembly design globally optimizes the preliminary assembly design scheme according to the global constraint information to obtain the optimized assembly design scheme;

[0417] 23. The large language model for assembly design returns the optimized assembly design scheme to the assembly design system;

[0418] 24. User 2 sends feedback information on the optimized assembly design scheme to the assembly design system;

[0419] 25. The assembly design system formulates an optimization strategy for the large language model for assembly design and the knowledge graph for assembly design according to the feedback information;

[0420] 26. Optimize the large language model for assembly design;

[0421] 27. Optimize the knowledge graph for assembly design.

[0422] The above introduced the design code generation method provided by the embodiments of the present application from multiple aspects. Next, in conjunction with the accompanying drawings, the design code generation device 11 and the knowledge graph construction device 12 provided by the embodiments of the present application will be introduced.

[0423] As Figure 11 shown, the embodiments of the present application provide a design code generation device 11 based on cloud services, which is applied to a cloud management platform. The cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple regions, each region includes at least one cloud data center, and the cloud services run on at least one server in at least one cloud data center located in multiple regions. The device 11 includes:

[0424] An interface module 1101, configured to provide a configuration interface, and the configuration interface is used to obtain information about the design intention input by the user, and the design intention is used to indicate the generation of a target design object;

[0425] A processing module 1102, configured to:

[0426] Retrieve the target design knowledge corresponding to the design intention from the knowledge graph. The knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. The target design knowledge is used to describe the implicit design intention for the target design object;

[0427] Process the design intention and the target design knowledge through a large language model to obtain a target design code, and the target design code is used to generate the target design object.

[0428] Optionally, the interface module 1101 is configured to:

[0429] Obtain multimodal design materials. The multimodal design materials include the first design object, and the multimodal design materials also include one or more of the following: the first text and the first image;

[0430] The processing module 1102 is configured to:

[0431] Obtain the first geometric feature of the first component among one or more components included in the first design object according to the first design object. The first geometric feature is used to describe the features related to the geometric elements in the first component;

[0432] Obtain the design knowledge corresponding to the first design object according to the first geometric feature;

[0433] After obtaining the design knowledge corresponding to the design materials of each modality, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a target joint characterization;

[0434] According to the target joint characterization, a knowledge graph is constructed.

[0435] Optionally, the target design code includes target triple information and information of at least one design code module, and the at least one design code module is used to generate a target design object, and the target triple information includes an operation object, an operation instruction, and operation parameters corresponding to at least one design code module.

[0436] Optionally, the processing module 1102 is used to:

[0437] Process the design intention and the target design knowledge through a large language model to obtain a target module identifier and target triple information, and the target module identifier includes module identifiers corresponding to at least one design code module;

[0438] According to the target module identifier and the target triple information, obtain the target design code through a large language model.

[0439] Optionally, the design intention includes an intention obtained based on a second design object,

[0440] The interface module 1101 is used to: obtain the second geometric features of the second component associated with the target design object among one or more components included in the second design object, and the second geometric features are used to describe the features related to geometric elements in the second component;

[0441] The processing module 1102 is used to: obtain the intention obtained based on the second design object according to the second geometric features.

[0442] Optionally, the processing module 1102 is used to:

[0443] Decompose the second design object to obtain one or more levels of components in the second design object;

[0444] From one or more levels of components in the second design object, obtain the components at the target level as the second component;

[0445] Obtain the second geometric features of the second component.

[0446] Optionally, the design intention further includes an intention obtained based on a second text, and the processing module 1102 is used to:

[0447] Jointly characterize the design information in the second geometric features and the second text to obtain a first joint characterization;

[0448] According to the first combined representation, obtain the first combined representation intention, which includes the intention obtained based on the second design object and the intention obtained based on the second text.

[0449] Optionally, the design intention further includes the intention obtained based on the second image. The processing module 1102 is configured to:

[0450] Identify the design information in the second image through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, attributes of entities in the second image;

[0451] Perform a combined representation of the second geometric feature and the design information in the second image to obtain a second combined representation;

[0452] According to the second combined representation, obtain the second combined representation intention, which includes the intention obtained based on the second design object and the intention obtained based on the second image.

[0453] Optionally, the second geometric feature includes one or more of the following:

[0454] Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, gradient features of the second component.

[0455] As Figure 12 shown, an embodiment of the present application provides a knowledge graph construction device 12, which is applied to a cloud management platform. The cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple regions, each region includes at least one cloud data center, and cloud services run on at least one server in at least one cloud data center located in multiple regions. The device 12 includes:

[0456] An interface module 1201, configured to obtain multi-modal design materials. The multi-modal design materials include a first design object, and the multi-modal design materials further include one or more of the following: a first text, a first image;

[0457] A processing module 1202, configured to:

[0458] According to the first design object, obtain the first geometric feature of the first component in one or more components included in the first design object. The first geometric feature is used to describe the features related to geometric elements in the first component;

[0459] According to the first geometric feature, obtain the design knowledge corresponding to the first design object;

[0460] After obtaining the design knowledge corresponding to the design materials of each modality, the design knowledge corresponding to the design materials of each modality is jointly characterized to obtain a target joint characterization;

[0461] According to the target joint characterization, a knowledge graph is constructed.

[0462] Optionally, the interface module 1201 is configured to: provide a configuration interface, and the configuration interface is used to obtain information about the design intention input by the user, and the design intention is used to indicate the generation of a target design object;

[0463] The processing module 1202 is configured to:

[0464] Retrieve the target design knowledge corresponding to the design intention from the knowledge graph, and the target design knowledge is used to describe the implicit design intention of the target design object;

[0465] Process the design intention and the target design knowledge through a large language model to obtain a target design code, and the target design code is used to generate a target design object.

[0466] Optionally, the number of the first design objects is multiple, and the processing module 1202 is configured to:

[0467] According to the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects, determine the target first geometric feature from the first geometric features corresponding to the multiple first design objects;

[0468] Obtain the design knowledge corresponding to the first design object according to the target first geometric feature.

[0469] Optionally, in the knowledge graph, the weight of the design knowledge corresponding to the target first geometric feature is determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship.

[0470] Optionally, the knowledge graph includes a first sub-graph and at least one second sub-graph. The first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to the multi-modal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one first text, design knowledge from at least one first image, design knowledge from multiple first design objects and the co-occurrence probability in the multiple first design objects meets the specified conditions. Any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more first design objects, design knowledge corresponding to the first text associated with one or more first design objects, design knowledge corresponding to the first image associated with one or more first design objects.

[0471] Among them, both the processing module and the interface module can be implemented by software or by hardware. Exemplarily, next, taking the processing module as an example, the implementation manner of the processing module will be introduced. Similarly, the implementation manner of the interface module can refer to the implementation manner of the processing module.

[0472] As an example of a software functional unit, the processing module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the processing module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code may be distributed in the same region, or may be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ), or may be distributed in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Among them, generally, one region may include multiple AZs.

[0473] Similarly, the multiple hosts / virtual machines / containers for running the code may be distributed in the same virtual private cloud (VPC), or may be distributed in multiple VPCs. Among them, generally, one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC, and the interconnection between VPCs is realized through the communication gateway.

[0474] As an example of a hardware functional unit, the processing module may include at least one computing device, such as a server. Alternatively, the processing module may also be implemented using a central processing unit (CPU), or may be implemented using an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. Among them, the above PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offloading card, an acceleration card, or any combination thereof.

[0475] The multiple computing devices included in the processing module may be distributed in the same region or in different regions. The multiple computing devices included in the processing module may be distributed in the same availability zone (AZ) or in different AZs. Similarly, the multiple computing devices included in the processing module may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, the multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, acceleration cards, etc.

[0476] It should be noted that in other embodiments, the processing module may be used to execute any step in the design code generation method or the knowledge graph construction method based on cloud services, and the interface module may be used to execute any step in the design code generation method or the knowledge graph construction method based on cloud services. The steps to be implemented by the processing module and the interface module can be specified as needed. By implementing different steps in the design code generation method or the knowledge graph construction method based on cloud services through the processing module and the interface module respectively, all functions of the design code generation device or the knowledge graph construction device based on cloud services can be realized.

[0477] The embodiments of the present application also provide a computing device 130. As Figure 13As shown, the computing device 130 includes: a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other via the bus 132. The computing device 130 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 130.

[0478] The bus 132 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. Among them, the unified bus is also known as the Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 13 only one line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The bus 132 can include a path for transmitting information between various components of the computing device 130 (for example, the memory 136, the processor 134, and the communication interface 138).

[0479] The processor 134 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, and other computing devices.

[0480] The memory 136 may include volatile memory, such as random access memory (RAM). The memory 136 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD), or one or more of them. In addition, the memory 136 may also be implemented by storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0481] It is worth noting that in the same computing device, the function of the memory 136 may be implemented by configuring the same type of storage medium, or two or more types of storage media may be configured to implement the function of the memory 136. This application does not make any limitations in this regard.

[0482] The memory 136 stores executable program code, and the processor 134 executes the executable program code to implement the functions of the foregoing interface module and processing module respectively, so as to implement the design code generation method or knowledge graph construction method based on cloud services applied to the cloud management platform in the foregoing embodiments. That is, the memory 136 stores instructions for executing the design code generation method or knowledge graph construction method based on cloud services applied to the cloud management platform in the foregoing embodiments.

[0483] The communication interface 138 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 130 and other devices or a communication network.

[0484] The embodiments of this application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0485] As Figure 14 shown, the computing device cluster includes at least one computing device 130. The memory 136 in one or more of the computing devices 130 in the computing device cluster may store the same instructions for executing the design code generation method or the knowledge graph construction method.

[0486] In some possible implementations, the memories 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the design code generation method based on cloud services or the knowledge graph construction method respectively. In other words, a combination of one or more computing devices 130 can jointly execute the instructions for executing the design code generation method based on cloud services or the knowledge graph construction method.

[0487] It should be noted that the memories 136 in different computing devices 130 in the computing device cluster can store different instructions, which are respectively used to execute partial functions of the design code generation method based on cloud services or the knowledge graph construction method. That is to say, the instructions stored in the memories 136 of different computing devices 130 can implement the functions of one or more modules in the interface module and the processing module.

[0488] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. Among them, the network can be a wide area network or a local area network, etc. Figure 15 A possible implementation is shown. As Figure 15 shown, the computing device 130A and the computing device 130B are connected through a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 136 in the computing device 130A may store instructions for executing the functions of the interface module. At the same time, the memory 136 in the computing device 130B may store instructions for executing the functions of the processing module. Or, in other examples, the memory 136 in the computing device 130A may store instructions for executing partial functions of the processing module. At the same time, the memory 136 in the computing device 130B may store instructions for executing another part of the functions of the processing module.

[0489] It should be understood that Figure 15 the functions of the computing device 130A shown in

[0490] can also be completed by multiple computing devices 130. Similarly, the functions of the computing device 130B can also be completed by multiple computing devices 130. Figure 14 and Figure 15 The connection manner of the computing device cluster. The difference is that the memories 136 of one or more computing devices 130 in this computing device cluster may store the same instructions for executing the design code generation method based on cloud services or the knowledge graph construction method.

[0491] In some possible implementations, the memories 136 of one or more computing devices 130 in the computing device cluster may also store some instructions for executing the design code generation method based on cloud services or the knowledge graph construction method respectively. In other words, a combination of one or more computing devices 130 may jointly execute the instructions for executing the design code generation method based on cloud services or the knowledge graph construction method.

[0492] It should be noted that the memories 136 in different computing devices 130 in the computing device cluster may store different instructions for implementing some functions of the design code generation method based on cloud services or the knowledge graph construction method. That is, the instructions stored in the memories 136 of different computing devices 130 may implement the functions of one or more modules in the interface module and the processing module.

[0493] The embodiments of the present application also provide a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute the design code generation method based on cloud services or the knowledge graph construction method.

[0494] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute the design code generation method based on cloud services or the knowledge graph construction method.

[0495] The embodiments of the present application also provide a chip system. The chip system includes a processor for implementing the steps executed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0496] Those skilled in the art can clearly understand that for the sake of convenience and brevity of description, the specific working processes of the above-described systems, devices, and units may refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0497] In several embodiments provided by this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0498] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0499] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0500] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs and other various media that can store program codes.

Claims

1. A design code generation method based on cloud services, characterized in that, The method is applied to a cloud management platform for managing the infrastructure that provides cloud services. The infrastructure includes multiple regions, each region including at least one cloud data center. The cloud services run on at least one server in at least one cloud data center located in multiple regions. The method includes: Providing a configuration interface for obtaining information about the design intent input by a user, where the design intent is used to indicate the generation of a target design object; Retrieving, from a knowledge graph, target design knowledge corresponding to the design intent. The knowledge graph includes one or more of the following: design knowledge from a first text, design knowledge from a first design object, and design knowledge from a first image. The target design knowledge is used to describe the implicit design intent for the target design object, and the first design object is a design object generated in a historical design task; Processing the design intent and the target design knowledge through a large language model to obtain target design code for generating the target design object.

2. The method according to claim 1, wherein Before retrieving the design knowledge corresponding to the design intent from the knowledge graph, the method further includes: Obtaining multi-modal design materials, where the multi-modal design materials include the first design object, and the multi-modal design materials further include one or more of the following: the first text and the first image; Obtaining, according to the first design object, first geometric features of a first component among one or more components included in the first design object, where the first geometric features are used to describe features related to geometric elements in the first component; Obtaining, according to the first geometric features, design knowledge corresponding to the first design object; After obtaining the design knowledge corresponding to each modality of the design materials, performing joint representation on the design knowledge corresponding to each modality of the design materials to obtain a target joint representation; Constructing the knowledge graph according to the target joint representation.

3. The method according to claim 1 or 2, characterized in that, The target design code includes target triple information and information of at least one design code module for generating the target design object. The target triple information includes the operation object, operation instruction, and operation parameter corresponding to the at least one design code module.

4. The method according to claim 3, wherein The processing the design intent and the target design knowledge through a large language model to obtain target design code includes: Processing the design intent and the target design knowledge through the large language model to obtain a target module identifier and the target triple information, where the target module identifier includes the module identifier corresponding to the at least one design code module; Obtaining the target design code through the large language model according to the target module identifier and the target triple information.

5. The method according to claim 1, characterized in that The design intent includes an intent obtained based on a second design object, and the method further includes: Obtain the second geometric feature of the second component associated with the target design object among one or more components included in the second design object, where the second geometric feature is used to describe the features related to geometric elements in the second component; Obtain the intention obtained based on the second design object according to the second geometric feature.

6. The method according to claim 5, wherein The obtaining the second geometric feature of the second component associated with the target design object among one or more components included in the second design object includes: Decompose the second design object to obtain one or more levels of components in the second design object; Obtain the components at the target level from one or more levels of components in the second design object as the second component; Obtain the second geometric feature of the second component.

7. The method according to claim 5, characterized in that The design intention further includes the intention obtained based on the second text. The obtaining the intention obtained based on the second design object according to the second geometric feature includes: Jointly represent the design information in the second geometric feature and the second text to obtain a first joint representation; Obtain a first joint representation intention according to the first joint representation, where the first joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second text.

8. The method according to claim 5, characterized in that The design intention further includes the intention obtained based on the second image. The method further includes: Identify the design information in the second image through object detection and / or semantic segmentation. The design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, attributes of entities in the second image; The obtaining the intention obtained based on the second design object according to the second geometric feature includes: Jointly represent the second geometric feature and the design information in the second image to obtain a second joint representation; Obtain a second joint representation intention according to the second joint representation, where the second joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second image.

9. The method according to any one of claims 5 - 8, characterized in that, The second geometric feature includes one or more of the following: Spatial information of geometric elements in the second component, semantic information used to describe the second component and / or geometric elements in the second component, structural features of the second component, gradient features of the second component.

10. A method for constructing a knowledge graph, characterized in that, The method is applied to a cloud management platform, which is used to manage the infrastructure for providing cloud services. The infrastructure includes being set in multiple regions, and each region includes at least one cloud data center. The cloud services run on at least one server in at least one cloud data center located in multiple regions. The method includes: Obtain multimodal design materials, where the multimodal design materials include a first design object, and the multimodal design materials further include one or more of the following: a first text, a first image. The first design object is a design object generated in a historical design task; Based on the first design object, obtain the first geometric feature of the first component among one or more components included in the first design object, where the first geometric feature is used to describe the features related to geometric elements in the first component; Based on the first geometric feature, obtain the design knowledge corresponding to the first design object; After obtaining the design knowledge corresponding to the design materials of each modality, perform joint representation on the design knowledge corresponding to the design materials of each modality to obtain a target joint representation; Construct a knowledge graph based on the target joint representation.

11. The method according to claim 10, characterized in that, The method further includes: Provide a configuration interface for obtaining information on the design intent input by the user, where the design intent is used to indicate the generation of a target design object; Retrieve the target design knowledge corresponding to the design intent from the knowledge graph, where the target design knowledge is used to describe the implicit design intent for the target design object; Process the design intent and the target design knowledge through a large language model to obtain a target design code, where the target design code is used to generate the target design object.

12. The method according to claim 10, wherein The number of the first design objects is multiple, and the step of obtaining the design knowledge corresponding to the first design object according to the first geometric feature includes: Determine a target first geometric feature from the first geometric features corresponding to the multiple first design objects according to the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects; Obtain the design knowledge corresponding to the first design object according to the target first geometric feature.

13. The method according to claim 12, wherein In the knowledge graph, the weight of the design knowledge corresponding to the target first geometric feature is determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship.

14. The method according to any one of claims 10-13, characterized in that, The knowledge graph includes a first sub-graph and at least one second sub-graph. The first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to the multi-modal design materials, and the common design knowledge includes one or more of the following: design knowledge from at least one of the first texts, design knowledge from at least one of the first images, design knowledge from the multiple first design objects and whose co-occurrence probability among the multiple first design objects meets the specified conditions. Any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more of the first design objects, design knowledge corresponding to the first texts associated with one or more of the first design objects, design knowledge corresponding to the first images associated with one or more of the first design objects.

15. A design code generation device based on cloud services, characterized in that, Applied to a cloud management platform for managing the infrastructure that provides cloud services. The infrastructure includes settings in multiple regions, each region includes at least one cloud data center, and the cloud services run on at least one server in at least one cloud data center located in multiple regions. The device includes: An interface module for providing a configuration interface for obtaining information on the design intent input by the user, where the design intent is used to indicate the generation of a target design object; A processing module for: Retrieve the target design knowledge corresponding to the design intention from the knowledge graph, where the knowledge graph includes one or more of the following: design knowledge from the first text, design knowledge from the first design object, and design knowledge from the first image. The target design knowledge is used to describe the implicit design intention for the target design object, and the first design object is a design object generated in a historical design task; Process the design intention and the target design knowledge through a large language model to obtain a target design code, where the target design code is used to generate the target design object.

16. The apparatus according to claim 15, wherein The interface module is configured to: Obtain multimodal design materials, where the multimodal design materials include the first design object, and the multimodal design materials further include one or more of the following: the first text and the first image; The processing module is configured to: According to the first design object, obtain the first geometric feature of the first component among one or more components included in the first design object, where the first geometric feature is used to describe the features related to geometric elements in the first component; According to the first geometric feature, obtain the design knowledge corresponding to the first design object; After obtaining the design knowledge corresponding to the design materials of each modality, perform joint representation on the design knowledge corresponding to the design materials of each modality to obtain a target joint representation; Construct the knowledge graph according to the target joint representation.

17. The device according to claim 15, characterized in that, The target design code includes target triple information and information of at least one design code module. The at least one design code module is used to generate the target design object, and the target triple information includes the operation object, operation instruction, and operation parameter corresponding to the at least one design code module.

18. The apparatus according to claim 17, wherein The processing module is configured to: Process the design intention and the target design knowledge through the large language model to obtain a target module identifier and the target triple information, where the target module identifier includes the module identifier corresponding to the at least one design code module; According to the target module identifier and the target triple information, obtain the target design code through the large language model.

19. The device according to claim 15, characterized in that, The design intention includes an intention obtained based on a second design object, The interface module is configured to: obtain the second geometric feature of the second component associated with the target design object among one or more components included in the second design object, where the second geometric feature is used to describe the features related to geometric elements in the second component; The processing module is configured to: according to the second geometric feature, obtain the intention obtained based on the second design object.

20. The device according to claim 19, characterized in that, The processing module is configured to: Decompose the second design object to obtain one or more levels of components in the second design object; From one or more levels of components in the second design object, obtain the components of the target level as the second component; Obtain the second geometric feature of the second component.

21. The device according to claim 19, wherein, The design intention further includes an intention obtained based on the second text, and the processing module is configured to: Jointly represent the design information in the second geometric feature and the second text to obtain a first joint representation; Based on the first joint representation, obtain a first joint representation intention, where the first joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second text.

22. The device according to claim 19, wherein The design intention further includes an intention obtained based on the second image, and the processing module is configured to: Identify the design information in the second image through object detection and / or semantic segmentation, where the design information in the second image includes one or more of the following: entities in the second image, relationships between entities in the second image, and attributes of entities in the second image; Jointly represent the second geometric feature and the design information in the second image to obtain a second joint representation; Based on the second joint representation, obtain a second joint representation intention, where the second joint representation intention includes the intention obtained based on the second design object and the intention obtained based on the second image.

23. The device according to any one of claims 19 - 22, characterized in that, The second geometric feature includes one or more of the following: Spatial information of geometric elements in the second component, semantic information for describing the second component and / or geometric elements in the second component, structural features of the second component, and gradient features of the second component.

24. A knowledge graph construction device, characterized in that, Applied to a cloud management platform, the cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes settings in multiple regions, each region includes at least one cloud data center, and the cloud services run on at least one server in at least one cloud data center located in multiple regions. The device includes: An interface module for obtaining multimodal design materials. The multimodal design materials include a first design object, and the multimodal design materials further include one or more of the following: a first text, a first image. The first design object is a design object generated in a historical design task; A processing module for: Based on the first design object, obtain a first geometric feature of a first component in one or more components included in the first design object. The first geometric feature is used to describe features related to geometric elements in the first component; Based on the first geometric feature, obtain design knowledge corresponding to the first design object; After obtaining the design knowledge corresponding to each modality of the design materials, jointly represent the design knowledge corresponding to each modality of the design materials to obtain a target joint representation; Based on the target joint representation, construct a knowledge graph.

25. The device according to claim 24, wherein The interface module is configured to: provide a configuration interface for obtaining information on a design intention input by a user, where the design intention is used to indicate the generation of a target design object; The processing module is configured to: Retrieve the target design knowledge corresponding to the design intent from the knowledge graph, where the target design knowledge is used to describe the implicit design intent for the target design object; Process the design intent and the target design knowledge through a large language model to obtain a target design code, where the target design code is used to generate the target design object.

26. The device according to claim 24, characterized in that The number of the first design objects is multiple, and the processing module is configured to: Determine a target first geometric feature from the first geometric features corresponding to the multiple first design objects according to the co-occurrence relationship of the first geometric features corresponding to the multiple first design objects; Obtain the design knowledge corresponding to the first design object according to the target first geometric feature.

27. The device according to claim 26, characterized in that, In the knowledge graph, the weight of the design knowledge corresponding to the target first geometric feature is determined according to the co-occurrence probability corresponding to the target first geometric feature in the co-occurrence relationship.

28. The device according to any one of claims 24-27, characterized in that, The knowledge graph includes a first sub-graph and at least one second sub-graph. The first sub-graph is used to describe the common design knowledge in the design knowledge corresponding to the multi-modal design materials. The common design knowledge includes one or more of the following: design knowledge from at least one of the first texts, design knowledge from at least one of the first images, design knowledge from the multiple first design objects and whose co-occurrence probability among the multiple first design objects meets a specified condition. Any one of the second sub-graphs is obtained according to one or more of the following information: design knowledge corresponding to one or more of the first design objects, design knowledge corresponding to the first text associated with one or more of the first design objects, design knowledge corresponding to the first image associated with one or more of the first design objects.

29. A cluster of computing devices, characterized in that, Including at least one computing device, the at least one computing device includes a processor and a memory; The processor is configured to execute the instructions stored in the memory to cause the computing device cluster to execute the method according to any one of claims 1-9, or execute the method according to any one of claims 10-14.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program runs on the processor, it causes the processor to execute the method according to any one of claims 1-14.

31. A computer program product comprising instructions, characterized in that, When the instructions are executed by the processor, the method according to any one of claims 1-14 is implemented.

Images