Origami prediction method and system based on multimodal graph learning

Abstract

An origami prediction method and system based on multimodal graph learning is provided. [Solution] In this invention, folding design parameters and an initial fold line image are obtained, a graph topology structure and a two-dimensional fold line image are generated using the folding design parameters, feature extraction is performed on the graph topology structure and the two-dimensional fold line image using a graph convolutional neural network to obtain first geometric features, the initial fold line image and the first geometric features are sent to a graph attention network for feature learning, second geometric features and image features are obtained, feature fusion is performed on the second geometric features and image features, and the fused features are analyzed using a multilayer perceptron to output an origami prediction result. This allows the invention to achieve efficient and highly accurate prediction of origami results and improve the versatility of the prediction model.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This invention claims priority to a Chinese patent application bearing application number 202510213165.5 and entitled "Origami prediction method and system based on multimodal graph learning," filed with the State Intellectual Property Office of the People's Republic of China on February 26, 2025, the entire contents of which are incorporated herein by reference and constitute a part of the present invention for all purposes.

[0002] The present invention relates to the technical field of image processing, and in particular to an origami prediction method and system based on multimodal graph learning. [Background technology]

[0003] The discussion in this section merely provides background information related to the present invention and does not necessarily constitute prior art.

[0004] Origami structures are a novel structural system that can be folded along predefined fold lines (mountain folds, valley folds, etc.) on a two-dimensional plane without the need for cutting or stretching. Origami structures are characterized by their flexible modality, high design flexibility, excellent rigidity, and excellent folding and unfolding ratio, demonstrating excellent performance in device structures. Therefore, origami structures are being widely used in fields such as aerospace, medical devices, and robotics. For example, a satellite solar panel has a diameter of only 2.5 m when folded and can reach a diameter of 27 m when fully unfolded, providing a large folding and unfolding ratio and enabling a larger practical area in a small storage space.

[0005] The key to origami prediction is to simulate and predict the folding process and final result based on the initial conditions of the origami, such as the size, shape, and crease distribution of the folding material, while being constrained by the folding rules. Calculating and inferring the 3D folded shape from 2D crease images not only enables prediction of artistic origami results, but also demonstrates the advantages of origami structures in the industrial field.

[0006] The inventors have discovered that the conventional methods for predicting results on origami structures have several technical problems, including the following: (1) Current common origami implementation methods rely primarily on manual experience and limited simulation inference. For example, designers first design the folding lines, and then manually or mechanically implement them according to the predefined folding lines. However, this method requires a long and labor-intensive process. Furthermore, designing relatively precise folding structures requires highly accurate manufacturing equipment, making it difficult to implement on a large scale. (2) Traditional origami structure design methods mainly rely on traditional origami rules such as the Huzita-Hatori axioms, Maekawa's conditions, Maekawa's theorem, and the 2-colorable condition. Although these theories clearly constrain the geometric transformations in the folding process, the computational process is very complicated, which results in low computational efficiency. (3) Conventional simulation and prediction software (e.g., Origami Simulator) that can perform inference calculations on the folding process of origami structures accelerates prediction speed to a certain extent. However, such simulation software requires recalculation for each prediction, lacks learning and generalization capabilities, and is significantly limited in the case of complex origami structures. At the same time, there has been an increasing amount of research exploring new origami prediction methods using deep learning technology. However, current deep learning-based models have difficulty simultaneously capturing the geometric and shape features of origami structures when used in the field of origami result prediction. As a result, the resulting prediction results are inaccurate in both modality and geometric structure, resulting in unsatisfactory prediction results.

[0007] To summarize, existing origami prediction methods generally suffer from the problems of low versatility, low prediction efficiency, and low accuracy of prediction results when dealing with large-volume, high-complexity prediction tasks. Summary of the Invention

[0008] To overcome the drawbacks of the prior art mentioned above, the present invention provides an origami prediction method and system based on multimodal graph learning, which can realize efficient and highly accurate prediction of origami results and improve the versatility of the prediction model.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions.

[0010] In a first aspect of the present invention, an origami prediction method based on multimodal graph learning is provided.

[0011] obtaining folding design parameters and an initial fold line image; generating a graph topology structure and a two-dimensional fold line image according to the folding design parameters; performing feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features; transmitting the initial crease line image and the first geometric features to a graph attention network to perform feature learning and obtain second geometric features and image features; and performing feature fusion on the second geometric features and image features, analyzing the fused features using a multi-layer perceptron, and outputting origami prediction results.

[0012] Furthermore, the folding design parameters include vertex coordinates, geometric constraints, fold line edge sets, and fold line types.

[0013] Furthermore, the vertex coordinates are used as node features, and the geometric constraints and fold line types are used as edge features.

[0014] Furthermore, performing feature extraction on the graph topology structure and the two-dimensional fold line image using a graph convolutional neural network includes setting a convolutional stride, using the graph convolutional neural network to generate a subgraph with the stride as a distance from the current node of the graph topology structure, transmitting information to the node features and edge features of the subgraph, and aggregating the feature information represented in the two-dimensional fold line image to obtain a first geometric feature.

[0015] Furthermore, the graph attention network performs global weight calculation and feature learning on the node features and edge features in the initial fold line image and the first geometric features to obtain the second geometric features and image features.

[0016] Furthermore, before using the graph convolutional neural network and the graph attention network to perform origami prediction, the graph convolutional neural network and the graph attention network need to be trained, and in order to ensure the accuracy of the model training, an augmentation operation is performed on the training data to increase the amount of training data.

[0017] Further, training the graph convolutional neural network and the graph attention network includes: defining a fusion loss function according to a prediction task; comparing the prediction result of the fused features with a ground truth label to calculate a loss value; then backpropagating the loss value to the graph convolutional neural network and the graph attention network based on a backpropagation algorithm; and iteratively updating parameters in the graph convolutional neural network and the graph attention network based on a gradient descent algorithm.

[0018] In a second aspect of the present invention, an origami prediction system based on multimodal graph learning is provided.

[0019] a data acquisition module configured to acquire folding design parameters and initial fold line images; an object generation module configured to generate a graph topology structure and a two-dimensional foldline image according to the folding design parameters; a feature extraction module configured to perform feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features, and to transmit the initial crease line image and the first geometric features to a graph attention network for feature learning to obtain second geometric features and image features; an origami prediction module configured to perform feature fusion on the second geometric features and image features, analyze the fused features using a multi-layer perceptron, and output origami prediction results.

[0020] In a third aspect of the present invention, there is provided a computer-readable storage medium having a program stored thereon, the program, when executed by a processor, realizing steps in the origami prediction method based on multimodal graph learning described in the first aspect of the present invention.

[0021] In a fourth aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a program stored in the memory and executable by the processor, wherein when the program is executed by the processor, steps in the origami prediction method based on multimodal graph learning described in the first aspect of the present invention are realized.

[0022] The above one or more technical measures have the following beneficial effects: (1) This invention uses a graph convolutional neural network to extract features from a graph topology structure and a two-dimensional fold line image to obtain geometric features. It then uses a graph attention network to further learn the geometric features, learning image features. Finally, it fuses the learned image features with the geometric features. Therefore, this invention can accurately fuse multimodal features while satisfying the constraints in the origami theorem and exhibits good adaptability to different types of origami scenes. This network architecture design makes it possible to meet the challenges of various origami scenes and improves the versatility of the prediction model. (2) By combining a graph convolutional neural network and a graph attention network, the present invention can effectively reduce the amount of computation while preserving key information. At the same time, the model has excellent generalization ability, which significantly reduces the computational power consumption in the case of similar structures and improves computational efficiency. (3) The present invention not only extracts geometric structure features but also analyzes image features, ensuring highly accurate prediction results for both shape and specific geometric structures. At the same time, the present invention performs an expansion operation on the training data, increasing the amount of training data, thereby giving the trained model better predictive capabilities. Therefore, the present invention can achieve highly accurate prediction of origami results.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the invention. [Brief explanation of the drawings]

[0024] The specification drawings that form a part of this invention are intended to provide a further understanding of the invention, and the illustrative embodiments of the invention and their descriptions are intended to interpret the invention and are not intended to unduly limit the invention.

[0025] [Figure 1] 1 is a flowchart of an origami prediction method based on multimodal graph learning in Example 1 of the present invention. [Figure 2] FIG. 1 is a schematic diagram of data relationships in the origami prediction method based on multimodal graph learning in Example 1 of the present invention. [Figure 3] FIG. 2 is a schematic diagram of model input data in the first embodiment of the present invention. [Figure 4] FIG. 1 is a schematic diagram of an information transmission process of a graph convolutional neural network in Example 1 of the present invention. [Figure 5] FIG. 1 is a schematic diagram of the information transmission process of the graph attention network in Example 1 of the present invention. [Figure 6] FIG. 2 is a schematic diagram of a feature fusion process in Example 1 of the present invention. [Figure 7] 1 is an exemplary view of a two-dimensional fold line design diagram according to a first embodiment of the present invention; [Figure 8] FIG. 2 is a schematic diagram of a fold line result prediction corresponding to a two-dimensional fold line design drawing in the first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] It should be pointed out that the following detailed description is exemplary and is intended to further explain the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0027] It should be noted that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit example embodiments according to the present invention.

[0028] Unless contradictory, the embodiments of the present invention and the features of the embodiments can be combined with each other. Example 1

[0029] This embodiment discloses an origami prediction method based on multimodal graph learning.

[0030] As shown in Figures 1 and 2, this origami prediction method is based on multimodal graph learning and can be applied to predicting the two-dimensional folding structure of complex origami models, such as the folding pattern of an origami crane. By inputting folding design parameters such as the vertex coordinates, crease type, and geometric constraints of the origami crane, the model generates a corresponding graph topology structure. A graph convolutional neural network (GCN) can then perform feature extraction on the crease image and geometric information. A graph attention network (GAT) then performs global weight calculations on these features to obtain accurate origami results. This method not only effectively predicts the two-dimensional structure of complex origami models such as origami cranes, but also provides detailed shape predictions at each stage of the folding process, thereby helping designers optimize the origami process and improve design accuracy. Specifically, Step S1 of acquiring folding design parameters and an initial fold line image; Step S2 of generating a graph topology structure and a two-dimensional fold line image according to the folding design parameters; Step S3 of performing feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features, and transmitting the initial crease line image and the first geometric features to a graph attention network to perform feature learning and obtain second geometric features and image features; and step S4 of performing feature fusion on the second geometric features and image features, analyzing the fused features using a multi-layer perceptron, and outputting the origami prediction result.

[0031] According to the above process, the present invention can improve the versatility of the prediction model and realize efficient and highly accurate prediction of origami results. In order to better understand the technical means of the present invention, the specific implementation steps of the present invention will be further explained below.

[0032] In step S1, folding design parameters and an initial fold line image are obtained.

[0033] As shown in Figure 3, the main input data used in the model include vertex coordinates, geometric constraints, fold line edge sets, fold line types, and folding degrees, etc. Here, the geometric constraints are various geometric constraints in the folding theorem, and the fold line types mainly include mountain folds, valley folds, and boundary folds.

[0034] The user inputs the folding design parameters and initial fold line image themselves. Specifically, in this embodiment, the folding design parameters include vertex coordinates, geometric constraints, fold line edge set, and fold line type, and the constraints required for calculation are selected here as Maekawa's theorem and Kawasaki's theorem.

[0035] Furthermore, Maekawa's theorem states that the difference between the number of mountain folds and the number of valley folds in the folding direction of each fold line that crosses any vertex in the fold design is always 2. Maekawa's theorem states that

number

number

number

number

[0036] Furthermore, Kawasaki's theorem states that when the included angles of the fold lines around any vertex in the fold line design are numbered, the sum of the odd-numbered included angles and the even-numbered included angles is equal and always 180°.

number

number

[0037] In step S2, a graph topology structure and a two-dimensional fold line image are generated based on the folding design parameters.

[0038] Based on a Python algorithm, a graph topology structure is generated from the vertex coordinates of the 2D crease image, the crease edge set, the geometric constraints, and the crease type. The 2D crease image is then drawn. The vertex coordinates are stored as node features, and the geometric constraints and crease type are stored as edge features in the graph topology structure. The crease edge set is used to distinguish the types of edge features. This contributes to improving the feature extraction capability of the model and achieving more accurate prediction results.

[0039] In step S3, a graph convolutional neural network is used to perform feature extraction on the graph topology structure and the two-dimensional fold line image to obtain first geometric features. The initial fold line image and the first geometric features are sent to a graph attention network for feature learning to obtain second geometric features and image features. That is, the graph attention network performs global weight calculation and feature learning on the node features and edge features in the initial fold line image and the first geometric features to obtain second geometric features and image features.

[0040] In step S3-1, before performing origami prediction using the graph convolutional neural network and the graph attention network, it is necessary to train the graph convolutional neural network and the graph attention network.

[0041] In step S3-1-1, in order to ensure the accuracy of model training, an augmentation operation is performed on the training data to increase the amount of training data.

[0042] In step S3-1-2, a fusion loss function is defined according to the prediction task, and the prediction result of the fused features is compared with the ground truth label to calculate a loss value. Then, the loss value is backpropagated to the graph convolutional neural network and the graph attention network based on the backpropagation algorithm, and the parameters in the graph convolutional neural network and the graph attention network are iteratively updated based on the gradient descent algorithm.

[0043] In step S3-2, origami prediction is performed using a trained graph convolutional neural network and a graph attention network, which can be achieved by the following steps:

[0044] In step S3-2-1, feature extraction is performed on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network, which includes the steps of setting a convolutional stride, generating a subgraph using the graph convolutional neural network with the stride as a distance from the current node of the graph topology structure, transmitting information to the node features and edge features of the subgraph, and aggregating the feature information represented by the two-dimensional crease line image to obtain a first geometric feature, i.e., setting the stride of the convolutional layer in the graph convolutional neural network as

number

number

[0045] In step S3-2-2, the initial fold line image and the first geometric features (node ​​features and edge features after information aggregation) are sent to the self-attention layer of the graph attention network. As shown in Figure 5, the graph attention network (abbreviated as GAT) globally finds the node with the highest correlation score for each node i, and combines the features of the node to obtain the second geometric features and image features.

[0046] During computation, GAT uses a shared learnable parameter matrix

number

number

number

number

number

number

number

number

number

number

number

number

[0047] In step S4, feature fusion is performed on the second geometric features and image features, and the fused features are analyzed using a multi-layer perceptron to output the origami prediction result.

[0048] In step S4-1, feature aggregation is performed using the attention coefficient calculated in step S3-2, and a new feature representation for node i is generated.

number

number

number

[0049] Furthermore, the pooling layer is part of the graph attention network and adopts self-attention graph pooling. This layer first processes the convolutional layer and activation function in the graph convolutional neural network to obtain the self-attention score of each node, then sorts the self-attention scores and retains the n highest-scoring nodes in front to form a mask. Finally, the mask retains the features of the specified node and the topology information between these nodes.

[0050] In step S4-2, feature fusion is performed on the second geometric feature and the image feature, and the image feature vector corresponding to the image feature is defined as I, and the geometric feature vector corresponding to the second geometric feature is defined as R.

number

number

[0051] In feature transformation, the dimension transformation is performed by fully connected layer mapping, so that the second geometric feature and image feature have more appropriate dimension and more accurate representativeness. The calculation formula of the feature after mapping is:

number

number

number

number

number

number

number

number

number

number

[0052] In step S4-3, a multilayer perceptron is used to analyze the fused features and output the origami prediction results.

[0053] In step S4-3-1, an attention matrix is ​​introduced to dynamically determine the importance weights in the fusion process of the image features and the second geometric features. Specifically, the attention module is A, and the attention module A is

number

number

number

number

number

number

number

number

[0054] Furthermore, attention module A is a simple multi-layer perceptron (MLP), which includes two input layers (for receiving image features and second geometric features, respectively), one hidden layer, and two output layers (for outputting attention scores of image features and second geometric features, respectively). The hidden layer performs nonlinear transformation using the function tanh, and the output layer performs normalization using the softmax function.

[0055] In step S4-3-2, the image feature and the second geometric feature are weighted fused according to the calculated attention score to obtain a fusion feature F, that is, the calculation formula is:

number

number

number

number

number

number

number

number

[0056] In step S4-3-3, the fused features are finally transformed through the output layer to adapt to the demands of the subsequent prediction task. The output layer can be a simple fully connected layer that maps the fused features to the target dimension. Specifically, as shown in Figure 6, the weight matrix of the output layer is

number

number

number

number

number

[0057] This embodiment discloses an origami prediction system based on multimodal graph learning.

[0058] a data acquisition module configured to acquire folding design parameters and initial fold line images; an object generation module configured to generate a graph topology structure and a two-dimensional foldline image according to the folding design parameters; a feature extraction module configured to perform feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features, and to transmit the initial crease line image and the first geometric features to a graph attention network for feature learning to obtain second geometric features and image features; an origami prediction module configured to perform feature fusion on the second geometric features and image features, analyze the fused features using a multi-layer perceptron, and output origami prediction results. Example 3

[0059] The purpose of this embodiment is to provide a computer-readable storage medium.

[0060] The computer-readable storage medium stores a computer program, which, when executed by a processor, realizes the steps in the origami prediction method based on multimodal graph learning described in Example 1 of the present disclosure. Example 4

[0061] The purpose of this embodiment is to provide an electronic device.

[0062] The electronic device comprises a memory, a processor, and a program stored in the memory and executable by the processor, and when the program is executed by the processor, steps in the origami prediction method based on multimodal graph learning described in Example 1 of the present disclosure are realized.

[0063] The steps involving the apparatuses in the above Examples 2, 3 and 4 correspond to those in the method Example 1, and for specific embodiments, please refer to the relevant description in Example 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media containing one or more sets of commands, and should also be understood to include any medium that can store, encode or carry the sets of commands to be executed by a processor, causing the processor to perform any of the methods in the present invention.

[0064] As will be appreciated by those skilled in the art, each module or step of the present invention described above can be implemented by a general-purpose computer device, and optionally, they can be implemented by a program code executable by a computing device, stored in a storage device and executed by the computing device, or they can be implemented as individual integrated circuit modules, or multiple modules or steps can be implemented by implementing them as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0065] Although specific embodiments of the present invention have been described above with reference to the drawings, they do not limit the scope of protection of the present invention, and as will be understood by those skilled in the art, various modifications and variations made by those skilled in the art based on the technical means of the present invention without requiring creative efforts are still included within the scope of protection of the present invention.

Claims

1. obtaining folding design parameters and an initial fold line image; generating a graph topology structure and a two-dimensional fold line image according to the folding design parameters; performing feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features; transmitting the initial crease line image and the first geometric features to a graph attention network to perform feature learning and obtain second geometric features and image features; and performing feature fusion on the second geometric features and image features, analyzing the fused features using a multi-layer perceptron, and outputting an origami prediction result.

2. The origami prediction method based on multimodal graph learning of claim 1, wherein the folding design parameters include vertex coordinates, geometric constraints, fold line edge sets, and fold line types.

3. The origami prediction method based on multimodal graph learning described in claim 2, characterized in that vertex coordinates are node features, and geometric constraints and fold line types are edge features.

4. The origami prediction method based on multimodal graph learning of claim 1, characterized in that performing feature extraction on the graph topology structure and the two-dimensional fold line image using a graph convolutional neural network includes setting a convolutional stride, using the graph convolutional neural network to generate a subgraph with the stride as a distance from the current node of the graph topology structure, transmitting information on the node features and edge features of the subgraph, and aggregating feature information represented in the two-dimensional fold line image to obtain a first geometric feature.

5. The origami prediction method based on multimodal graph learning described in claim 1, characterized in that the graph attention network performs global weight calculation and feature learning on node features and edge features in the initial fold line image and first geometric features to obtain second geometric features and image features.

6. The origami prediction method based on multimodal graph learning according to claim 1, characterized in that before performing origami prediction using the graph convolutional neural network and the graph attention network, it is necessary to train the graph convolutional neural network and the graph attention network, and in order to ensure the accuracy of the model training, an augmentation operation is performed on the training data to increase the amount of training data.

7. 7. The origami prediction method based on multimodal graph learning of claim 6, wherein training the graph convolutional neural network and the graph attention network includes: defining a fusion loss function according to a prediction task; comparing the prediction result of the fused features with a ground truth label to calculate a loss value; then backpropagating the loss value to the graph convolutional neural network and the graph attention network based on a backpropagation algorithm; and iteratively updating parameters in the graph convolutional neural network and the graph attention network based on a gradient descent algorithm.

8. a data acquisition module configured to acquire folding design parameters and initial fold line images; an object generation module configured to generate a graph topology structure and a two-dimensional foldline image according to the folding design parameters; a feature extraction module configured to perform feature extraction on the graph topology structure and the two-dimensional crease line image using a graph convolutional neural network to obtain first geometric features, and to transmit the initial crease line image and the first geometric features to a graph attention network for feature learning to obtain second geometric features and image features; an origami prediction module configured to perform feature fusion on the second geometric features and image features, analyze the fused features using a multilayer perceptron, and output origami prediction results.

9. A computer-readable storage medium having a program stored thereon, the computer-readable storage medium being characterized in that, when the program is executed by a processor, steps in the origami prediction method based on multimodal graph learning according to any one of claims 1 to 7 are realized.

10. 8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable by the processor, wherein, when the program is executed by the processor, steps in the origami prediction method based on multimodal graph learning according to any one of claims 1 to 7 are realized.

Images