Deep Learning-based Reverse Deduction and Dynamic Reproduction Method of Embroidery Stitches

Abstract

The present invention discloses a method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning, belonging to the technical field of embroidery stitch recognition. The method includes: collecting stitch data; using a point cloud registration algorithm to fuse three-dimensional point cloud data collected from multiple perspectives to construct a multi-modal data set; constructing a dynamic graph spatio-temporal interaction network, and extracting feature vectors in the multi-modal data based on the dynamic graph spatio-temporal interaction network; constructing a stitch inverse deduction model to obtain stitch spatial layout instructions, dynamic process parameters, and an optimized stitch sequence; constructing a dynamic reproduction model to obtain an actual embroidery action sequence. The present invention can automatically learn the deep features of the embroidery method in the embroidery image, and has significantly improved the accuracy of stitch inverse deduction and the authenticity of dynamic reproduction; it can achieve accurate inverse deduction of various embroidery methods under complex textures, different lighting conditions, and diverse embroidery angles.

Description

Technical Field

[0001] The present invention relates to a method for reproducing embroidery stitches, specifically a method for reverse deduction and dynamic reproduction of embroidery stitches based on deep learning, belonging to the technical field of embroidery stitch recognition. Background Art

[0002] The method for reverse deduction and dynamic reproduction of embroidery based on machine learning focuses on the reverse analysis of embroidery techniques and the restoration of dynamic processes, breaking through the technical bottlenecks of traditional manual feature analysis in the logical restoration of three-dimensional stitch patterns and the modeling of dynamic embroidery timing.

[0003] Traditional reverse methods rely on manual experience to visually infer the stitch direction and the relationship between embroidery layers of embroidery finished products. Generally, a basic camera is used to capture the stitch execution process and 3D modeling technology is used to achieve virtual restoration. It is difficult to accurately analyze the stitch trajectories, detailed features, and stitch logics of various embroidery pictures, and the reverse deduction process of each picture requires labor costs and 3D rendering costs.

[0004] In the research of the method for reverse deduction and dynamic reproduction of embroidery stitches, traditional simple feature extraction and matching algorithms based on image processing often face problems such as inaccurate reverse deduction of stitches, distorted dynamic reproduction, and sensitivity to noise interference in the image, and redundant feature extraction due to the diversity of stitch textures, rich changes in thread colors, and inconsistencies in embroidery angles. Summary of the Invention

[0005] Object of the Invention: Aiming at the above problems, the object of the present invention is to provide a method for reverse deduction and dynamic reproduction of embroidery stitches based on deep learning to achieve high-precision intelligent restoration of the embroidery process from static finished products to dynamic embroidery processes.

[0006] Technical Solution: The method for reverse deduction and dynamic reproduction of embroidery stitches based on deep learning of the present invention includes the following steps:

[0007] Step 1, collecting stitch data, including:

[0008] Collecting three-dimensional point cloud data of the embroidery finished product;

[0009] Collecting spectral data and RGB images of the embroidery finished product;

[0010] Collecting the microsecond-level dynamic details of the movement of the embroidery needle during the embroidery process;

[0011] Step 2: Use the point cloud registration algorithm to fuse the 3D point cloud data collected from multiple perspectives. Align the fused 3D point cloud data, spectral data, and RGB images in the coordinate system through spatial registration, extract geometric features, material features, and texture features respectively, and use the attention-weighted feature-level fusion strategy for fusion to construct a multi-modal dataset;

[0012] Step 3: Construct a dynamic graph spatio-temporal interaction network. Extract the feature vectors from the multi-modal data based on the dynamic graph spatio-temporal interaction network, model the embroidery stitches as a dynamic graph structure in combination with the feature vectors, and update the edge weights in real time according to the spatio-temporal continuity of the stitch movement trajectory;

[0013] Step 4: Construct a stitch reverse inference model based on a physical simulation engine and a generative adversarial network. Input the feature vectors and embroidery process constraint parameters in Step 3 into the stitch reverse inference model to obtain the stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequence;

[0014] Step 5: Construct a dynamic reproduction model. Input the stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequence into the dynamic reproduction model to obtain the actual embroidery action sequence.

[0015] Furthermore, the dynamic graph spatio-temporal interaction network includes a dynamic graph neural network and a spatio-temporal graph neural network.

[0016] Furthermore, the dynamic graph structure includes multiple nodes and the edges connecting the nodes. Among them, the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes.

[0017] Furthermore, the process of real-time updating the edge weights according to the spatio-temporal continuity of the stitch movement trajectory in Step 3 includes:

[0018] When new stitch data is input, calculate the spatial distance and time interval between the new stitch and the existing stitches, and dynamically adjust the edge weights according to the calculation results.

[0019] Furthermore, the construction of the stitch reverse inference model based on a physical simulation engine and a generative adversarial network in Step 4 includes:

[0020] Construct a dual-path generator, including a structure generation path and a physical parameter generation path. The generator converts the feature vectors into stitch spatial layout instructions through the structure generation path, and parallelly predicts the dynamic process parameters using the physical parameter generation path; the dynamic process parameters include thread tension and thread entry angle;

[0021] Construct a differentiable physical engine. Input the dynamic process parameters into the differentiable physical engine for forward simulation and iterative optimization until the thread tension satisfies the physical constraints and stop the iteration to obtain the optimized stitch sequence;

[0022] Construct a global discriminator and a local discriminator. Use the global discriminator to evaluate the process consistency of the stitch sequence, and use the local discriminator to evaluate the rationality of the single stitch action.

[0023] Furthermore, the dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-real transfer learning training module, a real-time adaptive module, and an embroidery thread material recognition and adaptation module.

[0024] Furthermore, the geometric features include the three-dimensional coordinates of the stitches, the curvature of the embroidery thread, the stacked thickness of the embroidery layers, the stitch spacing, and the stitch angle;

[0025] The material features include the reflectivity of the embroidery thread material, the surface friction coefficient, and the material type;

[0026] The texture features include the texture direction, texture density, texture contrast, and texture roughness.

[0027] Furthermore, the feature vector in step 3 includes geometric features, material features, and spatio-temporal encoding.

[0028] Beneficial effects: Compared with the prior art, the significant advantages of the present invention are:

[0029] By constructing a stitch reverse inference model, the present invention can automatically learn the deep features of the embroidery method in the embroidery image, and has significantly improved the accuracy of stitch reverse inference and the authenticity of dynamic reproduction;

[0030] The present invention has a wider applicability. It can achieve accurate reverse inference of various embroidery methods such as even stitches, interlocking stitches, and joining stitches under complex textures, different lighting conditions, and diverse embroidery angles, and can accurately and effectively perform dynamic reproduction. For embroidery images under different colors, backgrounds, and lighting conditions, it can stably output the stitch steps and dynamic demonstrations that conform to the actual embroidery process, providing strong technical support for the inheritance and innovation of embroidery culture, overcoming many limitations of traditional methods in practical applications, and having high practical value and promotion prospects. Description of the Drawings

[0031] Figure 1 is a flowchart of the embroidery stitch reverse inference and dynamic reproduction method based on deep learning;

[0032] Figure 2 is a structural diagram of the dynamic graph spatio-temporal interaction network;

[0033] Figure 3 is a structural schematic diagram of the stitch reverse inference model;

[0034] Figure 4 is a structural schematic diagram of the dynamic reproduction model;

[0035] Figure 5 It is the recognition effect diagram of random stitch method. Specific implementation manner

[0036] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0037] In this embodiment, the inverse deduction and dynamic reproduction method of embroidery stitch based on deep learning has a flow chart as Figure 1 shown, and this method includes the following steps:

[0038] Step 1, collect stitch data, including:

[0039] Collect the three-dimensional point cloud data of the embroidered finished product;

[0040] Collect the spectral data and RGB images of the embroidered finished product;

[0041] Collect the microsecond-level dynamic details of the movement of the embroidery needle during the embroidery process.

[0042] In one example, the three-dimensional point cloud data of the embroidered finished product is collected from multiple perspectives by a three-dimensional laser scanner. This three-dimensional point cloud data includes the three-dimensional coordinates of the stitches, the curvature of the embroidery thread, and the thickness of the embroidered layer stack. The spectral data of the embroidered finished product is collected by a hyperspectral microscopy imaging system. This spectral data includes the reflectivity of the embroidery thread material, the surface friction coefficient, and the material type. Use a camera to collect the RGB image of the embroidered finished product; introduce an event camera to capture the microsecond-level dynamic details of the high-speed movement of the embroidery needle during the embroidery process on the basis of the traditional RGB image, including the jitter trajectory of the embroidery needle, etc., to make up for the motion blur defect of the conventional camera and further enrich the dimension and accuracy of data collection.

[0043] Step 2, use the point cloud registration algorithm to fuse the three-dimensional point cloud data collected from multiple perspectives, align the fused three-dimensional point cloud data, spectral data and RGB image in the coordinate system through spatial registration, extract geometric features, material features and texture features respectively, and adopt an attention-weighted feature-level fusion strategy for fusion to construct a multi-modal data set.

[0044] Use the point cloud registration algorithm to fuse the three-dimensional point cloud data collected from multiple perspectives to eliminate noise and improve the analysis accuracy of tiny stitch details. Align the fused three-dimensional point cloud data, spectral data and RGB image in the coordinate system through spatial registration, extract geometric features, material features and texture features respectively, where the geometric features come from the three-dimensional point cloud data, the material features come from the spectral data, and the texture features come from the RGB image. The geometric features, material features and texture features are fused using an attention-weighted feature set fusion strategy to highlight the information that is more critical for the analysis of embroidery techniques, and a multi-modal data set is obtained.

[0045] This multimodal dataset usually exists in the form of multi-dimensional structured data or enhanced point clouds. Among them, the multi-dimensional structured data integrates the feature information of different modalities into a high-dimensional feature vector. Each vector contains geometric features, material features, and texture features. These feature vectors can be organized into a matrix or tensor, facilitating subsequent batch processing and feature learning by deep learning models.

[0046] The enhanced point cloud form means expanding the traditional three-dimensional point cloud data. Each point not only contains spatial coordinate information but also additional rich attributes extracted from spectral data and RGB images, such as material reflectivity, color information, and local texture features. This enhanced point cloud can more comprehensively describe the physical and visual characteristics of the embroidery work, providing more abundant details for the analysis and dynamic reproduction of complex stitches.

[0047] Furthermore, the geometric features include the three-dimensional coordinates of the stitches, the curvature of the embroidery thread, the thickness of the stacked embroidery layers, the stitch spacing, and the stitch angle;

[0048] The material features include the reflectivity of the embroidery thread material, the surface friction coefficient, and the material type;

[0049] The texture features include the texture direction, texture density, texture contrast, and texture roughness.

[0050] Step 3: Construct a dynamic graph spatio-temporal interaction network, extract the feature vectors from the multimodal data based on the dynamic graph spatio-temporal interaction network, model the embroidery stitches as a dynamic graph structure in combination with the feature vectors, and update the edge weights in real time according to the spatio-temporal continuity of the stitch movement trajectory.

[0051] Combined with Figure 2 , furthermore, the dynamic graph spatio-temporal interaction network includes a dynamic graph neural network and a spatio-temporal graph neural network.

[0052] Construct a dynamic graph spatio-temporal interaction network by combining a dynamic graph neural network (DGNN) and a spatio-temporal graph neural network (STGNN). Input the data in the multimodal dataset into the dynamic graph spatio-temporal interaction network for feature extraction. The extracted feature vectors include geometric features, material features, and spatio-temporal encoding, etc. Model the embroidery stitches as a dynamic graph structure in combination with the extracted feature vectors.

[0053] Among them, the spatio-temporal encoding is a comprehensive encoding for time and space features, used to capture the dynamic temporal characteristics such as the sequence, speed, and acceleration of the stitches changing over time during the embroidery process. At the same time, combined with the spatial distribution and geometric relationship of the stitches, it comprehensively reflects the spatio-temporal continuity of the embroidery actions.

[0054] In the dynamic graph spatio-temporal interaction network, the dynamic graph neural network is responsible for constructing the dynamic graph structure. Taking embroidery stitches as nodes and the relationships between nodes as edge weights, it updates the edge weights in real time according to the spatio-temporal continuity of the stitch movement trajectories to reflect the dynamic relationship changes between stitches. The spatio-temporal graph neural network focuses on capturing the spatio-temporal dependence relationships of stitch movements, analyzing the mutual influences of stitches in time and space to understand the movement patterns and spatio-temporal characteristics of stitches during the embroidery process. The process of fusing feature information adopts an attention-weighted feature-level fusion strategy, integrating geometric features, material features, texture features, and spatio-temporal encodings to form a comprehensive feature vector. These feature vectors not only contain the geometric, material, and texture information of embroidery stitches but also the dynamic characteristics of stitches changing over time, such as sequence, speed, and acceleration. Finally, the feature vectors output by the dynamic graph spatio-temporal interaction network are used for stitch reverse inference and dynamic reproduction, providing a basis for the generation of subsequent stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequences. In this way, the dynamic graph spatio-temporal interaction network can not only capture the static features of embroidery stitches but also understand their dynamic characteristics, thus providing a comprehensive and accurate feature representation for the reverse inference and dynamic reproduction of embroidery processes.

[0055] The dynamic graph neural network and the spatio-temporal graph neural network are integrated in the dynamic graph spatio-temporal interaction network and jointly act on the feature extraction and modeling of multi-modal data. The dynamic graph neural network updates the edge weights, and the spatio-temporal graph neural network captures spatio-temporal dependencies. The two cooperate with each other to enhance the model's understanding and analysis ability of the dynamic changes of embroidery stitches, and can provide a more accurate feature representation for subsequent stitch reverse inference and dynamic reproduction.

[0056] Furthermore, the dynamic graph structure includes multiple nodes and the edges connecting two nodes. Among them, the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes. The connection strength or similarity between the two connected nodes is reflected through the edge weight. The role of updating the edge weight is to dynamically adjust the connection strength between nodes in the graph structure, so as to more accurately reflect the spatio-temporal relationship between embroidery stitches, including the relevance and sequence between stitches. The updating mechanism is crucial for improving the parsing accuracy and dynamic reproduction ability of the model. The relevance and sequence between stitches are important bases for constructing the stitch reverse inference model, helping the model to generate stitch spatial layout instructions and dynamic process parameters more accurately. At the same time, the edge weight information helps to construct a more realistic embroidery action sequence, enabling the dynamic reproduction model to generate action instructions that conform to the actual embroidery process according to the spatio-temporal relationship between stitches, thereby improving the authenticity and accuracy of the dynamic reproduction of the embroidery process.

[0057] Furthermore, the process of updating the edge weight in real time according to the spatio-temporal continuity of the stitch movement trajectory in step 3 includes:

[0058] When new stitch data is input, calculate the spatial distance and time interval between the new stitch and the existing stitches, and dynamically adjust the edge weights according to the calculation results.

[0059] Among them, the new stitch data refers to the stitch information collected in real time during embroidery, including three-dimensional point cloud data, spectral data, RGB images, and microsecond-level dynamic details of the movement of the embroidery needle. Input the new stitch data into the dynamic graph neural network to calculate the spatial distance and time interval between the new stitch and the existing stitches, so as to dynamically adjust the edge weights to reflect the spatio-temporal continuity between the stitches.

[0060] The update of the edge weights is based on the combined effect of the spatial distance decay factor and the time interval decay factor, and an exponential decay function is used to model the spatio-temporal correlation. The update formula is:

[0061] ,

[0062] In the formula, represents the edge weight between node i and node j. The larger the weight value, the stronger the correlation; represents the spatial distance between the new stitch and the existing stitches, in meters; represents the time interval between the new stitch and the existing stitches, in seconds; α, β, γ, δ represent learnable parameters or preset weight coefficients. In this example, set: , , , .

[0063] For example, if the calculated spatial distance is 5 cm and the time interval is 5 ms, the edge weight can be calculated as: which is 1.0917. If the preset edge weight range is [0,1], the result needs to be normalized. For example, use the Sigmoid function for normalization, and the final weight is: .

[0064] Step 4, construct a stitch reverse inference model based on a physical simulation engine and a generative adversarial network, and input the feature vector and embroidery process constraint parameters in Step 3 into the stitch reverse inference model to obtain the stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequence.

[0065] Combined with Figure 3 , further, the construction of the stitch reverse inference model based on the physical simulation engine and the generative adversarial network in Step 4 includes:

[0066] Construct a generator based on a dual-path, including a structure generation path and a physical parameter generation path. The generator converts the feature vector into stitch spatial layout instructions through the structure generation path, and uses the physical parameter generation path to predict dynamic process parameters in parallel; the dynamic process parameters include thread tension and thread entry angle, etc.

[0067] Construct a differentiable physics engine, input the dynamic process parameters into the differentiable physics engine for forward simulation and iterative optimization, and stop the iteration until the embroidery thread tension meets the physical constraints, so as to obtain the optimized stitch sequence;

[0068] Construct a discriminator module, including a global discriminator and a local discriminator, use the global discriminator to evaluate the process consistency of the stitch sequence, and use the local discriminator to evaluate the rationality of the single stitch action.

[0069] Among them, the embroidery process constraint parameters include the three-dimensional point cloud data, spectral data, RGB images, and dynamic detail parameters of the embroidery needle movement collected in the data acquisition stage, the geometric features, material features, texture features, and fusion parameters in the feature extraction and fusion stage, the dynamic graph structure and stitch movement trajectory parameters in the dynamic graph spatio-temporal interaction network construction stage, the feature vectors, embroidery process constraints, generator dual paths, and discriminator evaluation parameters in the stitch inverse inference model construction stage, the stitch space layout instructions, dynamic process, optimized stitch sequence, multi-objective reward function, real-time adaptive module, embroidery thread material recognition, and embroidery thread tension closed-loop control parameters in the dynamic reproduction model construction stage.

[0070] The generator includes a structure generation path and a physical parameter generation path. The structure generation path receives the feature vector through the input layer, performs feature encoding using a multi-layer perceptron or a convolutional neural network, then captures the global dependencies by combining the self-attention mechanism or the Transformer structure, and finally generates the stitch space layout instructions through the decoder. The physical parameter generation path also receives the feature vector, extracts features through the encoder, and uses a fully connected layer to predict the dynamic process parameters, such as the embroidery thread tension and the thread entry angle.

[0071] The dual-path structure of the generator refers to some common architectures in the generative adversarial network (GAN) in its design, including the encoder and decoder structures of the deep convolutional generative adversarial network, and the self-attention mechanism of the Transformer-based GAN model, which is customized for the embroidery stitch inverse inference task to achieve the joint optimization of the stitch space layout and physical parameters.

[0072] The differentiable physics engine can use an embroidery plugin such as PyBullet. The differentiable physics engine structure includes a parameter input and parsing module, a physical simulation module, an optimization and iteration module, and an output module.

[0073] The parameter input and parsing module receives the dynamic process parameters, such as the embroidery thread tension and the stitch speed, etc., and receives the embroidery scene configuration parameters, such as the embroidery cloth material and the embroidery thread specification, etc., and parses and maps these parameters to the corresponding variables in the physical simulation model.

[0074] The physical simulation module includes the embroidery thread dynamics model, the embroidery cloth mechanical model and the stitch interaction model. It uses numerical integration methods and discretization technology to solve the differential equations in the physical simulation model, simulates the movement, deformation and interaction of the embroidery thread with the embroidery cloth during the embroidery process, and outputs simulation results, such as the shape of the embroidery thread on the embroidery cloth, and the force distribution.

[0075] The input of the embroidery thread dynamics model includes the initial position and speed of the embroidery thread, the physical parameters of the embroidery thread, and the external force information. The initial position and speed determine the geometric state and movement trend of the embroidery thread at the beginning of the simulation, which are usually expressed in the form of coordinates and vectors; the physical parameters of the embroidery thread, such as mass, density, elastic modulus, etc., reflect the material characteristics and mechanical properties of the embroidery thread and affect its response when subjected to force; the external force information includes the traction applied by the embroidery needle, the friction and normal force of the embroidery cloth on the embroidery thread, etc. The magnitude and direction of these external forces determine the movement and deformation of the embroidery thread. The output of the embroidery thread dynamics model is the position and speed of the embroidery thread at different times and the tension distribution of the embroidery thread. The position and speed give the movement trajectory of the embroidery thread in the embroidery process in the form of a time series, which is expressed as a set of coordinate points and the corresponding velocity vector, which can be used for subsequent analysis of the embroidery thread's alignment; the tension distribution is the tension of each point along the length of the embroidery thread, reflecting the tightness of the embroidery thread when subjected to force, which is of great significance for judging the embroidery quality, such as whether the embroidery thread is too tight or too loose.

[0076] The embroidery thread dynamics model uses the existing LSTM long short-term memory network and Transformer network. LSTM is a neural network suitable for processing time series data. It can be used to simulate the dynamic behavior of embroidery thread. Its input is the initial state of the embroidery thread and external force information, and its output is the position, speed and tension of the embroidery thread at different times. Transform is a neural network architecture based on the self-attention mechanism, which is suitable for processing complex sequence data. Its input is the initial state of the embroidery thread and external force, and its output is the movement trajectory and tension distribution of the embroidery thread. The advantage of using these two networks is that they can capture global dependencies and are suitable for processing complex dynamic systems. SoftBodyDynamics is used as a plug-in, which is a tool specifically used for soft body dynamics simulation. It can be used to simulate the movement and deformation of embroidery thread. It needs to be used in conjunction with a physical engine such as PyBullet.

[0077] The input of the embroidery fabric mechanical model includes the initial geometry and boundary conditions of the embroidery fabric, the material properties of the embroidery fabric, and the external forces acting on the embroidery fabric. Among them, the initial geometry and boundary conditions define the size, shape, and fixing method of the embroidery fabric at the beginning of the simulation, such as the length, width, thickness of the embroidery fabric, and whether the four peripheral boundaries are fixed; the material properties of the embroidery fabric, such as Young's modulus, Poisson's ratio, yield strength, etc., determine the elastic deformation and plastic deformation characteristics of the embroidery fabric when stressed; the external forces acting on the embroidery fabric include the pulling force of the embroidery thread on the embroidery fabric, the pressure generated when the stitch penetrates the embroidery fabric, etc. The distribution position and magnitude of the external forces affect the deformation mode of the embroidery fabric. The output of the embroidery fabric mechanical model is the deformation of the embroidery fabric and the stress distribution inside the embroidery fabric. The deformation is manifested as the displacement field and strain field of the embroidery fabric at different times. The displacement field describes the position change of each point on the embroidery fabric, and the strain field reflects the microscopic deformation degree of the embroidery fabric, which can be used to evaluate the change in the shape of the embroidery fabric caused by embroidery; the stress distribution reflects the magnitude and direction of the stress received by each point inside the embroidery fabric, helps to understand the stress state of the embroidery fabric during the embroidery process, and has reference value for predicting possible damage or relaxation of the embroidery fabric.

[0078] In the embroidery thread mechanical model, the finite element method (FEM) is used to handle the complex geometry and boundary conditions formed by the embroidery thread. The principle is to divide the embroidery fabric into many small elements, such as triangles or quadrilaterals, and approximately solve the mechanical equations on each element. This method is very suitable for simulating the complex deformation and stress distribution of the embroidery fabric during the embroidery process and can handle complex geometry and boundary conditions with high precision. The embroidery thread mechanical model can use graph neural networks or convolutional neural networks. Among them, graph neural networks are particularly suitable for processing data with spatial relationships, such as the grid structure of the embroidery fabric. Its input is the initial geometry, boundary conditions, material properties, etc. of the embroidery fabric, and the output is the deformation and stress distribution of the embroidery fabric. The advantage of this network is that it can capture spatial relationships and can handle complex grid structures. Convolutional neural networks can be used to process image data. The deformation and stress distribution of the embroidery fabric can be regarded as image data for processing. Its input and output are the same as those of the graph neural network. The advantage is that it can capture local features and is suitable for processing image data. For the plug-in, ABAQUS can be selected. This is a widely used finite element analysis software that can be used to simulate the mechanical behavior of the embroidery fabric. Its advantages are high precision and the ability to provide rich material models and boundary conditions.

[0079] The input of the stitch interaction model includes the geometric parameters of the stitch, the motion state of the stitch, and the state information of the embroidery fabric and thread. Among them, the geometric parameters of the stitch, such as the diameter and length of the needle and the thickness of the thread, affect the penetration depth of the stitch in the fabric and the contact between the threads; the motion state of the stitch includes the insertion speed, withdrawal speed, insertion angle, etc. of the stitch, which determine the interaction mode and time between the stitch and the fabric; the state information of the fabric and thread, such as the current deformation of the fabric and the distribution state of the thread on the fabric, provides the environmental background for the stitch interaction model, enabling the model to consider the mutual influence between the stitch and the existing embroidery parts. The output of the stitch interaction model is the mutual force between the stitch and the fabric, as well as the local deformation and force of the thread under the action of the stitch. The mutual force includes the magnitude and direction of the frictional force, normal force, etc., reflecting the resistance and support force received by the stitch during embroidery, which helps to analyze the difficulty of embroidery and the force on the fabric; the local deformation and force of the thread under the action of the stitch describe the small deformation and force change of the thread near the stitch due to the movement of the stitch, which is of great significance for evaluating the wear of the thread and the fineness of the embroidery pattern.

[0080] The stitch interaction model can adopt a multi-layer perceptron (MLP) and Transformer. The MLP can be used to learn the complex non-linear relationship between the stitch and the fabric. Its input is the geometric parameters, motion state of the stitch, and state information of the fabric, and the output is the mutual force between the stitch and the fabric, the local deformation and force of the thread. The advantage of this network is that it is simple and easy to implement and is suitable for dealing with complex non-linear relationships. The Transformer network can be used to process sequence data in this model and is suitable for simulating the dynamic interaction between the stitch and the fabric. Its input is the motion trajectory of the stitch and time series data, and the output is the mutual force between the stitch and the fabric, the local deformation and force of the thread. The advantage of this network is that it can capture long-range dependencies in time series. For the plug-in, the PyTorch deep learning framework can be used, which can be used to build a neural network model to learn the complex interaction relationship between the stitch and the fabric. Or use Plotly, which is a Python library for data visualization that supports interactive charts and can be used to visualize the interaction results between the stitch and the fabric.

[0081] The optimization and iteration module defines the loss function to measure the difference between the simulation results and the physical constraints. It adopts the gradient descent optimization algorithm and uses the automatic differentiation mechanism to calculate the gradient of the loss function with respect to the physical parameters to guide the adjustment direction of the physical parameters. Through multiple iterations of optimization until the physical parameters such as the thread tension meet the physical constraints, such as the thread tension reaches a reasonable range, etc. Among them, the physical parameters refer to those physical properties and mechanical characteristics involved in the embroidery process and that can affect the embroidery effect, including thread tension, thread deformation coefficient, fabric elastic modulus, stitch angle, friction coefficient, and thread density, etc.

[0082] In one example, the expression of the loss function is:

[0083] ,

[0084] wherein, represents the tension of the embroidery thread in the i-th simulation; represents the target embroidery thread tension; represents the tolerance range of the tension; N represents the number of simulation times or data points; represents the weight coefficient, which is used to balance the influence of different constraint terms; represents other physical constraint terms, such as the deformation coefficient, friction, etc.

[0085] The output module sorts out and formats the optimized physical parameters and simulation results, and outputs them to the subsequent formation sequence generation module through the data interface, providing accurate physical support for the inverse deduction and dynamic reproduction of the embroidery stitches. The entire differentiable physical engine combines physical simulation and machine learning technologies and is customized according to the specific requirements of the embroidery process. The formation sequence generation module is a part of the stitch inverse deduction model, which is implemented through the dual-path structure of the generator in the stitch inverse deduction model. It is responsible for generating stitch space layout instructions and dynamic process parameters according to the optimized physical parameters and simulation results, and outputting these instructions and parameters to the dynamic reproduction model to generate the actual embroidery action sequence.

[0086] Among them, the global discriminator is responsible for evaluating the overall process consistency of the stitch sequence. The input of the global discriminator is the feature vector of the entire stitch sequence, and these feature vectors cover the stitch space layout instructions, dynamic process parameters, and the optimized formation sequence. In the feature extraction layer, the global discriminator adopts a convolutional neural network (CNN) or a Transformer structure to extract the global features of the stitch sequence. The CNN can effectively capture the spatial features in the stitch sequence, while the Transformer is better at dealing with the long-distance dependencies in the sequence data. After feature extraction, the global discriminator further processes the extracted global features through a fully connected layer to generate an evaluation of the overall process consistency of the stitch sequence. The global discriminator will finally output a probability value, and the closer this value is to 1, the better the process consistency of the stitch sequence.

[0087] Among them, the local discriminator is mainly used to evaluate the rationality of a single stitch action. The input of the local discriminator is the feature vector of a single stitch, and these feature vectors include the geometric features, material features, dynamic process parameters, etc. of the stitch. In the feature extraction layer, the local discriminator adopts a multi-layer perceptron (MLP) or a convolutional neural network (CNN) structure to extract the local features of a single stitch. The MLP can perform non-linear transformation on the features of the stitch to capture the complex relationships between the features. After feature extraction, the local discriminator further processes the extracted local features through a fully connected layer to generate an evaluation of the rationality of a single stitch action. The local discriminator outputs a probability value, and the closer this value is to 1, the higher the rationality of the stitch action.

[0088] Step 5, construct a dynamic reproduction model, input the stitch space layout instruction, dynamic process parameters, and the optimized stitch sequence into the dynamic reproduction model to obtain the actual embroidery action sequence.

[0089] Combined with Figure 4 , where the dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-real transfer learning training module, a real-time adaptive module, and an embroidery thread material recognition and adaptation module. Each module in the dynamic reproduction model works together to complete the dynamic reproduction task of embroidery stitches.

[0090] The multi-objective reinforcement learning control module integrates virtual-real transfer learning and multi-objective optimization mechanisms. Through the neural differential equation modeling module, the stitch features are mapped into the differential equation of the embroidery needle movement in the continuous time domain, realizing the continuous modeling of the embroidery needle trajectory. The multi-objective reinforcement learning control module takes the stitch space layout instructions, dynamic process parameters, and the optimized stitch sequence as inputs. It encodes the stitch features through a Transformer-based policy network and decodes to generate multi-objective action policies. Its value network uses a multi-layer perceptron (MLP) to evaluate the comprehensive value of actions. Then, through the reinforcement learning (RL) control strategy, with the stitch space layout instructions, dynamic process parameters, and the optimized stitch sequence as inputs, the Transformer policy network uses its self-attention mechanism to deeply encode these stitch features, comprehensively capturing the long-range dependencies between features, and then decodes to generate multi-objective action policies. These policies cover key action decisions such as embroidery needle movement trajectory planning and embroidery thread tension control. The generated multi-objective action policies are passed to the value network using MLP. MLP performs non-linear transformations through multiple neurons, deeply mining the complex relationships between action features, and evaluates the comprehensive value of the action policies in multiple aspects such as embroidery accuracy, efficiency, and safety according to the multi-objective reward function including accuracy reward, efficiency reward, and safety reward. The RL control strategy optimizes the policy network in the virtual environment using the proximal policy optimization algorithm based on the evaluation results of the value network. If the comprehensive value evaluated by the value network is high, it indicates that the policy performs well in achieving multiple objectives, and the algorithm will strengthen this policy. Otherwise, it will adjust the parameters of the policy network to generate a better policy. After continuous iterative optimization, the RL control strategy can output virtual embroidery needle movement instructions and digital tension control parameters that better meet the actual embroidery requirements, effectively improving the accuracy and stability of the dynamic reproduction of embroidery stitches.

[0091] The multi-objective reward function includes accuracy reward, efficiency reward, and safety reward. The accuracy reward is the mean square error between the virtual embroidery needle trajectory and the theoretical path. The efficiency reward is the negative weighted sum of the simulation embroidery time and the computational resource consumption. The safety reward is the difference penalty between the virtual embroidery thread tension and the fracture threshold. The proximal policy optimization algorithm is used to collaboratively optimize the policy in the virtual environment, and finally output virtual embroidery needle movement instructions and digital tension control parameters.

[0092] In one example, the expression of the multi-objective reward function is:

[0093] ,

[0094] In the formula, R represents the comprehensive reward value; represents the weight coefficient, which is used to balance the influence of different reward terms; represents the accuracy reward, which is defined as the negative value of the mean square error between the virtual embroidery needle trajectory and the theoretical path, and the expression is:

[0095] ,

[0096] wherein, T is the length of the time series, represents the position vector of the predicted trajectory at time t, represents the position vector of the true trajectory at time t;

[0097] represents the efficiency reward, defined as the negative weighted sum of the simulation embroidery time and the computational resource consumption, and the formula is:

[0098] ,

[0099] wherein, w 1, w 2 is the weight coefficient, represents the simulation embroidery time, represents the computational resource consumption;

[0100] represents the safety reward, defined as the negative weighted sum of the simulation embroidery time and the computational resource consumption, and the formula is:

[0101] ,

[0102] wherein, is the virtual embroidery thread tension, that is, the magnitude of the tension borne by the simulated embroidery thread; is the fracture threshold, which refers to the maximum tension value that the embroidery thread can withstand. When the virtual embroidery thread tension exceeds the fracture threshold, there is a risk of the embroidery thread breaking.

[0103] When the reward value of the multi-objective reward function is in the ideal situation, that is, when the error is 0, the time is 0, and the tension is 0, the comprehensive reward value R is 0. At this time, the multi-objective reinforcement learning control module has no penalty and the reward value is maximized;

[0104] When the reward value is in the general situation, that is, when the error, time, and tension are within a reasonable range, the comprehensive reward value R is a relatively small negative number. At this time, the model performance is qualified, but there is room for optimization;

[0105] When the reward value is in the dangerous situation, that is, when the tension exceeds the limit and the error is large, the comprehensive reward value R is a relatively large negative number. At this time, the model generation result is not feasible and needs to be optimized again.

[0106] Starting from the spatio-temporal features of the needlework method, including geometry, material, and spatio-temporal encoding, the neural differential equation modeling module extracts spatio-temporal correlation features through a graph convolutional network (GCN), constructs a neural ordinary differential equation (Neural ODE) driven by an LSTM-Transformer hybrid network, and maps the spatio-temporal correlation features to the parameters of the virtual embroidery needle motion equation in the continuous time domain. The fourth-order Runge-Kutta method (RK4) with an adaptive step size is used for high-precision solution, supporting sub-millimeter trajectory interpolation such as generating trajectory points every 0.1 mm and microsecond-level dynamic response, ensuring the smoothness of the virtual embroidery needle motion in the simulation environment. When processing the fine flat stitch of Suzhou embroidery, for example, the neural differential equation modeling module captures the minute displacement changes of the embroidery needle, such as 0.05 mm-level jitter, through the differential equation, and adjusts the interpolation density in real time to output high-fidelity continuous motion parameters.

[0107] The expression of the neural ordinary differential equation is:

[0108] ,

[0109] In the formula, represents the hidden state vector, which contains the pose parameters of the embroidery needle, such as the position x(t), velocity v(t), acceleration a(t), and dynamic process parameters, such as the embroidery thread tension and the thread entry angle; represents the dynamic function parameterized by the LSTM-Transformer hybrid network, with the input being the hidden state h(t) and the spatio-temporal features ; represents the information extracted by the dynamic graph spatio-temporal interaction network, including geometry, material, and spatio-temporal encoding information.

[0110] The virtual-real transfer learning training module first constructs a million-level embroidery action chain in the physical simulation engine, pre-trains the control strategy through the proximal policy optimization algorithm, and then uses the real data collected by the actual device combined with meta-learning to quickly fine-tune the policy network. These real data can come from the records of the manual embroidery process or the operation data of other automated embroidery devices, such as the actual motion trajectory of the embroidery needle, the embroidery thread tension, the stitch angle, the embroidery needle speed, etc. Specifically, the virtual-real transfer learning training module is divided into two stages: simulation pre-training and virtual data fine-tuning. In the simulation pre-training stage, in the virtual embroidery physical engine built based on PyBullet, a million-level stitch interaction action chain is generated, and the general control strategy is pre-trained through the PPO algorithm. In the virtual data fine-tuning stage, the stitch motion data simulated by the virtual embroidery machine, such as the preset friction coefficient and the virtual motor response delay, are introduced, and the meta-learning framework MAML is used to quickly adjust the parameters of the policy network to achieve low-attenuation transfer from the simulation environment to the virtual reproduction scene on the network side. All training and optimization are based on digital parameters and do not require actual hardware support.

[0111] The real-time adaptive module uses a high-precision laser rangefinder to provide real-time feedback on the embroidery needle position deviation, dynamically adjusts the temporal attention weight of the LSTM-Transformer predictor, and realizes online error compensation. Specifically, the real-time adaptive module drives the adaptive mechanism through virtual sensor data, such as the embroidery needle position deviation and embroidery thread tension error in the simulation environment, and then calculates the virtual embroidery needle position deviation and virtual embroidery thread tension error. According to the calculation results, the adaptive controller uses the LSTM-Transformer predictor to dynamically adjust the temporal attention weight. For example, when the simulation trajectory offset exceeds 0.1 mm, the real-time adaptive module generates a virtual trajectory correction instruction within 5 milliseconds, and realizes error compensation by adjusting the neural differential equation parameters to ensure the sub-millimeter accuracy of the network-side simulation.

[0112] The embroidery thread material recognition and adaptation module identifies the embroidery thread material based on multispectral imaging data, automatically switches the mechanical model parameters and links the embroidery thread tension closed-loop control system. Specifically, based on multispectral imaging data such as reflectivity, texture, and spectral features, the ResNet-50 backbone network is used to extract deep features and classify material types such as silk, cotton thread, or metal thread through a fully connected layer. The elastic modulus, friction coefficient, and fracture threshold of the corresponding material are then dynamically loaded from the preset mechanical parameter library to the virtual differentiable physics engine. For example, when a metal thread is identified, the virtual thread lifting speed is automatically limited to 0.5 m / s in the simulation environment, and the process consistency is maintained by adjusting the virtual embroidery thread tension parameters. All control logic runs completely in a closed loop in the digital twin environment without the need to link physical equipment.

[0113] The preliminary results of the dynamic reproduction authenticity are evaluated based on two criteria, including trajectory similarity (TS) and deformation error (DE). Trajectory similarity measures the similarity between the generated motion trajectory and the real trajectory, and the calculation formula is:

[0114] ,

[0115] In the formula, T Indicates the length of the time series; and They are the predicted trajectory and the true trajectory in time T The trajectory similarity measures the similarity of trajectories by calculating the cosine similarity between the direction vectors of the predicted trajectory and the true trajectory. The closer the cosine similarity value is to 1, the more similar the trajectories are.

[0116] The deformation error is used to measure the difference between the generated deformation and the real deformation, and the calculation formula is:

[0117] ,

[0118] In the formula, is the total number of deformation points; and are the values of the predicted deformation and the real deformation at the th point respectively. This deformation error calculates the average of the absolute differences between the predicted deformation and the real deformation, and is used to measure the accuracy of the deformation. The smaller the deformation error, the closer the predicted deformation is to the real deformation.

[0119] Figure 5 The figure shows the recognition effect diagram of the random stitch method, i.e., the picture of the grabbing stitch, after the multi-modal data is processed by the deep learning network described in the present invention. The recognition confidence level in the figure is 0.94, indicating that it can achieve accurate inverse deduction of various embroidery stitch methods such as the flat stitch, the couching stitch, and the joining stitch under complex textures, different lighting conditions, and diverse embroidery angles, and can accurately and effectively perform dynamic reproduction. For embroidery images under different colors, backgrounds, and lighting conditions, it can stably output the stitch method steps and dynamic demonstrations that conform to the actual embroidery process, providing strong technical support for the inheritance and innovation of embroidery culture, overcoming many limitations of traditional methods in practical applications, and having high practical value and promotion prospects.

Claims

1. A method for inverse inference and dynamic reproduction of embroidery stitches based on deep learning, characterized in that It includes the following steps: Step 1, collect pin data, including: Collect the three-dimensional point cloud data of the embroidered finished product; Collect the spectral data and RGB images of the embroidered finished product; Collect the microsecond-level dynamic details of the movement of the embroidery needle during the embroidery process; Step 2, use the point cloud registration algorithm to fuse the three-dimensional point cloud data collected from multiple perspectives, align the fused three-dimensional point cloud data, spectral data, and RGB images with the coordinate system through spatial registration, extract geometric features, material features, and texture features respectively, and use an attention-weighted feature-level fusion strategy for fusion to construct a multi-modal dataset; Step 3, construct a dynamic graph spatio-temporal interaction network, extract the feature vectors in the multi-modal data based on the dynamic graph spatio-temporal interaction network, model the embroidery stitches as a dynamic graph structure in combination with the feature vectors, and update the edge weights in real time according to the spatio-temporal continuity of the stitch movement trajectory; Step 4, construct a stitch reverse inference model based on a physical simulation engine and a generative adversarial network, input the feature vectors in Step 3 and the embroidery process constraint parameters into the stitch reverse inference model to obtain the stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequences; Step 5, construct a dynamic reproduction model, input the stitch spatial layout instructions, dynamic process parameters, and optimized stitch sequences into the dynamic reproduction model to obtain the actual embroidery action sequence; The construction of the stitch reverse inference model based on the physical simulation engine and the generative adversarial network in Step 4 includes: Construct a generator based on a dual-path, including a structure generation path and a physical parameter generation path. The generator converts the feature vectors into stitch spatial layout instructions through the structure generation path, and uses the physical parameter generation path to parallelly predict the dynamic process parameters; the dynamic process parameters include thread tension and thread entry angle; Construct a differentiable physical engine, input the dynamic process parameters into the differentiable physical engine for forward simulation and iterative optimization, and stop the iteration until the thread tension meets the physical constraints to obtain the optimized stitch sequence; Construct a global discriminator and a local discriminator.

2. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to claim 1, wherein The dynamic graph spatio-temporal interaction network includes a dynamic graph neural network and a spatio-temporal graph neural network.

3. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to claim 2, wherein The dynamic graph structure includes multiple nodes and the edges connecting the nodes. Among them, the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes.

4. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to claim 3, wherein The process of real-time updating the edge weights according to the spatio-temporal continuity of the stitch movement trajectory in Step 3 includes: When new stitch data is input, calculate the spatial distance and time interval between the new stitch and the existing stitches, and dynamically adjust the edge weights according to the calculation results.

5. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to claim 4, characterized in that, The dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-real transfer learning training module, a real-time adaptive module, and a thread material recognition and adaptation module.

6. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to any one of claims 1 to 5, characterized in that, Geometric features include stitch three-dimensional coordinates, thread curvature, stitch layer stacking thickness, stitch spacing, and stitch angle; Material features include thread material reflectivity, surface friction coefficient, and material type; Texture features include texture direction, texture density, texture contrast, and texture roughness.

7. The method for inverse deduction and dynamic reproduction of embroidery stitches based on deep learning according to any one of claims 1 to 5, characterized in that The feature vectors in Step 3 include geometric features, material features, and spatio-temporal encoding.

Images