Deep learning-based embroidery stitch backstepping and dynamic reproduction method

Abstract

The invention discloses an embroidery stitching method backstepping and dynamic reproduction method based on deep learning, and belongs to the technical field of embroidery stitching method identification, and the method comprises the steps: collecting stitch data; fusing the three-dimensional point cloud data acquired from multiple view angles by using a point cloud registration algorithm, and constructing a multi-modal data set; constructing a dynamic graph space-time interaction network, and extracting feature vectors in the multi-modal data based on the dynamic graph space-time interaction network; constructing a stitch inverse model to obtain a stitch space layout instruction, dynamic process parameters and an optimized array sequence; and constructing a dynamic reproduction model to obtain an actual embroidering action sequence. According to the invention, the deep features of the embroidery method in the embroidery image can be automatically learned, and the accuracy of the embroidery method backstepping and the authenticity of the dynamic reproduction are obviously improved; the method can achieve the precise reverse deduction of a plurality of embroidery methods under the conditions of complex textures, different light conditions and various embroidery angles.

Description

Technical Field

[0001] The invention relates to an embroidery needle reproduction method, in particular to an embroidery needle reverse deduction and dynamic reproduction method based on deep learning, belonging to the technical field of embroidery needle recognition. Background Art

[0002] The embroidery reverse deduction and dynamic reproduction method based on machine learning focuses on the reverse analysis and dynamic process restoration of embroidery technology, breaking through the technical bottleneck of traditional manual feature analysis in the three-dimensional needle logic restoration and dynamic embroidery timing modeling.

[0003] Traditional reverse methods rely on manual experience to visually infer the stitch direction and layer stacking relationship of embroidery products. Generally, basic cameras are used to capture the needle execution process and 3D modeling technology is used to achieve virtual restoration. It is difficult to accurately analyze the needle trajectory, detailed characteristics and needle logic of various embroidery images. In addition, the needle reverse inference process of each image requires manpower costs and 3D rendering costs.

[0004] In the research on embroidery reverse deduction and dynamic reproduction methods, the traditional simple feature extraction and matching algorithm based on image processing, when faced with complex embroidery images, often leads to inaccurate embroidery reverse deduction, dynamic reproduction distortion and other problems due to the diversity of embroidery textures, rich variations in embroidery thread colors and inconsistencies in embroidery angles. It is also sensitive to noise interference in the image and has redundant feature extraction. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide an embroidery needle technique reverse inference and dynamic reproduction method based on deep learning, so as to realize high-precision intelligent restoration of the embroidery process from static finished products to dynamic embroidery process.

[0006] Technical solution: The embroidery needle method reverse deduction and dynamic reproduction method based on deep learning of the present invention comprises the following steps: Step 1: Collect pin data, including: Collect 3D point cloud data of finished embroidery products; Collect spectral data and RGB images of finished embroidery products; Collect microsecond-level dynamic details of the embroidery needle movement during the embroidery process; Step 2: Use the point cloud registration algorithm to fuse the 3D point cloud data collected from multiple perspectives, align the coordinate system of the fused 3D point cloud data, spectral data and RGB image through spatial registration, extract geometric features, material features and texture features respectively, and fuse them using the feature-level fusion strategy of attention weighting to construct a multimodal dataset; Step 3, construct a dynamic graph spatiotemporal interaction network, extract feature vectors from multimodal data based on the dynamic graph spatiotemporal interaction network, model the embroidery stitches as a dynamic graph structure based on the feature vectors, and update the edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory; Step 4: construct a needle method reverse inference model based on a physical simulation engine and a generative adversarial network, input the feature vector and embroidery process constraint parameters in step 3 into the needle method reverse inference model, and obtain needle method spatial layout instructions, dynamic process parameters, and optimized array sequence; Step 5: construct a dynamic reproduction model, input the needle space layout instructions, dynamic process parameters and the optimized array sequence into the dynamic reproduction model to obtain the actual embroidery action sequence.

[0007] Furthermore, the dynamic graph spatiotemporal interaction network includes a dynamic graph neural network and a spatiotemporal graph neural network.

[0008] Furthermore, the dynamic graph structure includes a plurality of nodes and edges connecting the nodes, wherein the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes.

[0009] Furthermore, the process of updating edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory in step 3 includes: When new pin data is input, the spatial distance and time interval between the new pin and the existing pins are calculated, and the edge weight is dynamically adjusted based on the calculation results.

[0010] Furthermore, in step 4, constructing a needle method reverse inference model based on the physical simulation engine and the generative adversarial network includes: A dual-path generator is constructed, including a structure generation path and a physical parameter generation path. The generator converts feature vectors into stitch space layout instructions through the structure generation path, and uses the physical parameter generation path to predict dynamic process parameters in parallel; the dynamic process parameters include embroidery thread tension and thread entry angle. Construct a differentiable physics engine, input dynamic process parameters into the differentiable physics engine for forward simulation and iterative optimization, and stop iteration when the embroidery thread tension meets the physical constraints to obtain the optimized array sequence; A global discriminator and a local discriminator are constructed. The global discriminator is used to evaluate the process consistency of the stitch sequence, and the local discriminator is used to evaluate the rationality of the single stitch action.

[0011] Furthermore, the dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-reality transfer learning training module, a real-time adaptive module, and an embroidery thread material recognition and adaptation module.

[0012] Furthermore, the geometric features include three-dimensional coordinates of stitches, curvature of embroidery thread, thickness of embroidery layer stacking, stitch spacing, and stitch angle; Material characteristics include embroidery thread material reflectivity, surface friction coefficient, and material type; Texture features include texture direction, texture density, texture contrast, and texture roughness.

[0013] Furthermore, the feature vector in step 3 includes geometric features, material features and spatiotemporal coding.

[0014] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: By constructing a needle method reverse inference model, the present invention can automatically learn the deep-level features of embroidery methods in embroidery images, and has significantly improved the accuracy of embroidery method reverse inference and the authenticity of dynamic reproduction; The present invention has wider applicability and can realize accurate reverse deduction of various embroidery methods such as even stitch, loop stitch, and connecting stitch under complex textures, different lighting conditions, and various embroidery angles, and can accurately and effectively perform dynamic reproduction. For embroidery images under different colors, backgrounds, and lighting conditions, the present invention can stably output embroidery steps and dynamic demonstrations that conform to actual embroidery technology, providing strong technical support for the inheritance and innovation of embroidery culture, overcoming many limitations of traditional methods in practical applications, and having high practical value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flowchart of the embroidery needle method reverse inference and dynamic reproduction method based on deep learning; Figure 2 It is a structural diagram of the dynamic graph spatiotemporal interaction network; Figure 3 It is a structural schematic diagram of the needle method reverse deduction model; Figure 4 It is a structural diagram of the dynamic reproduction model; Figure 5 This is the recognition effect diagram of random stitch method. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0017] The embroidery needle method reverse deduction and dynamic reproduction method based on deep learning described in this embodiment is as follows: Figure 1 As shown, the method comprises the following steps: Step 1: Collect pin data, including: Collect 3D point cloud data of finished embroidery products; Collect spectral data and RGB images of finished embroidery products; Capture microsecond-level dynamic details of the embroidery needle movement during the embroidery process.

[0018] In one example, a 3D laser scanner is used to collect 3D point cloud data of an embroidery product from multiple perspectives. The 3D point cloud data includes the 3D coordinates of the stitches, the curvature of the embroidery thread, and the thickness of the embroidery layer stack. A multispectral microscopic imaging system is used to collect spectral data of the embroidery product. The spectral data includes the reflectivity of the embroidery thread material, the surface friction coefficient, and the material type. A camera is used to collect RGB images of the embroidery product. An event camera is introduced to capture the microsecond-level dynamic details of the high-speed motion of the embroidery needle during the embroidery process, including the needle jitter trajectory, etc., based on the traditional RGB image, to make up for the motion blur defects of conventional cameras and further enrich the dimension and accuracy of data collection.

[0019] Step 2: Use the point cloud registration algorithm to fuse the 3D point cloud data collected from multiple perspectives. Align the coordinate system of the fused 3D point cloud data, spectral data, and RGB image through spatial registration, extract geometric features, material features, and texture features respectively, and use the attention-weighted feature-level fusion strategy to fuse them and construct a multimodal dataset.

[0020] The point cloud registration algorithm is used to fuse the 3D point cloud data collected from multiple perspectives to eliminate noise and improve the accuracy of analyzing tiny stitch details. The fused 3D point cloud data, spectral data and RGB image are aligned to the coordinate system through spatial registration to extract geometric features, material features and texture features, respectively. The geometric features come from the 3D point cloud data, the material features come from the spectral data, and the texture features come from the RGB image. The geometric features, material features and texture features are fused using the attention-weighted feature set fusion strategy to highlight the more critical information for embroidery process analysis and obtain a multimodal dataset.

[0021] This multimodal dataset usually exists in the form of multidimensional structured data or enhanced point cloud. Multidimensional structured data integrates the feature information of different modalities into a high-dimensional feature vector. Each vector contains geometric features, material features, and texture features. These feature vectors can be organized into a matrix or tensor to facilitate batch processing and feature learning in subsequent deep learning models.

[0022] The enhanced point cloud format refers to the expansion of traditional 3D point cloud data. Each point not only contains spatial coordinate information, but also has additional rich attributes extracted from spectral data and RGB images, such as material reflectivity, color information, and local texture features. This enhanced point cloud can more comprehensively describe the physical and visual characteristics of embroidery works, and provide richer details for the analysis and dynamic reproduction of complex needlework.

[0023] Furthermore, the geometric features include three-dimensional coordinates of stitches, curvature of embroidery thread, thickness of embroidery layer stacking, stitch spacing, and stitch angle; Material characteristics include embroidery thread material reflectivity, surface friction coefficient, and material type; Texture features include texture direction, texture density, texture contrast, and texture roughness.

[0024] Step 3: construct a dynamic graph spatiotemporal interaction network, extract feature vectors from multimodal data based on the dynamic graph spatiotemporal interaction network, model the embroidery stitches as a dynamic graph structure based on the feature vectors, and update the edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory.

[0025] Combination Figure 2 ,Furthermore, the dynamic graph spatiotemporal interaction network includes a dynamic graph neural network and a spatiotemporal graph neural network.

[0026] A dynamic graph spatiotemporal interaction network is constructed by combining the dynamic graph neural network (DGNN) and the spatiotemporal graph neural network (STGNN). The data in the multimodal dataset is input into the dynamic graph spatiotemporal interaction network for feature extraction. The extracted feature vectors include geometric features, material features, and spatiotemporal coding. The embroidery stitches are modeled as a dynamic graph structure based on the extracted feature vectors.

[0027] Among them, spatiotemporal coding is a comprehensive coding of time and space characteristics, which is used to capture the dynamic timing characteristics of the sequence, speed, acceleration, etc. of stitches changing over time during the embroidery process. At the same time, it combines the spatial distribution and geometric relationship of the stitches to fully reflect the spatiotemporal continuity of the embroidery action.

[0028] In the dynamic graph spatiotemporal interaction network, the dynamic graph neural network is responsible for constructing the dynamic graph structure, taking embroidery stitches as nodes and the relationship between nodes as edge weights. The edge weights are updated in real time according to the spatiotemporal continuity of the stitch motion trajectory to reflect the dynamic relationship changes between stitches. The spatiotemporal graph neural network focuses on capturing the spatiotemporal dependency of stitch motion, analyzing the mutual influence of stitches in time and space, and thus understanding the motion pattern and spatiotemporal characteristics of stitches during embroidery. The process of fusing feature information adopts the feature-level fusion strategy of attention weighting, integrating geometric features, material features, texture features and spatiotemporal coding to form a comprehensive feature vector. These feature vectors not only contain the geometry, material and texture information of embroidery stitches, but also the dynamic characteristics of stitches changing over time, such as sequence, speed and acceleration. Finally, the feature vectors output by the dynamic graph spatiotemporal interaction network are used for needle method inversion and dynamic reproduction, which provides a basis for the generation of subsequent needle method spatial layout instructions, dynamic process parameters and optimized array sequences. In this way, the dynamic graph spatiotemporal interaction network can not only capture the static features of embroidery stitches, but also understand their dynamic characteristics, thereby providing a comprehensive and accurate feature representation for the reverse and dynamic reproduction of embroidery technology.

[0029] The dynamic graph neural network and the spatiotemporal graph neural network are integrated in the dynamic graph spatiotemporal interaction network, and work together to extract and model the features of multimodal data. The dynamic graph neural network updates the edge weights, and the spatiotemporal graph neural network captures spatiotemporal dependencies. The two work together to improve the model's understanding and analysis of the dynamic changes of embroidery stitches, and can provide more accurate feature representation for subsequent stitch reverse deduction and dynamic reproduction.

[0030] Furthermore, the dynamic graph structure includes multiple nodes and edges connecting two nodes, wherein the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes, and the edge weight reflects the association strength or similarity between the two connected nodes. The role of updating the edge weight is to dynamically adjust the association strength between the nodes in the graph structure so as to more accurately reflect the spatiotemporal relationship between embroidery stitches, including the correlation and sequence between stitches. The update mechanism is crucial to improving the parsing accuracy and dynamic reproduction ability of the model. The correlation and sequence between stitches are important bases for constructing a needle method reverse model, helping the model to more accurately generate needle method spatial layout instructions and dynamic process parameters. At the same time, the edge weight information helps to construct a more realistic embroidery action sequence, so that the dynamic reproduction model can generate action instructions that conform to the actual embroidery process based on the spatiotemporal relationship between stitches, thereby improving the authenticity and accuracy of the dynamic reproduction of the embroidery process.

[0031] Furthermore, the process of updating edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory in step 3 includes: When new pin data is input, the spatial distance and time interval between the new pin and the existing pins are calculated, and the edge weight is dynamically adjusted based on the calculation results.

[0032] The new stitch data refers to the stitch information collected in real time during the embroidery process, including 3D point cloud data, spectral data, RGB images, and microsecond-level dynamic details of the embroidery needle movement. The new stitch data is input into the dynamic graph neural network to calculate the spatial distance and time interval between the new stitch and the existing stitch, so as to dynamically adjust the edge weight to reflect the spatiotemporal continuity between the stitches.

[0033] The edge weight update is based on the combined effect of the spatial distance decay factor and the time interval decay factor, and uses an exponential decay function to model the spatiotemporal correlation. The update formula is: , In the formula, Represents the edge weight between node i and node j. The larger the weight value, the stronger the association. Indicates the spatial distance between the new pin and the existing pin, in meters; Indicates the time interval between the new pin and the existing pin, in seconds; α, β, γ, δ represent learnable parameters or preset weight coefficients, which are set in this example: , , , .

[0034] If the calculated spatial distance is 5 cm and the time interval is 5 milliseconds, the edge weight can be calculated as: is 1.0917. If the preset edge weight range is [0,1], the result needs to be normalized. For example, if the Sigmoid function is used for normalization, the final weight is: .

[0035] Step 4: construct a needle method reverse inference model based on the physical simulation engine and the generative adversarial network, input the feature vector and embroidery process constraint parameters in step 3 into the needle method reverse inference model, and obtain the needle method spatial layout instructions, dynamic process parameters and optimized array sequence.

[0036] Combination Figure 3 , further, in step 4, constructing a needle method reverse inference model based on the physical simulation engine and the generative adversarial network includes: A dual-path generator is constructed, including a structure generation path and a physical parameter generation path. The generator converts feature vectors into stitch space layout instructions through the structure generation path, and uses the physical parameter generation path to predict dynamic process parameters in parallel; dynamic process parameters include embroidery thread tension and thread entry angle, etc. Construct a differentiable physics engine, input dynamic process parameters into the differentiable physics engine for forward simulation and iterative optimization, and stop iteration when the embroidery thread tension meets the physical constraints to obtain the optimized array sequence; A discriminator module is constructed, including a global discriminator and a local discriminator. The global discriminator is used to evaluate the process consistency of the stitch sequence, and the local discriminator is used to evaluate the rationality of the single stitch action.

[0037] Among them, the embroidery process constraint parameters include three-dimensional point cloud data, spectral data, RGB images and dynamic detail parameters of embroidery needle motion collected in the data acquisition stage, geometric features, material features, texture features and fusion parameters in the feature extraction and fusion stage, dynamic graph structure and stitch motion trajectory parameters in the dynamic graph spatiotemporal interactive network construction stage, feature vectors, embroidery process constraints, generator dual paths and discriminator evaluation parameters in the needle method reverse model construction stage, needle method space layout instructions, dynamic process, optimized array sequence, multi-objective reward function, real-time adaptive module, embroidery thread material recognition and embroidery thread tension closed-loop control parameters in the dynamic reproduction model construction stage.

[0038] The generator includes a structure generation path and a physical parameter generation path. The structure generation path receives feature vectors through the input layer, uses a multi-layer perceptron or a convolutional neural network to encode features, then combines the self-attention mechanism or the Transformer structure to capture global dependencies, and then generates stitch space layout instructions through the decoder. The physical parameter generation path also receives feature vectors, extracts features through the encoder, and uses the fully connected layer to predict dynamic process parameters, such as embroidery thread tension and thread entry angle.

[0039] The dual-path structure of the generator is designed with reference to some common architectures in generative adversarial networks (GANs), including the encoder and decoder structures of deep convolutional generative adversarial networks and the self-attention mechanism of the Transformer-based GAN model. It is customized for the embroidery stitch inversion task to achieve the joint optimization of the stitch spatial layout and physical parameters.

[0040] The differentiable physics engine can use embroidery plug-ins such as PyBullet. The differentiable physics engine structure includes parameter input and parsing module, physics simulation module, optimization and iteration module and output module.

[0041] The parameter input and analysis module receives dynamic process parameters, such as embroidery thread tension and stitch speed, as well as embroidery scene configuration parameters, such as embroidery cloth material and embroidery thread specifications, and analyzes and maps these parameters to the corresponding variables in the physical simulation model.

[0042] The physical simulation module includes the embroidery thread dynamics model, the embroidery cloth mechanical model and the stitch interaction model. It uses numerical integration methods and discretization technology to solve the differential equations in the physical simulation model, simulates the movement, deformation and interaction of the embroidery thread with the embroidery cloth during the embroidery process, and outputs simulation results, such as the shape of the embroidery thread on the embroidery cloth, and the force distribution.

[0043] The input of the embroidery thread dynamics model includes the initial position and speed of the embroidery thread, the physical parameters of the embroidery thread, and the external force information. The initial position and speed determine the geometric state and movement trend of the embroidery thread at the beginning of the simulation, which are usually expressed in the form of coordinates and vectors; the physical parameters of the embroidery thread, such as mass, density, elastic modulus, etc., reflect the material characteristics and mechanical properties of the embroidery thread and affect its response when subjected to force; the external force information includes the traction applied by the embroidery needle, the friction and normal force of the embroidery cloth on the embroidery thread, etc. The magnitude and direction of these external forces determine the movement and deformation of the embroidery thread. The output of the embroidery thread dynamics model is the position and speed of the embroidery thread at different times and the tension distribution of the embroidery thread. The position and speed give the movement trajectory of the embroidery thread in the embroidery process in the form of a time series, which is expressed as a set of coordinate points and the corresponding velocity vector, which can be used for subsequent analysis of the embroidery thread's alignment; the tension distribution is the tension of each point along the length of the embroidery thread, reflecting the tightness of the embroidery thread when subjected to force, which is of great significance for judging the embroidery quality, such as whether the embroidery thread is too tight or too loose.

[0044] The embroidery thread dynamics model uses the existing LSTM long short-term memory network and Transformer network. LSTM is a neural network suitable for processing time series data. It can be used to simulate the dynamic behavior of embroidery thread. Its input is the initial state of the embroidery thread and external force information, and its output is the position, speed and tension of the embroidery thread at different times. Transform is a neural network architecture based on the self-attention mechanism, which is suitable for processing complex sequence data. Its input is the initial state of the embroidery thread and external force, and its output is the movement trajectory and tension distribution of the embroidery thread. The advantage of using these two networks is that they can capture global dependencies and are suitable for processing complex dynamic systems. SoftBodyDynamics is used as a plug-in, which is a tool specifically used for soft body dynamics simulation. It can be used to simulate the movement and deformation of embroidery thread. It needs to be used in conjunction with a physical engine such as PyBullet.

[0045] The input of the embroidery cloth mechanical model includes the initial geometry and boundary conditions of the embroidery cloth, the material properties of the embroidery cloth, and the external forces acting on the embroidery cloth. The initial geometry and boundary conditions define the size, shape, and fixing method of the embroidery cloth at the beginning of the simulation, such as the length, width, thickness, and whether the four boundaries are fixed. The material properties of the embroidery cloth, such as Young's modulus, Poisson's ratio, and yield strength, determine the elastic deformation and plastic deformation characteristics of the embroidery cloth when subjected to force. The external forces acting on the embroidery cloth include the tension of the embroidery thread on the embroidery cloth, the pressure generated when the needle penetrates the embroidery cloth, etc. The distribution position and size of the external force affect the deformation mode of the embroidery cloth. The output of the embroidery cloth mechanical model is the deformation of the embroidery cloth and the stress distribution inside the embroidery cloth. The deformation is manifested as the displacement field and strain field of the embroidery cloth at different times. The displacement field describes the position change of each point on the embroidery cloth, and the strain field reflects the degree of microscopic deformation of the embroidery cloth, which can be used to evaluate the change of the shape of the embroidery cloth caused by embroidery; the stress distribution reflects the magnitude and direction of the stress on each point inside the embroidery cloth, which helps to understand the stress state of the embroidery cloth during the embroidery process and has a reference value for predicting possible damage or relaxation of the embroidery cloth.

[0046] The finite element method (FEM) is used in the embroidery thread mechanical model to deal with the complex geometric shapes and boundary conditions formed by the embroidery thread. The principle is to divide the embroidery cloth into many small units, such as triangles or quadrilaterals, and approximately solve the mechanical equations on each unit. This method is very suitable for simulating the complex deformation and stress distribution of the embroidery cloth during the embroidery process, and can handle complex geometric shapes and boundary conditions with high precision. The embroidery thread mechanical model can use a graph neural network or a convolutional neural network. The graph neural network is particularly suitable for processing data with spatial relationships, such as the grid structure of the embroidery cloth. Its input is the initial geometry, boundary conditions, material properties, etc. of the embroidery cloth, and its output is the deformation and stress distribution of the embroidery cloth. The advantage of this network is that it can capture spatial relationships and can handle complex grid structures. The convolutional neural network can be used to process image data. The deformation and stress distribution of the embroidery cloth can be treated as image data for processing. Its input and output are the same as those of the graph neural network. Its advantage is that it can capture local features and is suitable for processing image data. As for plug-ins, you can choose ABAQUS, which is a widely used finite element analysis software that can be used to simulate the mechanical behavior of embroidery cloth. Its advantages are high precision and the ability to provide rich material models and boundary conditions.

[0047] The input of the stitch interaction model includes the geometric parameters of the stitch, the motion state of the stitch, and the state information of the embroidery cloth and embroidery thread. The geometric parameters of the stitch, such as the diameter, length, thickness of the embroidery thread, etc., affect the penetration depth of the stitch in the embroidery cloth and the contact between the embroidery threads; the motion state of the stitch, including the insertion speed, withdrawal speed, insertion angle, etc., determines the interaction mode and time between the stitch and the embroidery cloth; the state information of the embroidery cloth and the embroidery thread, such as the current deformation of the embroidery cloth and the distribution of the embroidery thread on the embroidery cloth, provides the environment background for the stitch interaction model, so that the model can consider the mutual influence between the stitch and the existing embroidery part. The output of the stitch interaction model is the interaction force between the stitch and the embroidery cloth, as well as the local deformation and force of the embroidery thread under the action of the stitch. The interaction force includes the magnitude and direction of friction force, normal force, etc., which reflects the resistance and support force encountered by the stitches during the embroidery process, and helps to analyze the difficulty of embroidery and the stress conditions of the embroidery cloth. The local deformation and force of the embroidery thread under the action of the stitches describe the tiny deformation and force changes of the embroidery thread near the stitches due to the movement of the stitches, which is of great significance for evaluating the wear of the embroidery thread and the fineness of the embroidery pattern.

[0048] The stitch interaction model can use multi-layer perceptron (MLP) and Transformer. MLP can be used to learn the complex nonlinear relationship between stitches and embroidery cloth. Its input is the geometric parameters, motion state, and state information of the stitches. The output is the interaction force between the stitches and the embroidery cloth, and the local deformation and force of the embroidery thread. The advantage of this network is that it is simple and easy to implement, and it is suitable for processing complex nonlinear relationships. The Transformer network can be used to process sequence data in this model, which is suitable for simulating the dynamic interaction between stitches and embroidery cloth. Its input is the motion trajectory of the stitches and time series data, and the output is the interaction force between the stitches and the embroidery cloth, and the local deformation and force of the embroidery thread. The advantage of this network is that it can capture long-distance dependencies in time series. In terms of plug-ins, the pytorch deep learning framework can be used to build neural network models and learn the complex interaction between stitches and embroidery cloth. Or use Plotly, a pytho library for data visualization that supports interactive charts and can be used to visualize the interaction results between stitches and embroidery cloth.

[0049] The optimization and iteration module defines the loss function, measures the difference between the simulation results and the physical constraints, uses the gradient descent optimization algorithm, and uses the automatic derivation mechanism to calculate the gradient of the loss function with respect to the physical parameters to guide the adjustment direction of the physical parameters. It optimizes through multiple iterations until the physical parameters such as the embroidery thread tension meet the physical constraints, such as the embroidery thread tension reaching a reasonable range. The physical parameters refer to the physical properties and mechanical characteristics involved in the embroidery process and that can affect the embroidery effect, including embroidery thread tension, embroidery thread deformation coefficient, embroidery cloth elastic modulus, stitch angle, friction coefficient, and embroidery thread density.

[0050] In one example, the loss function is expressed as: , In the formula, represents the tension of the embroidery thread in the i-th simulation; Indicates the target embroidery thread tension; Indicates the tolerance range of tension; N indicates the number of simulations or data points; Represents the weight coefficient, which is used to balance the influence of different constraints; Represents other physical constraints, such as deformation coefficient, friction, etc.

[0051] The output module organizes and formats the optimized physical parameters and simulation results, and outputs them to the subsequent array sequence generation module through the data interface, providing accurate physical support for the reverse inference and dynamic reproduction of embroidery stitches. The entire differentiable physics engine combines physical simulation and machine learning technology, and is customized according to the specific needs of embroidery technology. The array sequence generation module is part of the needle reverse inference model. It is implemented in the needle reverse inference model through the dual-path structure of the generator. It is responsible for generating needle spatial layout instructions and dynamic process parameters based on the optimized physical parameters and simulation results, and outputting these instructions and parameters to the dynamic reproduction model to generate the actual embroidery action sequence.

[0052] The global discriminator is responsible for evaluating the overall process consistency of the needle sequence. The input of the global discriminator is the feature vector of the entire needle sequence, which covers the needle spatial layout instructions, dynamic process parameters, and optimized array sequence. In the feature extraction layer, the global discriminator uses a convolutional neural network (CNN) or a Transformer structure to extract the global features of the needle sequence. CNN can effectively capture the spatial features in the needle sequence, while Transformer is better at processing long-distance dependencies in sequence data. After feature extraction, the global discriminator further processes the extracted global features through a fully connected layer to generate an evaluation of the overall process consistency of the needle sequence. The global discriminator will eventually output a probability value. The closer the value is to 1, the better the process consistency of the needle sequence.

[0053] The local discriminator is mainly used to evaluate the rationality of a single stitch action. The input of the local discriminator is the feature vector of a single stitch, which includes the geometric features, material features, dynamic process parameters, etc. of the stitch. In the feature extraction layer, the local discriminator uses a multi-layer perceptron (MLP) or convolutional neural network (CNN) structure to extract the local features of a single stitch. MLP can perform nonlinear transformations on the features of the stitches to capture the complex relationship between the features. After feature extraction, the local discriminator further processes the extracted local features through a fully connected layer to generate an evaluation of the rationality of a single stitch action. The local discriminator outputs a probability value. The closer the value is to 1, the higher the rationality of the stitch action.

[0054] Step 5: construct a dynamic reproduction model, input the needle space layout instructions, dynamic process parameters and the optimized array sequence into the dynamic reproduction model to obtain the actual embroidery action sequence.

[0055] Combination Figure 4 The dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-reality transfer learning training module, a real-time adaptive module, and an embroidery thread material recognition and adaptation module. The modules in the dynamic reproduction model work together to complete the dynamic reproduction task of embroidery stitches.

[0056] The multi-objective reinforcement learning control module integrates virtual-real transfer learning and multi-objective optimization mechanisms. Through the neural differential equation modeling module, the needle characteristics are mapped into the embroidery needle motion differential equation in the continuous time domain, realizing the continuous modeling of the embroidery needle trajectory. The multi-objective reinforcement learning control module takes the needle spatial layout instructions, dynamic process parameters and optimized array sequence as inputs, encodes the needle characteristics through the Transformer-based policy network and decodes to generate a multi-objective action strategy. Its value network uses a multi-layer perceptron (MLP) to evaluate the comprehensive value of the action. Then, through the reinforcement learning (RL) control strategy, with the needle spatial layout instructions, dynamic process parameters and optimized array sequence as inputs, the Transformer policy network uses its self-attention mechanism to deeply encode these needle characteristics, comprehensively capture the long-distance dependencies between features, and then decode to generate a multi-objective action strategy. These strategies cover key action decisions such as embroidery needle motion trajectory planning and embroidery thread tension control. The generated multi-objective action strategy will be passed to the value network using MLP. MLP performs nonlinear transformations through multiple neurons, deeply explores the complex relationship between action features, and evaluates the comprehensive value of action strategies in embroidery accuracy, efficiency, safety and other aspects based on the multi-objective reward function including accuracy reward, efficiency reward and safety reward. The RL control strategy optimizes the policy network in a virtual environment with the help of the proximal policy optimization algorithm based on the evaluation results of the value network. If the comprehensive value of the value network evaluation is high, it means that the strategy performs well in achieving multiple goals. The algorithm will strengthen the strategy. Otherwise, the policy network parameters will be adjusted to generate a better strategy. After continuous iterative optimization, the RL control strategy can output virtual embroidery needle motion instructions and digital tension control parameters that are more in line with actual embroidery needs, effectively improving the accuracy and stability of dynamic reproduction of embroidery needle techniques.

[0057] The multi-objective reward function includes precision reward, efficiency reward and safety reward. The precision reward is the mean square error between the virtual embroidery needle trajectory and the theoretical path, the efficiency reward is the negative weighted sum of the simulated embroidery time and the computing resource consumption, and the safety reward is the difference penalty between the virtual embroidery thread tension and the break threshold. The proximal strategy optimization algorithm is used to coordinate the optimization strategy in the virtual environment, and finally outputs the virtual embroidery needle motion instructions and digital tension control parameters.

[0058] In one example, the expression of the multi-objective reward function is: , In the formula, R represents the comprehensive reward value; Represents the weight coefficient, which is used to balance the impact of different reward items; It represents the accuracy reward, which is defined as the negative value of the mean square error between the virtual embroidery needle trajectory and the theoretical path. The expression is: , In the formula, T is the length of the time series, represents the position vector of the predicted trajectory at time t, represents the position vector of the true trajectory at time t; represents the efficiency reward, which is defined as the negative weighted sum of the simulation embroidery time and the computing resource consumption. The formula is: , In the formula, w 1 , w 2 is the weight coefficient, Indicates the simulation embroidery time. Indicates computing resource consumption; represents the safety reward, which is defined as the negative weighted sum of the simulation embroidery time and the computing resource consumption. The formula is: , In the formula, It is the virtual embroidery thread tension, that is, the tension of the simulated embroidery thread; The breaking threshold refers to the maximum tension that the embroidery thread can withstand. When the virtual embroidery thread tension exceeds the breaking threshold, the embroidery thread is at risk of breaking.

[0059] When the reward value of the multi-objective reward function is in an ideal situation, that is, when the error is 0, the time is 0, and the tension is 0, the comprehensive reward value R is 0. At this time, the multi-objective reinforcement learning control module has no penalty and the reward value is maximized; When the reward value is in normal condition, that is, when the error, time and tension are within a reasonable range, the comprehensive reward value R is a small negative number. At this time, the model performs well, but there is room for optimization. When the reward value is in a dangerous situation, that is, the tension exceeds the limit and the error is large, the comprehensive reward value R is a large negative number. At this time, the model generation result is not feasible and needs to be re-optimized.

[0060] The neural differential equation modeling module starts from the spatiotemporal features of the needle method, including geometry, material, and spatiotemporal coding, extracts spatiotemporal correlation features through the graph convolutional network GCN, and constructs a neural ordinary differential equation Neural ODE driven by an LSTM-Transformer hybrid network, mapping the spatiotemporal correlation features into virtual embroidery needle motion equation parameters in the continuous time domain. The fourth-order Runge-Kutta method RK4 with adaptive step size is used for high-precision solution, supporting sub-millimeter trajectory interpolation, such as generating trajectory points and microsecond dynamic response every 0.1 mm, to ensure the smoothness of virtual embroidery needle motion in the simulation environment. For example, when processing the fine flat stitches of Suzhou embroidery, the neural differential equation modeling module captures the tiny displacement changes of the embroidery needle through differential equations, such as 0.05 mm jitter, and adjusts the interpolation density in real time to output high-fidelity continuous motion parameters.

[0061] The expression of the Neural Ordinary Differential Equation is: , In the formula, Represents the hidden state vector, which includes the needle's posture parameters, such as position x(t), velocity v(t), acceleration a(t) and dynamic process parameters, such as thread tension and thread entry angle; represents the dynamics function parameterized by the LSTM-Transformer hybrid network, with input as hidden state h(t) and spatiotemporal features ; The representation is extracted from the spatiotemporal interaction network of the dynamic graph, and contains geometry, material, and spatiotemporal encoding information.

[0062] The virtual-real transfer learning training module first builds a million-level embroidery action chain in the physical simulation engine, pre-trains the control strategy through the proximal strategy optimization algorithm, and then uses the real data collected by the actual equipment combined with meta-learning to quickly fine-tune the policy network. These real data can come from the records of the manual embroidery process or the operating data of other automated embroidery equipment, such as the actual motion trajectory of the embroidery stitches, embroidery thread tension, stitch angle, embroidery needle speed and other information. Specifically, the virtual-real transfer learning training module is divided into two stages: simulation pre-training and virtual data fine-tuning. In the simulation pre-training stage, a million-level stitch interaction action chain is generated in the virtual embroidery physical engine built based on PyBullet, and the general control strategy is pre-trained through the PPO algorithm. In the virtual data fine-tuning stage, the stitch motion data simulated by the virtual embroidery machine is introduced, such as the preset friction coefficient and the virtual motor response delay. The meta-learning framework MAML is used to quickly adjust the policy network parameters to achieve low-attenuation migration from the simulation environment to the virtual reproduction scene on the network end. All training and optimization are based on digital parameters and do not require actual hardware support.

[0063] The real-time adaptive module uses a high-precision laser rangefinder to provide real-time feedback on the embroidery needle position deviation, dynamically adjusts the temporal attention weight of the LSTM-Transformer predictor, and realizes online error compensation. Specifically, the real-time adaptive module drives the adaptive mechanism through virtual sensor data, such as the embroidery needle position deviation and embroidery thread tension error in the simulation environment, and then calculates the virtual embroidery needle position deviation and virtual embroidery thread tension error. According to the calculation results, the adaptive controller uses the LSTM-Transformer predictor to dynamically adjust the temporal attention weight. For example, when the simulation trajectory offset exceeds 0.1 mm, the real-time adaptive module generates a virtual trajectory correction instruction within 5 milliseconds, and realizes error compensation by adjusting the neural differential equation parameters to ensure the sub-millimeter accuracy of the network-side simulation.

[0064] The embroidery thread material recognition and adaptation module identifies the embroidery thread material based on multispectral imaging data, automatically switches the mechanical model parameters and links the embroidery thread tension closed-loop control system. Specifically, based on multispectral imaging data such as reflectivity, texture, and spectral features, the ResNet-50 backbone network is used to extract deep features and classify material types such as silk, cotton thread, or metal thread through a fully connected layer. The elastic modulus, friction coefficient, and fracture threshold of the corresponding material are then dynamically loaded from the preset mechanical parameter library to the virtual differentiable physics engine. For example, when a metal thread is identified, the virtual thread lifting speed is automatically limited to 0.5 m / s in the simulation environment, and the process consistency is maintained by adjusting the virtual embroidery thread tension parameters. All control logic runs completely in a closed loop in the digital twin environment without the need to link physical equipment.

[0065] The preliminary results of the dynamic reproduction authenticity are evaluated based on two criteria, including trajectory similarity (TS) and deformation error (DE). Trajectory similarity measures the similarity between the generated motion trajectory and the real trajectory, and the calculation formula is: , In the formula, T Indicates the length of the time series; and They are the predicted trajectory and the true trajectory in time T The trajectory similarity measures the similarity of trajectories by calculating the cosine similarity between the direction vectors of the predicted trajectory and the true trajectory. The closer the cosine similarity value is to 1, the more similar the trajectories are.

[0066] The deformation error is used to measure the difference between the generated deformation and the real deformation, and the calculation formula is: , In the formula, is the total number of deformation points; and The predicted deformation and the real deformation are The deformation error calculates the average value of the absolute difference between the predicted deformation and the actual deformation, which is used to measure the accuracy of the deformation. The smaller the deformation error, the closer the predicted deformation is to the actual deformation.

[0067] Figure 5 The figure shows the recognition effect diagram of random stitch, i.e., grab stitch, after multimodal data is processed by the deep learning network described in the present invention. 0.94 in the figure is the recognition confidence, indicating that it is possible to accurately reverse a variety of embroidery methods such as even stitch, loop stitch, and connecting stitch under complex textures, different lighting conditions, and various embroidery angles, and can accurately and effectively reproduce them dynamically. For embroidery images under different colors, backgrounds, and lighting conditions, it can stably output embroidery steps and dynamic demonstrations that conform to actual embroidery technology, which provides strong technical support for the inheritance and innovation of embroidery culture, overcomes many limitations of traditional methods in practical applications, and has high practical value and promotion prospects.

Claims

1. A method for reverse inference and dynamic reproduction of embroidery needlework based on deep learning, characterized in that: The steps include: Step 1: Collect pin data, including: Collect 3D point cloud data of finished embroidery products; Collect spectral data and RGB images of finished embroidery products; Collect microsecond-level dynamic details of the embroidery needle movement during the embroidery process; Step 2: Use the point cloud registration algorithm to fuse the 3D point cloud data collected from multiple perspectives, align the coordinate system of the fused 3D point cloud data, spectral data and RGB image through spatial registration, extract geometric features, material features and texture features respectively, and fuse them using the feature-level fusion strategy of attention weighting to construct a multimodal dataset; Step 3, construct a dynamic graph spatiotemporal interaction network, extract feature vectors from multimodal data based on the dynamic graph spatiotemporal interaction network, model the embroidery stitches as a dynamic graph structure based on the feature vectors, and update the edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory; Step 4: construct a needle method reverse inference model based on a physical simulation engine and a generative adversarial network, input the feature vector and embroidery process constraint parameters in step 3 into the needle method reverse inference model, and obtain needle method spatial layout instructions, dynamic process parameters, and optimized array sequence; Step 5: construct a dynamic reproduction model, input the needle space layout instructions, dynamic process parameters and the optimized array sequence into the dynamic reproduction model to obtain the actual embroidery action sequence.

2. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to claim 1 is characterized in that: The dynamic graph spatiotemporal interaction network includes dynamic graph neural network and spatiotemporal graph neural network.

3. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to claim 2 is characterized in that: The dynamic graph structure includes multiple nodes and edges connecting the nodes, wherein the nodes are composed of geometric features and material features, and the edge weight refers to the weight value of the edge connecting two nodes.

4. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to claim 3 is characterized in that: The process of updating edge weights in real time according to the spatiotemporal continuity of the stitch motion trajectory in step 3 includes: When new pin data is input, the spatial distance and time interval between the new pin and the existing pins are calculated, and the edge weight is dynamically adjusted based on the calculation results.

5. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to claim 4 is characterized in that: In step 4, the needle method reverse inference model is constructed based on the physical simulation engine and the generative adversarial network, including: A dual-path generator is constructed, including a structure generation path and a physical parameter generation path. The generator converts feature vectors into stitch space layout instructions through the structure generation path, and uses the physical parameter generation path to predict dynamic process parameters in parallel; the dynamic process parameters include embroidery thread tension and thread entry angle. Construct a differentiable physics engine, input dynamic process parameters into the differentiable physics engine for forward simulation and iterative optimization, and stop iteration when the embroidery thread tension meets the physical constraints to obtain the optimized array sequence; Construct global discriminator and local discriminator.

6. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to claim 5 is characterized in that: The dynamic reproduction model includes a multi-objective reinforcement learning control module, a neural differential equation modeling module, a virtual-reality transfer learning training module, a real-time adaptive module, and an embroidery thread material recognition and adaptation module.

7. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to any one of claims 1 to 6, characterized in that: The geometric features include the three-dimensional coordinates of the stitches, the curvature of the embroidery thread, the thickness of the embroidery layer stacking, the stitch spacing and the stitch angle; Material characteristics include embroidery thread material reflectivity, surface friction coefficient, and material type; Texture features include texture direction, texture density, texture contrast, and texture roughness.

8. The embroidery needle reverse deduction and dynamic reproduction method based on deep learning according to any one of claims 1 to 6, characterized in that: The feature vector in step 3 includes geometric features, material features and spatiotemporal coding.

Images