Row needle trajectory generation system, method based on multi-modal input

Abstract

The application provides a system and method for generating a needle track based on multi-modal input, which comprises a multi-modal input module, an alignment module and a needle track generation module; the multi-modal input module is used for receiving multi-modal data; the alignment module is used for extracting data features of each modal data respectively; the needle track generation module is used for automatically generating an embroidery pattern based on the data features, and quantifying the color of the embroidery pattern to determine the needle track corresponding to the multi-modal data. The system provided by the application can express the design intention more freely and flexibly by receiving multi-modal data, and can fuse the multi-modal data by extracting data features of each modal data respectively through the alignment module, so as to automatically generate an embroidery pattern corresponding to the multi-modal data, and quantifying the color of the embroidery pattern to determine the needle track corresponding to the multi-modal data, thereby realizing the automatic generation of the needle track of the embroidery machine conveniently, efficiently and accurately.

Description

Technical Field

[0001] This invention relates to the field of embroidery machine control technology, and in particular to a needle trajectory generation system and method based on multimodal input. Background Technology

[0002] With the rapid development of information technology, the embroidery industry is increasingly demanding automation, intelligence, and personalization. Traditional methods for generating needle tracks on embroidery machines mainly rely on manual design and pattern making, whereby professional technicians design corresponding embroidery patterns and their corresponding needle tracks based on the embroidery requirements.

[0003] However, manual methods are inefficient, make it difficult to accurately generate complex patterns, and cannot adapt to changing market demands. Summary of the Invention

[0004] This invention provides a needle trajectory generation system and method based on multimodal input, which solves the shortcomings of the existing technology that is inefficient, difficult to accurately generate complex patterns, and unable to adapt to changing market demands due to the manual method.

[0005] This invention provides a needle trajectory generation system based on multimodal input, comprising: a multimodal input module, an alignment module, and a needle trajectory generation module;

[0006] The multimodal input module is used to receive multimodal data, which includes at least one of voice, text, and image.

[0007] The alignment module is used to extract the data features of each modality data respectively;

[0008] The needle trajectory generation module is used to automatically generate the embroidery pattern corresponding to the multimodal data based on the data characteristics of each modal data, and to perform color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data.

[0009] According to the present invention, a needle trajectory generation system based on multimodal input is provided, wherein the needle trajectory generation module includes a color quantization unit, a shape generation unit, a needle pattern matching unit, and a needle trajectory generation unit.

[0010] The color quantization unit is used to perform color clustering on the embroidery pattern to obtain a quantized pattern.

[0011] The shape generation unit is used to obtain the pattern shape based on the color blocks in the quantized pattern;

[0012] The needle pattern matching unit is used to determine the needle pattern that matches the shape of the pattern.

[0013] The needle path generation unit is used to fill the pattern shape based on the needle pattern corresponding to the pattern shape to obtain the needle path.

[0014] According to the needle trajectory generation system based on multimodal input provided by the present invention, the color quantization unit is further specifically used to perform color clustering on the embroidery pattern, and filter the clustered colors based on the colors required for the needlework to obtain the quantized pattern.

[0015] According to the present invention, a needle trajectory generation system based on multimodal input is provided. The needle trajectory generation module further includes an embroidery pattern generation unit, which is used to automatically generate the embroidery pattern step by step based on the data features of each modal data and noise image information.

[0016] According to the needle trajectory generation system based on multimodal input provided by the present invention, the embroidery pattern generation unit is further specifically used to automatically generate the embroidery pattern step by step based on the data features of each modal data, deep learning algorithms, and noise image information.

[0017] According to the present invention, a needle trajectory generation system based on multimodal input is provided, wherein the embroidery pattern generation unit includes an optimization unit, which is used to optimize the embroidery pattern based on needlework rules and / or process requirements to obtain an optimized embroidery pattern.

[0018] According to the multimodal input-based needle trajectory generation system provided by the present invention, the optimization unit is further configured to receive user input and adjust the embroidery pattern based on the user input to obtain the adjusted embroidery pattern.

[0019] According to the present invention, a needle trajectory generation system based on multimodal input is provided, wherein the needle trajectory generation module further includes an output module, the output module being used to convert the needle trajectory into needle instructions recognizable by the embroidery machine, and to send the needle instructions to the embroidery machine.

[0020] According to the present invention, a needle trajectory generation system based on multimodal input is provided, wherein the alignment module includes a data preprocessing unit and an alignment unit;

[0021] The data preprocessing unit is used to transcribe the speech in the multimodal data into corresponding transcribed text, and to scale, crop, and normalize the images in the multimodal data to obtain clean modal data.

[0022] The alignment unit is used to extract the data features of each cleaning modality data respectively.

[0023] The present invention also provides a method for generating needle trajectory based on multimodal input, comprising:

[0024] Acquire multimodal data, wherein the multimodal data includes at least one of speech, text, and images;

[0025] Extract the data features of each modality separately;

[0026] Based on the data characteristics of each modal data, an embroidery pattern corresponding to the multimodal data is automatically generated, and the embroidery pattern is color-quantified to determine the needle trajectory corresponding to the multimodal data.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the needle trajectory generation method based on multimodal input as described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for generating needle trajectory based on multimodal input.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for generating needle trajectory based on multimodal input.

[0030] The present invention provides a needle trajectory generation system and method based on multimodal input. It receives multimodal data through a multimodal input module, enabling users to express their design intentions more freely and flexibly. An alignment module extracts the data features of each modality to fuse the multimodal data, which is then used by the needle trajectory generation module to automatically generate embroidery patterns corresponding to the multimodal data. Furthermore, color quantification of the embroidery patterns determines the needle trajectory corresponding to the multimodal data, achieving convenient, efficient, and user-compliant automated generation of embroidery machine needle trajectories. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the structure of the needle trajectory generation system based on multimodal input provided by the present invention;

[0033] Figure 2This is one of the flowcharts illustrating the needle trajectory generation method based on multimodal input provided by the present invention;

[0034] Figure 3 This is the second flowchart of the needle trajectory generation method based on multimodal input provided by the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] To address the above problems, this invention provides a needle trajectory generation system based on multimodal input, which enables fast and universal needle trajectory generation. Figure 1 This is a schematic diagram of the structure of the needle trajectory generation system based on multimodal input provided by the present invention, as shown below. Figure 1 As shown, the system includes: a multimodal input module 110, an alignment module 120, and a needle trajectory generation module 130;

[0038] The multimodal input module 110 is used to receive multimodal data, which includes at least one of voice, text, and image.

[0039] The alignment module 120 is used to extract the data features of each modality data respectively;

[0040] The needle trajectory generation module 130 is used to automatically generate the embroidery pattern corresponding to the multimodal data based on the data characteristics of each modal data, and to perform color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data.

[0041] Specifically, in the needle trajectory generation system based on multimodal input, the multimodal input module 110, alignment module 120, and needle trajectory generation module 130 are connected sequentially. First, the multimodal input module 110 can receive multimodal data input by the user. This multimodal data includes mainstream data types used in human-computer interaction applications, namely voice, text, and images. It should be noted that the received multimodal data includes at least one of voice, text, and images. Furthermore, the data modality of the received multimodal data is not limited to documents, images, and voice.

[0042] Understandably, compared to existing technologies that rely on single-modal input, generating needle paths for embroidery machines based on multimodal data allows for a wider range of applications for needle path generation systems. Furthermore, compared to the information provided by single-modal input, needle paths generated based on multimodal data are more accurate and are better suited for generating needle paths for more complex embroidery patterns.

[0043] Next, the pre-trained CLIP (Contrastive Language-Image Pre-training) model can be loaded through the alignment module 120. Data from each modality is input into the CLIP model, and the CLIP model extracts the embedding vectors for each modality, which are denoted as data features. Specifically, if the received multimodal data includes speech data, the speech data can be transcribed into text first. Then, the text and image modal data are input into the CLIP model respectively. The text encoder in the CLIP model extracts embedding vectors from the text, and the image encoder extracts embedding vectors from the image, thus obtaining the data features corresponding to each modality.

[0044] It should be noted that during the pre-training phase of CLIP, a large number of text-image pairs can be constructed as sample datasets to pre-train the initial CLIP model. Each training sample contains a matching pair of text and image. The training objective of the CLIP model is to learn a text encoder and an image encoder, such that the extracted text embedding vectors and image embedding vectors of the text-image pairs are as close as possible in the representation space, while the embedding vectors of mismatched text and image pairs are as far apart as possible. Thus, in practical applications, the CLIP model has learned how to map text and images to the representation space.

[0045] It should also be noted that, to better achieve the training objectives of the CLIP model, a contrastive loss function can be used to iterate the parameters of the initial CLIP model. For each text-image pair, the similarity between the text embedding and the image embedding, such as cosine similarity, can be calculated and compared with all other mismatched text-image pairs. Thus, the contrastive loss function encourages the CLIP model to increase the similarity of matching pairs while decreasing the similarity of mismatched pairs. This ensures that the embedding vectors corresponding to input data with similar pattern meanings represent similar semantics, while the embedding vectors corresponding to input data with inconsistent pattern meanings represent different semantics. This guarantees that the data features extracted from each modality based on the CLIP model can accurately represent the pattern information contained in each modality.

[0046] Furthermore, the generation model in the needle trajectory generation module 130 can automatically generate embroidery patterns corresponding to the multimodal data based on the data characteristics of each modality. For example, it could be a lotus flower, including the petals, stems, and colors of each area. Thus, the embroidery pattern here can be an image containing pattern texture and pattern colors. It should be noted that, compared to manually generated patterns, automatically generating embroidery patterns that match the multimodal data through the generation model can flexibly and conveniently generate embroidery patterns of various complexities without relying on professional technicians.

[0047] After obtaining the embroidery pattern, color quantization can be performed. This can be done using K-means color quantization, dividing the color space of the embroidery pattern into K regions. The color of each pixel in the image is then replaced with the color of its corresponding region, effectively clustering the colors in the embroidery image, with K clusters. By adjusting the colors in the embroidery pattern, the outlines of different color blocks can be used as the various embroidery shapes, thus obtaining the needlework path corresponding to each shape. This needlework path can be obtained by filling each embroidery shape according to its corresponding needlework pattern.

[0048] It should be noted that after generating the embroidery pattern based on the generative model, the corresponding needlework trajectory is automatically generated based on the embroidery pattern, further realizing flexible and de-professional needlework trajectory generation, making the application of the needlework trajectory generation system based on multimodal input more convenient.

[0049] The system provided in this invention receives multimodal data through a multimodal input module, enabling users to express their design intentions more freely and flexibly. An alignment module extracts the data features of each modality to fuse the multimodal data, which is then used by a needle trajectory generation module to automatically generate embroidery patterns corresponding to the multimodal data. Furthermore, color quantification of the embroidery patterns determines the needle trajectory corresponding to the multimodal data, achieving convenient, efficient, and user-compliant automated generation of embroidery machine needle trajectories.

[0050] Based on any of the above embodiments, the needle path generation module includes a color quantization unit, a shape generation unit, a needle pattern matching unit, and a needle path generation unit:

[0051] The color quantization unit is used to perform color clustering on the embroidery pattern to obtain a quantized pattern.

[0052] The shape generation unit is used to obtain the pattern shape based on the color blocks in the quantized pattern;

[0053] The needle pattern matching unit is used to determine the needle pattern that matches the shape of the pattern.

[0054] The needle path generation unit is used to fill the pattern shape based on the needle pattern corresponding to the pattern shape to obtain the needle path.

[0055] Specifically, the needle path generation module includes a color quantization unit, a shape generation unit, a needle pattern matching unit, and a needle path generation unit. These units are connected sequentially.

[0056] After the embroidery pattern is generated by the corresponding unit in the needle trajectory generation module, the needle trajectory corresponding to the embroidery pattern is generated by the color quantization unit, shape generation unit, needle pattern matching unit, and needle trajectory generation unit.

[0057] First, color quantization units can be used to cluster the colors in the embroidery pattern, resulting in a quantized pattern. For example, the colors contained in the embroidery pattern can be clustered into K clusters. It should be noted that the quantized pattern contains fewer colors than the embroidery pattern itself, ensuring that the colors in the pattern remain within a manageable range and reducing production difficulty. Furthermore, the colors in the quantized pattern are clustered based on the distribution of pixel colors in the embroidery pattern, thus obtaining the most representative colors. This helps optimize color selection while maintaining the overall visual effect of the pattern, making the embroidery result more aesthetically pleasing and harmonious. Moreover, embroidery often has specific requirements for the thread colors used; for example, certain thread colors may be easier to obtain or more suitable for specific embroidery materials. K-means color quantization maps the colors in the pattern to available thread colors, ensuring the feasibility and practicality of the embroidery process.

[0058] Next, the outlines of the color blocks in the quantized pattern can be used as pattern shapes by the shape generation unit. Further, after obtaining the pattern shapes, the matching stitch pattern can be determined based on each pattern shape in the image. For example, the stitch pattern can be determined by attributes such as the type and size of the pattern shape. Here, the stitch pattern can include tatami stitch, flat stitch, and insert stitch. For example, when the pattern shape is large, the matching stitch pattern could be tatami stitch. Then, the stitch trajectory generation unit fills each pattern shape with a non-repeating, non-empty fill according to the stitch pattern corresponding to each pattern shape, and uses the fill trajectory as the stitch trajectory of the embroidery machine.

[0059] The system provided in this invention performs color clustering on the embroidery pattern using a color quantization unit to obtain a quantified pattern. This ensures that the colors of the pattern are within an operable range, reducing production difficulty. Furthermore, by clustering to obtain the most representative colors, it helps optimize color selection while maintaining the overall visual effect of the pattern, making the embroidery result more aesthetically pleasing and harmonious. Additionally, a needle pattern matching unit determines a needle pattern that matches the shape of the pattern in the embroidery pattern, obtaining a needle trajectory that meets actual production needs, further improving the quality of the needle trajectory.

[0060] Based on any of the above embodiments, the color quantization unit is further specifically used to perform color clustering on the embroidery pattern, and filter the clustered colors based on the colors required for needlework to obtain the quantized pattern.

[0061] Specifically, when quantizing the color of an embroidery pattern based on a color quantization unit, in order to further ensure that the color of the quantized pattern is within the operable range of the embroidery machine, the embroidery pattern can be clustered by color, and the clustered colors can be filtered based on the color required for the needlework to obtain a quantized pattern, thereby further ensuring the practicality of the generated needlework trajectory in the actual embroidery process.

[0062] Based on any of the above embodiments, the needle trajectory generation module further includes an embroidery pattern generation unit, which is used to automatically generate the embroidery pattern step by step based on the data features of each modal data and noise image information.

[0063] Specifically, the needlework trajectory generation module also includes an embroidery pattern generation unit. This unit can fuse textual and image data features, along with an image information array composed of random noise, to generate image information in several steps. The Step parameter is typically 50 or 100. It can consist of a mesh network and a scheduling algorithm, proceeding in a progressive manner (diffusion), adding relevant information at each step. It should be noted that it is precisely this step-by-step processing of the data features from each modality that ultimately generates a high-quality image, resulting in a high-quality embroidery pattern, thereby improving the quality of the needlework trajectory generated based on the embroidery pattern.

[0064] Based on any of the above embodiments, the embroidery pattern generation unit is further specifically used to automatically generate the embroidery pattern step by step based on the data features of each modality data, deep learning algorithms, and noise image information.

[0065] Specifically, in the embroidery pattern generation unit, a powerful deep learning algorithm is used to learn the complex mapping relationship between the input and the embroidery pattern, thereby achieving automatic and accurate generation of embroidery patterns based on multimodal input.

[0066] Based on any of the above embodiments, the embroidery pattern generation unit includes an optimization unit, which is used to optimize the embroidery pattern based on stitch rules and / or process requirements to obtain an optimized embroidery pattern.

[0067] Specifically, after the embroidery pattern generation unit, there is an optimization unit. The optimization unit can optimize and adjust the embroidery pattern generated by the needle trajectory generation module according to the needlework rules of the embroidery machine and / or the process requirements in actual production, so as to obtain an optimized embroidery pattern. This ensures the accuracy and feasibility of the needle trajectory generated based on the optimized embroidery pattern, and further improves the embroidery quality and production efficiency.

[0068] Based on any of the above embodiments, the optimization unit is further configured to receive user input and adjust the embroidery pattern based on the user input to obtain the adjusted embroidery pattern.

[0069] Specifically, the optimization unit can not only optimize and adjust the general needlework rules and process requirements, but also receive user input to adjust the embroidery pattern according to the user input, so as to obtain the adjusted embroidery pattern, making the adjusted embroidery pattern more in line with the user's needs, and thus the accuracy and effectiveness of the needlework trajectory are higher.

[0070] Based on any of the above embodiments, the needle trajectory generation module further includes an output module, which is used to convert the needle trajectory into needle instructions that can be recognized by the embroidery machine, and send the needle instructions to the embroidery machine.

[0071] Specifically, the output module here follows the needle trajectory generation module. This module converts the needle trajectory into an instruction format recognizable by the embroidery machine and sends it to the machine for execution via a dedicated interface. Simultaneously, the output module also features real-time monitoring and feedback functions to ensure the smooth progress of the embroidery process.

[0072] Based on any of the above embodiments, the alignment module includes a data preprocessing unit and an alignment unit;

[0073] The data preprocessing unit is used to transcribe the speech in the multimodal data into corresponding transcribed text, and to scale, crop, and normalize the images in the multimodal data to obtain clean modal data.

[0074] The alignment unit is used to extract the data features of each cleaning modality data respectively.

[0075] Specifically, the alignment module includes a data preprocessing unit and an alignment unit. The data preprocessing unit transcribes the speech in the multimodal data into corresponding transcribed text. Therefore, the alignment unit only needs to process the text and image modal data. The data preprocessing unit also scales, crops, and normalizes the images in the multimodal data to obtain clean modal data, thereby improving the accuracy of the data features extracted by the alignment unit. Furthermore, the alignment unit extracts the data features of each clean modal data segment separately.

[0076] Based on any of the above embodiments Figure 2 This is one of the flowcharts illustrating the needle trajectory generation method based on multimodal input provided by the present invention, such as... Figure 2 As shown, the method includes:

[0077] Step 210: Obtain multimodal data, wherein the multimodal data includes at least one of speech, text, and images;

[0078] Step 220: Extract the data features of each modality data respectively;

[0079] Step 230: Based on the data features of each modal data, automatically generate the embroidery pattern corresponding to the multimodal data, and perform color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data.

[0080] The method provided in this invention receives multimodal data, enabling users to express their design intentions more freely and flexibly. By extracting the data features of each modality, the multimodal data is fused. Based on the data features of each modality, the embroidery pattern corresponding to the multimodal data is automatically generated. By quantifying the color of the embroidery pattern, the needle trajectory corresponding to the multimodal data is determined, thereby achieving convenient, efficient, and user-compliant automated generation of embroidery machine needle trajectories.

[0081] In one embodiment, Figure 3 This is the second flowchart of the needle trajectory generation method based on multimodal input provided by the present invention, as shown below. Figure 3 As shown, the method includes: First, receiving audio input, image input, and text input. After receiving the audio input, performing speech recognition on the audio input to obtain recognized text. Next, inputting the recognized text from the speech recognition, the image input, and the text input together into the CLIP model, and outputting the CLIP representation vector corresponding to each input through the CLIP model. Here, the encoder in the CLIP model can be a variational autoencoder.

[0082] Next, the CLIP representation vector is input into the generative model for image information generation (Diffusion), and the fused features are then decoded to obtain the embroidery pattern. Specifically, the image is generated by the image decoder based on the data features obtained from the preceding information generator. The image decoder runs only once at the end to generate the final pixel image (dimensions: (3, 512, 512) where 3 represents red / green / blue; 512 represents width; 512 represents height), i.e., the embroidery pattern. Furthermore, the generative model can be trained based on a generative adversarial network.

[0083] Next, the colors of the embroidery pattern are quantized to obtain a quantized pattern. Shapes are then generated based on the quantized pattern to obtain the pattern shapes within the embroidery pattern. Next, needle pattern matching is performed on each pattern shape to obtain the corresponding needle pattern. Finally, based on each pattern shape and its corresponding needle pattern, a needle trajectory is generated to obtain the needle trajectory for the embroidery machine.

[0084] The method provided in this invention utilizes advanced deep learning architectures such as generative adversarial networks and variational autoencoders to learn from the fused data features and automatically generate embroidery patterns that meet user needs. Furthermore, the generative model can generate creative and diverse embroidery patterns based on input speech, text, or image information.

[0085] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a needlework trajectory generation method based on multimodal input. This method includes: acquiring multimodal data, including at least one of voice, text, and image; extracting data features from each modality of data; automatically generating an embroidery pattern corresponding to the multimodal data based on the data features of each modality of data; and performing color quantization on the embroidery pattern to determine the needlework trajectory corresponding to the multimodal data.

[0086] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the needle trajectory generation method based on multimodal input provided by the above methods. The method includes: acquiring multimodal data, the multimodal data including at least one of voice, text, and image; extracting data features of each modality data respectively; automatically generating an embroidery pattern corresponding to the multimodal data based on the data features of each modality data; and performing color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data.

[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the needle trajectory generation method based on multimodal input provided by the above methods. The method includes: acquiring multimodal data, the multimodal data including at least one of speech, text, and image; extracting data features of each modality data respectively; automatically generating an embroidery pattern corresponding to the multimodal data based on the data features of each modality data; and performing color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A needle trajectory generation system based on multimodal input, characterized in that, include: Multimodal input module, alignment module, and needle trajectory generation module; The multimodal input module is used to receive multimodal data, which includes at least one of voice, text, and image. The alignment module is used to extract data features of each modality of data respectively; wherein, the alignment module is used to load a pre-trained contrastive language image pre-training model, input each modality of data into the contrastive language image pre-training model respectively, and extract the embedding vectors of each modality of data through the contrastive language image pre-training model to obtain the data features; the training objective of the contrastive language image pre-training model is to learn a text encoder and an image encoder, such that the text embedding vectors and image embedding vectors extracted based on the text encoder and the image encoder are close in the representation space; The needle trajectory generation module is used to automatically generate the embroidery pattern corresponding to the multimodal data based on the data features of each modal data, and to perform color quantization on the embroidery pattern to determine the needle trajectory corresponding to the multimodal data. The needle trajectory generation module includes an embroidery pattern generation unit, which is used to automatically generate the embroidery pattern step by step based on the data features of each modal data and noise image information; the embroidery pattern is an image containing pattern texture and pattern color; The needle trajectory generation module further includes a color quantization unit, a shape generation unit, a needle pattern matching unit, and a needle trajectory generation unit: The color quantization unit is used to perform color clustering on the embroidery pattern and filter the clustered colors based on the colors required for the needlework to obtain a quantized pattern; the quantized pattern contains fewer colors than the embroidery pattern. The shape generation unit is used to obtain the pattern shape based on the color blocks in the quantized pattern; The needle pattern matching unit is used to determine the needle pattern that matches the shape of the pattern. The needle path generation unit is used to fill the pattern shape based on the needle pattern corresponding to the pattern shape to obtain the needle path; The embroidery pattern generation unit is also specifically used to automatically generate the embroidery pattern step by step based on the data features of each modality data, deep learning algorithms, and noisy image information.

2. The needle trajectory generation system based on multimodal input according to claim 1, characterized in that, The embroidery pattern generation unit is followed by an optimization unit, which is used to optimize the embroidery pattern based on stitch rules and / or process requirements to obtain an optimized embroidery pattern.

3. The needle trajectory generation system based on multimodal input according to claim 2, characterized in that, The optimization unit is also used to receive user input and adjust the embroidery pattern based on the user input to obtain the adjusted embroidery pattern.

4. The needle trajectory generation system based on multimodal input according to any one of claims 1 to 3, characterized in that, The needle path generation module further includes an output module, which is used to convert the needle path into needle instructions that the embroidery machine can recognize, and send the needle instructions to the embroidery machine.

5. The needle trajectory generation system based on multimodal input according to any one of claims 1 to 3, characterized in that, The alignment module includes a data preprocessing unit and an alignment unit; The data preprocessing unit is used to transcribe the speech in the multimodal data into corresponding transcribed text, and to scale, crop, and normalize the images in the multimodal data to obtain clean modal data. The alignment unit is used to extract the data features of each cleaning modality data respectively.

6. A method for generating a needle trajectory based on a multimodal input-based needle trajectory generation system according to any one of claims 1 to 5, characterized in that, include: Acquire multimodal data, wherein the multimodal data includes at least one of speech, text, and images; Extract the data features of each modality separately; Based on the data features of each modal data, an embroidery pattern corresponding to the multimodal data is automatically generated, and the embroidery pattern is color quantified to determine the needle trajectory corresponding to the multimodal data. The automatic generation of the embroidery pattern corresponding to the multimodal data based on the data features of each modal data includes: automatically generating the embroidery pattern step by step based on the data features of each modal data and noise image information; the embroidery pattern is an image containing pattern texture and pattern color; The step of color quantization of the embroidery pattern to determine the needle trajectory corresponding to the multimodal data includes: performing color clustering on the embroidery pattern, and filtering the clustered colors based on the colors required for the needlework to obtain a quantized pattern; the quantized pattern contains fewer colors than the embroidery pattern. The pattern shape is obtained based on the color blocks in the quantized pattern; Determine the stitch pattern that matches the shape of the pattern; Based on the needlework pattern corresponding to the pattern shape, the pattern shape is filled to obtain the needlework trajectory.

Images