A clothing sewing control method and system based on artificial intelligence

Abstract

The application is suitable for the field of garment sewing control technology, and particularly relates to a garment sewing control method and system based on artificial intelligence. The method comprises the following steps: acquiring an RGB image, a multispectral image and a vibration modal image of a current sewing area; extracting feature points from the RGB image, the multispectral image and the vibration modal image respectively to obtain first feature points, second feature points and third feature points; determining a multi-modal quality index based on the RGB image, the multispectral image and the vibration modal image, and determining an optimal fusion weight based on the multi-modal quality index; fusing the first feature points, the second feature points and the third feature points according to the optimal fusion weight to obtain fused feature points; predicting an actual position of a first target point according to the fused feature points to obtain predicted feature points; and generating a platform control instruction according to an error between the predicted feature points and a second target point. The method effectively reduces the target positioning error and improves the real-time performance and accuracy of the sewing path tracking.

Description

Technical Field

[0001] This application belongs to the field of garment sewing control technology, and in particular relates to a garment sewing control method and system based on artificial intelligence. Background Technology

[0002] As textile processes become more complex and the demand for mass production increases, traditional hand sewing is gradually becoming unable to meet the requirements of fast, high-precision, and highly consistent production. Therefore, automatic control technology has emerged, mainly using technologies such as computer numerical control (CNC), PLC (programmable logic controller), and servo motors to control the sewing machine's operation in real time with precision.

[0003] In existing automated sewing systems, visual servo control systems are often used to guide the sewing path. However, when dealing with highly reflective (such as silk, coated fabrics) or transparent / semi-transparent (such as tulle, chiffon) fabrics, the performance of traditional computer vision algorithms (such as feature extraction based on SIFT and ORB) drops sharply. The sewing position is easily lost or drifted due to light interference, resulting in sewing path tracking errors and producing defective products.

[0004] In summary, during the process of controlling the sewing of garments, there is a problem that the extraction of visual features is unstable due to special fabrics, which in turn causes errors in sewing path tracking. Summary of the Invention

[0005] This application provides an artificial intelligence-based garment sewing control method and system, which can solve the problem in related technologies where unstable visual feature extraction due to special fabrics leads to sewing path tracking errors during garment sewing.

[0006] In a first aspect, embodiments of this application provide an artificial intelligence-based garment sewing control method, comprising: Acquire the RGB image, multispectral image, and vibration mode map of the current sewing area; Feature points are extracted from the RGB image, the multispectral image, and the vibration modal diagram to obtain a first feature point, a second feature point, and a third feature point; wherein, the first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, the third feature point is the position of the first target point in the vibration modal diagram, and the first target point is the position of the preset sewing path point in the current sewing area; Based on the RGB image, the multispectral image, and the vibration mode map, a multimodal quality index is determined, and based on the multimodal quality index, an optimal fusion weight is determined; wherein, the multimodal quality index includes the quality index of the RGB image, the quality index of the multispectral image, and the quality index of the vibration mode map, and the optimal fusion weight includes the fusion weight of the first feature point, the fusion weight of the second feature point, and the fusion weight of the third feature point; According to the optimal fusion weight, the first feature point, the second feature point and the third feature point are fused to obtain the fused feature point; Based on the fused feature points, the actual position of the first target point is predicted to obtain the predicted feature points; Based on the error between the predicted feature point and the second target point, a platform control command is generated, and the XY platform is controlled to move according to the platform control command; wherein, the second target point is the actual position of the needle landing point.

[0007] The technical solutions described in this application embodiment have at least the following technical effects: This application provides an artificial intelligence-based garment sewing control method. First, it acquires the RGB image, multispectral image, and vibration modal diagram of the current sewing area. Then, it extracts feature points from the RGB image, multispectral image, and vibration modal diagram to obtain a first feature point (the position of a first target point in the RGB image, where the first target point is the position of a preset sewing path point in the current sewing area), a second feature point (the position of the first target point in the multispectral image), and a third feature point (the position of the first target point in the vibration modal diagram). Finally, based on the RGB image, multispectral image, and vibration modal diagram, it determines the multimodal quality index (the quality index of the RGB image). This method utilizes multispectral images and vibration modal maps to achieve multi-angle and multi-level perception of sewing materials, overcoming the limitations of single-modal perception and comprehensively improving the system's ability to understand complex fabrics. The method employs an adaptive weight fusion mechanism that automatically adjusts perception weights based on environmental and fabric changes. When the quality of information in one modality decreases, the weights of other modalities are automatically increased, thereby improving overall recognition stability and system robustness. System latency is a common problem in sewing systems, especially during high-speed sewing, where the target point may have changed position, leading to path tracking deviation. This method compensates for system response latency through a predictive mechanism, effectively reducing target positioning errors and improving the real-time performance and accuracy of path tracking, achieving high-precision synchronization between the needle and the preset path. This method ensures that every stitch in the sewing process falls in the predetermined position, thereby improving production efficiency and sewing quality. It not only reduces human intervention but also enables the efficient operation of automated sewing equipment.

[0008] Secondly, embodiments of this application provide an artificial intelligence-based garment sewing control system, comprising: The acquisition unit is used to acquire the RGB image, multispectral image, and vibration mode diagram of the current sewing area; The feature point extraction unit is used to extract feature points from the RGB image, the multispectral image, and the vibration modal diagram respectively to obtain a first feature point, a second feature point, and a third feature point; wherein, the first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, the third feature point is the position of the first target point in the vibration modal diagram, and the first target point is the position of a preset sewing path point in the current sewing area; The weight determination unit is used to determine a multimodal quality index based on the RGB image, the multispectral image, and the vibration mode map, and to determine the optimal fusion weight based on the multimodal quality index; wherein the multimodal quality index includes the quality index of the RGB image, the quality index of the multispectral image, and the quality index of the vibration mode map, and the optimal fusion weight includes the fusion weight of the first feature point, the fusion weight of the second feature point, and the fusion weight of the third feature point; The feature point fusion unit is used to fuse the first feature point, the second feature point and the third feature point according to the optimal fusion weight to obtain fused feature points; The prediction unit is used to predict the actual position of the first target point based on the fused feature points to obtain the predicted feature points; The sewing unit is used to generate platform control instructions based on the error between the predicted feature point and the second target point, and to control the movement of the XY platform according to the platform control instructions; wherein the second target point is the actual position of the needle drop point.

[0009] Thirdly, embodiments of this application provide an artificial intelligence-based garment sewing control device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in any of the embodiments of the first aspect.

[0010] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a schematic flowchart of an artificial intelligence-based garment sewing control method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an artificial intelligence-based garment sewing control device provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] In related technologies, visual servo control systems are widely used to guide sewing paths in automated garment sewing processes. These systems utilize cameras to identify feature points or edges on the fabric, thereby precisely controlling the sewing machine head or robot to sew. This technology significantly improves sewing efficiency and accuracy, playing a crucial role, especially in large-scale production. However, with the diversification of fabric types, particularly the use of highly reflective and transparent / semi-transparent fabrics, traditional computer vision algorithms face significant challenges, making precise guidance of sewing paths difficult to achieve.

[0020] For example, in highly reflective fabrics (such as silk and coated fabrics), the intensity and angle of reflected light on the fabric surface can vary greatly, making it difficult for traditional computer vision algorithms (such as feature extraction methods based on SIFT and ORB) to reliably identify feature points or edges of the fabric. These algorithms rely on stable local features in the image for matching and tracking, but changes in lighting and reflection phenomena can interfere with feature extraction, leading to errors in the recognition results and thus affecting the guidance of the sewing path.

[0021] Furthermore, for transparent or semi-transparent fabrics (such as tulle and chiffon), traditional computer vision algorithms cannot effectively identify their detailed features because light passes through the fabric surface and the edges of the fabric itself are not clear. Even if the algorithm can extract edges or feature points under certain conditions, the extracted features are often not stable enough due to the thinness and light refraction properties of the fabric. They are easily affected by ambient light and fabric deformation, leading to drift or loss in path tracking.

[0022] While existing methods attempt to improve visual effects by optimizing lighting conditions or using specific filters, these methods have certain limitations. First, optimizing lighting cannot solve light interference in all situations, especially given its poor adaptability to different fabrics and lighting environments. Second, while the use of filters can reduce reflection or light transmission to some extent, this solution lacks universality and cannot comprehensively address variations in various fabrics.

[0023] To complicate matters further, fabric undergoes dynamic deformation during sewing, especially for materials with high elasticity or thinness. Traditional feature extraction algorithms may fail to adapt to these changes in real time, leading to the loss or misalignment of feature points and further exacerbating errors in sewing path tracking. These errors not only affect sewing accuracy but can also result in defective products on the production line, increasing rework and quality control costs.

[0024] To address the aforementioned problems, this application provides an artificial intelligence-based garment sewing control method and system. The method first acquires an RGB image, a multispectral image, and a vibration modal diagram of the current sewing area. Then, feature points are extracted from the RGB image, multispectral image, and vibration modal diagram to obtain a first feature point (the position of a first target point in the RGB image, where the first target point is the position of a preset sewing path point in the current sewing area), a second feature point (the position of the first target point in the multispectral image), and a third feature point (the position of the first target point in the vibration modal diagram). Finally, based on the RGB image, multispectral image, and vibration modal diagram, a multimodal quality index (quality index of the RGB image, quality index of the multispectral image, and quality index of the vibration modal diagram) is determined. This method utilizes multi-spectral images and vibration modal maps to achieve multi-angle and multi-level perception of sewing materials, overcoming the limitations of single-modal perception and comprehensively improving the system's ability to understand complex fabrics. The method employs an adaptive weight fusion mechanism to automatically adjust perception weights based on environmental and fabric changes. When the quality of information in one modality decreases, the weights of other modalities are automatically increased, thereby improving overall recognition stability and system robustness. System latency is a common problem in sewing systems, especially during high-speed sewing, where the target point may have changed position, leading to path tracking deviation. This method compensates for system response latency through a predictive mechanism, effectively reducing target positioning errors and improving the real-time performance and accuracy of path tracking, achieving high-precision synchronization between the needle and the preset path. This method ensures that every stitch in the sewing process falls in the predetermined position, thereby improving production efficiency and sewing quality. It not only reduces human intervention but also enables the efficient operation of automated sewing equipment.

[0025] The AI-based garment sewing control method provided in this application embodiment can be applied to AI-based garment sewing control equipment. In this case, the AI-based garment sewing control equipment is the executing entity of the AI-based garment sewing control method provided in this application embodiment. This application embodiment does not impose any restrictions on the specific type of AI-based garment sewing control equipment.

[0026] For example, an AI-based garment sewing control device may include an RGB image acquisition device, a multispectral image acquisition device, a vibration signal acquisition device, an XY platform, a sewing needle, and a control device that is communicatively connected to the RGB image acquisition device, the multispectral image acquisition device, the vibration signal acquisition device, the XY platform, and the sewing needle. An RGB image acquisition device is a device capable of acquiring RGB images of the sewing area, and can be an industrial camera, a regular high-definition camera, etc.; a multispectral image acquisition device is a device capable of acquiring multispectral images of the sewing area, and can be a multispectral camera, a near-infrared camera, etc.; a vibration signal acquisition device is a device capable of acquiring vibration signals of the sewing area, and can be a laser Doppler vibrometer, a laser displacement sensor, etc.; an XY platform is a device capable of driving the fabric to move on a plane, and can be a flat XY platform (for sewing), an XY electric displacement platform, etc.; a needle is a device capable of performing the sewing action, and can be an electric sewing machine head (such as JUKI, Brother, etc.); a control device is a device capable of data processing and controlling the RGB image acquisition device, multispectral image acquisition device, vibration signal acquisition device, XY platform, and needle, and can be an industrial PC (industrial control computer), a high-performance laptop / desktop computer, an edge computing device, a combination of PLC and industrial PC, etc.

[0027] To better understand the AI-based garment sewing control method provided in this application, the specific implementation process of the AI-based garment sewing control method provided in this application will be described below by way of example.

[0028] Figure 1 A schematic flowchart of an AI-based garment sewing control method provided in an embodiment of this application is shown. The AI-based garment sewing control method includes: S100: Acquire the RGB image, multispectral image, and vibration mode diagram of the current sewing area.

[0029] As can be understood, an RGB image is an image that uses three basic colors—red, green, and blue—to represent colors, with each pixel consisting of the intensity values ​​of three color channels.

[0030] Multispectral images include information on the reflection or radiation of an object or region under multiple different wavelengths of light (including but not limited to visible light, infrared light, near-infrared light, etc.), with each band providing different reflection or absorption characteristics.

[0031] A vibration modal diagram is an image that can show the vibration behavior of an object or region at different frequencies.

[0032] For example, an RGB image acquisition device can capture an image of the current sewing area within the visible light range using a lighting source (which can be an artificial light source, such as an LED light), and convert the image into RGB three primary color values, with each pixel having a corresponding red, green, and blue value, thus obtaining an RGB image. The captured RGB image can be transmitted to the control system via a data interface (such as USB, Ethernet, or fiber optic connection).

[0033] The multispectral image acquisition device can acquire multispectral images of the current sewing area in multiple bands. For example, an infrared filter can be used to capture information in the infrared band, or a near-infrared filter can be used to capture the thermal reaction of the material. The image of each band can be stored as a separate image file. Similar to RGB images, the acquired multispectral images can be transmitted to the control system via a data interface.

[0034] By illuminating the material surface of the current sewing area with a laser beam, the vibration signal acquisition device can capture the material's displacement and vibration velocity, converting them into vibration signals. The acquired vibration signals are then processed through filtering and analysis to form a vibration modal diagram, displaying the dynamic characteristics of the current sewing area (such as vibration frequency and amplitude). The processed vibration modal diagram can be transmitted to the control system via a data interface for further analysis.

[0035] This step allows for the acquisition of multimodal data of the current sewing area. Each image and signal provides different physical information. Comprehensive analysis of multimodal data can better capture the state of the sewing area, thereby enabling precise control of the sewing process.

[0036] S200: Extract feature points from the RGB image, multispectral image, and vibration modal diagram respectively to obtain a first feature point, a second feature point, and a third feature point. The first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, and the third feature point is the position of the first target point in the vibration modal diagram. The first target point is the position of a preset sewing path point in the current sewing area.

[0037] For example, classic feature extraction algorithms (such as SIFT and ORB) can be used to extract the first feature point from an RGB image: the RGB image can be converted to a grayscale image, and Gaussian blur can be applied to the image to reduce the impact of noise on feature point extraction; by constructing an image pyramid, the image is processed at multiple scales from high resolution to low resolution. In each layer of the Gaussian pyramid, Differential Gaussian (DoG) is used to detect local extrema, remove extrema with excessive edge response, and retain extrema with strong stability as key points; the gradient direction and magnitude of the local region around each key point can be calculated to generate a 128-dimensional descriptor for each key point. Based on the descriptor of each key point, feature point matching can be performed, and the successfully matched feature point is the first feature point.

[0038] A convolutional neural network (CNN) can be pre-trained to extract the first feature point. A large number of RGB images from the sewing process can be collected, and the location of the first target point can be manually labeled on the RGB images to obtain a training dataset. This training dataset can then be used to train the CNN model. The model automatically extracts feature points by learning local and global features in the images. During training, the loss function is optimized to enable the model to accurately predict the location of the first feature point.

[0039] Similarly, a convolutional neural network can be pre-trained as a multispectral branch network. Inputting a multispectral image into this network allows it to extract the second feature point. Likewise, a convolutional neural network can be pre-trained as a vibration branch network. Inputting a vibration modal map into this network enables it to extract the second feature point.

[0040] This step enables the precise location of the first target point in different modal data, providing a solid data foundation for subsequent precise control and automated sewing.

[0041] In one possible implementation, S200 extracts feature points from the RGB image, multispectral image, and vibration mode diagram respectively to obtain a first feature point, a second feature point, and a third feature point, including: S210, based on the RGB branch network, extracts feature points from the RGB image to obtain the first feature point.

[0042] It is understandable that classic feature extraction algorithms (such as SIFT and ORB) can extract geometric features from RGB images, which emphasize the local precision of the image; deep neural networks can extract deep semantic features from images, which provide global semantic information about the image. An RGB branch network can be pre-built to integrate traditional geometric features and deep semantic features to achieve stronger feature representation capabilities.

[0043] Feature distillation can guide the RGB branch network to learn the geometric details and accuracy of geometric features. Simultaneously, an attention module is introduced within the network, enabling it to adaptively balance the local geometric information of geometric features with the global semantic information of deep semantic features. This attention mechanism helps the network focus on important regions in the image, thereby improving the effective fusion of different feature types. The specific method for constructing the RGB branch network is detailed in subsequent steps S201-S205.

[0044] For example, an RGB image can be input into an RGB branch network, and the forward propagation of the RGB branch network outputs a heatmap with the same size as the RGB image. The maximum point on the heatmap is the first feature point.

[0045] Optionally, in step S210, feature points are extracted from the RGB image based on the RGB branch network to obtain the first feature points, including: S211 is an encoder based on an RGB branch network that extracts shallow and deep feature maps from RGB images.

[0046] As can be understood, the encoder is the front-end part of the RGB branch network, responsible for extracting multi-level features from the input image. The encoder of the RGB branch network consists of multiple convolutional layers, pooling layers, and activation functions. Each convolutional operation can be seen as extracting increasingly complex features from the original image.

[0047] Shallow feature maps contain low-level information about an image, such as edges, textures, and simple shapes. Extracted early in the encoder process, shallow feature maps capture local details of the image and are suitable for high-frequency information.

[0048] Deep feature maps contain high-level semantic information about an image and can represent more complex patterns such as objects and scenes. Deep feature maps are extracted in the later stages of the network and can help to understand the overall image by progressively processing low-level features in the image.

[0049] For example, in the encoder of the RGB branch network, shallow feature maps are extracted through multiple convolutional layers. These convolutional layers perform convolution operations on the RGB image using small kernels (e.g., 3×3 or 5×5) to extract edges, textures, and other low-level features. To increase non-linear transformation, the output after the convolutional layers can be activated using a non-linear activation function (e.g., ReLU), thereby increasing the network's expressive power. To reduce computational complexity, pooling layers (e.g., max pooling or average pooling) can be used after the shallow convolutions. Pooling layers reduce the spatial size of the image by pooling local regions, while retaining its most important features. After processing through multiple convolutional and pooling layers, the shallow feature maps are obtained.

[0050] Deep feature map extraction involves more convolutional and pooling layers. Similar to shallow feature map extraction, deep feature map extraction also involves processing the image through consecutive convolutional layers. The number and size of the convolutional kernels increase with the depth of the network (e.g., 5×5 or 7×7). After processing through multiple convolutional and pooling layers, deep feature maps are obtained.

[0051] Shallow feature maps and deep feature maps carry local and global information of the image, respectively, laying the foundation for subsequent feature fusion and decoding operations.

[0052] S212, a fusion module based on the RGB branch network, fuses shallow feature maps and deep feature maps to obtain the first fused feature map.

[0053] It is understandable that the main task of the fusion module in the RGB branch network is to effectively fuse the shallow feature maps extracted by the encoder with the deep feature maps, thereby combining the advantages of both. The fusion module introduces a spatial attention mechanism, enabling the RGB branch network to adaptively and dynamically balance the contributions of the two types of features in different image regions.

[0054] For example, bilinear interpolation or transposed convolution can be used to upsample deep feature maps to the same resolution as shallow feature maps, aligning the spatial resolution of the two sets of feature maps for pixel-by-pixel feature fusion.

[0055] Two sets of feature maps can be concatenated along the channel dimension. The concatenated feature map contains both shallow local geometric information and deep semantic context information, providing a rich data foundation for the learning of attention mechanisms.

[0056] The concatenated feature map can be input into a small convolutional network (such as consisting of two 3×3 convolutional layers and a sigmoid activation function) to generate a spatial attention map. This spatial attention map is used to assign a weight value to each pixel location, with the weight ranging from [0, 1]. The weight is used to indicate whether the model should rely more on shallow or deep features at a certain location.

[0057] Spatial attention maps can be used to weight and fuse shallow and deep feature maps to obtain a first fused feature map. When the value at a certain position in the spatial attention map is close to 1, it indicates that the region has rich texture details and clear edges, and the model should rely more on shallow features; when the attention value is close to 0, it indicates that the region is blurry or occluded, and the model should rely more on deep semantic features to maintain stability.

[0058] Through the fusion module, the network can more effectively combine local geometric information and global semantic information. The first fused feature map not only maintains high-precision spatial features, but also has stronger stability and generalization ability at the semantic level.

[0059] S213, Based on the decoder of the RGB branch network and the first fused feature map, a first heat map of the same size as the RGB image is generated, and the maximum point in the first heat map is determined as the first feature point.

[0060] It can be understood that the decoder is the inverse process of the encoder, used to restore high-dimensional, low-resolution feature maps into high-resolution images. The task of the decoder is to gradually recover spatial information while preserving the semantic information in the fused features.

[0061] The decoder consists of an upsampling layer (which gradually increases the resolution of the feature map through nearest neighbor interpolation, bilinear interpolation, or deconvolution), a convolutional layer, an activation function (ReLU or Leaky ReLU), and a normalization layer.

[0062] For example, the first fused feature map can be input into the decoder. The decoder upsamples layer by layer. After each upsampling layer, the spatial size of the first fused feature map gradually approaches the size of the RGB image. When the size of the feature map output by the last layer of the decoder is the same as that of the RGB image, the final spatial response map is obtained. In the decoder output stage, a 1×1 convolutional layer can be added as a classification head to map the high-dimensional features into a single-channel response map. This channel represents the confidence level of each pixel in the image as a feature point. After these steps, a first heatmap with the same size as the RGB image is obtained, where the value of each pixel represents the confidence level of that location as a target point.

[0063] A local extremum (i.e., confidence) search can be performed on the first heatmap to find the pixel position with the highest confidence. This position is the first feature point, and the coordinates of this position are the coordinates of the first feature point.

[0064] This step not only achieves accurate mapping from multi-layer features to spatial coordinates, but also restores detailed localization accuracy while maintaining high semantic understanding capabilities. This enables high-precision recognition and robust localization of the first feature point, and can stably determine the target point position even under conditions such as complex textures, changes in lighting, or partial occlusion.

[0065] In one possible implementation, the AI-based garment sewing control method also includes: S201, Obtain multiple training samples. Each training sample includes a sample image of the sample sewing area, a teacher feature map, and a sample heatmap. The sample image is an RGB image, the sample heatmap is a heatmap of the sample image, and the sample heatmap includes manually labeled actual feature points. The teacher feature map is obtained by extracting features from the sample image using a classic feature extraction algorithm.

[0066] For example, an RGB image acquisition device can be used to photograph the sewing area under different fabrics and sewing conditions, and the acquired RGB images can be formatted and resized (e.g., uniformized to 256×256 or 512×512 pixels) to obtain sample images as input data for the RGB branch network.

[0067] On the sample images, key points (i.e., actual feature points) along the sewing path can be annotated manually or using semi-automatic annotation tools. These key points are preset sewing path points. Each sample image may include one or more key points, and the coordinates of these key points can be pixel positions. A two-dimensional Gaussian distribution centered on the keypoint coordinates can be calculated to obtain a sample heatmap, i.e. ,in, Represents a sample heatmap. Represents the coordinates of any pixel in the sample image. The standard deviation of the Gaussian kernel is used to control the diffusion range and can be set to 1-3% of the image width. If there are multiple key points in the image, the heatmaps can be superimposed and the maximum value taken. The final generated sample heatmap has the same size as the sample image and has high response values ​​at the actual feature point locations labeled manually.

[0068] Classic feature extraction algorithms (such as SIFT and ORB) can be used to extract the set of key points in the sample image and calculate the corresponding response values ​​(such as corner responses or DoG values). The response value of each key point is mapped to a two-dimensional grid of the same size as the sample image to obtain a response map. If a pixel is a key point location, a high response value is assigned to that location; otherwise, a low value or 0 is assigned. The response distribution of key points can be smoothed (e.g., Gaussian filtering) to enhance continuity, and the response map can be normalized to unify its numerical range within the [0, 1] interval to obtain the teacher feature map.

[0069] It can be checked whether the sample image, teacher feature map, and sample heatmap are completely consistent in size. If there are resolution differences, bilinear interpolation can be used to adjust the alignment. Each sample image, teacher feature map, and sample heatmap is combined into a training sample.

[0070] These steps preserve the accuracy of manual annotation while introducing interpretable feature representations from traditional algorithms, providing high-quality input and teacher signals for the subsequent training of the RGB branch network. This enables the network to learn geometric and semantic features collaboratively, improving the accuracy and stability of feature point localization.

[0071] For each training sample: S202, based on the RGB branch network, extracts shallow feature maps, deep feature maps, and sample feature points from sample images.

[0072] For example, the method for extracting shallow feature maps, deep feature maps, predicted heatmaps, and sample feature points from sample images is the same as steps S211 to S213, and will not be repeated here. The predicted heatmap is obtained through the decoder output of the RGB branch network.

[0073] S203, Calculate the principal loss based on the actual feature points and sample feature points in the sample heatmap. The principal loss characterizes the error between the actual feature points and the sample feature points.

[0074] For example, it can be assumed that the sample heatmap is as follows: The actual feature points in the sample heatmap are The predicted heat map is The predicted feature points of the samples in the heatmap are .

[0075] Mean square error can be used to calculate and The error between them, i.e. ,in, , These represent the height and width of the heatmap, respectively. The error between the actual feature points and the sample feature points can be calculated, i.e. or .

[0076] The main loss can be obtained by weighting and fusing the errors between heatmaps and the errors between feature points. , , This represents the weighting coefficients, which can be adjusted based on the performance of the validation set. You can choose 1. It can be taken as 0.1).

[0077] S204, Calculate the distillation loss based on the shallow feature map and the teacher feature map. The distillation loss characterizes the error between the shallow feature map and the teacher feature map.

[0078] This step understandably utilizes the teacher's feature map as a guiding signal, using distillation loss to guide the shallow feature learning of the RGB branch network to enhance its geometric representation capabilities, thereby improving its accuracy and stability in feature point localization. Distillation loss is essentially a form of network knowledge transfer, ensuring that the shallow feature representation of the deep model remains consistent with the geometric features extracted by classic feature extraction algorithms.

[0079] For example, if the shallow feature map and the teacher feature map have different resolutions, bilinear interpolation or deconvolution can be used to upsample or downsample them so that they are spatially aligned; if the number of channels is different, a 1×1 convolutional layer can be used to project the teacher feature map or the shallow feature map into the same channel space.

[0080] The error between the shallow feature map and the teacher feature map can be calculated using the mean squared error loss (MSE), which is called distillation loss. ,in, Indicates distillation loss, , Represents the height and width of the feature map. This represents the number of channels in the feature map. This represents a shallow feature map. This represents a teacher's characteristic graph.

[0081] This step guides the network's shallow features to inherit the edge and corner sensitivity of the teacher's features, enabling the network to maintain high localization accuracy in complex textures and high-frequency regions.

[0082] S205: Calculate the total loss based on the main loss and distillation loss, and update the parameters in the RGB branch network using the backpropagation algorithm based on the total loss.

[0083] For example, the principal loss and distillation loss can be combined using a linear weighting method to form the total loss, i.e. ,in, Indicates the total loss. The weight hyperparameters representing the distillation loss (ranging from 0.1 to 1.0, and can be tuned using a validation set) can be dynamically adjusted during training (emphasizing distillation in the early stages and focusing on predicting the first feature point in the later stages), such as training for the first 20 epochs. =1.0, and then gradually decreased to 0.1.

[0084] The automatic differentiation function of deep learning frameworks (such as PyTorch's loss.backward()) can be used to calculate the partial derivatives (i.e., gradients) of the total loss with respect to all parameters. ,in, This represents all learnable parameters in the RGB branch network. These parameters can be updated using optimizers such as Adam and SGD. ,in, It is the learning rate.

[0085] Steps S202 to S205 constitute one iteration. All training samples are iterated through, which is called one epoch. The network training process executes multiple epochs (such as 100, 200) until the network performance is stable and the trained RGB branch network is obtained.

[0086] The above steps, through combined main supervision and distillation supervision, effectively integrate high-quality manually labeled positional supervision signals with the geometric prior knowledge of classical feature extraction algorithms during the training of the RGB branch network. The main loss guides the network to accurately learn the spatial distribution of feature points, improving target localization accuracy; while the distillation loss allows the shallow network to inherit the local structural representation ability of traditional features during the learning process, enhancing the network's perception of details such as edges and textures. This synergistic optimization enables the RGB branch network to maintain high accuracy, strong robustness, and good generalization ability even when facing complex sewing patterns, weakened textures, occlusion, or noise, thereby significantly improving the reliability of sewing path point detection and subsequent control systems.

[0087] S220, based on a multispectral branch network, extracts feature points from a multispectral image to obtain the second feature point.

[0088] For example, a multispectral image can be input into a multispectral branch network, which propagates forward and performs cross-band information fusion in the inner layers of the network. Finally, the coordinates of the second feature point are regressed in the output layer of the multispectral branch network.

[0089] The training process of the multispectral branch network: Multispectral image acquisition devices can be used to photograph the sewing area under different fabrics and sewing conditions to obtain a multispectral image sample set. Each sample includes multiple band images (such as visible light band images (RGB images) and near-infrared band images (NIR images)). All band images in each sample can be registered and aligned to ensure consistent pixel positions of the same physical point in different images.

[0090] The first target point can be manually labeled in the RGB image, and the location of this point can be mapped to the images of each band as the actual location of the second feature point. A set of sample pairs is constructed by combining each sample in the multispectral image sample set with its corresponding labeled point.

[0091] A convolutional neural network (CNN) specifically designed for processing multispectral images (i.e., a multispectral branch network) can be constructed. The input layer of the network can be adapted to the number of channels in the multispectral image; for example, if four bands (R, G, B, NIR) are used, the number of input channels is four. The intermediate layers of the network can be constructed using two methods: one is early fusion, where multi-band images are directly stacked together as input to the network, allowing the network to learn how to combine information from each band at the front end. Assuming four bands are used, the first layer of the network is a convolutional layer. The network performs convolution operations on the four band images, learning the relationships between the channels. The size of the convolution kernel can be 3×3 or 5×5. After convolution, activation functions (such as ReLU), and pooling, the network gradually extracts low-level features (such as edges and textures) and high-level features (such as objects or structures) from the image. Finally, the output layer of the network can output feature maps, feature point locations, or heatmaps, etc.

[0092] Second, there's dual-stream fusion, which processes information from different spectral bands separately, using two different network branches to learn the features of each band, and then fusing the features from the two branches. These two branches can be an RGB branch and a NIR branch. The RGB branch uses standard convolutional neural networks (such as ResNet, VGG, etc.) to process RGB images and extract information such as color and texture; the NIR branch also uses a convolutional neural network to process NIR images and learn information about material and translucency. Fusion can be achieved in the intermediate layers of the two branches through feature concatenation or attention mechanisms. The fused feature map is then fed into a common backend network for feature point prediction (e.g., outputting the coordinates of feature points through fully connected layers or regression layers).

[0093] Multispectral images from a set of sample pairs can be input into a multispectral branch network. The network outputs predicted feature points (or heatmaps). The error (loss) between the predicted feature points and the labeled points is calculated using the Euclidean distance loss function (the loss for heatmaps can be calculated using mean squared error (MSE)). Using an Adam or SGD optimizer, backpropagation is performed based on the loss to update network parameters such as convolutional kernels, attention parameters, and fusion modules. Multiple epochs of training are performed on the entire set of sample pairs until the network parameters converge, resulting in a trained multispectral branch network.

[0094] This step not only provides high-precision localization in areas with complex textures, but also maintains robustness in challenging environments such as changes in lighting, occlusion, or changes in material translucency, significantly improving the accuracy and stability of feature point extraction.

[0095] S230, based on the vibration branch network, extracts feature points from the vibration modal diagram to obtain the third feature point. The vibration branch network is either MobileNetV2 or U-Net.

[0096] For example, the vibration modal graph can be input into a trained vibration branch network, which can then predict the location of the third feature point.

[0097] The training process of the vibration branch network: Vibration signals of the sewing area under different fabrics and sewing conditions can be collected using a vibration signal acquisition device and converted into vibration mode diagrams. The positions of the actual third feature points in each vibration mode diagram can be manually labeled. Each training sample can consist of a vibration mode diagram and the coordinates of the corresponding actual feature points. All training samples can be divided into a training set and a validation set.

[0098] The network architecture can be MobileNetV2, which uses an inverse residual structure and linear bottlenecks to improve the network's performance under low computational resources; or a smaller U-Net can be chosen, which is suitable for tasks that require the recovery of fine image information. Although the original U-Net is large, a lightweight smaller version can be built by adjusting the network depth and width.

[0099] Initializing network weights using MobileNetV2 or a smaller U-Net architecture can be done using pre-trained models. For MobileNetV2, leveraging weights pre-trained on datasets like ImageNet can help improve training performance.

[0100] The training set can be loaded using batch processing, with each batch containing multiple training samples. The vibration modal maps from each batch are input into the network, where features are extracted using convolutional and pooling layers. The network then outputs the coordinates of the predicted feature points through fully connected layers. The mean squared error (MSE) loss function can be used to calculate the error (loss) between the predicted and actual feature points. Based on the loss, the gradient of the network parameters is calculated using backpropagation. Based on the gradient, the parameters in the network are updated using an optimizer (such as Adam or SGD). These steps are repeated until the network loss on the validation set converges, resulting in a trained vibration branch network.

[0101] Through training, the network learns local and global features in the vibration modal diagram, enabling it to accurately locate and extract feature points even under conditions of vibration mode changes, noise interference, or local occlusion. This step improves the accuracy and robustness of vibration signal analysis, providing reliable feature point information for subsequent analysis and decision-making.

[0102] In one possible implementation, step S100 involves obtaining the vibration modal diagram, including: S110, acquire the original vibration signal.

[0103] For example, the vibration signal of the current sewing area can be collected by a vibration signal acquisition device, which is the original vibration signal.

[0104] S120 performs filtering and detrending processing on the original vibration signal to obtain the processed vibration signal.

[0105] For example, a digital bandpass filter (such as an FIR filter or an IIR filter) can be used to remove low-frequency drift and high-frequency noise from the original vibration signal, retaining the effective vibration signal frequency band to obtain the filtered signal. The digital bandpass filter can be set in the frequency range of 50Hz to 1000Hz.

[0106] Linear detrending (such as fitting a straight line using the least squares method) or high-order polynomial detrending can be used to remove the low-frequency components from the filtered signal. For example, the detrended signal can be obtained by fitting the trend of the filtered signal and subtracting it from the filtered signal; this is the processed vibration signal. The purpose of detrending is to remove long-period variation trends from the signal (such as temperature drift or slowly changing mechanical fault signals), allowing the signal to focus more on short-term dynamic vibrations.

[0107] S130 performs a short-time Fourier transform on the processed vibration signal to obtain the vibration mode diagram.

[0108] For example, a short-time Fourier transform can be applied to the processed signal, dividing it into overlapping segments using a window function. Each segment undergoes a Fourier transform to obtain its corresponding frequency components. The short-time Fourier transform formula is as follows: ,in, The window function can be selected, such as the Hanning window, which can smooth signal edges and reduce spectral leakage. Indicates time offset. Indicates frequency, Represents the imaginary unit. The integral variable is represented by . The resulting vibration modal diagram is a two-dimensional matrix (or time-frequency diagram), which is the time-frequency domain representation of the vibration signal. It shows the energy distribution of the signal at each time point and at each frequency component, and provides the spectral characteristics of the vibration signal and its changes over time.

[0109] The vibration mode diagrams obtained through the above steps provide rich information for subsequent feature point extraction, which can help detect and analyze the dynamic vibration characteristics of the fabric.

[0110] S300 determines multimodal quality indices based on RGB images, multispectral images, and vibration modal maps, and then determines the optimal fusion weights based on these indices. The multimodal quality indices include the quality indices of the RGB images, multispectral images, and vibration modal maps, while the optimal fusion weights include the fusion weights of the first feature point, the second feature point, and the third feature point.

[0111] For example, image processing libraries (such as OpenCV and PIL) can be used to convert RGB images to grayscale images. Grayscale images simplify calculations and improve processing efficiency by preserving the image's brightness information (i.e., brightness channels). ,in, Represents a grayscale image. , , These represent the pixel values ​​of the red, green, and blue channels of the image, respectively. To reduce the impact of noise in the image on subsequent quality calculations, a 3x3 Gaussian kernel (or another kernel of suitable size) can be applied to the image. Perform convolution to obtain the smoothed image, i.e. ,in, Indicates to Image after Gaussian blurring The standard deviation is expressed as The Gaussian kernel represents the image smoothing operation.

[0112] It can analyze grayscale images. The number of pixels corresponding to each gray level, and the number of pixels corresponding to each gray level are calculated together. The ratio between the total number of pixels yields the frequency of each gray level, which is called the gray-level histogram. The gray-level histogram represents the frequency distribution of different gray levels (0~255). We can assume... There is Each gray level has several gray levels. The frequency is ,calculate Information entropy : Information entropy is a standard for measuring the amount of information in an image. Images with higher entropy values ​​contain more information, while images with lower entropy values ​​are blurry, overexposed, or underexposed.

[0113] The Sobel operator can be used to calculate separately. Gradient of each pixel in the x and y directions and And calculate the gradient magnitude of each pixel. ,Right now Calculate the gradient magnitude of all pixels. The average gradient magnitude of the entire image is obtained by averaging the values ​​of the gradients. The average gradient magnitude can reflect the sharpness and edge strength of an image. The larger the gradient magnitude, the richer the edge information of the image, indicating that the image is sharper.

[0114] The quality index of an RGB image can be obtained by combining information entropy and average gradient magnitude using a weighted average method. The weights corresponding to information entropy and average gradient magnitude can be 1, meaning that the quality index of an RGB image is the mean of information entropy and average gradient magnitude.

[0115] Near-infrared band images can be extracted from multispectral images. The signal-to-noise ratio (SNR) can be obtained by calculating the ratio between the average value (representing signal) and the standard deviation (representing noise) of pixels in the near-infrared band image. The local standard deviation of each pixel in the near-infrared band image can be calculated, and the average of the local standard deviations of all pixels can be taken to obtain the contrast ratio. The average of the SNR and contrast ratio can be calculated as a quality indicator of the multispectral image.

[0116] A bandpass filter (e.g., 50Hz~500Hz) can be applied to the vibration signal to remove low-frequency drift and high-frequency noise, resulting in a filtered vibration signal. The signal energy can be obtained by summing the squares of the filtered vibration signal. ,in, Indicates signal energy. This represents the filtered vibration signal. Signal energy reflects the overall intensity of the vibration signal. In vibration modal diagram quality assessment, excessively low energy may indicate that the fabric has not been effectively excited or that there is a problem with the vibration sensor.

[0117] Can be Perform a fast Fourier transform to obtain The frequency domain representation, i.e. And calculate the power spectrum: The dominant frequency significance can be obtained by identifying the main peak power (maximum power value) and the second-highest peak power (second-largest power value) in the power spectrum and calculating their ratio. Dominant frequency significance is an important indicator of the stability and purity of a vibration signal. The main frequency components in a vibration mode determine the signal characteristics; a stronger dominant frequency indicates a more stable and pure vibration signal. The mean of the signal energy and dominant frequency significance can be calculated as a quality indicator of the vibration mode diagram.

[0118] The quality indices of RGB images, multispectral images, and vibration modal graphs have different dimensions and ranges, and these indices can be standardized. The min-max normalization method can be used to map the values ​​of the quality indices of all three modes to the range [0, 1]. ,in, These represent quality indicators for different modalities. This represents the minimum value among the three modalities of quality index. This represents the maximum value among the three modalities of quality; it can be standardized using Z-score, i.e. ,in, This represents the mean of the quality index for the three modes. The standard deviation of the quality index for the three modes is represented.

[0119] Based on the quality index of each modality, a weighted average method can be used to calculate the fusion weight of feature points for each modality. The fusion weight can be proportional to the quality index; the higher the image quality, the greater its fusion weight. ,in, This represents the fusion weights of feature points from different modalities.

[0120] This step, by quantifying the quality of each modal data, adaptively adjusts its weight in the final result based on the effectiveness of each image or signal, thereby achieving more accurate and stable target localization and path tracking. It not only optimizes the fusion process of different modal information but also effectively reduces the impact of noise and interference, contributing to the reliable performance of automated systems in complex environments.

[0121] In one possible implementation, in step S300, multimodal quality indicators are determined based on the RGB image, multispectral image, and vibration modal diagram, including: S310 converts the RGB image to a grayscale image; based on the Laplacian operator, it calculates the Laplacian variance of the grayscale image to obtain the sharpness; it calculates the local standard deviation of the grayscale image to obtain the first contrast ratio. The mean of the sharpness and the first contrast ratio is calculated to obtain the quality index of the RGB image.

[0122] For example, an RGB image can be converted to a grayscale image using an image processing library such as OpenCV or PIL. The Laplacian operator is a second-order derivative operator used to detect rapidly changing regions in an image. In practical applications, a Laplacian filter can be used to... Perform a convolution operation to obtain The Laplace response.

[0123] We can use the Laplace variance formula: The variance of the Laplacian response is calculated, which represents the sharpness. A larger variance indicates richer details and higher sharpness in the image; a smaller variance indicates a blurrier image. express The Laplacian response of each pixel in the image. This represents the average Laplacian response of all pixels. express The number of all pixels in the array.

[0124] A grayscale image can be divided into multiple small blocks (such as 8×8 or 16×16 pixel blocks). The standard deviation of each block can be calculated. The larger the standard deviation, the higher the contrast of that block. The mean of the standard deviations of all small blocks is taken as the average. The contrast ratio (i.e., the first contrast ratio). The standard deviation of each small block is calculated using the following formula: ,in, Indicates standard deviation, This represents the pixel value of a small region. This represents the average value of all pixels in a small region. This indicates the number of pixels in a small area.

[0125] The average of sharpness and first contrast can be calculated as a quality indicator for RGB images.

[0126] Using the average of sharpness and first contrast as a quality indicator for RGB images can quantify the overall quality of the image and provide information about the sharpness and contrast of image details.

[0127] S320: Calculate the signal-to-noise ratio (SNR) of the near-infrared image in the multispectral image, and calculate the local standard deviation of the near-infrared image to obtain the second contrast ratio; calculate the mean of the SNR and the second contrast ratio to obtain the quality index of the multispectral image.

[0128] For example, a multispectral image contains images of multiple bands, from which the near-infrared band image (i.e., the near-infrared image) can be extracted. The mean of all pixel values ​​in the near-infrared image can be calculated; based on the mean of all pixel values, the standard deviation of the near-infrared image can be calculated; the ratio between the mean of all pixel values ​​and the standard deviation of the near-infrared image is the signal-to-noise ratio.

[0129] Similar to the calculation method for the first contrast ratio, the near-infrared image can be divided into multiple small blocks, the standard deviation of each small block can be calculated, and the mean of the standard deviations of all small block regions can be taken as the contrast ratio of the near-infrared image, i.e., the second contrast ratio.

[0130] The signal-to-noise ratio and the average of the second contrast ratio can be calculated as a quality indicator for multispectral images.

[0131] Signal-to-noise ratio (SNR) measures the ratio of signal to noise in an image, while local standard deviation (FSD) reflects brightness variations in different regions of the image. By combining the mean of SNR and secondary contrast ratio, the resulting quality metric provides a comprehensive image quality assessment, which can be effectively used for image quality monitoring, object detection, and subsequent analysis tasks.

[0132] S330: Calculate the average amplitude and standard deviation of the vibration modal graph, and invert the standard deviation; calculate the mean of the average amplitude and the inverted standard deviation to obtain the quality index of the vibration modal graph.

[0133] For example, it can be assumed that the vibration modal diagram is a Two-dimensional matrix ,in, Indicates the height of the vibration modal diagram. Indicates the width of the vibration modal diagram. This represents the time-frequency distribution of the vibration mode diagram.

[0134] The amplitude of a vibration modal diagram represents the energy at each time frequency point. The amplitude can be expressed as the absolute value at that time frequency point. ,in, Representing time and frequency points The amplitude. The mean amplitude can be calculated for all time-frequency points, yielding the average amplitude, i.e. ,in, This represents the average amplitude.

[0135] The standard deviation can be calculated based on the average amplitude, i.e. ,in, This represents the standard deviation. To emphasize areas of small variation in an image, the standard deviation can be inverted. A larger standard deviation indicates a less smooth image, while a smaller standard deviation indicates a smoother image. Inverting the standard deviation can highlight smooth areas and small variations in the image. .

[0136] The mean of the average amplitude and the inverted standard deviation can be used as a quality indicator for the vibration modal diagram. This step effectively quantifies the quality of vibration modal maps, providing a comprehensive indicator of image sharpness and detail.

[0137] In one possible implementation, in step S300, the optimal fusion weights are determined based on the multimodal quality index, including: S301, based on multimodal quality metrics, obtains the optimal fusion weights through forward propagation using a decision model. The decision model is an MLP network.

[0138] For example, multimodal quality metrics can be input into a trained decision model, which then performs a forward propagation and outputs the optimal fusion weights.

[0139] Build a decision-making model, including: S3001, Obtain the initial sample set. Each initial sample includes a multimodal quality index, a first feature point, a second feature point, a third feature point, and a true feature point, which is obtained through manual annotation.

[0140] For example, sample images (RGB images, multispectral images, and vibration modal images) from the training process of the RGB branch network, multispectral branch network, and vibration branch network can be used. Image registration and alignment are performed on the sample images of the three modalities to ensure that the pixel position of the same physical point is consistent in different images. The multimodal quality index of the sample images of the three modalities is calculated through steps S310 to S330, and the first feature point, second feature point, and third feature point are extracted through the trained RGB branch network, multispectral branch network, and vibration branch network, respectively.

[0141] Since the sample images of the three modalities have been registered and aligned, any image can be selected, and real feature points can be manually labeled on the image.

[0142] By following these steps, we can obtain the multimodal quality index and corresponding feature points of the sample images in three modalities, as well as the true feature points.

[0143] S3002: Based on the first feature point, second feature point, third feature point and true feature point in each initial sample, construct an optimization function for each initial sample, and solve the optimization function for each initial sample through an optimization algorithm to obtain the optimal fusion weight for each initial sample.

[0144] For example, an optimization function can be constructed: ,in, Represents the first feature point The fusion weight, Indicates the second feature point The fusion weight, Represents the third feature point The fusion weight, This represents the true feature points. The goal of the optimization function is to minimize the error between the weighted fused feature points and the true feature points.

[0145] The constraints in the optimization process can be ,as well as , , The two constraints of 0 ensure that the weights of each mode are reasonable and that the sum of all weights is 1, which conforms to the basic principle of weighted average.

[0146] The optimization algorithm can be gradient descent, which is used to find the minimum value of the optimization function. The weights can be adjusted by calculating the gradient of the optimization function. , , To minimize the error, we can calculate the partial derivative of the optimization function with respect to each weight, update the weights using gradient descent, and continue until convergence to the optimal solution. , , ,in, This represents the learning rate, which controls the step size for each update.

[0147] For each initial sample, the corresponding optimal fusion weights are solved using the method described above.

[0148] S3003: Construct the final sample set based on the multimodal quality index in each initial sample and the optimal fusion weight of each initial sample.

[0149] For example, the multimodal quality metrics in each initial sample and the optimal fusion weights of each initial sample can be paired to construct the final sample set for training the decision model.

[0150] S3004, based on the final sample set, iteratively trains the initial model to obtain the decision model.

[0151] For example, the initial model can be an MLP (Multilayer Perceptron) model, where the number of nodes in the input layer can be the same as the dimension of the multimodal quality metric. The hidden layers are the core of the MLP, consisting of multiple neurons and using the ReLU activation function to capture complex nonlinear relationships. The first hidden layer can be designed with 64 neurons, and the second hidden layer with 32 neurons. These two hidden layers help the model learn the complex mapping relationship between input features and output weights. The output layer can consist of three neurons and the Softmax activation function, with the three neurons corresponding to three fusion weights. , , .

[0152] The final sample set can be divided into a training set and a validation set. The multimodal quality index of each sample in the training set is input into the initial model. After a nonlinear transformation through two hidden layers, three weight values ​​are output. The error between the predicted weights output by the initial model and the optimal fused weights in the samples is calculated using the loss function (mean squared error, MSE). The gradient of the loss function is passed to each model parameter through the backpropagation algorithm, and the gradient of each parameter is calculated. The Adam optimizer can be used to adjust the model parameters (weights and biases) based on the gradients, gradually reducing the loss and optimizing the model.

[0153] At the end of each training epoch, the training loss is calculated and monitored. If the loss value no longer decreases significantly over multiple epochs, or if a preset stopping condition is met, training can be terminated, resulting in the decision model. A validation set can be used to monitor the training process, prevent overfitting, and evaluate the model's performance on new data.

[0154] The decision model obtained through these steps can automatically adjust the fusion weights according to different quality indicators, achieve high-precision feature point fusion, and provide accurate decision support for the automated processing and analysis of multimodal data.

[0155] S400, based on the optimal fusion weight, fuse the first feature point, the second feature point, and the third feature point to obtain the fused feature point.

[0156] For example, it can be assumed that the position of the first feature point in the RGB image is The location of the second feature point in the multispectral image is The position of the third feature point in the vibration modal diagram is: The formula for calculating the fused feature points is as follows: ,in, Indicates fused feature points, , , They represent , , The fusion weight.

[0157] This step effectively integrates key feature points extracted from RGB images, multispectral images, and vibration modal diagrams. By using the quality information of each modal data for weighted fusion, it reduces the errors and uncertainties that may be caused by a single mode, thus providing a more reliable basis for the accurate positioning of the first target point.

[0158] S500 predicts the actual position of the first target point based on the fused feature points, thus obtaining the predicted feature points.

[0159] It's understandable that the XY platform is constantly moving during the sewing process, and there's a certain time delay in image acquisition, processing, and control command generation. In this situation, directly relying on the currently fused feature points for sewing control might lead to positional errors because the target point has already shifted. Therefore, this delay can be compensated for by predicting the future position.

[0160] For example, it can be assumed that the sewing process follows a linear model, i.e., the speed remains constant. Ignoring external disturbances and complex dynamic factors, the future position of the first target point can be predicted through linear extrapolation based on the current speed and position. The current speed can be manually set before sewing begins; or it can be calculated based on the position of the current fused feature point and the position of the fused feature point from the previous control cycle. ,in, Indicates the current speed. Indicates the position of the currently fused feature point. This indicates the position of the fusion feature point from the previous control cycle. Indicates the duration of the control cycle.

[0161] The actual position of the first target point after the delay time can be calculated using a linear extrapolation formula. ,in, Indicates the predicted feature points, The delay time (i.e., the time required from image acquisition to the generation of control commands) can be measured or estimated depending on the system's processing speed and hardware performance.

[0162] In one possible implementation, S500 predicts the actual position of the first target point based on the fused feature points, obtaining predicted feature points, including: S510 uses the fused feature points as observations and calculates the current state vector based on the observations using the Kalman filter algorithm.

[0163] As can be understood, Kalman filtering is a recursive filter used to estimate the state of a linear dynamic system. Its core idea is to continuously correct the estimate of the system state by fusing observations (fusing feature points) and predictions (the algorithm's estimate).

[0164] For example, the state of a control system can consist of position and velocity, and the state vector can be represented as follows: ,in, Indicates the location of the fused feature points. This represents the speed at which the feature points are fused. The fused feature points can be used as observations, which can be represented as... .

[0165] The predicted state vector for the current time step can be calculated from the state vector of the previous time step using the state transition model in Kalman filtering. The state transition model is ,in, This represents the state prediction vector at the current moment. This represents the state vector from the previous time step. This represents the control input matrix (if there is a control input). In this step, it can be assumed that there is no control input. Zero, Indicates control input, The state transition matrix represents the state transition relationship from the previous time step to the current time step. For a uniform linear motion model, the state transition matrix is... It can be ,in, The time interval represents the time difference between the previous moment and the current moment.

[0166] The Kalman gain can be calculated using the Kalman gain calculation formula. It can be determined based on the Kalman gain. and observed values Correct the current state prediction vector, i.e. ,in, This represents the corrected state prediction vector for the current moment. This represents the observation matrix, where the state is directly observable. It is the identity matrix. Through this correction step, the Kalman filter can correct the current state prediction vector (i.e., the current state vector) based on the observations, resulting in a more accurate position and velocity estimate.

[0167] S520 projects the current state vector forward based on the delay time to obtain the predicted state vector. The delay time is the time from image acquisition to needle drop.

[0168] For example, this can be achieved through the state transition matrix. For the current state vector By projecting forward, we obtain the predicted state vector after the delay time, i.e. ,in, This represents the power of the state transition matrix.

[0169] S530 extracts the position component from the predicted state vector to obtain the predicted feature points.

[0170] For example, the positional components can be directly extracted from the predicted state vector to obtain the predicted feature points, i.e., from... Directly extract positional components This refers to the predicted feature points.

[0171] Kalman filtering continuously optimizes state estimation through prediction and correction steps, enhancing the system's adaptability to dynamic changes. These steps effectively incorporate time delay factors, providing strong support for accurate dynamic prediction and control.

[0172] S600 generates platform control commands based on the error between the predicted feature point and the second target point, and controls the movement of the XY platform according to the platform control commands. The second target point is the actual position of the needle's landing point.

[0173] As can be understood, in this method, the position of the needle remains fixed, while the XY platform moves along with the fabric during sewing (i.e., the fabric is fixed on the XY platform). This step achieves precise sewing by adjusting the position of the XY platform so that the sewing path points on the XY platform are aligned with the needle's landing point.

[0174] For example, the error between the predicted feature point and the second target point in the X-axis and Y-axis directions can be calculated, i.e. , The coordinates of the predicted feature points are: The coordinates of the second target point are .

[0175] The movement of the XY platform can be calculated using proportional control based on the errors in the X and Y axes. , ,in, This represents the amount of movement of the XY platform along the X-axis. This represents the amount of movement of the XY platform along the Y-axis. , This represents the control gain coefficient, used to adjust the XY platform's response to errors. The control gain coefficient can be set experimentally or through parameter tuning to improve the accuracy and stability of the platform's movement. The platform control commands for the XY platform can be expressed as follows: , .

[0176] Platform control commands can be sent to the XY platform's drive system. Upon receiving the commands, the drive system adjusts the X and Y axis motor drives accordingly to precisely move the platform. Through the drive motors, the platform moves the corresponding distance along the X and Y axes, respectively. After the platform adjustment is complete, the first target point aligns with the second target point (i.e., the needle's landing point), achieving precise positioning of the target point.

[0177] This step not only improves the path tracking accuracy during the sewing process, but also effectively overcomes the system delay problem, which is beneficial to the stability and reliability of the automated system.

[0178] In one possible implementation, the AI-based garment sewing control method also includes: S10: Acquire the RGB image, multispectral image, and vibration modal diagram of the current sewing area; S20: Extract feature points from the RGB image, multispectral image, and vibration modal diagram to obtain the first feature point, the second feature point, and the third feature point; S30: Determine the multimodal quality index based on the RGB image, multispectral image, and vibration modal diagram, and determine the optimal fusion weight based on the multimodal quality index; S40: Fuse the first feature point, the second feature point, and the third feature point according to the optimal fusion weight to obtain the fused feature point; S50: Generate needle control instructions based on the fused feature point, and control the needle movement according to the needle control instructions.

[0179] As can be understood, in this method, the XY platform is fixed (i.e., a fixed platform). During the sewing process, all movements and adjustments are achieved through changes in the position of the needle. Since the platform is stationary, the movement control of the needle relies entirely on the fused feature points to determine the target position. The system does not need to consider the delay time between image acquisition and needle descent; the target position at each moment can be obtained in real time by directly calculating the position of the fused feature points.

[0180] For example, the implementation methods of steps S10 to S40 are the same as those of steps 100 to S400.

[0181] In practical applications, the position of the machine needle may be relative to a coordinate system of a certain working area (such as an image coordinate system or a machine coordinate system). Therefore, the positions of the fused feature points can be converted into the actual robot's control coordinates (such as machine tool coordinates). Machine needle control commands can be generated based on these control coordinates. These commands are sent to the machine control system using communication protocols (such as G-code, Modbus, CAN, etc.). The generation of these commands depends on the type of control system used. After receiving the commands, the machine control system drives the machine needle to move along the specified trajectory to the target position (control coordinates).

[0182] These steps enable the sewing needle to perform efficient and accurate automated sewing operations within complex sewing areas by precisely fusing multimodal image data, thereby improving the system's robustness and precision.

[0183] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0184] Corresponding to the AI-based garment sewing control method described in the above embodiments, this application also provides an AI-based garment sewing control device, the various units of which can implement the various steps of the AI-based garment sewing control method.

[0185] The device includes: The acquisition unit is used to acquire the RGB image, multispectral image, and vibration mode diagram of the current sewing area.

[0186] The feature point extraction unit is used to extract feature points from the RGB image, multispectral image, and vibration modal diagram respectively, to obtain a first feature point, a second feature point, and a third feature point. The first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, and the third feature point is the position of the first target point in the vibration modal diagram. The first target point is the position of a preset sewing path point in the current sewing area.

[0187] The weight determination unit is used to determine multimodal quality indices based on RGB images, multispectral images, and vibration modal maps, and to determine the optimal fusion weights based on these indices. The multimodal quality indices include the quality indices of the RGB images, the multispectral images, and the vibration modal maps. The optimal fusion weights include the fusion weights of the first feature point, the second feature point, and the third feature point.

[0188] The feature point fusion unit is used to fuse the first feature point, the second feature point, and the third feature point according to the optimal fusion weight to obtain the fused feature point.

[0189] The prediction unit is used to predict the actual position of the first target point based on the fused feature points, and obtain the predicted feature points.

[0190] The sewing unit generates platform control commands based on the error between the predicted feature point and the second target point, and controls the movement of the XY platform according to the platform control commands. The second target point is the actual position of the needle's landing point.

[0191] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0192] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0193] This application also provides an artificial intelligence-based garment sewing control device. Figure 2 This is a schematic diagram of an AI-based garment sewing control device according to an embodiment of this application. The AI-based garment sewing control device includes an RGB image acquisition device, a multispectral image acquisition device, a vibration signal acquisition device, an XY platform, a sewing needle, and a control device communicatively connected to the RGB image acquisition device, the multispectral image acquisition device, the vibration signal acquisition device, the XY platform, and the sewing needle. Figure 2 As shown, the control device 6 of the AI-based garment sewing control equipment in this embodiment includes: at least one processor 60 ( Figure 2 Only one is shown in the image), at least one memory 61 ( Figure 2 (Only one is shown in the image) and a computer program 62 stored in the at least one memory 61 and executable on the at least one processor 60, wherein when the processor 60 executes the computer program 62, it causes the AI-based garment sewing control device to implement the steps in any of the above-described AI-based garment sewing control method embodiments, or causes the AI-based garment sewing control device to implement the functions of each unit in the above-described device embodiments.

[0194] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 62 in the control device 6 of the artificial intelligence-based garment sewing control equipment.

[0195] The control device 6 of the AI-based garment sewing control equipment can be an industrial PC (industrial control computer), a high-performance laptop / desktop computer, an edge computing device, or a combination of a PLC and an industrial PC. This AI-based garment sewing control equipment may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 2 This is merely an example of an AI-based garment sewing control device and does not constitute a limitation on AI-based garment sewing control devices. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0196] The processor 60 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0197] In some embodiments, the memory 61 may be an internal storage unit of the control device 6 of the AI-based garment sewing control device, such as a hard drive or memory of the AI-based garment sewing control device. In other embodiments, the memory 61 may be an external storage device of the AI-based garment sewing control device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the AI-based garment sewing control device. Furthermore, the memory 61 may include both internal and external storage units of the AI-based garment sewing control device. The memory 61 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0198] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0199] This application provides a computer program product that, when run on an AI-based garment sewing control device, enables the AI-based garment sewing control device to implement the steps described in any of the above method embodiments.

[0200] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to an AI-based garment sewing control device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0201] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0202] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] In the embodiments provided in this application, it should be understood that the disclosed AI-based garment sewing control device, equipment, and method can be implemented in other ways. For example, the AI-based garment sewing control device and equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A garment sewing control method based on artificial intelligence, characterized in that, include: Acquire the RGB image, multispectral image, and vibration mode map of the current sewing area; Feature points are extracted from the RGB image, the multispectral image, and the vibration modal diagram to obtain a first feature point, a second feature point, and a third feature point; wherein, the first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, the third feature point is the position of the first target point in the vibration modal diagram, and the first target point is the position of the preset sewing path point in the current sewing area; Based on the RGB image, the multispectral image, and the vibration mode map, a multimodal quality index is determined, and based on the multimodal quality index, an optimal fusion weight is determined; wherein, the multimodal quality index includes the quality index of the RGB image, the quality index of the multispectral image, and the quality index of the vibration mode map, and the optimal fusion weight includes the fusion weight of the first feature point, the fusion weight of the second feature point, and the fusion weight of the third feature point; According to the optimal fusion weight, the first feature point, the second feature point and the third feature point are fused to obtain the fused feature point; Based on the fused feature points, the actual position of the first target point is predicted to obtain the predicted feature points; Based on the error between the predicted feature point and the second target point, a platform control command is generated, and the XY platform is controlled to move according to the platform control command; wherein, the second target point is the actual position of the needle landing point.

2. The garment sewing control method based on artificial intelligence as described in claim 1, characterized in that, The step of extracting feature points from the RGB image, the multispectral image, and the vibration mode diagram to obtain a first feature point, a second feature point, and a third feature point includes: Based on the RGB branch network, feature points are extracted from the RGB image to obtain the first feature point; Based on a multispectral branch network, feature points are extracted from the multispectral image to obtain second feature points; Based on the vibration branch network, feature points are extracted from the vibration modal diagram to obtain the third feature point; wherein, the vibration branch network is MobileNetV2 or U-Net.

3. The garment sewing control method based on artificial intelligence as described in claim 2, characterized in that, The step of extracting feature points from the RGB image based on the RGB branch network to obtain the first feature point includes: The encoder based on the RGB branch network extracts the shallow feature map and deep feature map of the RGB image; The fusion module based on the RGB branch network fuses the shallow feature map and the deep feature map to obtain a first fused feature map; Based on the decoder of the RGB branch network and the first fused feature map, a first heatmap of the same size as the RGB image is generated, and the maximum point in the first heatmap is determined as the first feature point.

4. The garment sewing control method based on artificial intelligence as described in claim 3, characterized in that, The method further includes: Multiple training samples are obtained; each training sample includes a sample image of the sample sewing area, a teacher feature map, and a sample heatmap. The sample image is an RGB image, the sample heatmap is a heatmap of the sample image, the sample heatmap includes manually labeled actual feature points, and the teacher feature map is obtained by extracting features from the sample image using a classic feature extraction algorithm. For each training sample: Based on the RGB branch network, shallow feature maps, deep feature maps, and sample feature points of the sample image are extracted; The principal loss is calculated based on the actual feature points in the sample heatmap and the sample feature points; wherein the principal loss is used to characterize the error between the actual feature points and the sample feature points. Distillation loss is calculated based on the shallow feature map and the teacher feature map; wherein, the distillation loss is used to characterize the error between the shallow feature map and the teacher feature map; The total loss is calculated based on the main loss and the distillation loss, and the parameters in the RGB branch network are updated using the backpropagation algorithm based on the total loss.

5. The garment sewing control method based on artificial intelligence as described in claim 2, characterized in that, Obtaining the vibration modal diagram includes: Acquire the raw vibration signal; The original vibration signal is filtered and detrended to obtain the processed vibration signal; The processed vibration signal is subjected to a short-time Fourier transform to obtain the vibration mode diagram.

6. The garment sewing control method based on artificial intelligence as described in claim 1, characterized in that, The determination of multimodal quality indicators based on the RGB image, the multispectral image, and the vibration mode diagram includes: The RGB image is converted to a grayscale image; the Laplacian variance of the grayscale image is calculated based on the Laplacian operator to obtain the sharpness; the local standard deviation of the grayscale image is calculated to obtain the first contrast ratio; the mean of the sharpness and the first contrast ratio is calculated to obtain the quality index of the RGB image. Calculate the signal-to-noise ratio (SNR) of the near-infrared image in the multispectral image, and calculate the local standard deviation of the near-infrared image to obtain the second contrast ratio; calculate the mean of the SNR and the second contrast ratio to obtain the quality index of the multispectral image. Calculate the average amplitude and standard deviation of the vibration modal graph, and invert the standard deviation; calculate the mean of the average amplitude and the inverted standard deviation to obtain the quality index of the vibration modal graph.

7. The garment sewing control method based on artificial intelligence as described in claim 1, characterized in that, The determination of the optimal fusion weights based on the multimodal quality index includes: Based on the aforementioned multimodal quality index, the optimal fusion weights are obtained through forward propagation via a decision model; wherein, the decision model is an MLP network. Constructing the decision model includes: Obtain an initial sample set; wherein each initial sample includes a multimodal quality index, a first feature point, a second feature point, a third feature point, and a true feature point, wherein the true feature points are obtained through manual annotation; Based on the first feature point, second feature point, third feature point and true feature point in each initial sample, an optimization function for each initial sample is constructed, and the optimization function for each initial sample is solved by an optimization algorithm to obtain the optimal fusion weight for each initial sample. The final sample set is constructed based on the multimodal quality index of each initial sample and the optimal fusion weight of each initial sample; Based on the final sample set, the initial model is trained iteratively to obtain the decision model.

8. The garment sewing control method based on artificial intelligence as described in claim 1, characterized in that, The step of predicting the actual position of the first target point based on the fused feature points to obtain predicted feature points includes: The fused feature points are used as observations, and the current state vector is calculated using the Kalman filter algorithm based on the observations. Based on the delay time, the current state vector is projected forward to obtain the predicted state vector; wherein, the delay time is the time from the acquisition of the image to the fall of the needle; The positional component is extracted from the predicted state vector to obtain the predicted feature points.

9. The garment sewing control method based on artificial intelligence as described in claim 1, characterized in that, The method further includes: Acquire the RGB image, multispectral image, and vibration mode diagram of the current sewing area; Feature points are extracted from the RGB image, the multispectral image, and the vibration mode diagram respectively to obtain the first feature point, the second feature point, and the third feature point; Based on the RGB image, the multispectral image, and the vibration mode diagram, the multimodal quality index is determined, and based on the multimodal quality index, the optimal fusion weight is determined; According to the optimal fusion weight, the first feature point, the second feature point and the third feature point are fused to obtain the fused feature point; Based on the fused feature points, a needle control command is generated, and the needle is moved according to the needle control command.

10. A garment sewing control system based on artificial intelligence, characterized in that, include: The acquisition unit is used to acquire the RGB image, multispectral image, and vibration mode diagram of the current sewing area; The feature point extraction unit is used to extract feature points from the RGB image, the multispectral image, and the vibration modal diagram respectively to obtain a first feature point, a second feature point, and a third feature point; wherein, the first feature point is the position of the first target point in the RGB image, the second feature point is the position of the first target point in the multispectral image, the third feature point is the position of the first target point in the vibration modal diagram, and the first target point is the position of a preset sewing path point in the current sewing area; The weight determination unit is used to determine a multimodal quality index based on the RGB image, the multispectral image, and the vibration mode map, and to determine the optimal fusion weight based on the multimodal quality index; wherein the multimodal quality index includes the quality index of the RGB image, the quality index of the multispectral image, and the quality index of the vibration mode map, and the optimal fusion weight includes the fusion weight of the first feature point, the fusion weight of the second feature point, and the fusion weight of the third feature point; The feature point fusion unit is used to fuse the first feature point, the second feature point and the third feature point according to the optimal fusion weight to obtain fused feature points; The prediction unit is used to predict the actual position of the first target point based on the fused feature points to obtain the predicted feature points; The sewing unit is used to generate platform control instructions based on the error between the predicted feature point and the second target point, and to control the movement of the XY platform according to the platform control instructions; wherein the second target point is the actual position of the needle drop point.

Images