Computer vision automatic detection method and system for clothing structure defects based on multi-modal AI perception

Abstract

The application provides a garment structure defect computer vision automatic detection method and system based on multi-modal AI perception, and relates to the technical field of intelligent quality inspection. The method first acquires garment perception data of at least two modes and extracts multi-level feature representations; aligns and enhances the features through an inter-modal cross-attention interaction network to generate multi-modal joint features; inputs the multi-modal joint features into a defect detection network to obtain a defect existence score map of each pixel region; estimates a pixel-by-pixel uncertainty heat map of the detection result through a Monte Carlo random inactivation sampling method, and dynamically triggers a review strategy according to the uncertainty heat map; and finally outputs the defect detection result and corresponding uncertainty evaluation information. Through the collaborative mechanism of inter-modal cross-attention interaction and uncertainty quantification evaluation, the precision and reliability of garment structure defect detection under complex working conditions are significantly improved.

Description

Technical Field

[0001] This application relates to the field of intelligent quality inspection technology, and more specifically, to a computer vision automatic detection method and system for garment structural defects based on multimodal AI perception. Background Technology

[0002] The intelligent upgrading of the garment manufacturing industry has placed higher demands on product quality inspection. Traditional garment defect detection mainly relies on manual visual inspection or a single visible light vision system. The former is inefficient and inconsistent, while the latter suffers from a high rate of missed detections in scenarios involving complex fabric textures, reflective materials, and hidden defects. In recent years, multimodal sensing technologies, such as structured light 3D imaging, infrared thermal imaging, and multispectral imaging, have been introduced into the field of garment quality inspection. By acquiring complementary information through sensing channels with different physical characteristics, they provide new solutions to overcome the limitations of single-modal methods.

[0003] However, existing multimodal garment defect detection methods generally employ simple channel splicing or element-by-element addition as early fusion strategies during multimodal information fusion, or perform equal-weighted fusion of features across modalities. This leads to two prominent problems: First, simple linear fusion strategies cannot adequately model the complex nonlinear interactions between different modal features. When a modality is affected by noise (such as drastic changes in illumination or overexposure of fabric reflection), its low-quality features directly contaminate the fusion results, lowering the overall detection performance. Second, existing fusion methods lack explicit modeling of semantic alignment between modalities. Features from different modalities may originate from the same physical location but represent completely different semantic concepts. Direct splicing can cause semantic confusion and reduce the representational power of the features.

[0004] Furthermore, existing detection models generally only output deterministic defect predictions, lacking a quantitative assessment of prediction uncertainty. In actual production environments, when faced with new defect patterns of unknown distribution or suspicious areas with ambiguous edges, detection models often make high-confidence erroneous judgments. Operators find it difficult to know which detection results require manual verification, leading to a serious trust crisis in quality inspection systems in unattended scenarios. Summary of the Invention

[0005] This application provides a computer vision-based automatic detection method and system for garment structural defects based on multimodal AI perception. Through the collaborative mechanism of intermodal cross-attention interaction and uncertainty quantification assessment, the accuracy and reliability of garment structural defect detection under complex working conditions are significantly improved.

[0006] In a first aspect, this application provides a computer vision-based automatic detection method for garment structural defects based on multimodal AI perception, including: Acquire clothing perception data of at least two modalities, and extract multi-level feature maps of each modality through a multi-scale feature pyramid network. The multi-level feature maps contain semantic information with different spatial resolutions. Multi-level feature maps of each modality are input into the inter-modal cross-attention interaction network. The cross-attention calculation is performed with the feature of one modality as the query and the features of the other modalities as the key and value to generate semantically aligned enhanced feature representations between modalities. The enhanced feature representations of each modality are then fused layer by layer at the same scale to obtain a multi-modal joint feature map. The multimodal joint feature map is input into a defect detection network based on a fully convolutional architecture, and a defect existence score map for each spatial location is output through dense prediction. The defect existence score map represents the confidence level of each pixel belonging to a preset defect category. During the testing phase, a random deactivation layer is enabled in the defect detection network. Multiple forward inferences are performed on the same input to obtain multiple defect existence score maps. The pixel-level mean of multiple score maps is calculated as the final defect detection result. The pixel-level variance of multiple score maps is calculated to generate an uncertainty heatmap. Based on the uncertainty heatmap, areas that require manual review are marked.

[0007] Preferably, acquiring clothing perception data of at least two modalities, and extracting multi-level feature maps for each modality using a multi-scale feature pyramid network, specifically includes: For each mode, the corresponding raw sensing data is collected; among them, the two-dimensional visible light mode collects high-resolution visible light images of clothing fabrics, the three-dimensional structured light mode collects depth point cloud data of clothing pieces, and the short-wave infrared mode collects short-wave infrared spectral images of the sewing area. One or more data samples can be collected in each modality; The collected raw sensing data is preprocessed, including illumination correction, spatial registration, and scale normalization. Initial features are extracted by a pre-trained convolutional neural network backbone network and fed into a feature pyramid network. Through top-down path enhancement and lateral connections, feature maps of multiple scales are output and combined into the multi-level feature map.

[0008] Preferably, the multi-level feature maps of each modality are input into the inter-modal cross-attention interaction network. Using the features of one modality as the query and the features of the remaining modalities as keys and values, cross-attention calculation is performed to generate semantically aligned enhanced feature representations between modalities. The enhanced feature representations of each modality are then fused layer by layer at the same scale to obtain the multi-modal joint feature map. Specifically, this includes: For each of the preset defect categories, initialize a set of learnable category prototype query vectors, wherein the defect category includes at least two of the following: fabric defects, sewing defects, size deviations, and structural asymmetry. For each scale of the feature pyramid, cross-attention is calculated using features from at least two modalities as queries and features from the remaining modalities as keys and values. Using the feature map of the visible light modality at that scale as the query, and the feature maps of the corresponding scales of the 3D structured light modality and the shortwave infrared modality as keys and values, the first set of cross-attention weights is calculated. Then, using the feature map of the 3D structured light modality at that scale as the query, and the feature maps of the corresponding scales of the visible light modality and the shortwave infrared modality as keys and values, the second set of cross-attention weights is calculated. The first set of cross-attention weights is multiplied and summed with the feature maps of the 3D structured light mode and the short-wave infrared mode, respectively. The second set of cross-attention weights is multiplied and summed with the feature maps of the visible light mode and the short-wave infrared mode, respectively. The two sets of results are then fused to obtain the cross-modal interaction features at this scale. The similarity between the cross-modal interaction features and the category prototype query vector is calculated to obtain the category attention weights. The cross-modal interaction features are weighted and converged using category attention weights, and then added to the original features through residual connections to generate the enhanced feature representation; The enhanced feature representations at each scale are concatenated along the channel dimension and then dimensionality reduced using a convolutional fusion layer to obtain the multimodal joint feature map.

[0009] Preferably, inputting the multimodal joint feature map into a defect detection network based on a fully convolutional architecture, and outputting a defect presence score map for each spatial location through dense prediction, specifically includes: A decoder-led fully convolutional defect detection network is constructed, wherein the decoder fuses features at different scales through skip connections and gradually restores spatial resolution; The multi-scale, multi-modal joint feature maps are input into the corresponding layers of the decoder, and cross-scale fusion and refinement are performed through upsampling blocks and convolutional layers. After refining through multiple upsampling blocks and convolutional layers, the original input resolution is gradually restored. A 1×1 convolutional layer and a sigmoid activation function are added to the last layer of the decoder to output the defect presence score map, where each pixel value is between 0 and 1, representing the confidence level that the pixel has a defect.

[0010] Preferably, during the testing phase, a random deactivation layer is enabled in the defect detection network to perform multiple forward inferences on the same input, resulting in multiple defect existence score maps. The pixel-level mean of the multiple score maps is calculated as the final defect detection result. The pixel-level variance of the multiple score maps is calculated to generate an uncertainty heatmap. Based on the uncertainty heatmap, areas requiring manual review are marked, specifically including: During the testing phase, a random deactivation layer was enabled after all convolutional layers of the defect detection network, and the deactivation probability was set. Multiple forward inferences were performed on the multimodal joint feature map of the same input, and different neurons were randomly deactivated each time to obtain the sampling results of multiple defect existence score maps. Calculate the mean and variance of multiple sampling results at each pixel location, use the mean as the final defect presence score at that location, and use the variance as the uncertainty estimate at that location to generate an uncertainty heatmap; For each pixel location, if its uncertainty estimate exceeds a preset first threshold, the location is marked as a high uncertainty region; if the average uncertainty of the entire image exceeds a preset second threshold, the sample is marked as a sample requiring manual review.

[0011] Preferably, the areas requiring manual verification based on the uncertainty heatmap marking also include: The bounding box coordinates, defect category prediction results, and uncertainty values ​​of high-uncertainty regions are associated and stored. Generate a review report containing the original multimodal images, defect detection results, and uncertainty heatmap; push the review report to the manual review terminal for quality inspectors to confirm and correct.

[0012] Preferably, the method further includes: based on the final defect detection results and the uncertainty heatmap, automatically marking samples whose uncertainty exceeds a preset alarm threshold as samples requiring manual review, and pushing the multimodal raw data and detection results of the sample to the manual review terminal.

[0013] Secondly, this application provides a computer vision-based automatic detection system for garment structural defects based on multimodal AI perception, comprising: The perception acquisition module is used to acquire clothing perception data of at least two modalities, and extract multi-level feature maps of each modality through a multi-scale feature pyramid network. The multi-level feature maps contain semantic information with different spatial resolutions. The cross-modal fusion module is used to input multi-level feature maps of each modality into the inter-modal cross-attention interaction network. It uses the features of one modality as the query and the features of the other modalities as the keys and values ​​to perform cross-attention calculation, generate semantically aligned enhanced feature representations between modalities, and fuse the enhanced feature representations of each modality layer by layer at the same scale to obtain a multi-modal joint feature map. The defect detection module is used to input the multimodal joint feature map into a defect detection network based on a fully convolutional architecture, and output a defect existence score map for each spatial location through dense prediction. The defect existence score map represents the confidence level of each pixel belonging to a preset defect category. The uncertainty assessment module is used to enable the random deactivation layer in the defect detection network during the testing phase, perform multiple forward inferences on the same input to obtain multiple defect existence score maps, calculate the pixel-level mean of multiple score maps as the final defect detection result, calculate the pixel-level variance of multiple score maps to generate an uncertainty heatmap, and mark the areas that need to be manually reviewed based on the uncertainty heatmap.

[0014] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described automatic computer vision detection method for clothing structural defects based on multimodal AI perception.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described automatic computer vision detection method for clothing structural defects based on multimodal AI perception.

[0016] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: This application overcomes the technical bottlenecks of traditional multimodal fusion and detection by employing a collaborative design of intermodal cross-attention interaction network and Monte Carlo uncertainty quantification assessment. First, a bidirectional intermodal cross-attention mechanism is introduced, using features of different modalities as queries and keys to achieve bidirectional flow and complementarity of information between modalities. This effectively solves the semantic confusion and noise pollution problems caused by simple splicing, fully leveraging the complementary advantages of each modality in defect representation. Second, a multiple inference framework based on Monte Carlo random deactivation is adopted. By simulating random perturbations of model parameters, the statistical distribution of detection results is obtained. Variance quantifies the cognitive and accidental uncertainties at the pixel level, achieving accurate assessment of the reliability of detection results. Furthermore, the system links uncertainty heatmaps with automatic verification strategies, automatically triggering manual intervention for high-uncertainty samples and high-uncertainty regions, constructing a closed-loop quality inspection system with human-machine collaboration. In summary, this application significantly improves the detection accuracy and reliability of a multimodal garment defect detection system under complex working conditions through the collaborative mechanism of intermodal cross-attention interaction and uncertainty quantification assessment. Attached Figure Description

[0017] Figure 1 is an exemplary flowchart of a computer vision-based automatic detection method for garment structural defects based on multimodal AI perception, according to some embodiments of this application. Figure 2 is a schematic diagram of the architecture of an intermodal cross-attention interaction network according to some embodiments of this application; Figure 3 is a schematic diagram of the Monte Carlo uncertainty quantification evaluation process according to some embodiments of this application; Figure 4 is a schematic diagram of the structure of a computer vision automatic detection system for clothing structural defects based on multimodal AI perception, according to some embodiments of this application. Figure 5 is a schematic diagram of the structure of a computer device for implementing the automatic detection method according to some embodiments of this application. Detailed Implementation

[0018] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] refer to Figure 1 This figure is an exemplary flowchart of a computer vision-based automatic detection method for garment structural defects based on multimodal AI perception, according to some embodiments of this application. The method mainly includes the following steps: In step S101: acquire clothing perception data of at least two modalities, and extract multi-level feature maps of each modality through a multi-scale feature pyramid network.

[0020] The multimodal approach in this application specifically refers to at least two sensing modes with different physical properties, such as two-dimensional visible light images, three-dimensional structured light depth data, and short-wave infrared spectral images. Taking an actual garment production line as an example, a multi-sensor array consisting of a high-speed line scan camera, a structured light scanner, and a short-wave infrared camera is deployed after the sewing process: the two-dimensional visible light mode acquires color texture images of the garment fabric with a resolution of not less than 4096×2048; the three-dimensional structured light mode simultaneously acquires depth point clouds of the same area with a point spacing of not more than 0.2 mm; the short-wave infrared mode detects the difference in reflection / absorption of short-wave infrared light in the sewing stitch area, and utilizes the semi-transparent properties of the fabric in this wavelength band to penetrate the surface and capture the latent features of internal stitch defects.

[0021] The raw data for each modality are preprocessed: visible light images undergo gamma correction to compensate for fluctuations in production line lighting; 3D point clouds are spatially registered with visible light images using an iterative nearest-point algorithm; and shortwave infrared images are normalized to the 0-1 range after blackbody correction. Then, the data for each modality enters a feature extraction channel within a shared backbone architecture: a ResNet-50 pre-trained on a large industrial image dataset is used as the backbone network. Feature maps at different spatial resolutions (1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the corresponding modal's original input image, respectively) are extracted from the ResNet-50 at four residual stages and fed into a feature pyramid network. Through a top-down upsampling fusion path and lateral connections, a multi-level feature map set with a uniform scale for each modality is obtained. Each level of feature map simultaneously contains local details and high-level semantic information at that scale.

[0022] In step S102: The multi-level feature maps of each modality are input into the inter-modal cross-attention interaction network. Cross-attention calculation is performed using the features of one modality as the query and the features of the other modalities as the keys and values ​​to generate semantically aligned enhanced feature representations between modalities. The enhanced feature representations of each modality are then fused layer by layer at the same scale to obtain a multi-modal joint feature map. refer to Figure 2 The core idea of ​​the intermodal cross-attention interaction network is to achieve bidirectional information interaction between features of different modalities. Specifically, during the training phase, a set of prototype vectors with fixed dimensions are randomly initialized for each defect category (such as holes, broken needles, skipped stitches, color difference, size deviation, etc.) in the preset defect categories. These prototype vectors are gradually learned as the semantic centers of each defect category through backpropagation during the training process.

[0023] For each scale of the feature pyramid, perform the following: Using the feature map of the visible light mode at this scale as the query, and using the feature maps of the corresponding scales of the three-dimensional structured light mode and the short-wave infrared mode as the keys and values ​​respectively, the first set of cross-attention weights is calculated; Then, using the feature map of the three-dimensional structured light mode at this scale as the query, and using the feature maps of the visible light mode and the short-wave infrared mode at the corresponding scales as the keys and values, respectively, the second set of cross-attention weights is calculated. The first set of cross-attention weights is multiplied by the feature maps of the three-dimensional structured light mode and the short-wave infrared mode respectively and then added together. The second set of cross-attention weights is multiplied by the feature maps of the visible light mode and the short-wave infrared mode respectively and then added together. The two sets of results are then fused to obtain the cross-modal interaction features at this scale. The cosine similarity between cross-modal interaction features and category prototype query vectors is calculated, and the category attention weights are obtained after Softmax normalization. We use category attention weights to weight and converge cross-modal interaction features, and then add them to the original features through residual connections to generate an enhanced feature representation at this scale.

[0024] The above interaction process is repeated for each level of the feature pyramid. Finally, the enhanced feature representations of all scales are concatenated in the channel dimension and reduced to a fixed number of channels through a 1×1 convolutional fusion layer to obtain the multimodal joint feature map.

[0025] In step S103: The multimodal joint feature map is input into a defect detection network based on a fully convolutional architecture, and the defect presence score map at each spatial location is output through dense prediction.

[0026] This application employs a decoder-led fully convolutional defect detection network, which utilizes the multi-scale, multi-modal joint feature map output in step S102 to gradually restore spatial resolution through skip connections. The multi-modal joint feature maps at different scales of the feature pyramid are input into the corresponding layers of the decoder, and cross-scale fusion and refinement are performed through upsampling blocks and convolutional layers. After refinement through multiple upsampling blocks and convolutional layers, the original input resolution is gradually restored. A 1×1 convolutional layer and a sigmoid activation function are added to the last layer of the decoder to output the defect presence score map, where each pixel value is between 0 and 1, representing the confidence level that the pixel contains a defect.

[0027] During the training phase, the network employs a combination of weighted binary cross-entropy loss and Dice loss for supervision to address the severe imbalance between defective and background pixels. Each sample in the training data has a pixel-level defect annotation mask.

[0028] In step S104: During the testing phase, a random deactivation layer is enabled in the defect detection network. Multiple forward inferences are performed on the same input to obtain multiple defect existence score maps. The pixel-level mean of the multiple score maps is calculated as the final defect detection result. The pixel-level variance of the multiple score maps is calculated to generate an uncertainty heatmap. Based on the uncertainty heatmap, areas requiring manual review are marked. refer to Figure 3 This application employs Monte Carlo random deactivation as an approximate Bayesian inference method to quantify the uncertainty of model predictions. Specifically, a random deactivation mechanism is enabled after all convolutional layers of the defect detection network, with a deactivation probability set to 0.2. During the testing phase, the randomly deactivated layers are kept active.

[0029] The same multimodal joint feature map is input into a defect detection network with random deactivation enabled, and 30 independent forward inferences are performed. During each inference, neurons within the network are randomly discarded according to a preset deactivation probability, resulting in 30 sampled defect existence score maps. The mean and variance of the 30 sampled results at each pixel location are calculated. The mean is used as the final defect existence score at that location, and the variance is used as the uncertainty estimate at that location, generating an uncertainty heatmap.

[0030] For each pixel location, if its uncertainty estimate exceeds a preset first threshold of 0.15, the location is marked as a high uncertainty region; if the average uncertainty of the entire image exceeds a preset second threshold of 0.08, the sample is marked as a sample requiring manual review.

[0031] Furthermore, the bounding box coordinates, defect category prediction results, and uncertainty values ​​of the high-uncertainty region are associated and stored; a review report containing the original multimodal image, defect detection results, and uncertainty heatmap is generated; and the review report is pushed to the manual review terminal for quality inspectors to confirm and correct.

[0032] In addition, the system can set a defect confidence threshold of 0.5. For areas with a final defect existence score higher than 0.5 and an uncertainty lower than 0.15, the system will directly identify them as defective areas and output the detection results. For areas with a final defect existence score higher than 0.5 but an uncertainty higher than 0.15, the system will mark them as suspicious defective areas and push them to the manual review terminal. For areas with a final defect existence score lower than or equal to 0.5 and an uncertainty lower than 0.15, the system will directly identify them as non-defective areas. For areas with a final defect existence score lower than or equal to 0.5 but an uncertainty higher than 0.15, the system will mark them as suspicious non-defective areas and push them to the manual review terminal.

[0033] On the other hand, in some embodiments, this application provides a computer vision-based automatic detection system for garment structural defects based on multimodal AI perception, referencing... Figure 4 The figure is a schematic diagram of a computer vision automatic detection system for garment structural defects based on multimodal AI perception, according to some embodiments of this application. The system includes: a perception acquisition module 401, a cross-modal fusion module 402, a defect detection module 403, and an uncertainty assessment module 404, which are described below: The perception acquisition module 401 in this application is mainly used to acquire clothing perception data of at least two modalities. It extracts multi-level feature maps of each modality through a multi-scale feature pyramid network. The multi-level feature maps contain semantic information with different spatial resolutions. The cross-modal fusion module 402 in this application is mainly used to input the multi-level feature maps of each modality into the cross-modal attention interaction network, use the feature of one modality as the query and the features of the other modalities as the key and value to perform cross-attention calculation, generate the enhanced feature representation with semantic alignment between modalities, and fuse the enhanced feature representations of each modality layer by layer at the same scale to obtain the multi-modal joint feature map; In addition, the defect detection module 403 in this application is mainly used to input the multimodal joint feature map into a defect detection network based on a fully convolutional architecture, and output the defect existence score map of each spatial location through dense prediction. The defect existence score map represents the confidence level of each pixel belonging to a preset defect category. In addition, the uncertainty assessment module 404 in this application is mainly used to enable the random deactivation layer in the defect detection network during the testing phase, perform multiple forward inferences on the same input to obtain multiple defect existence score maps, calculate the pixel-level mean of multiple score maps as the final defect detection result, calculate the pixel-level variance of multiple score maps to generate an uncertainty heat map, and mark the areas that need to be manually reviewed based on the uncertainty heat map.

[0034] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described computer vision automatic detection method for clothing structural defects based on multimodal AI perception.

[0035] In some embodiments, reference Figure 5 The figure is a schematic diagram of the structure of a computer device implementing a computer vision-based automatic detection method for clothing structural defects based on multimodal AI perception, according to some embodiments of this application. The computer vision-based automatic detection method for clothing structural defects based on multimodal AI perception in the above embodiments can... Figure 5 The computer device shown is used to implement this, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503, and at least one communication interface 504.

[0036] Processor 501 can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0037] The communication bus 502 can be used to transmit information between the aforementioned components.

[0038] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 503 may exist independently and be connected to processor 501 via communication bus 502. Memory 503 may also be integrated with processor 501.

[0039] The memory 503 stores program code for executing the scheme of this application, and its execution is controlled by the processor 501. The processor 501 executes the program code stored in the memory 503. The program code may include one or more software modules. In the above embodiments, the computer vision automatic detection method for clothing structural defects based on multimodal AI perception can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0040] Communication interface 504 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0041] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single CPU) processor or a multi-core (multi CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0042] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0043] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described automatic computer vision detection method for clothing structural defects based on multimodal AI perception.

[0044] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0045] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A computer vision-based automatic detection method for garment structural defects based on multimodal AI perception, characterized in that, Includes the following steps: Acquire clothing perception data of at least two modalities, and extract multi-level feature maps of each modality through a multi-scale feature pyramid network. The multi-level feature maps contain semantic information with different spatial resolutions. Multi-level feature maps of each modality are input into the inter-modal cross-attention interaction network. The cross-attention calculation is performed with the feature of one modality as the query and the features of the other modalities as the key and value to generate semantically aligned enhanced feature representations between modalities. The enhanced feature representations of each modality are then fused layer by layer at the same scale to obtain a multi-modal joint feature map. The multimodal joint feature map is input into a defect detection network based on a fully convolutional architecture, and a defect existence score map for each spatial location is output through dense prediction. The defect existence score map represents the confidence level of each pixel belonging to a preset defect category. During the testing phase, a random deactivation layer is enabled in the defect detection network. Multiple forward inferences are performed on the same input to obtain multiple defect existence score maps. The pixel-level mean of multiple score maps is calculated as the final defect detection result. The pixel-level variance of multiple score maps is calculated to generate an uncertainty heatmap. Based on the uncertainty heatmap, areas that require manual review are marked.

2. The method as described in claim 1, characterized in that, Acquire clothing perception data for at least two modalities, and extract multi-level feature maps for each modality using a multi-scale feature pyramid network. Specifically, this includes: For each mode, the corresponding raw sensing data is collected; among them, the two-dimensional visible light mode collects high-resolution visible light images of clothing fabrics, the three-dimensional structured light mode collects depth point cloud data of clothing pieces, and the short-wave infrared mode collects short-wave infrared spectral images of the sewing area. One or more data samples can be collected in each modality; The collected raw sensing data is preprocessed, including illumination correction, spatial registration, and scale normalization. Initial features are extracted by a pre-trained convolutional neural network backbone network and fed into a feature pyramid network. Through top-down path enhancement and lateral connections, feature maps of multiple scales are output and combined into the multi-level feature map.

3. The method as described in claim 1, characterized in that, Multi-level feature maps from each modality are input into an inter-modal cross-attention interaction network. Using features from one modality as the query and features from the remaining modalities as keys and values, cross-attention computation is performed to generate semantically aligned enhanced feature representations between modalities. These enhanced feature representations from each modality are then fused layer by layer at the same scale to obtain a multi-modal joint feature map. Specifically, this includes: For each of the preset defect categories, initialize a set of learnable category prototype query vectors, wherein the defect category includes at least two of the following: fabric defects, sewing defects, size deviations, and structural asymmetry. For each scale of the feature pyramid, cross-attention is performed using features from at least two modalities as queries and features from the remaining modalities as keys and values. Using the feature map of the visible light mode at this scale as the query, and the feature maps of the corresponding scales of the three-dimensional structured light mode and the short-wave infrared mode as the keys and values ​​respectively, the first set of cross-attention weights is calculated; then using the feature map of the three-dimensional structured light mode at this scale as the query, and the feature maps of the corresponding scales of the visible light mode and the short-wave infrared mode as the keys and values ​​respectively, the second set of cross-attention weights is calculated. The first set of cross-attention weights is multiplied and summed with the feature maps of the 3D structured light mode and the short-wave infrared mode, respectively. The second set of cross-attention weights is multiplied and summed with the feature maps of the visible light mode and the short-wave infrared mode, respectively. The two sets of results are then fused to obtain the cross-modal interaction features at this scale. The similarity between the cross-modal interaction features and the category prototype query vector is calculated to obtain the category attention weights. The cross-modal interaction features are weighted and converged using category attention weights, and then added to the original features through residual connections to generate the enhanced feature representation; The enhanced feature representations at each scale are concatenated along the channel dimension and then dimensionality reduced using a convolutional fusion layer to obtain the multimodal joint feature map.

4. The method as described in claim 1, characterized in that, The multimodal joint feature map is input into a defect detection network based on a fully convolutional architecture, and the defect presence score map for each spatial location is output through dense prediction. Specifically, this includes: A decoder-led fully convolutional defect detection network is constructed, wherein the decoder fuses features at different scales through skip connections and gradually restores spatial resolution; The multi-scale, multi-modal joint feature maps are input into the corresponding layers of the decoder, and cross-scale fusion and refinement are performed through upsampling blocks and convolutional layers. After refining through multiple upsampling blocks and convolutional layers, the original input resolution is gradually restored. A 1×1 convolutional layer and a sigmoid activation function are added to the last layer of the decoder to output the defect presence score map, where each pixel value is between 0 and 1, representing the confidence level that the pixel has a defect.

5. The method as described in claim 1, characterized in that, During the testing phase, a random deactivation layer is enabled in the defect detection network. Multiple forward inferences are performed on the same input to obtain multiple defect presence score maps. The pixel-level mean of these score maps is calculated as the final defect detection result. The pixel-level variance of these score maps is calculated to generate an uncertainty heatmap. Based on this uncertainty heatmap, areas requiring manual review are marked, specifically including: During the testing phase, a random deactivation layer was enabled after all convolutional layers of the defect detection network, and the deactivation probability was set. Multiple forward inferences were performed on the multimodal joint feature map of the same input, and different neurons were randomly deactivated each time to obtain the sampling results of multiple defect existence score maps. Calculate the mean and variance of multiple sampling results at each pixel location, use the mean as the final defect presence score at that location, and use the variance as the uncertainty estimate at that location to generate an uncertainty heatmap; For each pixel location, if its uncertainty estimate exceeds a preset first threshold, the location is marked as a high uncertainty region; if the average uncertainty of the entire image exceeds a preset second threshold, the sample is marked as a sample requiring manual review.

6. The method as described in claim 5, characterized in that, The areas requiring manual verification based on the uncertainty-based heatmap marking also include: The bounding box coordinates, defect category prediction results, and uncertainty values ​​of high-uncertainty regions are associated and stored. Generate a review report containing the original multimodal images, defect detection results, and uncertainty heatmap; push the review report to the manual review terminal for quality inspectors to confirm and correct.

7. The method as described in claim 1, characterized in that, Also includes: Based on the final defect detection results and uncertainty heatmap, samples with uncertainty exceeding the preset alarm threshold are automatically marked as samples requiring manual review, and the multimodal raw data and detection results of the sample are pushed to the manual review terminal.

8. A computer vision-based automatic detection system for garment structural defects based on multimodal AI perception, characterized in that, include: The perception acquisition module is used to acquire clothing perception data of at least two modalities, and extract multi-level feature maps of each modality through a multi-scale feature pyramid network. The multi-level feature maps contain semantic information with different spatial resolutions. The cross-modal fusion module is used to input multi-level feature maps of each modality into the inter-modal cross-attention interaction network. It uses the features of one modality as the query and the features of the other modalities as the keys and values ​​to perform cross-attention calculation, generate semantically aligned enhanced feature representations between modalities, and fuse the enhanced feature representations of each modality layer by layer at the same scale to obtain a multi-modal joint feature map. The defect detection module is used to input the multimodal joint feature map into a defect detection network based on a fully convolutional architecture, and output a defect existence score map for each spatial location through dense prediction. The defect existence score map represents the confidence level of each pixel belonging to a preset defect category. The uncertainty assessment module is used to enable the random deactivation layer in the defect detection network during the testing phase, perform multiple forward inferences on the same input to obtain multiple defect existence score maps, calculate the pixel-level mean of multiple score maps as the final defect detection result, calculate the pixel-level variance of multiple score maps to generate an uncertainty heatmap, and mark the areas that need to be manually reviewed based on the uncertainty heatmap.

9. A computer device comprising a memory and a processor, the memory storing code, characterized in that, The processor is configured to acquire the code and execute the computer vision automatic detection method for clothing structural defects based on multimodal AI perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the automatic computer vision detection method for clothing structural defects based on multimodal AI perception as described in any one of claims 1 to 7.

Images