A defect detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application relate to the field of visual detection, and disclose a defect detection method and device, electronic equipment and a storage medium, which comprise: acquiring a to-be-detected image containing a target object; performing inference on the to-be-detected image based on a single-stage target detection network to obtain a candidate defect region represented by a bounding box; obtaining a visual feature map of the to-be-detected image based on a visual coding branch of a visual multi-modal large model; mapping the candidate defect region to a coordinate system of the visual feature map, and extracting semantic features corresponding to the candidate defect region in the visual feature map; and performing defect category determination and bounding box regression on the candidate defect region based on the semantic features corresponding to the candidate defect region. The present application solves the problems of limited positioning accuracy, insufficient fine-grained discrimination ability, insufficient generalization ability and high maintenance cost of the defect detection scheme, and at least achieves the effect of balancing detection speed, positioning accuracy, fine-grained discrimination ability and engineering deployability.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of visual inspection, and particularly to a defect detection method, apparatus, electronic device and storage medium. Background Technology

[0002] In the manufacturing and operational support of slender and flexible workpieces such as cables, wire harnesses, and wire ropes, it is often necessary to automatically detect and classify surface defects (including ablation, wear, fracture, indentation, foreign matter adhesion, and sheath damage) online to support quality control and safety early warning. Existing industrial vision inspection solutions include the following categories: rule-based machine vision inspection methods, single-stage deep learning object detection methods, and two-stage object detection methods. With the development of Visual-Language Models (VLM) / Large Vision-Language Models (LVLM), a new architecture has emerged that deeply integrates visual encoders with large-scale language models (such as the Qwen2-VL series). These models can jointly understand local textures, crack morphology, ablation marks on material surfaces, and character / imprint information (OCR (Optical Character Recognition) level details) on a single high-resolution image, possessing contextual analysis and semantic interpretation capabilities approaching those of human inspectors. However, there are still significant obstacles to directly applying multimodal large models to online industrial defect detection.

[0003] In summary, the existing technology has the following main shortcomings: (1) Positioning accuracy is limited While existing single-stage real-time target detection networks (such as the YOLO series) offer high inference speeds, when dealing with small, finely shaped surface defects, the bounding boxes they regress often suffer from positional offsets or excessively large bounding boxes, making it difficult to accurately fit the defect area. Such detection results fall short of the millimeter-level precision required for defect location in industrial quality inspection scenarios.

[0004] (2) Insufficient ability to distinguish fine-grained defects Traditional detection networks primarily rely on convolutional features or lightweight classification heads for category determination, which struggles to fully express high-semantic information such as differences in material surface texture and thermal damage morphology. Therefore, existing methods often fail to reliably distinguish between defects that appear similar but exhibit significant differences in process risk levels (e.g., mild wear versus high-risk ablation / discharge breakdown marks), leading to misclassification or underreporting of high-risk defects.

[0005] (3) Insufficient generalization ability and high maintenance cost Defect morphology on industrial production lines exhibits continuous evolution, with new defect patterns constantly emerging depending on production conditions, material batches, and external environments (such as overheating, pressure, and corrosion). Traditional single-stage detectors typically require retraining or iterative fine-tuning of the detection head and even the backbone network when introducing new defect categories. This increases data annotation and training costs and also risks interfering with existing converged categories. Furthermore, while multimodal visual language models possess strong semantic discrimination capabilities, their complete inference process is computationally intensive and structurally complex, making direct deployment as online detectors in real-time on production lines extremely difficult.

[0006] (4) It is difficult to simultaneously satisfy real-time performance and high semantic judgment. Current industrial detection solutions lack a unified, engineering-feasible workflow that can simultaneously achieve two capabilities within the same inference chain: firstly, high-recall, rapid candidate region generation for the entire image to meet the throughput requirements of real-time online detection; and secondly, high-semantic-level fine-grained judgment and localization of candidate regions to achieve reliable classification and bounding box quality. In other words, existing methods have not yet effectively achieved the organic coupling of "real-time candidate box generation" and "high-semantic-level fine-grained decision-making." Summary of the Invention

[0007] The purpose of this invention is to provide at least one defect detection method, device, electronic device, and storage medium, which can at least solve the problems of limited positioning accuracy, insufficient fine-grained discrimination ability, insufficient generalization ability, and high maintenance cost of defect detection schemes in related technologies, and can at least achieve the effect of balancing detection speed, positioning accuracy, fine-grained discrimination ability, and engineering deployability in defect detection schemes.

[0008] To address the aforementioned technical problems, at least one embodiment of this application provides a defect detection method, comprising: acquiring an image to be detected containing a target object; reasoning on the image to be detected based on a single-stage object detection network to obtain candidate defect regions represented by bounding boxes; obtaining a visual feature map of the image to be detected based on a visual encoding branch of a visual multimodal large model; mapping the candidate defect regions to the coordinate system of the visual feature map, and extracting semantic features corresponding to the candidate defect regions from the visual feature map; and performing defect category determination and bounding box regression on the candidate defect regions based on the semantic features corresponding to the candidate defect regions.

[0009] At least one embodiment of this application also provides a defect detection device, comprising: an image acquisition module for acquiring an image to be detected containing a target object; a candidate box generation module for reasoning on the image to be detected based on a single-stage object detection network to obtain candidate defect regions represented by bounding boxes; a visual encoding module for obtaining a visual feature map of the image to be detected based on a visual encoding branch of a visual multimodal large model; a semantic feature extraction module for mapping the candidate defect regions to the coordinate system of the visual feature map and extracting semantic features corresponding to the candidate defect regions from the visual feature map; and a defect refinement module for performing defect category determination and bounding box regression on the candidate defect regions based on the semantic features corresponding to the candidate defect regions.

[0010] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the defect detection method described above.

[0011] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the defect detection method described above.

[0012] The defect detection method, apparatus, electronic device, and storage medium provided in the embodiments of this application infer candidate defect regions represented by bounding boxes from an image to be detected containing a target object based on a single-stage target detection network; obtain a visual feature map of the image to be detected based on the visual coding branch of a visual multimodal large model; map the candidate defect regions to the coordinate system of the visual feature map and extract the semantic features corresponding to the candidate defect regions from the visual feature map; and perform defect category determination and bounding box regression based on the semantic features corresponding to the candidate defect regions. The embodiments of this application employ a phased, cross-model paradigm defect detection approach. The first phase uses a lightweight, real-time single-stage target detection network to rapidly scan the entire image to be detected, solely responsible for generating candidate defect regions to ensure high recall and meet the latency requirements of online production lines. The second phase, without altering the backbone parameters of the visual multimodal large model, calls upon its visual encoding results to perform fine-grained defect category determination and bounding box refinement on each candidate defect region output from the first phase. The two phases are connected through geometric coordinate mapping and region alignment operations, forming a cascaded detection link of "rapid candidate → semantic refinement," which balances detection speed, positioning accuracy, fine-grained discrimination capability, and engineering deployability.

[0013] In some optional embodiments, mapping the candidate defect region to the coordinate system of the visual feature map and extracting the semantic features corresponding to the candidate defect region from the visual feature map includes: mapping the candidate defect region to the input coordinate system of the visual coding branch to obtain a first mapping result; mapping the first mapping result to the coordinate system of the visual feature map to obtain a second mapping result; and extracting the semantic features of the second mapping result as the semantic features corresponding to the candidate defect region.

[0014] In some optional embodiments, based on the semantic features corresponding to the candidate defect region, defect category determination and bounding box regression are performed on the candidate defect region, including: using a pre-trained defect refinement model to process the semantic features corresponding to the candidate defect region to obtain the defect category and bounding box regression results of the candidate defect region; wherein, the defect refinement model includes a shared projection layer, a classification branch and a bounding box regression branch, the shared projection layer is used to project the semantic features corresponding to the candidate defect region onto an intermediate dimension to obtain a shared projection result; the classification branch is used to project the shared projection result onto a defect category space of a target dimension to determine the defect category, and the bounding box regression branch is used to obtain the regression offset of the candidate defect region based on the shared projection result, and obtain the corresponding target defect region based on the regression offset.

[0015] In some optional embodiments, the shared projection result expression is as follows:

[0016] The expression for the defect category determination result is as follows:

[0017]

[0018] The regression offset expression for the candidate defect region is as follows:

[0019] in, It is a shared projection result; , , These are trainable parameters; It is the first Semantic feature vectors corresponding to each candidate defect region; It is any defect category; It includes all defect categories; It is the number of defect categories; Indicates the first Each candidate defect region belongs to category c The probability of; This represents the regression offset of the candidate defect region.

[0020] In some optional embodiments, the defect detection method further includes: training or fine-tuning a single-stage detection network and a defect refinement model while freezing the parameters of the visual multimodal large model.

[0021] In some optional embodiments, the defect detection method further includes: outputting defect detection results after defect category determination and bounding box regression, wherein the defect detection results include the coordinates of the target bounding box after bounding box regression, the defect category label, and the corresponding confidence score.

[0022] In some optional embodiments, the defect detection method further includes: performing post-processing operations on the target defect regions obtained by bounding box regression, wherein the post-processing operations include: deleting target defect regions with confidence scores lower than a first set threshold, and retaining the target defect regions with the highest confidence scores among target defect regions of the same defect category with intersection-union ratios higher than a second set threshold. Attached Figure Description

[0023] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0024] Figure 1 This is a flowchart of a defect detection method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating an application example of the defect detection method provided in the embodiments of this application; Figure 3 This is a schematic diagram of a defect detection device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0026] To facilitate understanding of the embodiments of this application, relevant content on industrial visual inspection will be introduced first.

[0027] In the manufacturing and operational support of slender, flexible workpieces such as cables, wire harnesses, and wire ropes, online automatic detection and classification of surface defects (including ablation, wear, fracture, indentation, foreign matter adhesion, and sheath damage) are typically required to support quality control and safety early warning. Among related technologies, industrial vision inspection solutions can be mainly divided into the following three categories.

[0028] (1) Rule-based machine vision detection method Basic principle: This type of method relies on a fixed image processing flow: First, a linear or area array industrial camera is used to image the surface of the workpiece being tested; then, the image is processed by classic visual operators such as threshold segmentation, edge detection, template matching, and geometric measurement to identify abnormal areas such as bright / dark spots, abrupt contour changes, and structural fractures.

[0029] The main limitations include: a. Limited generalization ability. Such methods rely on manually set thresholds, filter kernels, and geometric rules, so they can often only detect predefined defect patterns. When faced with novel defects or complex working conditions (such as different materials or different surface textures), the detection performance drops significantly.

[0030] b. Insufficient robustness. This type of detection method is highly sensitive to lighting conditions, surface reflectivity, noise, and background texture, and is prone to false alarms or missed detections.

[0031] c. High migration costs. To adapt to different production lines, cable models, or shooting lighting conditions, it is often necessary to readjust or rewrite the rules. The parameter tuning process relies on experienced engineers, which is time-consuming and has high maintenance costs.

[0032] (2) Single-stage deep learning object detection method Basic Principle: In recent years, single-stage object detection networks (such as the YOLOv5, YOLOv8, YOLOv9 series models; YOLO stands for You Only Look Once) have been applied in various industrial vision scenarios, including metal surface scratch detection, solder joint anomaly detection, and cable surface ablation trace detection. These methods extract features from the entire image through an end-to-end convolutional / Transformer backbone network and directly regress the bounding box coordinates and corresponding class probability distribution of the target directly on the feature map, thereby obtaining candidate defect regions and their class confidence in a single forward inference.

[0033] The main limitations include: a. Limited accuracy in locating small-scale defects. For targets with extremely small volume and indistinct boundaries (such as tiny ablation points, pinhole-level damage, or minor local abrasions on cable sheaths), single-stage detectors often provide oversized bounding boxes or exhibit significant center offsets, making it difficult to meet the industrial quality inspection requirements for high-precision quantification of defect ranges.

[0034] b. Difficulty in distinguishing fine-grained categories. Single-stage detectors typically perform classification directly on the backbone feature map, making it difficult to reliably differentiate defect types that are highly similar in appearance but have significantly different process risks. For example, minor wear and high-risk high-temperature ablation / discharge breakdown marks may appear as similar bright / dark spot textures on a grayscale background or monochrome sheath material, making them prone to misclassification.

[0035] c. Expanding to new defect categories is costly. When new defect patterns appear on the production line, but the sample size is often very small (e.g., only a few dozen sample images are collected), directly retraining or fine-tuning the entire single-stage detection model can easily lead to underfitting or catastrophic forgetting of existing defect categories, resulting in high maintenance costs.

[0036] (3) Two-stage object detection methods (typical examples include Faster R-CNN, Faster Region-based Convolutional Neural Network). Basic principle: A two-stage detection framework typically includes two steps: The first step is for the Region Proposal Network (RPN) to generate a series of potential target regions (candidate boxes, proposals) across the entire image. The second step is to perform ROI Pooling (Region of Interest Pooling) or ROI Align (Region of Interest Align) on each candidate box, extract the local features of the candidate region and standardize them into a fixed-size feature representation, which is then fed into the classification branch and the bounding box regression branch to achieve category determination and fine localization correction.

[0037] This type of method is known for its high target localization accuracy and robust small target detection capability in traditional visual detection tasks.

[0038] The main limitations include: a. High training cost. Two-stage frameworks typically require end-to-end joint optimization of the RPN (Region Proposal Network) and subsequent classification / regression heads, which is highly dependent on the number of high-quality labeled samples, and the labeling cost is often high in industrial scenarios.

[0039] b. Limited transferability. RPN and subsequent classification heads are typically trained for specific target categories. When a completely new defect pattern is introduced into the production line, retraining or significant fine-tuning of the detection pipeline is often required, and the transfer efficiency remains unsatisfactory.

[0040] c. It is difficult to directly utilize the semantic representation capabilities of general multimodal models. Traditional two-stage detectors mainly rely on convolutional or visual Transformer-style features, lacking multimodal cross-domain semantic understanding capabilities, and making it difficult to reuse the prior knowledge of large-scale pre-trained visual-language models in recent years regarding material properties, texture semantics, and damage type interpretation.

[0041] In related technologies, the application of multimodal vision-language models in industrial inspection scenarios has also been explored. With the development of Visual-Language Models (VLM) / Large Vision-Language Models (LVLM), a class of architectures that deeply integrate visual encoders with large-scale language models has emerged (such as the Qwen2-VL series). These models can jointly understand local textures, crack morphology, ablation marks on material surfaces, and character / engraving information (OCR (Optical Character Recognition) level details) on a single high-resolution image, possessing contextual analysis and semantic interpretation capabilities approaching those of human inspectors.

[0042] Basic Mechanism: This type of model typically divides the input image into visual patches with fixed step sizes and encodes these patches as high-dimensional visual tokens. These visual tokens, along with text tokens, are input into a unified cross-modal model structure. During the training phase, the model learns rich semantic priors through large-scale cross-visual and language alignment tasks (description generation, region understanding, reasoning question answering, etc.), thus enabling it to identify material anomaly patterns and perform semantic attribution, such as "burn marks," "outward-curving fracture edges," and "exposed metal wire bundles."

[0043] However, there are still significant obstacles to directly applying multimodal large models to online industrial defect detection: a. High computational cost. The inference overhead of this type of model is much higher than that of classical detectors. Performing region-by-region multimodal inference directly on the entire image will be difficult to meet the real-time requirements (millisecond to tens of milliseconds response) of high-speed production lines.

[0044] b. The output format does not directly match the needs of industrial quality inspection. Multimodal large models are usually based on natural language descriptions, question-and-answer outputs, or regional indicators, lacking the ability to perform high-precision, engineering-usable bounding box (location, width, and height) regression for small-scale defects. In other words, it can "understand" defects and describe their properties, but it does not naturally provide the "specific coordinates + dimensions" format required for industrial inspection.

[0045] c. End-to-end retraining is extremely costly. Transforming such large multimodal models directly into complete detectors (i.e., generating accurate bounding boxes and categories from the original image end-to-end) requires large-scale supervised training. Since these models typically have an enormous number of parameters (on the order of billions), collecting a sufficient number of labeled bounding boxes in industrial quality inspection scenarios is very difficult, making the training cost often unacceptable in engineering practice.

[0046] In summary, the existing technology has the following main shortcomings: (1) Positioning accuracy is limited While existing single-stage real-time target detection networks (such as the YOLO series) offer high inference speeds, when dealing with small, finely shaped surface defects, the bounding boxes they regress often suffer from positional offsets or excessively large bounding boxes, making it difficult to accurately fit the defect area. Such detection results fall short of the millimeter-level precision required for defect location in industrial quality inspection scenarios.

[0047] (2) Insufficient ability to distinguish fine-grained defects Traditional detection networks primarily rely on convolutional features or lightweight classification heads for category determination, which struggles to fully express high-semantic information such as differences in material surface texture and thermal damage morphology. Therefore, existing methods often fail to reliably distinguish between defects that appear similar but exhibit significant differences in process risk levels (e.g., mild wear versus high-risk ablation / discharge breakdown marks), leading to misclassification or underreporting of high-risk defects.

[0048] (3) Insufficient generalization ability and high maintenance cost Defect morphology on industrial production lines exhibits continuous evolution, with new defect patterns constantly emerging depending on production conditions, material batches, and external environments (such as overheating, pressure, and corrosion). Traditional single-stage detectors typically require retraining or iterative fine-tuning of the detection head and even the backbone network when introducing new defect categories. This increases data annotation and training costs and also risks interfering with existing converged categories. Furthermore, while multimodal visual language models possess strong semantic discrimination capabilities, their complete inference process is computationally intensive and structurally complex, making direct deployment as online detectors in real-time on production lines extremely difficult.

[0049] (4) It is difficult to simultaneously satisfy real-time performance and high semantic judgment. Current industrial detection solutions lack a unified, engineering-feasible workflow that can simultaneously achieve two capabilities within the same inference chain: firstly, high-recall, rapid candidate region generation for the entire image to meet the throughput requirements of real-time online detection; and secondly, high-semantic-level fine-grained judgment and localization of candidate regions to achieve reliable classification and bounding box quality. In other words, existing methods have not yet effectively achieved the organic coupling of "real-time candidate box generation" and "high-semantic-level fine-grained decision-making."

[0050] To address the technical problems of limited positioning accuracy, insufficient fine-grained discrimination ability, insufficient generalization ability, and high maintenance cost in the aforementioned defect detection schemes, this invention proposes a defect detection method. The implementation details of the defect detection method in this embodiment are described below. The following content is only for ease of understanding and is not necessary for implementing this solution.

[0051] Example 1: The defect detection method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes: Step 101: Obtain the image to be detected that contains the target object.

[0052] Specifically, the method in this embodiment is geared towards online appearance quality inspection scenarios for slender workpieces such as cables, wire ropes, and wire harnesses. The target object can be slender workpieces such as cables, wire ropes, and wire harnesses, and the image to be inspected is the original inspection image obtained by imaging the outer surface of the object to be inspected (e.g., slender and flexible workpieces such as cables, wire harnesses, and wire ropes) using an industrial camera.

[0053] In practical implementation, the image to be detected can be a grayscale image or a color image, and the resolution can reach [resolution value missing]. Pixels or higher to ensure that minute defects (such as ablation points, minor sheath damage, broken strands, etc.) are distinguishable at the pixel scale.

[0054] Let the original image to be detected be denoted as . , in, This represents the image to be detected. These are the height and width of the image to be detected, respectively. It represents the set of real numbers, and its superscript indicates the dimension of the real number vector space.

[0055] Step 102: Based on the single-stage object detection network, reason about the image to be detected to obtain candidate defect regions represented by bounding boxes.

[0056] In practical implementation, the single-stage object detection network can adopt the YOLOv model, such as the YOLOv9 model fine-tuned with industrial data. Based on the single-stage object detection network, it performs fast inference on the entire original image I to be detected in real time, and outputs... K There are 10 candidate defect regions (proposals), each represented by a bounding box as follows:

[0057]

[0058] in, Indicates the first k One candidate defect region, They represent the first k The top-left and bottom-right coordinates of each candidate defect region are provided. The single-stage target detection network also provides preliminary category prediction results and confidence scores for each candidate defect region (i.e., coarse-grained judgment, such as "suspected ablation / suspected wear / suspected breakage").

[0059] The goal of this step is to achieve high recall and real-time performance, that is, to cover as many potential defect locations as possible while ensuring that the inference speed meets the requirements of online detection, rather than focusing on the final bounding box accuracy.

[0060] Step 103: Obtain the visual feature map of the image to be detected based on the visual coding branch of the visual multimodal large model.

[0061] In some cases, the visual multimodal large model can be the Qwen2-VL series visual multimodal large model.

[0062] In the specific implementation, based on the visual coding branch of the large visual multimodal model (such as the visual backbone network in the Qwen2-VL series), the entire input image (the original image to be detected I) is modeled with high semantic features as follows: First, the original image to be detected, I, is normalized in size (e.g., scaled proportionally and possibly padded) to...

[0063] in, This represents the size-normalized image to be inspected. These represent the standard input sizes of the visual encoding branches of a large visual multimodal model (e.g., ...). The height and width (approximately square resolution).

[0064] Secondly, the visual coding branch (visual encoder) will The image is divided into fixed-step patches, and each patch is encoded as a high-dimensional visual token. Rearranging all visual tokens into a two-dimensional grid yields a high-order visual feature map. :

[0065] in, Representing channel dimension (e.g.) ); Indicates the spatial resolution of the visual token grid (e.g.) The grid size is ); This indicates the position of the input image in the grid. The high-level visual semantic representation can describe local material, ablation trace morphology, fracture texture, etc. u Indicates the horizontal position (column). v Indicates the vertical position (row).

[0066] In this embodiment, the parameters of the visual encoding branch remain frozen, eliminating the need for large-scale retraining for production line scenarios.

[0067] Step 104: Map the candidate defect region to the coordinate system of the visual feature map, and extract the semantic features corresponding to the candidate defect region from the visual feature map.

[0068] In the specific implementation, the bounding boxes B (defined in the original image coordinate system) of the candidate defect regions output by the single-stage object detection network are accurately mapped to the visual feature map. In the coordinate system, and in the visual feature map Local semantic features corresponding to the candidate defect region are extracted.

[0069] In the specific implementation, a fixed-length feature vector f for each candidate defect region is obtained through coordinate scaling, normalization, projection mapping, and region alignment sampling. k It is used for subsequent defect repair.

[0070] This embodiment establishes an explicit mapping relationship from the original image coordinate system to the visual patch-token grid coordinate system within the large visual multimodal model, projecting the candidate defect regions output by the single-stage object detection network onto the visual feature map of the large visual multimodal model. Corresponding regions are aligned and sampled on this high-dimensional semantic feature map to extract local high-semantic feature representations describing the candidate regions. Bounding box refinement regression is then performed based on these representations, thereby refining and correcting the candidate box positions at the millimeter scale, achieving cross-scale coordinate alignment and region-level feature extraction.

[0071] Step 105: Based on the semantic features corresponding to the candidate defect regions, perform defect category determination and bounding box regression on the candidate defect regions.

[0072] In the specific implementation, based on the semantic features corresponding to the candidate defect region, defect category determination and bounding box regression are performed on the candidate defect region. This includes: using a pre-trained defect refinement model to process the semantic features corresponding to the candidate defect region to obtain the defect category and bounding box regression results of the candidate defect region; wherein, the defect refinement model includes a shared projection layer, a classification branch and a bounding box regression branch. The shared projection layer is used to project the semantic features corresponding to the candidate defect region onto an intermediate dimension to obtain a shared projection result; the classification branch is used to project the shared projection result onto the defect category space of the target dimension to determine the defect category; the bounding box regression branch is used to obtain the regression offset of the candidate defect region based on the shared projection result, and obtain the corresponding target defect region based on the regression offset.

[0073] The defect detection method in this embodiment uses a single-stage object detection network as a real-time detection module to provide candidate defect regions represented by bounding boxes that are "highly recalled but potentially inaccurate." These candidate defect regions are projected onto the high-semantic visual feature space of a large visual multimodal model, and then fine-grained classification and precise localization regression are performed within this high-semantic visual feature space. While maintaining the high-speed inference capability of the single-stage real-time object detection network, the high-semantic visual features of the large visual multimodal model are further introduced to perform fine-grained category discrimination and bounding box refinement on the candidate defect regions, thereby obtaining high-precision small target defect detection results and achieving highly timely fine-grained defect detection and precise localization.

[0074] In some embodiments, the candidate defect region is mapped to the coordinate system of the visual feature map, and the semantic features corresponding to the candidate defect region are extracted from the visual feature map, including: Step 201: Map the candidate defect region onto the input coordinate system of the visual coding branch to obtain the first mapping result; Step 202: Map the first mapping result to the coordinate system of the visual feature map to obtain the second mapping result; Step 203: Extract the semantic features of the second mapping result as the semantic features corresponding to the candidate defect region.

[0075] This embodiment obtains a first mapping result by mapping the candidate defect region to the input coordinate system of the visual coding branch; and obtains a second mapping result by mapping the first mapping result to the coordinate system of the visual feature map, thereby realizing multi-scale coordinate mapping of the candidate defect region bounding box and achieving coordinate alignment by mapping the candidate defect region to the coordinate system of the visual feature map.

[0076] In some examples, it is assumed that the original image to be detected is of size 1. The size of the input image after preprocessing by the visual coding branch is Preprocessing can involve scaling proportionally and then padding, so that... .

[0077] Define a linear scaling factor from the original coordinate system to the input coordinate system of the visual encoding branch. , :

[0078] For any candidate defect region bounding box output by a single-stage target detection network :

[0079] Its corresponding bounding box in the input coordinate system of the visual coding branch can be represented as:

[0080] in:

[0081]

[0082] Visual feature map output by the visual encoding branch Spatial resolution To be k Projected to grid coordinate system Further define the downsampling step size , :

[0083] Then the candidate defect region bounding box The coordinates in the visual feature map coordinate system can be written as:

[0084] in:

[0085]

[0086] The above coordinate mapping process realizes the transformation of the candidate defect region bounding box from the original pixel coordinate system. Input coordinate system via visual encoding branch Then, in the visual feature map (grid) coordinate system The system uses a stepwise projection of the visual feature map, which enables the system to... Accurate positioning and The corresponding area provides a foundation for subsequent ROI alignment and refinement.

[0087] In the specific implementation, the ROI Align algorithm is used to extract the semantic features of the second mapping result as the semantic features corresponding to the candidate defect region, thus realizing region alignment and feature extraction.

[0088] In some examples, in visual feature maps Above, corresponding The regions in question are typically not integer grid boundaries. To extract features from these regions without introducing significant quantization errors, the ROI Align algorithm is employed. The basic idea of ​​the ROI Align algorithm is to... Sampling is done using a fixed-size grid, for example... Grid. Regarding this Each cell of the grid Calculate its in The corresponding floating-point coordinates and to Bilinear interpolation is performed on adjacent integer grid points to obtain the feature vector of the cell location, ultimately forming an aligned tensor: ,in, , is the set value; then for Perform global average pooling in the spatial dimension:

[0089] get That is, the first A fixed-length high semantic representation vector for each candidate defect region. Semantically, this vector can be regarded as the "descriptive feature" of the visual multimodal large model backbone for the local defect.

[0090] In some embodiments, the defect refinement model (ROI Refiner Head) includes a shared linear projection layer, as well as classification and bounding box regression branches; each branch contains a normalization layer and a linear prediction layer, the parameters of which can be trained independently, while the backbone of the large visual multimodal model remains frozen. For example, the architecture of the defect refinement model includes five parameterized sublayers: One shared linear layer (shared projection layer): project = nn.LazyLinear(512) Normalization layer for one classification branch: LayerNorm(512) A linear layer with one classification branch (linear prediction layer): Linear(512, num_classes) Normalization layer for one regression branch: LayerNorm(512) A linear layer with one regression branch (linear prediction layer): Linear(512, 4).

[0091] In some examples, the defect refinement model performs lightweight neural network inference on the semantic feature vector f of each candidate defect region. The data flow can be: the output of ROI Align (7×7) is first subjected to global average pooling, then enters a shared projection layer (output dimension is 512), and the output of the shared projection layer then enters the classification branch and the bounding box regression branch respectively, outputting the _th k Probability distribution of fine-grained defect categories for each candidate defect region k and bounding box regression k Used for k The position and size are calibrated with high precision.

[0092] In some embodiments, the above defect detection method further includes: outputting defect detection results after defect category determination and bounding box regression, wherein the defect detection results include the coordinates of the target bounding box after bounding box regression, the defect category label, and the corresponding confidence score. The refined results of all candidate defect regions are fused together to finally output the final category label (fine-grained category), confidence score, and refined bounding box coordinates for each defect.

[0093] In some embodiments, before outputting the defect detection results after defect category determination and bounding box regression, the above-described defect detection method further includes: performing post-processing operations on the target defect regions obtained by bounding box regression. The post-processing operations include: deleting target defect regions with confidence scores below a first preset threshold; and retaining the target defect region with the highest confidence score among target defect regions of the same defect category with an intersection-union ratio (IoU) higher than a second preset threshold. By filtering low-confidence targets and merging overlapping boxes of similar targets (e.g., non-maximum suppression (NMS) based on IoU), the accuracy of the output results is further improved.

[0094] In some embodiments, for high-risk defect categories (such as severe sheath damage, high-temperature ablation, exposed broken strands, etc.) in the defect detection results after defect category determination and bounding box regression, online alarms, rejection, rework orders, or quality traceability records can be directly triggered.

[0095] In some cases, various defect categories can be pre-classified into high, medium, and low risk. Industrial cameras installed on the production line image the outer surface of the object to be inspected (such as cables, wire harnesses, steel wire ropes, and other slender and flexible workpieces) to obtain the original inspection image, which is then uploaded to the cable quality inspection integrated machine to execute the above defect detection method in real time. When a high-risk defect category appears in the defect detection results, feedback is sent to the production line to locate the object to be inspected, thereby triggering online alarms, rejection, rework orders, or quality traceability records.

[0096] This embodiment outputs the final defect category label, confidence score, and refined high-precision bounding box coordinates for each candidate defect region during the inference phase. This enables qualitative (defect type determination) and quantitative (spatial location and morphological range) characterization of defects, providing structured inspection results that can be directly used for industrial decision-making. The output can also be directly integrated with production line alarms, sorting and rejection systems, rework guidelines, and quality traceability systems, supporting automated closed-loop control of online quality inspection.

[0097] In some embodiments, the above-described defect detection method further includes: training or fine-tuning a single-stage detection network and a defect refinement model while freezing the parameters of a large visual multimodal model.

[0098] During the inference phase, the parameters of the backbone network of the large visual multimodal model are frozen, meaning the main body of the large visual multimodal model is not retrained. Supervised training is only performed on the lightweight ROI Refiner Head, which is responsible for classifying candidate defect regions and adjusting their locations. This structure allows for rapid adaptation to new defect types and new production line conditions with less data and lower training overhead, significantly reducing subsequent upgrade and maintenance costs, lowering model iteration costs, and improving transferability.

[0099] In some embodiments, the specific implementation method for determining the defect category by the classification branch (classification head) is as follows: For the Feature vectors of candidate defect regions The classification head will Projected onto intermediate dimension (For example And further projected onto The category space of dimension can be formalized as:

[0100] in: Indicates shared projection results. These are the trainable parameters of the shared projection layer used as a linear dimensionality reduction / feature adaptation layer; This refers to the number of defect categories (e.g., "ablation / discharge damage", "wear", "fracture", "bulge / peeling", etc.). For example, with a value of 3, there are 3 categories of defects, including "scratches, damage, and exposed copper".

[0101] Will The multi-class probability distribution is obtained through the softmax function and used as the defect category determination result. The expression is as follows:

[0102]

[0103] in, It is the first The original classification scores (classification score vectors) of each candidate defect region before softmax normalization. Indicates the first Each candidate defect region belongs to category c The probability of; These are the trainable parameters of the classification layer (classification branch); It is the first The semantic feature vectors corresponding to each candidate defect region; are classification layer parameters; d It is the intermediate dimension; It is any defect category; This includes all defect categories.

[0104] by For example, [2.1, 0.3, -1.4] represents three raw scores for three defect categories, with category 1 having the highest score (2.1) and category 3 having the lowest score (-1.4). After softmax normalization, these scores are converted into probabilities, such as: = [0.82, 0.15, 0.03] It is a length of A vector, where each dimension corresponds to a score for a defect category. Then it means the first k Each candidate defect region belongs to category c The original fraction.

[0105] In some cases, when training the defect refinement model, the first... Each candidate defect region is assigned a label representing the true defect category. (This label is derived from the best match between the manually labeled bounding box and the bounding box of the candidate defect region). The classification loss uses cross-entropy loss:

[0106] This loss enables the defect refinement model to distinguish fine-grained defect categories that are similar in appearance but have different levels of risk.

[0107] In some embodiments, the specific implementation of bounding box regression (refinement) is as follows: To achieve high-precision localization, the bounding box of each candidate defect region is... Learning a regression offset Used to generate refined bounding boxes The bounding box is represented by its center point coordinates and width and height, and its parametric form in the original image coordinate system is defined as follows:

[0108] in:

[0109]

[0110]

[0111]

[0112] Bounding box regression branch (regression head) for the first The semantic feature vector corresponding to each candidate defect region Output four-dimensional continuous quantities:

[0113] in, express x The regression value in the direction, express y The regression value in the direction, The regression value representing the width. The regression value representing the height, Indicates the first k The regression output vector of each candidate defect region.

[0114] No. The regression offset expression for each candidate defect region is as follows:

[0115] in, It is a shared projection result; the above , , These are trainable parameters.

[0116] In the specific implementation, the projection results will be shared. This intermediate feature, shared with the classification head, is input into the regression layer (bounding box regression branch), which undergoes a linear transformation to obtain the aforementioned four-dimensional output vector. .

[0117] For training We need to provide a "true" regression target. Let the first... k The actual bounding boxes corresponding to each candidate defect region are:

[0118] The regression objective is then defined as the proportion-normalized difference, for example:

[0119]

[0120] During training, smooth L1 loss or L1 / L2 loss pairs are used. and Supervision:

[0121] In the reasoning stage, a prediction is obtained. Then, the reversible transformation yields the refined bounding box parameters:

[0122]

[0123]

[0124]

[0125] Then restore it to the top left and bottom right corners:

[0126]

[0127]

[0128]

[0129] Obtain the final refined bounding box

[0130] When training the defect refinement model, the classification head and regression head are jointly optimized, and the total loss is defined as:

[0131] in, This is a weighting coefficient, which can be set according to business metrics (such as classification accuracy, positioning accuracy, etc.). This prompts the model to output the correct fine-grained defect category; This enables the model to output a high-precision bounding box that matches the actual defect area.

[0132] Since the visual feature backbone (the visual encoder of the large visual multimodal model) remains frozen, only the trainable parameters of the defect-refining model are available. (etc.) need to be updated, therefore, the training cost is controllable, and it can quickly adapt to new defect types under small sample conditions.

[0133] Example 2: This embodiment provides an application example of the defect detection method described in the foregoing embodiments.

[0134] The training phase of the aforementioned defect detection method can be viewed as a collaborative training and alignment process between two networks: (1) The first stage is a single-stage target detection network (real-time candidate region generation module); (2) ROI refinement head (a defect refinement model trained on the frozen visual multimodal large model backbone).

[0135] Step 1: Data Preparation Collect a large number of real production line images (Resolution 1024×1024).

[0136] Defects in the image are labeled with rectangular boxes by manual or semi-automatic annotation systems, and fine-grained defect category labels are given.

[0137] Each annotation record (annotation box) It can be represented as

[0138] in, These are defect category labels (such as ablation, wear, broken strands, etc.).

[0139] The annotations can use common detection formats such as COCO, where each annotation box is defined by... or equivalent express.

[0140] Step 2: Fine-tuning the single-stage object detection network in the first stage Using the labeled data mentioned above, a single-stage object detection network (e.g., YOLOv9) is trained for standard object detection, enabling it to regress a set of bounding boxes for candidate defect regions across the entire image. And its base category scores. The goal of this network is to maintain a high recall rate during the inference phase, that is, to cover as many suspected defective areas as possible.

[0141] The detection training includes classification loss (e.g., based on cross-entropy or Focal Loss) and bounding box regression loss (e.g., IoU Loss, GIoU, DIoU, CIoU, etc.). After training, a single-stage object detection network for the first stage is obtained, which is used as a "teacher-style candidate generator".

[0142] Step 3: ROI Refinement Training During the training phase of the ROI refinement head, the bounding boxes of the candidate defect regions output by the single-stage object detection network are used. Compared with the actual annotation box ( Matching is performed to construct a monitoring signal.

[0143] bounding box of candidate defect region The matching with the actual bounding box is as follows: For each Calculate its relationship with all ground truth boxes Intersection over Union (IoU):

[0144] Select the true frame that maximizes IoU.

[0145] As The source of oversight.

[0146] from Read its defect category label and geometric parameters As the regression target.

[0147] Feature extraction and ROI Alignment are as follows: Image The visual feature map is obtained by feeding it into the visual encoding branch of the frozen visual multimodal large model.

[0148] According to the description in the foregoing embodiments, Projected to The coordinate system obtained And obtain the corresponding feature vector through ROIAlign. .

[0149] This process is repeated for each sample image and each candidate bounding box during training, thereby constructing training samples for all candidate defect regions. .

[0150] Step 4: Multi-task supervision right After classification head calculation , and tags Calculate cross-entropy loss .right After bounding box regression head calculation Calculate smoothed L1 loss Joint losses:

[0151] Parameter update strategy: Freeze the multimodal visual encoding backbone (do not update the backbone parameters of the large model), and only train the shared projection layer, classification branch, and bounding box regression branch in the ROI refinement head. Since the number of parameters to be trained is much smaller than that of the complete visual multimodal large model, the computational resources and sample size required for training are significantly reduced, making it suitable for incremental iteration in industrial settings.

[0152] Through the above process, the ROI refinement head learned that: (1) How to distinguish fine-grained defect categories based on high semantic visual features; (2) How to select candidate boxes in the first stage Accurate regression to This achieves high positioning accuracy.

[0153] After training, the resulting deployable model includes: the weights of the first-stage single-stage object detection network; the frozen weights of the visual multimodal large model; the ROI refinement head (parameters of the trained lightweight shared projection layer, classification branch, and bounding box regression branch); and the coordinate alignment and ROI Align mapping strategy. These models and strategies together constitute a complete defect detection system.

[0154] like Figure 2 As shown, the implementation method for the inference phase is described below: During the inference phase, the execution flow of the defect detection system when it is actually running online on the production line is as follows: The first step, image acquisition: An industrial camera (used as an image acquisition module) periodically acquires images of the surface of the cable or wire harness. Send to the system.

[0155] The second step is to generate bounding boxes for candidate defect regions: This involves processing the image... Inputting the first-stage single-stage object detection network (YOLOv9 fine-tuned model) yields a set of bounding boxes for candidate defect regions. And its rough confidence level. This stage requires high recall, that is, covering as many possible anomalous areas as possible.

[0156] The third step, semantic feature extraction and ROI alignment, includes: (1) Image The visual encoder (used as a visual feature extraction module) of the frozen visual multimodal large model is fed into it to obtain a full-image level visual feature map. .

[0157] (2) Bounding box for each candidate defect region The coordinate mapping method described in the foregoing embodiment is used to obtain .

[0158] (3) In high semantic visual feature maps Top Performing ROI Align yields the visual feature vectors corresponding to the candidate defect regions. .

[0159] Step 4: Refined ROI Reasoning Will Input the trained ROI refinement head to obtain: fine-grained defect category probability distribution and regression offset The decoding formula described in the previous embodiment is used to recover the data. .

[0160] This step outputs:

[0161] in

[0162]

[0163] Step 5: Post-processing and output For all Perform the following operations: (1) Low confidence filtering: If Set a threshold (e.g.) If ), then remove it.

[0164] (2) Overlap suppression: For overlaps between the same category For excessively tall frames, retain the highest value. The box (i.e., nonmaximum suppression) ).

[0165] (3) Structured output: Output the defect category, final bounding box coordinates and confidence level as the final detection result.

[0166] For targets identified as high-risk defects, the system can directly trigger alarm / rejection actions, or upload the results to the host computer quality traceability system to achieve closed-loop control of the production line.

[0167] The beneficial effects of this embodiment are as follows: (1) A phased, cross-model paradigm defect detection process: The first stage uses a lightweight, real-time single-stage target detection network to quickly scan the entire industrial image, which only undertakes the responsibility of "generating candidate defect regions" to ensure high recall and online production line latency requirements; The second stage, without changing the backbone parameters of the visual multimodal large model, calls its visual encoding results to perform fine-grained defect category determination and bounding box fine regression on each candidate defect region output by the first stage; The two stages are connected by a predefined geometric coordinate mapping relationship and region alignment (ROIAlign) operation to form a cascaded detection link of "fast candidate → semantic refinement". This cascaded detection process is different from the traditional end-to-end single detection network and belongs to a cross-paradigm, modular industrial surface defect detection scheme.

[0168] (2) Explicit coordinate alignment of the bounding boxes of candidate defect regions to the visual token grid: The bounding boxes of candidate defect regions output by the first-stage detection network are aligned with the original image coordinate system. Input coordinate system mapped to the visual encoder of a large visual multimodal model This is then further mapped to a visual token grid. and in the feature map corresponding to that grid. The method performs ROI Align to obtain fixed-length feature vectors that correspond one-to-one with the candidate defect regions. This coordinate alignment process is explicit, computable, and reproducible. It unifies the candidate boxes generated by the detector with the high semantic visual features output by the visual multimodal large model into the same space, providing accurate local semantic input for subsequent classification and regression. This method is decoupled from specific one-stage detector types and specific visual multimodal large models, and can be transferred to scenarios with different image resolutions, different surface materials, and different production line imaging conditions.

[0169] (3) The incremental adaptation mechanism of freezing the visual backbone of the visual multimodal large model and training only the lightweight ROI fine-tuning head: The visual multimodal large model as a whole is used as a "general visual feature generator". Its parameters are frozen during the training and inference stages to preserve its ability to express visual patterns such as multi-type textures, cracks, characters, and ablation marks. Based on freezing the backbone, only the lightweight shared projection layer, classification branch and bounding box regression branch located after ROI alignment are trained or fine-tuned, so that the system can quickly adapt to new defect categories or new field conditions under small sample and short cycle conditions. This training paradigm significantly reduces the computing power consumption and data requirements in the industrial implementation process, and takes into account the semantic capabilities of the large model and the engineering constraints of the industrial site.

[0170] (4) Structured inspection output for industrial quality inspection: After semantic refinement of candidate defect regions, the output includes: defect category label (which can be refined to risk level), corresponding confidence score, and target bounding box coordinates after regression refinement; this output format can be directly consumed by online quality inspection system, host computer or production line control unit for automatic alarm, sorting / rejection, rework location and quality traceability, reflecting the completeness of this application for industrial application scenarios; compared with multimodal question answering system that only outputs text description, or one-stage detector that only outputs coarse box, the output of this application has clear geometric semantic dual constraints.

[0171] Example 3: Another embodiment of this application relates to a defect detection device. The implementation details of the defect detection device in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. A schematic diagram of the defect detection device in this embodiment can be seen as follows: Figure 3 As shown, it includes an image acquisition module 301, a candidate box generation module 302, a visual encoding module 303, a semantic feature extraction module 304, and a defect refinement module 305.

[0172] Image acquisition module 301 is used to acquire an image to be detected containing the target object; The candidate box generation module 302 is used to infer candidate defect regions represented by bounding boxes based on the image to be detected using a single-stage object detection network. The visual coding module 303 is used to obtain the visual feature map of the image to be detected based on the visual coding branch of the large visual multimodal model. The semantic feature extraction module 304 is used to map the candidate defect region to the coordinate system of the visual feature map and extract the semantic features corresponding to the candidate defect region in the visual feature map. The defect refinement module 305 is used to determine the defect category and perform bounding box regression on the candidate defect region based on the semantic features corresponding to the candidate defect region.

[0173] In some embodiments, mapping candidate defect regions to the coordinate system of a visual feature map and extracting semantic features corresponding to candidate defect regions from the visual feature map includes: mapping candidate defect regions to the input coordinate system of a visual coding branch to obtain a first mapping result; mapping the first mapping result to the coordinate system of a visual feature map to obtain a second mapping result; and extracting semantic features of the second mapping result as semantic features corresponding to candidate defect regions.

[0174] In some embodiments, based on the semantic features corresponding to the candidate defect region, defect category determination and bounding box regression are performed on the candidate defect region. This includes using a pre-trained defect refinement model to process the semantic features corresponding to the candidate defect region to obtain the defect category and bounding box regression results of the candidate defect region. The defect refinement model includes a shared projection layer, a classification branch, and a bounding box regression branch. The shared projection layer is used to project the semantic features corresponding to the candidate defect region onto an intermediate dimension to obtain a shared projection result. The classification branch is used to project the shared projection result onto the defect category space of the target dimension to determine the defect category. The bounding box regression branch is used to obtain the regression offset of the candidate defect region based on the shared projection result, and obtain the corresponding target defect region based on the regression offset.

[0175] In some embodiments, the shared projection result expression is as follows:

[0176] The expression for the defect category determination result is as follows:

[0177]

[0178] The regression offset expression for the candidate defect region is as follows:

[0179] in, It is a shared projection result; , , These are trainable parameters; It is a candidate defect region The corresponding semantic feature vector; d It is the intermediate dimension; It is any defect category; It includes all defect categories; It represents the number of defect categories.

[0180] In some embodiments, the defect detection apparatus further includes a model training module for training or fine-tuning a single-stage detection network and a defect refinement model while freezing the parameters of a large visual multimodal model.

[0181] In some embodiments, the defect detection device further includes a result output module for outputting the defect detection result after defect category determination and bounding box regression. The defect detection result includes the coordinates of the target bounding box after bounding box regression, the defect category label, and the corresponding confidence score.

[0182] In some embodiments, the result output module is further configured to: before outputting the defect detection result after defect category determination and bounding box regression, perform post-processing operations on the target defect region obtained by bounding box regression, the post-processing operations including: deleting target defect regions with confidence scores lower than a first set threshold, and retaining the target defect region with the highest confidence score among target defect regions in the same defect category with a crossover ratio higher than a second set threshold.

[0183] The specific implementation methods of each module are described in the aforementioned embodiments, and will not be repeated in this embodiment.

[0184] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0185] Example 4: Another embodiment of this application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the defect detection methods in the above embodiments.

[0186] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0187] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0188] Example 5: Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the above-described defect detection method embodiment.

[0189] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A defect detection method, characterized in that, include: Acquire the image to be detected that contains the target object; Based on a single-stage object detection network, the image to be detected is inferred to obtain candidate defect regions represented by bounding boxes; The visual feature map of the image to be detected is obtained based on the visual coding branch of the visual multimodal large model. The candidate defect region is mapped onto the coordinate system of the visual feature map, and the semantic features corresponding to the candidate defect region are extracted from the visual feature map. Based on the semantic features corresponding to the candidate defect regions, the candidate defect regions are used to determine the defect category and perform bounding box regression.

2. The defect detection method according to claim 1, characterized in that, Mapping the candidate defect region onto the coordinate system of the visual feature map, and extracting the semantic features corresponding to the candidate defect region from the visual feature map, including: The candidate defect region is mapped onto the input coordinate system of the visual coding branch to obtain the first mapping result; The first mapping result is mapped onto the coordinate system of the visual feature map to obtain the second mapping result; The semantic features of the second mapping result are extracted and used as the semantic features corresponding to the candidate defect region.

3. The defect detection method according to claim 1, characterized in that, Based on the semantic features corresponding to the candidate defect regions, defect category determination and bounding box regression are performed on the candidate defect regions, including: Using a pre-trained defect refinement model, the semantic features corresponding to the candidate defect regions are processed to obtain the defect category and bounding box regression results of the candidate defect regions; The defect refinement model includes a shared projection layer, a classification branch, and a bounding box regression branch. The shared projection layer projects the semantic features corresponding to the candidate defect region onto an intermediate dimension to obtain a shared projection result. The classification branch projects the shared projection result onto the defect category space of the target dimension to determine the defect category. The bounding box regression branch obtains the regression offset of the candidate defect region based on the shared projection result and obtains the corresponding target defect region based on the regression offset.

4. The defect detection method according to claim 3, characterized in that, The expression for the shared projection result is as follows: The expression for the defect category determination result is as follows: The regression offset expression for the candidate defect region is as follows: in, It is a shared projection result; , , These are trainable parameters; It is the first Semantic feature vectors corresponding to each candidate defect region; It is any defect category; It includes all defect categories; It is the number of defect categories; Indicates the first Each candidate defect region belongs to category c The probability of; This represents the regression offset of the candidate defect region.

5. The defect detection method according to claim 3, characterized in that, Also includes: With the parameters of the large visual multimodal model frozen, train or fine-tune the single-stage detection network and the defect refinement model.

6. The defect detection method according to claim 1, characterized in that, Also includes: Output the defect detection results after defect category determination and bounding box regression. The defect detection results include the coordinates of the target bounding box after bounding box regression, the defect category label, and the corresponding confidence score.

7. The defect detection method according to claim 1, characterized in that, Also includes: Post-processing operations are performed on the target defect regions obtained by bounding box regression. The post-processing operations include: deleting target defect regions with confidence scores lower than a first set threshold, and retaining the target defect regions with the highest confidence scores among target defect regions of the same defect category with intersection-union ratios higher than a second set threshold.

8. A defect detection device, characterized in that, include: The image acquisition module is used to acquire the image to be detected that contains the target object; The candidate box generation module is used to infer the image to be detected based on a single-stage object detection network to obtain candidate defect regions represented by bounding boxes; The visual coding module is used to obtain the visual feature map of the image to be detected based on the visual coding branch of the large visual multimodal model. The semantic feature extraction module is used to map the candidate defect region to the coordinate system of the visual feature map, and extract the semantic features corresponding to the candidate defect region in the visual feature map. The defect refinement module is used to determine the defect category and perform bounding box regression on the candidate defect regions based on the semantic features corresponding to the candidate defect regions.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the defect detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the defect detection method according to any one of claims 1 to 7.

Images