Intelligent visual image detection system based on pre-trained large model
Patent Information
- Application Number
- CN202611209523.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-10-30
AI Technical Summary
[0005]本发明要解决的技术问题是：如何在保证工业产线高并发、毫秒级检测节拍的前提下，将预训练视觉大模型的泛化能力和语义理解能力应用于超高分辨率工业图像的缺陷检测，克服大模型固定小尺寸输入与超高分辨率图像之间的分辨率矛盾，以及大模型算力开销与工业现场边缘算力有限、实时性要求高之间的效率矛盾，从而降低复杂背景、反光材质及未知缺陷场景下的漏检率和误检率
[0013]本发明能够达到的有益效果包括：
Smart Images
Figure CN123015469A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence visual inspection technology, specifically to an intelligent visual image inspection system based on a pre-trained large model. It is particularly suitable for production line scenarios that require defect detection of ultra-high resolution industrial images, such as the inspection of new energy battery electrode sheets, the inspection of 3C precision structural components, and the inspection of semiconductor wafers. It can achieve intelligent visual image inspection that balances the generalization ability of the pre-trained large model with the millisecond-level inspection cycle requirements of industrial production lines. Background Technology
[0002] As industrial manufacturing moves towards high-end development, intelligent vision inspection systems have become an important part of production line quality control. Traditional machine vision solutions rely on manually designed image processing operators to identify defects, requiring repeated parameter tuning for specific products and exhibiting weak generalization ability. Deep learning small models based on convolutional neural networks are widely used in the field of industrial inspection, but they are small-sample closed-set models. Their detection capability depends on the distribution of training data, and they are prone to missed detections and false detections when faced with complex backgrounds, reflective materials, and unknown defect types. Furthermore, when products are changed or new defect categories are added, it is necessary to re-collect labeled data and retrain the model, resulting in high iteration costs.
[0003] The pre-trained visual large model is pre-trained with massive amounts of natural image data and has strong zero-shot generalization ability and semantic understanding ability. The closest existing technology to this invention is to apply the pre-trained visual large model directly or after simple fine-tuning to industrial image detection. This scheme first scales the original high-resolution image to the fixed input size of the large model, or uses a fixed-size sliding window to divide the original image into multiple sub-images, and then inputs them into the visual encoder of the pre-trained large model to extract features. Finally, the classification head or detection head outputs the judgment result and summarizes it to obtain the detection conclusion of the whole image.
[0004] The above-mentioned approach indiscriminately feeds all input images into a large model for full inference, which has the following problems: minor defects are easily compressed and lost during overall scaling, leading to missed detections; the sliding window slicing loses global topological context information; normal structures such as mounting holes and chamfered edges are easily misjudged as abnormal, leading to false detections; and inferring all sub-images one by one increases the time consumption significantly with the increase of resolution, making it difficult to meet the millisecond-level detection cycle requirements of the production line. Therefore, there is an urgent need for an intelligent vision image detection solution that can both leverage the generalization advantages of the large model and meet the real-time requirements of the production line. Summary of the Invention
[0005] The technical problem to be solved by this invention is: how to apply the generalization ability and semantic understanding ability of pre-trained large visual models to defect detection of ultra-high resolution industrial images while ensuring high concurrency and millisecond-level detection cycle of industrial production lines, overcome the resolution contradiction between the fixed small-size input of large models and ultra-high resolution images, and the efficiency contradiction between the computing power consumption of large models and the limited edge computing power and high real-time requirements of industrial sites, thereby reducing the false negative rate and false positive rate in complex backgrounds, reflective materials and unknown defect scenarios.
[0006] To address the aforementioned technical problems, this invention provides an intelligent visual image detection system based on a pre-trained large model, comprising an image acquisition module, an anomaly perception routing module, an adaptive cropping routing module, a dual-stream large model precision judgment module, and a result output module.
[0007] The image acquisition module is used to acquire high-resolution original images of the production line products and generate a low-resolution global image corresponding to the high-resolution original image.
[0008] The anomaly-aware routing module uses a lightweight convolutional neural network to extract region-level features from the low-resolution global map, obtaining anomaly confidence scores for each image region. Based on a preset low confidence threshold, it judges each image region: image regions with anomaly confidence scores lower than the low confidence threshold are processed early and directly judged as qualified regions, and no further calculations are performed on these regions; image regions with anomaly confidence scores not lower than the low confidence threshold are marked as suspected anomaly regions and enter the subsequent cropping process.
[0009] The adaptive cropping routing module dynamically generates cropping bounding boxes for the suspected abnormal regions based on class activation heatmaps and connected component analysis. After proportional expansion and coordinate mapping, it crops the corresponding high-resolution local patches from the high-resolution original image, realizing the transformation from a coarse response region to a precise high-resolution patch, thus avoiding the global topological fragmentation problem caused by the traditional fixed sliding window slicing strategy.
[0010] The dual-stream large model precision judgment module includes a pre-trained visual large model with its backbone network parameters frozen, and a dual-stream cue adapter module inserted into the cross-attention layer of the pre-trained visual large model. The global label sequence corresponding to the low-resolution global map and the local label sequence corresponding to the high-resolution local map are mapped to query vector, key vector, and value vector respectively through the low-rank projection layer of the dual-stream cue adapter module. Then, through bidirectional cross-attention calculation, the global semantic stream and local texture stream are fused. This allows the pre-trained visual large model to balance global topological context understanding and accurate localization of minor defects without adjusting the backbone network parameters. Based on the fused features, it outputs the detection results of the suspected abnormal regions and the semantic interpretation of the defect type.
[0011] The result output module is used to summarize the qualified conclusions corresponding to the qualified areas and the detection results output by the dual-flow large model fine judgment module to obtain the detection conclusion of the entire high-resolution original image, and output the detection conclusion to the production line control system.
[0012] In a preferred embodiment, the system of the present invention further includes a knowledge distillation training module, which uses the deep feature map and logical attribution vector output by the dual-stream large model precision judgment module as supervision signals, and trains the network parameters of the anomaly perception routing module by jointly using feature-level distillation loss and logical-level distillation loss, so that the anomaly perception routing module can obtain a defect sensitivity close to that of the pre-trained visual large model without increasing the number of parameters, thereby further reducing the false negative rate and false positive rate.
[0013] The beneficial effects that this invention can achieve include:
[0014] Firstly, the anomaly perception routing module performs millisecond-level preliminary screening on the high-resolution original image and performs early exit processing on the qualified areas. Only the suspected abnormal areas are cropped into high-resolution local patches and sent to the pre-trained visual large model for fine judgment, thereby greatly reducing the average computational load of the system and enabling the overall detection cycle to meet the real-time requirements of industrial production lines.
[0015] Secondly, the adaptive pruning routing module dynamically locates suspected abnormal areas based on the class activation heatmap and performs outward expansion and coordinate mapping, avoiding the loss of global topology information caused by the traditional fixed sliding window slicing strategy and reducing false detections caused by the lack of global context.
[0016] Third, by injecting global low-resolution semantic stream and local high-resolution texture stream into the cross-attention layer of the pre-trained visual large model through the dual-stream cue adapter module, the pre-trained visual large model can take into account both global semantic understanding and local detail perception without increasing the computational load of the backbone network, thereby further reducing the false negative rate and false positive rate.
[0017] Fourth, since the dual-stream large model precision judgment module freezes the backbone network parameters of the pre-trained visual large model and only trains the dual-stream cue adapter module, it avoids the catastrophic forgetting problem caused by full parameter fine-tuning, and at the same time significantly reduces the cost of GPU memory and computing power required for training, making it feasible for engineering implementation. Attached Figure Description
[0018] Figure 1 This is the overall business flow diagram of the system of the present invention;
[0019] Figure 2This is a flowchart of the bidirectional cross-attention fusion process of the dual-stream cue adapter of the present invention;
[0020] Figure 3 This is a flowchart of the knowledge distillation training closed loop of the present invention. Detailed Implementation
[0021] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0022] In this embodiment, the intelligent visual image detection system provided by the present invention is deployed on an edge computing device in an industrial production line. This edge computing device can be an industrial control computer equipped with a graphics processing accelerator card, communicating with industrial cameras, the production line PLC control system, and the MES system. The system as a whole includes an image acquisition module, an anomaly perception routing module, an adaptive cropping routing module, a dual-stream large model precision judgment module, a result output module, and a knowledge distillation training module. The data flow between these modules is as follows: After acquiring a high-resolution original image of the industrial production line, the image acquisition module performs downsampling processing on the image to generate a low-resolution global image, which is then sent to the anomaly perception routing module. Simultaneously, the original high-resolution image is stored in an image cache unit for later use. The anomaly perception routing module performs regional-level anomaly scoring on the low-resolution global image and performs enhancement processing on the vast majority of normal areas. The system performs pre-exit processing and outputs a qualified conclusion. For suspected abnormal regions whose scores reach a preset threshold, the adaptive cropping routing module locates and crops the corresponding high-resolution local patches on the original high-resolution image. Subsequently, the low-resolution global image and the high-resolution local patches are fed into the dual-stream large model fine judgment module as dual-stream inputs. After feature fusion is completed in the cross-attention layer of the pre-trained visual large model by the dual-stream cue adapter module, the final detection result and semantic interpretation are output. The result output module summarizes the qualified conclusion of the pre-exited regions and the detection result output by the dual-stream large model fine judgment module to obtain the final detection conclusion of the entire image, and outputs it to the production line control system. At the same time, the deep feature map and semantic logic output by the dual-stream large model fine judgment module serve as supervision signals. The knowledge distillation training module continuously optimizes the network parameters of the anomaly perception routing module, forming a closed-loop training circuit.
[0023] In a specific example, the high-resolution original image acquired by the industrial camera has a resolution of 3840×2160 pixels. The image acquisition module performs proportional downsampling processing on the original image to obtain a low-resolution global image with a resolution of 384×384 pixels. It should be understood that the above resolution values are only examples. The present invention does not limit the specific resolution of the original image and the global image, and can be adjusted according to the camera model and defect size requirements in the actual industrial inspection scenario.
[0024] The anomaly-aware routing module uses a lightweight convolutional neural network to divide the low-resolution global image into blocks, resulting in multiple image regions. It then extracts feature vectors from each region. For the i-th image region, the anomaly-aware routing module calculates an anomaly confidence score based on the region's feature vector. The calculation method is as follows:
[0025] ;
[0026] In the formula, This represents the feature vector extracted from the i-th image region using a lightweight convolutional backbone. , These are the weight matrix and bias vector of the first fully connected layer, respectively. , These are the weight matrix and bias vector of the second fully connected layer, respectively. It is a non-linear activation function. The sigmoid function maps the output to the range of 0 to 1. This represents the anomaly confidence score of the i-th image region, with a value ranging from 0 to 1. The larger the value, the higher the probability that there is an anomaly in the region.
[0027] After obtaining the anomaly confidence score for each image region, the anomaly detection routing module uses a preset low confidence threshold. Perform routing determination for each image region: when When the anomaly detection routing module determines that the image region is qualified, it terminates all subsequent calculations for that region and no longer sends the region to the dual-stream large model fine judgment module, thereby achieving early exit and significantly reducing the average computational load of the system. At this time, the anomaly detection routing module marks the image region as a suspected anomaly region and passes the coordinate information of the region to the adaptive cropping routing module for further processing. In a specific example, the low confidence threshold... A value of 0.3 means that image regions with an anomaly confidence score below 0.3 are considered qualified regions and exited early. It should be understood that this threshold can be adjusted according to different requirements for false negative rate and computational efficiency in actual detection scenarios. For example, it can be taken in the range of 0.2 to 0.4.
[0028] For suspected anomalous regions marked by the anomaly-aware routing module, the adaptive cropping routing module performs a transformation from coarse response regions to precise high-resolution patches. This avoids the global topological fragmentation problem caused by the traditional fixed sliding window tiling strategy. Specifically, the adaptive cropping routing module first performs gradient-weighted activation mapping calculation on the feature map output from the last layer of the lightweight convolutional backbone in the anomaly-aware routing module, obtaining a spatial activation heatmap corresponding to the suspected anomalous region. The response intensity of each pixel in this heatmap reflects the contribution of that location to the anomaly confidence score. Then, a binarization threshold is set for this activation heatmap. The response strength is greater than Pixels in the active region are marked as active pixels, and the remaining pixels are marked as inactive pixels, thus obtaining a binarized activation mask. Next, connected component analysis is performed on this binarized activation mask to extract the bounding rectangle corresponding to each connected region, which is used as a candidate anomaly bounding box, denoted as . ,in , , , These represent the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner of the candidate bounding box, respectively. Then, the bounding box is expanded according to a preset expansion ratio. The candidate bounding box is proportionally expanded to obtain the expanded cropped bounding box. The purpose of expansion is to preserve the contextual information around the candidate anomaly region and avoid losing defect edge features due to the bounding box being too compact. Finally, the cropped bounding box is mapped to coordinates according to the ratio between the low-resolution global image and the original high-resolution image to obtain the precise cropping coordinates in the coordinate system of the original high-resolution image. Based on this, the corresponding region is cropped from the original high-resolution image cache to obtain high-resolution local patches, i.e., the image corresponding to the local label. For cases where there are multiple suspected anomaly regions in the same image, the adaptive cropping routing module repeats the above connected component analysis and subsequent processing for each suspected anomaly region to crop multiple high-resolution local patches. Each patch and its relative coordinates in the original image are then passed to the dual-stream large model fine judgment module. In a specific example, the expansion ratio... The value is 0.2, which means expanding the candidate bounding box outward by 20% in both length and width directions.
[0029] Alternatively, in another implementation of the adaptive cropping routing module, in addition to using the above-mentioned method of generating cropping bounding boxes based on class activation heatmaps and connected component analysis, a lightweight object detection network can be used to perform bounding box regression prediction on suspected abnormal regions in the low-resolution global map, replacing the class activation heatmap scheme. This can also achieve the same technical effect of moving from coarse suspected region location to precise cropping coordinate generation.
[0030] The dual-stream large-scale model precision judgment module freezes all parameters of the backbone network of the pre-trained visual large-scale model, inserting only lightweight, trainable dual-stream cue adapter modules at the cross-attention positions of each layer of the backbone network to achieve the fusion of global semantic stream and local texture stream. Specifically, the dual-stream large-scale model precision judgment module first inputs a low-resolution global map into the image block embedding layer of the pre-trained visual large-scale model to obtain a global label sequence, denoted as... ,in This represents the global number of tags. As the feature dimension, each high-resolution local patch output by the adaptive cropping routing module is input into the same image block embedding layer to obtain a local label sequence, denoted as . ,in This represents the number of local markers.
[0031] At each cross-attention position of the pre-trained large visual model, the dual-stream cue adapter module uses a linear mapping layer to project the global and local label sequences into query vectors, key vectors, and value vectors, respectively. During the projection process, this linear mapping layer employs low-rank matrix factorization to reduce the number of trainable parameters. The query vector corresponding to the local label sequence is used as the query, and the key and value vectors corresponding to the global label sequence are used as the key and value, respectively. Local-to-global cross-attention is calculated, enabling local high-resolution patches to acquire global topological context information. The calculation method is as follows:
[0032] ;
[0033] In the formula, This represents the query matrix obtained by projecting the local labeled sequence. , These represent the key matrix and value matrix obtained by projecting the global marker sequence, respectively. This represents the dimension of the key vector, used to scale the attention score to avoid excessively large values that could lead to unstable gradients. This represents the output of local features after incorporating global context information.
[0034] Using the query vector corresponding to the global label sequence as the query, and the key vector and value vector corresponding to the local label sequence as the key and value, we calculate the global-to-local cross-attention, enabling the global features to obtain detailed information about minor local defects. The calculation method is as follows:
[0035] ;
[0036] In the formula, This represents the query matrix obtained by projecting the global label sequence. , Let these represent the key matrix and value matrix obtained by projecting the local labeled sequence, respectively. This represents the global feature output after incorporating local detail information.
[0037] The bidirectional fusion features obtained above and The residuals are added to the original local label sequence and the original global label sequence respectively, and then fed into the next layer of the pre-trained visual large model network to continue forward propagation until feature extraction of all layers is completed. The fused features output by the final layer of the pre-trained visual large model are sent to the detection head and the semantic interpretation head. The detection head outputs the detection result of the suspected abnormal region based on the fused features. The detection result includes a qualified conclusion and a unqualified conclusion. The semantic interpretation head outputs a logical interpretation of the defect type based on the label semantics based on the fused features, such as a description of defect types such as scratches, missing parts, and different colors.
[0038] Alternatively, in another implementation of the dual-stream cue adapter module, in addition to inserting the dual-stream cue adapter module at the cross-attention position of each layer of the pre-trained visual large model, the dual-stream cue adapter module can also be inserted only in the middle several layers of the pre-trained visual large model. By reducing the number of inserted layers, the number of trainable parameters and inference latency are further reduced, and higher inference efficiency is achieved with less impact on the fusion effect. This can be used as a supplementary implementation of the present invention in scenarios with limited computing resources.
[0039] To enable the anomaly-aware routing module to possess defect sensitivity approaching that of a pre-trained large-scale visual model, the system of this invention may further include a knowledge distillation training module. This module performs joint feature-level and semantic-level distillation training using the pre-trained large-scale visual model as the teacher network and the anomaly-aware routing module as the student network. Specifically, for the same batch of industrial image samples, the anomaly-aware routing module and the pre-trained large-scale visual model are input separately to obtain intermediate feature maps output by the anomaly-aware routing module in the corresponding regions. And deep feature maps output by pre-trained large visual models in the same region. Because the feature dimensions of the student network and the teacher network are inconsistent, the knowledge distillation training module introduces a trainable linear projection layer. The deep feature map output by the teacher network is projected into the same dimensional space as the feature map of the student network, and the feature-level distillation loss is calculated:
[0040] ;
[0041] In the formula, This represents the intermediate feature map output by the student network. This represents the deep feature map output by the teacher network. Indicates a dimension-aligned projection layer. The squared L2 norm represents the feature difference and is used to measure the feature level difference between the two.
[0042] Furthermore, the knowledge distillation training module obtains the defect type logical attribution vector from the teacher's network output. Corresponding logical vector to the student network output And introduce temperature coefficient After softening both components separately, the logic-level distillation loss was calculated:
[0043] ;
[0044] ;
[0045] In the formula, , These represent the logical output values of the teacher network and the student network for the c-th defect category, respectively. This is a temperature coefficient used to control the smoothness of the probability distribution. , Let represent the probabilities of the teacher network and student network in class c after softening. This represents the logical level distillation loss as measured in the form of KL divergence.
[0046] The knowledge distillation training module weights and combines the feature-level distillation loss and the logic-level distillation loss according to preset weight coefficients to obtain the total distillation loss:
[0047] ;
[0048] In the formula, , These are the weighting coefficients for the feature-level distillation loss and the logic-level distillation loss, respectively, used to balance their respective impacts on student network training. In a specific example, the weighting coefficients... The value is 0.5, and the weighting coefficient is... The value is 0.5.
[0049] Furthermore, the knowledge distillation training module will reduce the total distillation loss. The task supervision loss of the student network itself on labeled samples By weighting factor Combining these, we obtain the total loss ultimately used for backpropagation to update the student network parameters:
[0050] ;
[0051] In the formula, This represents the task supervision loss of the student network on labeled samples, which can specifically be the binary classification cross-entropy loss for defect detection. This is the overall weighting coefficient for distillation losses. This represents the total loss function used for the final backpropagation. In a specific example, the weight coefficients... When the value is 1, the knowledge distillation training module uses a gradient descent-type optimization algorithm to iteratively update the parameters of the student network, repeatedly executing the above-mentioned feature extraction, loss calculation, and parameter update process until the total loss converges or reaches the preset training rounds. Finally, an anomaly perception routing module with pre-trained visual large model-level defect sensitivity is obtained. It should be noted that during the training process, the parameters of the teacher network are always kept frozen, and only the parameters of the student network are updated to avoid the teacher network being affected by the training process.
[0052] In summary, the intelligent visual image detection method provided by this invention includes the following process: First, a high-resolution original image of the product on the production line is acquired using an industrial camera, and a corresponding low-resolution global image is generated simultaneously. Then, the low-resolution global image is input into the anomaly perception routing module to calculate the anomaly confidence score of each image region. Regions with scores below the low confidence threshold are processed for early exit and are deemed qualified. Next, for suspected anomaly regions with scores not below the low confidence threshold, the corresponding high-resolution local patches are cropped from the original high-resolution image using an adaptive cropping routing module. Finally, the low-resolution global image and each high-resolution local patch are input together. The dual-stream large model precision judgment module completes feature fusion and inference through the dual-stream prompt adapter module, outputting the final detection results and defect type semantic explanations for each suspected area. Finally, it summarizes the qualified conclusions of the areas that exited early and the judgment results output by the dual-stream large model precision judgment module to form the final detection conclusion of the entire image. The detection conclusion is then output to the production line PLC control system and MES system. During system operation, the deep feature maps and logical attribution semantics output by the dual-stream large model precision judgment module are continuously collected as training samples. The anomaly perception routing module is periodically updated online or offline according to the above knowledge distillation training mechanism to continuously improve the detection sensitivity of the anomaly perception routing module.
[0053] As can be seen from the above embodiments, the present invention performs early exit processing on the vast majority of normal areas through a lightweight anomaly perception routing module at the front end, and only crops a small number of suspected abnormal areas into high-resolution local patches before sending them to the pre-trained visual large model with the backbone frozen for fine judgment. This can control the average computational load of the system within the range that meets the real-time requirements of the production line while preserving the generalization ability and semantic understanding ability of the large model. At the same time, the dual-stream prompt adapter module introduces global semantic stream and local texture stream at the cross-attention layer, which can effectively avoid false detections caused by the lack of global context in the traditional sliding window scheme. In addition, the cross-architecture knowledge distillation training mechanism can transfer the defect sensitivity of the large model to the lightweight routing network at the front end, continuously improving the detection accuracy of the system without significantly increasing the number of parameters.
[0054] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.
Claims
1. An intelligent visual image detection system based on a pre-trained large model, characterized in that, include: The image acquisition module is used to acquire high-resolution original images of the products on the production line and generate a low-resolution global image corresponding to the high-resolution original image. The anomaly-aware routing module is used to extract region-level features from the low-resolution global map, obtain anomaly confidence scores for each image region, and determine each image region based on a preset low-confidence threshold. Image regions with an abnormal confidence score lower than the low confidence threshold are processed for early exit and determined to be qualified regions; image regions with an abnormal confidence score not lower than the low confidence threshold are marked as suspected abnormal regions. An adaptive cropping routing module is used to locate and crop high-resolution local patches corresponding to the suspected abnormal regions on the high-resolution original image. The dual-stream large model precision judgment module includes a pre-trained visual large model with its backbone network parameters frozen, and a dual-stream cue adapter module inserted into the cross-attention layer of the pre-trained visual large model. The dual-stream cue adapter module is used to perform bidirectional cross-attention fusion of the global label sequence corresponding to the low-resolution global map and the local label sequence corresponding to the high-resolution local map patch to obtain fusion features, and output the detection result of the suspected abnormal region based on the fusion features. The detection result includes a qualified conclusion and a unqualified conclusion. The result output module is used to summarize the qualified conclusions corresponding to the qualified regions and the detection results output by the dual-stream large model fine judgment module to obtain the detection conclusions of the entire high-resolution original image.
2. The system according to claim 1, characterized in that, The adaptive pruning routing module is specifically used for: Gradient-weighted class activation mapping is calculated on the feature map output by the convolutional backbone network in the anomaly perception routing module to obtain a spatial activation heatmap corresponding to the suspected anomaly region. The spatial activation heatmap is binarized using a preset activation threshold to obtain a binarized activation mask. Connectivity analysis is performed on the binarized activation mask to extract the bounding rectangles corresponding to each connected region as candidate anomaly bounding boxes; The candidate anomaly bounding box is expanded proportionally according to a preset expansion ratio to obtain a clipped bounding box. The cropping bounding box is mapped to coordinates according to the ratio between the low-resolution global image and the high-resolution original image to obtain the cropping coordinates in the coordinate system of the high-resolution original image, and the high-resolution local patch is cropped from the high-resolution original image according to the cropping coordinates.
3. The system according to claim 1, characterized in that, The dual-stream notification adapter module is specifically used for: The global label sequence and the local label sequence are respectively mapped to the corresponding query vector, key vector and value vector through a linear projection layer. The linear projection layer uses low-rank matrix factorization to achieve parametric projection. Using the query vector corresponding to the local label sequence as the query, and the key vector and value vector corresponding to the global label sequence as the key and value, the local-to-global cross attention is calculated to obtain the local feature output that integrates global context information; Using the query vector corresponding to the global label sequence as the query, and the key vector and value vector corresponding to the local label sequence as the key and value, global cross-attention to local is calculated to obtain a global feature output that incorporates local detail information; The local feature output and the global feature output are respectively added to the corresponding original local label sequence and the original global label sequence by residual addition to obtain the fused feature.
4. The system according to claim 3, characterized in that, The dual-stream large model precision judgment module also includes a detection head and a semantic interpretation head. The detection head is used to output the detection result of the suspected abnormal region based on the fusion features, and the semantic interpretation head is used to output the defect type semantic interpretation of the suspected abnormal region based on the fusion features.
5. The system according to claim 1, characterized in that, It also includes a knowledge distillation training module, which is used to train the network parameters of the anomaly-aware routing module using the deep feature map and logistic attribution vector output by the dual-stream large model precision judgment module as supervision signals. Obtain the intermediate feature map output by the anomaly perception routing module and the deep feature map output by the dual-stream large model fine judgment module. Calculate the feature-level distillation loss by projecting the deep feature map with the intermediate feature map after dimension alignment. The logical vector output by the anomaly perception routing module and the logical attribution vector output by the dual-stream large model fine judgment module are obtained, and the logical level distillation loss is calculated after temperature coefficient softening processing. The total distillation loss is obtained by weighting and combining the feature-level distillation loss with the logic-level distillation loss, and then combined with the task supervision loss of the anomaly-aware routing module to obtain the total loss. The network parameters of the anomaly-aware routing module are iteratively updated based on the total loss.
6. A method for intelligent visual image detection based on a pre-trained large model, characterized in that, include: Acquire high-resolution original images of the products on the production line and generate a low-resolution global map corresponding to the high-resolution original images; Regional feature extraction is performed on the low-resolution global map to obtain the anomaly confidence score of each image region. Image regions with anomaly confidence scores lower than a preset low confidence threshold are processed for early exit and determined to be qualified regions. Image regions with anomaly confidence scores not lower than the low confidence threshold are marked as suspected anomaly regions. For the suspected abnormal region, a high-resolution local patch corresponding to the suspected abnormal region is located and cropped from the high-resolution original image; The global label sequence corresponding to the low-resolution global map and the local label sequence corresponding to the high-resolution local map are input into a pre-trained visual large model whose backbone network parameters are frozen. The two-stream cue adapter module inserted into the cross-attention layer of the pre-trained visual large model performs bidirectional cross-attention fusion to obtain fusion features. The detection results of the suspected abnormal region are output based on the fusion features. The detection results include qualified and unqualified conclusions. By summarizing the qualified conclusions corresponding to the qualified areas and the detection results corresponding to the suspected abnormal areas, the detection conclusion of the entire high-resolution original image is obtained.
7. The method according to claim 6, characterized in that, The step of locating and cropping a high-resolution local patch corresponding to the suspected abnormal region on the high-resolution original image includes: Gradient-weighted class activation mapping is calculated on the feature map output by the convolutional backbone network that performs the region-level feature extraction to obtain a spatial activation heatmap corresponding to the suspected abnormal region. The spatial activation heatmap is binarized using a preset activation threshold to obtain a binarized activation mask. Connectivity analysis is performed on the binarized activation mask to extract the bounding rectangles corresponding to each connected region as candidate anomaly bounding boxes; The candidate anomaly bounding box is expanded proportionally according to a preset expansion ratio to obtain a clipped bounding box. The cropping bounding box is mapped to coordinates according to the ratio between the low-resolution global image and the high-resolution original image to obtain the cropping coordinates in the coordinate system of the high-resolution original image, and the high-resolution local patch is cropped from the high-resolution original image according to the cropping coordinates.
8. The method according to claim 6, characterized in that, The steps for bidirectional cross-attention fusion via the dual-stream cue adapter module include: The global label sequence and the local label sequence are respectively mapped to the corresponding query vector, key vector and value vector through a linear projection layer. The linear projection layer uses low-rank matrix factorization to achieve parametric projection. Using the query vector corresponding to the local label sequence as the query, and the key vector and value vector corresponding to the global label sequence as the key and value, the local-to-global cross attention is calculated to obtain the local feature output that integrates global context information; Using the query vector corresponding to the global label sequence as the query, and the key vector and value vector corresponding to the local label sequence as the key and value, global cross-attention to local is calculated to obtain a global feature output that incorporates local detail information; The local feature output and the global feature output are respectively added to the corresponding original local label sequence and the original global label sequence by residual addition to obtain the fused feature.
9. The method according to claim 6, characterized in that, Also includes: Using the deep feature map and logistic attribution vector output by the pre-trained large visual model as supervision signals, the network parameters for performing the region-level feature extraction are trained: Obtain the intermediate feature map output by the network that performs the region-level feature extraction, and calculate the feature-level distillation loss by projecting the deep feature map with the intermediate feature map after dimension alignment. Obtain the logical vector and the logical attribution vector output by the network performing the regional feature extraction, and calculate the logical distillation loss after temperature coefficient softening processing; The feature-level distillation loss and the logic-level distillation loss are weighted and combined to obtain the total distillation loss, which is then combined with the task supervision loss to obtain the total loss. The network parameters for the region-level feature extraction are iteratively updated based on the total loss.
10. The method according to claim 8, characterized in that, The step of outputting the detection result of the suspected abnormal region based on the fusion features further includes: Based on the fusion features, output the semantic interpretation of the defect type of the suspected abnormal region.