An industrial image logical anomaly detection method based on promptable concept segmentation and spatial hierarchical matching

Abstract

The application discloses an industrial image logical anomaly detection method based on a promptable concept segmentation and spatial hierarchical matching, and relates to the technical field of image processing. The method comprises the following steps: acquiring a to-be-detected image and at least one normal reference image; determining a target component or sub-component category involved in a to-be-detected task, and constructing a candidate prompt list for each target component or sub-component category; performing component instance extraction on the to-be-detected image and the normal reference image respectively to obtain an instance mask set corresponding to each candidate prompt; performing effectiveness judgment on the instance mask set corresponding to each candidate prompt, and respectively forming a test sub-instance set corresponding to the to-be-detected image and a reference sub-instance set corresponding to the normal reference image; constructing a spatial hierarchical structure of components and sub-components; extracting visual features of the to-be-detected image and the normal reference image, and performing mask pooling on the visual features at each level of the spatial hierarchical structure to obtain test instance visual features and reference instance visual features corresponding to each level; performing quantity comparison and instance matching on test instances in the to-be-detected image and reference instances in the normal reference image at corresponding levels of the spatial hierarchical structure; performing logical consistency detection to obtain a combined statistical anomaly score and a scene constraint anomaly score; fusing the spatial hierarchical matching anomaly score, the combined statistical anomaly score and the scene constraint anomaly score to generate an image-level anomaly score, and outputting a logical anomaly detection result of the to-be-detected image according to the image-level anomaly score.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial inspection technology, and in particular to an image detection method for logical anomalies in industrial scenarios, specifically an industrial image logical anomaly detection method based on suggestible concept segmentation and spatial hierarchy matching. Background Technology

[0002] In recent years, with the continuous improvement of intelligent manufacturing and industrial automation, image analysis-based automated inspection technology has become an important technical means to ensure product quality, improve production efficiency, and reduce labor costs. Compared with traditional inspection methods that rely on human experience, machine vision-based inspection methods have advantages such as fast inspection speed, good repeatability, and ease of deployment, and have been widely used in industrial scenarios such as electronics manufacturing, packaging inspection, assembly inspection, and automotive parts inspection.

[0003] In industrial vision inspection tasks, anomalies can generally be divided into two categories: structural anomalies and logical anomalies. Structural anomalies typically manifest as localized appearance defects such as scratches, cracks, gaps, stains, and dents. These anomalies usually exhibit obvious changes in texture, color, or shape in the image, making them relatively easy to identify using traditional image processing methods or deep learning methods. In contrast, logical anomalies often do not manifest as obvious localized damage, but rather as missing components, inconsistent component quantities, mismatched component categories, incorrect component combination relationships, incorrect component connection relationships, or abnormal local spatial relationships. These anomalies rely more heavily on the semantics of the components and their relationships, making them significantly more difficult to detect.

[0004] Existing technologies for addressing logical anomalies in industrial images mainly fall into the following categories. One type of method relies on supervised segmentation models or pseudo-label generation techniques. It first segments the components in the image and then determines logical relationships based on the segmentation results. While this type of method can achieve component-level representations to some extent, it often requires pixel-level annotation, pseudo-label generation, or additional model training, making it difficult to adapt to industrial application environments characterized by high annotation costs, limited sample sizes, and frequent scene changes. Another type of method attempts to directly perform semantic judgment on images using visual language models, multimodal models, or the reasoning capabilities of large models. This type of method can perform a certain degree of logical analysis in open-vocabulary scenarios, but it still suffers from insufficient accuracy in determining quantitative relationships, distinguishing fine-grained components, and modeling complex spatial relationships.

[0005] Furthermore, existing segmentation-based logical anomaly detection techniques suffer from the following drawbacks. First, existing segmentation methods are prone to semantic consistency and instance granularity in complex industrial scenarios. For target components belonging to the same category, the segmentation model may obtain inconsistent instance masks under different sample images or background conditions. When using different text prompts or visual language matching methods, issues such as component category confusion, missed segmentation, missegmentation, or oversegmentation may also occur, thereby reducing the reliability of subsequent logical anomaly judgments. Second, existing methods typically lack explicit modeling of the hierarchical relationships between components and sub-components, making it difficult to uniformly compare and recursively analyze the number, category, and local relationships of components at different levels in complex structures. Third, for industrial scenarios with combinatorial constraints, geometric statistical constraints, or specific topological relationship constraints, existing methods often lack a unified logical verification mechanism, making it difficult to simultaneously address the detection needs of component category mismatch, combinatorial relationship conflicts, geometric statistical deviations, and topological relationship anomalies.

[0006] In real-world industrial inspection scenarios, there are often situations where the number of normal samples is small, abnormal samples are difficult to collect, and operating conditions vary significantly. Relying on large-scale training samples and complex training processes not only results in high deployment costs but also easily leads to performance degradation when migrating across different scenarios. Therefore, how to achieve stable extraction of component instances from industrial images with few samples or even no additional training, and further effectively model their spatial hierarchical relationships and logical consistency to complete the detection of complex logical anomalies, remains a pressing technical problem to be solved in this field. Summary of the Invention

[0007] This invention aims to address the problems of unstable component instance extraction, insufficient modeling of hierarchical relationships between components and sub-components, difficulty in uniformly judging different types of logical anomalies, and reliance on additional training in existing industrial image logical anomaly detection technologies. It provides an industrial image logical anomaly detection method based on suggestible concept segmentation and spatial hierarchy matching.

[0008] This invention first constructs a candidate suggestion list based on a normal reference image and target industrial scene information. Then, using a segmentation model that supports concept suggestions, it extracts component instances from the image to be detected and the normal reference image according to the candidate suggestion list. By selecting the first valid instance mask set, it determines the instance set corresponding to each target component or sub-component category, thereby improving the semantic consistency and extraction stability of the component instance mask. Subsequently, it constructs a spatial hierarchy structure based on the inclusion relationship, spatial relationship, or semantic subordination relationship between components and sub-components, and performs quantity comparison, instance feature similarity calculation, and recursive matching at the corresponding levels. Furthermore, it combines combination consistency detection, geometric statistical comparison, and scene topology constraint checking to uniformly identify component quantity differences, category mismatches, combination relationship conflicts, geometric statistical deviations, and topological relationship anomalies, and finally generates logical anomaly detection results for the image to be detected.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching includes the following steps: S1. Obtain the image to be detected and at least one normal reference image; S2. Based on the normal reference image and the target industrial scene information, determine the target component or sub-component category involved in the task to be detected, and construct a candidate prompt list for each target component or sub-component category; S3. Using a segmentation model that supports concept cues, component instance extraction is performed on the image to be detected and the normal reference image according to the candidate cue list to obtain the instance mask set corresponding to each candidate cue; wherein, the segmentation model that supports concept cues preferably adopts SAM3, or an equivalent model with natural language concept cue segmentation capability; for the input image and candidate cue, the segmentation model that supports concept cues extracts image features and cue features, obtains conditional feature representation through cross-modal fusion, and outputs the instance mask set corresponding to the candidate cue by the mask decoder.

[0010] S4. Based on the candidate suggestion list, the validity of the instance mask set corresponding to each candidate suggestion is judged, and a test sub-instance set corresponding to the image to be detected and a reference sub-instance set corresponding to the normal reference image are formed respectively.

[0011] S5. Based on the test sub-instance set and the reference sub-instance set, construct a spatial hierarchy structure of components and sub-components; wherein the spatial hierarchy structure uses the target product or target region as the root node, component instances as first-level nodes, and sub-component instances within the components as lower-level nodes, and determines the parent-child node relationship based on the inclusion relationship between instance masks, the relationship of instance center positions, and the component hierarchy relationship in the target industrial scene. The component hierarchy relationship can be determined by manually preset component hierarchy rules, or by the scene structure rules generated by the multimodal model based on the normal reference image.

[0012] S6. Visual features of the image to be detected and the normal reference image are extracted using an image encoder, and at each level of the spatial hierarchy, the visual features are masked and pooled based on the instance masks in the test sub-instance set and the reference sub-instance set to obtain the test instance visual features and reference instance visual features corresponding to each level; wherein, the image encoder is used to output the multi-layer visual features of the image, and the mask pooling is used to aggregate the visual features within the coverage area of ​​each component instance mask into the corresponding instance visual features.

[0013] S7. At the corresponding level of the spatial hierarchy, perform quantity comparison and instance matching between the test instances in the image to be detected and the reference instances in the normal reference image.

[0014] S8. Based on the logical constraint information determined by the normal reference image, perform logical consistency detection to obtain the combined statistical anomaly score and the scene constraint anomaly score.

[0015] S9. The spatial hierarchical matching anomaly score, combined statistical anomaly score, and scene constraint anomaly score are fused to generate an image-level anomaly score, and the logical anomaly detection result of the image to be detected is output based on the image-level anomaly score.

[0016] Preferably, in step S1, the image to be detected is an industrial image for which it is necessary to determine whether there is a logical anomaly, and the normal reference image is a sample image without logical anomalies in the target industrial scene, used to provide reference information for normal component categories, normal component quantities, normal component spatial layout, and subsequent logical constraint construction.

[0017] Preferably, in step S2, the target component or sub-component category is determined based on the composition structure of the target product, visible parts in the normal reference image, and the type of logical anomaly to be detected; the candidate prompt list includes at least the category name of the target component or sub-component and several appearance description prompts generated by the multimodal large language model based on the normal reference image, used to describe the same target component or sub-component from different expressions; the candidate prompts in the candidate prompt list are ordered according to the generation results of the multimodal large language model or the results preset by humans, and are sequentially input into the segmentation model supporting concept prompts in subsequent steps according to the order.

[0018] Preferably, in step S2, the candidate suggestion list is generated by the multimodal large language model based on the target component or sub-component category name, template instructions, and normal reference image; wherein, the template instructions are used to require the multimodal large language model to combine the appearance, color, shape, material, position, and scene context of the target component or sub-component in the normal reference image to output multiple candidate suggestions describing the same target component or sub-component.

[0019] Alternatively, the candidate suggestion list can also be pre-defined manually based on the component definitions in the target industrial scenario.

[0020] Preferably, the segmentation model supporting concept cues described in step S3 is SAM3, or an equivalent model with natural language concept cues segmentation capabilities.

[0021] Preferably, in step S4, for each target component or sub-component category, according to the order of the candidate prompts in the candidate prompt list, it is sequentially determined whether the instance mask set corresponding to each candidate prompt is a valid instance mask set, and the first valid instance mask set is taken as the instance set of the target component or sub-component category; the valid instance mask set is a mask set output by the segmentation model supporting concept prompts for the corresponding candidate prompts, containing at least one component instance mask; the instance sets corresponding to each target component or sub-component category in the image to be detected are summarized to form a test sub-instance set; the instance sets corresponding to each target component or sub-component category in the normal reference image are summarized to form a reference sub-instance set; the component category, mask region, and instance position of each instance mask are recorded.

[0022] Preferably, in step S7, for each level, the number of test instances and the number of reference instances of the same component category under the same parent node are first counted; when the two numbers are inconsistent, the corresponding branch of the level is marked as an anomaly, and the level anomaly score of the current level is set to the preset maximum anomaly value; when the two numbers are consistent, the cost matrix is ​​calculated based on the visual features of the test instances and the visual features of the reference instances, and the Hungarian matching algorithm is used to determine the one-to-one correspondence between the test instances and the reference instances; for each pair of matched instances, the matching anomaly score of the current level is generated according to whether their component categories are consistent and the size of the matching cost; for the pair of matched instances with child components, the number comparison and instance matching are recursively performed at the level of its child components until there are no child components or the preset level depth is reached; the level anomaly scores of each level are summed up by the maximum value to obtain the spatial level matching anomaly score.

[0023] Preferably, in steps S6 to S7, the image encoder is an image encoder using a visual basic model, used to output multi-layer visual features of the image to be detected and the normal reference image; The image encoder is preferably the CLIP ViT-L / 14 image encoder implemented by OpenCLIP, and the image block features output from the 6th, 12th, 18th and 24th Transformer layers are fused to obtain a dense visual feature map.

[0024] For any component instance, a corresponding mask region is determined in the dense visual feature map based on the instance mask of that component instance. The visual features within the mask region are then subjected to average pooling and normalization to obtain the instance visual features of that component instance. For test instances and reference instances within the same level, a cost matrix is ​​calculated based on the similarity between the visual features of the test instance and the visual features of the reference instance. Hungarian matching is then performed based on the cost matrix; where a higher matching cost indicates a greater visual difference between the test instance and the reference instance, and a higher corresponding anomaly score.

[0025] Preferably, in step S8, the logical constraint information includes component combination rules, component geometric statistics rules, and scene topology rules. For component combination rules, it is determined whether the combination of categories, colors, attributes, or functions between related components satisfies the normal correspondence. If not, a combination anomaly score is generated. For component geometric statistics rules, normal statistical values ​​are calculated based on the area, height, length, width, spacing, or angle of the corresponding components in the normal reference image, and it is determined whether the corresponding components in the image to be detected exceed the preset tolerance range. If they exceed the tolerance range, a statistical anomaly score is generated. The maximum value between the combination anomaly score and the statistical anomaly score is taken as the combination statistical anomaly score. For scene topology rules, it is determined whether the component topology in the image to be detected satisfies the normal structure based on the spatial connection relationship, left-right correspondence, endpoint correspondence, or nearest neighbor relationship between component instances. If not, a scene constraint anomaly score is generated.

[0026] Preferably, in step S9, the image-level anomaly score is preferably the maximum value among the spatial hierarchy matching anomaly score, the combined statistical anomaly score, and the scene constraint anomaly score; when the image-level anomaly score is greater than a preset threshold, it is determined that the image to be detected has a logical anomaly; when the image-level anomaly score is not greater than the preset threshold, it is determined that the image to be detected is a normal image.

[0027] Preferably, in steps S8 to S9, the combined anomaly score, statistical anomaly score, and scene constraint anomaly score are used to characterize different types of logical anomaly signals. Specifically, the combined anomaly score characterizes whether the combination of categories, colors, attributes, or functions between related components violates normal correspondence; the statistical anomaly score characterizes the degree of deviation of the component's geometric attributes from the statistical values ​​of the normal reference image; and the scene constraint anomaly score characterizes whether the spatial connection relationship, left-right correspondence relationship, endpoint correspondence relationship, or nearest neighbor relationship between components violates normal topology. The maximum value among the combined anomaly score and the statistical anomaly score is taken as the combined statistical anomaly score, and the maximum value among the spatial hierarchy matching anomaly score, the combined statistical anomaly score, and the scene constraint anomaly score is taken as the image-level anomaly score, so that any spatial hierarchy matching anomaly, component combination anomaly, component geometric statistical anomaly, or scene topology anomaly can be reflected in the final logical anomaly detection result.

[0028] This invention has the following beneficial effects: It constructs a candidate suggestion list using a normal reference image and target industrial scene information, and utilizes a segmentation model that supports concept suggestions to extract component instances from the image to be detected and the normal reference image according to the candidate suggestion list, reducing the problems of missed segmentation, missegmentation, and unstable instance extraction caused by single-category name suggestions; by judging the validity of the instance mask set corresponding to each candidate suggestion and selecting the first valid instance mask set as the instance set of the corresponding category, the test sub-instance set and the reference sub-instance set have a consistent suggestion construction method and instance extraction method, providing a stable foundation for subsequent matching; through image encoder and mask pool... By obtaining the visual features of instances and combining spatial hierarchy, quantity comparison, cost matrix, and Hungarian matching, the system achieves instance correspondence modeling at the component level and sub-component level, enabling the identification of component quantity differences, category mismatches, and local structural anomalies. Logical consistency detection is performed through component combination rules, geometric statistical rules, and scene topology rules, enabling unified judgment of various logical anomalies such as attribute combination conflicts, geometric statistical deviations, and spatial topology anomalies. Furthermore, this invention does not require retraining of the segmentation model or anomaly detection model for the target scene, and can complete industrial image logical anomaly detection with a small number of normal reference images, exhibiting good cross-scene adaptability and deployment convenience. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the overall process of an industrial image logic anomaly detection method based on suggestible concept segmentation and spatial hierarchy matching, according to the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] A method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching includes the following steps: S1. Obtain the image to be detected and at least one normal reference image; In step S1, an image to be detected and a normal reference image are acquired or read in the target industrial scene, and both are used as different inputs for subsequent processing. Specifically, the image to be detected is input into the subsequent component instance extraction and anomaly detection process; the normal reference image is input into the subsequent reference instance extraction, spatial hierarchy matching, and logical constraint construction process. The image to be detected is an industrial image for which logical anomalies need to be determined, and the normal reference image is a sample image without logical anomalies in the target industrial scene.

[0032] S2. Based on the normal reference image and the target industrial scene information, determine the target component or sub-component category involved in the task to be detected, and construct a candidate prompt list for each target component or sub-component category; In step S2, based on the visible components in the normal reference image, the composition structure of the target product, and the type of logical anomaly to be detected, the target component or sub-component category to be involved in logical anomaly detection is determined. For example, for a product containing a container, label, and internal area, the target component or sub-component category may include the container body, label area, and internal filling area; for a product containing connectors, cables, and clamps, the target component or sub-component category may include connectors, cables, clamps, and connection ends. After determining the target component or sub-component category, a candidate prompt list is generated for each category, allowing the same component or sub-component to be described by multiple different expressions. The candidate prompt list includes at least the category name of the target component or sub-component and several appearance description prompts. The appearance description prompts can be generated by a multimodal large language model combined with a normal reference image and templated instructions, or they can be pre-defined manually based on the component definition in the target industrial scene. The candidate prompts in the candidate prompt list are ordered according to the generation result of the multimodal large language model or the result pre-defined manually, and are used to subsequently drive the segmentation model supporting concept prompts to extract component instances in the image to be detected and the normal reference image.

[0033] S3. Using a segmentation model that supports concept cues, perform component instance extraction on the image to be detected and the normal reference image according to the candidate cues list to obtain the instance mask set corresponding to each candidate cues; In step S3, for each target component or sub-component category, candidate prompts are sequentially input into the concept-supporting segmentation model according to the candidate prompt list corresponding to that category. For the input image and candidate prompts, the concept-supporting segmentation model extracts image features and prompt features, obtains conditional feature representations through cross-modal fusion, and outputs an instance mask set corresponding to the candidate prompt by a mask decoder. The above process is performed on the image to be detected to obtain the instance mask set corresponding to each candidate prompt in the image to be detected; the above process is also performed on the normal reference image to obtain the instance mask set corresponding to each candidate prompt in the normal reference image.

[0034] S4. Based on the candidate suggestion list, the validity of the instance mask set corresponding to each candidate suggestion is judged, and a test sub-instance set corresponding to the image to be detected and a reference sub-instance set corresponding to the normal reference image are formed respectively. In step S4, for each target component or sub-component category, the instance mask set corresponding to each candidate prompt is sequentially determined to be a valid instance mask set according to the order in which the candidate prompts appear in the candidate prompt list. When the instance mask set corresponding to a candidate prompt is a valid instance mask set, that instance mask set is used as the instance set of that target component or sub-component category, and the selection of subsequent candidate prompts for that category is stopped. The valid instance mask set is the mask set output by the segmentation model supporting concept prompts for the corresponding candidate prompt, which contains at least one component instance mask.

[0035] The instance sets corresponding to each target component or sub-component category in the image to be detected are summarized to form a test sub-instance set; the instance sets corresponding to each target component or sub-component category in the normal reference image are summarized to form a reference sub-instance set. For each instance mask, its component category, mask area, and instance position are recorded. The test sub-instance set is used to represent the components and sub-components actually extracted in the image to be detected, and the reference sub-instance set is used to represent the normal components and sub-components in the normal reference image, and serves as a reference object for subsequent spatial hierarchy construction, instance matching, and logical consistency detection.

[0036] By using the above method, when a single category name suggestion cannot reliably extract component instances, the appearance description suggestions in the candidate suggestion list can be used to supplement semantic information, thereby improving the stability of component instance extraction.

[0037] S5. Based on the test sub-instance set and the reference sub-instance set, construct the spatial hierarchy of components and sub-components; In step S5, based on the instance masks, instance positions, and component categories in the test sub-instance set and the reference sub-instance set, respectively, a test spatial hierarchy corresponding to the image to be detected and a reference spatial hierarchy corresponding to the normal reference image are constructed. The spatial hierarchy uses the target product or target region as the root node, component instances as first-level nodes, and sub-component instances within a component as lower-level nodes. For two instances, if the mask region of one instance is located inside the mask region of another instance, or its center position is located within the spatial range of another instance, then the two are determined to have a parent-child relationship. For component hierarchies that cannot be determined solely by geometric inclusion relationships, they are determined according to the component hierarchy rules in the target industrial scene or the scene structure rules generated by the multimodal model based on the normal reference image. Through the above method, global components, local components, and sub-components are organized into a recursively matching hierarchical structure.

[0038] S6. Visual features of the image to be detected and the normal reference image are extracted using an image encoder, and at each level of the spatial hierarchy, the visual features are masked and pooled based on the instance masks in the test sub-instance set and the reference sub-instance set to obtain the test instance visual features and reference instance visual features corresponding to each level. In step S6, an image encoder is used to extract features from the image to be detected and the normal reference image respectively, obtaining corresponding multi-layer visual features. In some embodiments, the image encoder uses the CLIP ViT-L / 14 image encoder implemented in OpenCLIP, and fuses the image patch features output from the 6th, 12th, 18th, and 24th layers of the Transformer to obtain a dense visual feature map. Specifically, let the image encoder be in the... The image patch features output by the layer are ,in Indicates the number of image patches. Indicates the feature dimension. The selected... The image patch features output from each feature layer are concatenated along the feature dimension to obtain a fused feature map. The fused feature map is then reshaped and interpolated to the original image resolution to obtain a dense visual feature map. ,in and This represents the image height and width. For any instance in the test sub-instance set or the reference sub-instance set, let its instance mask be... , Represents the dense visual feature map In position The visual features at that location, then the instance visual features of that instance. satisfy:

[0039] in, Indicates the instance mask at position The value at the specified location, Norm indicates normalization. The instance visual features corresponding to each instance in the image to be detected constitute the test instance visual feature set, and the instance visual features corresponding to each instance in the normal reference image constitute the reference instance visual feature set.

[0040] S7. At the corresponding level of the spatial hierarchy, perform quantity comparison and instance matching between the test instances in the image to be detected and the reference instances in the normal reference image. In step S7, within the corresponding levels of the test space hierarchy and the reference space hierarchy, test instances and reference instances are organized according to component categories. For each set of instances of the same category within each level, the number of test instances is first compared with the number of reference instances; when the number of test instances is inconsistent with the number of reference instances, it is determined that there is a quantity anomaly in the current level, and the level anomaly score of the current level is assigned. Set the maximum outlier value as a preset value; when the number of test instances is the same as the number of reference instances, calculate the similarity matrix based on the visual features of the instances obtained in step S6, and construct the cost matrix based on the similarity matrix. Specifically, let the first... The visual features of each test instance are: , No. The visual features of the reference instances are Then the similarity between the two satisfy:

[0041] Where <·,·> represent inner product similarity. Based on the aforementioned similarity... Construct the cost matrix The cost matrix elements in satisfy:

[0042] Based on the cost matrix Perform Hungarian matching to obtain a one-to-one correspondence between test instances and reference instances. For each matched instance pair, its corresponding matching cost is used as the matching anomaly score. If the component categories of the matched instance pair are found to be inconsistent during the matching result verification, the matching anomaly score of that matched instance pair is set to the preset maximum anomaly value. For the current level, the maximum value among the matching anomaly scores of all matched instance pairs is used as the level anomaly score. For matching instance pairs with child components, the above process of quantity comparison, cost matrix calculation, and instance matching is repeated at the child component level. If the number of test instances in any child component level is inconsistent with the number of reference instances, the level anomaly score of the corresponding child component level is adjusted. Set the maximum outlier value to the preset limit. This will ultimately determine the outlier scores for all levels. The maximum value in the range is used as the spatial hierarchical matching anomaly score:

[0043] in, The number of levels in the spatial hierarchy is indicated, and the preset maximum anomaly value is used to represent the maximum anomaly score that indicates the existence of a logical anomaly. In some embodiments, the value is 1.

[0044] S8. Based on the logical constraint information determined by the normal reference image, perform logical consistency detection to obtain the combined statistical anomaly score and the scene constraint anomaly score; In step S8, component combination rules, component geometric statistics rules, and scene topology rules are determined based on the normal reference image, and combination anomaly score, statistical anomaly score, and scene constraint anomaly score are calculated respectively.

[0045] For component combination rules, it is determined whether the combination of categories, colors, attributes, or functions between related components in the image to be detected satisfies the normal correspondence; if the normal correspondence is satisfied, the combination anomaly score is calculated. If the normal correspondence is not satisfied, then the combined abnormal score is used. .

[0046] For component geometric statistics rules, normal statistical values ​​are calculated based on the area, height, length, width, spacing, or angle of the corresponding component in the normal reference image. Let the observed attribute value of the corresponding component in the image to be detected be v, and the mean of this attribute in the normal reference image be v. The standard deviation is Then, outlier scores are counted. satisfy:

[0047] in, For scale parameters, This is to avoid positive numbers with a denominator of zero. If the number of normal reference images is small, it will affect the standard deviation. If unstable, then based on the observed attribute value Does it fall within the mean? Statistical anomaly scores are generated within the preset tolerance range.

[0048] For scene topology rules, the system determines whether the component topology in the image to be detected satisfies the normal structure based on the spatial connection relationship, left-right correspondence, endpoint correspondence, or nearest neighbor relationship between component instances; if it satisfies the normal topology structure, the scene constraint anomaly score is calculated. If the normal topology is not satisfied, the scene constraint anomaly score will be calculated. .

[0049] The combined abnormal score and statistical outlier scores The maximum value in the range is used as the combined statistical outlier score:

[0050] in, The scene constraint anomaly score is used to characterize the maximum degree of anomaly in component combination relationship anomalies and component geometric statistical deviations. Used to characterize whether the topological relationship of a component violates the normal topology.

[0051] S9. The spatial hierarchical matching anomaly score, combined statistical anomaly score, and scene constraint anomaly score are fused to generate an image-level anomaly score, and the logical anomaly detection result of the image to be detected is output based on the image-level anomaly score. In step S9, the spatial hierarchy matching anomaly score obtained in step S7 is used. The combined statistical anomaly score obtained in step S8 and scene constraint anomaly score The images are fused to obtain image-level anomaly scores. In some embodiments, the image-level anomaly score satisfy:

[0052] When the image-level anomaly score When the error score exceeds a preset threshold, a detection result indicating a logical anomaly in the image to be detected is output; when the image-level anomaly score exceeds a preset threshold, a detection result indicating a logical anomaly in the image to be detected is output. If the value is not greater than a preset threshold, the detection result of the image to be detected is output as a normal image. The logical anomaly detection result includes image-level anomaly markers, image-level anomaly scores, and the component level or logical constraint type that triggered the anomaly.

[0053] Performance testing: The effects of the present invention are further illustrated by the following experiments.

[0054] 1) Test conditions

[0055] This invention was tested on a server with an NVIDIA GeForce RTX 4090 graphics card and 24GB of video memory. The development environment was based on Python and the PyTorch deep learning framework. Component instance extraction was implemented using SAM3, and instance visual feature extraction was implemented using the CLIP ViT-L / 14 image encoder implemented by OpenCLIP. No additional parameters were trained for SAM3, the image encoder, or other related models during the testing process.

[0056] 2) Test Content

[0057] This embodiment tests the method of the present invention on the MVTec LOCO AD dataset. The MVTec LOCO AD dataset is a public dataset for industrial logic anomaly detection, containing five categories: Breakfast Box, Juice Bottle, Pushpins, Screw Bag, and Splicing Connectors. These categories respectively contain different component structures, component quantity relationships, combinatorial constraint relationships, and spatial topological relationships, and can be used to evaluate the logic anomaly detection method's ability to identify missing components, inconsistent component quantities, category mismatches, combinatorial constraint conflicts, geometric statistical biases, and topological relationship anomalies.

[0058] This embodiment is tested according to the following process: First, the image to be detected and a normal reference image are acquired, and the target component or sub-component category involved in the detection task is determined based on the normal reference image and the target industrial scene information; then, a candidate prompt list is constructed for each target component or sub-component category, and a segmentation model that supports concept prompts is used to extract component instances from the image to be detected and the normal reference image according to the candidate prompt list; for each category, the first valid instance mask set is used as the instance set corresponding to that category, and test sub-instance set and reference sub-instance set are formed respectively; then, a spatial hierarchy structure is constructed according to the inclusion relationship, positional relationship and semantic relationship between components and sub-components, and quantity comparison, cost matrix calculation and Hungarian matching are performed at the corresponding level; finally, the spatial hierarchy matching anomaly score, the combined statistical anomaly score and the scene constraint anomaly score are fused to generate an image-level anomaly score, and the logical anomaly detection result is output according to the image-level anomaly score.

[0059] This embodiment uses F1-max and AUROC as evaluation metrics. For F1-max, the proportion of truly anomalous images among all images judged as anomalous is considered precision; the proportion of correctly identified anomalous images among all truly anomalous images is considered recall. The F1 score is used to comprehensively measure precision and recall, and F1-max represents the maximum F1 score obtained under different anomaly judgment thresholds, used to evaluate the overall detection capability of the method under optimal threshold conditions. For AUROC, a receiver operating characteristic curve is plotted with false positive rate on the horizontal axis and true positive rate on the vertical axis. AUROC represents the area under the curve, used to measure the overall ability of the method to distinguish normal images from logically anomalous images under different judgment thresholds. The higher the values ​​of F1-max and AUROC, the better the logical anomaly detection performance. Table 1 shows the logical anomaly detection results of the method of this invention on different categories of the MVTec LOCO AD dataset.

[0060] Table 1. Logical anomaly detection results of the method of the present invention on different categories of the MVTec LOCO AD dataset.

[0061] As shown in Table 1, the method of this invention achieved corresponding logical anomaly detection results in five categories of the MVTec LOCO AD dataset. Specifically, the Breakfast Box and Juice Bottle categories include containers, region partitions, and hierarchical relationships between internal components; the Pushpins and Screw Bag categories include multiple repeating components and their quantity, category, and positional relationships; and the Splicing Connectors category includes topological constraints between clamps, cables, and connectors. Experimental results demonstrate that by using SAM3 for suggestible concept segmentation, combined with a candidate suggestion list, spatial hierarchical instance matching, and logical consistency detection, logical anomalies in industrial images can be detected with a small number of samples and without requiring additional training for the target scene.

[0062] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching, characterized in that, Includes the following steps: S1. Obtain the image to be detected and at least one normal reference image; S2. Based on the normal reference image and the target industrial scene information, determine the target component or sub-component category involved in the task to be detected, and construct a candidate prompt list for each target component or sub-component category; S3. Using a segmentation model that supports concept cues, component instance extraction is performed on the image to be detected and the normal reference image according to the candidate cues list to obtain the instance mask set corresponding to each candidate cues. For the input image and the candidate cues, the segmentation model that supports concept cues extracts image features and cues features, obtains conditional feature representations through cross-modal fusion, and outputs the instance mask set corresponding to the candidate cues by the mask decoder. S4. Based on the candidate suggestion list, the validity of the instance mask set corresponding to each candidate suggestion is judged, and a test sub-instance set corresponding to the image to be detected and a reference sub-instance set corresponding to the normal reference image are formed respectively. S5. Based on the test sub-instance set and the reference sub-instance set, construct a spatial hierarchy structure of components and sub-components; wherein, the spatial hierarchy structure takes the target product or target area as the root node, the component instance as the first-level node, and the sub-component instances within the component as the lower-level nodes, and determines the parent-child node relationship according to the inclusion relationship between instance masks, the relationship of instance center positions, and the component hierarchy relationship in the target industrial scene; the component hierarchy relationship can be determined by manually preset component hierarchy rules, or it can be determined by the scene structure rules generated by the multimodal model based on the normal reference image; S6. Visual features of the image to be detected and the normal reference image are extracted using an image encoder, and at each level of the spatial hierarchy, the visual features are masked and pooled based on the instance masks in the test sub-instance set and the reference sub-instance set to obtain the test instance visual features and reference instance visual features corresponding to each level; wherein, the image encoder is used to output the multi-layer visual features of the image, and the mask pooling is used to aggregate the visual features within the coverage area of ​​each component instance mask into the corresponding instance visual features. S7. At the corresponding level of the spatial hierarchy, perform quantity comparison and instance matching between the test instances in the image to be detected and the reference instances in the normal reference image. S8. Based on the logical constraint information determined by the normal reference image, perform logical consistency detection to obtain the combined statistical anomaly score and the scene constraint anomaly score; S9. The spatial hierarchical matching anomaly score, combined statistical anomaly score, and scene constraint anomaly score are fused to generate an image-level anomaly score, and the logical anomaly detection result of the image to be detected is output based on the image-level anomaly score.

2. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, The image to be detected in step S1 is an industrial image for which it is necessary to determine whether there is a logical anomaly. The normal reference image is a sample image without logical anomalies in the target industrial scene, used to provide reference information for normal component categories, normal component quantities, normal component spatial layout, and subsequent logical constraint construction.

3. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, In step S2, the target component or sub-component category is determined based on the composition structure of the target product, visible parts in the normal reference image, and the type of logical anomaly to be detected. The candidate suggestion list includes at least the category name of the target component or sub-component and several appearance description suggestions generated by the multimodal large language model based on the normal reference image, used to describe the same target component or sub-component from different expressions. The candidate suggestions in the candidate suggestion list are ordered according to the generation results of the multimodal large language model or the results preset by humans, and are sequentially input into the segmentation model supporting concept suggestions in subsequent steps according to the order of the generated suggestions. The candidate suggestion list mentioned in step S2 is generated by a multimodal large language model based on the target component or sub-component category name, template instructions, and normal reference image; wherein, the template instructions are used to require the multimodal large language model to combine the appearance, color, shape, material, position, and scene context of the target component or sub-component in the normal reference image to output multiple candidate suggestions describing the same target component or sub-component; or, the candidate suggestion list is pre-set manually according to the component definition in the target industrial scene.

4. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, The segmentation model supporting concept cues described in step S3 is preferably SAM3, or an equivalent model with natural language concept cues segmentation capabilities.

5. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, In step S4, for each target component or sub-component category, according to the order of the candidate prompts in the candidate prompt list, it is sequentially determined whether the instance mask set corresponding to each candidate prompt is a valid instance mask set, and the first valid instance mask set is taken as the instance set of the target component or sub-component category; the valid instance mask set is the mask set output by the segmentation model supporting concept prompts for the corresponding candidate prompt, which contains at least one component instance mask; the instance sets corresponding to each target component or sub-component category in the image to be detected are summarized to form a test sub-instance set; The set of instances corresponding to each target component or sub-component category in the normal reference image is summarized to form a reference sub-instance set; for each instance mask, its component category, mask area and instance position are recorded.

6. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, In steps S6 to S7, the image encoder adopts a visual basic model image encoder to output multi-layer visual features of the image to be detected and the normal reference image. The image encoder is a CLIP ViT-L / 14 image encoder implemented with OpenCLIP, and it fuses the image patch features output from the 6th, 12th, 18th and 24th Transformer layers to obtain a dense visual feature map. For any component instance, the corresponding mask region is determined in the dense visual feature map based on the instance mask of the component instance, and the visual features in the mask region are subjected to average pooling and normalization to obtain the instance visual features of the component instance. For test instances and reference instances within the same level, a cost matrix is ​​calculated based on the similarity between the visual features of the test instance and the visual features of the reference instance, and Hungarian matching is performed based on the cost matrix; wherein, the larger the matching cost, the greater the visual difference between the test instance and the reference instance, and the higher the corresponding matching anomaly score.

7. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, In step S7, for each level, the number of test instances and reference instances of the same component category under the same parent node are first counted. When the two numbers are inconsistent, the corresponding branch of the level is marked as an anomaly, and the level anomaly score of the current level is set to the preset maximum anomaly value. When the two numbers are consistent, the cost matrix is ​​calculated based on the visual features of the test instances and the visual features of the reference instances, and the Hungarian matching algorithm is used to determine the one-to-one correspondence between the test instances and the reference instances. For each pair of matched instances, the matching anomaly score of the current level is generated according to whether their component categories are consistent and the size of the matching cost. For a pair of matched instances with child components, the number comparison and instance matching are recursively performed at the child component level until there are no child components or the preset level depth is reached. The maximum value of the level anomaly scores of each level is summed to obtain the spatial level matching anomaly score.

8. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 1, characterized in that, In step S8, the logical constraint information includes component combination rules, component geometric statistics rules, and scene topology rules. For component combination rules, it is determined whether the combination of categories, colors, attributes, or functions between related components satisfies the normal correspondence. If not, a combination anomaly score is generated. For component geometric statistics rules, normal statistical values ​​are calculated based on the area, height, length, width, spacing, or angle of the corresponding component in the normal reference image, and it is determined whether the corresponding component in the image to be detected exceeds the preset tolerance range. If the value exceeds the limit, a statistical outlier score is generated; the maximum value between the combined outlier score and the statistical outlier score is taken as the combined statistical outlier score. For scene topology rules, the system determines whether the component topology in the image to be detected satisfies the normal structure based on the spatial connection relationship, left-right correspondence, endpoint correspondence, or nearest neighbor relationship between component instances; if it does not satisfy the normal structure, a scene constraint anomaly score is generated.

9. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 8, characterized in that, In step S9, the image-level anomaly score is preferably the maximum value among the spatial hierarchy matching anomaly score, the combined statistical anomaly score, and the scene constraint anomaly score; when the image-level anomaly score is greater than a preset threshold, it is determined that the image to be detected has a logical anomaly; when the image-level anomaly score is not greater than the preset threshold, it is determined that the image to be detected is a normal image.

10. The method for detecting logical anomalies in industrial images based on suggestible concept segmentation and spatial hierarchy matching as described in claim 9, characterized in that, In steps S8 to S9, the combined anomaly score, statistical anomaly score, and scene constraint anomaly score are used to characterize different types of logical anomaly signals. Specifically, the combined anomaly score characterizes whether the combination of categories, colors, attributes, or functions between related components violates normal correspondence; the statistical anomaly score characterizes the degree of deviation of the component's geometric attributes from the statistical values ​​of the normal reference image; and the scene constraint anomaly score characterizes whether the spatial connection relationship, left-right correspondence, endpoint correspondence, or nearest neighbor relationship between components violates normal topology. The maximum value among the combined anomaly score and the statistical anomaly score is taken as the combined statistical anomaly score, and the maximum value among the spatial hierarchy matching anomaly score, the combined statistical anomaly score, and the scene constraint anomaly score is taken as the image-level anomaly score, so that any spatial hierarchy matching anomaly, component combination anomaly, component geometric statistical anomaly, or scene topology anomaly can be reflected in the final logical anomaly detection result.

Images