An Unsupervised Image Anomaly Detection System and Method Based on Semantic Multimodal Background Modeling
This unsupervised image anomaly detection method based on semantic multimodal background modeling solves the problem of false detection caused by the multimodal characteristics of the background in industrial visual quality inspection. It achieves high-precision and low-false-detection anomaly detection and is suitable for unsupervised inspection of complex backgrounds and rapidly iterating products.
Patent Information
- Application Number
- CN202611189484.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-03-30
- Filing Date
- 2026-08-06
- Publication Date
- 2026-11-10
Smart Images

Figure CN123071431A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and intelligent industrial inspection technology, and in particular to an unsupervised image anomaly detection system and method based on semantic multimodal background modeling. Background Technology
[0002] In the field of industrial visual quality inspection, unsupervised anomaly detection technology has become a research hotspot in academia and industry due to its advantage of not requiring a large number of anomaly-labeled samples. This type of method typically models a large number of normal samples to learn the distribution or representation of normal images. In the inference phase, it generates a pixel-level raw anomaly score map by calculating the difference between the test sample and the built normal model, and then segments out the anomaly regions using a global threshold or a simple region normalization strategy. However, the above-mentioned technical solutions still face the following shortcomings in practical industrial applications: (1) The multimodal characteristics of the background region itself lead to false detections: Industrial product images often contain complex background structures, such as screw holes, seams, heat dissipation teeth, reflective areas, etc. These different functional regions have significant differences in visual texture and reflective properties, resulting in different baseline levels of the anomaly scores generated by the same anomaly detection model in different background regions. For example, the edge response of reflective areas may be naturally higher than that of matte panel areas. In this case, if a single global threshold is used for segmentation, normal background regions with high response are very likely to be misjudged as anomalies; even if simple partitioning normalization is used, if the partitioning is based only on the low-level features of the image rather than engineering semantics, it is impossible to fundamentally solve the problem of inconsistent score scales between different background modes.
[0003] (2) The reliance on foreground priors limits the universality of the methods: To improve detection accuracy, some advanced post-processing methods introduce an initial foreground mask as a guide to suppress background interference. However, in real production line environments, especially in scenarios with rapid iteration of new product models, it is often impossible to obtain any labeled information or reliable foreground priors in advance. This reliance on foreground masks makes it difficult to deploy and apply the above methods in purely unsupervised scenarios with zero labels.
[0004] (3) Existing modeling granularity cannot adapt to the internal heterogeneity of the background: Mainstream unsupervised anomaly detection methods usually model the image as a whole or simply distinguish between the foreground and the background, failing to deeply explore the unique distribution patterns of different semantic regions within the background. This coarse-grained modeling approach makes it difficult for the model to accurately distinguish between normal high responses and true local anomalies within a region, thus limiting the upper limit of detection accuracy.
[0005] Therefore, there is an urgent need for an anomaly detection mechanism that is completely unsupervised, does not rely on any prior information, and can perform fine-grained modeling of multimodal characteristics within complex backgrounds, so as to achieve high-precision, low-false-detection defect localization and segmentation in real industrial scenarios. Summary of the Invention
[0006] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies, such as ignoring background multimodal characteristics and falsely detecting normal high-response regions, and relying on foreground priors. This invention provides an unsupervised image anomaly detection system and method based on semantic multimodal background modeling. Without requiring any foreground priors or anomaly annotations, it accurately models the normal patterns of different semantic regions within the background, achieving localized anomaly determination and effectively avoiding false detections caused by background multimodal characteristics. Furthermore, its lightweight modular design allows for flexible integration into any existing detection process, providing a high-precision, low-false-detection, easy-to-deploy, and highly versatile unsupervised anomaly detection solution for real-world industrial environments.
[0007] The technical solution adopted by this invention to solve its technical problem is: an unsupervised image anomaly detection method based on semantic multimodal background modeling, comprising the following steps: Obtain input data, which includes template library diagrams and the diagram to be tested; Workpiece semantic region extraction is performed on the image under test to obtain the background semantic partition mask set. ;in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image; The template library graph and the graph under test are processed using an anomaly segmentation model to generate the original anomaly score graph of the graph under test. ; Original anomaly score map and background semantic partition mask set The input is fed into a multimodal semantic region threshold predictor, which performs the following operations: For each background sub-region Based on the original anomaly score map and the corresponding mask Region-aware normalization is performed to obtain a normalized fractional map. ; Normalized fractional graph and its mask Input to a sub-region with background Corresponding, trainable region-specific threshold prediction subnetwork Generate a spatially adaptive threshold map. The background sub-region is calculated through differentiable soft binarization. Anomaly probability diagram ; Integrate all Anomaly probability map of each background sub-region The final anomaly probability map is obtained. ; Based on the final anomaly probability plot Obtain the anomaly mask and generate a binarized anomaly region mask image; Extract the abnormal location from the abnormal region mask image and output the outline or location box information of the abnormal region.
[0008] By acquiring input data, extracting the semantic region of the workpiece to obtain a background semantic partition mask set, generating an original anomaly score map through an anomaly segmentation model, and inputting both into a semantic multimodal threshold prediction module to sequentially perform region-aware normalization, region-specific threshold prediction, differentiable soft binarization, and anomaly fusion to output the final anomaly probability map, and then through the steps of acquiring anomaly masks and extracting anomaly locations, a complete, end-to-end, unsupervised image anomaly detection method based on background multimodal modeling is realized.
[0009] Furthermore, the background semantic partitioning mask set Satisfying mutually exclusive and fully covered constraints: , where 1 is a full-1 tensor of the same size as the input image.
[0010] By limiting the background semantic partition mask set to satisfy mutually exclusive and full coverage constraints, each pixel in the image is uniquely assigned to a background sub-region, laying a standardized and unambiguous partitioning foundation for subsequent refined modeling.
[0011] Furthermore, the region-aware normalization process specifically includes: Calculate the first Mean local anomaly score of each background sub-region : , in, For the original anomaly score map at location The value, For the mask at position The indicated value, To prevent division by zero of small constants; Based on the mean of local anomaly scores and learnable parameters For the original anomaly score map Perform nonlinear normalization to obtain the first... Normalized score map of each background subregion : , in, .
[0012] By calculating the mean local anomaly score for each background sub-region and performing nonlinear normalization on the original anomaly score map based on this mean and learnable parameters, adaptive adjustment to eliminate score scale differences between different background regions is achieved.
[0013] Furthermore, the region-specific threshold prediction subnetwork This is a lightweight convolutional neural network, and its network structure includes: First layer: a two-dimensional convolutional layer with a kernel size of 3×3 and 64 output channels, followed by batch normalization and ReLU activation function; The second layer is a two-dimensional convolutional layer with a kernel size of 3×3 and 32 output channels, followed by batch normalization and ReLU activation function; The third layer is a two-dimensional convolutional layer with a kernel size of 1×1 and an output channel of 1, followed by a Sigmoid activation function.
[0014] By configuring a lightweight neural network consisting of three convolutional layers as a region-specific threshold prediction subnetwork for each background sub-region, the ability to generate spatially adaptive threshold maps for different background regions is achieved.
[0015] Furthermore, the region-specific threshold prediction subnetwork The training was conducted under unsupervised conditions, using only normal samples. The training process included: Freeze the anomaly segmentation model used to generate the original anomaly score map, and update only the region-specific threshold prediction subnetwork. and learnable parameters ; Constructing an unsupervised loss function This is used to minimize the final anomaly probability map corresponding to normal samples. Overall response: , in, For batch size, This refers to the image resolution.
[0016] By freezing the anomaly segmentation model during unsupervised training, updating only the threshold prediction subnetwork and learnable parameters, and constructing a loss function that minimizes the overall response of the final anomaly probability map of normal samples, pure unsupervised training without any anomaly labeling or foreground masking is achieved.
[0017] Furthermore, the differentiable soft binarization operation is as follows: , in, Normalized fractional graph The pixel values in For spatially adaptive threshold maps Threshold at the corresponding position The preset transition band width; the first Anomaly probability map of each background sub-region Obtained through the following formula: .
[0018] By defining a differentiable soft binarization function that includes a transition band, and calculating the anomaly probability map of each sub-region based on the normalized score map, threshold map, and mask, end-to-end differentiable soft thresholding segmentation is achieved, avoiding the training instability problem caused by hard thresholding.
[0019] Furthermore, integrate all Anomaly probability maps for each background sub-region are generated by taking the maximum value pixel by pixel. .
[0020] By fusing the anomaly probability maps of all background sub-regions pixel by pixel and taking the maximum value, a localized anomaly determination logic is achieved where each pixel is determined independently by the model of its respective background sub-region.
[0021] Furthermore, the anomaly segmentation model includes an image feature extractor and a similarity calculator, which generate the original anomaly score map in the following manner. : The template library image and the image to be tested are input into the image feature extractor to obtain the corresponding feature representations. and ; Calculate using a similarity calculator and The similarity is used to generate the original anomaly score map. .
[0022] This paper presents a specific implementation method for an optional anomaly segmentation model by using an image feature extractor to extract feature representations of the template image and the image to be tested, and generating an anomaly score map as the original anomaly score map using a similarity calculator.
[0023] Furthermore, the abnormal locations are extracted from the abnormal region mask image, specifically including: Perform connectivity analysis on the mask image of abnormal regions; Execute the contour extraction algorithm to obtain the set of closed contour coordinates of all abnormal regions marked as foreground; The contour is filtered based on the contour area or the size of the circumscribed rectangle to remove small or invalid contours caused by noise; Output the contour coordinates of the valid abnormal region or the location box information defined by the circumscribed rectangle.
[0024] By performing connected component analysis, contour extraction, and geometric feature-based contour filtering on the mask map of abnormal regions, a complete post-processing solution is achieved to transform the anomaly probability map into defect contours and location box information that can be used for quality inspection.
[0025] A system for implementing the unsupervised image anomaly detection method based on semantic multimodal background modeling as described above includes: The semantic region extraction module is used to extract the semantic region of the workpiece from the image under test, and obtain the background semantic partition mask set. ;in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image; Anomaly segmentation models are used to process template library graphs and the graph under test to generate raw anomaly score maps of the graph under test. ; The semantic multimodal threshold prediction module, connected to the semantic region extraction module and the anomaly segmentation model, is used to receive the original anomaly score map. and background semantic partition mask set And output the final anomaly probability map. The semantic multimodal threshold prediction module includes: Region-aware normalization units are used to normalize each background sub-region. Based on the original anomaly score map and the corresponding mask Region-aware normalization is performed to obtain a normalized fractional map. ; Each region-specific threshold prediction subnetwork is connected to a region-aware normalization unit. Corresponding to a background sub-region Used to receive normalized fractional graphs and its mask It outputs a spatially adaptive threshold map. ; Differentiable soft binarization unit, and region-aware normalization unit and Each region-specific threshold prediction subnetwork is connected to, used to predict thresholds based on the normalized score map. Spatial adaptive threshold map and mask Each background sub-region is calculated through a differentiable soft binarization operation. Anomaly probability diagram ; Anomaly fusion units, connected to differentiable soft binarization units, are used to fuse all... Anomaly probability map of each background sub-region Output the final anomaly probability map. ; The post-processing module, connected to the semantic multimodal threshold prediction module, is used to predict the anomaly probability based on the final anomaly probability map. Obtain the anomaly mask, generate a binarized anomaly region mask image, extract the anomaly location from the anomaly region mask image, and output the contour or location box information of the anomaly region.
[0026] An electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the unsupervised image anomaly detection method based on semantic multimodal background modeling described above.
[0027] A computer-readable storage medium is also provided, on which a computer program is stored, characterized in that, when the program is executed by a processor, it implements the unsupervised image anomaly detection method based on semantic multimodal background modeling described above.
[0028] The beneficial effects of this invention are: This invention does not rely on any prior foreground information or abnormal sample annotations during the training and inference phases. During training, it only needs to collect easily obtainable normal product images. By constructing an unsupervised loss function that minimizes the abnormal response of normal samples, the model is driven to autonomously learn the normal fluctuation range and abnormal judgment threshold of each background sub-region. During inference, any image to be tested is processed directly without the need to pre-estimate abnormal regions or input foreground masks. This allows the invention to be directly deployed in real production line environments, and is especially suitable for harsh scenarios such as rapid iteration of new product models, scarcity of abnormal samples, and zero-labeled data. It significantly shortens the data preparation cycle and improves development efficiency and production line changeover response speed.
[0029] This invention divides the image background into multiple mutually exclusive sub-regions based on engineering semantics, independently calculates local statistical features for each sub-region, and eliminates the inherent fractional scale differences between different regions through learnable nonlinear normalization. On this basis, an independent threshold prediction network is configured for each sub-region to generate a spatially adaptive threshold map, which can accurately characterize the normal fluctuation range under different texture patterns. This enables the model to accurately distinguish between normal high responses and true local anomalies within the region, effectively avoiding false detection problems caused by the multimodal characteristics of the background.
[0030] This invention employs a localized anomaly determination logic. The final anomaly probability map is obtained by fusing the maximum value of the anomaly probability maps of each sub-region pixel by pixel. That is, the anomaly of each pixel is determined independently only by the normal mode of its background sub-region. No cross-mode comparison is performed between different regions. This localized determination ensures the contextual relevance of the anomaly definition, making the detection results more reasonable in a physical sense and significantly improving the reliability of the model in detecting complex structural workpieces.
[0031] This invention employs a lightweight, modular design. The region-aware normalization unit and the region-specific threshold prediction subnetwork have extremely small parameter counts and low computational overhead, allowing them to be embedded as plug-and-play post-processing plugins into any existing unsupervised anomaly detection workflow. Regardless of whether the method is based on template comparison, feature reconstruction, or distribution modeling, the output original anomaly score map can be directly used as input to this invention. After optimization by the semantic multimodal threshold prediction module, more accurate anomaly detection results are obtained. This modular design ensures the method's versatility and scalability, enabling it to interface with continuously evolving industrial detection technologies. Attached Figure Description
[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0033] Figure 1 This is a structural block diagram of the system of the present invention.
[0034] Figure 2 This is a structural block diagram of the method of the present invention.
[0035] Figure 3 This is a block diagram of the multimodal semantic region threshold predictor in the method of this invention.
[0036] Figure 4 This is a schematic diagram of the background partition of a specific example of the present invention.
[0037] Figure 5 This is a comparison diagram of the results of the method of the present invention and existing methods that do not distinguish between semantic regions for anomaly detection. Detailed Implementation
[0038] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0039] Example 1: like Figure 1 As shown, an unsupervised image anomaly detection system based on semantic multimodal background modeling includes a semantic region extraction module, an anomaly segmentation model, a semantic multimodal threshold prediction module, and a post-processing module. The semantic region extraction module extracts the semantic regions of the image under test and outputs a background semantic partition mask set. .in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image.
[0040] The semantic region of a workpiece refers to a localized area on an industrial part or product that has specific engineering or functional significance, such as a positioning reference surface, bolt mounting holes, heat dissipation fins, electrical interfaces, or brand identification areas. These areas not only have clear geometric boundaries but also carry the technological semantics necessary for design, manufacturing, assembly, or quality inspection processes. The core objective of extracting these semantic regions is to transform human-understood engineering knowledge into spatial information that machines can accurately recognize and manipulate, thereby providing structured input for subsequent automated processes.
[0041] For semantic region extraction, an appropriate technical path can be selected based on the constraints of the application scenario. In situations with diverse workpiece types and a lack of fixed design rules, data-driven adaptive methods are typically used, such as cluster analysis of image features or semantic segmentation using deep learning models. Algorithms learn the visual patterns and semantic associations of regions from samples. However, in typical large-scale, high-paced industrial settings, most workpieces are standardized and serialized, with relatively fixed positions, shapes, and dimensions of their functional areas. Therefore, a more direct and reliable method is often employed: based on the workpiece's CAD drawings or reference images, precise masks for each functional area are manually drawn in advance, forming reusable semantic templates. This method embeds domain knowledge into the template, ensuring unambiguous semantic definitions and zero deviation in positional information.
[0042] Regardless of the extraction method used, obtaining an accurate semantic region of the workpiece can bring significant benefits to downstream tasks. It provides a clear area to be inspected for the workpiece and provides stable guidance for industrial anomaly judgment. Figure 4 A specific example is given, in which an image of an industrial product is divided into six mutually exclusive background sub-regions according to its functional structure: background area, left edge of the workpiece, workpiece panel area, hole area, right functional area, and constant / variable character area.
[0043] Background semantic partition mask set Mutually exclusive and fully covered constraints: , where 1 is a tensor of all 1s of the same size as the input image. This constraint ensures that each pixel in the image is assigned to exactly one and only one background sub-region, meaning that the sub-regions are mutually exclusive and together cover the entire image, with no omissions or overlaps.
[0044] The anomaly segmentation model is used to process the template library graph and the graph under test to generate the original anomaly score graph of the graph under test. The core objective of anomaly segmentation models is to generate a pixel-level anomaly score map to quantify the probability of an anomaly at each location in the image under test.
[0045] To achieve this goal, mainstream methods primarily rely on two technical paths: one is a template-based comparison path, which measures differences by calculating the similarity or reconstruction residual between the test image and the normal template in the feature space; the other is a distribution-based modeling path, which determines anomalies by assessing the probability that the test feature deviates from the normal feature distribution. Regardless of whether a supervised, unsupervised, or zero-shot implementation is used, the final output is a high-resolution anomaly score map, providing a direct and reliable quantitative basis for subsequent threshold segmentation and precise localization.
[0046] This embodiment provides an example using a template comparison-based technique: the workpiece image is 2456×1800 pixels, and the template image and the image to be tested are the same size, both with a resolution of 2456×1800. The anomaly segmentation model consists of an image feature extractor and a similarity calculator. The image to be tested and the template image are processed by the feature extractor to obtain their corresponding feature representations. , The similarity calculator calculates the similarity between the image to be tested and the template image. The smaller the similarity value, the more abnormal it is; the larger the value, the more normal it is. Finally, an anomaly map between the template image and the image to be tested is obtained. , represented as ,in, For batch size, This refers to the image resolution.
[0047] The semantic multimodal threshold prediction module is the core innovation of this invention. It is connected to the semantic region extraction module and the anomaly segmentation model and is used to receive the original anomaly score map. and background semantic partition mask set And output the final anomaly probability map. .like Figure 3 As shown, this module specifically includes: a region-aware normalization unit, Each region-specific threshold prediction subnetwork, differentiable soft binarization unit, and anomaly fusion unit.
[0048] Region-aware normalization units are used for each background sub-region Based on the original anomaly score map and the corresponding mask Region-aware normalization is performed to obtain a normalized fractional map. ; First, calculate the... Mean local anomaly score of each background sub-region : , in, For the original anomaly score map at location The value at that location, To determine whether the corresponding position belongs to the first Background indicator value (1 indicates belonging, 0 indicates not belonging). The numerator represents the sum of the anomaly scores of all pixels in the region, and the denominator represents the total number of pixels in the region; To prevent small constants from being divided by zero.
[0049] Subsequently, based on the mean of local outlier scores and learnable parameters For the original anomaly score map Perform nonlinear normalization to obtain the first... Normalized score map of each background subregion : , in, These are learnable parameters that control the sensitivity of the normalization function. The main body of the function is a scaled Sigmoid function, which... Mapping to the range of approximately [0.3, 0.8] results in pixels slightly above the mean being moderately enhanced within this background region, while pixels much above the mean approach saturation, thus highlighting local anomalies. At the same time, it suppresses the fractional scale differences between different background regions, fundamentally eliminating the inherent fractional scale inconsistency problem between different background patterns, and laying the foundation for subsequent threshold determination.
[0050] This embodiment includes Region-specific threshold prediction subnetwork Each region has its own dedicated threshold prediction subnetwork Corresponding to a background sub-region Connected to the region-aware normalization unit, it is used to receive the normalized fractional map. and its mask It outputs a spatially adaptive threshold map. .
[0051] The threshold prediction subnetwork employs a lightweight convolutional neural network structure, specifically including: First layer: A two-dimensional convolutional layer with a kernel size of 3×3 and 64 output channels, denoted as... This is followed by Batch Normalization (BN) and the ReLU activation function; The second layer: a two-dimensional convolutional layer with a kernel size of 3×3 and 32 output channels, denoted as... Similarly, it is followed by BN and ReLU activation functions; The third layer: a two-dimensional convolutional layer with a kernel size of 1×1 and 1 output channel, denoted as... The output value is then compressed to the interval [0,1] by the Sigmoid activation function, which serves as the final spatial adaptive threshold map.
[0052] All 3×3 convolutional layers employ a zero-padding strategy with padding=1 to ensure consistent spatial resolution between the input and output feature maps. This network structure has a very small number of parameters and low computational cost, making it a plug-and-play module that can be embedded into any backbone anomaly detection process. The prediction branches corresponding to each background sub-region can share network weights to further reduce model complexity, or they can be set independently to improve the ability to model different texture patterns.
[0053] Differentiable soft binarization unit and region-aware normalization unit and Each region-specific threshold prediction subnetwork is connected to, used to predict thresholds based on the normalized score map. Threshold map and mask Each background sub-region is calculated through a differentiable soft binarization operation. Anomaly probability diagram .
[0054] The definition of a differentiable soft binarization function is: , in, Normalized fractional graph The pixel values in For spatially adaptive threshold maps Threshold at the corresponding position The preset transition band width; for example =0.1. When the fraction Significantly higher than the threshold ( When the score is significantly below the threshold, the output is 1, indicating an anomaly; when the score is significantly below the threshold, the output is 1. When the output is 0, it is considered normal; in the intermediate transition interval [ , The output changes linearly. This design ensures that the function is differentiable everywhere, enabling the entire network to support end-to-end gradient backpropagation, facilitating joint training, and avoiding training instability caused by hard thresholding.
[0055] No. Anomaly probability map of each background sub-region Obtained through the following formula: , in, This indicates element-wise multiplication by a mask. Ensure that the anomaly probability is calculated only within the current background sub-region.
[0056] The anomaly fusion unit is connected to a differentiable soft binarization unit to fuse the anomaly probability maps of all K background sub-regions. The final anomaly probability map .
[0057] The blending operation specifically involves taking the maximum value pixel by pixel: This operation achieves localized anomaly detection: the anomaly of each pixel is determined independently by the model of its respective background sub-region, without comparison with other regions. This strategy achieves localized anomaly detection, effectively adapting to the multimodal characteristics of complex backgrounds. A pixel is judged as defective as long as it exhibits sufficiently high anomaly in its normal background mode, without comparison with other regions, which is more in line with physical intuition and engineering logic.
[0058] The semantic multimodal threshold prediction module contains learnable parameters (in region-aware normalization). The weights of the region-specific threshold prediction subnetwork need to be determined through training. Crucially, this training process can be completed under completely unsupervised conditions, relying solely on normal samples without any anomaly annotations or foreground masks, which aligns with real-world industrial deployment scenarios.
[0059] The specific training process is as follows: (i) Collect good product data and input it into the anomaly detection model to obtain the corresponding raw anomaly score map; (ii) Obtain the semantic region of the workpiece; (iii) Formulate the unsupervised optimization objective: On normal images, ideally, the final anomaly probability map should approach zero, i.e., there should be no anomalous response. Therefore, the unsupervised loss function is defined as: This loss encourages the model to output the lowest possible anomaly confidence on normal samples.
[0060] (iv) Training strategy: The backbone anomaly detection model is kept frozen, and only the learnable parameters in the region-specific threshold prediction subnetwork and the normalization module are trained. The optimizer can be either Adam or SGD, and the learning rate is set to... Once trained, the model can be used for any image during the inference phase.
[0061] This technique primarily works by minimizing anomalous responses on normal data; the threshold predictor automatically learns the threshold at the [missing information]. In the context of the background, what level of normalization score is still considered within the normal fluctuation range, thus assigning reasonable adaptive thresholds to different texture regions.
[0062] The post-processing module is connected to the semantic multimodal threshold prediction module to predict the anomaly probability based on the final anomaly probability map. Obtain the anomaly mask, generate a binarized anomaly region mask image, extract the anomaly location from the anomaly region mask image, and output the contour or location box information of the anomaly region.
[0063] First, the final anomaly probability map... The image is converted into a binary anomaly mask. Specifically, during deployment, areas with anomaly probabilities greater than 0.5 are set as foreground (abnormal), and those less than or equal to 0.5 are set as background (normal). Through this process, an anomaly mask image of the same size as the input image is generated, clearly distinguishing suspected abnormal areas from the normal background in binary form, laying the foundation for subsequent precise location positioning.
[0064] Secondly, the spatial contour information of the abnormal regions is accurately located and output from the binary anomaly mask. Specifically, connected component analysis is performed on the obtained anomaly mask image, and a contour extraction algorithm is executed to obtain the set of closed contour coordinates of all anomaly regions marked as foreground. Each contour represents the boundary pixel position of an independent anomaly region. The processing may include a contour filtering step, such as filtering out small or invalid contours that may be caused by noise based on geometric features such as contour area and bounding rectangle size, to improve the robustness of the detection results. Finally, the contour coordinates of all extracted valid anomaly regions, or the location box information defined by their bounding rectangles, are used as the anomaly detection results of this method, and are structuredly output and returned to the calling side for subsequent display, recording, or decision-making processes.
[0065] Example 2: like Figure 2 As shown, the unsupervised image anomaly detection method based on semantic multimodal background modeling includes the following steps: Step 1: Input Data There are two types of input data: template images and images to be tested. Template images refer to images captured at fixed points on normal workpieces; here, the template image is a good product image, and the template image library typically uses a normal workpiece for pre-collection. Images to be tested refer to images of the workpiece taken at fixed points to determine whether it is abnormal. In this embodiment, the image size is 2456×1800 pixels, with RGB three channels.
[0066] Step 2: Extract the semantic region of the workpiece Workpiece semantic region extraction is performed on the image under test to obtain the background semantic partition mask set. ;in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image.
[0067] The semantic region of a workpiece refers to a localized area on an industrial part or product that has specific engineering or functional significance, such as a positioning reference surface, bolt mounting holes, heat dissipation fins, electrical interfaces, or brand identification areas. These areas not only have clear geometric boundaries but also carry the technological semantics necessary for design, manufacturing, assembly, or quality inspection processes. The core objective of extracting these semantic regions is to transform human-understood engineering knowledge into spatial information that machines can accurately recognize and manipulate, thereby providing structured input for subsequent automated processes.
[0068] For semantic region extraction, an appropriate technical path can be selected based on the constraints of the application scenario. In situations with diverse workpiece types and a lack of fixed design rules, data-driven adaptive methods are typically used, such as cluster analysis of image features or semantic segmentation using deep learning models. Algorithms learn the visual patterns and semantic associations of regions from samples. However, in typical large-scale, high-paced industrial settings, most workpieces are standardized and serialized, with relatively fixed positions, shapes, and dimensions of their functional areas. Therefore, a more direct and reliable method is to manually create precise masks for each functional area in advance based on the workpiece's CAD drawings or reference images, forming a reusable semantic template. This method embeds domain knowledge into the template, ensuring unambiguous semantic definitions and zero deviation in positional information.
[0069] In this embodiment, based on the CAD drawings of the workpiece, precise masks for six functional areas are pre-drawn manually, such as... Figure 4 As shown, it includes: background area, left edge of the workpiece, workpiece panel area, hole area, right functional area, and constantly changing character area. Background semantic partition mask set. Satisfying mutually exclusive and fully covered constraints: , where 1 is a tensor of all 1s of the same size as the input image. This constraint ensures that each pixel in the image is assigned to exactly one and only one background sub-region, meaning that the sub-regions are mutually exclusive and together cover the entire image, with no omissions or overlaps.
[0070] Step 3: Anomaly Segmentation Model The template library graph and the graph to be tested are processed using an anomaly segmentation model to generate the original anomaly score graph of the graph to be tested. The core objective of anomaly segmentation models is to generate a pixel-level anomaly score map to quantify the probability of an anomaly at each location in the image under test.
[0071] To achieve this goal, the mainstream methods mainly rely on two technical paths: one is the template-based comparison path, which measures the difference by calculating the similarity or reconstruction residual between the image to be tested and the normal template in the feature space; the other is the distribution-based modeling path, which determines anomalies by assessing the probability that the feature to be tested deviates from the normal feature distribution.
[0072] This embodiment provides an example using a template comparison-based technique: the template image of the workpiece is the same size as the image to be tested. The anomaly segmentation model consists of an image feature extractor and a similarity calculator. The image feature extractor uses a pre-trained network such as CLIP or ResNet50 capable of extracting image features. After forward propagation of the network, the template image and the image to be tested obtain their respective corresponding image features. , The similarity calculator uses cosine similarity, and the calculation formula is: The test image and the template image are processed by a feature extractor to obtain their respective feature representations. , The similarity calculator calculates the similarity between the image to be tested and the template image. The smaller the similarity value, the more abnormal it is; the larger the value, the more normal it is. Finally, an anomaly map between the template image and the image to be tested is obtained. The generated original anomaly score map The image size is 77×77, and then upsampled to the original image size of 2456×1800 using bilinear interpolation, which is represented as follows: ,in, This refers to the batch size.
[0073] Original anomaly score map and background semantic partition mask set The input is fed into a multimodal semantic region threshold predictor, which performs the following operations: (1) Region perception normalization For each background sub-region Based on the original anomaly score map and the corresponding mask Perform region-aware normalization to obtain a normalized score map. .
[0074] First, calculate the... Mean local anomaly score of each background sub-region : , in, For the original anomaly score map at location The value at that location, To determine whether the corresponding position belongs to the first Background indicator value (1 indicates belonging, 0 indicates not belonging). The numerator represents the sum of the anomaly scores of all pixels in the region, and the denominator represents the total number of pixels in the region; To prevent small constants from being divided by zero.
[0075] Subsequently, based on the mean of local outlier scores and learnable parameters For the original anomaly score map Perform nonlinear normalization to obtain the first... Normalized score map of each background subregion : , in, These are learnable parameters that control the sensitivity of the normalization function. The main body of the function is a scaled Sigmoid function, which... Mapping to the interval of approximately [0.3, 0.8] results in pixels slightly above the mean being moderately enhanced within this background region, while pixels much above the mean are nearly saturated, thus highlighting local anomalies while suppressing fractional scale differences between different background regions.
[0076] (2) Region-specific threshold prediction Normalized fractional graph and its mask Input to a sub-region with background Corresponding, trainable region-specific threshold prediction subnetwork Generate a spatially adaptive threshold map. .
[0077] The threshold prediction subnetwork employs a lightweight convolutional neural network structure, specifically including: The first layer is a two-dimensional convolutional layer with a kernel size of 3×3, a stride of 1, padding of 1, and 64 output channels, followed by batch normalization (BN) and ReLU activation function, and the output feature map size is kept at 2456×1800×64. The second layer is a two-dimensional convolutional layer with a kernel size of 3×3, a stride of 1, padding of 1, and 32 output channels, followed by BN and ReLU activation functions. The output feature map size is maintained at 2456×1800×32. The third layer is a two-dimensional convolutional layer with a kernel size of 1×1, a stride of 1, padding of 1, and 1 output channel. It is followed by a sigmoid activation function, and the output feature map size is kept at 2456×1800×1, which serves as the final spatial adaptive threshold map.
[0078] In this embodiment, Each region-specific threshold prediction subnetwork employs a weight-sharing strategy to further reduce the number of parameters. The number of parameters for a single region-specific threshold prediction subnetwork is calculated as follows: First layer: 3×3×1×64+64=640 parameters; The second layer contains 3×3×64×32+32=18464 parameters; The third layer: 1×1×32×1+1=33 parameters; Batch normalization layer: Each layer has 2 × number of channels and parameters, totaling 2 × (64 + 32) = 192 parameters. The total number of parameters in a single sub-network is approximately 19.3K. When the parameter value is 6, the total number of parameters is approximately 115.8K if independent weights are used; if shared weights are used, the total number of parameters can be reduced to 19.3K. This embodiment adopts a shared weight strategy to achieve multi-region threshold prediction with extremely low computational overhead. Actual testing shows that the inference time for a single image is approximately 35ms (on an NVIDIA GeForce RTX 3080 graphics card), meeting the real-time detection requirements of industrial production lines.
[0079] (3) Differentiable soft binarization According to the normalized fraction graph Threshold map and mask Each background sub-region is calculated through a differentiable soft binarization operation. Anomaly probability diagram A differentiable soft binarization function is defined as: , in, Normalized fractional graph The pixel values in For spatially adaptive threshold maps Threshold at the corresponding position The preset transition band width; for example =0.1. When the fraction Significantly higher than the threshold ( When the score is significantly below the threshold, the output is 1, indicating an anomaly; when the score is significantly below the threshold, the output is 1. When the output is 0, it is considered normal; in the intermediate transition interval [ , The output changes linearly. This design ensures that the function is differentiable everywhere, enabling the entire network to support end-to-end gradient backpropagation, facilitating joint training, and avoiding training instability caused by hard thresholding.
[0080] No. Anomaly probability map of each background sub-region Obtained through the following formula: , in, This indicates element-wise multiplication by a mask. Ensure that the anomaly probability is calculated only within the current background sub-region.
[0081] (4) Abnormal fusion Anomaly probability map fused from all K background sub-regions The final anomaly probability map The blending operation specifically involves taking the maximum value pixel by pixel: This operation enables localized anomaly detection: the anomaly of each pixel is determined independently by the model of its respective background sub-region, without comparison with other regions. A pixel is identified as a defect as long as it exhibits sufficiently high anomaly within its normal background mode, thus achieving localized anomaly detection and effectively adapting to the multimodal characteristics of complex backgrounds.
[0082] Step 4: Obtain the anomaly mask This step will generate the final anomaly probability map. The image is converted into a binary anomaly mask. Specifically, during deployment, areas with anomaly probabilities greater than 0.5 are set as foreground (abnormal), and areas with probabilities less than or equal to 0.5 are set as background (normal). Through this process, an anomaly mask image with the same size as the input image (2456×1800) is generated, clearly distinguishing suspected abnormal areas from the normal background in binary form, laying the foundation for subsequent precise location positioning.
[0083] Step Six: Extracting Abnormal Locations This step aims to accurately locate and output the spatial contour information of anomalous regions from a binary anomaly mask. Specifically, connected component analysis is performed on the anomaly mask image obtained in step five, and a contour extraction algorithm is executed to obtain the set of closed contour coordinates of all anomaly regions marked as foreground. Each contour represents the boundary pixel position of an independent anomaly region. The processing may include a contour filtering step, such as filtering out small or invalid contours that may be caused by noise based on geometric features such as contour area and bounding rectangle size, to improve the robustness of the detection results. Finally, the contour coordinates of all extracted valid anomaly regions, or the location box information defined by their bounding rectangles, are used as the anomaly detection results of this method, and are structuredly output and returned to the calling side for subsequent display, recording, or decision-making processes.
[0084] Unsupervised training process The semantic multimodal threshold prediction module in this invention includes learnable parameters (in region-aware normalization). The weights of the region-specific threshold prediction subnetwork need to be determined through training. Crucially, this training process can be completed under completely unsupervised conditions, relying solely on normal samples without any anomaly annotations or foreground masks, which aligns with real-world industrial deployment scenarios.
[0085] This embodiment uses the following training configuration: Dataset partitioning: Collect 1000 normal and good product images and partition them into 80% training set, 10% validation set, and 10% test set; Optimizer: The Adam optimizer is selected, and the initial learning rate is set to... Weight decay is set to ; Batch size: 8; Training rounds: A total of 100 rounds of training are conducted, with evaluation on the validation set every 5 rounds. The model with the smallest loss on the validation set is selected as the final model. Data augmentation: Randomly flip, rotate (±5°), and adjust brightness (±10%) the training images to enhance the robustness of the model.
[0086] The specific training process is as follows: (i) Collect good product data and input it into the anomaly detection model to obtain the corresponding raw anomaly score map; (ii) Obtain the semantic region of the workpiece; (iii) Formulate the unsupervised optimization objective: On normal images, ideally, the final anomaly probability map should approach zero (i.e., no anomaly response). Therefore, the unsupervised loss function is defined as: This loss encourages the model to output the lowest possible anomaly confidence on normal samples.
[0087] (iv) Training strategy: The backbone anomaly detection model is kept frozen, and only the learnable parameters in the region-specific threshold prediction subnetwork and the normalization module are trained. During training, the loss is calculated on the validation set every 5 epochs, and the model with the smallest validation loss is saved. If the validation loss does not fall below the historical best value for 10 consecutive epochs, training is terminated early. After training is complete, the saved best model is used for inference.
[0088] This technique primarily works by minimizing anomalous responses on normal data; the threshold predictor automatically learns the threshold at the [missing information]. In the context of the background, what level of normalization score is still considered within the normal fluctuation range, thus assigning reasonable adaptive thresholds to different texture regions.
[0089] Example Effects like Figure 4 As shown in the schematic diagram of the background partitioning in this embodiment, an image of an industrial product is divided into six mutually exclusive background sub-regions according to its functional structure: background area, left edge of the workpiece, workpiece panel area, hole area, right functional area, and constant / variable character area.
[0090] like Figure 5 As shown, the verification effect of the method in this embodiment in an actual production line project is demonstrated. Figure 5 Figure (a) shows the anomaly detection results without distinguishing semantic regions, indicating that there are many false detections; Figure 5Figure (b) shows the anomaly detection effect after distinguishing semantic regions using the method of this embodiment. The detected contours are more accurate, and false detections are effectively suppressed. Specific quantitative indicators show that compared with a fixed hard threshold for dividing the foreground and background, the method of this embodiment reduces the false detection rate by 41% and increases the recall rate by 3.2%, verifying the technical effect of the present invention.
[0091] Example 3: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the unsupervised image anomaly detection method based on semantic multimodal background modeling as described in Embodiment 2. The electronic device can be an industrial computer, an embedded device, a server, or other hardware device with computing capabilities. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA). The memory can be a storage medium such as RAM, a hard disk, or a solid-state drive (SSD).
[0092] By embedding the anomaly detection method of the present invention into electronic devices, real-time or near-real-time anomaly detection of industrial product images on the production line can be achieved, meeting the high-efficiency and accurate requirements of industrial quality inspection.
[0093] Example 4: This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the unsupervised image anomaly detection method based on semantic multimodal background modeling as described in Embodiment 2. The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drive (SSD), optical disc (CD, DVD), etc. The computer program is stored in the medium in software form and, when loaded and executed by a processor, can drive hardware devices to complete the anomaly detection method of this invention.
[0094] This invention organically combines a series of innovative techniques, including the structuring of engineering semantic knowledge, region-adaptive learnable normalization, region-specific neural network threshold prediction, and localized anomaly judgment logic, to propose a novel unsupervised anomaly detection paradigm: semantic multimodal background modeling. Its core principle lies in recognizing the multimodal characteristics of industrial image backgrounds and the inherent differences in anomaly score benchmarks across different functional areas (such as screw hole areas, panel areas, and character areas). By explicitly dividing and independently modeling these areas, the differences in score scales between regions are eliminated. Through a learnable threshold prediction network, the model can autonomously learn the normal fluctuation range of different locations within each region from normal data. The localized anomaly judgment logic ensures that each pixel is compared only within the normal range of its own background mode, rather than across regions. This series of designs enables the model to accurately distinguish between normal high responses within a region and true local anomalies, fundamentally solving the problem of false detection caused by the multimodal characteristics of the background, while maintaining high sensitivity to subtle anomalies.
[0095] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.
Claims
1. An unsupervised image anomaly detection method based on semantic multimodal background modeling, characterized in that, Includes the following steps: Obtain input data, which includes template library diagrams and the diagram to be tested; Workpiece semantic region extraction is performed on the image under test to obtain the background semantic partition mask set. ;in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image; The template library graph and the graph under test are processed using an anomaly segmentation model to generate the original anomaly score graph of the graph under test. ; Original anomaly score map and background semantic partition mask set The input is fed into a multimodal semantic region threshold predictor, which performs the following operations: For each background sub-region Based on the original anomaly score map and the corresponding mask Region-aware normalization is performed to obtain a normalized fractional map. ; Normalized fractional graph and its mask Input to a sub-region with background Corresponding, trainable region-specific threshold prediction subnetwork Generate a spatially adaptive threshold map. The background sub-region is calculated through differentiable soft binarization. Anomaly probability diagram ; Integrate all Anomaly probability map of each background sub-region The final anomaly probability map is obtained. ; Based on the final anomaly probability plot Obtain the anomaly mask and generate a binarized anomaly region mask image; Extract the abnormal location from the abnormal region mask image and output the outline or location box information of the abnormal region.
2. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: The background semantic partition mask set Satisfying mutually exclusive and fully covered constraints: , where 1 is a full-1 tensor of the same size as the input image.
3. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: The region-aware normalization process specifically includes: Calculate the first Mean local anomaly score of each background sub-region : , in, For the original anomaly score map at location The value, For the mask at position The indicated value, To prevent division by zero of small constants; Based on the mean of local anomaly scores and learnable parameters For the original anomaly score map Perform nonlinear normalization to obtain the first... Normalized score map of each background subregion : , in, .
4. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 3, characterized in that: The region-specific threshold prediction subnetwork It is a lightweight convolutional neural network, and its network structure includes: First layer: a two-dimensional convolutional layer with a kernel size of 3×3 and 64 output channels, followed by batch normalization and ReLU activation function; The second layer is a two-dimensional convolutional layer with a kernel size of 3×3 and 32 output channels, followed by batch normalization and ReLU activation function; The third layer is a two-dimensional convolutional layer with a kernel size of 1×1 and an output channel of 1, followed by a Sigmoid activation function.
5. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 4, characterized in that: The region-specific threshold prediction subnetwork The training was conducted under unsupervised conditions, using only normal samples. The training process included: Freeze the anomaly segmentation model used to generate the original anomaly score map, and update only the region-specific threshold prediction subnetwork. and learnable parameters ; Constructing an unsupervised loss function This is used to minimize the final anomaly probability map corresponding to normal samples. Overall response: , in, For batch size, This refers to the image resolution.
6. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: The differentiable soft binarization operation is as follows: , in, Normalized fractional graph The pixel values in For spatially adaptive threshold maps Threshold at the corresponding position The preset transition band width; the first Anomaly probability map of each background sub-region Obtained through the following formula: .
7. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: Integrate all Anomaly probability maps for each background sub-region are generated by taking the maximum value pixel by pixel. .
8. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: The anomaly segmentation model includes an image feature extractor and a similarity calculator, and generates the original anomaly score map in the following manner. : The template library image and the image to be tested are input into the image feature extractor to obtain the corresponding feature representations. and ; Calculate using a similarity calculator and The similarity is used to generate the original anomaly score map. .
9. The unsupervised image anomaly detection method based on semantic multimodal background modeling according to claim 1, characterized in that: Extracting anomaly locations from an anomaly region mask image specifically includes: Perform connectivity analysis on the mask image of abnormal regions; Execute the contour extraction algorithm to obtain the set of closed contour coordinates of all abnormal regions marked as foreground; The contour is filtered based on the contour area or the size of the circumscribed rectangle to remove small or invalid contours caused by noise; Output the contour coordinates of the valid abnormal region or the location box information defined by the circumscribed rectangle.
10. A system for implementing the unsupervised image anomaly detection method based on semantic multimodal background modeling as described in any one of claims 1 to 9, characterized in that, include: The semantic region extraction module is used to extract the semantic region of the workpiece from the image under test, and obtain the background semantic partition mask set. ;in, ≥2, each mask This corresponds to a mutually exclusive background sub-region based on engineering semantics, and all masks together cover the entire image; Anomaly segmentation models are used to process template library graphs and the graph under test to generate raw anomaly score maps of the graph under test. ; The semantic multimodal threshold prediction module, connected to the semantic region extraction module and the anomaly segmentation model, is used to receive the original anomaly score map. and background semantic partition mask set And output the final anomaly probability map. The semantic multimodal threshold prediction module includes: Region-aware normalization units are used to normalize each background sub-region. Based on the original anomaly score map and the corresponding mask Region-aware normalization is performed to obtain a normalized fractional map. ; Each region-specific threshold prediction subnetwork is connected to a region-aware normalization unit. Corresponding to a background sub-region Used to receive normalized fractional graphs and its mask It outputs a spatially adaptive threshold map. ; Differentiable soft binarization unit, and region-aware normalization unit and Each region-specific threshold prediction subnetwork is connected to, used to predict thresholds based on the normalized score map. Spatial adaptive threshold map and mask Each background sub-region is calculated through a differentiable soft binarization operation. Anomaly probability diagram ; Anomaly fusion units, connected to differentiable soft binarization units, are used to fuse all... Anomaly probability map of each background sub-region Output the final anomaly probability map. ; The post-processing module, connected to the semantic multimodal threshold prediction module, is used to predict the anomaly probability based on the final anomaly probability map. Obtain the anomaly mask, generate a binarized anomaly region mask image, extract the anomaly location from the anomaly region mask image, and output the contour or location box information of the anomaly region.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the unsupervised image anomaly detection method based on semantic multimodal background modeling as described in any one of claims 1 to 9.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the unsupervised image anomaly detection method based on semantic multimodal background modeling as described in any one of claims 1 to 9.
