A Method for Generating Abnormal Pill Images Based on Region Attention Mechanism

Abstract

The present application relates to the technical field of tablet quality detection, and specifically discloses a tablet abnormal image generation method based on a regional attention mechanism, comprising the following steps: S1, obtaining a tablet normal image dataset and a tablet abnormal image dataset; S2, constructing an SDDRAM-CycleGAN model comprising a generator and a double-discriminator regional attention module; S3, taking the tablet normal image dataset as a source domain and the tablet abnormal image dataset as a target domain, and performing bidirectional mapping training using the double-discriminator regional attention module, wherein the SDDRAM-CycleGAN model parameters are optimized by calculating a loss function during the training process; and S4, generating a tablet normal image to be converted into an abnormal image using the trained SDDRAM-CycleGAN model; by introducing the double-discriminator regional attention module and the residual guidance mechanism, the SDDRAM-CycleGAN model's ability to generate abnormal regions is enhanced, thereby achieving accurate conversion from a normal tablet image to a high-quality abnormal image.

Description

Technical Field

[0001] This invention relates to the field of tablet quality inspection technology, and more specifically to a method for generating abnormal tablet images based on a region attention mechanism. Background Technology

[0002] Pill quality inspection is a crucial step in the pharmaceutical industry, with testing standards far exceeding those for ordinary industrial products. Compared to general target inspection, pills have more stringent quality standards, more multi-dimensional defect identification requirements, and stricter safety compliance to ensure zero defects in pharmaceuticals. Current visual inspection solutions for pills typically involve data acquisition, manual annotation, feature extraction, defect identification, and result output. These solutions need to identify subtle differences in appearance, shape, and color, but suffer from significant drawbacks such as strong dependence on annotation quality, insufficient algorithm generalization ability, and high resource consumption.

[0003] In recent years, deep learning technology has made significant progress in the field of computer vision, providing new solutions for tablet quality inspection. Image classification and detection are fundamental tasks. ResNet's residual connections effectively solve the gradient vanishing problem in deep networks. Vision Transformer (ViT) introduces the attention mechanism from natural language processing into computer vision. Single-stage methods, represented by YOLO and SSD, directly predict bounding boxes through regression, which is faster. Two-stage methods, represented by the R-CNN series, employ a cascaded strategy of "region proposal + classification regression," resulting in superior detection accuracy. These groundbreaking models have laid the foundation for subsequent tasks.

[0004] With the rapid development of computer vision technology, 3D object visual detection has gradually become one of the research hotspots. Qi et al. proposed PointNet for processing point cloud data. By designing a unique network structure, it can learn the global features of point clouds, significantly improving accuracy and efficiency. Lang et al. proposed PointPillars to convert point clouds into voxel format, thereby using a two-dimensional convolutional neural network (CNN) to process 3D data. The Transformer architecture, with its unique self-attention mechanism, has achieved great success in fields such as natural language processing, and this advantage is also reflected in 3D visual detection tasks. Sun et al. proposed a stereo perception 3D object detector. By introducing the Transformer architecture, it can better capture the relationships between complex 3D point cloud data and mine long-distance dependencies and global information in the data.

[0005] However, these methods still face a key problem in practical application: data imbalance. In pharmaceutical manufacturing, the proportion of abnormal samples is usually extremely low, while deep learning models tend to be biased towards the majority of normal pills during training, resulting in a severe deficiency in their ability to identify rare defects. In addition, although traditional data augmentation methods can alleviate the problem of insufficient data to some extent, the generated samples often lack realistic defect characteristics and cannot simulate the complex anomalies in actual production. Summary of the Invention

[0006] The purpose of this invention is to provide a method for generating abnormal images of pills based on a region attention mechanism. By introducing a dual-discrimination region attention module and a residual guidance mechanism, the SDDRAM-CycleGAN model is enhanced to generate abnormal regions, thereby achieving accurate conversion from normal pill images to high-quality abnormal images.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for generating abnormal pill images based on a region attention mechanism includes the following steps:

[0009] S1. Obtain the dataset of normal images of pills and the dataset of abnormal images of pills;

[0010] S2. Construct an SDDRAM-CycleGAN model that includes a generator and a dual-discriminant region attention module. The generator is used to generate corresponding abnormal images of the tablet from normal images. The dual-discriminant region attention module includes a dual discriminator and an abnormal region attention module, which are used to comprehensively evaluate the authenticity of the generated abnormal images of the tablet. By calculating the residual between the normal image and the abnormal image of the tablet, the SDDRAM-CycleGAN model is guided to focus on the generation process of the abnormal regions of the tablet.

[0011] S3. The normal image dataset of pills is used as the source domain and the abnormal image dataset of pills is used as the target domain. The dual-discrimination region attention module is used for bidirectional mapping training. During the training process, the parameters of the SDDRAM-CycleGAN model are optimized by calculating the loss function.

[0012] S4. Using the trained SDDRAM-CycleGAN model, the normal image of the pill to be converted is transformed into an abnormal image.

[0013] Furthermore, in S2, the generator adopts the SegNeXt architecture, specifically including:

[0014] A convolutional encoder includes a multi-scale convolutional attention module, which extracts multi-scale features from normal images of pills and generates attention weights through depth convolution, multi-branch depth strip convolution and 1×1 convolution.

[0015] The decoder aggregates the multi-scale features of the normal image of the pill with the convolutional encoder and generates abnormal images of the pill by modeling the global context through the lightweight Hamburger module.

[0016] Furthermore, in S2, the dual-discrimination region attention module includes:

[0017] The block discriminator is used to divide the generated abnormal pill images into blocks and output the abnormal classification result matrix of each image block.

[0018] The region attention discriminator is used to perform pixel-level discrimination on the generated abnormal images of pills, and outputs a discrimination matrix of the same size as the input image.

[0019] Furthermore, the loss function in S3 includes:

[0020] Adversarial loss is used to compute the generator's adversarial loss against the block discriminator and the region attention discriminator based on the output of the block discriminator and the region attention discriminator;

[0021] Cyclic consistency loss is used to constrain the consistency between the generated abnormal pill image reconstructed by the inverse generator and the source domain data.

[0022] Identity loss is used to constrain the generator's output to the target domain data to remain unchanged;

[0023] The region attention loss is used to guide the CycleGAN model to pay more attention to the generation and transformation of abnormal regions of the tablet by focusing on the gradient changes of abnormal regions rather than normal regions.

[0024] Furthermore, the convolutional encoder adopts a pyramid structure and includes four downsampling stages, each of which consists of a downsampling module and a cascaded multi-scale convolutional attention module.

[0025] Furthermore, the multi-scale convolutional attention module extracts multi-scale features as attention weights through convolution operations, and then performs weighted processing on the attention weights and the input of the multi-scale convolutional attention module, specifically as follows:

[0026] The output of the 1×1 convolution is used as the final attention weight to recalibrate the features of the input to the multi-scale convolutional attention module, as expressed in the following expression:

[0027] (1)

[0028] (2)

[0029] In the formula, Here, Att represents the input features, Att is the attention weight map, and Out is the output feature. DW-Conv represents element-wise matrix multiplication, while DW-Conv represents depthwise convolution. .

[0030] Furthermore, the block discriminator consists of five 4×4 convolutional layers, ultimately dividing the generated abnormal pill image into... The image block matrix.

[0031] Furthermore, the region attention discriminator consists of three 1×1 convolutional layers and outputs a discrimination matrix of the same size as the input image, which is used to judge whether each pixel of the generated abnormal image of the pill belongs to the abnormal image category.

[0032] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0033] This invention effectively improves the quality and realism of converting normal to abnormal images of pharmaceutical tablets by introducing an SDDRAM-CycleGAN model trained on an unpaired dataset and combining it with a dual-discriminator region attention module. The dual-discriminator structure enhances the global and local discrimination capabilities of the generated images, while the abnormal region attention module guides the model to focus on the generation of abnormal regions through residual calculation, significantly improving the detail accuracy and controllability of abnormal features in the generated images. The bidirectional mapping training mechanism not only strengthens the model's stability but also optimizes the mapping relationship between the source and target domains, making the generated abnormal images more visually realistic and structurally more consistent with actual defect characteristics. This technical solution achieves high-quality abnormal image synthesis without the need for paired samples, providing strong support for sample augmentation and model training in fields such as industrial quality inspection and medical image analysis. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] The following description, in conjunction with the accompanying drawings, further illustrates a method for generating abnormal tablet images based on a regional attention mechanism according to the present invention.

[0036] Figure 1 This is a flowchart illustrating the method for generating abnormal tablet images based on a regional attention mechanism in Embodiment 1 of the present invention.

[0037] Figure 2 This is a structural composition diagram of the SDDRAM-CycleGAN model in Embodiment 1 of the present invention;

[0038] Figure 3 This is a model architecture diagram of GAN1 in the SDDRAM-CycleGAN model in Embodiment 1 of the present invention;

[0039] Figure 4 This is an architecture diagram of the convolutional encoders MSCA and MSCAN in Embodiment 1 of the present invention; wherein (a) is the architecture diagram of MSCA; and (b) is the architecture diagram of MSCAN.

[0040] Figure 5 These are different decoder design architecture diagrams in Embodiment 1 of the present invention; where (a) is the architecture diagram of SegFormer; (b) is the architecture diagram of the mainstream CNN method; and (c) is the architecture diagram of SegNeXt.

[0041] Figure 6 This is a schematic diagram of the SDDRAM-CycleGAN training and testing process in Embodiment 2 of the present invention;

[0042] Figure 7 The above are the existing CycleGAN generation results in Embodiment 2 of the present invention; where (a) is the original image and (b) is the abnormal image of generated spots;

[0043] Figure 8 These are the data preprocessing results from Embodiment 2 of the present invention; where (a) is a normal image of a real tablet; (b) is a speckled image of a real tablet; (c) is a normal image of a CG tablet; and (d) is a speckled image of a CG tablet.

[0044] Figure 9 This is a diagram of the SDDRAM-CycleGAN iterative process in Embodiment 2 of the present invention;

[0045] Figure 10 This is a schematic diagram of the training process of SED-CycleGAN in Embodiment 2 of the present invention.

[0046] Figure 11 These are the image generation results of the medicine tablet in Embodiment 2 of the present invention; where (a) is a normal image of a real medicine tablet, and (b) is the result of generating a speckled image;

[0047] Figure 12 These are the CG image generation results in Embodiment 2 of the present invention; where (a) is a normal CG tablet image, (b) is a near-real normal image, and (c) is the result of speckle image generation.

[0048] Figure 13 This is a comparison chart of the ablation experiment results in Example 2 of the present invention. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0051] Example 1

[0052] like Figure 1 As shown, the specific process of constructing the SDDRAM-CycleGAN model, which includes a generator and a dual-discrimination region attention module, in the method for generating abnormal pill images based on the region attention mechanism provided by this invention is as follows:

[0053] 1. Network structure of the SDDRAM-CycleGAN model:

[0054] The SDDRAM-CycleGAN model in this invention features significant architectural improvements, exhibiting marked differences compared to the standard CycleGAN. For example... Figure 2 As shown in (a), the model in this invention employs a dual generator. And four discriminators An enhanced architecture was developed. Normal and anomalous images were used as the source domains of CycleGAN, respectively. and target domain Perform bidirectional mapping.

[0055] The model training process includes two symmetrical paths: the forward training process GAN1 and the backward training process GAN2. Figure 2 Image (b) shows GAN1, in which Accept normal images and generate abnormal images Images that follow a uniform distribution , Conversely accept As input, it generates near-normal images. Data distribution , Responsible for distinguishing and .and Figure 2 (c) reflects GAN2, and Figure 2 (b) has a similar process and model structure, but processes the image input from the opposite direction, i.e., with anomalous images. Input is the input, and the final output is the output. .

[0056] In this embodiment, the main function of SDDRAM-CycleGAN is training. This enables it to transform normal images into defective, aberrant images. GAN1, in particular,... and All are used to assist in participation Training process: Discriminator and They are responsible for distinguishing during the forward training process. and ;and Training is aided by constructing a cycle consistency loss, and adversarial training is also required during reverse training to optimize its performance.

[0057] GAN1 , and The model structure is as follows Figure 3 As shown. SegNeXt is used as the generator. It is a block discriminator. It is a region attention discriminator that constructs soft labels. DDRAM was designed to increase the ability to synthesize anomalous regions.

[0058] 2. Architecture of the SegNeXt generator:

[0059] Generator The main objective is to transform normal images into anomalous images. This invention uses SegNeXt as the generator. First, the working principle of the SegNeXt semantic segmentation network is explained. Then, by analyzing the similarity between semantic segmentation and generator networks, the feasibility of achieving pixel-level optimization in image generation tasks using the SegNeXt architecture is demonstrated.

[0060] (1) SegNeXt semantic segmentation network principle

[0061] This invention summarizes four key characteristics of an ideal semantic segmentation model: a powerful backbone network, multi-scale information fusion, spatial attention mechanism, and low computational overhead. To meet these requirements, SegNeXt employs an efficient architecture: a convolutional encoder is designed with a pyramid structure and a multi-scale convolutional attention (MSCA) module to achieve efficient extraction of multi-scale features; the decoder further improves performance and efficiency by aggregating features from the last three stages and combining them with global context modeling. Details are as follows:

[0062] 1) Convolutional encoder

[0063] SegNeXt's encoder is built using a pyramid structure, and its overall architecture is similar to ViT, but the key difference is that it replaces the self-attention mechanism with a newly designed MSCA module. The MSCA module consists of three core components, such as... Figure 4 As shown in (a): a) Depthwise convolution, responsible for local feature aggregation, denoted as in the figure. , , indicating the use a) Deep convolution with kernels of varying sizes; b) Multi-branch deep strip convolution to extract contextual features at multiple scales; c) 1×1 convolution to establish inter-channel dependencies.

[0064] The innovation of the MSCA module lies in its approach: first, it extracts multi-scale features as attention weights through convolutional operations, and then reweights these weights with the module input. Specifically, the output of a 1×1 convolution is used as the final attention weight to recalibrate the features of the MSCA input, expressed as follows:

[0065] (1)

[0066] (2)

[0067] In the above formula, Here, Att represents the input features, Att is the attention weight map, and Out is the output feature. DW-Conv represents element-wise matrix multiplication, while DW-Conv represents depthwise convolution. correspond Figure 4 The four branches in (a) (For identity connections). Each branch uses two cascaded depth strip convolutions to equivalently simulate a large-kernel standard depth convolution, with kernel sizes configured as 7×7, 11×11, and 21×21, respectively. The advantages of using depth strip convolutions are: a) computational efficiency, for example, a 7×7 standard convolution can be achieved by combining 7×1 and 1×7 strip convolutions; b) adaptability to striped target features in detection scenarios, complementing conventional mesh convolutions. Finally, by stacking the MSCA module with other functional blocks, the complete convolutional encoder MSCAN is constructed, such as... Figure 4 As shown in (b).

[0068] The input image processing flow in the encoder consists of four progressive downsampling stages, with the feature map sizes output from each stage being: (H / W represents the height / width of the input image). Each stage consists of the following two core components: a) a downsampling module, which can use 3×3 convolutions with a stride of 2 and batch normalization to halve the resolution; b) ... A cascaded MSCAN convolutional encoder. Based on different numbers of channels. and stacking layers Based on the configuration, this invention constructs four variants: , , , .

[0069] 2) Decoder

[0070] In semantic segmentation models, the encoder is typically initialized based on an ImageNet pre-trained model. To effectively fuse multi-scale features and enhance semantic understanding, a decoder structure is needed after the encoder. This invention compares three typical decoder designs, such as... Figure 5 As shown: 1) SegFormer uses a pure MLP structure; 2) mainstream CNN methods (such as ASPP, PSPNet, DANet, etc.) directly use the encoder's final output as the decoding input; 3) the structure proposed by SegNeXt—innovatively integrates the features of the last three stages of the encoder and models the global context through a lightweight Hamburger module. Experiments show that design 3), when working in conjunction with a convolutional encoder, can significantly improve segmentation performance while maintaining computational efficiency.

[0071] It is worth noting that, unlike the SegFormer decoder which fuses features from all four stages, the decoder proposed in this invention only integrates features from the last three stages. This design is based on two factors: First, in the convolution-based SegNeXt architecture, the first-stage features contain too much low-level visual information, and introducing a decoder would simultaneously impair model performance and increase computational burden; second, experimental results show that this selective feature fusion strategy makes the convolution-based SegNeXt significantly outperform the Transformer-based SegFormer and HRFormer in terms of performance.

[0072] (2) Semantic segmentation and generator network similarity

[0073] The initial design of the standard CycleGAN had some limitations in its generator. It typically employs a fully convolutional structure, progressively extracting features through downsampling, then using a residual network to enhance the effect, and finally upsampling to restore the original resolution. However, this design has some significant drawbacks. First, the generator's backbone architecture is relatively basic, failing to fully utilize advanced feature extraction techniques, resulting in shortcomings in handling contextual information and multi-scale features. A large amount of detail is lost during the downsampling stage, and although upsampling can restore the image size, the loss of detail and contextual information cannot be fully compensated for. Second, the lack of a spatial attention mechanism prevents the network from effectively focusing on certain important regions in the image, ultimately affecting the quality of the generated image. These problems lead to CycleGAN often failing to meet higher detail standards in high-precision image reconstruction or detection tasks. An abnormal image of a pill generated by the standard CycleGAN is shown below. Figure 7 As shown, the image quality is poor, the generated spots cannot be well embedded in the surface of the tablet, and there are a large number of checkerboard artifacts.

[0074] Semantic segmentation networks, such as SegNeXt, are typically used for pixel-level tasks. Their goal is to classify each pixel in an image to identify different instance categories, scene information, or other semantic information. Therefore, SegNeXt can be used for pixel-level optimization in image generation, mainly in the following aspects:

[0075] 1) Low computational complexity: Recognizing high-resolution images usually requires huge computational resources, but SegNeXt reduces computational complexity by optimizing the network architecture, which can ensure that the model maintains high accuracy and significantly improves inference speed when generating large-scale image data.

[0076] 2) Powerful backbone network: Compared with traditional CNNs, SegNeXt uses a more efficient backbone network that can capture richer information during feature extraction. Its advantage lies in its ability to extract more details from images, thereby providing higher accuracy and finer granularity in complex image generation tasks.

[0077] 3) Multi-scale feature fusion: This invention proposes an abnormal image generation task, requiring the CycleGAN generator to accurately identify and locate the surface of the pill in the image, while SegNeXt can enhance its adaptability to various scenarios through optimized multi-scale information interaction design, and generate images with greater generalization.

[0078] 4) Spatial Attention Mechanism: Attention mechanisms improve network performance by giving the model focus on important regions of the image. Introducing the SegNeXt spatial attention mechanism into the CycleGAN generator allows the model to focus more on semantic information in the image when generating anomalous images.

[0079] Therefore, this invention employs SegNeXt as the generator, a convolutional encoder, and MSCA as a building block in each downsampling stage. The decoder aggregates features from the last three downsampling stages and uses a lightweight Hamburger module to generate a new image. The convolutional encoder design significantly reduces computational complexity and captures richer contextual information during feature extraction, preserving information about non-abnormal regions in the generated image. Through spatial attention mechanisms and the multi-scale feature fusion capabilities of MSCA, the generator focuses more on generating anomalous regions, achieving a smooth transition of anomalous regions in the generated image, ultimately resulting in a more refined image. and abnormal images The data domain distribution tends to be consistent.

[0080] 3. Dual discriminator structure:

[0081] The SDDRAM-CycleGAN proposed in this invention employs a dual discriminator structure, such as... Figure 3 The DD shown is shown. and The work involves division of labor and collaboration, evaluating the generated image at different levels. If the input image... The resolution is :

[0082] 1) It is a block discriminator, mainly composed of 5 4×4 convolutional layers, which ultimately divides the image into blocks. This is an image patch matrix. It's equivalent to performing local feature analysis on each image patch and outputting a classification result matrix to determine whether any anomalous features exist in each region.

[0083] 2) This is the pixel discriminator, mainly composed of three 1×1 convolutional layers. This design preserves the dimensional information of the input image and outputs a pixel discriminator. The discrimination matrix. The generated image is subjected to pixel-level discrimination, which is mainly used to determine whether each pixel of the generated image belongs to the abnormal image category.

[0084] also, Also known as the region attention discriminator, its output and the residual of the generator image together constitute DDRAM, which further enhances the model's ability to focus on local features.

[0085] (1) Abnormal Area Attention Module

[0086] To enhance the generator To enhance the ability to construct anomalous features, this invention designs a Region Attention Module (RAM), which, combined with a dual discriminator design, forms a dual-discriminative Region Attention Module, such as... Figure 7 The DDRAM shown. The original label of the discriminator in this module is... (This represents an abnormal image, filled with all 1s) (matrix) or (This represents a normal image, filled with zeros) matrix).

[0087] A key aspect of DDRAM design is the precise targeting of abnormal regions. In the generator... During the training process, the normal image is first calculated. Its generated abnormal images The residual image Res between the two is used to obtain the difference information between them, as shown in Equation (3). To avoid the influence of the contrast difference between the defect area and the background, the residual of each image is normalized. To ensure a more stable and comparable distribution of residual characteristics, the function expression is:

[0088] (3)

[0089] Next, label the abnormal images. Add the soft label to the defect residual image Res. This causes the label value of the abnormal region to be greater than the label value of the normal region. Furthermore, the parameters in the formula... The weight used to control the DDRAM is set to 0.5 in this embodiment.

[0090] (4)

[0091] Compared with regional attention discrimination results A region attention loss was constructed using the L2 loss function, which enhances the model's focus on anomalous regions during training, forcing the discriminator's response to these regions to be closer to the target value, thus guiding the generator. The model strengthens the construction of anomalous region features to meet the requirements of the discriminator, while maintaining only slight variations in surrounding regions. Since the generator's output is initially close to its input, the initial residual image value is zero. As training progresses, the labels of anomalous regions gradually become larger than those of normal regions, allowing the model to focus more intently on anomalous areas.

[0092] (2) Loss function module

[0093] This invention defines a loss function for training SDDRAM-CycleGAN, using all the loss functions and expressions of the standard CycleGAN. Due to the introduction of a dual discriminator structure and a region attention module, the loss function needs to be updated, and a new training method is proposed. The specific loss function is as follows:

[0094] 1) Adversarial Loss: Due to the introduction of dual discriminators, the adversarial loss of the model changes, and is divided into a forward process (GAN1) and a backward process (GAN2). GAN1's... , and Corresponding to adversarial losses , and GAN2 , and Corresponding to adversarial losses , and The specific formula is as follows:

[0095] (5)

[0096] (6)

[0097] (7)

[0098] (8)

[0099] (9)

[0100] (10)

[0101] 2) Cycle Consistency Loss: Loss during forward and backward training. and They are represented as follows:

[0102] (11)

[0103] (12)

[0104] 3) Identity loss: the loss during forward training and backward training. and They are represented as follows:

[0105] (13)

[0106] (14)

[0107] 4) Region Attention Loss: Unlike the standard CycleGAN, this invention proposes DDRAM and constructs soft labels. It uses L2 loss and anomaly attention loss functions in both forward and backward training. and See equations (15) and (16). In In the middle, soft tags The calculation principle and Consistent. During backpropagation, the gradient in the abnormal region is significantly higher than that in the normal region, therefore the model focuses more on the changes in the abnormal region than the changes in the normal region.

[0108] (15)

[0109] (16)

[0110] To more clearly express the application method of the model, this invention uses... Figure 6 This describes the training and testing process of SDDRAM-CycleGAN.

[0111] (3) Image quality assessment

[0112] To systematically evaluate the generated abnormal images of pills, this invention establishes an evaluation system from two dimensions: pixel-level features and domain distribution features. Pixel-level comparison focuses more on the pixel details at the physical level of a single image, but cannot measure distribution consistency; while data distribution level measures the authenticity and diversity of the generated dataset from the perspective of probability distribution, but has no ability to perceive local details.

[0113] 1) Pixel-based image quality assessment

[0114] Pixel-based image quality assessment focuses on the physical level, emphasizing pixel-by-pixel comparison of individual images, and is used to evaluate real-world images. and the corresponding generated image Similarity between them. The following are common metrics for pixel-level quality assessment:

[0115] a. Mean Absolute Error (MAE)

[0116] MAE uses the mean absolute deviation of predicted values ​​from true values ​​as a quantification metric. It assigns equal weight to all errors, thus having a relatively small impact on outliers. MAE is commonly used to measure the error in image reconstruction; a smaller MAE value indicates a smaller error and that the reconstructed image is closer in quality to the original image. The calculation formula is:

[0117] (17)

[0118] In the formula, , These are the original images. and generating images The first in pixel value, It is the total number of pixels in the image.

[0119] b. Mean Squared Error (MSE)

[0120] MSE (Mean Squared Error), a commonly used metric for evaluating the difference between predicted and actual values, measures the deviation by calculating the average of the squared errors. Compared to MAE (Maximum Error), MSE is more sensitive to larger errors because the squaring operation amplifies their impact. MSE is widely used in image reconstruction, denoising, and compression; a smaller value indicates a smaller prediction error and higher image quality. However, because MSE is more sensitive to outliers, it may be less intuitive than MAE in certain scenarios. The calculation formula is as follows:

[0121] (18)

[0122] In the formula, , These are the original images. and generating images The first in pixel value, It is the total number of pixels in the image.

[0123] c. Peak Signal-to-Noise Ratio (PSNR)

[0124] PSNR is a classic metric for evaluating the degree of image distortion. It measures image quality by calculating the ratio between the maximum possible value of an image and the noise level. It is commonly used to analyze compressed or reconstructed images; a higher PSNR value indicates better image quality and a smaller difference from the original image. The calculation formula is:

[0125] (19)

[0126] In the formula, It is the maximum value of pixels in the image (usually 255 for 8-bit images). It is the mean square error.

[0127] d. Structural Similarity Index Measure (SSIM)

[0128] SSIM is an evaluation metric that quantifies the similarity between two images by analyzing their brightness, contrast, and structural features. Its value ranges from 0 to 1, where 1 indicates that the two images are identical, and 0 indicates that they are completely different. SSIM is suitable for detecting structural differences in images, especially changes in contrast and brightness. The calculation formula is:

[0129] (20)

[0130] In the formula, , These are images and The average brightness, , Especially their brightness variance, It is their brightness covariance. , It is a stability constant to prevent the denominator from reaching zero.

[0131] e. Multi-Scale Structural Similarity Index Measure (MS-SSIM)

[0132] MS-SSIM, an advanced version of SSIM, improves the accuracy of image quality assessment by analyzing the structural similarity of images at multiple scales. It comprehensively considers local structural information at different scales, thus providing a more holistic assessment of the similarity between images. Compared to traditional SSIM, MS-SSIM, through multi-scale processing, can better capture the structural features of images at different resolutions, making it particularly suitable for high-resolution or complex images. Its calculation process consists of three steps: first, the image is decomposed into multiple scales, with progressive downsampling to obtain images at different scales; then, the SSIM value is calculated at each scale; finally, the SSIM values ​​at each scale are weighted and averaged to obtain the final MS-SSIM value. The MS-SSIM value ranges from 0 to 1, where 1 indicates that the images are completely identical, and 0 indicates that they are completely different. Compared to SSIM, MS-SSIM provides a more refined image quality assessment, especially suitable for processing images with complex or multi-layered structures. The calculation formula is:

[0133] (twenty one)

[0134] In the formula, It is the number of scale layers. It is the first SSIM values ​​on each scale It represents the weight of each scale.

[0135] 2) Image quality assessment based on data domain distribution

[0136] Data domain distribution-based methods measure the authenticity and diversity of data from a probability distribution perspective, focusing on the overall characteristics of batch data. They are suitable for evaluating the global distribution similarity between real and generated image sets. These metrics typically employ pre-trained models such as InceptionV3 and ResNet34 for feature extraction during computation. The following are common metrics:

[0137] a. Fréchet Inception Distance (FID)

[0138] FID primarily focuses on the overall quality of the generated images, by randomly selecting from both the real image set and the generated image set. A set of images is used to compare the distribution of generated and real images. These images are then fed into a truncated Inception network (with the final classification layer removed) to extract a 64-dimensional depth feature vector. The final calculation formula is as follows:

[0139] (twenty two)

[0140] In the formula, and These represent the mean vectors of the feature spaces of the real image and the generated image, respectively. and Let represent the feature space covariance matrices of the real image and the generated image, respectively; This is the matrix trace operator. A lower FID score indicates better image quality, while a higher FID score suggests that the generated image may contain random noise, which could lead to blurring or distortion, thus reducing its quality.

[0141] b. Bulldozer Distance (Wasserstein Distance, WD)

[0142] Difference of probability distributions (WD) is a metric used to quantify the difference between two probability distributions. A smaller WD value indicates higher distribution similarity, and in GANs, it directly reflects the realism of the generated images. In WGAN, the core function of the discriminator is to guide the generator's optimization by learning the WD metric between the real and generated image sets. When used as an evaluation metric, the calculation of WD relies on a well-trained discriminator model. Given... A real image sample and One generated image sample The WD estimation formula is:

[0143] (twenty three)

[0144] In the formula, and Representing the distributions of real images and generated data, respectively. This is the discriminator feature mapping function.

[0145] c. Inception Score (IS)

[0146] Image quality index (IS) is used to evaluate the quality and diversity of a single image set. Its core relies on the Inception network to analyze the classification distribution of generated images and quantifies image quality by calculating the KL divergence between the conditional label distribution (the class probability given an image) and the marginal label distribution (the average class probability of all images). Specifically, a higher IS value indicates that the generated images are more categorically clear and diverse, meaning the generator can produce more recognizable and varied images. However, IS focuses more on the class information of the images and may overlook details of image quality. Therefore, IS is often used in conjunction with other evaluation metrics to provide a more comprehensive assessment of generated images. By comparing the IS values ​​of generated and real image sets, the distributional similarity between the two can be evaluated. The calculation formula is as follows:

[0147] (twenty four)

[0148] In the formula, It is the number of images. It is an image Conditional distribution, It is the marginal distribution of the generated image set.

[0149] Example 2

[0150] The present invention also provides a training and verification process for the CycleGAN model constructed in Example 1:

[0151] 1. Data Collection:

[0152] The data used in this embodiment comes from a tablet visual inspection machine, and is specifically for one type of... The white pills were photographed, and then converted into normal images and speckled images.

[0153] The data acquisition process involved two aspects. First, all collected real images were manually labeled by professionals, resulting in 971 normal images and 735 speckled images. Second, CG technology was used to generate CG tablets and a batch of images (referred to as CG images), yielding 1500 normal and speckled images respectively. All original images had a resolution of 480×480 pixels. Irrelevant areas of the tablet images were cropped while keeping the tablet centered in the image, resulting in images approximately 120×120 pixels. Several preprocessed samples were randomly selected for display, such as... Figure 8 As shown.

[0154] To verify the performance of the anomaly generation method proposed in this invention, the dataset was first divided. Real images were divided into training and test sets in a 1:1 ratio, while all CG images were retained for testing. As shown in Table 1, four subsets were ultimately formed: training set trainA (normal images), trainingB (blob images), and test sets testA (normal images) and testB (blob images).

[0155] Table 1 Distribution of Experimental Data

[0156]

[0157] After completing data preprocessing and dataset partitioning, the experiment was conducted on the Ubuntu 20.04 operating system. The specific configuration was as follows: the experiment was conducted on a hardware platform with a 12-core Intel Xeon Platinum 8255C processor and an NVIDIA RTX 2080 Ti graphics card. The PyTorch 2.0.0 deep learning framework was used for model development, and CUDA 11.8 driver was configured to fully utilize the GPU acceleration performance, providing a stable and efficient computing environment for the experiment.

[0158] 2. CycleGAN model training process:

[0159] This invention uses the SDDRAM-CycleGAN model to generate pill dot images. The network initialization employs a normal distribution and Dropout network optimization techniques, with an initial gain set to 0.005, instance normalization used, a learning rate of 0.0002, and a linear decay strategy to adjust the learning rate. The batch size is set to 2, and the Adam optimizer is used for training during iterations.

[0160] Generator This is a key module for generating blob images and is primarily trained in GAN1. For example... Figure 9 As shown, the experiment demonstrates the adversarial loss of GAN1. , and Cyclic consistency loss Loss of identity and regional attention loss The iterative process. (By...) Figure 9 As can be seen, by the 100th round, the losses of each module have fully converged, indicating that the model training is stable and effective. It is worth noting the anomaly region attention loss proposed in this invention. It also converged.

[0161] During the training of the SDDRAM-CycleGAN model, different images were recorded at different epochs to comprehensively evaluate the model's performance. For example... Figure 10 As shown, these image samples include: original normal images Abnormal images generated by the model Reconstructed normal image and identity mapping image Experimental results show that when the number of training epochs is less than or equal to 20, the model is still in the initial learning stage, and its performance is poor, resulting in inferior generated images. and reconstructed images The image quality was initially poor; however, as the number of training epochs increased to 40, the quality of the generated images began to improve, with a significant increase in detail and clarity, and the generated blob features exhibiting a more natural transition effect; when the training epochs reached 100, the model had fully converged, and the generated images... The image clarity is high and the image quality is excellent. Throughout the training process, the SDDRAM-CycleGAN model consistently maintains a high degree of consistency in image surface features other than the generated speckles, which fully demonstrates the rationality of the SDDRAM-CycleGAN model structure and the effectiveness of the training strategy.

[0162] 3. Display of generated results:

[0163] After 100 iterations, the model uses 1933 images from the test set testA to generate blob images from normal images. For example... Figure 11 As shown, the results demonstrate the generation of some abnormal images from real pill images. Experimental results show that the generated images have clear, natural, and diverse spots, without altering other features of the original image. This indicates that the abnormal region attention module of the proposed algorithm can accurately locate and generate abnormal features.

[0164] To verify the practical application effect of the proposed method, this invention uses normal images of CG tablets in the test set testA for testing. The experimental results are as follows: Figure 12 As shown. The specific experimental procedure is as follows: First, the SED-CycleGAN method proposed in Chapter 3 is used to... Figure 12The original CG image in (a) is converted to Figure 12 The near-realistic image in (b) is then further generated using SDDRAM-CycleGAN. Figure 10 The image in (c) shows anomalies with spots. Observation reveals that the generated result and... Figure 11 The quality of data in (b) is comparable, and the dataset is more diverse, indicating that the model has good generalization ability and can simultaneously utilize both CG and real images to generate anomalous images in batches. This also verifies the CycleGAN model's ability to generate anomalous images, solving problems such as low production efficiency and data imbalance in tablet detection.

[0165] 4. Analyze the impact of each module of SDDRAM-CycleGAN on the generated results:

[0166] This embodiment conducts ablation experiments to analyze the impact of each module of SDDRAM-CycleGAN on the generated results. Different generation models were constructed by changing the model architecture, and then these models were used to generate images. Next, image quality assessment methods at the pixel level and data domain distribution level were used to analyze the specific contributions of different modules.

[0167] The SDDRAM-CycleGAN anomaly generation method employs a multi-module architecture, with its core modules including a SegNeXt generator, a dual discriminator (DD), and a region attention module (RAM). To verify the effectiveness of each module, this embodiment uses a progressive combination strategy to conduct ablation experiments, sequentially constructing the following four variants of the generation method:

[0168] (1) CycleGAN: The standard CycleGAN is adopted, whose generator consists of six residual blocks and is equipped with a single discriminator as the basic contrast model;

[0169] (2) RAM-CycleGAN: RAM is introduced, the generator still consists of six residual blocks, and a single discriminator is retained to verify the effect of RAM on the generation of anomalies;

[0170] (3) DDRAM-CycleGAN: DDRAM is introduced, and the generator still uses six residual blocks to form an architecture that combines DD and RAM, which is used to evaluate the effect of DD and RAM working together;

[0171] (4) SDDRAM-CycleGAN: Based on DDRAM-CycleGAN, the generator is replaced with the SegNeXt model, and the resulting complete architecture is used to verify the performance improvement of the SegNeXt model in the anomaly generation task.

[0172] Let the normal image of test set testA be denoted as The generation results of the above generation method variants (1) to (4) are respectively denoted as , , , The above image generation results are as follows: Figure 13 As shown

[0173] Depend on Figure 13 It can be seen that, The image quality is poor, the ability to generate anomalies is poor, and the spots on the surface of the tablet cannot be naturally embedded. It then completely lacks the ability to generate abnormal features; in comparison, The proposed SDDRAM-CycleGAN method achieved good generation results, with a significant improvement in image clarity. The highest quality not only boasts the best clarity and contrast but also generates the most diverse range of defects.

[0174] To quantitatively evaluate the quality of the generated images, a pixel-based evaluation method was first used to calculate the similarity between the generated image and the corresponding original image. It is known that the smaller the MAE and MSE values, and the larger the PSNR, SSIM, and MS-SSIM values, the better the image quality. Experimental results are shown in Table 2. and It outperforms other comparison methods in all indicators. It is worth noting that, combined with... The experimental results of generating defect-free images lead to the conclusion that the functional modules in the SDDRAM-CycleGAN method work together and are indispensable, jointly ensuring the effectiveness of abnormal image generation.

[0175] Table 2. Pixel-level quality comparison of generated images

[0176]

[0177] This invention further employs an image quality assessment method based on data domain distribution. It comprehensively evaluates the image quality by calculating the IS value of speckle image sets generated by different methods, as well as their domain distribution distances (FID and WD) with real drug speckle image sets. It is known that the larger the IS, the smaller the FID and WD, and the better the image quality, and the closer it is to the real speckle image. Experimental results are shown in Table 3. The fact that it performed best across all evaluation metrics demonstrates that SDDRAM-CycleGAN can most effectively generate anomalous images that conform to the characteristics of the real distribution.

[0178] Table 3. Comparison of domain distribution between generated image sets and real images

[0179]

[0180] 5. Comparison with existing generation methods:

[0181] To demonstrate the superiority of the method in this invention, this embodiment compares and analyzes the proposed method with other abnormal image generation methods. The image generation layer was fully trained on a single RTX 2080 Ti GPU in an offline environment, and then compared using two evaluation methods: 1) Generation model efficiency comparison: the training time and inference time of each model were statistically analyzed to assess efficiency; 2) Image quality comparison: evaluation metrics FID and IS based on data domain distribution were used to quantify the similarity between the speckle images generated by different methods and real pill speckle images.

[0182] The experimental results are shown in Table 4. The SDDRAM-CycleGAN method proposed in this invention demonstrates significant advantages in multiple dimensions. Regarding model efficiency, this method not only shortens the training time compared to DCGAN and DDPM, but also achieves an inference speed of 57.87 ms / frame, a significant improvement over DDPM. In the quality evaluation, SDDRAM-CycleGAN achieves the best results in both FID and IS, indicating that its generated images exhibit excellent quality and diversity.

[0183] A comparative analysis of the experimental results revealed that DDPM has significant limitations in practical applications due to its slow inference speed; while DAT-Net and DD-CycleGAN offer improvements in speed, their image quality is significantly worse; and DCGAN was eliminated due to its overall poor performance. In conclusion, SDDRAM-CycleGAN, through the design of the SegNeXt generator and DDRAM, achieves a good balance between image quality and computational efficiency. This method performs exceptionally well in anomalous image generation tasks, providing an effective solution to the data imbalance problem in drug visual inspection.

[0184] Table 4 Test results of different image generation models

[0185]

[0186] 6. Conclusion:

[0187] To address the data imbalance problem in the visual inspection of pharmaceutical tablets, this paper proposes the SDDRAM-CycleGAN anomalous image generation algorithm, which transforms normal tablet images into anomalous images with speckled patterns. Finally, the practical value of this proposed method in anomalous image generation is verified through image quality evaluation, ablation experiments of the generated model, and comparison with state-of-the-art methods. The conclusions are as follows:

[0188] (1) This invention innovatively designs a SegNeXt generator and a CycleGAN with a dual discriminator structure, and designs a region attention module to increase the ability to learn abnormal region features;

[0189] (2) Experiments have shown that the inference speed of the method in this invention reaches 57.87ms / frame, which is significantly better than other existing image generation methods. Furthermore, quantitative calculation of the two key indicators, FID and IS, shows that the generated images perform well in terms of quality and diversity.

[0190] The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A tablet abnormal image generation method based on a region attention mechanism, characterized in that, The method comprises the following steps: S1, obtaining a tablet normal image data set and a tablet abnormal image data set; S2, constructing an SDDRAM-CycleGAN model comprising a generator and a double-discriminator region attention module, wherein the generator is used to generate a corresponding tablet abnormal image from a tablet normal image, the double-discriminator region attention module comprises a double discriminator and an abnormal region attention module, and is used to comprehensively judge the authenticity of the generated tablet abnormal image; and the SDDRAM-CycleGAN model is guided to focus on the generation process of the abnormal region of the tablet by calculating the residual error between the tablet normal image and the tablet abnormal image; The S2 comprises: A block discriminator is used to perform block processing on the generated tablet abnormal image and output an abnormal classification result matrix of each image block; A region attention discriminator is used to perform pixel-level discrimination on the generated tablet abnormal image and output a discrimination matrix with the same size as the input image; S3, taking the tablet normal image data set as a source domain and the tablet abnormal image data set as a target domain, and performing bidirectional mapping training by using the double-discriminator region attention module; in the training process, the SDDRAM-CycleGAN model parameters are optimized by calculating a loss function; The S3 comprises: An adversarial loss is used to calculate the adversarial loss of the generator and the block discriminator and the region attention discriminator based on the output of the block discriminator and the region attention discriminator; A cycle consistency loss is used to constrain the consistency of the generated tablet abnormal image after being reconstructed by a reverse generator with the source domain data; An identity loss is used to constrain the output of the generator to remain unchanged for the target domain data; A region attention loss is used to guide the CycleGAN model to pay more attention to the generation and conversion of the abnormal region of the tablet by focusing on the gradient change of the abnormal region rather than the normal region; S4, generating a tablet normal image to be converted into an abnormal image by using the trained SDDRAM-CycleGAN model. 2.The tablet abnormal image generation method based on the region attention mechanism according to claim 1, characterized in that, In the S2, the generator adopts a SegNeXt architecture, and specifically comprises: A convolutional encoder comprises a multi-scale convolutional attention module, which extracts multi-scale features in the tablet normal image and generates attention weights through deep convolution, multi-branch deep strip convolution and 1×1 convolution; A decoder aggregates the multi-scale features of the tablet normal image extracted by the convolutional encoder, models the global context through a lightweight Hamburger module, and generates a tablet abnormal image. 3.The tablet abnormal image generation method based on the region attention mechanism according to claim 2, characterized in that, The convolutional encoder adopts a pyramid structure and comprises four down-sampling stages, each of which is composed of a down-sampling module and a cascaded multi-scale convolutional attention module. 4.The tablet abnormal image generation method based on the region attention mechanism according to claim 2, characterized in that, The multi-scale convolutional attention module extracts multi-scale features as attention weights through convolution operation, and performs weighted processing on the input of the multi-scale convolutional attention module by using the attention weights, specifically as follows: The output of the 1×1 convolution is used as the final attention weight to recalibrate the features of the input of the multi-scale convolutional attention module, and the expression is as follows: wherein, denotes input features, Att is an attention weight map, Out is output features, denotes an element-wise matrix multiplication operation, DW-Conv represents a depthwise convolution operation, represents a 1 x 1 convolution operation, denotes a multi-branch strip convolution, wherein . 5.The tablet abnormal image generation method based on the region attention mechanism according to claim 1, characterized in that, The patch discriminator consists of 5 layers of 4x4 convolutional layers, which finally divides the generated tablet abnormal image into a matrix of image blocks. 6.The tablet abnormal image generation method based on the region attention mechanism according to claim 1, characterized in that, The regional attention discriminator is composed of 3 layers of 1*1 convolutional layers, and outputs a discrimination matrix with the same size as the input image, which is used to judge whether each pixel of the generated tablet abnormal image belongs to the abnormal image category.

Images