Image defect detection method based on memory tree structure
By constructing a memory tree structure encoding network, potential vector decoding network and feature reconstruction network, the problems of abnormal information inflow and feature relationship loss are solved, the fusion of information within and between blocks is realized, and the efficiency and quality of image defect detection are improved. .
Patent Information
- Application Number
- CN202411238047.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-05
AI Technical Summary
In the method based on the reconstruction image generation, abnormal information is prone to flow into the feature space of the image, resulting in difficulty in normal recovery of defect images, and loss of features and relationships between blocks in image blocks, affecting the effect of image defect detection.
Build a memory tree structure encoding network, latent vector decoding network and feature reconstruction network. Through memory tree structure encoding and latent vector decoding, and combine feature reconstruction networks, the fusion of information within and between image blocks and enhance image reconstruction effect.
Effectively limit the inflow of abnormal information, enhance the robustness of image reconstruction, improve the efficiency and quality of image defect detection, improve the connection between the relationship between information and blocks in image blocks, and improve the image reconstruction effect.
Smart Images

Figure SMS_3 
Figure SMS_6 
Figure SMS_8
Abstract
Description
Technical Field
[0001] The invention belongs to an image defect detection method, and relates to the technical fields of memory modules, image vision processing, and computer vision industrial defect detection. Background Art
[0002] With the development of the industrial and computer fields, image detection technology in the industrial field has gradually matured. Image defect detection refers to a computer vision task that identifies the defective parts of an image among a large number of normal images. Image defect detection mainly includes three levels: whether the image has defects, where the defective parts of the image are, and how big the defects are. The current mainstream image defect detection technologies include two types of methods: image restoration and image reconstruction. Image restoration refers to simulating defects with noise as input in the network, converting the image defect detection task into an image denoising task, and completing image defect detection with image denoising as the goal. However, studies in recent years have shown that the ability of noise to simulate defects is limited, so the image restoration method limits its further development. The image reconstruction-based method takes learning the behavior pattern of normal images as the training goal to avoid the model being disturbed by abnormal information, so that even if a defective image is input, it can still be reconstructed into a normal image during image defect detection. The pixels between the input image and the output image are distinguished to determine whether the image contains defects. If the reconstruction effect is good, it can even determine the location and size of the image defects, completing the multi-level task of image defect detection.
[0003] However, the current methods based on reconstructed image generation face two major problems: First, abnormal information easily flows into the feature space of the image, making it difficult to restore the defective image to normal. Pidhorskyi, Perera and others constrained the latent space in an attempt to suppress the expression of abnormal information from the latent space. However, the powerful generalization ability of the autoencoder made it difficult to achieve strict spatial constraints. Subsequently, MemoryAE proposed a memory mechanism and tried to reorganize image features by establishing a memory pool. This method integrates the memory mechanism into anomaly detection and restricts features in the early stage of feature mapping. Second, the features within the image block and the relationship between blocks are lost. To this end, GANomaly attempts to encode the reconstructed image to achieve feature-assisted image reconstruction. Zhou et al. introduced a pre-trained module to strengthen the structural feature association between the input data and the image, trying to capture the structural information of the image. Summary of the invention
[0004] The purpose of the present invention is to solve the problem that abnormal information easily flows into the feature space of the image in the task of reconstructing the image, resulting in the difficulty of restoring the defective image to normality and the loss of the features within the image block and the relationship between the blocks. By solving the above two problems, the efficient expression of normal information and the connection between the information within the image block and the relationship between the blocks can be achieved, thereby enhancing the reconstruction effect of the reconstructed image.
[0005] The technical solution adopted by the present invention to solve the above technical problems is:
[0006] S1. Construct a memory tree structure encoding network for image encoding representation and potential vector generation;
[0007] S2. Construct a latent vector decoding network for latent vector decoding, image information collection during reconstruction, and reconstructed image generation;
[0008] S3. Construct a feature reconstruction network to reconstruct the image intra-block and inter-block information fusion;
[0009] S4. Combine the memory tree structure encoding network in S1, the latent vector decoding network in S2 and the feature reconstruction network in S3 to construct an image defect detection method architecture based on the memory tree structure;
[0010] S5. Training of image defect detection method architecture based on memory tree structure and defect image testing.
[0011] 1. The image defect detection method based on the memory tree structure according to claim 1, characterized in that the specific process of S1 is:
[0012] The memory tree structure encoding network is composed of a four-layer convolution block structure to perform preliminary compression on the input image. Through the combination of convolution and LeakyReLU activation function, the input image with 3 channels is compressed into a preliminary compressed representation of the input image with a feature dimension of H×W×C. This representation is denoted as Q. In the memory tree structure, PO memory pools are included, and IT memory items are set in each memory pool. By reading the memory items in the memory pool in a tree structure through Q, a post-compression representation of the corresponding memory pool will be generated. At the same time, after each reading is completed, the memory pool will be updated according to the previous query. Since PO memory pools are set, PO post-compression representations with Q as the root node will be generated. The expansion of the memory tree is shown below. First, Q is used as the reading unit, and the memory tree with a height of 2 is obtained through memory pools MP1 and MP2. and
[0013]
[0014] Then, and These two branch nodes are used as reading units, and read the two post-representations with a height of 3 in the memory tree structure through the corresponding memory pool:
[0015]
[0016] Then, As a reading unit, the memory leaf node at the end of the memory tree structure is queried through four different memory pools:
[0017]
[0018] Since the reading methods of the PO memory pools are consistent, only the input is different, the following takes Q as an example to introduce in detail the reading and updating methods in the memory tree structure.
[0019] Reading: Before reading, Q must first be channel-separated along the channel direction, so that it changes from the original size H×W×C to K=H×W C-dimensional vectors, each of which is denoted as query k During the reading process, each query in Q k Read the IT memory items ITEMS in the memory pool MP1 in sequence. Obtain a correlation graph of size IT×K by calculating the correlation between Q and ITEMS, and then apply the softmax function to convert the correlation value into a matching probability.
[0020]
[0021] Then, Q is re-expressed using the matching probability as the weight coefficient. Specifically, each vector is converted to:
[0022]
[0023] According to formula (12), each query k They are all re-represented by the IT memory items ITEMS in the memory pool MP1, thus completing the reading operation of Q.
[0024] Update: According to the matching probability calculated by softmax, the query can be measured k The relevance of all memory items can be sorted according to the relevance, and an index can be created to indicate it. Then the query closest to each memory item is selected. k Each memory item is updated. In the update operation, the previous correlation graph of size IT×K is continued to be used and normalized in the horizontal direction.
[0025]
[0026] Finally, the initial compressed representation Q and the PO post-compressed representations are fused through convolution, and the fusion result is a latent vector z with a dimension of 100, which completes the memory tree structure encoding of the input image.
[0027] 2. The image defect detection method based on the memory tree structure according to claim 1, characterized in that the specific process of S2 is:
[0028] A latent vector decoding network is constructed to decode the latent vector compressed by the memory tree structure encoder into a reconstructed image, and at the same time, the reconstruction information of the vector decoding process is collected for identification by the discriminator.
[0029] First, the latent vector z is input into the latent vector decoding network, and the latent vector z is decoded into a reconstructed image through the upsampling structure of four deconvolution stacks. During the decoding process, the decoding network will generate 256×4×4, 128×8×8, and 64×16×16 decoding process features in the deconvolution stack blocks, respectively, which are recorded as Decoder 256 , Decoder 128 , Decoder 64 .
[0030] Decoder 256 =ConvTrans(BN(RELU(z))) (16)
[0031] Decoder 128 =ConvTrans(BN(RELU(Decoder 256 ))) (17)
[0032] Decoder 64 =ConvTrans(BN(RELU(Decoder 128 ))) (18)
[0033] Then, Decoder 64 is fed into the fourth deconvolution block, where the latent vector z is finally reconstructed into the reconstructed image.
[0034] Recons=ConvTrans(Tanh(Decoder 64 )) (19)
[0035] In the latent vector decoding network, Decoder 256 , Decoder 128 , Decoder 64 They will be collected and stored separately to complete feature alignment, and the decoded features will be uniformly adjusted to a size of 64×16×16. The original decoder of this size 64No adjustments are made.
[0036] Decoder 256to64 =ConvTrans(Upsample(Decoder 256 )) (20)
[0037] Decoder 128to64 =ConvTrans(Decoder 128 ) (twenty one)
[0038] After aligning the features, Decoder 64 、Decoder 256to64 、Decoder 128to64 The corresponding positions of the discriminator are spliced along the channel direction and fused into the discriminator network, so that the discriminator learns the corresponding reconstruction process information.
[0039] 3. The image defect detection method based on the memory tree structure according to claim 1, characterized in that the specific process of S3 is:
[0040] After building the S1 memory tree structure encoding network and the S2 latent vector decoding network, a feature reconstruction network is constructed to extract the features of the reconstructed image. Referring to the latent vector dimension, the intra-block and inter-block information of the reconstructed image is fused to encode the reconstructed image.
[0041] The feature reconstruction network uses a two-way feature extraction method to extract features from the reconstructed image Recons. One way of feature extraction is to use the traditional convolution block to divide the reconstructed image Recons into 3×3 image blocks and model the information in the image blocks, which is recorded as F IN_BLO The other feature is the inter-block information feature extracted by the mixed convolution block MCB, denoted as F OUT_BLO The hybrid convolution block mainly contains three layers of features to complete the modeling of the reconstructed image Recons: pooling information F POOL Used to reduce invalid information of reconstructed image blocks, 1×1 convolution information F Conv Used to fuse the information between different channels of the reconstructed image, 3×3 hole convolution information F Dilation It is used to model the relationship information between different image blocks. Finally, the three layers of features of the mixed convolution block are fused through convolution, and the fusion method is as follows:
[0042] F OUT_BLO =Conv 1×1 Conv 3×3 (CONCAT(F POOL ,F Conv ,F Dilation )) (twenty two)
[0043] After the intra-block features of the traditional convolutional block and the inter-block features of the hybrid convolutional block are extracted, the dual-channel features are fused to generate feature reconstruction results.
[0044] 4. The image defect detection method based on the memory tree structure according to claim 1, characterized in that the specific process of S4 is:
[0045] S1, S2, and S3 respectively construct a memory tree structure encoding network, a latent vector decoding network, and a feature reconstruction network. Combining these three module networks, an image defect detection method based on a memory tree structure is constructed.
[0046] The present invention uses GAN as the basic architecture, and the memory tree structure encoding network and the potential vector decoding network jointly construct the generator network, which is used to complete the compression encoding and compression decoding of the input image and restore the normal form of the input image. The feature reconstruction network is used to reconstruct the compressed features as a feature supplement of the generator. At the same time, a convolutional discriminator is constructed to complete the generation mode discrimination of the input image and the memory tree structure generation network, and promote network training.
[0047] Specifically, the training of the image defect detection method based on the memory tree structure mainly includes three loss functions.
[0048] Feature matching loss: This completes the adversarial training of the generator G and the discriminator D, where x is the input image and f(·) maps the input image x to the intermediate feature representation of the discriminator.
[0049]
[0050] Generator reconstruction image loss: capture the context information of the input image and complete the reconstruction of the input image.
[0051]
[0052] Feature reconstruction loss: captures the compressed features z of the input image and the compressed features of the reconstructed image The context information of the network is optimized and trained using feature distance.
[0053]
[0054] Finally, the training objective of the image defect detection method based on the memory tree structure is defined, and each loss function is regulated by the regulation factor w:
[0055] L=w adv L adv +w G-Recon L G-Recon +w F-Recon LF-Recon (26)
[0056] 5. The image defect detection method based on the memory tree structure according to claim 1, characterized in that the specific process of S5 is:
[0057] The training method of the image defect detection method based on the memory tree structure is as follows:
[0058] This method is an unsupervised detection method for image defect detection. Only normal images are input during the training phase. The dimension of the input image is set to 32×32. The input image will be compressed into a latent vector of dimension 100 after passing through the encoder network of the memory tree structure. The latent vector is then input into the latent vector decoding network for decoding, so that the latent vector of dimension 100 is restored to a reconstructed image of 32×32. The 32×32 reconstructed image will be input into the feature reconstruction network for recompression, and the compressed dimension after recompression is also 100. During the training process, the input image and the reconstructed image will be alternately used for identification by the discriminator, thereby promoting the generator to generate more realistic images. During training, Pytorch 1.0 is used to build and implement the network, and the Adam optimizer is used to train the network. The main parameter settings during training are as follows:
[0059] The learning rate is set to 0.002; the training batch is set to 15; the potential vector dimension is set to 100; the training batch size is set to 64; the number of memory pools is set to 10; the memory items in the memory pool are set to 10; the weight coefficient of the training target is w G-Recon As the main optimization target, it is set to 50, and other coefficients are set to 1.
[0060] Compared with the prior art, the present invention has the following beneficial effects:
[0061] 1. The present invention proposes a new image defect detection method based on a memory tree structure. The network adopts a memory tree structure and uses multi-level memory compression features to fuse the input information, realizing a multi-level prototype model to re-represent the image and limit the influx of abnormal information.
[0062] 2. The present invention proposes a latent vector decoding network and a feature reconstruction network. The former enhances the discriminative ability of the discriminator by decoding the latent vector and collecting the hierarchical information of the image decoding process; the latter assists image reconstruction by fusing the intra-block information and inter-block information of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 Schematic diagram of the structure of the image defect detection method based on the memory tree structure.
[0064] Figure 2 and Figure 3 This is a comparison chart of the results of multi-class anomaly detection based on the memory tree structure image defect detection method.
[0065] Figure 4 This is a comparison chart of image defect detection results of image defect detection methods based on memory tree structure.
[0066] Figure 5 This is the visualization result diagram of image defect detection based on the image defect detection method based on the memory tree structure. DETAILED DESCRIPTION
[0067] The accompanying drawings are only used for illustrative purposes and should not be construed as limiting the present patent.
[0068] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0069] Figure 1 Figure 1 is a schematic diagram of the structure of an image defect detection method based on a memory tree structure. Figure 1 As shown, the entire structure consists of four parts: a memory tree structure encoder network, a latent vector decoding network, a feature reconstruction network, and a discriminator network.
[0070] exist Figure 1 In the example, the image is first input to the memory tree structure encoder network. The input image is initially compressed into Q through the structure of the four-layer convolutional block combination. Then, the memory items in the memory pool are read in a tree structure through Q, and the post-compression representation of the corresponding memory pool is generated. At the same time, after each reading, the memory pool will be updated according to the previous query. The expansion of the memory tree is as follows. First, Q is used as the reading unit, and the memory items with a height of 2 in the memory tree are obtained through memory pools MP1 and MP2. and
[0071]
[0072] Then, and These two branch nodes are used as reading units, and read the two post-representations with a height of 3 in the memory tree structure through the corresponding memory pool:
[0073]
[0074] Then, As a reading unit, the memory leaf node at the end of the memory tree structure is queried through four different memory pools:
[0075]
[0076] Since the reading methods of the PO memory pools are consistent, only the input is different, the following takes Q as an example to introduce in detail the reading and updating methods in the memory tree structure.
[0077] Reading: Before reading, Q must first be channel-separated along the channel direction, so that it changes from the original size H×W×C to K=H×W C-dimensional vectors, each of which is denoted as query k During the reading process, each query in Q k Read the IT memory items ITEMS in the memory pool MP1 in sequence. Obtain a correlation graph of size IT×K by calculating the correlation between Q and ITEMS, and then apply the softmax function to convert the correlation value into a matching probability.
[0078]
[0079] Then, Q is re-expressed using the matching probability as the weight coefficient. Specifically, each vector is converted to:
[0080]
[0081] According to formula (12), each query k They are all re-represented by the IT memory items ITEMS in the memory pool MP1, thus completing the reading operation of Q.
[0082] Update: According to the matching probability calculated by softmax, the query can be measured k The relevance of all memory items can be sorted according to the relevance, and an index can be created to indicate it. Then the query closest to each memory item is selected. k Each memory item is updated. In the update operation, the previous correlation graph of size IT×K is continued to be used and normalized in the horizontal direction.
[0083]
[0084] Finally, the initial compressed representation Q and the PO post-compressed representations are fused through convolution, and the fusion result is a latent vector z with a dimension of 100, which completes the memory tree structure encoding of the input image.
[0085] The latent vector decoding network is used to decode the latent vector compressed by the memory tree structure encoder into a reconstructed image, and at the same time collect the reconstruction information of the vector decoding process for the identification of the discriminator.
[0086] First, the latent vector z is input into the latent vector decoding network, and the latent vector z is decoded into a reconstructed image through the upsampling structure of four deconvolution stacks. During the decoding process, the decoding network will generate 256×4×4, 128×8×8, and 64×16×16 decoding process features in the deconvolution stack blocks, respectively, which are recorded as Decoder 256 , Decoder 128 , Decoder 64 .
[0087] Decoder 256 =ConvTrans(BN(RELU(z))) (16)
[0088] Decoder 128 =ConvTrans(BN(RELU(Decoder 256 ))) (17)
[0089] Decoder 64 =ConvTrans(BN(RELU(Decoder 128 ))) (18)
[0090] Then, Decoder 64 is fed into the fourth deconvolution block, where the latent vector z is finally reconstructed into the reconstructed image.
[0091] Recons=ConvTrans(Tanh(Decoder 64 )) (19)
[0092] In the latent vector decoding network, Decoder 256 , Decoder 128 , Decoder 64 They will be collected and stored separately to complete feature alignment, and the decoded features will be uniformly adjusted to a size of 64×16×16. The original decoder of this size 64 No adjustments are made.
[0093] Decoder 256to64 =ConvTrans(Upsample(Decoder 256 )) (20)
[0094] Decoder 128to64 =ConvTrans(Decoder 128 ) (twenty one)
[0095] After aligning the features, Decoder 64 、Decoder256to64 、Decoder 128to64 The corresponding positions of the discriminator are spliced along the channel direction and fused into the discriminator network, so that the discriminator learns the corresponding reconstruction process information.
[0096] After building the S1 memory tree structure encoding network and the S2 latent vector decoding network, a feature reconstruction network is constructed to extract the features of the reconstructed image. Referring to the latent vector dimension, the intra-block and inter-block information of the reconstructed image is fused to encode the reconstructed image.
[0097] The feature reconstruction network uses a two-way feature extraction method to extract features from the reconstructed image Recons. One way of feature extraction is to use the traditional convolution block to divide the reconstructed image Recons into 3×3 image blocks and model the information in the image blocks, which is recorded as F IN_BLO The other feature is the inter-block information feature extracted by the mixed convolution block MCB, denoted as F OUT_BLO The hybrid convolution block mainly contains three layers of features to complete the modeling of the reconstructed image Recons: pooling information F POOL Used to reduce invalid information of reconstructed image blocks, 1×1 convolution information F Conv Used to fuse the information between different channels of the reconstructed image, 3×3 hole convolution information F Dilation It is used to model the relationship information between different image blocks. Finally, the three layers of features of the mixed convolution block are fused through convolution, and the fusion method is as follows:
[0098] F OUT_BLO =Conv 1×1 Conv 3×3 (CONCAT(F POOL ,F Conv ,F Dilation )) (twenty two)
[0099] After the intra-block features of the traditional convolutional block and the inter-block features of the hybrid convolutional block are extracted, the dual-channel features are fused to generate feature reconstruction results.
[0100] Finally, a convolutional discriminator is constructed to complete the generation mode discrimination of the input image and the memory tree structure generation network, and promote network training.
[0101] Figure 2 and Figure 3 This is a comparison chart of the results of multi-class anomaly detection based on the memory tree structure image defect detection method. Figure 2 The results of multi-class anomaly detection on the MNIST dataset are demonstrated by the image defect detection method based on the memory tree structure. Figure 3The results of multi-class anomaly detection of this method on the CIFAR10 dataset are shown. From the figure, it can be seen that compared with the current advanced methods, the present invention has a significant improvement in both the overall results and the results of each single class.
[0102] Figure 4 The following is a comparison chart of the image defect detection results of the image defect detection method based on the memory tree structure. The MVTec AD dataset contains 5 texture class images and 10 target class images. In the comparison with the current advanced methods, it can be seen that the overall results of the present invention in both texture class and target class are superior to the current advanced methods.
[0103] Figure 5 This is a visualization result diagram of image defect detection based on the memory tree structure. The present invention selects six categories for visualization. The left side is the defect of the target, and the right side is the reconstruction of the defect by the present invention. Figure 5 It can be seen that the present invention can repair image defect areas and realize image defect detection.
[0104] The present invention proposes an image defect detection method based on a memory tree structure, which is divided into four parts: a memory tree structure encoding network, a latent vector decoding network, a feature reconstruction network, and a discriminator network. The memory tree structure encoding network uses the characteristics of the tree structure to control abnormal information and achieves high-quality input image feature mapping. The latent vector decoding network collects and aligns the hierarchical features in the reconstruction process, and migrates them to the discriminator network for information flow between modules. The feature reconstruction network pays attention to enhancing the fusion of information within the image block and between blocks, and uses feature contrast to assist the image in reconstruction, thereby enhancing the robustness of image reconstruction, improving the image reconstruction effect, and further improving the efficiency and quality of image defect detection. At the same time, the present invention also verifies the method for multiple types of abnormality detection and image defect detection data, and this method has great advantages over the current advanced methods.
[0105] Finally, the details of the above examples of the present invention are only examples for explaining the present invention. For those skilled in the art, any modification, improvement and replacement of the above embodiments should be included in the protection scope of the claims of the present invention.
Claims
1. An image defect detection method based on a memory tree structure, characterized in that: The method comprises the following steps: S1. Construct a memory tree structure encoding network for image encoding representation and potential vector generation; S2. Construct a latent vector decoding network for latent vector decoding, image information collection during reconstruction, and reconstructed image generation; S3. Construct a feature reconstruction network to reconstruct the image intra-block and inter-block information fusion; S4. Combine the memory tree structure encoding network in S1, the latent vector decoding network in S2 and the feature reconstruction network in S3 to construct an image defect detection method architecture based on the memory tree structure; S5. Training of image defect detection method architecture based on memory tree structure and defect image testing.
2. The image defect detection method based on the memory tree structure according to claim 1, characterized in that: The specific process of S1 is as follows: The memory tree structure encoding network is composed of a four-layer convolution block structure to perform preliminary compression on the input image. Through the combination of convolution and LeakyReLU activation function, the input image with 3 channels is compressed into a preliminary compressed representation of the input image with a feature dimension of H×W×C. This representation is denoted as Q. In the memory tree structure, PO memory pools are included, and IT memory items are set in each memory pool. By reading the memory items in the memory pool in a tree structure through Q, a post-compression representation of the corresponding memory pool will be generated. At the same time, after each reading is completed, the memory pool will be updated according to the previous query. Since PO memory pools are set, PO post-compression representations with Q as the root node will be generated. The expansion of the memory tree is shown below. First, Q is used as the reading unit, and the memory tree with a height of 2 is obtained through memory pools MP1 and MP2. and Then, and These two branch nodes are used as reading units, and read the two post-representations with a height of 3 in the memory tree structure through the corresponding memory pool: Then, As a reading unit, the memory leaf node at the end of the memory tree structure is queried through four different memory pools: Since the reading methods of the PO memory pools are consistent, only the input is different, the following takes Q as an example to introduce in detail the reading and updating methods in the memory tree structure. Reading: Before reading, Q must first be channel-separated along the channel direction, so that it changes from the original size H×W×C to K=H×W C-dimensional vectors, each of which is denoted as query k During the reading process, each query in Q k Read the IT memory items ITEMS in the memory pool MP1 in sequence. Obtain a correlation graph of size IT×K by calculating the correlation between Q and ITEMS, and then apply the softmax function to convert the correlation value into a matching probability. Then, Q is re-expressed using the matching probability as the weight coefficient. Specifically, each vector is converted to: According to formula (12), each query k They are all re-represented by the IT memory items ITEMS in the memory pool MP1, thus completing the reading operation of Q. Update: According to the matching probability calculated by softmax, the query can be measured k The relevance of all memory items can be sorted according to the relevance, and an index can be created to indicate it. Then the query closest to each memory item is selected. k Each memory item is updated. In the update operation, the previous correlation graph of size IT×K is continued to be used and normalized in the horizontal direction. Finally, the initial compressed representation Q and the PO post-compressed representations are fused through convolution, and the fusion result is a latent vector z with a dimension of 100, which completes the memory tree structure encoding of the input image.
3. The image defect detection method based on the memory tree structure according to claim 1, characterized in that: The specific process of S2 is: A latent vector decoding network is constructed to decode the latent vector compressed by the memory tree structure encoder into a reconstructed image, and at the same time, the reconstruction information of the vector decoding process is collected for identification by the discriminator. First, the latent vector z is input into the latent vector decoding network, and the latent vector z is decoded into a reconstructed image through the upsampling structure of four deconvolution stacks. During the decoding process, the decoding network will generate 256×4×4, 128×8×8, and 64×16×16 decoding process features in the deconvolution stack blocks, respectively, which are recorded as Decoder 256 , Decoder 128 , Decoder 64 . Decoder 256 =ConvTrans(BN(RELU(z))) (16) Decoder 128 =ConvTrans(BN(RELU(Decoder 256 ))) (17) Decoder 64 =ConvTrans(BN(RELU(Decoder 128 ))) (18) Then, Decoder 64 is fed into the fourth deconvolution block, where the latent vector z is finally reconstructed into the reconstructed image. Recons=ConvTrans(Tanh(Decoder 64 )) (19) In the latent vector decoding network, Decoder 256 , Decoder 128 , Decoder 64 They will be collected and stored separately to complete feature alignment, and the decoded features will be uniformly adjusted to a size of 64×16×16. The original decoder of this size 64 No adjustments are made. Decoder 256to64 =ConvTrans(Upsample(Decoder 256 )) (20) Decoder 128to64 =ConvTrans(Decoder 128 ) (21) After aligning the features, Decoder 64 、Decoder 256to64 、Decoder 128to64 The corresponding positions of the discriminator are spliced along the channel direction and fused into the discriminator network, so that the discriminator learns the corresponding reconstruction process information.
4. The image defect detection method based on the memory tree structure according to claim 1, characterized in that: The specific process of S3 is as follows: After building the S1 memory tree structure encoding network and the S2 latent vector decoding network, a feature reconstruction network is constructed to extract the features of the reconstructed image. Referring to the latent vector dimension, the intra-block and inter-block information of the reconstructed image is fused to encode the reconstructed image. The feature reconstruction network uses a dual-path feature extraction method to extract features from the reconstructed image Recons. One feature is to use the traditional convolution block to divide the reconstructed image Recons into 3×3 image blocks and model the information in the image blocks, denoted as F IN_BLO The other feature is the inter-block information feature extracted by the mixed convolution block MCB, denoted as F OUT_BLO The hybrid convolution block mainly contains three layers of features to complete the modeling of the reconstructed image Recons: pooling information F POOL Used to reduce invalid information of reconstructed image blocks, 1×1 convolution information F Conv Used to fuse the information between different channels of the reconstructed image, 3×3 hole convolution information F Dilation It is used to model the relationship information between different image blocks. Finally, the three layers of features of the mixed convolution block are fused through convolution, and the fusion method is as follows: F OUT_BLO =Conv 1×1 Conv 3×3 (CONCAT(F POOL ,F Conv ,F Dilation )) (22) After the intra-block features of the traditional convolutional block and the inter-block features of the hybrid convolutional block are extracted, the dual-channel features are fused to generate feature reconstruction results.
5. The image defect detection method based on the memory tree structure according to claim 1, characterized in that: The specific process of S4 is as follows: S1, S2, and S3 respectively construct a memory tree structure encoding network, a latent vector decoding network, and a feature reconstruction network. Combining these three module networks, an image defect detection method based on a memory tree structure is constructed. The present invention uses GAN as the basic architecture, and the memory tree structure encoding network and the potential vector decoding network jointly construct the generator network, which is used to complete the compression encoding and compression decoding of the input image and restore the normal form of the input image. The feature reconstruction network is used to reconstruct the compressed features as a feature supplement of the generator. At the same time, a convolutional discriminator is constructed to complete the generation mode discrimination of the input image and the memory tree structure generation network, and promote network training. Specifically, the training of the image defect detection method based on the memory tree structure mainly includes three loss functions. Feature matching loss: This completes the adversarial training of the generator G and the discriminator D, where x is the input image and f(·) maps the input image x to the intermediate feature representation of the discriminator. Generator reconstruction image loss: capture the context information of the input image and complete the reconstruction of the input image. Feature reconstruction loss: captures the compressed features z of the input image and the compressed features of the reconstructed image The context information of the network is optimized and trained using feature distance. Finally, the training objective of the image defect detection method based on the memory tree structure is defined, and each loss function is regulated by the regulation factor w: L=w adv L adv +w G-Recon L G-Recon +w F-Recon L F-Recon (26)。 6. The image defect detection method based on the memory tree structure according to claim 1, characterized in that: The specific process of S5 is as follows: The training method of the image defect detection method based on the memory tree structure is as follows: This method is an unsupervised detection method for image defect detection. Only normal images are input during the training phase. The dimension of the input image is set to 32×32. The input image will be compressed into a latent vector of dimension 100 after passing through the encoder network of the memory tree structure. The latent vector is then input into the latent vector decoding network for decoding, so that the latent vector of dimension 100 is restored to a reconstructed image of 32×32. The 32×32 reconstructed image will be input into the feature reconstruction network for recompression, and the compressed dimension after recompression is also 100. During the training process, the input image and the reconstructed image will be alternately used for identification by the discriminator, thereby promoting the generator to generate more realistic images. During training, Pytorch 1.0 is used to build and implement the network, and the Adam optimizer is used to train the network. The main parameter settings during training are as follows: The learning rate is set to 0.002; the training batch is set to 15; the potential vector dimension is set to 100; the training batch size is set to 64; the number of memory pools is set to 10; the memory items in the memory pool are set to 10; the weight coefficient of the training target is w G-Recon As the main optimization target, it is set to 50, and other coefficients are set to 1.
Citation Information
Patent Citations
Image anomaly detection method based on memory network
CN114882007A
Monitoring video anomaly detection method based on automatic encoder and memory network
CN116434132A
