Banknote paper defect classification and evaluation method and device based on twin neural network

Abstract

The invention provides a currency paper defect classification and evaluation method and device based on a twin neural network, and relates to the technical field of artificial intelligence and defect detection and recognition, and the method comprises the steps: segmenting defects from a detection image through employing a defect segmentation network model, and obtaining a defect binary image; performing masking processing on the defect binary image, obtaining a defect foreground image from the detection image, and inputting the defect foreground image into a defect classification network model for defect type identification and classification to obtain defect type data of the detection image; performing image processing and defect evaluation on the defect binary image, and evaluating the overall damage degree of the detection image according to morphological characteristics of defects to obtain a quality risk score value of the detection image; and according to the quality risk score value and the defect type data of the detection image, detecting the defects of the banknote paper, analyzing to obtain the defect grade of the banknote paper, and according to the defect type data and the quality risk score value of the whole paper reel, evaluating and optimizing the production process of the production line.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and defect detection and recognition technology, and in particular to a method and device for classifying and evaluating banknote defects based on a twin neural network. Background Art

[0002] This section is intended to provide a background or context for embodiments of the present invention. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] In the existing banknote production appearance quality defect detection system, the appearance quality of banknotes is based on the gray value of the product image for abnormal detection, such as template matching, subtracting the image from the template mean image to obtain a residual image, and calculating the position and size of the defect through the BLOB algorithm. However, in the existing technology, the accuracy of the area statistics result is low, the detection effect is affected by the detection threshold, and the detected defects cannot accurately correspond to the defect names in the actual production process, resulting in the inability of technicians to quickly and efficiently evaluate and improve the production line.

[0004] In summary, there is an urgent need for a technical solution that can overcome the above-mentioned defects, effectively improve the detection rate of banknote defects, provide effective support for evaluating and analyzing production line problems, and improve production quality. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention proposes a method and device for classifying and evaluating banknote defects based on a twin neural network. The present invention obtains a defect binary image by pixel-level segmentation of the original image, and performs single defect quality risk assessment through morphological analysis; further obtains a defect foreground image based on the defect binary image, and classifies the defect foreground image; evaluates the quality risk and cause of the paper shaft and the production process problems based on the defect quantity and quality risk score of the entire shaft, providing strong technical support for analyzing banknote defects and evaluating the improvement of production line processes.

[0006] In a first aspect of an embodiment of the present invention, a method for classifying and evaluating banknote defects based on a twin neural network is proposed, the method comprising:

[0007] The defect segmentation network model is used to segment defects from the inspection image to obtain a defect binary image;

[0008] Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder;

[0009] Masking the defect binary image to obtain a defect foreground image from the detection image, and inputting the defect foreground image into the defect classification network model to identify and classify the defect type to obtain defect type data of the detection image;

[0010] Performing image processing and defect assessment on the defect binary image, assessing the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtaining a quality risk score value of the inspection image;

[0011] Based on the quality risk score value and defect type data of the detection image, banknote defects are detected, and the defect classification of the banknote is obtained by analysis. Based on the defect type data and quality risk score value of the entire paper shaft, the production process of the production line is evaluated and optimized.

[0012] In a second aspect of an embodiment of the present invention, a device for classifying and evaluating banknote defects based on a twin neural network is proposed, the device comprising:

[0013] A defect segmentation processing module is used to segment defects from the detection image using a defect segmentation network model to obtain a defect binary image;

[0014] Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder;

[0015] A defect type recognition and classification module is used to perform mask processing on the defect binary image, obtain a defect foreground image from the detection image, and input the defect foreground image into the defect classification network model to recognize and classify the defect type, so as to obtain defect type data of the detection image;

[0016] A quality risk analysis module is used to perform image processing and defect assessment on the defect binary image, assess the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtain a quality risk score value of the inspection image;

[0017] The defect analysis and evaluation module is used to detect banknote defects based on the quality risk score value and defect type data of the detection image, analyze and obtain the defect grade of the banknote paper, and evaluate and optimize the production process of the production line based on the defect type data and quality risk score value of the entire paper shaft.

[0018] In a third aspect of an embodiment of the present invention, a computer device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for classifying and evaluating banknote defects based on a twin neural network is implemented.

[0019] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method for classifying and evaluating banknote defects based on a twin neural network is implemented.

[0020] In a fifth aspect of an embodiment of the present invention, a computer program product is proposed, which includes a computer program, and when the computer program is executed by a processor, it implements a method for classifying and evaluating banknote defects based on a twin neural network.

[0021] The paper money defect classification and evaluation method and device based on twin neural network proposed in the present invention obtains a defect binary image by pixel-level segmentation of the original image, and performs single defect quality risk evaluation through morphological analysis; further obtains a defect foreground image based on the defect binary image, and classifies the defect foreground image; evaluates the quality risk and cause of the paper shaft according to the number of defects and quality risk score of the whole shaft, so as to evaluate the production process problems. Based on the evaluation results, technicians can quickly and efficiently locate paper money defects, improve and optimize the production line process, optimize the production process flow, improve equipment accuracy, strengthen quality control, etc., so as to improve the production efficiency and product quality of the production line, realize the improvement and optimization of the production line process, and provide strong technical support for analyzing paper money defects and evaluating the improved production line process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a schematic flow chart of a method for classifying and evaluating banknote defects based on a twin neural network according to an embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of a process of segmenting defects from a detection image using a defect segmentation network model according to an embodiment of the present invention.

[0025] Figure 3 It is a schematic diagram of a flow chart of inputting a defect foreground image into a defect classification network model to identify and classify defect types according to an embodiment of the present invention.

[0026] Figure 4 It is a flowchart of a process for calculating defect risk scores, analyzing scores, and conducting alarm and evaluation according to an embodiment of the present invention.

[0027] Figure 5 4 is a schematic diagram of a defect segmentation network according to an embodiment of the present invention.

[0028] Figure 6 It is a schematic diagram of the structure of an attention comparison module according to an embodiment of the present invention.

[0029] Figure 7 It is a schematic diagram of the architecture of a banknote defect classification and evaluation device based on a twin neural network according to an embodiment of the present invention.

[0030] Figure 8 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0033] According to an embodiment of the present invention, a method and device for classifying and evaluating banknote defects based on a twin neural network are proposed, which relates to the field of artificial intelligence and defect detection and identification technology.

[0034] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0035] Figure 1 FIG. 1 is a flow chart of a method for classifying and evaluating banknote defects based on a twin neural network according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0036] S101, segmenting defects from the inspection image using a defect segmentation network model to obtain a defect binary image;

[0037] Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder;

[0038] S102, performing mask processing on the defect binary image to obtain a defect foreground image from the detection image, and inputting the defect foreground image into the defect classification network model to perform defect type recognition and classification to obtain defect type data of the detection image;

[0039] S103, performing image processing and defect assessment on the defect binary image, assessing the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtaining a quality risk score value of the inspection image;

[0040] S104, detecting banknote defects based on the quality risk score and defect type data of the detection image, analyzing to obtain a defect grade for the banknote, and evaluating and optimizing the production process of the production line based on the defect type data and quality risk score of the entire paper shaft.

[0041] In actual application scenarios, the banknote defect classification and evaluation method based on twin neural networks proposed in the present invention has the following working principle: using a defect segmentation network model to segment defects from a detection image to obtain a defect binary image; masking the defect binary image to obtain a defect foreground image from the detection image, and inputting the defect foreground image into the defect classification network model to identify and classify the defect type to obtain defect type data of the detection image; performing image processing and defect evaluation on the defect binary image, and evaluating the overall degree of damage of the detection image according to the morphological characteristics of the defects to obtain a quality risk score value of the detection image; detecting banknote defects according to the quality risk score value and defect type data of the detection image, analyzing to obtain a defect grade for the banknote, and evaluating and optimizing the production process of the production line according to the defect type data and quality risk score value of the entire paper shaft. The overall solution obtains a binary defect map by pixel-level segmentation of the original image, and performs a single defect quality risk assessment through morphological analysis; further, the defect foreground image is obtained based on the defect binary map, and the defect foreground image is classified; according to the number of defects and quality risk scores of the entire axis, the quality risk and cause of the paper axis are assessed to evaluate the production process problems. Based on the evaluation results, technicians can quickly and efficiently locate banknote defects, improve and optimize the production line process, optimize the production process, improve equipment accuracy, strengthen quality control, etc., to improve the production efficiency and product quality of the production line, and achieve the improvement and optimization of the production line process, providing strong technical support for analyzing banknote defects and evaluating the improvement of production line processes.

[0042] In order to explain more clearly the above-mentioned banknote defect classification and evaluation method based on twin neural network, each step is explained in detail below.

[0043] S101:

[0044] The defect segmentation network model is used to segment defects from the inspection image and obtain a defect binary image.

[0045] An attention comparison module is provided in the defect segmentation network model, and the attention comparison module is used for input feature comparison between twin neural networks, dividing the image into slices, and setting weights from high to low for the contribution scores of small slices from the center to the edge of the image.

[0046] The resolution size of the template image is larger than the defect image. Accordingly, the defect segmentation network model adds a hole convolution layer in the last layer of the encoder to increase the receptive field of the defect segmentation network model without image downsampling.

[0047] refer to Figure 2 , is a flow chart of segmenting defects from a detection image using a defect segmentation network model according to an embodiment of the present invention. Figure 2 As shown, the specific process is:

[0048] S201, collecting template images and defect images; wherein, a template image is prepared, wherein the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is collected by a detection system, defect position information is obtained during collection, and a corresponding template image is marked on the template image according to the defect position information;

[0049] S202, cleaning the collected template images corresponding to the defects, deleting the image groups with incorrect annotations, or the image groups with a deviation between the template image and the defect image greater than a deviation threshold;

[0050] S203, annotating the defect positions in the defect data at the pixel level to generate a binary label image, and putting the binary label image into the folders of to-be-detected, template, and label, respectively, with the data corresponding one to one;

[0051] S204, constructing and training a twin neural network, loading the data in the folder, performing enhancement processing on the data to expand the data set, and inputting the processed training data into the twin neural network for training;

[0052] S205, optimizing network parameters through training learning, establishing a mapping relationship between defective image pixels in the sample data set and corresponding template pixels; during model testing, calculating the difference between the current image to be detected and the template image through the trained network parameters, and determining the probability that the pixel position is a defective pixel;

[0053] S206, using the trained twin neural network, by inputting the image to be inspected transmitted on the production line and the template image stored in the template library, after the model forward calculation, the defect binary image containing the defect is obtained through the softmax normalization function.

[0054] Specifically, for (S204), the twin neural network is constructed and trained, the data in the folder is loaded, the data is enhanced and processed to expand the data set, and the processed training data is input into the twin neural network for training. In the model training process, the cross entropy of the binary classification is used as the loss function, and the formula of the loss function is expressed as:

[0055]

[0056] In the formula, y is the calculated value of the loss function; N is the total number of samples; p i Y is the probability that the network predicts a positive sample; i Represents the label of the i-th sample, the positive class is 0 and the negative class is 1;

[0057] Specifically, for (S206), the trained twin neural network is used to input the image to be detected transmitted on the production line and the template image stored in the template library. After the model is forward calculated, the defect binary image containing the defect is obtained by processing with the softmax normalization function, including:

[0058] The softmax normalization function is expressed using the following formula:

[0059] Y = softmax(z(w,h))

[0060] Where Y is a binary image; z is the tensor of the input image; w and h are the position indexes of the pixels in the image.

[0061] S102:

[0062] The defect foreground image is input into the defect classification network model to identify and classify the defect type, so as to obtain the defect type data of the detection image.

[0063] The defect foreground image is input into a defect classification network model to identify and classify the defect type, and the corresponding names of the banknote production process defects are obtained. According to different defect corresponding names, different defect assessment calculation methods are used to obtain defect assessment data, wherein the defect assessment data obtained using different defect assessment calculation methods at least includes: calculating the foreground area based on spot defects; calculating the number of foreground pixels and the edge gradient of the foreground original image based on missing defects; calculating the ratio of defect foreground to macro-area template based on deformation defects; and calculating the foreground and background grayscale variance values ​​based on uniformity defects.

[0064] refer to Figure 3 , is a flow chart of inputting a defect foreground image into a defect classification network model to identify and classify defect types according to an embodiment of the present invention. Figure 3 As shown, the specific process is:

[0065] S301, marking defect types according to the collected defect corresponding template diagrams;

[0066] S302, cleaning the labeled data and deleting the images with incorrect labels, wherein the labeling method is to save the images in the form of image naming, with different labels representing different defect types;

[0067] S303, constructing and training a defect classification network model, and expanding the data set by using an enhanced processing method; reducing the gap between the predicted category and the actual category through training, and obtaining the optimal parameters of the defect classification network model by training the model until convergence;

[0068] S304, using the trained defect classification network model, inputting the defect image, and outputting an n-dimensional vector, each dimension of the n-dimensional vector represents a specific defect type, and n is the total number of defect types.

[0069] Specifically, for (S303), the defect classification network model is constructed and trained, and the data set is expanded by an enhanced processing method; the gap between the predicted category and the actual category is reduced by training, and the optimal parameters of the defect classification network model are obtained by training the model until convergence. The enhanced processing method of the expanded data set includes at least the use of random scale transformation, HSV spatial jitter, and random cropping when inputting into the network, which is described by the following formula:

[0070] G=g(x,ω)

[0071] In the formula, g represents the classification network model; x represents the input defect image; ω represents the model parameter; G represents an n-dimensional vector of the output, and n represents the number of defect categories;

[0072] Through training, the gap between the predicted category and the actual category is reduced, which can be described by the following formula:

[0073] index=argmin(1+∑ i∈pos e -G[i] )

[0074] Among them, G[i] represents the element of vector G; i is the defect type serial number; index is the category number finally predicted by the network; pos represents the real defect type set of the input image;

[0075] The optimal parameters of the defect classification network model are obtained by training the model until convergence.

[0076] The processing procedures of S103 and S104 are described in detail below.

[0077] S103:

[0078] The defect binary image is subjected to image processing and defect assessment, and the overall damage degree of the detection image is assessed according to the morphological characteristics of the defects to obtain a quality risk score value of the detection image.

[0079] S104:

[0080] Based on the quality risk score value and defect type data of the detection image, banknote defects are detected, and the defect classification of the banknote is obtained by analysis. Based on the defect type data and quality risk score value of the entire paper shaft, the production process of the production line is evaluated and optimized.

[0081] refer to Figure 4, is a flow chart of the process of calculating defect risk scores, analyzing scores, and performing alarm and evaluation according to an embodiment of the present invention. Figure 4 As shown, the specific process is:

[0082] S401, calculating the risk score of each type of defect of the paper shaft according to the classification result and the segmentation defect binary map;

[0083] When the score of one type of defect risk exceeds a preset first threshold, an alarm is issued;

[0084] S402, according to the score of each type of defect risk of the paper shaft, a total score of the defect risk of the paper shaft is obtained by summarizing;

[0085] S403, judging the overall quality risk of the paper shaft based on the total score of the paper shaft defect risk;

[0086] If the total score exceeds the preset second threshold, an alarm is issued, and a manual random inspection or full inspection is initiated when the total score reaches the manual inspection threshold;

[0087] S404, guide the process and adjust the technology according to the alarm of defect type.

[0088] Specifically, for (S401), the risk score of each type of defect of the paper shaft is calculated according to the classification result and the segmentation defect binary map. The specific algorithm for scoring is:

[0089] According to the defect type of the classification result, the foreground area, the edge gradient of the foreground original image, the ratio of the foreground and macro area, and the grayscale variance of the foreground and background are selected as indicators. According to the preset tolerance threshold, the severity score is calculated by linear fitting. The severity score formula is:

[0090]

[0091] Where S is the severity score of the defect image; Thd_S is the severity threshold of one type of defect; Mes_S is the calculation result of the defect image index; when the severity score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification;

[0092] Using foreground area, foreground grayscale variance, and foreground-background contrast as indicators, and according to the preset tolerance threshold, linear fitting is used to calculate the undetectability score. The undetectability score formula is:

[0093]

[0094] Where D is the undetectability score of the defect image; Thd_D is the undetectability threshold of one type of defect; Mes_D is the calculation result of the defect image index; when the undetectability score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification.

[0095] Specifically, for (S403) judging the overall quality risk of the paper shaft based on the total score of the paper shaft defect risk, the detailed algorithm and judgment logic are as follows:

[0096] According to the defect score of the entire axis, the overall risk score of the axis is calculated; if it exceeds the tolerance threshold, an alarm is issued; the calculation formula for the overall risk score of the entire axis is:

[0097]

[0098] Where Z is the overall risk score of the shaft; N' is the total number of large sheets produced; i is the defect type number; n i is the total number of defects of type i; S i is the mean severity score of the i-th defect; D i is the undetectability score of the i-th defect.

[0099] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0100] The processing process of the present invention can be considered as a process of classifying and evaluating banknote paper defects. First, the original image is segmented at the pixel level to obtain a defect binary image, and a single defect quality risk assessment is performed through morphological analysis; further, a defect foreground image is obtained based on the defect binary image, and the defect foreground image is classified; finally, based on the number of defects and quality risk scores of the entire axis, the quality risk and cause of the paper axis are evaluated, thereby evaluating the production process problems.

[0101] Among them, a deep learning-based defect segmentation network model is used for image processing, and a defect classification network model is used for classification. The defect risk score, defect type, quantity and other indicators are combined to evaluate the quality of the paper shaft and improve the production process.

[0102] Specifically, the defects are segmented from the input detection image through a deep learning-based defect segmentation network to obtain a defect binary map. Compared with other open source twin neural networks, this network targets the characteristics of banknote defect images, that is, the defect center is generally located in the center of the image, and the defect area size is approximately Gaussian distributed; an attention comparison module (LACB, Local Attention Contrastive Block) is designed for input feature comparison between twin neural networks, dividing the image into slices, and setting weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image.

[0103] In addition, in order to avoid incorrect segmentation caused by the position floating of banknote anti-counterfeiting elements due to the flexible changes in paper printing, the resolution size of the template image is larger than the defect image. Therefore, a larger receptive field is required to find the anti-counterfeiting element features of the banknote background from the larger resolution image to avoid segmenting the anti-counterfeiting element features as the defect foreground. The network adds a hole convolution layer (ASPP, Atrous Spatial Pyramid Pooling) to the last layer of the encoder to increase the receptive field of the network model without downsampling the image.

[0104] The defect image is segmented into a binary image to obtain a defect foreground image, which is then passed into the defect classification network for classification. The detected image is divided into names corresponding to defects in the banknote production process. Different defect evaluation calculation methods are used according to different defect names. For example, for spot defects, the foreground area is mainly calculated; for missing defects, the number of foreground pixels and the edge gradient of the foreground original image are mainly calculated; for deformation defects, the ratio of defect foreground to macro-area template is calculated; for uniformity defects, the foreground and background grayscale variance values ​​are calculated.

[0105] At the same time, the defect binary image can also be input into the defect assessment module based on image processing. According to the morphological characteristics of the defects, such as the exact number of pixels of foreground defects, the average grayscale of foreground pixels, edge gradient, and contrast with the background, the overall damage degree of the inspection image is evaluated and a quality risk score value of the inspection image is given.

[0106] According to the defect grading of banknote paper and the defect classification results of the entire paper shaft and the risk score value of the image quality of the entire paper shaft detection, the problems existing in the production process of the production line are evaluated. According to the evaluation results, the production line process is improved and optimized, such as optimizing the production process flow, improving equipment accuracy, strengthening quality control, etc., to improve the production efficiency and product quality of the production line and realize the improvement and optimization of the production line process.

[0107] 1. Through the deep learning-based defect segmentation network, the defects are segmented from the input detection image to obtain a defect binary image.

[0108] In actual application scenarios, the template image is obtained by averaging the image of the banknote to be tested and multiple defect-free banknotes, and the template image is input into the twin neural network composed of the backbone network with the same structure. The banknote to be tested and the template banknote have the same resolution. The network detects defects by comparing multi-layer feature information and finding differences to obtain a defect binary image. In particular, if there is no defect in the background image, the final output defect binary image is an image of all 0s. If there is a defect in the background, the pixel value of the defective part segmented by the network is 1, and the pixel value of other defect-free positions is 0.

[0109] The specific process is:

[0110] Step A1: Data collection:

[0111] Collect template images and defect images; the template image is made by superimposing and averaging multiple defect-free banknote images, and the defect image is collected by a traditional detection system, and the defect location information is obtained while collecting, and the corresponding template image is marked on the template image according to the defect location information. The collected images contain defect sample data of different products and different regions.

[0112] Step A2, data cleaning and labeling: First, clean the collected defect template data, delete the image groups with obvious incorrect labels or the image groups with large deviations between the template image and the defect image, and ensure that the number of defects of various types is as balanced as possible; then, use the labeling software to label the defect positions in the defect data at the pixel level to generate a binary label image. Put the processed data into the three folders of to-be-detected, template, and label, and the data corresponds one to one.

[0113] Step A3, build and train the twin neural network: after loading the data, perform random scale transformation, HSV spatial jitter, random cropping and other enhancement processing on the data to expand the data set, and input the processed training data into the twin neural network improved and optimized for banknote defects. The model includes an attention comparison module, and adds a void space pyramid pooling module after the encoder, such as Figure 5 shown.

[0114] exist Figure 5 The terms that need to be explained are:

[0115] Example image Flaw, a sample image with defects input to the model.

[0116] Sample image Temp, a sample image of a normal state input into the model for comparison.

[0117] First_Convl_Flaw, the initial convolution layer, is used to process sample images with defects.

[0118] First_Convl_Temp, the initial convolution layer, is used to process sample images in the normal state.

[0119] MaxPooling, the maximum pooling layer, reduces the data dimension and extracts important features through sampling operations.

[0120] ResNetLayer, the residual network layer, is a special convolutional layer that solves the gradient vanishing problem of deep neural networks by introducing skip connections. Figure 5In the figure, corresponding to the example image Flaw and the example image Temp, four layers of ResNetLayer are set, namely ResNetLayer1_Flaw, ResNetLayer2_Flaw, ResNetLayer3_Flaw, ResNetLayer4_Flaw, ResNetLayer1_Temp, ResNetLayer2_Temp, ResNetLayer3_Temp, and ResNetLayer4_Temp.

[0121] LACB, Local Attention Contrastive Block, is an attention comparison module used to compare input features between twin neural networks. It divides the image into slices and sets weights for the contribution scores of small slices from the center to the edge of the image from high to low.

[0122] refer to Figure 5 ,In the twin neural network encoder (dashed box 501, dashed box 502), a LACB is set between the first three ResNetLayer layers respectively.

[0123] Skip, the skip connection unit, is used to retain more spatial information in the network structure. Figure 5 In the example, three Skips are set, namely, Skip1, Skip2, and Skip3, corresponding to three LACBs respectively.

[0124] ASPP, Atrous Spatial Pyramid Pooling, is an asymmetric convolution layer used in tasks such as semantic segmentation. It captures multi-scale information through convolution kernels of different sizes. ASPP is connected to the fourth LACB.

[0125] DecoderModel, model decoder, is used to decode the encoded features into output images in the model. Figure 5 It contains three layers of decoders, namely, DecoderModel1, DecoderModel2, and DecoderModel3.

[0126] Conv,3×3,3×3 convolution kernel, the convolution kernel of set size in convolutional neural network is used to extract local features.

[0127] UpSample, a sampling unit, is used to increase the data dimension and restore the low-resolution feature map to high resolution in the model. Figure 5 It contains two layers of sampling units, namely, UpSample1 and UpSample2.

[0128] Output image (defect), the output image with defects.

[0129] exist Figure 5 In the figure, the column pointed to by the dotted box 501 and the column pointed to by the dotted box 502 are twin neural network encoders with the same structure, which pass the obtained feature maps to the attention comparison module layer by layer. The attention comparison module calculates the similarity, compares the differences, and then jumps the result to the decoder. On the feature map of the bottom layer, the features are integrated through the hole space pyramid pooling module to obtain a larger receptive field, and finally sent to the decoder. The three modules of the dotted box 503 are model decoders, which have three layers, each of which contains a 3×3 convolution, three hole convolution layers with different expansion coefficients, and an upsampling module. The feature map input by jumping fuses the feature information of the bottom scale. Image recognition, classification, segmentation and generation can be achieved using twin neural networks.

[0130] The attention comparison module divides two feature maps of different sizes into slices of the same number. The smaller defect image slice queries the larger template image slice for similar features, and compares the slices for similarity using the method of high weight in the center and low weight around to find the area with the lowest similarity, which is considered to be a defect. Figure 6 As shown, X flaw , X temp They represent the defect image feature map and the comparison model feature map respectively. Input represents the output end, which inputs the original data; correspondingly, Output represents the output end, which outputs the result data.

[0131] After passing through the nn.Liner fully connected layer, three feature atlases q, k, and v are obtained respectively. nn.Liner is used to implement linear transformation in the twin neural network.

[0132] refer to Figure 6 , corresponding to the defect image feature map (X flaw ), the feature atlas obtained is q flaw , k flaw 、v flaw , corresponding to the contrast model feature map (X temp ), the feature atlas obtained is q temp , k temp 、v temp .

[0133] Furthermore, B is the number of slices, C is the number of channels after slicing, T1 and T2 are the different feature map sizes of the template image and the defect image after slicing, where T1 is greater than T2. Represents matrix multiplication.

[0134] q flaw With k tempThe result is then multiplied by the transpose of v temp The transpose of the matrix is ​​multiplied, and finally passed through the custom Attention module ( Figure 6 The custom Attention module is used to enhance the attention of the twin neural network to the important parts of the input data.

[0135] The cross entropy of binary classification is used as the loss function during training, which can be expressed by the following formula:

[0136]

[0137] In the formula, y is the calculated value of the loss function; N is the total number of samples; p i Y is the probability that the network predicts a positive sample; i Represents the label of the i-th sample, the positive class is 0 and the negative class is 1;

[0138] The network parameters θ are continuously optimized through training and learning to establish a direct mapping relationship between the defective image pixels in the sample data set and the corresponding template pixels. During model testing, the difference between the current image to be detected and the template image is calculated through the trained network parameters θ, thereby inferring the probability that the pixel position is a defective pixel.

[0139] Step A4, model deployment and post-processing: The parameters of the most efficient network model trained in step A3 are deployed on the on-site server. By inputting the image to be inspected transmitted on the production line and the template image stored in the template library, a tensor with a shape of [M, N, 2] is obtained after forward inference of the model. After a softmax normalization function, a binary image containing defects is obtained. The process can be expressed by the following formula:

[0140] The softmax normalization function is expressed using the following formula:

[0141] Y = softmax(z(w,h))

[0142] Where Y is a binary image; z is the tensor of the input image; w and h are the position indexes of the pixels in the image.

[0143] 2. Defect classification based on defect classification network model.

[0144] In actual application scenarios, the defect binary image of the defect segmentation result is used as a mask, and the pixel-level grayscale value is multiplied with the original detection image to obtain the defect foreground image, which is passed into the classification network to accurately classify the defect image so that the defect type is consistent with the defect name in the process, which can more accurately guide the previous process to make process adjustments and reduce the scrap rate.

[0145] The specific process is:

[0146] Step B1, labeling the defect type of the data collected in step A1.

[0147] Step B2: clean the data, delete the pictures with obvious wrong labels, and ensure that the number of sample data of each type is balanced. The labeling method is to save them in the form of image naming, with different numbers representing different defect types.

[0148] Step B3, build a classification network model and train it. When inputting into the network, use random scale transformation, HSV spatial jitter, random cropping and other enhancement processing to expand the data set. The following formula can be used to describe it:

[0149] G=g(x,ω)

[0150] In the formula, g represents the classification network model; x represents the input defect image; ω represents the model parameter; G represents an n-dimensional vector of the output, and n represents the number of defect categories;

[0151] The training process can be viewed as an optimization problem to reduce the gap between the predicted category and the actual category, which can be expressed by the following formula:

[0152] index=argmin(1+∑ i∈pos e -G[i] )

[0153] Among them, G[i] represents the element of vector G; i is the defect type serial number; index is the category number finally predicted by the network; pos represents the real defect type set of the input image;

[0154] The optimal parameters of the model are obtained by training the model until convergence.

[0155] Step B4, deploy the optimal model obtained through training to the production site server. The model inputs the defect image and outputs an n-dimensional vector. Each dimension of the n-dimensional vector represents a specific defect type, and n is the total number of defect types.

[0156] Specifically, the defect types include at least sticky paste, thread ends, dirty paste, rubbed paste, ink, perforation, holes, sticky paste seeping through the curtain, incomplete watermark, deformed watermark, white watermark hanging paste, interrupted security thread, floating line of fully buried thread, bending of fully buried thread, etc. In actual application scenarios, the defect types are not limited to these, and this is only for illustrative purposes.

[0157] 3. Calculate the risk score of each type of defect of the paper shaft based on the classification result and the segmentation defect binary map. If it exceeds the preset threshold, an alarm will be triggered. Summarize the risk score of the paper shaft defects and trigger an alarm if it exceeds the preset threshold. The factory determines the overall quality risk of the paper shaft based on the shaft risk value, determines whether the paper shaft needs manual sampling or full inspection, and can also guide the previous process to make process adjustments based on the alarm defect type.

[0158] The specific process is:

[0159] Step C1, according to the defect type of the classification result, the foreground area, the edge gradient of the foreground original image, the area ratio of the foreground and macro area, and the grayscale variance of the foreground and background are selected as indicators. According to the preset tolerance threshold, the severity score is calculated by linear fitting. If the score value exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification.

[0160] The severity score formula is:

[0161]

[0162] Where S is the severity score of the defect image; Thd_S is the severity threshold of one type of defect; Mes_S is the calculation result of the defect image index; when the severity score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification;

[0163] Step C2, using foreground area, foreground grayscale variance, and foreground-background contrast as indicators, and according to the preset tolerance threshold, linear fitting is performed to calculate the undetectability score. If the score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification.

[0164]

[0165] Where D is the undetectability score of the defect image; Thd_D is the undetectability threshold of one type of defect; Mes_D is the calculation result of the defect image index; when the undetectability score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification.

[0166] Step C3, based on the number of defects of this type, a threshold judgment alarm is performed, and if the tolerance threshold is exceeded, an entire axis alarm is performed.

[0167] Step C4, calculate the overall risk score of the paper axis according to the defect score of the entire axis. If it exceeds the tolerance threshold, an alarm is issued.

[0168] The calculation formula for the overall risk score of the entire axis is:

[0169]

[0170] Where Z is the overall risk score of the shaft; N' is the total number of large sheets produced; i is the defect type number; n i is the total number of defects of type i; S i is the mean severity score of the i-th defect; D i is the undetectability score of the i-th defect.

[0171] Based on the image quality risk score of the single axis and the entire paper axis, the production process problems of the production line are evaluated to find out the bottlenecks and problems in the production process. According to the evaluation results, the production line process is improved and optimized, such as optimizing the production process, improving equipment accuracy, strengthening quality control, etc., to improve the production efficiency and product quality of the production line.

[0172] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 7 A banknote defect classification and evaluation device based on a twin neural network according to an exemplary embodiment of the present invention is introduced.

[0173] The implementation of the banknote defect classification and evaluation device based on the twin neural network can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" or "unit" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0174] Based on the same inventive concept, the present invention also proposes a banknote defect classification and evaluation device based on a twin neural network, such as Figure 7 As shown, the device comprises:

[0175] The defect segmentation processing module 710 is used to segment defects from the detection image using a defect segmentation network model to obtain a defect binary image;

[0176] Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder;

[0177] The defect type recognition and classification module 720 is used to perform mask processing on the defect binary image, obtain a defect foreground image from the detection image, and input the defect foreground image into the defect classification network model to recognize and classify the defect type, so as to obtain defect type data of the detection image;

[0178] The quality risk analysis module 730 is used to perform image processing and defect assessment on the defect binary image, assess the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtain a quality risk score value of the inspection image;

[0179] The defect analysis and evaluation module 740 is used to detect banknote defects based on the quality risk score value and defect type data of the detection image, analyze and obtain the defect grade of the banknote, and evaluate and optimize the production process of the production line based on the defect type data and quality risk score value of the entire paper shaft.

[0180] In one embodiment, the defect segmentation processing module 710 uses a defect segmentation network model to segment defects from a detection image to obtain a defect binary image, including:

[0181] Collecting template images and defect images; wherein, the template image is produced, and the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is collected by a detection system, and defect position information is obtained during collection, and a corresponding template image is marked on the template image according to the defect position information;

[0182] Clean the collected template images corresponding to the defects, and delete the image groups with incorrect annotations or the image groups whose deviation between the template image and the defect image is greater than the deviation threshold;

[0183] The defect positions in the defect data are annotated at the pixel level to generate a binary label image, and the binary label image is placed in the folders of to-be-detected, template, and label, respectively, with one-to-one correspondence between the data;

[0184] Build and train the twin neural network, load the data in the folder, perform data enhancement processing on the data to expand the data set, and input the processed training data into the twin neural network for training;

[0185] Through training and learning to optimize network parameters, a mapping relationship between defective image pixels and corresponding template pixels in the sample data set is established; during model testing, the difference between the current image to be detected and the template image is calculated through the trained network parameters to determine the probability that the pixel position is a defective pixel;

[0186] Using the trained twin neural network, the image to be inspected transmitted on the production line and the template image stored in the template library are input. After the model forward calculation, the defect binary image containing the defect is obtained through the softmax normalization function.

[0187] In one embodiment, the defect segmentation processing module 710 constructs and trains a twin neural network, loads data in a folder, performs enhancement processing on the data to expand the data set, and inputs the processed training data into the twin neural network for training, including:

[0188] During the model training process, the cross entropy of the two categories is used as the loss function. The formula of the loss function is expressed as:

[0189]

[0190] In the formula, y is the calculated value of the loss function; N is the total number of samples; p i Y is the probability that the network predicts a positive sample; i Represents the label of the i-th sample, the positive class is 0 and the negative class is 1;

[0191] Using the trained twin neural network, the image to be inspected transmitted on the production line and the template image stored in the template library are input. After the model is forward calculated, the defect binary image containing the defect is obtained through the softmax normalization function, including:

[0192] The softmax normalization function is expressed using the following formula:

[0193] Y = softmax(z(w,h))

[0194] Where Y is a binary image; z is the tensor of the input image; w and h are the position indexes of the pixels in the image.

[0195] In one embodiment, the defect type recognition and classification module 720 inputs the defect foreground image into the defect classification network model to perform defect type recognition and classification to obtain defect type data of the detection image, including:

[0196] The defect foreground image is input into a defect classification network model to identify and classify the defect type, and the corresponding names of the banknote production process defects are obtained. According to different defect corresponding names, different defect assessment calculation methods are used to obtain defect assessment data, wherein the defect assessment data obtained using different defect assessment calculation methods at least includes: calculating the foreground area based on spot defects; calculating the number of foreground pixels and the edge gradient of the foreground original image based on missing defects; calculating the ratio of defect foreground to macro-area template based on deformation defects; and calculating the foreground and background grayscale variance values ​​based on uniformity defects.

[0197] In one embodiment, the apparatus further comprises: a model training module;

[0198] The model training module is used to:

[0199] Mark the defect type according to the corresponding template diagram of the collected defects;

[0200] Clean the labeled data and delete the images with incorrect labels. The labeling method is to save the images in the form of image naming, with different labels representing different defect types.

[0201] Construct and train a defect classification network model, and use enhanced processing to expand the data set; reduce the gap between the predicted category and the actual category through training, and obtain the optimal parameters of the defect classification network model by training the model until convergence;

[0202] Using the trained defect classification network model, the defect image is input and an n-dimensional vector is output. Each dimension of the n-dimensional vector represents a specific defect type, and n is the total number of defect types.

[0203] In one embodiment, the model training module constructs and trains a defect classification network model, and expands the data set by using an enhanced processing method; reduces the gap between the predicted category and the actual category through training, and obtains the optimal parameters of the defect classification network model by training the model until convergence, including:

[0204] The enhancement processing method of the expanded data set includes at least the use of random scale transformation, HSV spatial jitter, and random cropping when inputting into the network, which is described by the following formula:

[0205] G=g(x,ω)

[0206] In the formula, g represents the classification network model; x represents the input defect image; ω represents the model parameter; G represents an n-dimensional vector of the output, and n represents the number of defect categories;

[0207] Through training, the gap between the predicted category and the actual category is reduced, which can be described by the following formula:

[0208] index=argmin(1+∑ i∈pos e -G[i] )

[0209] Among them, G[i] represents the element of vector G; i is the defect type serial number; index is the category number finally predicted by the network; pos represents the real defect type set of the input image;

[0210] The optimal parameters of the defect classification network model are obtained by training the model until convergence.

[0211] In one embodiment, the quality risk analysis module 730 performs image processing and defect evaluation on the defect binary image, evaluates the overall damage degree of the detection image according to the morphological characteristics of the defects, and obtains the quality risk score value of the detection image, and the defect analysis and evaluation module 740 detects banknote defects according to the quality risk score value and defect type data of the detection image, analyzes and obtains the defect classification of the banknote, and evaluates and optimizes the production process of the production line according to the defect type data and quality risk score value of the entire paper shaft, including:

[0212] An alarm unit is used to calculate the risk score of each type of defect of the paper shaft according to the classification result and the segmentation defect binary map; when the score of one type of defect risk exceeds a preset first threshold, an alarm is issued;

[0213] According to the score of each type of defect risk of the paper shaft, the total score of the defect risk of the paper shaft is obtained by summarizing;

[0214] Based on the total score of the paper shaft defect risk, the overall quality risk of the paper shaft is judged; if the total score exceeds the preset second threshold, an alarm is issued, and when the total score reaches the manual inspection threshold, a manual random inspection or full inspection is initiated;

[0215] Based on the alarm of defect type, the process is guided and the technology is adjusted.

[0216] In one embodiment, the alarm unit calculates the risk score of each type of defect of the paper shaft according to the classification result and the segmentation defect binary map; and issues an alarm when the score of one type of defect risk exceeds a preset first threshold, including:

[0217] According to the defect type of the classification result, the foreground area, the edge gradient of the foreground original image, the ratio of the foreground and macro area, and the grayscale variance of the foreground and background are selected as indicators. According to the preset tolerance threshold, the severity score is calculated by linear fitting. The severity score formula is:

[0218]

[0219] Where S is the severity score of the defect image; Thd_S is the severity threshold of one type of defect; Mes_S is the calculation result of the defect image index; when the severity score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification;

[0220] Using foreground area, foreground grayscale variance, and foreground-background contrast as indicators, and according to the preset tolerance threshold, linear fitting is used to calculate the undetectability score. The undetectability score formula is:

[0221]

[0222] Where D is the undetectability score of the defect image; Thd_D is the undetectability threshold of one type of defect; Mes_D is the defect image index calculation result; when the undetectability score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification;

[0223] Based on the total score of the paper shaft defect risk, the overall quality risk of the paper shaft is judged; if the total score exceeds the preset second threshold, an alarm is issued, and when the total score reaches the manual inspection threshold, a manual random inspection or full inspection is initiated, including:

[0224] According to the defect score of the entire axis, the overall risk score of the axis is calculated; if it exceeds the tolerance threshold, an alarm is issued; the calculation formula for the overall risk score of the entire axis is:

[0225]

[0226] Where Z is the overall risk score of the shaft; N' is the total number of large sheets produced; i is the defect type number; n i is the total number of defects of type i; S i is the mean severity score of the i-th defect; D i is the undetectability score of the i-th defect.

[0227] It should be noted that, although several modules of the device for classifying and evaluating banknote defects based on twin neural networks are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment.

[0228] Based on the above invention concept, Figure 8 As shown, the present invention also proposes a computer device 800, including a memory 810, a processor 820, and a computer program 830 stored in the memory 810 and executable on the processor 820, wherein the processor 820 implements the aforementioned banknote defect classification and evaluation method based on a twin neural network when executing the computer program 830.

[0229] Based on the aforementioned inventive concept, the present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the aforementioned banknote defect classification and evaluation method based on a twin neural network.

[0230] Based on the aforementioned inventive concept, the present invention proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements a method for classifying and evaluating banknote defects based on a twin neural network.

[0231] The paper money defect classification and evaluation method and device based on twin neural network proposed in the present invention obtains a defect binary image by pixel-level segmentation of the original image, and performs single defect quality risk evaluation through morphological analysis; further obtains a defect foreground image based on the defect binary image, and classifies the defect foreground image; evaluates the quality risk and cause of the paper shaft according to the number of defects and quality risk score of the whole shaft, so as to evaluate the production process problems. Based on the evaluation results, technicians can quickly and efficiently locate paper money defects, improve and optimize the production line process, optimize the production process flow, improve equipment accuracy, strengthen quality control, etc., so as to improve the production efficiency and product quality of the production line, realize the improvement and optimization of the production line process, and provide strong technical support for analyzing paper money defects and evaluating the improved production line process.

[0232] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of laws and regulations.

[0233] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] The present invention is described with reference to flowcharts and / or block diagrams of methods and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0235] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0236] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0237] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A banknote defect classification and evaluation method based on twin neural network, characterized in that: The method includes: The defect segmentation network model is used to segment defects from the inspection image to obtain a defect binary image; Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder; Masking the defect binary image to obtain a defect foreground image from the detection image, and inputting the defect foreground image into the defect classification network model to identify and classify the defect type to obtain defect type data of the detection image; Performing image processing and defect assessment on the defect binary image, assessing the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtaining a quality risk score value of the inspection image; Based on the quality risk score value and defect type data of the detection image, banknote defects are detected, and the defect classification of the banknote is obtained by analysis. Based on the defect type data and quality risk score value of the entire paper shaft, the production process of the production line is evaluated and optimized.

2. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 1 is characterized in that: The defect segmentation network model is used to segment defects from the inspection image to obtain a defect binary map, including: Collect template images and defect images; Clean the collected template images corresponding to the defects, and delete the image groups with incorrect annotations or the image groups whose deviation between the template image and the defect image is greater than the deviation threshold; The defect positions in the defect data are annotated at the pixel level to generate a binary label image, and the binary label image is placed in the folders of to-be-detected, template, and label, respectively, with one-to-one correspondence between the data; Build and train the twin neural network, load the data in the folder, enhance the data to expand the data set, and input the processed training data into the twin neural network for training; Through training and learning to optimize network parameters, a mapping relationship between defective image pixels and corresponding template pixels in the sample data set is established; during model testing, the difference between the current image to be detected and the template image is calculated through the trained network parameters to determine the probability that the pixel position is a defective pixel; Using the trained twin neural network, the image to be inspected transmitted on the production line and the template image stored in the template library are input. After the model forward calculation, the defect binary image containing the defect is obtained through the softmax normalization function.

3. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 2 is characterized in that: Build and train the twin neural network, load the data in the folder, enhance the data to expand the data set, and input the processed training data into the twin neural network for training, including: During the model training process, the cross entropy of the two categories is used as the loss function. The formula of the loss function is expressed as: In the formula, y is the calculated value of the loss function; N is the total number of samples; p i Y is the probability that the network predicts a positive sample; i Represents the label of the i-th sample, the positive class is 0 and the negative class is 1; Using the trained twin neural network, the image to be inspected transmitted on the production line and the template image stored in the template library are input. After the model is forward calculated, the defect binary image containing the defect is obtained through the softmax normalization function, including: The softmax normalization function is expressed using the following formula: Y = softmax(z(w,h)) Where Y is a binary image; z is the tensor of the input image; w and h are the position indexes of the pixels in the image.

4. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 2 is characterized in that: Inputting the defect foreground image into the defect classification network model to identify and classify the defect type, and obtaining the defect type data of the detection image, including: The defect foreground image is input into a defect classification network model to identify and classify the defect type, and the corresponding names of the banknote production process defects are obtained. According to different defect corresponding names, different defect assessment calculation methods are used to obtain defect assessment data, wherein the defect assessment data obtained using different defect assessment calculation methods at least includes: calculating the foreground area based on spot defects; calculating the number of foreground pixels and the edge gradient of the foreground original image based on missing defects; calculating the ratio of defect foreground to macro-area template based on deformation defects; and calculating the foreground and background grayscale variance values ​​based on uniformity defects.

5. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 4 is characterized in that: The method includes: Mark the defect type according to the corresponding template diagram of the collected defects; Clean the labeled data and delete the images with incorrect labels. The labeling method is to save the images in the form of image naming, with different labels representing different defect types. Construct and train a defect classification network model, and use enhanced processing to expand the data set; reduce the gap between the predicted category and the actual category through training, and obtain the optimal parameters of the defect classification network model by training the model until convergence; Using the trained defect classification network model, the defect image is input and an n-dimensional vector is output. Each dimension of the n-dimensional vector represents a specific defect type, and n is the total number of defect types.

6. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 5 is characterized in that: Construct and train a defect classification network model, and use enhanced processing to expand the data set; reduce the gap between the predicted category and the actual category through training, and obtain the optimal parameters of the defect classification network model by training the model until convergence, including: The enhancement processing method of the expanded data set includes at least the use of random scale transformation, HSV spatial jitter, and random cropping when inputting into the network, which is described by the following formula: G=g(x,ω) In the formula, g represents the classification network model; x represents the input defect image; ω represents the model parameter; G represents an n-dimensional vector of the output, and n represents the number of defect categories; Through training, the gap between the predicted category and the actual category is reduced, which can be described by the following formula: index=argmin(1+∑ i∈pos e -G[i] ) Among them, G[i] represents the element of vector G; i is the defect type serial number; index is the category number finally predicted by the network; pos represents the real defect type set of the input image; The optimal parameters of the defect classification network model are obtained by training the model until convergence.

7. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 1, characterized in that: The defect binary image is subjected to image processing and defect evaluation, and the overall damage degree of the detection image is evaluated according to the morphological characteristics of the defects to obtain the quality risk score value of the detection image, and the defects of the banknote paper are detected according to the quality risk score value and defect type data of the detection image, and the defect classification of the banknote paper is obtained by analysis, and the production process of the production line is evaluated and optimized according to the defect type data and quality risk score value of the entire paper shaft, including: According to the classification results and the segmentation defect binary map, the risk score of each type of defect of the paper shaft is calculated; when the score of one type of defect risk exceeds the preset first threshold, an alarm is issued; According to the score of each type of defect risk of the paper shaft, the total score of the defect risk of the paper shaft is obtained by summarizing; Based on the total score of the paper shaft defect risk, the overall quality risk of the paper shaft is judged; if the total score exceeds the preset second threshold, an alarm is issued, and when the total score reaches the manual inspection threshold, a manual random inspection or full inspection is initiated; Based on the alarm of defect type, the process is guided and the technology is adjusted.

8. The method for classifying and evaluating banknote defects based on twin neural networks according to claim 7 is characterized in that: According to the classification results and the segmentation defect binary map, the risk score of each type of defect of the paper shaft is calculated; When the score of one type of defect risk exceeds the preset first threshold, an alarm is issued, including: According to the defect type of the classification result, the foreground area, the edge gradient of the foreground original image, the ratio of the foreground and macro area, and the grayscale variance of the foreground and background are selected as indicators. According to the preset tolerance threshold, the severity score is calculated by linear fitting. The severity score formula is: Where S is the severity score of the defect image; Thd_S is the severity threshold of one type of defect; Mes_S is the calculation result of the defect image index; when the severity score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification; Using foreground area, foreground grayscale variance, and foreground-background contrast as indicators, and according to the preset tolerance threshold, linear fitting is used to calculate the undetectability score. The undetectability score formula is: Where D is the undetectability score of the defect image; Thd_D is the undetectability threshold of one type of defect; Mes_D is the defect image index calculation result; when the undetectability score exceeds the alarm threshold, a single alarm is issued and displayed on the main interface of the program for manual verification; Based on the total score of the paper shaft defect risk, the overall quality risk of the paper shaft is judged; if the total score exceeds the preset second threshold, an alarm is issued, and when the total score reaches the manual inspection threshold, a manual random inspection or full inspection is initiated, including: According to the defect score of the entire axis, the overall risk score of the axis is calculated; if it exceeds the tolerance threshold, an alarm is issued; the calculation formula for the overall risk score of the entire axis is: Where Z is the overall risk score of the shaft; N' is the total number of large sheets produced; i is the defect type number; n i is the total number of defects of type i; S i is the mean severity score of the i-th defect; D i is the undetectability score of the i-th defect.

9. A banknote defect classification and evaluation device based on twin neural network, characterized in that: The device includes: A defect segmentation processing module is used to segment defects from the detection image using a defect segmentation network model to obtain a defect binary image; Among them, the defect segmentation network model is obtained by analyzing the mapping relationship between the defect image and the template image and training the model; the template image is obtained by superimposing and averaging multiple defect-free banknote images; the defect image is obtained by collecting defect position information through the detection system and marking it on the template image according to the defect position information; the defect segmentation network model is provided with an attention comparison module, which is used for input feature comparison between twin neural networks, divides the image into slices, and sets weights from high to low for the contribution scores of small slices from the center of the image to the edge of the image; the resolution size of the template image is larger than that of the defect image, and the defect segmentation network model adds a hole convolution layer in the last layer of the encoder; A defect type recognition and classification module is used to perform mask processing on the defect binary image, obtain a defect foreground image from the detection image, and input the defect foreground image into the defect classification network model to recognize and classify the defect type, so as to obtain defect type data of the detection image; A quality risk analysis module is used to perform image processing and defect assessment on the defect binary image, assess the overall damage degree of the inspection image according to the morphological characteristics of the defects, and obtain a quality risk score value of the inspection image; The defect analysis and evaluation module is used to detect banknote defects based on the quality risk score value and defect type data of the detection image, analyze and obtain the defect grade of the banknote paper, and evaluate and optimize the production process of the production line based on the defect type data and quality risk score value of the entire paper shaft.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images