A pharmaceutical packaging defect detection method and system based on artificial intelligence

Abstract

The present invention provides a method and system for detecting defects in pharmaceutical packaging based on artificial intelligence. The method extracts multiple ROI regions in the image through a mask and identifies the defect type of each ROI region. Each image block in the pre-processed image block sequence, the ROI ratio of each image block, and the number of each image block are encoded to obtain image block encoding, weight encoding, and position encoding. The second fusion result is used as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the encoder. The first fusion result is used as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the encoder. The image block encoding is used as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the encoder, and the recognition result of the DETR model is output. The present invention can specifically identify defects in blisters and substrates.

Description

Technical Field

[0001] The present invention relates to the field of pharmaceutical packaging, and in particular to a pharmaceutical packaging defect detection method and system based on artificial intelligence. Background Art

[0002] The integrity of pharmaceutical packaging is crucial for pharmaceuticals, directly impacting their environmental stability and protection within the packaging. As medical products, their quality and effectiveness are crucial to patient health and treatment outcomes. Packaging integrity effectively prevents the intrusion of moisture, oxygen, and other harmful substances, thereby reducing the likelihood of moisture, oxidation, and deterioration within the packaging. Maintaining drug stability extends their shelf life and ensures they maintain their efficacy and effectiveness. Packaging integrity helps protect pharmaceuticals from external environmental contamination, preventing dust, bacteria, viruses, and other contaminants from entering the packaging, thereby maintaining drug purity. This is particularly true in specialized environments such as hospitals and pharmacies.

[0003] Currently, some companies still rely on manual visual inspection on their production lines, which is not only inefficient but also prone to missed and false detections. Defect detection technology can help improve inspection speed. Commonly used defect detection technologies include machine vision, image processing, and deep learning. While these technologies can detect defects in pharmaceutical packaging to a certain extent, their effectiveness in detecting minor or hidden defects still needs to be improved. Detection is particularly challenging for aluminum-plastic blisters with darker colors or highly reflective surfaces, and the accuracy and robustness of these technologies need to be improved. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method for drug packaging defect detection based on artificial intelligence, which includes the following steps:

[0005] Taking a photo of the drug packaging and preprocessing the photo to obtain an image of a preset size that only includes the drug packaging, obtaining a drug packaging template input by a user, determining a mask corresponding to the drug packaging template, extracting multiple ROI regions in the image using the mask, and identifying the defect type of each ROI region;

[0006] The image is divided into a plurality of image blocks to obtain an image block sequence, the image blocks are numbered according to the order of the image blocks, and a proportion r1 of the ROI region in each image block in the image block is calculated; the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence;

[0007] Each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence are encoded to obtain image block code, weight code and position code, the weight code and the image block code are fused to obtain a first fusion result, the position code and the image block code are fused to obtain a second fusion result, the second fusion result is used as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the Encoder, the first fusion result is used as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the Encoder, and the image block code is used as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the Encoder; and the recognition result of the DETR model is output.

[0008] Preferably, the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically:

[0009] If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

[0010] Preferably, the dilation operation is performed on the ROI region in the image so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI region, specifically:

[0011] Divide the image into a plurality of rectangular regions, each rectangular region having a ROI region, and identify the edge between the ROI region and the non-ROI region in each rectangular region;

[0012] Obtaining a maximum value of the proportion in the rectangular area, and determining a convolution kernel size for the dilation operation according to the maximum value;

[0013] The convolution kernel is used to perform a dilation operation on the edge so that the proportion of all the image blocks in the rectangular area after the dilation reaches 1.

[0014] Preferably, the fusing of the weight coding and the image block coding to obtain a first fusion result is specifically:

[0015] The image block is encoded to obtain a vector of length M, the proportion corresponding to the image block is encoded to obtain a weighted coding vector of length N, and the vector of length M and the weighted coding vector of length N are vector-concatenated to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number.

[0016] Preferably, the fusing of the position code and the image block code to obtain the second fusion result is specifically:

[0017] Determine the row number and column number of the image block according to the position, encode the row number and column number into vectors of length M / 2, concatenate the two vectors of length M / 2 to obtain a position vector of length M to obtain a position code;

[0018] Adding the image block code and the position code bit by bit to obtain a first vector;

[0019] Determine the center point of each ROI area, calculate the distance between the image block and the nearest center point, encode the distance into a vector of length N, and concatenate the vector of length M and the first vector to obtain a second fusion result of length M+N.

[0020] Preferably, h1 is determined as follows:

[0021] Obtain a ratio of the number of image blocks whose proportion is not 0 in the preprocessed image block sequence to the number of image blocks in the image block sequence, and determine h1 based on the ratio and the number of heads of the first layer multi-head attention mechanism in the encoder.

[0022] In addition, the present invention also provides a drug packaging defect detection system based on artificial intelligence, which includes the following modules:

[0023] An ROI region recognition module is configured to photograph the drug packaging, preprocess the photographed image to obtain an image of a preset size that only includes the drug packaging, obtain a drug packaging template input by a user, determine a mask corresponding to the drug packaging template, extract multiple ROI regions in the image using the mask, and identify the defect type in each ROI region;

[0024] a preprocessing module, configured to divide the image into a plurality of image blocks to obtain an image block sequence, number the image blocks in order, and calculate a proportion r1 of the ROI region in each image block in the image block; and preprocess the image block sequence according to the proportion to obtain a preprocessed image block sequence;

[0025] The second recognition module is used to encode each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence to obtain image block code, weight code and position code, fuse the weight code and the image block code to obtain a first fusion result, fuse the position code and the image block code to obtain a second fusion result, use the second fusion result as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the Encoder, use the first fusion result as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the Encoder, and use the image block code as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the Encoder; and output the recognition result of the DETR model.

[0026] Preferably, the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically:

[0027] If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

[0028] Preferably, the dilation operation is performed on the ROI region in the image so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI region, specifically:

[0029] Divide the image into a plurality of rectangular regions, each rectangular region having a ROI region, and identify the edge between the ROI region and the non-ROI region in each rectangular region;

[0030] Obtaining a maximum value of the proportion in the rectangular area, and determining a convolution kernel size for the dilation operation according to the maximum value;

[0031] The convolution kernel is used to perform a dilation operation on the edge so that the proportion of all the image blocks in the rectangular area after the dilation reaches 1.

[0032] Preferably, the fusing of the weight coding and the image block coding to obtain a first fusion result is specifically:

[0033] The image block is encoded to obtain a vector of length M, the proportion corresponding to the image block is encoded to obtain a weighted coding vector of length N, and the vector of length M and the weighted coding vector of length N are vector-concatenated to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number.

[0034] Preferably, the fusing of the position code and the image block code to obtain the second fusion result is specifically:

[0035] Determine the row number and column number of the image block according to the position, encode the row number and column number into vectors of length M / 2, concatenate the two vectors of length M / 2 to obtain a position vector of length M to obtain a position code;

[0036] Adding the image block code and the position code bit by bit to obtain a first vector;

[0037] Determine the center point of each ROI area, calculate the distance between the image block and the nearest center point, encode the distance into a vector of length N, and concatenate the vector of length M and the first vector to obtain a second fusion result of length M+N.

[0038] Preferably, h1 is determined as follows:

[0039] Obtain a ratio of the number of image blocks whose proportion is not 0 in the preprocessed image block sequence to the number of image blocks in the image block sequence, and determine h1 based on the ratio and the number of heads of the first layer multi-head attention mechanism in the encoder.

[0040] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0041] To address the difficulty in detecting minor defects in pharmaceutical packaging, particularly aluminum-plastic blister packaging, the present invention provides an artificial intelligence-based pharmaceutical packaging defect detection method and system capable of identifying minor defects in aluminum-plastic packaging. This method separates the blister and substrate portions of the aluminum-plastic packaging for detection. Since the blister portion may contain minor defects such as black spots, these defects can also be identified. Furthermore, the present invention separates the blister portion of the substrate to identify defects within the substrate. Incorporating the ROI percentage of the image block into the DETR input provides more targeted detection and significantly improves recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a flow chart of Example 1;

[0044] Figure 2 These are pictures of different aluminum-plastic blister medicine packaging;

[0045] Figure 3 Extract schematic diagram for ROI;

[0046] Figure 4 This is a segmentation diagram of the drug packaging;

[0047] Figure 5 Schematic diagram of the first layer input of DETR Encoder. DETAILED DESCRIPTION

[0048] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] Example 1

[0051] The present invention provides a drug packaging defect detection method based on artificial intelligence, such as Figure 1 As shown, the method includes the following steps:

[0052] S1, taking a photo of the drug package, and preprocessing the photographed image to obtain an image of a preset size that only includes the drug package, obtaining a drug package template input by a user, determining a mask corresponding to the drug package template, extracting multiple ROI regions in the image using the mask, and identifying the defect type of each ROI region;

[0053] There are many ways to package medicines, with aluminum-plastic blisters being the most common. These protect medications from moisture and other contaminants. However, due to the influence of various production line links and the production line environment, aluminum-plastic blister packaging can develop defects such as missing pills, black spots, broken pills, and deformation.

[0054] The camera installed on the assembly line captures the aluminum-plastic packaging after it is filled with medicine. Since the camera's shooting range is larger than the medicine packaging, it is necessary to identify the medicine packaging and obtain an image containing only the medicine packaging. There are many ways to pre-process the photographed image to obtain an image of a preset size that only includes the medicine packaging, such as line detection, template matching, etc., which are not specifically limited in the present invention.

[0055] Since the aluminum-plastic packaging method is not fixed, Figure 2 Two common aluminum-plastic blister packaging methods are shown. The present invention provides templates for various aluminum-plastic packaging. A corresponding mask is determined based on the template. A mask, also known as a film, is like a layer covering an image, filtering out unwanted areas and displaying the desired areas. The mask can be used to extract the ROI (Region of Interest). In one specific embodiment, the ROI is the blister portion of the aluminum-plastic blister. The aluminum-plastic blister package includes a blister portion and a substrate portion. The blister portion is the independent space where the drug is placed, and the substrate is the bottom portion of the aluminum-plastic blister package.

[0056] The ROI area is the focus of identification, and black spots or damaged pills are easily misjudged. In a preferred embodiment, the ViT model is used to identify each ROI area separately. After the ROI area is extracted, the remaining part is the substrate part of the aluminum-plastic blister, such as Figure 3 shown.

[0057] S2, dividing the image into a plurality of image blocks to obtain an image block sequence, numbering the image blocks in order, and calculating a proportion r1 of the ROI region in each image block in the image block; preprocessing the image block sequence according to the proportion to obtain a preprocessed image block sequence;

[0058] The DETR (DEtection TRanformer) model is an end-to-end target detection network, which mainly includes a backbone network composed of a convolutional neural network, an encoder, a decoder, and a prediction head. DETR is based on the target recognition structure of the Transformer. In a preferred embodiment, the image is divided into multiple image blocks, such as Figure 4As shown, the image blocks are numbered in order. Preferably, the numbering adopts the form of row number + column number. For example, the first image block is numbered 11, and the image block in the first row and second column is numbered 12.

[0059] The proportion of the ROI region in each image block is then calculated, and the ROI is preprocessed based on the proportion. The preprocessing primarily removes the ROI region from the image block while retaining the background portion. In one specific embodiment, the preprocessing involves replacing the ROI region in the image block with other pixel values ​​that are correlated with the pixel values ​​of the non-ROI region in the image block.

[0060] S3. Encode each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence to obtain image block code, weight code, and position code. Fuse the weight code and the image block code to obtain a first fusion result. Fuse the position code and the image block code to obtain a second fusion result. Use the second fusion result as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the Encoder. Use the first fusion result as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the Encoder. Use the image block code as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the Encoder. Output the recognition result of the DETR model.

[0061] The pre-processed image block sequence includes a plurality of image blocks, and image block encoding, encoding of the proportion, and encoding of the position of the image block are performed on each image block.

[0062] In one specific embodiment, encoding the image block means using convolution to convert the image block into a preset size, and then flattening the 2D image into a one-dimensional vector; in another embodiment, the image block is first flattened into a one-dimensional vector, and then the one-dimensional vector is converted into a vector of a preset length through a fully connected layer.

[0063] In another specific embodiment, encoding the proportion is to embed the proportion to obtain a one-dimensional vector; encoding the position of the image block can adopt the default position encoding method in DETR or the encoding method described below.

[0064] After the above three encodings, each image block corresponds to three vectors, namely: image block feature vector (image block encoding), weight vector (weight encoding), and position vector (position encoding).

[0065] The image block feature vector represents the feature information of the image block, the weight vector represents the proportion of the background in the image block, and the position vector represents the position of the image block in the image.

[0066] The weight coding and the image block coding are fused to obtain a first fusion result, and the position coding and the image block coding are fused to obtain a second fusion result.

[0067] DETR is based on Transformer, and the foundation of Transformer is the attention mechanism. The attention mechanism involves Q (Query), K (Key), and V (Value), where the Q vector is the query vector, K is the correlation vector between the queried information and other information, and V is the vector of the queried information.

[0068] Q=x*W q ;

[0069] K=x*W k ;

[0070] V=x*W v ;

[0071] Where x is the input, W q 、W k 、W v is a parameter matrix that can be learned during training;

[0072] The calculation formula of the attention mechanism is

[0073] The Q matrix or Q vector corresponds to an input and a W q Parameter matrix, similarly, the K matrix or K vector corresponds to an input and a k parameter matrix, and the V matrix or V vector corresponds to an input and a W v Parameter matrix. In DETR, the result of adding image features and spatial positional encoding is used as the input of K and Q, and image features are used as the input of V.

[0074] However, in drug packaging defect detection, after the ROI area is extracted, the substrate part is incomplete. The present invention fuses the weight information into the QKV input of the multi-head self-attention mechanism. Specifically, the weight code and the image block code are fused to obtain a first fusion result, and the position code and the image block code are fused to obtain a second fusion result. The second fusion result is used as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the encoder. The first fusion result is used as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the encoder. The image block code is used as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the encoder.

[0075] The Encoder refers to the Encoder in DETR, and the Encoder in DETR includes a multi-layer structure, and the structure of each layer is the same. The first fusion result and the second fusion result of the present invention are only input into the first layer multi-head self-attention mechanism in the Encoder, such as Figure 5 As shown, the other layers of the Encoder still use the DETR default input method. The input of the first layer of the multi-head attention mechanism is:

[0076] Q=x2*W q ;

[0077] K=x1*W k ;

[0078] V=x2*W v ;

[0079] Among them, x1 is the first fusion result, and x2 is the second fusion result.

[0080] The multi-head self-attention mechanism includes multiple self-attentions. To avoid excessive redundancy in the proportion information, the second fusion result in the present invention is used as the input of the K matrix corresponding to h1 attentions. The other attention mechanisms still use image block encoding. If the vector lengths of the first fusion result and the image block encoding are different, the length of the image block encoding is padded to the same length as the first fusion result, which facilitates calculation. Alternatively, 0 is encoded using the proportion encoding method, and then linked to the end of the image block encoding vector to obtain an image block encoding with the same length as the first fusion result.

[0081] After extracting the ROI area, the area of ​​the ROI in the image is blank. If the image blocks in the blank part are still input into the DETR model, not only the amount of calculation is increased, but also since this part is invalid information, it may affect the result. Based on this, in a specific embodiment, the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically:

[0082] If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

[0083] If an image block is completely within the ROI region, the image block is directly deleted from the image block sequence. If an image block does not contain the ROI region at all, it is not processed and is retained. If an image block contains part of the ROI region, a dilation operation in image morphology is used to expand the non-ROI region of the image block into the ROI region until the image block no longer contains the ROI region. The image block that does not contain or does not include the ROI region means that the image does not include the ROI region. Although the position of the image block remains unchanged, the dilation operation expands the image range of the non-ROI region.

[0084] The aluminum-plastic blister medicine package contains several pills, and the spaces between the blisters are relatively obvious. In a specific embodiment, the expansion operation is performed on the ROI area in the image so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area, specifically:

[0085] Divide the image into a plurality of rectangular regions, each rectangular region having a ROI region, and identify the edge between the ROI region and the non-ROI region in each rectangular region;

[0086] The image is divided into a plurality of rectangular regions, each of which has an ROI region. In this way, each rectangular region can be operated separately. There are many ways to divide the rectangular region, for example, find a segmentation method corresponding to the aluminum-plastic blister package. In another specific embodiment, the image is divided into a plurality of rectangular regions, each of which has an ROI region. This means that the ROI region in the rectangular region is removed to obtain the following: Figure 3 The substrate image in , and then segmented.

[0087] Obtaining a maximum value of the proportion in the rectangular area, and determining a convolution kernel size for the dilation operation according to the maximum value;

[0088] The rectangular region includes multiple image blocks, each with a different proportion. To include only the background image in the image block, the image block with the largest proportion after dilation must not contain the ROI region image, which also determines the size of the convolution kernel. One embodiment of determining the convolution kernel size for the dilation operation based on the maximum value is to establish a corresponding relationship between the maximum value and the convolution kernel size for the dilation operation.

[0089] The convolution kernel is used to perform a dilation operation on the edge so that the proportion of all the image blocks in the rectangular area after the dilation reaches 1.

[0090] The image blocks in the expanded rectangular area that originally accounted for less than 1 no longer contain the ROI image.

[0091] The weight coding reflects the size of the proportion in the image block, which determines the importance of this image block in identifying substrate defects. In a specific embodiment, the weight coding and the image block coding are fused to obtain a first fusion result, which is specifically:

[0092] The image block is encoded to obtain a vector of length M, the proportion corresponding to the image block is encoded to obtain a weighted coding vector of length N, and the vector of length M and the weighted coding vector of length N are vector-concatenated to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number.

[0093] For example, if the image block size is 10X10, it is converted into a vector of length 100 after encoding. The original proportion corresponding to the image block is 0.2. After encoding 0.2, it is converted into a vector of 1100, whose length is 4. Then the two are concatenated to obtain the first fusion result of length 100+4.

[0094] DETR needs to obtain global information of the substrate. Position embedding contains position information, and the distance from the ROI region contains information about the importance of the image block. The closer the image block is to the ROI region, the more important it is because the drug particles are closer to each other. In a specific embodiment, the position encoding and the image block encoding are fused to obtain a second fusion result, specifically:

[0095] Determine the row number and column number of the image block according to the position, encode the row number and column number into vectors of length M / 2, concatenate the two vectors of length M / 2 to obtain a position vector of length M to obtain a position code;

[0096] Adding the image block code and the position code bit by bit to obtain a first vector;

[0097] For example, row number 3 and column number 5 are encoded to obtain a vector with a length of 50, and column number 5 is encoded to obtain a vector with a length of 50. The sum of the two lengths is 100. It is added bit by bit with the image block code to obtain the first vector, and the length of the first vector is also 100.

[0098] Determine the center point of each ROI area, calculate the distance between the image block and the nearest center point, encode the distance into a vector of length N, and concatenate the vector of length M and the first vector to obtain a second fusion result of length M+N.

[0099] In order to keep the lengths of the Q, K, and V inputs in the attention mechanism the same, in a specific embodiment, the distance information between the image block and the ROI area is fused into the input vectors of K and Q. Specifically, the distance between the image block and the center point of the nearest ROI area is determined, and then the distance is encoded to obtain a vector of length N. The first vector and the vector of length N obtained by encoding the distance are then concatenated to obtain the second fusion result.

[0100] The more image blocks whose proportions are not 0 in the preprocessed image block sequence, the more important the proportions, i.e., the weight information, are. An extreme example is that the proportions of all image blocks are not 0. In this case, each attention mechanism obtains the proportion information, so that the global information of the image block can be obtained. The multi-head attention mechanism consists of multiple self-attentions. In a specific embodiment, the h1 is determined as follows:

[0101] Obtain a ratio of the number of image blocks whose proportion is not 0 in the preprocessed image block sequence to the number of image blocks in the image block sequence, and determine h1 based on the ratio and the number of heads of the first layer multi-head attention mechanism in the encoder.

[0102] After determining h1, the other self-attention mechanisms of the multi-head attention mechanism are used as h2. For example, the number of heads of the multi-head self-attention mechanism is 5. If h1=2, then h2=3.

[0103] Example 2

[0104] The present invention also provides a drug packaging defect detection system based on artificial intelligence, which includes the following modules:

[0105] An ROI region recognition module is configured to photograph the drug packaging, preprocess the photographed image to obtain an image of a preset size that only includes the drug packaging, obtain a drug packaging template input by a user, determine a mask corresponding to the drug packaging template, extract multiple ROI regions in the image using the mask, and identify the defect type in each ROI region;

[0106] a preprocessing module, configured to divide the image into a plurality of image blocks to obtain an image block sequence, number the image blocks in order, and calculate a proportion r1 of the ROI region in each image block in the image block; and preprocess the image block sequence according to the proportion to obtain a preprocessed image block sequence;

[0107] The second recognition module is used to encode each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence to obtain image block code, weight code and position code, fuse the weight code and the image block code to obtain a first fusion result, fuse the position code and the image block code to obtain a second fusion result, use the second fusion result as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the Encoder, use the first fusion result as the input of the K matrix corresponding to the h1 attention heads in the multi-head attention mechanism in the first layer of the Encoder, and use the image block code as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the Encoder; and output the recognition result of the DETR model.

[0108] Preferably, the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically:

[0109] If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

[0110] Preferably, the dilation operation is performed on the ROI region in the image so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI region, specifically:

[0111] Divide the image into a plurality of rectangular regions, each rectangular region having a ROI region, and identify the edge between the ROI region and the non-ROI region in each rectangular region;

[0112] Obtaining a maximum value of the proportion in the rectangular area, and determining a convolution kernel size for the dilation operation according to the maximum value;

[0113] The convolution kernel is used to perform a dilation operation on the edge so that the proportion of all the image blocks in the rectangular area after the dilation reaches 1.

[0114] Preferably, the fusing of the weight coding and the image block coding to obtain a first fusion result is specifically:

[0115] The image block is encoded to obtain a vector of length M, the proportion corresponding to the image block is encoded to obtain a weighted coding vector of length N, and the vector of length M and the weighted coding vector of length N are vector-concatenated to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number.

[0116] Preferably, the fusing of the position code and the image block code to obtain the second fusion result is specifically:

[0117] Determine the row number and column number of the image block according to the position, encode the row number and column number into vectors of length M / 2, concatenate the two vectors of length M / 2 to obtain a position vector of length M to obtain a position code;

[0118] Adding the image block code and the position code bit by bit to obtain a first vector;

[0119] Determine the center point of each ROI area, calculate the distance between the image block and the nearest center point, encode the distance into a vector of length N, and concatenate the vector of length M and the first vector to obtain a second fusion result of length M+N.

[0120] Preferably, h1 is determined as follows:

[0121] Obtain a ratio of the number of image blocks whose proportion is not 0 in the preprocessed image block sequence to the number of image blocks in the image block sequence, and determine h1 based on the ratio and the number of heads of the first layer multi-head attention mechanism in the encoder.

[0122] Example 3

[0123] The present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the method described in the first embodiment is implemented.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A drug packaging defect detection method based on artificial intelligence, characterized in that: The method comprises the following steps: Taking a photo of the drug packaging and preprocessing the photo to obtain an image of a preset size that only includes the drug packaging, obtaining a drug packaging template input by a user, determining a mask corresponding to the drug packaging template, extracting multiple ROI regions in the image using the mask, and identifying the defect type of each ROI region; The image is divided into a plurality of image blocks to obtain an image block sequence, the image blocks are numbered according to the order of the image blocks, and a proportion r1 of the ROI region in each image block in the image block is calculated; the image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence; Encode each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence to obtain an image block code, a weight code, and a position code, fuse the weight code and the image block code to obtain a first fusion result, fuse the position code and the image block code to obtain a second fusion result, use the second fusion result as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the encoder, use the first fusion result as the input of the K matrix corresponding to h1 attention heads in the multi-head attention mechanism in the first layer of the encoder, and use the image block code as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the encoder; output the recognition result of the DETR model; The fusing of the weight coding and the image block coding to obtain a first fusion result is specifically as follows: encoding the image block to obtain a vector of length M, encoding the proportion corresponding to the image block to obtain a weight coding vector of length N, and performing vector concatenation of the vector of length M and the weight coding vector of length N to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number; The fusion of the position code and the image block code to obtain the second fusion result is specifically as follows: determining the row number and column number of the image block according to the position code, encoding the row number and column number into vectors of length M / 2 respectively, and performing vector splicing on the two vectors of length M / 2 to obtain a position vector of length M to obtain the position code; adding the image block code and the position code bit by bit to obtain the first vector; determining the center point of each ROI area, calculating the distance between the image block and the nearest center point, encoding the distance into a vector of length N, and performing vector splicing on the vector of length M and the first vector to obtain a second fusion result of length M+N.

2. The method according to claim 1, wherein The image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically: If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

3. The method according to claim 2, wherein The dilation operation is performed on the ROI region in the image so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI region, specifically: Divide the image into a plurality of rectangular regions, each rectangular region having a ROI region, and identify the edge between the ROI region and the non-ROI region in each rectangular region; Obtaining a maximum value of the proportion in the rectangular area, and determining a convolution kernel size for the dilation operation according to the maximum value; The convolution kernel is used to perform a dilation operation on the edge so that the proportion of all the image blocks in the rectangular area after the dilation reaches 1.

4. The method according to claim 1, wherein The h1 is determined as follows: Obtain a ratio of the number of image blocks whose proportion is not 0 in the preprocessed image block sequence to the number of image blocks in the image block sequence, and determine h1 based on the ratio and the number of heads of the first layer multi-head attention mechanism in the encoder.

5. A drug packaging defect detection system based on artificial intelligence, characterized in that: The system includes the following modules: An ROI region recognition module is configured to photograph the drug packaging, preprocess the photographed image to obtain an image of a preset size that only includes the drug packaging, obtain a drug packaging template input by a user, determine a mask corresponding to the drug packaging template, extract multiple ROI regions in the image using the mask, and identify the defect type in each ROI region; a preprocessing module, configured to divide the image into a plurality of image blocks to obtain an image block sequence, number the image blocks in order, and calculate a proportion r1 of the ROI region in each image block in the image block; and preprocess the image block sequence according to the proportion to obtain a preprocessed image block sequence; The second recognition module is used to encode each image block itself, the proportion r1 of each image block, and the number of each image block in the preprocessed image block sequence to obtain an image block code, a weight code, and a position code, fuse the weight code and the image block code to obtain a first fusion result, fuse the position code and the image block code to obtain a second fusion result, use the second fusion result as the input of the Q matrix and V matrix corresponding to the multi-head attention mechanism in the first layer of the Encoder, use the first fusion result as the input of the K matrix corresponding to h1 attention heads in the multi-head attention mechanism in the first layer of the Encoder, and use the image block code as the input of the K matrix corresponding to the remaining h2 attention heads in the multi-head attention mechanism in the first layer of the Encoder; output the recognition result of the DETR model; The fusing of the weight coding and the image block coding to obtain a first fusion result is specifically as follows: encoding the image block to obtain a vector of length M, encoding the proportion corresponding to the image block to obtain a weight coding vector of length N, and performing vector concatenation of the vector of length M and the weight coding vector of length N to obtain a first fusion result of length M+N, where M and N are positive integers and M is an even number; The fusion of the position code and the image block code to obtain the second fusion result is specifically as follows: determining the row number and column number of the image block according to the position code, encoding the row number and column number into vectors of length M / 2 respectively, and performing vector splicing on the two vectors of length M / 2 to obtain a position vector of length M to obtain the position code; adding the image block code and the position code bit by bit to obtain the first vector; determining the center point of each ROI area, calculating the distance between the image block and the nearest center point, encoding the distance into a vector of length N, and performing vector splicing on the vector of length M and the first vector to obtain a second fusion result of length M+N.

6. The system according to claim 5, wherein: The image block sequence is preprocessed according to the proportion to obtain a preprocessed image block sequence, specifically: If the proportion is 1, the image block is deleted from the image block sequence and the corresponding image block number is deleted; if the proportion is 0, the image block is not processed; otherwise, the ROI area in the image is expanded so that all image blocks with a proportion less than 1 and greater than 0 do not include the ROI area.

Images