A method and system for detecting foreign objects in pharmaceutical raw materials based on production line images

Abstract

The present invention relates to the technical field of image detection, and discloses a method and system for detecting foreign objects in pharmaceutical raw materials based on production line images. First, the present invention performs edge extraction on the initial pharmaceutical raw material production line image, divides it based on an adaptive threshold, and obtains the processed pharmaceutical raw material production line image; secondly, uses the KNN background difference method to segment the background and foreground of the processed pharmaceutical raw material production line image, extracts and fills the foreground image area to obtain the final pharmaceutical raw material production line image; then constructs a MaskR-CNN network, uses the termite life cycle optimization algorithm to optimize the initial learning rate in the network to obtain the global optimal solution; finally, obtains an improved MaskR-CNN network model according to the global optimal solution, outputs the recognition result, and completes the detection of foreign objects in pharmaceutical raw materials. The present invention processes and analyzes through the pharmaceutical raw material production line image, and the purpose of detecting foreign objects in pharmaceutical raw materials is achieved, and the method is accurate and objective.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and particularly to a method and system for detecting foreign objects in pharmaceutical raw materials based on production line images. Background Technique

[0002] Chinese Patent CN114723678B discloses a method and detection system for detecting foreign objects on high-voltage wires based on video images. The method specifically includes: acquiring video images of foreign objects on high-voltage wires, acquiring the next frame of image every m minutes to obtain a set of video images of foreign objects on high-voltage wires; performing equal slicing processing on the video images of foreign objects on high-voltage wires in the set of video images of foreign objects on high-voltage wires to form a grid map, and obtaining the image of the area framed by the coordinate box; magnifying the grid map and sequentially stacking it along the X-axis and Y-axis to form a batch of image groups, dividing the image groups into positive samples and negative samples according to whether the foreign objects stay on the high-voltage wires, and inputting them into a deep learning network model for training to obtain an identification model with a binary classification result, inputting the image of the area framed by the coordinate box into the identification model with the binary classification result, outputting the binary classification result, and judging the foreign objects on the high-voltage wires and the foreign objects not on the high-voltage wires. This invention does not optimize the deep learning network model, and the convergence speed is slow.

[0003] Traditional methods for detecting foreign objects in pharmaceutical raw materials rely on camera groups for detection on the conveyor belt. However, high-tech means such as neural networks are not used in the detection process, and the detection results often have accuracy errors, which may lead to small foreign objects being mixed into the raw materials. At the same time, traditional methods for detecting foreign objects in pharmaceutical raw materials have a slow detection speed and a high missed detection rate, posing a great threat to the physical health of patients. Summary of the Invention

[0004] In view of the problems in the related art, the present invention provides a method and system for detecting foreign objects in pharmaceutical raw materials based on production line images to overcome the above technical problems existing in the existing related technologies.

[0005] To solve the above technical problems, the present invention is realized through the following technical solutions:

[0006] The present invention provides a method for detecting foreign objects in pharmaceutical raw materials based on production line images, including the following steps:

[0007] S1. Acquire production line images of pharmaceutical raw materials to obtain an initial set of production line images of pharmaceutical raw materials, and use an improved sobel operator edge detection algorithm to perform edge extraction on the initial production line images of pharmaceutical raw materials in the initial set of production line images of pharmaceutical raw materials to obtain a processed set of production line images of pharmaceutical raw materials;

[0008] S2. Segment the background and foreground of the processed pharmaceutical raw material production line images in the processed pharmaceutical raw material production line image set based on the KNN background difference method, extract the foreground image region, and then perform hole filling to obtain the final pharmaceutical raw material production line image set;

[0009] S3. Construct a MaskR-CNN network, establish an iterative formula for the stochastic gradient descent optimizer, and use the termite life cycle optimization algorithm to optimize the initial learning rate in the iterative formula of the stochastic gradient descent optimizer to obtain the global optimal solution of the iterative formula of the stochastic gradient descent optimizer;

[0010] S4. Obtain an improved MaskR-CNN network model according to the global optimal solution of the iterative formula of the stochastic gradient descent optimizer, and output the recognition result based on the final pharmaceutical raw material production line image set to complete the foreign object detection of pharmaceutical raw materials.

[0011] The present invention extracts the edges of the initial pharmaceutical raw material production line images by using an improved sobel operator edge detection algorithm, clusters the initial pharmaceutical raw material production line images by introducing the K-means algorithm to obtain an adaptive threshold, and then divides them according to the adaptive threshold to obtain the processed pharmaceutical raw material production line images; compared with the traditional edge detection algorithm, this algorithm effectively retains the edge information and has good robustness and accuracy; secondly, the KNN background difference method is used to segment the background and foreground of the processed pharmaceutical raw material production line images, and image segmentation is realized by establishing a probability distribution model. Then, hole filling is performed on the images processed by the KNN background difference method to make the foreground region more complete and obtain a better segmentation effect. At the same time, the KNN background difference method distinguishes pharmaceutical raw materials from foreign objects in pharmaceutical raw materials, laying a foundation for subsequent processing and improving the accuracy of foreign object judgment; then a MaskR-CNN network is constructed, and the termite life cycle optimization algorithm is used to optimize the initial learning rate in the neural network to obtain the global optimal solution during the iterative process of the neural network. This algorithm simulates the iterative process of the initial learning rate as the behavior of a termite colony to solve the optimization problem. The Lévy flight effectively improves the convergence speed, and different types of termites such as worker ants expand the spatial search ability, facilitating the finding of the global optimal value with the minimum error, further improving the detection speed and accuracy; finally, an improved MaskR-CNN network model is obtained according to the global optimal solution, and the recognition result is output to complete the foreign object detection of pharmaceutical raw materials.

[0012] Preferably, the S1 includes the following steps:

[0013] S11. Obtain each frame of pharmaceutical raw material production line image to obtain an initial pharmaceutical raw material production line image set, and perform quantization processing on the pixel points on the initial pharmaceutical raw material production line images in the initial pharmaceutical raw material production line image set; Set the size to An operator template, place the operator template on the image of the initial pharmaceutical raw material production line, establish a Cartesian grid, obtain the city distance of the pixel points in the operator template, and obtain a distance matrix;

[0014] Calculate the sum of the gradient vectors of the central pixel point of the operator template in the vertical direction, horizontal direction, 45-degree diagonal direction, and 135-degree diagonal direction, and calculate the average gradient component of the central pixel point of the operator template according to the gradient vector sum and the distance matrix; successively calculate the average gradient components of other pixel points in the operator template to obtain an average gradient component matrix, and perform a convolution operation on the average gradient component matrix to obtain an average gradient value matrix;

[0015] S12. Traverse the image of the initial pharmaceutical raw material production line, calculate the average gradient value of the pixel points in the image of the initial pharmaceutical raw material production line to form an image average gradient value matrix; according to the image average gradient value matrix, use the K-means algorithm to cluster the image of the initial pharmaceutical raw material production line to obtain an adaptive threshold. The specific steps are as follows:

[0016] S121. Select any 2 pixel points in the image of the initial pharmaceutical raw material production line, and record them as the first clustering center pixel point and the second clustering center pixel point respectively, and use them as the initial clustering centers; according to the image average gradient value matrix, calculate the average gradient differences between other pixel points in the image of the initial pharmaceutical raw material production line and the first clustering center pixel point and the second clustering center pixel point, and classify the pixel points into the nearest initial clustering center according to the minimum average gradient difference to form classes;

[0017] S122. After all the pixel points in the image of the initial pharmaceutical raw material production line are classified, set the number of pixel points in the class as a The pixel point coordinates are recorded as Divide the average gradient value of the class again to obtain a new average gradient value. The calculation formula is as follows:

[0018] ;

[0019] Among them, Represents the horizontal i 、Vertical j The coordinates of the th pixel point, Represents the pixel point coordinates The average gradient value at the place, Represents the class, A Represents the new average gradient value;

[0020] Obtain new clustering centers according to the new average gradient value, re-classify according to the new clustering centers until the clustering centers no longer change, stop classification, and obtain the final class; find the minimum average gradient value in the final class, and the minimum average gradient value is the adaptive threshold;

[0021] S13. The initial pharmaceutical raw material production line images in the set of initial pharmaceutical raw material production line images are marked according to the adaptive threshold. When the gray value of a pixel point in an initial pharmaceutical raw material production line image is greater than the adaptive threshold, the gray value of the pixel point is set to 255; otherwise, the gray value of the pixel point is set to 0, completing the edge extraction of the initial pharmaceutical raw material production line image, removing the images outside the production line, obtaining the processed pharmaceutical raw material production line image, and generating a set of processed pharmaceutical raw material production line images.

[0022] The present invention performs edge extraction on the initial pharmaceutical raw material production line image through an improved Sobel operator edge detection algorithm, and performs clustering on the initial pharmaceutical raw material production line image by introducing the K-means algorithm to obtain an adaptive threshold. Compared with traditional edge detection algorithms, it effectively retains edge information and has better robustness and accuracy.

[0023] Preferably, S2 includes the following steps:

[0024] S21. Select any processed pharmaceutical raw material production line image in the set of processed pharmaceutical raw material production line images, denoted as the current frame pharmaceutical raw material production line image, and then select the previous frame pharmaceutical raw material production line image of the current frame pharmaceutical raw material production line image; collect k nearest neighbor pixel points on the current frame pharmaceutical raw material production line image, calculate the norm difference between the nearest neighbor pixel points and the previous frame pharmaceutical raw material production line image, and establish a pixel point probability distribution model; set a regional index set , where represents the m th regional index. A regional index of 0 represents a foreground image area, and a regional index of 1 represents a background image area. According to the pixel point probability distribution model and the regional index set, calculate the background probability;

[0025] Set a foreground threshold and a background threshold. Assign a gray value of 255 to the foreground image area where the pixel points corresponding to the background probability less than the foreground threshold are located, assign a gray value of 0 to the background image area where the pixel points corresponding to the background probability greater than the background threshold are located, and assign a gray value of 128 to the image area where the pixel points corresponding to the background probability greater than or equal to the foreground threshold and less than or equal to the background threshold are located; regard the pharmaceutical raw material foreign matter as the foreground image area and the pharmaceutical raw material as the background image area, complete the background and foreground segmentation, obtain the segmented pharmaceutical raw material production line image, and form a set of segmented pharmaceutical raw material production line images;

[0026] S22. Perform opening operation on the segmented pharmaceutical raw material production line images in the segmented pharmaceutical raw material production line image set, and then perform closing operation to obtain a processed pharmaceutical raw material production line image set. Perform hole filling on the processed pharmaceutical raw material production line image set to obtain a final pharmaceutical raw material production line image set. The specific steps are as follows:

[0027] S221. For the foreground image area in the processed pharmaceutical raw material production line image, select any pixel point in the foreground image area and denote it as the seed pixel point. Taking the seed pixel point as the center, traverse other pixel points in the foreground image area, and set the gray values of other pixel points with the same gray value as the seed pixel point to 255 until the edge of the foreground image area is traversed to obtain a preliminarily extracted pharmaceutical raw material production line image;

[0028] S222. Invert the gray values of the pixel points in the preliminarily extracted pharmaceutical raw material production line image to obtain an inverted pharmaceutical raw material production line image. Perform bitwise AND operation on the preliminarily extracted pharmaceutical raw material production line image and the inverted pharmaceutical raw material production line image to fill the missing gray values of the pixel points in the preliminarily extracted pharmaceutical raw material production line image, complete hole filling, obtain a final pharmaceutical raw material production line image, and form a final pharmaceutical raw material production line image set.

[0029] The present invention uses the KNN background difference method to segment the background and foreground of the processed pharmaceutical raw material production line image, distinguish the pharmaceutical raw materials from the foreign objects in the pharmaceutical raw materials, lay a foundation for subsequent processing, and improve the accuracy of foreign object judgment; then perform hole filling on the image processed by the KNN background difference method to make the foreground area more complete and obtain a better segmentation effect.

[0030] Preferably, the S3 includes the following steps:

[0031] S31. Set the backbone network of the Mask R-CNN network as the ResNet 50 network. The ResNet50 network includes several two-dimensional convolutional layers, max pooling layers, average pooling layers, residual modules, and fully connected layers; introduce an attention mechanism, compress the input image in the average pooling layer, and then perform convolution to obtain a channel attention feature map, which is output to the fully connected layer; use transfer learning to transfer the training weights of the input image to the Mask R-CNN network to construct a Mask R-CNN network;

[0032] ​S32. Obtain the COCO dataset on the network, select images of foreign objects in pharmaceutical raw materials such as fibers, metal chips, and glass chips in the COCO dataset to form a sample set of pharmaceutical raw material foreign object images; extract the foreground image regions of the pharmaceutical raw material foreign object image samples in the sample set of pharmaceutical raw material foreign object images, and perform annotation to obtain an annotated sample set of pharmaceutical raw material foreign object images; then perform brightness transformation, rotation, and noise addition on the annotated sample set of pharmaceutical raw material foreign object images respectively to achieve data augmentation and obtain a final sample set of pharmaceutical raw material foreign object images;

[0033] Set the MaskR-CNN network to use the stochastic gradient descent optimizer. The number of samples in the final sample set of pharmaceutical raw material foreign object images is b , the weight parameter is , the initial learning rate is , for the b th sample in the final sample set of pharmaceutical raw material foreign object images, the label is , for the b th sample in the final sample set of pharmaceutical raw material foreign object images, the predicted label is , then the iteration formula of the stochastic gradient descent optimizer is as follows:

[0034] ;

[0035] Among them, represents the updated weight parameter;

[0036] S33. Input the final sample set of pharmaceutical raw material foreign object images into the MaskR-CNN network. As the iteration continues, the initial learning rate is updated, and the termite life cycle optimization algorithm is used to optimize the initial learning rate to obtain the global optimal solution of the iteration formula of the stochastic gradient descent optimizer. The specific steps are as follows:

[0037] S331. Take the iteration formula of the stochastic gradient descent optimizer as the fitness function. Assume that there is a termite population in the search space. The initial size of the termite population is F, and the termite individuals in the termite population represent the initial learning rate in the iteration formula of the stochastic gradient descent optimizer; when determining the initial step size of the termite individuals, Lévy flight is introduced. Set the Lévy flight index to , and the calculation formula for the initial step size of the termite individuals is as follows:

[0038] ;

[0039] Among them, E represents the initial step size of the termite individuals and follows the Lévy flight distribution, and represent the normal distribution coefficients;

[0040] When the Lévy flight index is between 1 and 2, the initial step length of the termite individual is controlled by the Lévy flight index; otherwise, the initial step length of the termite individual is improved. Set the current iteration number as n , the maximum iteration number as N , at the n -th iteration, the step length of the termite individual is denoted as , then the step length of the termite individual satisfies the following formula:

[0041] ;

[0042] S332. The termite population moves with the step length of the termite individual. Set the movement vector set , where , , and represent movement vectors, and the movement of the termite population is controlled by the movement vector set. Set the position of the termite individual at the n -th iteration as , the best position of the termite individual at the n -th iteration as , then the position n+ of the termite individual at the -th iteration is calculated as follows:

[0043] ;

[0044] Further, the position of the termite individual at the n -th iteration corresponds to the learning rate of the MaskR-CNN network at the n -th iteration, and the initial learning rate is updated to the learning rate at the n -th iteration;

[0045] S333. As the position of the termite individual is continuously updated, worker-soldier individuals are introduced to expand the search space. Set the position of the worker-soldier individual at the n -th iteration as , then the position n+ of the h -th worker-soldier individual at the -th iteration is calculated as follows:

[0046] ;

[0047] During the update process of the position of the worker-soldier individual, calculate the best fitness function value corresponding to the position of the n+ -th worker-soldier individual at the h -th iteration, compare it with the best fitness function value corresponding to the position of the termite individual at the n+ -th iteration, and select the n+The best fitness function value of the first iteration is assigned to the n learning rate of the iteration; when the current number of iterations reaches the maximum number of iterations, the iteration stops, and the final h positions of the worker ant soldiers are obtained, and the global optimal solution of the iteration formula of the stochastic gradient descent optimizer is obtained.

[0048] The invention constructs a MaskR-CNN network, uses the termite life cycle optimization algorithm to optimize the initial learning rate in the neural network, obtains an improved MaskR-CNN network model, and simulates the initial learning rate iteration process as the behavior of a termite colony to solve the optimization problem; among them, Levy flight effectively improves the convergence speed, and different types of termites such as worker ants expand the space search ability, facilitating the finding of the global optimal value with the minimum error, further improving the detection speed and accuracy.

[0049] Preferably, S4 includes the following steps:

[0050] S41. Obtain an improved MaskR-CNN network model according to the global optimal solution of the iteration formula of the stochastic gradient descent optimizer. After annotating, performing brightness transformation, rotation, and adding noise to the final pharmaceutical raw material production line image set, input it into the improved MaskR-CNN network model, and output the recognition result;

[0051] S42. Obtain the type of foreign matter in the pharmaceutical raw material according to the recognition result, and complete the detection of foreign matter in the pharmaceutical raw material.

[0052] This embodiment also discloses a system for a method for detecting foreign matter in pharmaceutical raw materials based on production line images, specifically including: an image edge extraction module, an image background and foreground segmentation and filling module, a neural network parameter optimization module, and a pharmaceutical raw material foreign matter detection module;

[0053] The image edge extraction module is used to extract the edges of the initial pharmaceutical raw material production line image using an improved sobel operator edge detection algorithm;

[0054] The image background and foreground segmentation and filling module is used to segment the background and foreground of the processed pharmaceutical raw material production line image based on the KNN background difference method, and then perform hole filling;

[0055] The neural network parameter optimization module is used to optimize the initial learning rate using the termite life cycle optimization algorithm;

[0056] The pharmaceutical raw material foreign matter detection module is used to output the recognition result using the improved MaskR-CNN network model.

[0057] The present invention has the following beneficial effects:

[0058] 1. The invention extracts edges from the initial pharmaceutical raw material production line image through an improved Sobel operator edge detection algorithm, and clusters the initial pharmaceutical raw material production line image by introducing the K-means algorithm to obtain an adaptive threshold. Compared with traditional edge detection algorithms, it effectively retains edge information and has better robustness and accuracy.

[0059] 2. The invention uses the KNN background difference method to segment the background and foreground of the processed pharmaceutical raw material production line image, distinguish pharmaceutical raw materials from foreign objects in pharmaceutical raw materials, lay a foundation for subsequent processing, and improve the accuracy of foreign object judgment; then fills the holes in the image processed by the KNN background difference method to make the foreground area more complete and obtain a better segmentation effect.

[0060] 3. The invention constructs a MaskR-CNN network and optimizes the initial learning rate in the neural network using the termite life cycle optimization algorithm to obtain an improved MaskR-CNN network model. By simulating the iterative process of the initial learning rate as the behavior of a termite colony, the optimization problem is solved; among them, Lévy flight effectively improves the convergence speed, and different types of termites such as worker ants expand the spatial search ability, facilitating the finding of the global optimal value with the minimum error, further improving the detection speed and accuracy.

[0061] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0063] Figure 1 It is a schematic flow diagram of the detection of foreign objects in pharmaceutical raw materials by a pharmaceutical raw material foreign object detection system based on the production line image provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0065] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner", etc. indicating the orientation or positional relationship are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the invention.

[0066] Embodiment 1

[0067] Please refer to Figure 1 , this embodiment discloses a foreign object detection method for pharmaceutical raw materials based on production line images, which specifically includes the following contents:

[0068] S1. Obtain production line images of pharmaceutical raw materials to obtain an initial set of production line images of pharmaceutical raw materials. Use an improved Sobel operator edge detection algorithm to perform edge extraction on the initial production line images of pharmaceutical raw materials in the initial set of production line images of pharmaceutical raw materials, and obtain a processed set of production line images of pharmaceutical raw materials;

[0069] The said S1 includes the following steps:

[0070] S11. Obtain each frame of production line images of pharmaceutical raw materials to obtain an initial set of production line images of pharmaceutical raw materials, and perform quantization processing on the pixel points on the initial production line images of pharmaceutical raw materials in the initial set of production line images of pharmaceutical raw materials; Set the size as operator template, place the operator template on the initial production line image of pharmaceutical raw materials, and establish a Cartesian grid to obtain the city distance of the pixel points in the operator template, and obtain a distance matrix;

[0071] Calculate the sum of the gradient vectors of the central pixel point of the operator template in the vertical direction, horizontal direction, 45-degree diagonal direction, and 135-degree diagonal direction, and calculate the average gradient component of the central pixel point of the operator template according to the sum of the gradient vectors and the distance matrix; Calculate the average gradient components of other pixel points in the operator template in turn to obtain an average gradient component matrix, and perform convolution operation on the average gradient component matrix to obtain an average gradient value matrix;

[0072] S12. Traverse the initial production line image of pharmaceutical raw materials, calculate the average gradient value of the pixel points in the initial production line image of pharmaceutical raw materials to form an image average gradient value matrix; According to the image average gradient value matrix, use the K-means algorithm to cluster the initial production line image of pharmaceutical raw materials to obtain an adaptive threshold. The specific steps are as follows:

[0073] S121, select any two pixels in the initial pharmaceutical raw material production line image, record them as the first cluster center pixel and the second cluster center pixel, and use them as the initial cluster center; calculate the average gradient difference between other pixels in the initial pharmaceutical raw material production line image and the first cluster center pixel and the second cluster center pixel according to the image average gradient value matrix, and classify the pixels to the nearest initial cluster center according to the minimum average gradient difference to form a class;

[0074] S122, after all pixels in the initial pharmaceutical raw material production line image are classified, the number of pixels in the class is set to a , the pixel coordinates are marked as , divide the average gradient value of the class again to get a new average gradient value, the calculation formula is as follows:

[0075] ;

[0076] in, Indicates horizontal i , vertical j Pixel coordinates, Represents pixel coordinates The average gradient value at Represents a class, A Represents the new average gradient value;

[0077] A new cluster center is obtained according to the new average gradient value, and the classification is re-performed according to the new cluster center until the cluster center no longer changes, and the classification is stopped to obtain the final class; the minimum average gradient value is found in the final class, and the minimum average gradient value is the adaptive threshold;

[0078] S13, the initial pharmaceutical raw material production line images in the initial pharmaceutical raw material production line image set are marked according to the adaptive threshold value, when the gray value of the pixel in the initial pharmaceutical raw material production line image is greater than the adaptive threshold value, the gray value of the pixel is set to 255, otherwise the gray value of the pixel is set to 0, the edge extraction of the initial pharmaceutical raw material production line image is completed, the images outside the production line are eliminated, the processed pharmaceutical raw material production line images are obtained, and the processed pharmaceutical raw material production line image set is generated;

[0079] S2. Based on the KNN background difference method, the processed pharmaceutical raw material production line images in the processed pharmaceutical raw material production line image set are segmented into background and foreground, the foreground image area is extracted, and then the hole filling is performed to obtain a final pharmaceutical raw material production line image set;

[0080] The S2 comprises the following steps:

[0081] S21. Select any processed pharmaceutical raw material production line image from the set of processed pharmaceutical raw material production line images, denoted as the current frame pharmaceutical raw material production line image, and then select the previous frame pharmaceutical raw material production line image of the current frame pharmaceutical raw material production line image; collect k nearest pixel points on the current frame pharmaceutical raw material production line image, calculate the norm difference between the nearest pixel points and the previous frame pharmaceutical raw material production line image, and establish a pixel point probability distribution model; set a regional index set , where represents the m th regional index. A regional index of 0 indicates a foreground image region, and a regional index of 1 indicates a background image region. According to the pixel point probability distribution model and the regional index set, calculate the background probability;

[0082] Set a foreground threshold and a background threshold. Assign a gray value of 255 to the foreground image region where the pixel points with a background probability less than the foreground threshold are located, assign a gray value of 0 to the background image region where the pixel points with a background probability greater than the background threshold are located, and assign a gray value of 128 to the image region where the pixel points with a background probability greater than or equal to the foreground threshold and less than or equal to the background threshold are located; regard the pharmaceutical raw material foreign matter as the foreground image region and the pharmaceutical raw material as the background image region to complete the background and foreground segmentation, obtain the segmented pharmaceutical raw material production line image, and form a set of segmented pharmaceutical raw material production line images;

[0083] S22. Perform an opening operation on the segmented pharmaceutical raw material production line images in the set of segmented pharmaceutical raw material production line images, and then perform a closing operation to obtain a set of processed pharmaceutical raw material production line images. Perform hole filling on the set of processed pharmaceutical raw material production line images to obtain a final set of pharmaceutical raw material production line images. The specific steps are as follows:

[0084] S221. For the foreground image region in the processed pharmaceutical raw material production line image, select any pixel point in the foreground image region, denoted as the seed pixel point. Taking the seed pixel point as the center, traverse other pixel points in the foreground image region, and set the gray values of other pixel points with the same gray value as the seed pixel point to 255 until the edge of the foreground image region is traversed to obtain a preliminarily extracted pharmaceutical raw material production line image;

[0085] S222. Invert the gray values of the pixel points in the preliminarily extracted pharmaceutical raw material production line image to obtain an inverted pharmaceutical raw material production line image. Perform a bitwise AND operation on the preliminarily extracted pharmaceutical raw material production line image and the inverted pharmaceutical raw material production line image to fill in the missing gray values of the pixel points in the preliminarily extracted pharmaceutical raw material production line image, complete the hole filling, obtain the final pharmaceutical raw material production line image, and form a set of final pharmaceutical raw material production line images;

[0086] S3. Build a Mask R-CNN network, establish an iterative formula for the stochastic gradient descent optimizer, and use the termite life cycle optimization algorithm to optimize the initial learning rate in the iterative formula of the stochastic gradient descent optimizer to obtain the global optimal solution of the iterative formula of the stochastic gradient descent optimizer;

[0087] The S3 includes the following steps:

[0088] S31. Set the backbone network of the Mask R-CNN network to the ResNet 50 network. The ResNet50 network includes several two-dimensional convolutional layers, max pooling layers, average pooling layers, residual modules, and fully connected layers; introduce an attention mechanism to compress the input image in the average pooling layer, and then perform convolution to obtain a channel attention feature map, which is output to the fully connected layer; use transfer learning to transfer the training weights of the input image to the Mask R-CNN network to construct the Mask R-CNN network; S32. Obtain the COCO dataset on the network, select images of foreign objects in pharmaceutical raw materials such as fibers, metal chips, and glass chips in the COCO dataset to form a sample set of images of foreign objects in pharmaceutical raw materials; extract the foreground image regions of the sample images of foreign objects in pharmaceutical raw materials in the sample set of images of foreign objects in pharmaceutical raw materials and perform annotation to obtain an annotated sample set of images of foreign objects in pharmaceutical raw materials; then perform brightness transformation, rotation, and noise addition on the annotated sample set of images of foreign objects in pharmaceutical raw materials respectively to achieve data augmentation and obtain the final sample set of images of foreign objects in pharmaceutical raw materials;

[0089] Set the Mask R-CNN network to use the stochastic gradient descent optimizer. The number of samples in the final sample set of images of foreign objects in pharmaceutical raw materials is

[0090] , the weight parameter is b , the initial learning rate is , the label of the th sample in the final sample set of images of foreign objects in pharmaceutical raw materials is b , the predicted label of the th sample in the final sample set of images of foreign objects in pharmaceutical raw materials is b , then the iterative formula of the stochastic gradient descent optimizer is as follows:

[0091] ;

[0092] Among them, represents the updated weight parameter;

[0093] S33. Input the set of final pharmaceutical raw material foreign object image samples into the MaskR-CNN network. As the iteration progresses, the initial learning rate is updated, and the termite life cycle optimization algorithm is used to optimize the initial learning rate to obtain the global optimal solution of the random gradient descent optimizer iteration formula. The specific steps are as follows:

[0094] S331. Take the random gradient descent optimizer iteration formula as the fitness function. Assume that there is a termite population in the search space, and the initial size of the termite population is F. The termite individuals in the termite population represent the initial learning rate in the random gradient descent optimizer iteration formula. When determining the initial step size of the termite individuals, Levy flight is introduced, and the Levy flight index is set to . The formula for the initial step size of the termite individuals is as follows:

[0095] ;

[0096] where E represents the initial step size of the termite individuals and follows the Levy flight distribution, and represent the normal distribution coefficients;

[0097] When the Levy flight index is between 1 and 2, the initial step size of the termite individuals is controlled by the Levy flight index. Otherwise, the initial step size of the termite individuals is improved. Assume the current iteration number is n , the maximum iteration number is N . At the n -th iteration, the step size of the termite individual is denoted as . Then the step size of the termite individual satisfies the following formula:

[0098] ;

[0099] S332. The termite population moves with the step size of the termite individuals. Assume the set of movement vectors is , where , , and represent the movement vectors, and the movement of the termite population is controlled by the set of movement vectors. Assume the position of the termite individual at the n -th iteration is , and the best position of the termite individual at the n -th iteration is . Then the position n+ of the termite individual at the -th iteration is calculated as follows:

[0100] ;

[0101] Furthermore, at the nAt the n th iteration, the learning rate corresponding to the termite individual position is used to update the initial learning rate to the learning rate at the n th iteration;

[0102] S333. As the positions of termite individuals are continuously updated, worker ant and soldier individuals are introduced to expand the search space. Set the position of the worker ant and soldier individuals at the n th iteration as , then the position of the n+ th worker ant and soldier individual at the h th iteration is . The calculation formula is as follows:

[0103] ;

[0104] During the update process of the positions of worker ant and soldier individuals, calculate the best fitness function value corresponding to the position of the n+ th worker ant and soldier individual at the h th iteration, compare it with the best fitness function value corresponding to the termite individual position at the n+ th iteration, select the best fitness function value at the n+ th iteration, and assign it to the learning rate at the n th iteration; When the current iteration number reaches the maximum iteration number, stop the iteration to obtain the final position of the h th worker ant and soldier individual, and obtain the global optimal solution of the random gradient descent optimizer iteration formula;

[0105] S4. Obtain an improved MaskR-CNN network model according to the global optimal solution of the random gradient descent optimizer iteration formula, and output the recognition result based on the final pharmaceutical raw material production line image set to complete the detection of foreign objects in pharmaceutical raw materials;

[0106] The S4 includes the following steps:

[0107] S41. Obtain an improved MaskR-CNN network model according to the global optimal solution of the random gradient descent optimizer iteration formula. After annotating, performing brightness transformation, rotation, and adding noise to the final pharmaceutical raw material production line image set, input it into the improved MaskR-CNN network model and output the recognition result;

[0108] S42. Obtain the type of foreign object in the pharmaceutical raw material according to the recognition result to complete the detection of foreign objects in the pharmaceutical raw material.

[0109] Example 2

[0110] This embodiment also discloses a system for a pharmaceutical raw material foreign object detection method based on production line images, specifically including: an image edge extraction module, an image background and foreground segmentation and filling module, a neural network parameter optimization module, and a pharmaceutical raw material foreign object detection module;

[0111] The image edge extraction module is used to extract edges from the initial pharmaceutical raw material production line image using an improved sobel operator edge detection algorithm;

[0112] The image background and foreground segmentation and filling module is used to segment the background and foreground of the processed pharmaceutical raw material production line image based on the KNN background difference method and then perform hole filling;

[0113] The neural network parameter optimization module is used to optimize the initial learning rate using the termite life cycle optimization algorithm;

[0114] The pharmaceutical raw material foreign object detection module is used to output the recognition result using an improved MaskR-CNN network model.

[0115] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0116] The preferred embodiments of the invention disclosed above are only used to help illustrate the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification in order to better explain the principles and practical applications of the invention, so that those skilled in the relevant technical fields can understand and utilize the invention well.

Claims

1. A foreign object detection method for pharmaceutical raw materials based on production line images, characterized in that, It includes the following steps: S1. Obtain the images of the pharmaceutical raw material production line to get the initial set of images of the pharmaceutical raw material production line. Use the improved Sobel operator edge detection algorithm to extract the edges of the initial images of the pharmaceutical raw material production line in the initial set of images of the pharmaceutical raw material production line, and obtain the processed set of images of the pharmaceutical raw material production line; S2. Based on the KNN background difference method, segment the background and foreground of the processed images of the pharmaceutical raw material production line in the processed set of images of the pharmaceutical raw material production line, extract the foreground image region, and then perform hole filling to obtain the final set of images of the pharmaceutical raw material production line; S3. Construct a Mask R-CNN network, establish an iterative formula for the stochastic gradient descent optimizer, and optimize the initial learning rate in the iterative formula of the stochastic gradient descent optimizer to obtain the global optimal solution of the iterative formula of the stochastic gradient descent optimizer; S4. Obtain an improved Mask R-CNN network model according to the global optimal solution of the iterative formula of the stochastic gradient descent optimizer. According to the final set of images of the pharmaceutical raw material production line, output the recognition result to complete the detection of foreign objects in pharmaceutical raw materials; The S3 includes the following steps: S31. Set the backbone network of the Mask R-CNN network as the ResNet 50 network. The ResNet 50 network includes several two-dimensional convolutional layers, max pooling layers, average pooling layers, residual modules, and fully connected layers. Introduce the attention mechanism and use transfer learning to transfer the training weights of the input image into the Mask R-CNN network to construct the Mask R-CNN network; The two-dimensional convolutional layer, max pooling layer, average pooling layer, residual module, and fully connected layer; introduce the attention mechanism and use transfer learning to transfer the training weights of the input image into the Mask R-CNN network to construct the Mask R-CNN network; S32. Obtain the COCO dataset on the network. Select the images of foreign objects in pharmaceutical raw materials such as fibers, metal chips, and glass chips in the COCO dataset to form a set of sample images of foreign objects in pharmaceutical raw materials. Extract the foreground image regions of the sample images of foreign objects in pharmaceutical raw materials in the set of sample images of foreign objects in pharmaceutical raw materials, and perform annotation, brightness transformation, rotation, and adding noise to obtain the final set of sample images of foreign objects in pharmaceutical raw materials; Set the Mask R-CNN network to use the stochastic gradient descent optimizer, and the final number of pharmaceutical raw material foreign object image sample sets is b , and the weight parameter is , the initial learning rate is , and the label of the b th sample in the final pharmaceutical raw material foreign object image sample set is , and the predicted label of the b th sample in the final pharmaceutical raw material foreign object image sample set is , then the iteration formula of the stochastic gradient descent optimizer is as follows: ; Among them, represents the updated weight parameter; S33. Input the final set of sample images of foreign objects in pharmaceutical raw materials into the Mask R-CNN network. As the iteration progresses, the initial learning rate is updated. Use the termite life cycle optimization algorithm to optimize the initial learning rate to obtain the global optimal solution of the iterative formula of the stochastic gradient descent optimizer; The S33 includes the following steps: S331. Take the random gradient descent optimizer iteration formula as the fitness function. Assume that there is a termite population in the search space, and the initial size of the termite population is F. The termite individuals in the termite population represent the initial learning rate in the random gradient descent optimizer iteration formula. During the process of determining the initial step size of the termite individuals, Levy flight is introduced, and the Levy flight index is set as , and the initial step size of the termite individuals is obtained. When the Lévy flight index is between 1 and 2, the initial step length of termite individuals is controlled by the Lévy flight index; otherwise, the initial step length of termite individuals is improved. Set the current iteration number as n , the maximum iteration number as N , at the n -th iteration, the step length of termite individuals is denoted as , then the step length of termite individuals satisfies the following formula: ; S332. The termite population moves at the step length of the termite individuals, and a set of movement vectors is set , where , , and represent movement vectors, and the movement of the termite population is controlled by the set of movement vectors; it is set that the position of the termite individual at the n -th iteration is , and the best position of the termite individual at the n -th iteration is , then the position n+ of the termite individual at the -th iteration is calculated as follows: ； Further, at the n th iteration, the position of the termite individual corresponds to the learning rate of the Mask R-CNN network at the nth iteration, and the initial learning rate is updated to the n th iteration learning rate; S333. As the position of the termite individuals is continuously updated, worker-soldier individuals are introduced to expand the search space. Set the position of the worker-soldier individuals at the n th iteration, and calculate the position of the n+ 1st worker-soldier individual at the h th iteration; During the update process of the positions of worker ant soldiers, calculate the best fitness function value corresponding to the position of the n+ first iteration of the h worker ant soldiers, compare the best fitness function value corresponding to the position of the termite individuals in the n+ first iteration, select the best fitness function value of the n+ first iteration, and assign it to the learning rate of the n iteration; when the current iteration number reaches the maximum iteration number, stop the iteration, and obtain the positions of the h worker ant soldiers, and obtain the global optimal solution of the iterative formula of the stochastic gradient descent optimizer.

2. The pharmaceutical raw material foreign object detection method based on the production line image according to claim 1, wherein, The S1 includes the following steps: S11. Obtain the images of the pharmaceutical raw material production line to get the initial set of images of the pharmaceutical raw material production line. Set an operator template on the initial images of the pharmaceutical raw material production line, and calculate the average gradient components of the pixel points in the operator template to obtain an average gradient value matrix; S12. Traverse the initial images of the pharmaceutical raw material production line to obtain the image average gradient value matrix. According to the image average gradient value matrix, use the K-means algorithm to cluster the initial images of the pharmaceutical raw material production line to obtain an adaptive threshold; S13. Use the adaptive threshold to extract the edges of the initial images of the pharmaceutical raw material production line to obtain the processed images of the pharmaceutical raw material production line, and generate a processed set of images of the pharmaceutical raw material production line.

3. The foreign object detection method for pharmaceutical raw materials based on production line images according to claim 2, characterized in that, The clustering of the initial images of the pharmaceutical raw material production line using the K-means algorithm includes the following steps: Select the initial clustering centers in the initial pharmaceutical raw material production line image. According to the image average gradient value matrix, calculate the average gradient difference. Classify the pixel points into the nearest initial clustering centers according to the minimum average gradient difference to form classes. Calculate the average gradient values of the classes again to obtain new average gradient values. Obtain new clustering centers according to the new average gradient values, and reclassify according to the new clustering centers until the clustering centers no longer change, then stop the classification to obtain the final classes. Find the minimum average gradient value in the final classes, and the minimum average gradient value is the adaptive threshold.

4. The foreign object detection method for pharmaceutical raw materials based on production line images according to claim 3, wherein, The S2 includes the following steps: S21. Select the current frame pharmaceutical raw material production line image and the previous frame pharmaceutical raw material production line image from the processed pharmaceutical raw material production line image set. Collect k nearest neighbor pixel points on the current frame pharmaceutical raw material production line image, calculate the norm difference between the nearest neighbor pixel points and the previous frame pharmaceutical raw material production line image, establish a pixel point probability distribution model, and calculate the background probability. Set the foreground threshold and the background threshold, and compare with the background threshold to achieve the segmentation of the background and the foreground, obtain the segmented pharmaceutical raw material production line image, and form the segmented pharmaceutical raw material production line image set. S22. Perform opening operation and closing operation on the segmented pharmaceutical raw material production line image set to obtain the processed pharmaceutical raw material production line image set, and perform hole filling on the processed pharmaceutical raw material production line image set to obtain the final pharmaceutical raw material production line image set.

5. A method for detecting foreign objects in pharmaceutical raw materials based on production line images according to claim 4, characterized in that, The hole filling of the processed pharmaceutical raw material production line image set includes the following steps: S221. Select seed pixel points in the foreground image area of the processed pharmaceutical raw material production line image, set the gray values of other pixel points with the same gray value as the seed pixel points to 255 to obtain the preliminarily extracted pharmaceutical raw material production line image. S222. After inverting the gray values of the pixel points in the preliminarily extracted pharmaceutical raw material production line image, perform bitwise AND operation with the preliminarily extracted pharmaceutical raw material production line image to complete the hole filling, obtain the final pharmaceutical raw material production line image, and form the final pharmaceutical raw material production line image set.

6. The foreign object detection method for pharmaceutical raw materials based on production line images according to claim 5, characterized in that, The S4 includes the following steps: S41. Obtain the improved Mask R-CNN network model according to the global optimal solution of the random gradient descent optimizer iteration formula. After performing annotation, brightness transformation, rotation and noise addition processing on the final pharmaceutical raw material production line image set, input it into the improved Mask R-CNN network model to output the recognition result. S42. Obtain the types of foreign matters in the pharmaceutical raw materials according to the recognition result to complete the detection of foreign matters in the pharmaceutical raw materials.

7. A system for implementing the pharmaceutical raw material foreign object detection method based on production line images as described in any one of claims 1-6, characterized in that, Specifically include: Image edge extraction module, image background and foreground segmentation and filling module, neural network parameter optimization module and foreign matter detection module for pharmaceutical raw materials; The image edge extraction module is used to perform edge extraction on the initial pharmaceutical raw material production line image using the improved sobel operator edge detection algorithm. The image background and foreground segmentation and filling module is used to segment the background and foreground of the processed pharmaceutical raw material production line image based on the KNN background difference method, and then perform hole filling; The neural network parameter optimization module is used to optimize the initial learning rate using the termite life cycle optimization algorithm; The foreign object detection module for pharmaceutical raw materials is used to output the recognition result using the improved Mask R-CNN network model.

Images