Coating film thickness detection method and system based on visual imaging and deep learning

Abstract

The present application relates to the technical field of quality control, and provides a coating film thickness detection method and system based on visual imaging and deep learning.The method comprises the following steps: acquiring visual images of the whole process of fluidized bed coating, marking target particles in the visual images, and outlining the edges of the target particles; training a Mask R-CNN model based on the marked visual images; detecting real-time visual images by using the trained Mask R-CNN model to obtain target particle images; processing the target particle images to obtain black-and-white rendering images of the target particles; calculating the sizes of the pellet particles based on the black-and-white rendering images, calculating the difference between the size of the pellet particles and the size of the blank particles, and obtaining the coating film thickness of the pellets.The present application can accurately calculate the coating film thickness of the pellets, and provides a stable and efficient solution for online monitoring of the coating film thickness of the particles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of quality control, and particularly relates to a coating film thickness detection method and system based on visual imaging and deep learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Coating of oral solid dosage forms such as tablets and granules is a critical step in pharmaceutical manufacturing, usually performed in coating pans or fluidized bed equipment. The main purposes of film coating include controlling the site and rate of drug release (for example, in the case of enteric coating, the drug is insoluble in the stomach but dissolves in the intestine), improving the appearance of the drug, masking the bitter taste and other unpleasant tastes, and improving the stability of the drug by providing a barrier against environmental and physiological factors that may degrade the active pharmaceutical ingredient. Therefore, the film thickness is a key quality attribute that significantly affects the performance of the coating. Even a slight change in film thickness will affect the quality, solubility and stability of the drug, which highlights the necessity of real-time monitoring during the coating process.

[0004] Common process analytical techniques (PAT) for measuring film thickness or particle size are generally divided into direct and indirect methods. Direct methods include optical coherence tomography (OCT), visual imaging, spatial filter velocimetry (SFV) and focused beam reflectance measurement (FBRM). Visual imaging is a promising tool with great potential in particle size detection, especially when combined with image analysis algorithms in pharmaceutical manufacturing. Compared with other methods, visual imaging can measure and analyze the shape factor and size distribution of particles, providing a more intuitive measurement method. Traditional image segmentation methods include threshold processing and edge segmentation. Threshold processing distinguishes, splits and merges corresponding regions according to pixel intensity. Edge segmentation identifies boundaries by detecting discontinuities in pixel intensity.

[0005] However, fluidized bed coating is a dynamic, multiphase process. Traditional imaging techniques have a certain delay in real-time dynamic monitoring, especially in the case of high-frequency data acquisition and real-time processing, how to ensure the imaging speed and accuracy at the same time is still a challenge. In addition, even if the imaging technology can capture high-definition images, the subsequent image processing and data analysis are still very complex. Due to the dynamic nature of the fluidized bed environment, the shape, size and motion state of the particles change frequently, making it difficult for automated image processing algorithms to cope with diversity and uncertainty. Therefore, it is necessary to combine deep learning algorithms to more accurately and effectively analyze complex background pictures. SUMMARY

[0006] To solve the technical problems in the background art, the present application provides a coating film thickness detection method and system based on visual imaging and deep learning, which adopts a Mask Region-based Convolutional Neural Network (Mask R-CNN) to online segment visual imaging pictures, so as to accurately calculate the micro-pellet coating film thickness, and provides a stable and efficient solution for online monitoring of the particle coating film thickness.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a coating film thickness detection method based on visual imaging and deep learning.

[0009] A coating film thickness detection method based on visual imaging and deep learning, comprising:

[0010] Obtaining visual images of the whole process of fluidized bed coating, marking target particles in the visual images, and outlining the edges of the target particles;

[0011] Training a Mask R-CNN model based on the marked visual images;

[0012] Detecting real-time visual images by using the trained Mask R-CNN model to obtain target particle images, processing the target particle images to obtain black-and-white rendering images of the target particles;

[0013] Based on the black-and-white rendering images, calculating the size of the micro-pellet particles, calculating the difference between the size of the micro-pellet particles and the size of the blank particles, and obtaining the micro-pellet coating film thickness.

[0014] Further, in the model training process, the following steps are included: obtaining samples corresponding to the time points of the visual images, offline measuring the micro-pellet coating film thickness; based on the offline measured micro-pellet coating film thickness and the calculated micro-pellet coating film thickness in the model training process, calculating a loss function, and training the hyperparameters of the Mask R-CNN model.

[0015] Further, the offline measurement of the micro-pellet coating film thickness includes: subtracting the equivalent diameter size of the blank particles from the equivalent diameter size of the particles measured by the Malvern laser method to obtain the micro-pellet coating film thickness.

[0016] Further, the offline measurement of the micro-pellet coating film thickness includes: calculating the micro-pellet coating film thickness by using the ultraviolet spectrophotometer method; and using the following formula to describe:

[0017]

[0018] In the formula, C API is the API concentration of each group; W pellet is the average weight of each pellet; PAPI API proportion in the coating agent; p coating coating solid content density; d is the average particle size of the blank pellets.

[0019] Further, the difference between the pellet particle size and the blank particle size is calculated to obtain the pellet coating film thickness; the following formula is used to express:

[0020]

[0021] ACT jb indicates the average coating film thickness of the photographed particle image at the jth time point, indicates the equivalent average diameter of the particles in the picture at the jth time point, indicates the equivalent average diameter of the particles in the photographed blank pellet picture.

[0022] Further, the pellet particle size is calculated based on the black and white rendering picture; the method comprises: based on the black and white rendering picture, using the cv.contourArea() function to calculate the pellet particle size.

[0023] The second aspect of the present application provides a coating film thickness detection system based on visual imaging and deep learning.

[0024] A coating film thickness detection system based on visual imaging and deep learning, comprising:

[0025] A data acquisition and processing module configured to acquire visual images of the whole process of fluidized bed coating, mark target particles in the visual images, and outline the edges of the target particles;

[0026] A model training module configured to train a Mask R-CNN model based on the marked visual images;

[0027] A detection module configured to detect real-time visual images using the trained Mask R-CNN model to obtain target particle images, process the target particle images to obtain black and white rendering pictures of the target particles;

[0028] An output module configured to calculate the pellet particle size based on the black and white rendering picture, calculate the difference between the pellet particle size and the blank particle size, and obtain the pellet coating film thickness.

[0029] The third aspect of the present application provides a computer readable storage medium.

[0030] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the coating film thickness detection method based on visual imaging and deep learning of the first aspect described above.

[0031] A fourth aspect of the present application provides a computer device.

[0032] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the coating film thickness detection method based on visual imaging and deep learning according to the first aspect above when executing the program.

[0033] A fifth aspect of the present application provides a computer program product or a computer program.

[0034] The present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the steps in the coating film thickness detection method based on visual imaging and deep learning according to the first aspect above.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The present application provides a coating film thickness detection method and system based on visual imaging and deep learning, which adopts a Mask R-CNN (Mask Region-based Convolutional Neural Network) to online segment a visual imaging picture, so as to accurately calculate the coating film thickness of a micro-pellet, and provides a stable and efficient solution for online monitoring of the coating film thickness of a particle. The present application can accurately predict the coating film thickness in a complex background, greatly improves the prediction accuracy of the model, and provides technical support for subsequent quality control of a fluidized bed coating, thereby improving the safety and effectiveness of a drug.

[0037] The present application can realize real-time and accurate online monitoring of the coating film thickness change in a fluidized bed coating process by combining online imaging technology with a deep learning algorithm. The establishment of the technical solution of the present application is helpful for timely monitoring of the dynamic behavior of the coating process in the fluidized bed coating process, ensures the consistency of the quality of the coating process, and thus improves the safety and effectiveness of a drug, so the present application has good practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0038] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0039] Figure 1 is a flowchart of the coating film thickness detection method based on visual imaging and deep learning shown by the present application;

[0040] Figure 2 Figure 1 is a diagram showing the fluidized bed coating experimental apparatus of the present application; (a) is a NIR sensor, (b) is an imaging probe, (c) is a NIR sensor control interface, (d) is an imaging probe control interface;

[0041] Figure 3 Figure 2 is a diagram showing the particle number in the first batch, (a) is the particle number in the first batch, (b) is the particle number in the second batch, (c) is the particle number in the third batch;

[0042] Figure 4 Figure 3 is a diagram showing the polygonal marker particles composed of many points of the present application; (a) is a clear particle, (b) is an agglomerated particle, (c) is an incomplete particle at the image boundary, (d) is powder sticking to the window, (e) is a defocused or blurred particle;

[0043] Figure 5 Figure 4 is a diagram showing the total loss as a function of the iteration step of the present application;

[0044] Figure 6 Figure 5 is a diagram showing the accuracy evaluation results of the Mask R-CNN model, (a) is the accuracy evaluation results of the Mask R-CNN model, (b) is the recall evaluation results of the Mask R-CNN model, (c) is the accuracy evaluation results of the Mask R-CNN model, (d) is the F1 score evaluation results of the Mask R-CNN model;

[0045] Figure 7 Figure 6 is a diagram showing the picture segmentation of the present application; (a) is the original image, (b) is the boundary map predicted using the Mask R-CNN model, (c) is the black and white background rendering map;

[0046] Figure 8 Figure 7 is a diagram showing the film thickness comparison of visual imaging, laser diffraction and ultraviolet-visible spectrophotometer of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described with reference to the drawings and examples.

[0048] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0049] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0050] It should be noted that the flow diagrams and block diagrams in the drawings are representative of the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of the present disclosure. It should also be noted that each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of code, which includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and

[0051] As mentioned before, common process analytical techniques (PAT) for measuring film thickness or particle size include optical coherence tomography (OCT), visual imaging, spatial filter velocimetry (SFV) and focused beam reflectance measurement (FBRM). Visual imaging is a promising tool with great potential in particle size detection, especially when combined with image analysis algorithms in pharmaceutical manufacturing. Compared with other methods, visual imaging can measure and analyze the shape factor and size distribution of particles, providing a more intuitive measurement method. In addition, when combined with Mask R-CNN, a deep learning method, it can accurately and effectively analyze the changes in the thickness of the coating film of particles under complex background pictures.

[0052] In view of this, the present application provides a coating film thickness detection method and system based on visual imaging and deep learning, which uses a Mask Region-based Convolutional Neural Network (Mask R-CNN) to online segment visual imaging pictures, so as to accurately calculate the coating film thickness of micro-pellets. The present application will be described in detail through several embodiments as follows:

[0053] Embodiment One

[0054] As Figure 1As shown, the embodiment provides a coating film thickness detection method based on visual imaging and deep learning,

[0055] Step (1): Collecting visual imaging picture data of the whole fluidized bed coating process;

[0056] Step (2): Collecting samples at different time points in the fluidized bed coating process (corresponding to the imaging picture time points) and measuring the pellet coating film thickness offline;

[0057] Step (3): Using a marking tool to mark the target particles in the picture, creating a polygon to outline the edges of the target particles, and effectively describing various object shapes;

[0058] Step (4): The labeled file after marking, which contains all the particle information, is used for subsequent training of the Mask R-CNN model;

[0059] Step (5): The performance of the model is judged by accuracy, precision, recall and F1 value indicators;

[0060] Step (6): The optimized Mask R-CNN model is used for unknown sample picture particle segmentation to generate clear particle boundaries, and then the black and white rendering picture with target particles is generated by removing other backgrounds;

[0061] Step (7): The particle size is calculated from the black and white rendering picture by using the cv.contourArea() function, and the size of each particle in the picture corresponding to each time point is calculated in turn, and then the blank particle size is subtracted to obtain the growth of the coating film thickness;

[0062] Step (8): Compared with the offline Malven laser method and ultraviolet spectrophotometry, the accuracy of the technology is verified.

[0063] In another specific embodiment of the application, the step (1) of collecting visual imaging picture data of the whole fluidized bed coating process comprises:

[0064] During the fluidized bed coating process, the online two-dimensional visual probe is installed on the outer wall of the fluidized bed window, and the camera shooting frequency is controlled by the imaging system, which can be adjusted to a maximum of 10 frames per second. In the present application, the sampling frequency is one frame per 10 seconds, and the image is displayed and recorded on the computer.

[0065] In another specific embodiment of the application, step (2) collects samples at different time points in the fluidized bed coating process (corresponding to the imaging picture time points) and measures the particle coating film thickness offline:

[0066] Wherein, particle size measurement uses Mastersizer 3000 (Malvern Panalytical, UK Malvern) equipped with Aero S dry powder dispersion device to analyze the particle size of the particles. Wherein, Mastersizer 3000 uses laser diffraction to measure the scattering mode of the particles. The software analyzes the scattering data to determine the particle size distribution, and three measurements are taken to ensure accuracy and repeatability. Both the visual imaging and the laser diffraction measured film thickness can be obtained by subtracting the blank particle equivalent diameter size from the particle equivalent diameter size at different time points, as shown in formulas (8)-(9).

[0067] The coating film thickness is measured using the ultraviolet spectrophotometer method as follows:

[0068] First, a certain weight of blank pellets is weighed and placed under a microscope to ensure proper dispersion. The number of pellets is calculated to calculate the average weight of each pellet, and this process is repeated three times to obtain an average value. Second, a certain weight of pellets is randomly weighed from each pellet sample collected during the coating process. Each group of pellets is mixed with pure water, and after ultrasonic treatment, it is ensured that the API is completely dissolved in the aqueous solution. Then the solution is filtered through a 0.45 μm filter, and the corresponding absorbance is measured using a UV-visible spectrophotometer. Before sample analysis, a calibration curve is established to determine the API concentration in the solution. The data related to each pellet core is obtained by understanding the following information: 1) coating ingredient ratio; 2) coating density (measured using an automatic true density analyzer, Biodiesel, BSD-TD); 3) average diameter of blank pellets. Therefore, the average film thickness (δ thickness,UV ) of each particle in the sample is calculated as shown in formula (1):

[0069]

[0070] In the formula, C API is the API concentration (w / w) of each group; W pellet is the average weight of each pellet; P API is the proportion of API in the coating agent; p coating is the coating solid content density; d is the average particle size of the blank pellets.

[0071] In another specific embodiment of the present application, step (3) uses a marking tool to mark the target particles in the picture, creates a polygon to outline the edges of the target particles, and effectively describes various object shapes;

[0072] Using VIA 2.0 (Visual-Geometry-Group, 2021) labeling tool, polygons were created to outline the edges of the target particles, effectively describing various object shapes. This involved manually placing discrete points along the curved edges of each particle, with lines automatically connecting adjacent points to form polygons. The number of points needed to accurately label each particle depended on its size, with larger particles typically requiring 20-30 points to describe their shape with polygons. Upon completion of labeling, a label file was generated containing the positional information of all particles, sorted by image file name, for use in subsequent training of the Mask R-CNN model. Note that, in order to reflect the true size of the particles as much as possible, the clumped granules, incomplete granules at the image boundary, powders sticking to the window, out-of-focus and blurred granules, etc. contained in the picture were not labeled, which would affect the calculation of the size of the reagent particles.

[0073] In another embodiment of the present application, step (4) uses the label file containing information of all particles after labeling for subsequent training of the Mask R-CNN model.

[0074] The Mask R-CNN model includes a backbone network, a region proposal network, a region of interest alignment layer, and a region convolutional neural network. The backbone network is used to extract feature maps, and the backbone network includes a residual network and a feature pyramid. Based on the feature maps, the region proposal network is used to obtain the region of interest and propose the candidate object bounding box. Based on the region of interest and the feature maps, the region of interest alignment layer is used for matching, completing feature aggregation of the feature maps, and pooling into a fixed size, and outputting to the region convolutional neural network through a fully connected layer. The region convolutional neural network includes a first branch, a second branch, and a third branch. The first branch realizes classification of the particles through a softmax classifier. The second branch realizes more accurate target positioning through a bounding box regressor. The third branch completes contour segmentation of the target particles through a fully convolutional network and generates a mask. Finally, the output information of each branch is integrated to obtain a target particle image containing the category, positioning bounding box, and segmentation mask, realizing accurate positioning and recognition of the target particles.

[0075] The configuration settings during model training are as follows. The number of steps per round is set to 100, and the number of validation steps is set to 30. The confidence level for detection is set to 0.7, which means that the Mask R-CNN model will only detect an object instance if the confidence level that the object instance exists is more than 70%. The learning rate and momentum of the gradient descent algorithm are set to 0.001 and 0.9, respectively, to ensure the stability of weight optimization and faster convergence behavior. The loss function can be represented as:

[0076] L total =L cls +Lbox +L mask (2)

[0077] where L cls , L box and L mask are the classification loss, the bounding box regression loss and the mask loss, respectively.

[0078] In yet another specific embodiment of the present application, step (5) evaluates the performance of the model by accuracy, precision, recall and F1 value indicators.

[0079] The accuracy, precision, recall and F1 value indicator formulas are as follows:

[0080]

[0081] where TP represents the number of samples correctly identified as positive; FN represents the number of samples incorrectly identified as negative; FP represents the number of samples incorrectly identified as positive; and TN represents the number of samples correctly identified as negative.

[0082] In yet another specific embodiment of the present application, step (6) uses the optimized model for unknown sample image particle detection to obtain a target particle image, and then removes other backgrounds to generate a black and white rendering image with target particles.

[0083] In yet another specific embodiment of the present application, step (7) uses the cv.contourArea() function to calculate the particle size from the black and white rendering image, and sequentially calculates the size of the particles in the image at each time point, and then subtracts the size of the blank particles to obtain the growth of the coating film thickness.

[0084] For the estimation of each particle diameter, the equivalent spherical diameter (ESD) is used. ED is calculated from the projected area (Area) of the particle:

[0085]

[0086] The equivalent average diameter of the particles in the image can be further calculated by the following formula:

[0087]

[0088] where ESD i represents the diameter of the i-th particle; and N is the total number of particles in the image.

[0089] The average coating thickness (ACT) of the particles in the picture at each time point can be further calculated by the following formula:

[0090]

[0091] wherein ACT jb represents the average coating film thickness of the particle image at the jth time point, represents the equivalent average diameter of the particles in the picture at the jth time point, represents the equivalent average diameter of the particles in the picture of the blank pellets.

[0092] In another specific embodiment of the present application, step (8) is compared with the offline Malvern laser method and the ultraviolet spectrophotometry method to verify the accuracy of the technology.

[0093] Example Two

[0094] The coating film thickness detection method based on visual imaging and deep learning described in the present embodiment comprises the following steps:

[0095] Step (1): Collect visual imaging picture data of the whole process of fluidized bed coating.

[0096] Before starting each coating batch, preheat the fluidized bed for half an hour to make the fluidized bed in a relatively stable state for granulation. After preheating, put in the material, mix for 10 minutes, and then start spraying the coating liquid.

[0097] During the fluidized bed coating process, install the online two-dimensional visual probe on the outer wall of the fluidized bed window, about 2 mm away from the window (see Figure 2 d). The camera parameter settings are as follows: the light source brightness is set to 800%, the current size is set to 0.6 A, and the pulse width is set to 0.02 ms. The camera shooting frequency is controlled by the imaging system, which can be adjusted up to 10 frames per second. In this study, the sampling frequency is one frame per 10 seconds, and the image is displayed and recorded on the computer. As shown in Figure 2 b, the computer interface displays the acquired particle image in real time.

[0098] Step (2): Collect samples at different time points during the fluidized bed coating process (corresponding to the imaging picture time points) to measure the coating film thickness of the particles offline.

[0099] Where particle size measurement uses Mastersizer 3000 (Malvern Panalytical, UK Malvern) equipped with an Aero S dry powder dispersion unit to analyze the particle size of the granules. About 10 grams of sample is loaded into the sample hopper and compressed air at 1 bar is used to disperse the particles into individual particles. Before introducing the sample, a background measurement is taken using compressed air only to address any background noise or contamination issues. The dispersed sample is then introduced into the laser beam path, where the Mastersizer 3000 measures the scattering pattern of the particles using laser diffraction. The software analyzes the scattering data to determine the particle size distribution, and three measurements are taken to ensure accuracy and reproducibility. Both visual imaging and laser diffraction measured film thickness can be calculated according to the particle equivalent diameter size at different time points minus the blank particle equivalent diameter size, as shown in Equations (8)-(9).

[0100] The coating film thickness is measured using the UV spectrophotometer method as follows:

[0101] First, a certain weight of blank pellets is weighed and placed under a microscope (see Figure 3 ) to ensure proper dispersion. The number of pellets is calculated to calculate the average weight of each pellet, and this process is repeated three times to obtain an average value. Second, a certain weight of pellets is randomly weighed from each sample of pellets collected from the coating process. Each group of pellets is mixed with pure water and sonicated for 20 minutes to completely dissolve the API in the aqueous solution. The solution is then filtered through a 0.45 pm filter, and the absorbance is measured at a maximum wavelength of 233 nm (lmax) using a UV-visible spectrophotometer (EU-2800D, Nirobiotech, China). Before sample analysis, a calibration curve is established to determine the API concentration in the solution. Third, data related to each pellet is obtained by understanding the following information: (i) coating ingredient ratio (ii) coating density (measured using an automatic true density analyzer, Beishide, BSD-TD) (iii) average diameter of blank pellets. Therefore, the average film thickness (d thickness,UV ) of each pellet in the sample is calculated by Equation (1).

[0102] Step (3): Using the marking tool, mark the target particles in the picture, create a polygon to outline the edges of the target granules, effectively describe various object shapes, as shown in Figure 4 .

[0103] The VIA 2.0 (Visual-Geometry-Group, 2021) labeling tool was used to create polygons to outline the edges of the target particles, effectively describing the shapes of various objects. This involved manually placing discrete points along the curved edges of each particle, with lines automatically connecting adjacent points to form polygons. The number of points required to accurately label each particle depended on its size, with larger particles typically requiring 20-30 points to describe their shape with polygons. Upon completion of labeling, a label file was generated containing the positional information of all particles, sorted by image file name, for use in subsequent training of the Mask R-CNN model. Note that, in order to reflect the true size of the particles as much as possible, clumped granules, incomplete granules at the image boundary, powders sticking to the window, out-of-focus and blurred granules, etc. contained in the picture were not labeled, which would affect the calculation of the size of the reagent particles.

[0104] Step (4): The labeled file containing all particle information after labeling was used for subsequent training of the Mask R-CNN model.

[0105] A total of 140 images were labeled, with 112 and 28 images in the training and validation sets, respectively. The configuration settings during model training were as follows. The number of steps per round was set to 100, and the number of validation steps was set to 30. The confidence level for detection was set to 0.7, which means that the Mask R-CNN model will only detect an object instance if the confidence level of its existence exceeds 70%. The learning rate and momentum of the gradient descent algorithm were set to 0.001 and 0.9, respectively, to ensure the stability of weight optimization and faster convergence behavior. Figure 5 The total loss of the Mask R-CNN model (containing 140 images) was shown to change with the training steps. Obviously, the loss decreased with the increase of the number of steps and tended to be stable after 100 steps. Therefore, the model weight after 100 iterations was used for new particle segmentation images in this study.

[0106] Step (5): The performance of the model was evaluated by accuracy, precision, recall, and F1 value. The accuracy, precision, recall, and F1 value were calculated using formulas (3)-(6), respectively. The test set results are shown in Table 2. Figure 6

[0107] The test set consisted of 112 images selected from the coating process of 14 batches (8 images per batch) in chronological order. The average number of particles per image was about 20. The accuracy, recall, precision, and F1 score of the Mask R-CNN algorithm were further evaluated on the test set. Figure 6 ​The evaluation results of the Mask R-CNN algorithm are shown, the accuracy is between 75% and 100%, the average is 97.9%, the recall rate is between 75% and 100%, the average is 98.0%, the accuracy is between 90.1% and 100%, the average is 98.4%, and the F1 score is between 85.7% and 100%, the average is 97.9%. The above results show that the performance of the Mask R-CNN model on the validation set is also excellent.

[0108] Step (6): The optimized model is used for unknown sample picture particle detection to obtain a target particle image, as shown in Figure 7 (b), and then a black and white rendering of the target particles is generated by removing other backgrounds, as shown in Figure 7 (c).

[0109] Step (7): The particle size is calculated from the black and white rendering image using the cv.contourArea() function, and the particle size in the image corresponding to each time point is calculated in turn, and then the blank particle size is subtracted to obtain the growth of the coating film thickness, as shown in formulas (7)-(9).

[0110] Step (8): The accuracy of the technology is verified by comparing with offline Malvern and ultraviolet spectrophotometry.

[0111] Figure 8 The comparison results of three methods (imaging, laser diffraction and ultraviolet visible spectrophotometer) for measuring film thickness are given. For the collected particle samples, the film thickness results measured by online visual imaging are consistent with the results measured by offline laser diffraction and ultraviolet visible spectrophotometer. Although the coating film thickness values measured by different methods are slightly different, the trend is consistent, proving that the imaging technology can be used for online monitoring of film thickness in the coating process.

[0112] Example Three

[0113] The embodiment provides a coating film thickness detection system based on visual imaging and deep learning.

[0114] A coating film thickness detection system based on visual imaging and deep learning comprises:

[0115] The data acquisition and processing module is configured to acquire visual images of the whole process of fluidized bed coating, and mark target particles in the visual images to outline edges of the target particles.

[0116] The model training module is configured to train a Mask R-CNN model based on the marked visual images.

[0117] The detection module is configured to: detect the real-time visual image by using the trained Mask R-CNN model to obtain a target particle image, and process the target particle image to obtain a black-and-white rendering image of the target particle.

[0118] The output module is configured to: calculate the size of the pellet particle based on the black-and-white rendering image, calculate the difference between the size of the pellet particle and the size of the blank particle, and obtain the coating film thickness of the pellet.

[0119] In some embodiments, the model training module is further configured to, in the model training process, include: obtaining a sample corresponding to a time point of a visual image, and measuring the coating film thickness of the pellet offline; calculating a loss function based on the coating film thickness of the pellet measured offline and the coating film thickness of the pellet calculated in the model training process, and training the hyperparameters of the Mask R-CNN model.

[0120] In some embodiments, the model training module is further configured to: subtract the equivalent diameter size of the blank particle from the equivalent diameter size of the particle measured by the Malvern laser method to obtain the coating film thickness of the pellet.

[0121] In some embodiments, the model training module is further configured to: calculate the coating film thickness of the pellet by using an ultraviolet spectrophotometer method; and describe the following formula:

[0122]

[0123] In the formula, C API is the API concentration of each group; W pellet is the average weight of each pellet; P API is the proportion of API in the coating agent; p coating is the density of the coating solid content; and d is the average particle size of the blank pellet.

[0124] In some embodiments, the output module is further configured to: calculate the coating film thickness of the pellet by using the following formula:

[0125]

[0126] In the formula, ACT jb represents the average coating film thickness of the photographed particle image at the jth time point, represents the equivalent average diameter of the particle in the picture at the jth time point, represents the equivalent average diameter of the particle in the photographed blank pellet picture.

[0127] In some embodiments, the output module is further configured to: calculate the size of the pellet particle by using a cv.contourArea() function based on the black-and-white rendering image.

[0128] Embodiment Four

[0129] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement steps in the coating film thickness detection method based on visual imaging and deep learning according to the first embodiment or the second embodiment.

[0130] Embodiment five

[0131] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements steps in the coating film thickness detection method based on visual imaging and deep learning according to the first embodiment or the second embodiment when executing the program.

[0132] Embodiment six

[0133] The embodiment provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes steps in the coating film thickness detection method based on visual imaging and deep learning according to the first embodiment or the second embodiment.

[0134] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.

[0135] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the function specified in one block or multiple blocks.

[0136] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0138] Those of ordinary skill in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0139] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those of ordinary skill in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting coating film thickness based on visual imaging and deep learning, characterized in that, include: Obtain visual images of the entire fluidized bed coating process, and mark and delineate the edges of the target particles in the visual images; The Mask R-CNN model is trained based on labeled visual images; The trained Mask R-CNN model is used to detect real-time visual images to obtain target particle images. The target particle images are then processed to obtain black and white rendering images of the target particles. Based on the black and white rendering image, the size of the micro-pellets is calculated, and the difference between the size of the micro-pellets and the size of the blank particles is calculated to obtain the thickness of the micro-pellet coating film. The model training process includes: acquiring samples at corresponding time points of visual images and measuring the thickness of the micro-pellet coating film offline; calculating the loss function and training the hyperparameters of the Mask R-CNN model based on the offline measured thickness of the micro-pellet coating film and the thickness of the micro-pellet coating film calculated during model training. The offline measurement of the microparticle coating thickness includes: calculating the microparticle coating thickness using an ultraviolet spectrophotometer; described by the following formula: In the formula, API concentration for each group; This represents the average weight of each pill; This refers to the proportion of API in the coating agent; The density of the coating solids content; The average particle size of the blank pellet is denoted as .

2. The coating film thickness detection method based on visual imaging and deep learning according to claim 1, characterized in that, The difference between the size of the micro-pellets and the size of the blank particles is calculated to obtain the thickness of the micro-pellet coating film; The following formula is used to express this: in, Indicates the first The average coating thickness of the particle images taken at each time point express The equivalent average diameter of particles in the image at the specified time point. This represents the equivalent average diameter of the particles in a blank microsphere image.

3. The coating film thickness detection method based on visual imaging and deep learning according to claim 1, characterized in that, The method for calculating the microparticle size based on a black-and-white rendering image includes: using the cv.contourArea() function to calculate the microparticle size based on the black-and-white rendering image.

4. A coating film thickness detection system based on visual imaging and deep learning, characterized in that, include: The data acquisition and processing module is configured to: acquire visual images of the entire fluidized bed coating process, and mark and delineate the edges of the target particles in the visual images; The model training module is configured to train a Mask R-CNN model based on labeled visual images. The detection module is configured to: use the trained Mask R-CNN model to detect real-time visual images to obtain target particle images, process the target particle images to obtain black and white rendering images of the target particles; The output module is configured to: calculate the size of the micro-particles based on the black and white rendering image, calculate the difference between the size of the micro-particles and the size of the blank particles, and obtain the thickness of the micro-particle coating film. The model training process includes: acquiring samples at corresponding time points of visual images and measuring the thickness of the micro-pellet coating film offline; calculating the loss function and training the hyperparameters of the Mask R-CNN model based on the offline measured thickness of the micro-pellet coating film and the thickness of the micro-pellet coating film calculated during model training. The offline measurement of the microparticle coating thickness includes: calculating the microparticle coating thickness using an ultraviolet spectrophotometer; described by the following formula: In the formula, API concentration for each group; This represents the average weight of each pill; This refers to the proportion of API in the coating agent; The density of the coating solids content; The average particle size of the blank pellet is denoted as .

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the coating film thickness detection method based on visual imaging and deep learning as described in any one of claims 1-3.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the coating film thickness detection method based on visual imaging and deep learning as described in any one of claims 1-3.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the coating film thickness detection method based on visual imaging and deep learning as described in any one of claims 1-3.

Images