Liquid medicine filling online intelligent detection system of pharmaceutical production line based on visual detection

Abstract

The application discloses a liquid medicine filling online intelligent detection system of a pharmaceutical production line based on visual detection and relates to the technical field of filling detection.The system comprises a polarization extinction module, which comprises a multifocal plane camera unit and a light source unit, and a polarization plate is additionally arranged based on the multifocal plane camera unit and the light source unit, so that the mirror reflection of a bottle body and a liquid surface is eliminated, the reflection and impurities are prevented from being confused, and a polarization installation item is obtained.The application effectively solves the reflection interference and motion blur problem from the physical layer through the design of the polarization extinction module and a time-sharing frequency flash module, fundamentally eliminates the interference of the mirror reflection of a transparent / amber glass, a PET bottle body and a liquid surface on imaging, and meanwhile, the time-sharing frequency flash module adopts a combination of a bottom blue light cold backlight, a side red light annular dark field light source and a top white light bright field surface light source, so that clear and low-noise original image data is provided for subsequent image analysis, and a physical foundation for high-precision impurity detection is laid.

Description

Technical Field

[0001] This invention relates to the field of filling inspection technology, specifically to an online intelligent inspection system for liquid drug filling in pharmaceutical production lines based on vision inspection. Background Technology

[0002] Liquid drug filling inspection refers to a series of quality control and monitoring measures implemented during the filling operation of liquid drugs such as injections and oral liquids in the pharmaceutical production process. Its core purpose is to ensure the accuracy, safety, and compliance of the filling process and to prevent problems such as dosage errors, contamination, or leakage. Common inspection items and methods include filling volume inspection, which uses gravimetric or volumetric methods, such as weighing sensors or flow meters, to ensure the accuracy of the liquid volume per unit container; or container integrity inspection, which checks the sealing performance, such as leakage testing, using vacuum or pressure methods; and visual defects, such as visual inspection systems to identify cracks or contamination, to ensure drug quality and patient safety.

[0003] The method, apparatus, and storage medium for detecting foreign objects inside glass bottles, disclosed in patent publication number CN119880962A, employs X-ray sources with different voltages at the same viewing angle. This ensures that all single-channel X-ray images have the same viewing angle, reducing registration difficulty and preventing new errors introduced by scaling and fusion. Furthermore, the use of different voltages effectively covers imaging requirements for regions with varying densities. The output image, obtained through high-low frequency separation, high-frequency enhancement, and subsequent stitching and convolution, clearly displays foreign objects suspended in the glass bottle. This improves detection accuracy and reliability while allowing the foreign object detection process to be placed close to the filling process, eliminating the need for post-filling settling and significantly increasing the overall production line efficiency.

[0004] When conducting drug filling inspections, the aforementioned and similar technical solutions generally lack reflection suppression and defect layering design. Furthermore, the movement of the filling line in the production line causes significant motion blur, which exacerbates the confusion between bottle mirror reflections, outer wall scratches, water mist, and stains and liquid impurities introduced during the filling process. The detection rate of micron-level filling impurities such as glass fragments, fibers, and white spots is insufficient, resulting in a high false detection rate. This leads to the wrong rejection of a large number of qualified products, while the risk of unqualified products entering the market is relatively high. Summary of the Invention

[0005] The purpose of this invention is to provide an online intelligent inspection system for liquid drug filling in pharmaceutical production lines based on visual inspection, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an online intelligent detection system for liquid drug filling in a pharmaceutical production line based on visual inspection, comprising: Polarization extinction module: includes a multi-focal plane camera unit and a light source unit. Polarizers are installed on the multi-focal plane camera unit and the light source unit respectively to eliminate specular reflections on the bottle body and liquid surface, and avoid confusion between reflections and impurities, thus obtaining a polarization installation item; Time-division stroboscopic module: The light source unit includes a bottom cold light source, a side ring dark field light source and a top bright field light source. The target filling bottle is irradiated by time-division stroboscopic irradiation based on the light source unit, and then irradiation data items are obtained. Multi-focal plane acquisition module: The multi-focal plane camera unit acquires image data of the target filling bottle based on the illumination data items to obtain the acquisition dataset; The edge reasoning module includes a focal plane semantic segmentation unit, a small target defect detection unit, and a filling volume sub-pixel calculation unit. The focal plane semantic segmentation unit segments the front wall area of ​​the bottle, the liquid area, the back wall area of ​​the bottle, and the liquid surface area. Interference is eliminated by matching the position of the focal plane where the defect is located, and a focal plane analysis item is obtained. The small target defect detection unit distinguishes impurities and obtains an impurity analysis item. The filling volume sub-pixel calculation unit obtains the filling volume information and obtains a filling analysis item. Finally, the focal plane analysis item, the impurity analysis item, and the filling analysis item are combined to output the detection result and obtain a comprehensive analysis item.

[0007] Furthermore, the method for obtaining the polarization mounting item includes: Set a first installation spacing, obtain the light source outlet position of the light source unit, obtain the first installation point, install the linear polarizer with the first installation spacing as the reference, set the wavelength coverage range, and obtain the first installation item; The position of the multi-focal plane camera unit is obtained to get the second mounting point. The mounting angle is obtained by taking the polarization direction of the light source unit as orthogonal. The mounting adjustment item is obtained. The polarizer is installed based on the mounting adjustment item. The wavelength coverage range is set to obtain the second mounting item. The first mounting item and the second mounting item are combined to obtain the polarization mounting item.

[0008] Furthermore, the method for obtaining the irradiation data items includes: Based on the setting of unit parameters for each light source unit, light source parameter items are obtained, and each light source parameter item is matched with a light source unit. Based on the light source parameters, time intervals are set, and time-division stroboscopic illumination is performed according to the time intervals. A cycle value is set, which is a fixed number of illuminations. The time-division stroboscopic illumination is performed cyclically based on the cycle value, thereby obtaining the illumination data.

[0009] Furthermore, the method for obtaining the dataset includes: The multi-focal plane camera unit is aligned with three focal planes: the outer surface of the front wall of the bottle, the center of the liquid layer, and the outer surface of the back wall of the bottle. Focal plane calibration is performed to obtain focal plane information items. Based on the focal plane information item, image data of the target filling bottle under the illumination data item is obtained to obtain the acquisition dataset.

[0010] Furthermore, the method for obtaining the focal plane analysis term includes: The focal plane semantic segmentation unit uses a lightweight U-Net model to obtain training data, including labeled samples of different bottle types, different liquid levels, different bottle scratches, and water mist interference, to obtain the first model training term; Based on the model training terms, SIFT rigid registration of feature points is performed on images with multiple focal planes. A threshold for the number of matching feature points and a threshold for the registration error are set. The threshold for the number of matching feature points is the minimum number of feature points, and the threshold for the registration error is the maximum pixel value, thus obtaining the feature point registration terms. Based on the feature point registration term, two types of filling detection related regions, namely liquid region and liquid surface region, are segmented and extracted. The masks of bottle body region and bottle mouth region are directly discarded. Only the features of liquid and liquid surface regions are passed to the subsequent detection module. Non-filling interference is eliminated by matching the position of the focal plane where the defect is located, and then the focal plane analysis term is obtained.

[0011] Furthermore, the method for obtaining the impurity analysis items includes: Using lightweight YOLOv8n as the backbone network, the large target detection head was removed and only the small target detection branch was retained. The window size of the small target Transformer detection head was set, and a small target Transformer detection head was added. Training data was obtained, including labeled samples of glass chips, fibers, white spots, and flocculent matter introduced during the filling process, to obtain the second training model term. During training, a small target copy-paste enhancement strategy is adopted to improve the recognition accuracy of small targets. An upper limit of the confidence threshold is set. The output impurity information includes coordinates, equivalent diameter, and type confidence. When the confidence threshold is greater than the upper limit of the confidence threshold, it is judged as positive, and thus the impurity analysis item is obtained.

[0012] Furthermore, the method for obtaining the filling analysis items includes: The Zernike subpixel edge detection algorithm is used to extract the liquid surface contour. Foam recognition logic is built in. A fluctuation range value is set. Based on the fluctuation range value, the edge gradient within the combined result of the liquid surface and the fluctuation range value is collected. The algorithm automatically filters the edges with low gradient values ​​and selects the continuous edge with the highest gradient as the real liquid surface reference to avoid foam interference with measurement accuracy. The filling volume conversion uses a pre-entered bottle type volume-height calibration curve to convert the actual liquid level height into the filling volume, compares it with the preset upper and lower limits of the filling volume tolerance, outputs the measured filling volume value and the filling volume qualification judgment result, and then obtains the filling analysis items.

[0013] Furthermore, the method for obtaining the comprehensive analysis items includes: Judgment criteria are set for coke surface analysis, impurity analysis, and filling analysis to obtain classification judgment criteria. Screening and judgment are performed based on the classification judgment criteria to obtain a judgment result set. The test results are then comprehensively output based on the judgment result set to obtain a comprehensive analysis item.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This vision-based online intelligent inspection system for liquid drug filling in pharmaceutical production lines effectively solves the problems of reflective interference and motion blur at the physical level through the design of a polarized extinction module and a time-division stroboscopic module. It fundamentally eliminates the interference of mirror reflections from transparent / amber glass, PET bottles, and liquid surfaces on imaging, solving the core pain point of misjudgment caused by confusion between reflections and impurities. At the same time, the time-division stroboscopic module adopts a combination of bottom blue cold backlight, side red ring dark field light source, and top white bright field light source, and performs cyclic time-division stroboscopic illumination at a time interval of <200μs and a single light source response speed of ≤1μs. This high-speed light source switching and precise timing control compresses the total exposure time of a single bottle to ≤600μs, greatly reducing the motion blur effect on high-speed production lines. It provides clear, low-noise raw image data for subsequent image analysis and lays the physical foundation for high-precision impurity detection.

[0015] Meanwhile, the multi-focal plane camera unit precisely aligns with three focal planes: the front wall of the bottle, the center of the liquid layer, and the back wall of the bottle, simultaneously acquiring image data at different depths. This provides a physical basis for layered analysis. The core of the edge inference module lies in the focal plane semantic segmentation unit, which uses a lightweight U-Net model. Through rigid registration of SIFT feature points, it ensures accurate alignment of multi-focal plane images and intelligently segments and extracts the liquid / liquid surface area, discarding masks of irrelevant areas such as the bottle body. This unit can effectively eliminate non-filling interference such as scratches and water mist based on the focal plane location where defects occur. Subsequently, the small target defect detection unit is based on a pruned and optimized lightweight YOLOv8n backbone network, equipped with a 7×7 window small target Transformer detection head, and trained using a small target copy-paste enhancement strategy. It is specifically optimized for 50-200μm micro-imperfections. This combination of layered positioning and intelligent recognition significantly improves the detection accuracy and classification accuracy of micron-level glass chips, fibers, white spots, and other micro-imperfections, effectively reducing the risk of missed detections. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the polarization extinction module of the present invention; Figure 3 This is a schematic diagram of the time-division strobe module of the present invention; Figure 4This is a schematic diagram of the multi-focal plane acquisition module of the present invention; Figure 5 This is a schematic diagram of the edge reasoning module of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the field of pharmaceutical filling and testing, current technologies have significant shortcomings in terms of reflection suppression, defect stratification, and adaptability to dynamic environments, severely limiting detection accuracy and reliability. More importantly, motion blur is particularly pronounced on high-speed filling production lines, further exacerbating the confusion between the aforementioned interfering factors and liquid impurities that may be introduced during the filling process. This severely affects the detection rate of micron-level impurities such as glass fragments, fibers, and white spots. First, because factors such as reflection, scratches, and water mist are similar in image features to liquid impurities, existing detection systems struggle to accurately distinguish them, leading to the misclassification and rejection of many seemingly qualified products. This misrepresentation not only reduces production efficiency and increases operating costs but also negatively impacts a company's reputation. Second, and more seriously, there is the risk of missed detection of substandard products. Micron-level impurities are inherently difficult to detect, and with the added interference of motion blur, they are even more easily overlooked by detection systems. Once drugs containing these impurities enter the market, they may pose potential health hazards to patients. For example, glass fragments may damage the digestive tract, fibers may trigger allergic reactions, and other unidentified impurities may produce unknown side effects. This not only harms patients' interests but also severely damages the brand image and market competitiveness of pharmaceutical companies. The technical solution provided in this application, through the design of a polarization extinction module and a time-division stroboscopic module, effectively solves the problems of reflective interference and motion blur at the physical level. The polarization extinction module, by precisely installing polarizers at both ends of the light source outlet and camera lens mount, covering wavelengths of 400-700nm and automatically calibrating the angle, achieves a specular reflection suppression rate of ≥98.5% and a reflective grayscale value of ≤20 in the bottle area, fundamentally eliminating the specular reflection interference of transparent / amber glass, PET bottles, and liquid surfaces. This addresses the core pain point of misjudgment caused by the confusion between reflections and impurities, effectively mitigating image interference. Simultaneously, the time-division strobe module employs a combination of bottom blue cold backlight, side red ring dark field light source, and top white bright field light source, performing cyclic time-division strobe illumination at time intervals of <200μs and a single light source response speed of ≤1μs. This high-speed light source switching and precise timing control compresses the total exposure time of a single bottle to ≤600μs, significantly reducing motion blur effects on high-speed production lines. This provides clear, low-noise raw image data for subsequent image analysis, laying the physical foundation for high-precision impurity detection. Figure 1 As shown, it includes a polarization extinction module, a time-division stroboscopic module, a multi-focal plane acquisition module, and an edge inference module.

[0019] Polarization extinction module: includes a multi-focal plane camera unit and a light source unit. Polarizers are installed on the multi-focal plane camera unit and the light source unit respectively to eliminate specular reflections on the bottle body and liquid surface, and avoid confusion between reflections and impurities, thus obtaining a polarization installation item.

[0020] It is important to note that, such as Figure 2As shown, the method for obtaining the polarization mounting item includes: setting a first mounting spacing, obtaining the light source outlet position of the light source unit, obtaining a first mounting point, installing a linear polarizer based on the first mounting spacing, setting a wavelength coverage range, and obtaining a first mounting item; obtaining the bayonet position of the multi-focal plane camera unit, obtaining a second mounting point, using the orthogonality of the polarization direction of the light source unit as the mounting angle, obtaining a mounting adjustment item, installing a polarizer based on the mounting adjustment item, setting a wavelength coverage range, and obtaining a second mounting item; the first mounting item and the second mounting item are combined to obtain a polarization mounting item.

[0021] Specifically, the first installation spacing is set to 1cm. The linear polarizer is installed using the light source outlet position of the light source unit as the first installation pad. The wavelength coverage range is set to 400-700nm visible light range, thus obtaining the first installation item. The polarizer of the multi-focal plane camera unit is installed on the bayonet position of the camera lens filter, thus obtaining the second installation point. The installation angle is set with the polarization direction of the light source unit as orthogonal, thus obtaining the installation adjustment item. The polarizer is installed based on the installation adjustment item, and the wavelength coverage range is also set to 400-700nm visible light range, thus obtaining the second installation item. The first installation item and the second installation item are combined to obtain the polarization installation item. During debugging, the polarization angle is automatically matched by gray value acquisition. After calibration, the gray value of the reflection in the bottle area is ≤20, and the specular reflection suppression rate is ≥98.5%, thus eliminating the interference of specular reflection from transparent / amber glass, PET bottle body and liquid surface on filling detection from the physical layer.

[0022] Time-division stroboscopic module: The light source unit includes a bottom cold light source, a side ring dark field light source, and a top bright field light source. The target filling bottle is irradiated by time-division stroboscopic illumination based on the light source unit, thereby obtaining irradiation data items.

[0023] It is important to note that, such as Figure 3 As shown, the method for obtaining illumination data items includes: setting unit parameters based on light source units to obtain light source parameter items, and matching the light source parameter items with light source units respectively; setting time intervals based on light source parameter items, performing time-division stroboscopic illumination based on time intervals, setting a cycle value, the cycle value being a fixed number of illuminations, and performing cyclic time-division stroboscopic illumination based on the cycle value to obtain illumination data items.

[0024] Specifically, since the light source unit includes a bottom cold light source, a side ring dark field light source, and a top bright field light source, the unit parameters are set as follows: the bottom is a 450nm blue cold backlight with a brightness ≥12000lm, used to collect the liquid surface outline and high-absorption liquid impurity features; the side is a 620nm red 360° ring dark field light source with a light output angle of 15° oblique, used to excite the scattered light spot of ≥50μm micro-filling impurities, without the need for an additional light source facing the bottle cap and label detection; the top is a 550nm white bright field light source, used to collect the appearance and coding features of the bottle cap and label, thus obtaining the light source parameters. At this time, the time interval is set to a maximum of 200μs, that is, the time interval <200μs, and the single-channel light source response time ≤1μs, so the total exposure time of a single bottle ≤600μs. The cycle value is set to 2, and the cycle value is used as the basis for cyclic time-division stroboscopic irradiation, thus obtaining the irradiation data.

[0025] Multi-focal plane acquisition module: The multi-focal plane camera unit acquires image data of the target filling bottle based on the illumination data items to obtain the acquisition dataset.

[0026] It is important to note that, such as Figure 4 As shown, the method for acquiring the dataset includes: the multi-focal plane camera unit is aligned with three focal planes: the outer surface of the front wall of the bottle, the center of the liquid layer, and the outer surface of the back wall of the bottle, and focal plane calibration is performed to obtain focal plane information items; based on the focal plane information items, image data of the target filling bottle under the illumination data items is acquired to obtain the acquired dataset.

[0027] Specifically, the three-focal-plane camera group of the multi-focal-plane camera unit consists of three global shutter CMOS cameras, all using Sony IMX264 sensors with a pixel size of 3.45μm, a resolution of 2448×2048, a frame rate of ≥320fps, and a global exposure time of ≤1μs. They are paired with an 8mm focal length F1.4 fixed-focus industrial lens. The three cameras are respectively aimed at the outer surface of the front wall of the bottle, the center of the liquid layer, and the outer surface of the back wall of the bottle. A 1mm thick standard glass calibration plate is used for focal plane calibration, resulting in a focal plane offset error of ≤0.05mm after calibration. The depth of field coverage of the liquid layer focal plane is ±18mm, supporting full-coverage imaging of a maximum liquid layer thickness of 60mm for bottles ranging from 10-500ml. The system is equipped with a Xilinx Artix-7 series FPGA hardware synchronous control unit, directly connecting to the production line encoder pulse signal, supporting NPN / PNP dual interface input, and supporting a maximum pulse input of 1MHz. The trigger delay between the light source and the camera is ≤1ms. At this time, image data of the target filling bottle under the illumination data item is acquired, resulting in the acquired dataset.

[0028] The edge reasoning module includes a focal plane semantic segmentation unit, a small target defect detection unit, and a filling volume sub-pixel calculation unit. The focal plane semantic segmentation unit segments the front wall area of ​​the bottle, the liquid area, the back wall area of ​​the bottle, and the liquid surface area. Interference is eliminated by matching the position of the focal plane where the defect is located, and a focal plane analysis item is obtained. The small target defect detection unit distinguishes impurities and obtains an impurity analysis item. The filling volume sub-pixel calculation unit obtains the filling volume information and obtains a filling analysis item. Finally, the focal plane analysis item, the impurity analysis item, and the filling analysis item are combined to output the detection result and obtain a comprehensive analysis item.

[0029] It is important to note that, such as Figure 5 As shown, the method for obtaining the focal plane analysis term includes: the focal plane semantic segmentation unit adopts a lightweight U-Net model to obtain training data, including labeled samples of different bottle types, different liquid levels, different bottle scratches, and water mist interference, to obtain the first model training term; based on the model training term, SIFT feature point rigid registration is performed on the images of multiple focal planes, and a threshold for the number of matching feature points and a threshold for registration error are set. The threshold for the number of matching feature points is the minimum number of feature points, and the threshold for registration error is the maximum pixel value, to obtain the feature point registration term; based on the feature point registration term, two types of filling detection related regions, namely liquid region and liquid surface region, are segmented and extracted. The masks of the bottle body region and bottle mouth region are directly discarded, and only the features of the liquid and liquid surface regions are passed into the subsequent detection module. Non-filling interference is eliminated by matching the position of the focal plane where the defect is located, thereby obtaining the focal plane analysis term.

[0030] Specifically, the training data includes 100,000 labeled samples with different bottle types, liquid levels, and bottle scratches / water mist interference. The single-image inference time is ≤1.8ms, and the segmentation accuracy is ≥99.85%. During inference, SIFT feature point rigid registration is first performed on the images of the three focal planes. The threshold for the number of matching feature points is set to 20, and the registration error threshold is 0.8 pixels. Therefore, the number of matching feature points is ≥20, and the registration error is ≤0.8 pixels. Then, two types of filling detection-related regions, namely liquid region and liquid surface region, are segmented and extracted. The pixel coverage of the mask region is ≥99.5%. The masks of the bottle body region and bottle mouth region are directly discarded, and only the features of the liquid / liquid surface region are passed to the subsequent detection module to reduce invalid calculations. Non-filling interference is eliminated by matching the position of the focal plane where the defect is located: bottle body scratches, water mist on the outer wall, and stains on the outer wall that only appear on the front / back focal planes are directly filtered out, and only the features of the liquid region and liquid surface region on the middle focal plane are retained for subsequent filling quality judgment.

[0031] In the specific implementation process, the pre-stored camera calibration parameters are first called, and the calibration board image is pre-acquired using the Zhang Zhengyou calibration method. Radial and tangential distortion corrections are performed on the three original images to eliminate pixel shifts caused by lens distortion. Then, SIFT feature points of the bottle outline are extracted from the three images, and ≥20 common feature points are matched for rigid registration in each image. Finally, the pixel alignment error of the three images is ≤0.8 pixels, ensuring that the same physical location corresponds to the same pixel coordinate in the three images. The three aligned images are simultaneously input into the lightweight U-Net model with INT8 quantization. The inference time for a single image is ≤1.8ms. The output is a pixel-level classification mask for each image: classifying all pixels into the front wall region of the bottle, the liquid region, and the back wall of the bottle. The system categorizes regions into four types: invalid regions, bottle mouth / background invalid regions, with a segmentation accuracy of ≥99.85%. The mask coverage of liquid regions is ≥99.5%. Invalid regions outside the bottle mouth and body are directly masked. Position matching is performed on the masks of the three images: if a defect only appears in the bottle body region mask of the front / back focal plane, it is identified as a bottle body scratch, external wall water mist, or external wall stain, and all features of the defect are directly filtered out. If a feature only appears in the liquid / liquid surface region mask of the middle focal plane, it is identified as a filling-related feature. The features of this region are extracted separately and output to the impurity-specific detection module and the filling volume sub-pixel calculation module for subsequent inference. Finally, the output retains only the feature set of the filling-related liquid region and liquid surface region, without any redundant features of the bottle body or background.

[0032] It is important to note that the method for obtaining the impurity analysis term includes: using a lightweight YOLOv8n as the backbone network, removing the large target detection head and retaining only the small target detection branch, setting the window size of the small target Transformer detection head, adding a small target Transformer detection head, and obtaining training data, including labeled samples of glass debris, fibers, white spots, and flocculent matter introduced during the filling process, to obtain the second training model term; during training, a small target copy-paste enhancement strategy is used to improve the small target recognition accuracy, and an upper limit of the confidence threshold is set. The output impurity information includes coordinates, equivalent diameter, and type confidence. When the confidence threshold is greater than the upper limit of the confidence threshold, it is judged as positive, thus obtaining the impurity analysis term.

[0033] Specifically, a lightweight, trimmed YOLOv8n network was used as the backbone. Large target detection heads were removed, leaving only small target detection branches. The window size of the small target Transformer detection head was set to 7×7, specifically optimizing feature extraction for tiny filling impurities of 50-200μm. The training data included 80,000 labeled samples of glass debris, fibers, white spots, and flocculent matter introduced during the filling process, with small target samples of 50-200μm accounting for ≥70%. During training, a small target copy-paste enhancement strategy was adopted: 20-200μm impurity samples were randomly pasted onto defect-free liquid areas, with 1-3 small targets pasted per image to improve the accuracy of small target recognition. The inference time per image was ≤4.5ms. The output impurity information included coordinates, equivalent diameter, and type confidence. A confidence threshold ≥0.8 was considered positive. The false negative rate for filling impurities ≥50μm was ≤0.03%, the false positive rate was ≤0.2%, and the impurity classification accuracy was ≥98.5%.

[0034] In the specific implementation process, firstly, adaptive contrast stretching is performed on the features of the liquid region in the central focal plane, increasing the grayscale difference between the scattered light spots of tiny impurities and the liquid background from an average of 3-5 to 15-20, improving the signal-to-noise ratio by more than 3 times, and preventing tiny impurities of 50-200μm from being submerged by background noise. The features are then input into the pruned lightweight YOLOv8n backbone network, retaining only the small target detection branch and removing the downsampling layer for large defects, outputting feature maps of two scales: 32×32 and 16×16; then, they are fed into the small target Transformer in a 7×7 window. The MERS detector performs local attention calculations on the feature map, focusing on capturing the edge and texture features of tiny impurities to avoid dilution of small target features by convolution operations. First, it verifies the focal plane position of the identified candidate defects: if there are no identical features at corresponding positions on the front and rear focal planes, and the defect exists only on the middle focal plane, it is determined to be an impurity in the liquid introduced during the filling process. Then, it calculates the confidence level of the defect. Positive defects with a confidence level ≥ 0.8 are further classified into four categories: glass fragments, fibers, white spots, and flocculent matter. The final output includes: the number of impurities in the liquid, the type / confidence level of each impurity, and the qualification result of the impurities in the liquid.

[0035] In the specific implementation process, the confidence score is used to determine the probability that the currently identified candidate region is a real impurity rather than noise. The small target Transformer detection head outputs two types of basic confidence scores: target presence confidence and classification confidence. The target presence confidence score, output by the YOLO detection branch, represents the probability that a real target exists in the current candidate box, rather than background noise or reflective residues, with a value range of 0-1. During training, the label for real impurity samples is 1, and the label for background regions is 0. The classification confidence score, output by the detection head's classification branch, represents the probability that the current target belongs to a certain type of impurity, with a value range of 0-1. Each type of impurity corresponds to an independent classification confidence score. The overall confidence score is calculated using a weighted fusion formula, where the weights for target presence confidence and classification confidence are 0.6 and 0.4, respectively. After fusion, non-maximum suppression is performed, with an IOU threshold set to 0.3, eliminating duplicate recognition boxes with the same defect, and retaining only the overall confidence score. Targets with a composite result ≥ 0.8 are considered positive defects and proceed to subsequent classification and particle size calculation stages; targets with a composite result < 0.8 are directly filtered as noise. This threshold is based on tests of 100,000 samples. When the threshold is set to 0.8, the false detection rate can be controlled within 0.2%, while the false negative rate is ≤ 0.03%. After the Transformer head extracts the core features of the target for small targets, it outputs the classification confidence scores of four types of impurities. The category corresponding to the maximum value is taken as the preliminary classification result. The feature matching logic of the four types of impurities is as follows: the core identification features of glass debris are sharp edges, irregular polygonal shapes, gray values ​​much higher than the background, and aspect ratio < 3; the core identification features of fiber are long strip structures, aspect ratio ≥ 3, medium gray values, and slight burrs on the edges; the core identification features of white spots are approximately circular, gray values ​​slightly higher than the background, blurred edges, and diameter generally ≤ 100 μm; the core identification features of flocculent matter are irregular shapes, fluffy edges, low gray values, and area generally ≥ 200 μm. 2 After the verification is passed, the final classification result is output. If the difference between the highest classification confidence and the second highest classification confidence of a certain target is <0.1, it is uniformly classified as other impurities and is not forced to be classified into the 4 categories.

[0036] It is important to note that the method for obtaining the filling analysis items includes: using the Zernike subpixel edge detection algorithm to extract the liquid surface contour, incorporating foam recognition logic, setting a fluctuation range value, and based on the fluctuation range value, collecting the edge gradient within the combined result of the liquid surface and the fluctuation range value. The algorithm automatically filters edges with low gradient values ​​and selects the continuous edge with the highest gradient as the real liquid surface reference to avoid foam interference with measurement accuracy. The filling volume conversion uses a pre-entered bottle type volume-height calibration curve to convert the height value of the real liquid surface into the filling volume, compares it with the preset upper and lower limits of the filling volume tolerance, and outputs the measured filling volume value and the filling volume qualification judgment result, thereby obtaining the filling analysis items.

[0037] Specifically, the fluctuation range is 2mm. The edge gradient within a 2mm range above and below the liquid surface is collected. The gradient value of the real liquid surface is ≥80, and the gradient value of the foam edge is ≤30. The algorithm automatically filters edges with gradient values ​​below 50 and selects the continuous edge with the highest gradient as the real liquid surface reference to avoid foam interference with measurement accuracy.

[0038] In the specific implementation process, Canny edge coarse extraction is performed on the original liquid surface image with bottom backlighting to locate the approximate upper and lower boundaries of the liquid surface. Only the area within 2mm above and below the liquid surface is retained for subsequent calculations, compressing the calculation range by more than 90% and significantly reducing inference time. The Zernike moment algorithm is used to perform sub-pixel-level fitting on the coarsely located liquid surface edges. Convolution calculation is performed on edge pixels using a 7×7 Zernike template. The final positioning accuracy of the liquid surface edges can reach 0.01 pixels, which is far higher than the accuracy of ordinary pixel-level detection. Gradient calculation is performed on all edge points obtained by fitting: the gradient value of the interface between real liquid and air is ≥80, while the gradient value of the interface between foam and air is generally ≤30 due to the loose texture. The algorithm automatically filters edge points with gradient values ​​below 50 and selects the edge with the longest continuous length and the highest average gradient as the real liquid surface reference, completely eliminating the interference of upper foam in foaming dosage forms. The pre-stored volume-height calibration curve of the corresponding bottle type is called and pre-generated by fitting with 20 standard sample bottles with different liquid levels. The fitted R 2 ≥0.9999, convert the actual liquid level height into filling volume; then compare it with the preset upper and lower limits of filling volume tolerance, output the measured filling volume, filling volume qualification judgment result, and output the measured filling volume with an accuracy of 0.0001ml, filling volume out-of-tolerance type, and filling volume qualification judgment result.

[0039] It should be noted that the method for obtaining the comprehensive analysis items includes: setting judgment criteria based on the coke surface analysis items, impurity analysis items, and filling analysis items to obtain classification judgment criteria items; screening and judging based on the classification judgment criteria items to obtain a judgment result set; and outputting the comprehensive analysis items based on the judgment result set.

[0040] Specifically, judgment criteria are set for the focal surface analysis, impurity analysis, and filling analysis items, with pass and fail criteria defined for each. The pass criterion for the focal surface analysis item is: if a defect only appears in the mask of the bottle body area on the front / rear focal surface, it is judged as a bottle body scratch, water mist on the outer wall, or stain on the outer wall; conversely, if a feature only appears in the mask of the liquid / liquid surface area on the middle focal surface, it is judged as fail. The pass criterion for the impurity analysis item is no impurities. The pass criterion for the filling analysis item is: the error between the filling volume and the preset upper and lower limits is ≤ ±0.1ml. At this time, the judgment is screened and judged based on the classification judgment criteria to obtain a judgment result set. Based on the judgment result set, the detection results are comprehensively output to obtain the comprehensive analysis item, and the target filled bottle is judged as qualified, thus realizing intelligent filling detection with visual inspection as the core.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A vision-based intelligent online inspection system for liquid drug filling in pharmaceutical production lines, including: Polarization extinction module: includes a multi-focal plane camera unit and a light source unit. Polarizers are installed on the multi-focal plane camera unit and the light source unit respectively to eliminate specular reflections on the bottle body and liquid surface, and avoid confusion between reflections and impurities, thus obtaining a polarization installation item; Its characteristic is that it further includes: Time-division stroboscopic module: The light source unit includes a bottom cold light source, a side ring dark field light source and a top bright field light source. The target filling bottle is irradiated by time-division stroboscopic irradiation based on the light source unit, and then irradiation data items are obtained. Multi-focal plane acquisition module: The multi-focal plane camera unit acquires image data of the target filling bottle based on the illumination data items to obtain the acquisition dataset; The edge reasoning module includes a focal plane semantic segmentation unit, a small target defect detection unit, and a filling volume sub-pixel calculation unit. The focal plane semantic segmentation unit segments the front wall area of ​​the bottle, the liquid area, the back wall area of ​​the bottle, and the liquid surface area. Interference is eliminated by matching the position of the focal plane where the defect is located, and a focal plane analysis item is obtained. The small target defect detection unit distinguishes impurities and obtains an impurity analysis item. The filling volume sub-pixel calculation unit obtains the filling volume information and obtains a filling analysis item. Finally, the focal plane analysis item, the impurity analysis item, and the filling analysis item are combined to output the detection result and obtain a comprehensive analysis item.

2. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The method for obtaining the polarization mounting item includes: Set a first installation spacing, obtain the light source outlet position of the light source unit, obtain the first installation point, install the linear polarizer with the first installation spacing as the reference, set the wavelength coverage range, and obtain the first installation item; The position of the multi-focal plane camera unit is obtained to get the second mounting point. The mounting angle is obtained by taking the polarization direction of the light source unit as orthogonal. The mounting adjustment item is obtained. The polarizer is installed based on the mounting adjustment item. The wavelength coverage range is set to obtain the second mounting item. The first mounting item and the second mounting item are combined to obtain the polarization mounting item.

3. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The method for obtaining the irradiation data items includes: Based on the setting of unit parameters for each light source unit, light source parameter items are obtained, and each light source parameter item is matched with a light source unit. Based on the light source parameters, time intervals are set, and time-division stroboscopic illumination is performed according to the time intervals. A cycle value is set, which is a fixed number of illuminations. The time-division stroboscopic illumination is performed cyclically based on the cycle value, thereby obtaining the illumination data.

4. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The methods for obtaining the dataset include: The multi-focal plane camera unit is aligned with three focal planes: the outer surface of the front wall of the bottle, the center of the liquid layer, and the outer surface of the back wall of the bottle. Focal plane calibration is performed to obtain focal plane information items. Based on the focal plane information item, image data of the target filling bottle under the illumination data item is obtained to obtain the acquisition dataset.

5. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The method for obtaining the focal plane analysis term includes: The focal plane semantic segmentation unit uses a lightweight U-Net model to obtain training data, including labeled samples of different bottle types, different liquid levels, different bottle scratches, and water mist interference, to obtain the first model training term; Based on the model training terms, SIFT rigid registration of feature points is performed on images with multiple focal planes. A threshold for the number of matching feature points and a threshold for the registration error are set. The threshold for the number of matching feature points is the minimum number of feature points, and the threshold for the registration error is the maximum pixel value, thus obtaining the feature point registration terms. Based on the feature point registration term, two types of filling detection related regions, namely liquid region and liquid surface region, are segmented and extracted. The masks of bottle body region and bottle mouth region are directly discarded. Only the features of liquid and liquid surface regions are passed to the subsequent detection module. Non-filling interference is eliminated by matching the position of the focal plane where the defect is located, and then the focal plane analysis term is obtained.

6. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The method for obtaining the impurity analysis items includes: Using lightweight YOLOv8n as the backbone network, the large target detection head was removed and only the small target detection branch was retained. The window size of the small target Transformer detection head was set, and a small target Transformer detection head was added. Training data was obtained, including labeled samples of glass chips, fibers, white spots, and flocculent matter introduced during the filling process, to obtain the second training model term. During training, a small target copy-paste enhancement strategy is adopted to improve the recognition accuracy of small targets. An upper limit of the confidence threshold is set. The output impurity information includes coordinates, equivalent diameter, and type confidence. When the confidence threshold is greater than the upper limit of the confidence threshold, it is judged as positive, and thus the impurity analysis item is obtained.

7. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The method for obtaining the filling analysis items includes: The Zernike subpixel edge detection algorithm is used to extract the liquid surface contour. Foam recognition logic is built in. A fluctuation range value is set. Based on the fluctuation range value, the edge gradient within the combined result of the liquid surface and the fluctuation range value is collected. The algorithm automatically filters the edges with low gradient values ​​and selects the continuous edge with the highest gradient as the real liquid surface reference to avoid foam interference with measurement accuracy. The filling volume conversion uses a pre-entered bottle type volume-height calibration curve to convert the actual liquid level height into the filling volume, compares it with the preset upper and lower limits of the filling volume tolerance, outputs the measured filling volume value and the filling volume qualification judgment result, and then obtains the filling analysis items.

8. The online intelligent detection system for liquid drug filling in a pharmaceutical production line based on vision detection according to claim 1, characterized in that: The methods for obtaining the comprehensive analysis items include: Judgment criteria are set for coke surface analysis, impurity analysis, and filling analysis to obtain classification judgment criteria. Screening and judgment are performed based on the classification judgment criteria to obtain a judgment result set. The test results are then comprehensively output based on the judgment result set to obtain a comprehensive analysis item.

Images