Drug segmentation system and method based on machine vision and semantic analysis

Abstract

The invention provides a medicine segmentation and quantification system and method based on machine vision. The system and method are oriented to complex optical environments and individualized medicine taking of children. The system obtains a standard image through multi-view collection and calibration preprocessing, generates traceability information in combination with segmentation and recognition, and intelligently reminds a dosage form which is not suitable for crushing. The bearing assembly integrates smashing, material sweeping and collecting, weighing and quantitative discharging, and closed-loop metering is formed through staged material supplementing according to weighing feedback. And evaluating the particle size and agglomeration of the powder to guarantee the stability of ingredients. According to the scheme, traceability and milligram-level distribution of the whole process from the whole tablet to the powder are achieved, accuracy is improved, personal errors are reduced, and the method is suitable for medical dispensing, pharmacy quality inspection and other scenes.

Description

Technical Field

[0001] This invention discloses a drug segmentation system and method based on machine vision and semantic analysis, belonging to the field of intelligent medicine. Background Technology

[0002] Currently, the demand for drug fractionation and quantitative dispensing is increasingly prominent in medical institutions, especially in pediatric medication scenarios. Due to the large differences in children's weight and weaker metabolic capacity, clinical practice often requires the precise fractionation of adult tablets to the milligram level according to body weight. Existing technologies in this field have the following shortcomings:

[0003] First, traditional manual tablet breaking methods rely on visual inspection and experience, resulting in dosage errors of up to ±30%. Furthermore, they cannot accurately record crucial traceability information such as batch numbers and expiration dates, posing potential medication safety risks. While mechanical pulverizing equipment can grind tablets into powder, it lacks a real-time visual feedback mechanism, making it impossible to assess the pulverization effect and powder quality. This leads to uneven particle size and frequent agglomeration, which in turn affects the accuracy of subsequent quantitative dispensing.

[0004] Secondly, existing pharmaceutical visual inspection systems mostly employ fixed optical architectures and single-phase image acquisition, focusing only on detecting cosmetic defects or packaging integrity, failing to organically combine pre-grinding drug identification with post-grinding powder quality assessment. Before grinding, strong reflective interference from blister packs and high-gloss packaging makes drug identification difficult, hindering the effective extraction of batch numbers, expiration dates, and dosage form information. After grinding, powder particles easily agglomerate under electrostatic forces, and existing algorithms lack targeted particle separation and particle size statistics mechanisms, making accurate assessment of grinding quality difficult.

[0005] Secondly, existing quantitative dispensing equipment relies solely on weighing feedback for closed-loop control, failing to consider the impact of powder flowability differences on feeding accuracy. When powder agglomeration is high and flowability is poor, blockages or intermittent gushing can easily occur during feeding, causing the actual dispensing dosage to deviate from the target value. Conversely, when powder flowability is good, a fixed, conservative feeding rate results in low production efficiency. Furthermore, existing systems lack operational flexibility, failing to bypass cumbersome automatic identification processes in scenarios requiring rapid response, such as emergency care, thus impacting clinical response speed. Existing technologies generally lack end-to-end traceability capabilities from whole tablets to quantitative powder. The batch number, expiration date, and dosage form information of the drug before pulverization are not correlated with the powder quality data, weighing data, and replenishment parameters after pulverization. This makes it impossible to trace the specific drug source and operational process in the event of medication errors or adverse reactions, failing to meet the traceability requirements of drug management. Summary of the Invention

[0006] In summary, existing technologies in the field of drug segmentation and quantification control suffer from insufficient visual recognition capabilities, lack of powder quality assessment, limited quantification control accuracy, limited operational modes, and weak traceability, making it difficult to meet the requirements of precision medicine, especially individualized medication for children, for dosage accuracy, operational safety, and full traceability. To address these issues, this invention provides a drug segmentation system and method based on machine vision and semantic analysis, comprising:

[0007] The algorithm and control integrated machine includes a computing module, a housing, and a human-machine interface;

[0008] A visual acquisition and scanning component includes a position adjustment module and a visual head, wherein the position adjustment module is used to switch the visual head between a pre-crushing position and a post-crushing position;

[0009] The drug carrying assembly includes a rotating / positioning platform, a replaceable carrying plate, an integrated weighing unit, and a drug physical processing and quantification component. The rotating / positioning platform provides an angle index signal. The integrated weighing unit is located below the replaceable carrying plate and outputs a weight signal to the calculation module in real time. The drug physical processing and quantification component includes a micro-pulverizing unit, a sweeping component, a plate edge collection area, and a quantification discharge actuator.

[0010] The calibration and reference area is set within the field of view of the replaceable carrier plate to provide a reference for color standardization and viewing angle registration;

[0011] The software processing module running on the computing module includes:

[0012] The image preprocessing module is used to perform color normalization, highlight suppression and viewpoint registration on the image sequence acquired by the vision head. The color normalization calls the normalization parameters of the calibration and reference areas.

[0013] The instance segmentation module is used to generate pixel-level segmentation masks for the drug body, packaging structure, and identification areas.

[0014] The cross-view instance association module is used to perform instance matching on the segmentation results under different views based on the angle index signal, maintain a global instance identifier table, and generate a unique traceability identifier for each drug instance.

[0015] The identification module is used to perform text recognition or barcode decoding within the identification area, extract batch number, expiration date and dosage form information and establish a binding relationship with the traceability identification;

[0016] The particle separation module is used to perform particle separation processing on the image of the pulverized powder and output the separated particle mask;

[0017] The particle size statistics module is used to calculate the equivalent diameter of each separated particle, output the particle size distribution characteristic value, and generate grinding quality assessment results and powder flowability rating based on particle size distribution and agglomeration degree.

[0018] The mode selection and dosage form discrimination module is used to: receive the operation mode selected by the user through the human-machine interface, including manual mode and automatic mode; receive the drug information input by the user in manual mode; and in automatic mode, call the recognition result of the identification module, perform suitability discrimination according to the preset contraindicated dosage form database, and output an interlock signal to the micro-pulverizing unit and the quantitative dispensing actuator when the dosage form is determined to be contraindicated, and output a warning message through the human-machine interface.

[0019] The quantitative control and data integration module is used to drive the micro-grinding unit to perform grinding when grinding is permitted, and to drive the sweeping component to push the powder to the collection area on the edge of the disc; to determine whether secondary grinding is required based on the grinding quality assessment result, and to prompt secondary grinding through the human-machine interface when the particle size exceeds a preset threshold; when the grinding quality is qualified, based on the real-time weighing data of the integrated weighing unit and the powder flowability rating, the quantitative discharge actuator is controlled through closed-loop feedback to perform staged feeding until the target dosage is reached; and the traceability identifier, identification information, image data before and after grinding, segmentation mask, weighing data and feeding parameters are integrated into structured traceability data output.

[0020] The micro-pulverizing unit is used to pulverize tablets. The sweeping component is an elastic brush or scraper that forms adjustable contact with the replaceable carrier plate. The powder is radially pushed to the plate edge collection area by the rotational motion of the rotating / positioning platform. The plate edge collection area is connected to the collection container through a detachable collection container interface.

[0021] The human-computer interaction interface includes a display screen and an input device, wherein the input device is a touch screen, a keyboard, or a combination thereof; the display screen is used to display the operation mode selection interface, visual recognition results, grinding quality assessment results, real-time weighing data, and warning information; the input device is used to receive the operation mode selected by the user and the drug information manually entered.

[0022] Highlight suppression in the image preprocessing module is achieved by detecting high-brightness areas and using a texture reconstruction algorithm to recover the information of those areas.

[0023] The cross-view instance association module is implemented in the following way: extracting mask features from adjacent views, calculating feature similarity between views and solving for the optimal match, verifying the consistency of the match, maintaining a global instance identifier table, and recording the mask position of each drug instance at different angles.

[0024] The particle separation module is implemented in the following ways: morphological processing of the initial segmentation mask; watershed segmentation based on distance transformation; and region merging for over-segmented regions using a clustering algorithm.

[0025] The instance segmentation module employs a deep convolutional neural network; the structured traceability data includes: traceability identifiers of drug instances, multi-view images, segmentation masks, recognition text, geometric dimensions before and after pulverization, powder particle size distribution, weighing data, and feeding process parameters, enabling full-process traceability from complete tablets to quantitative powder.

[0026] A drug segmentation method based on machine vision and semantic analysis includes the following steps:

[0027] S0: Receive the operation mode selected by the user through the human-computer interaction interface. If the manual mode is selected, receive the drug information entered by the user and jump to step S3. If the automatic mode is selected, proceed to step S1.

[0028] S1: Adjust the vision head to the pre-crushing position, drive the rotating / positioning stage to rotate the drug and packaging at multiple angles, and acquire an image sequence with angle index; call the image preprocessing module to perform color normalization, highlight suppression, and viewpoint registration on the image sequence; call the instance segmentation module to generate pixel-level segmentation masks for the drug body, packaging structure, and identification area; call the cross-viewpoint instance association module to perform instance matching based on the angle index signal for the segmentation results under different viewpoints, and assign a unique traceability identifier to each drug instance; call the identifier recognition module to perform text recognition or barcode decoding in the identification area, extract batch number, expiration date, and dosage form information, and establish a binding relationship with the traceability identifier;

[0029] S2: Perform suitability judgment based on the identified dosage form information, query the preset contraindicated dosage form database, and if it is determined to be a contraindicated dosage form, output an interlock signal to prevent the micro-pulverization unit and the quantitative discharge actuator from starting, and output a warning message through the human-machine interface, and return to step S0 after waiting for user confirmation; if it is determined to be a permitted pulverization dosage form, proceed to step S3;

[0030] S3: Drive the micro-pulverizing unit to pulverize the tablets, drive the sweeping assembly to push the powder to the edge collection area of ​​the disc, and the powder enters the weighing area of ​​the replaceable carrier disc;

[0031] S4: Adjust the vision head (32) to the position after crushing and acquire the powder distribution image; call the image preprocessing module to preprocess the powder distribution image; call the particle separation module to perform morphological processing, watershed segmentation based on distance transformation, and region merging of the clustering algorithm on the initial segmentation mask; call the particle size statistics module to calculate the equivalent diameter of each separated particle and output the characteristic value of the particle size distribution; output the quality assessment result and powder flowability rating based on the particle size distribution and agglomeration degree.

[0032] S5: Determine whether the grinding quality is qualified. If the median particle size exceeds the preset threshold, prompt for secondary grinding through the human-machine interface and return to step S3. If qualified, proceed to step S6.

[0033] S6: Based on the real-time weighing data of the integrated weighing unit (22a) and the powder flowability rating, the quantitative discharge actuator is controlled by closed-loop feedback to perform staged feeding, including coarse feeding stage, micro feeding stage and micro replenishment stage. The feeding rate is adjusted according to the flowability rating until the target dose is reached.

[0034] S7: Integrate the traceability identifier, identification information, image data before and after crushing, segmentation mask, powder particle size distribution, weighing data and feeding parameters into structured traceability data and output it.

[0035] The highlight region suppression in step S1 includes: detecting high-brightness regions, expanding the highlight region boundaries, and using a texture reconstruction algorithm to recover the highlight region information; the cross-viewpoint instance association includes: extracting mask features from adjacent viewpoints, calculating the feature similarity between viewpoints and solving for the optimal match, verifying the matching consistency, and maintaining a global instance identifier table.

[0036] The processing flow of the particle separation module in step S4 includes: performing morphological operations on the initial segmentation mask; calculating the distance transformation of the processed mask and detecting local maxima as seed points; executing the watershed algorithm starting from the seed points; calculating the feature vector for each watershed region, and using a clustering algorithm to determine independent particles and merge over-segmented regions.

[0037] The phased feeding control strategy described in step S6 is as follows: in the coarse feeding stage, the feeding rate is at the maximum rate, which is close to the first preset ratio of the target dose; in the micro feeding stage, the feeding rate is reduced to the second preset ratio of the coarse feeding stage, which is close to the third preset ratio of the target dose; in the micro supplementation stage, the feed rate is precisely supplemented to the target dose by pulsed intermittent feeding.

[0038] In step S6, the feeding rate is adjusted according to the powder flowability rating: when the flowability rating is poor, the feeding rate in the coarse feeding stage is reduced and the vibration auxiliary device is activated; when the flowability rating is good, the standard feeding rate is adopted; when the flowability rating is excellent, the feeding rate in the coarse feeding stage is increased.

[0039] The instance segmentation module in step S1 uses a deep convolutional neural network to simultaneously optimize the segmentation loss, edge detection loss, and instance discrimination loss through a multi-task learning strategy.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] Through the collaborative work of image preprocessing, instance segmentation, cross-view instance association, identification, particle separation, and particle size statistics modules, and by forming a closed-loop control chain with the micro-pulverization unit, sweeping component, disk edge collection area, integrated weighing unit, and quantitative dispensing actuator in the drug delivery assembly, the system can achieve stable visual inspection, particle quality assessment, and milligram-level quantitative dispensing before and after pulverization. The image preprocessing module provides standardized input under complex optical conditions such as blister packs and high-gloss packaging through color standardization and highlight suppression. The identification module binds batch number, expiration date, and dosage form information to instance masks, providing a reliable data foundation for subsequent suitability assessment and traceability.

[0042] This system forms a closed-loop metering process: after the micro-pulverizing unit pulverizes the tablets, the sweeping component, in conjunction with the rotation of the support plate, radially pushes the powder to the collection area at the edge of the plate, reducing manual intervention and dead zones; the integrated weighing unit outputs weight data in real time within the static weighing window. The particle separation module and particle size statistics module perform agglomerate separation and particle size distribution calculation on the powder image, outputting a grinding quality assessment and powder flowability rating; when the assessment is unqualified, the system can prompt for secondary grinding to ensure that the powder entering the metering stage has better consistency and flowability, reducing metering fluctuations from the source.

[0043] In terms of safety, the mode selection and dosage form identification module supports both manual and automatic operating modes. In automatic mode, the system performs suitability assessment based on the identification results and a pre-set contraindicated dosage form database. When the dosage form is identified as non-crushable or non-divisible, the risk of mishandling of special formulations such as sustained-release and enteric-coated formulations is significantly reduced. Under complex operating conditions, the system maintains stable identification accuracy, measurement accuracy, and traceability consistency. Especially for small-dose individualized medication scenarios, the system significantly improves the safety and operability of medication for children and special populations. Simultaneously, the modular drug delivery and positioning components and software processing module facilitate engineering deployment and maintenance, making it suitable for various application scenarios such as medical institutions, pharmacy repackaging, and pharmaceutical quality inspection. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the accompanying drawings used in the following description of the embodiments or examples will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the technical solutions shown in these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a drug segmentation system based on machine vision and semantic analysis.

[0046] Figure 2This is a structural diagram of the physical processing and quantification components of a drug segmentation system based on machine vision and semantic analysis.

[0047] Figure 3 This is a structural diagram of the material scanning component of a drug segmentation system based on machine vision and semantic analysis.

[0048] Figure 4 This is a flowchart of the image preprocessing module of a drug segmentation system based on machine vision and semantic analysis.

[0049] Explanation of reference numerals in the attached diagram: 1-Algorithm and control integrated machine; 11-Computation module; 12-Box; 2-Drug carrying component; 21-Rotating / positioning platform; 22-Replaceable carrying plate; 22a-Integrated weighing unit; 23-Drug physical processing and quantitative component; 231-Micro pulverizing unit; 232-Sweeping component; 233-Plate edge collection area; 3-Vision acquisition and scanning component; 31-Position adjustment module; 32-Vision head. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0053] Example 1

[0054] See Figure 1 This invention provides a machine vision-based drug traceability segmentation and quantification control system, comprising a hardware execution layer, an image acquisition layer, an algorithm processing layer, and a control execution layer. In this embodiment, the hardware execution layer provides a stable mechanical platform; the image acquisition layer includes a vision acquisition and scanning component 3, comprising a position adjustment module 31 and a vision head 32; the algorithm processing layer includes an algorithm and control integrated machine 1, comprising a computing module 11 and a housing 12; and the control execution layer includes a drug physical processing and quantification component 23.

[0055] The drug carrying and positioning component 2 includes a rotating / positioning platform 21, a replaceable support plate 22, and an integrated weighing unit 22a. The rotating / positioning platform 21 is driven by a stepper motor, has an angular resolution of 0.1 degrees, and can rotate within a range of 0-360 degrees. After each rotation, it outputs an angle index signal to the calculation module 11. The replaceable support plate 22 is selected according to the drug type and automatically reads label information and loads the corresponding scene template parameters when inserted into the rotating / positioning platform 21. The integrated weighing unit 22a uses a high-precision strain gauge sensor, which is mounted on the replaceable support plate 22, and transmits real-time weight data to the calculation module 11 via a signal line.

[0056] The position adjustment module 31 of the vision acquisition and scanning component 3 adopts a linear guide rail and stepper motor drive structure, enabling precise adjustment of the vision head 32 in the vertical direction by 50mm. The vision head 32 is equipped with a 5-megapixel industrial camera with a 16mm lens focal length, and its field of view covers the entire working area of ​​the replaceable carrier plate 22. The position adjustment module 31 presets two fixed poses: a pre-crushing position and a post-crushing position. In the pre-crushing position, the vision head 32 is 300mm away from the surface of the replaceable carrier plate 22, suitable for acquiring images of complete tablets and packaging; in the post-crushing position, the vision head 32 is 150mm away from the surface of the carrier plate, suitable for acquiring images of the microscopic distribution of powder. The pose switching is controlled by the calculation module 11, with a switching time of approximately 2 seconds and a positioning accuracy of ±0.1mm.

[0057] The pharmaceutical physical processing and quantitative assembly 23 includes a micro-pulverizing unit 231, a sweeping assembly 232, a pan edge collection area 233, and a quantitative dispensing actuator. The micro-pulverizing unit 231 uses a ceramic grinding head, driven by a DC motor with an adjustable speed of 3000-5000 rpm. The grinding head has a diameter of 8 mm and can finely pulverize single tablets. The sweeping assembly 232 is a silicone elastic brush with bristle length of 5 mm. A pneumatic drive device controls the contact pressure between the brush head and the surface of the carrier pan. When the rotating / positioning platform 21 rotates, the sweeping assembly 232 remains in a fixed position, with the bristles forming a 5-10 degree contact angle with the carrier pan surface, pushing the powder radially to the pan edge collection area 233. The pan edge collection area 233 is an annular groove on the edge of the carrier pan, 10 mm wide and 3 mm deep, connected to a standard medicine cup via a detachable collection container interface. The quantitative discharge actuator uses a vibrating micro-feeding nozzle with an adjustable vibration frequency of 50-200Hz and a discharge port diameter of 2mm. The vibration amplitude is controlled by a piezoelectric ceramic actuator.

[0058] The computing module 11 of the algorithm and control all-in-one machine 1 adopts an industrial-grade embedded computing platform, equipped with an Intel Core i7 processor, 16GB of memory, and an NVIDIA GPU accelerator card. The computing module 11 runs a Linux operating system and integrates image processing, deep learning inference, and motion control functions. The human-machine interface includes a 10.1-inch touchscreen display, mounted on the front panel of the enclosure 12, for operating mode selection, parameter setting, and status monitoring.

[0059] In clinical pediatric medication scenarios, physicians prescribe individualized medications based on the child's weight, age, and condition. For example, a 100mg tablet might need to be divided into 35mg doses. The pharmacist selects automatic mode via a human-machine interface, places the tablet on a replaceable tray 22, and inserts it into the rotating / positioning stage 21. In this embodiment, the tablet is kept in an aluminum-plastic blister pack, and the blister film surface has strong specular reflection, which is common in actual pharmacy operations. The system automatically starts, and the rotating / positioning stage 21 rotates sequentially to eight positions: 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees. The visual head 32 acquires one frame image for each position before pulverization, resulting in a total of eight image frames with angle indices. After the images are transmitted to the computing module 11, the image preprocessing module performs standardization processing on the image sequence.

[0060] To address the specular reflection issue in blister packaging, the image preprocessing module performs specular highlight suppression. First, the image's brightness gradient is calculated, and pixels with gradient values ​​exceeding a threshold of 200 are identified as candidate specular regions. Then, a region growing algorithm is executed using these candidate points as seeds. Pixels whose brightness is greater than 90% of the current pixel's brightness are included in the specular region, ultimately determining the specular mask. For the specular regions, a guided filtering algorithm is used to recover texture information. The filtering radius is set to 8 pixels, and the regularization parameter ε = 0.01. Weighted reconstruction is performed using texture features from neighboring non-spectral regions, ensuring that the tablet outline and surface markings beneath the blister film are clearly visible.

[0061] The color normalization operation calls the normalization parameters of the calibration and reference areas. At startup, the system acquires an image of the calibration area, calculating the RGB mean value of the white reference panel as (245, 248, 250) and the RGB mean value of the black reference panel as (12, 10, 11). For each pixel of the tablet image, a linear mapping formula is applied: R' = (R-12) / (245-12) × 255. The G' and B' channels are calculated similarly to eliminate the influence of light source color temperature differences. The viewing angle registration operation uses the 0-degree viewing angle as a reference, calculates the rotation transformation matrix for the images of the other seven viewing angles, uses a feature point matching algorithm to determine the rotation angle, and applies bilinear interpolation to achieve sub-pixel level registration.

[0062] The instance segmentation module generates pixel-level segmentation masks based on a deep convolutional neural network. The network input is a standardized image with dimensions of 2048×2048×3. The network uses a ResNet-50 encoder to extract features, and the decoder restores spatial resolution through upsampling and skip connections, outputting a 12-channel probability map corresponding to 12 categories, including the drug body, blister pack, label area, and background. For the tablet image in this embodiment, the drug body mask delineates the active ingredient area of ​​the tablet below the blister pack, with an area of ​​approximately 450 mm²; the packaging structure mask identifies the blister pack area, with an area of ​​approximately 800 mm²; and the label area mask delineates the location of the batch number "20241015" embossed on the tablet surface, with an area of ​​approximately 30 mm².

[0063] The cross-view instance association module fuses the segmentation results from eight viewpoints into a global instance based on the angle index signal. First, it extracts the drug body mask contours from adjacent viewpoints, calculates the Hu invariant moment feature vector, and determines whether two masks represent the same tablet using cosine similarity. In this embodiment, the tablet mask from the 0-degree viewpoint and the mask from the 45-degree viewpoint have a similarity of 0.92, indicating they are the same instance. The eight viewpoints are then sequentially associated, a global instance identifier table is maintained, and a unique traceability identifier "TRACK-20241021-001" is assigned to the tablet.

[0064] The identification module performs text recognition within the masked identification area. It uses an OCR algorithm to detect character outlines, extracts character images, and inputs them into a pre-trained character recognition network. The recognition result is the batch number "20241015", the expiration date "2026-10", and the dosage form "tablets". The recognition result is then linked to the traceability identifier "TRACK-20241021-001" and stored in the database.

[0065] The mode selection and dosage form identification module calls the identifier recognition result and queries the preset contraindicated dosage form database. The database contains a list of dosage forms that are not suitable for pulverization, such as sustained-release tablets and enteric-coated tablets. In this embodiment, the identified dosage form is "tablets," which is determined to be a dosage form that can be pulverized. The system releases the interlock signal, allowing subsequent pulverization operations.

[0066] The pharmacist presses the start button, and the quantitative control and data integration module drives the micro-grinding unit 231 to perform pulverization. The pharmacist removes the tablet from the blister packaging, places it in the micro-grinding unit 231, and starts the motor. The grinding head rotates at 4000 rpm to grind the tablet into powder. Subsequently, the rotating / positioning platform 21 rotates once at 10 rpm, and the silicone brush of the sweeping assembly 232 pushes the powder radially to the collection area 233 on the edge of the tray. After the powder is pushed, the integrated weighing unit 22a measures the total weight of the powder as 98.5 mg.

[0067] The position adjustment module 31 drives the vision head 32 to switch to the post-crushing position and acquires an image of the powder distribution. The image preprocessing module performs preprocessing on the powder image, and the particle separation module processes the initial segmentation mask. First, a morphological opening operation is performed on the powder mask, with a 3×3 circle as the structuring element, to remove isolated noise points. Then, the distance transform is calculated, where the value of each foreground pixel in the distance transform map represents its distance to the nearest background pixel. Local maxima points in the distance transform map are detected, resulting in 126 seed points, corresponding to 126 potential particles. Starting from the seed points, a watershed algorithm is performed on the negative values ​​of the distance transform map to obtain a preliminary separated particle mask.

[0068] For over-segmented particles, the particle separation module calculates the area, perimeter, and roundness characteristics of each region, and uses a hierarchical clustering algorithm to determine whether adjacent regions should be merged. The clustering distance threshold is set to 0.3, ultimately outputting 78 independent particle masks. The particle size statistics module calculates the equivalent diameter of each particle, defined as the diameter of a circle with the same area as the particle. Statistical results show: d10 = 0.35mm, d50 = 0.62mm, d90 = 1.15mm, indicating a uniform particle size distribution that meets the target requirements. Based on the particle size distribution and aggregation degree, the aggregation degree is defined as the proportion of multi-peak regions in the distance transformation graph, which is 8% in this embodiment.

[0069] The quantitative control and data integration module controls the quantitative dispensing actuator to perform staged feeding based on the flowability rating and the target dose of 35mg. The standard medicine cup is installed below the collection area 233 on the edge of the tray through a detachable collection container interface, and weighing begins after the integrated weighing unit 22a is zeroed.

[0070] The first stage is the coarse feeding stage, with a target of approximately 80% of 35mg, or 28mg. The vibration frequency of the quantitative dispensing actuator is set to 150Hz, with a moderate vibration amplitude. Powder in the collection area 233 at the edge of the disc flows into the medicine cup through the discharge port under the action of vibration. The integrated weighing unit 22a reads the weight data every 100ms. When the weight reaches 27.8mg, the system reduces the vibration frequency. The second stage is the micro feeding stage, with a target of approximately 95% of 35mg, or 33.25mg. The vibration frequency is reduced to 50Hz, and the vibration amplitude is reduced to 30% of that in the coarse feeding stage, with a significant reduction in the feeding rate. When the weight reaches 33.1mg, the system stops vibrating. The third stage is the micro replenishment stage, with a target of precisely replenishing to 35mg. The vibration frequency is reduced to 30Hz, and pulsed intermittent feeding is used, with each vibration lasting 0.2 seconds followed by a 0.5-second pause to allow the weighing to stabilize. When the weight reaches 34.8mg, a pulse feeding is executed, and feeding stops when the weight reaches 35.0mg. The micro-compensation phase took approximately 3 seconds. The final weighing result was 35.0 mg, with an error of 0 mg, meeting the accuracy requirements.

[0071] The quantitative control and data integration module integrates the data from the entire process into structured traceability data, including: traceability identifier "TRACK-20241021-001", eight-view image file path, instance segmentation mask, identification text: batch number "20241015", expiration date "2026-10", dosage form "tablets", and the geometric dimensions of the tablets before pulverization: area 450mm². 2 The dimensions are: circumference 85mm, major axis 12mm, minor axis 9mm, and powder particle size distribution. Structured data is stored in JSON format and output to an external database or printed as a traceability label via a human-computer interaction interface. The pharmacist delivers the medicine cup with the traceability label to the child's family, completing the individualized medication process.

[0072] This embodiment achieves end-to-end traceability control from whole tablets to quantitative powder through the collaborative work of image preprocessing, instance segmentation, cross-view association, identifier recognition, particle separation, particle size statistics, dosage form discrimination, and quantitative control modules. The traceability information is complete and reliable, meeting the safety and accuracy requirements of clinical pediatric medication.

[0073] Example 2

[0074] The image preprocessing module establishes a color normalization mapping based on the calibration and reference area 51, considering the influence of light source color temperature, camera response, and ambient lighting on image color. The normalization mapping is expressed as follows:

[0075] R'=(R-Rblack) / (Rwhite-Rblack)×255

[0076] G'=(G-Gblack) / (Gwhite-Gblack)×255

[0077] B'=(B-Bblack) / (Bwhite-Bblack)×255

[0078] Where: R', G', and B' are the standardized RGB channel values; R, G, and B are the original RGB channel values; Rwhite, Gwhite, and Bwhite are the average RGB values ​​of the white reference board; and Rblack, Gblack, and Bblack are the average RGB values ​​of the black reference board.

[0079] Standardized parameters are obtained through the calibration process. See [link / reference] Figure 2 The calibration process is as follows: The system performs calibration once each time it starts up or every 2 hours. The vision head 32 acquires images of the white reference board, black reference board, and standard color card in the calibration and reference area 51, respectively. The calculation module 11 calculates the RGB mean value of the white reference board as the white reference and calculates the RGB mean value of the black reference board as the dark reference. Table 1 lists the calibration parameter ranges under different lighting conditions.

[0080] Table 1: Calibration Parameter Range

[0081]

[0082] The model calculates images in a single frame and outputs standardized images in real time.

[0083] The core function of the image preprocessing module is to apply calibration parameters to the original image, eliminating differences in light source color temperature drift and camera response. Reflection suppression employs a three-level processing strategy:

[0084] Level 1: When the image brightness gradient is less than the threshold Tgrad, it is determined that there is no obvious highlight area, and the reflection suppression process is skipped.

[0085] Level 2: When Tgrad≤brightness gradient≤2×Tgrad, local highlights are identified. The region growing algorithm is used to expand the highlight region and guided filtering reconstruction is performed. The filtering radius r is set to 5–10 pixels.

[0086] Level 3: When the brightness gradient > 2 × Tgrad, it is determined that there is a large area of ​​highlight. Multi-scale guided filtering is used. First, a large window r = 15 is used to suppress the large area of ​​highlight, and then a small window r = 5 is used to preserve the details.

[0087] To verify the accuracy of standardization, comparative tests were conducted under laboratory conditions. A standard color chart was used as the test sample, and images were acquired under three different lighting conditions. The color difference ΔE before and after standardization was recorded. The results showed that the average color difference ΔE before standardization was 12.5, while the average color difference ΔE after standardization decreased to 2.3, demonstrating that the standardization strategy significantly improved color consistency.

[0088] Example 3

[0089] The semantic segmentation module employs a deep convolutional neural network with an encoder-decoder structure, optimized for drug segmentation scenarios. The network input is a standardized image with dimensions 2048×2048×3, and the network output is a segmentation mask with dimensions 2048×2048×C, where C represents the number of classes, set to 12–15 classes.

[0090] The network structure includes the following layers:

[0091] (1) Encoder: Multi-scale features are extracted based on ResNet-50 backbone network, which includes 5 residual block groups. The output feature map sizes are 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 times the input resolution, and the number of channels are 64, 256, 512, 1024 and 2048, respectively.

[0092] (2) Attention module: Spatial attention and channel attention modules are embedded in the 3rd, 4th and 5th layers of the encoder, and the weights are adaptively adjusted according to the spatial statistics and channel correlation of the feature map.

[0093] (3) Decoder: It adopts an upsampling and skip connection structure to gradually fuse the multi-scale features of the encoder and restore the spatial resolution. The upsampling method adopts bilinear interpolation or transposed convolution.

[0094] (4) Output layer: The feature map is mapped to a C-channel probability map through 1×1 convolution. Each channel corresponds to a category, and the softmax activation function is applied to normalize it into a probability distribution.

[0095] The network has approximately 45 million parameters and the model size is about 180MB.

[0096] The model training process is as follows: Collect labeled datasets covering 5 typical drug types, with 200–500 images labeled for each type, including blister packaging, bottled products, Chinese herbal medicine slices, powder granules, etc.

[0097] The training objective is to minimize the multi-task loss function:

[0098] Loss=w1×Lseg+w2×Ledge+w3×Linst

[0099] Wherein: Lseg is the segmentation loss, which is a weighted combination of cross-entropy loss and Dice loss; Ledge is the edge detection loss, calculated using the edge distance map; and Linst is the instance discrimination loss, calculated using a contrastive learning approach to lose the feature embedding space. The weight coefficients w1, w2, and w3 are dynamically adjusted according to the training stage, initially emphasizing segmentation loss and later increasing the weight of edge and instance losses.

[0100] Example 4

[0101] The cross-view instance association module, based on the angle index of the rotation / positioning platform 21, realizes instance consistency tracking and panoramic mask fusion among multi-view images. This embodiment adopts a combined strategy of feature matching, optical flow estimation, and global instance ID maintenance.

[0102] The association process includes the following steps:

[0103] S1: Feature extraction, extracting contour features and region features for each instance. Contour features include Hu invariant moments, Fourier descriptors, contour area, perimeter, roundness, aspect ratio, etc.; region features include HSV color histogram, texture histogram, average gray level, gray level standard deviation, etc. The total dimension of the concatenated feature vectors is approximately 120–150, which is reduced to 32–64 dimensions using PCA.

[0104] S2: For adjacent viewpoints k and k+1, calculate the feature similarity matrix S between all instances in the two viewpoints. The matrix element S(i,j) is the cosine similarity between instance i in viewpoint k and instance j in viewpoint k+1. Simultaneously, consider spatial position constraints: based on the angle index of the rotation / positioning platform 21, calculate the expected position offset of the instance in the two viewpoints. If the actual position offset from the expected position exceeds a threshold, the similarity is set to a minimum value.

[0105] S3 constructs a bipartite graph matching problem and uses the Hungarian algorithm to find the optimal matching relationship. The matching result is a pair, indicating that the instance of viewpoint k and the instance of viewpoint k+1 are the same physical instance.

[0106] S4: Optical flow estimation verification. For pairings with a matching confidence score below a threshold, a dense optical flow algorithm is used to calculate the pixel motion vector field between adjacent viewpoint images. For candidate matching pairs, the instance mask region is extracted, and the position of this region in viewpoint k+1 is predicted based on the optical flow field and compared with the actual position. If the overlap rate between the predicted and actual positions is greater than a threshold, the match is confirmed to be valid.

[0107] Global Instance ID Maintenance: A global instance identifier table is established to record the local instance ID mapping relationship of each physical instance under different perspectives. During initialization, a global ID is assigned to all instances in perspective 0, and subsequent perspectives update the mapping table based on the matching results.

[0108] S5: For each global instance, collect its mask from various viewpoints and project it onto a unified panoramic coordinate system. The fusion method employs a weighted voting strategy. For each pixel in the panoramic coordinate system, the segmentation labels from different viewpoints are counted, and weighted voting is performed based on the viewpoint visibility weights.

Claims

1. A drug segmentation system based on machine vision and semantic analysis, characterized in that, include: The algorithm and control integrated machine (1) includes a computing module (11), a housing (12), and a human-machine interface; The visual acquisition and scanning component (3) includes a position adjustment module (31) and a visual head (32), wherein the position adjustment module (31) is used to switch the visual head (32) between a pre-crushing position and a post-crushing position; The drug carrier assembly (2) includes a rotating / positioning platform (21), a replaceable carrier plate (22), an integrated weighing unit (22a), and a drug physical processing and quantification assembly (23). The rotating / positioning platform (21) provides an angle index signal. The integrated weighing unit (22a) is located below the replaceable carrier plate (22) and outputs a weight signal to the calculation module (11) in real time. The drug physical processing and quantification assembly (23) includes a micro-pulverizing unit (231), a sweeping assembly (232), a plate edge collection area (233), and a quantification discharge actuator. The calibration and reference area is set within the field of view of the replaceable carrier plate (22) to provide a reference for color standardization and viewing angle registration; The software processing module running on the computing module (11) includes: The image preprocessing module is used to perform color normalization, highlight suppression and viewpoint registration on the image sequence acquired by the visual head (32). The color normalization calls the normalization parameters of the calibration and reference areas. The instance segmentation module is used to generate pixel-level segmentation masks for the drug body, packaging structure, and identification areas. The cross-view instance association module is used to perform instance matching on the segmentation results under different views based on the angle index signal, maintain a global instance identifier table, and generate a unique traceability identifier for each drug instance. The identification module is used to perform text recognition or barcode decoding within the identification area, extract batch number, expiration date and dosage form information and establish a binding relationship with the traceability identification; The particle separation module is used to perform particle separation processing on the image of the pulverized powder and output the separated particle mask; The particle size statistics module is used to calculate the equivalent diameter of each separated particle, output the particle size distribution characteristic value, and generate grinding quality assessment results and powder flowability rating based on particle size distribution and agglomeration degree. The mode selection and dosage form discrimination module is used to: receive the operation mode selected by the user through the human-machine interface, including manual mode and automatic mode; receive the drug information input by the user in manual mode; in automatic mode, call the recognition result of the identification module, perform suitability discrimination according to the preset contraindicated dosage form database, and output an interlock signal to the micro-pulverizing unit (231) and the quantitative discharge execution unit when the dosage form is determined to be contraindicated, and output a warning message through the human-machine interface. The quantitative control and data integration module is used to drive the micro-grinding unit (231) to perform grinding when grinding is allowed, and drive the sweeping component (232) to push the powder to the disk edge collection area (233); determine whether secondary grinding is required based on the grinding quality assessment result, and prompt secondary grinding through the human-machine interface when the particle size exceeds the preset threshold; when the grinding quality is qualified, based on the real-time weighing data of the integrated weighing unit (22a) and the powder flowability rating, control the quantitative discharge actuator to perform staged feeding through closed-loop feedback until the target dose is reached; integrate the traceability mark, identification information, image data before and after grinding, segmentation mask, weighing data and feeding parameters into structured traceability data output.

2. The drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The micro-pulverizing unit (231) is used to pulverize the tablets. The sweeping component (232) is an elastic brush or scraper that forms adjustable contact with the replaceable carrier plate (22). The powder is pushed radially to the edge collection area (233) by means of the rotation of the rotating / positioning platform (21). The edge collection area (233) is connected to the collection container through a detachable collection container interface.

3. The drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The human-computer interaction interface includes a display screen and an input device, wherein the input device is a touch screen, a keyboard, or a combination thereof; The display screen is used to display the operation mode selection interface, visual recognition results, grinding quality assessment results, real-time weighing data, and warning information; the input device is used to receive the operation mode selected by the user and the drug information manually entered.

4. The drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The highlight suppression in the image preprocessing module is achieved by detecting high-brightness areas and using a texture reconstruction algorithm to recover the information of those areas.

5. A drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The cross-view instance association module is implemented in the following way: extracting mask features of adjacent views, calculating feature similarity between views and solving for optimal matching, verifying matching consistency, maintaining a global instance identifier table, and recording the mask position of each drug instance at different angles.

6. A drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The particle separation module is implemented in the following ways: morphological processing is performed on the initial segmentation mask; watershed segmentation is performed based on distance transformation; and over-segmented regions are merged using a clustering algorithm.

7. A drug segmentation system based on machine vision and semantic analysis according to claim 1, characterized in that, The instance segmentation module employs a deep convolutional neural network; the structured traceability data includes: traceability identifiers of drug instances, multi-view images, segmentation masks, recognition text, geometric dimensions before and after pulverization, powder particle size distribution, weighing data, and feeding process parameters.

8. A drug segmentation method based on machine vision and semantic analysis, characterized in that, Includes the following steps: S0: Receive the operation mode selected by the user through the human-computer interaction interface. If the manual mode is selected, receive the drug information entered by the user and jump to step S3. If the automatic mode is selected, proceed to step S1. S1: Adjust the vision head (32) to the position before crushing, drive the rotating / positioning stage (21) to rotate the medicine and packaging at multiple angles, and collect image sequences with angle indexes; call the image preprocessing module to perform color normalization, highlight suppression and view registration on the image sequences; The instance segmentation module is invoked to generate pixel-level segmentation masks for the drug body, packaging structure, and identification area; The cross-view instance association module is invoked to perform instance matching on the segmentation results under different views based on the angle index signal, and a unique traceability identifier is assigned to each drug instance. The identification module is invoked to perform text recognition or barcode decoding within the identification area, extracting batch number, expiration date, and dosage form information and establishing a binding relationship with the traceability identification; S2: Perform suitability judgment based on the identified dosage form information, query the preset contraindicated dosage form database, and if it is determined to be a contraindicated dosage form, output an interlock signal to prevent the micro-pulverizing unit (231) and the quantitative discharge actuator from starting, and output a warning message through the human-machine interface, and return to step S0 after waiting for user confirmation; if it is determined to be a permitted pulverizing dosage form, proceed to step S3; S3: Drive the micro-pulverizing unit (231) to pulverize the tablets, drive the sweeping assembly (232) to push the powder to the edge collection area (233), and the powder enters the weighing area of ​​the replaceable carrier plate (22); S4: Adjust the vision head (32) to the position after crushing and acquire the powder distribution image; call the image preprocessing module to preprocess the powder distribution image; call the particle separation module to perform morphological processing, watershed segmentation based on distance transformation, and region merging of the clustering algorithm on the initial segmentation mask; call the particle size statistics module to calculate the equivalent diameter of each separated particle and output the characteristic value of the particle size distribution; output the quality assessment result and powder flowability rating based on the particle size distribution and agglomeration degree. S5: Determine whether the grinding quality is qualified. If the median particle size exceeds the preset threshold, prompt for secondary grinding through the human-machine interface and return to step S3. If qualified, proceed to step S6. S6: Based on the real-time weighing data of the integrated weighing unit (22a) and the powder flowability rating, the quantitative discharge actuator is controlled by closed-loop feedback to perform staged feeding, including coarse feeding stage, micro feeding stage and micro replenishment stage. The feeding rate is adjusted according to the flowability rating until the target dose is reached. S7: Integrate the traceability identifier, identification information, image data before and after crushing, segmentation mask, powder particle size distribution, weighing data and feeding parameters into structured traceability data and output it; The highlight region suppression in S1 includes: detecting high-brightness regions, expanding the highlight region boundaries, and using a texture reconstruction algorithm to recover highlight region information; the cross-viewpoint instance association includes: extracting mask features from adjacent viewpoints, calculating feature similarity between viewpoints and solving for the optimal match, verifying matching consistency, and maintaining a global instance identifier table; the instance segmentation module uses a deep convolutional neural network, and simultaneously optimizes the segmentation loss, edge detection loss, and instance discrimination loss through a multi-task learning strategy; The processing flow of the particle separation module in S4 includes: performing morphological operations on the initial segmentation mask; calculating the distance transformation of the processed mask and detecting local maxima as seed points; executing the watershed algorithm with the seed points as the starting positions; calculating the feature vector for each watershed region, and using a clustering algorithm to determine independent particles and merge over-segmented regions; The phased feeding control strategy in S6 is as follows: in the coarse feeding stage, the feeding rate is at the maximum rate to approach the first preset ratio of the target dose; in the micro feeding stage, the feeding rate is reduced to the second preset ratio of the coarse feeding stage, approaching the third preset ratio of the target dose; in the micro supplementation stage, the feed rate is precisely supplemented to the target dose with pulsed intermittent feeding. In step S6, the feeding rate is adjusted according to the powder flowability rating: when the flowability rating is poor, the feeding rate in the coarse feeding stage is reduced and the vibration auxiliary device is activated; when the flowability rating is good, the standard feeding rate is adopted; when the flowability rating is excellent, the feeding rate in the coarse feeding stage is increased.

Images