Multi-specification medicine classification and sorting method based on ai visual recognition and computing device

Abstract

This invention relates to the field of intelligent drug classification and sorting technology, specifically disclosing a multi-specification drug classification and sorting method and computing device based on AI visual recognition. The method includes: in response to a trigger signal indicating that a drug has entered the sorting area, acquiring the current multimodal technical feature data of the drug; comparing the current multimodal technical feature data with a pre-set drug packaging benchmark feature library to determine whether the drug has undergone packaging changes. This invention introduces a dynamic elastic registration mechanism based on edge anchor points and utilizes a thin-plate spline interpolation model to perform non-rigid mesh deformation on the multimodal data. Without requiring re-physical calibration, the algorithm adaptively eliminates data alignment errors caused by soft packaging deformation during high-speed transmission, ensuring the purity of feature extraction.

Description

Technical Field

[0001] This invention relates to the field of intelligent drug classification and sorting technology, and in particular to a method and computing device for classifying and sorting multi-specification drugs based on AI visual recognition. Background Technology

[0002] Currently, automated sample receiving in pharmaceutical testing institutions and production line sorting in pharmaceutical manufacturing enterprises primarily rely on barcode scanning or static template matching technology based on traditional machine vision for classification and transfer. However, pharmaceutical testing institutions process massive amounts of samples from different manufacturers daily, and pharmaceutical manufacturers frequently update packaging designs, materials, or layouts during batch production changes, often resulting in a mix of old and new packaging for the same specification of drug. Existing visual recognition systems typically employ a pre-trained offline + online static deployment model. When faced with samples from testing institutions that haven't been updated with new packaging, or new packaging materials temporarily switched on the production line, the system often fails to recognize them or makes misjudgments, necessitating system shutdown to re-collect massive amounts of data for full model training. This rigid model update mechanism is significantly at odds with the dynamic nature of frequent packaging changes during pharmaceutical sampling and production, leading to high system maintenance costs and an inability to meet the continuity and compliance requirements of the entire pharmaceutical testing and production process.

[0003] Furthermore, in the pretreatment of pharmaceutical samples or the high-speed sorting and transport of finished products, a technical conflict exists between static rigid calibration and dynamic nonlinear deformation for widely available flexible-packaged pharmaceuticals, such as aluminum-plastic blister packs, infusion bags, or objects that undergo non-rigid deformation due to conveyor belt vibration. This is because existing multimodal sensor systems typically rely on a checkerboard-based initial static calibration matrix for spatial alignment. However, when flexible-packaged pharmaceuticals experience minute bending during high-speed sorting or when the sensor support experiences micron-level temperature drift, the static matrix cannot describe the local nonlinear deformation, leading to misalignment between spectral and texture data. For example, the spectrum of the conveyor belt background may be mismatched as the edge of the pharmaceutical product, severely affecting sorting accuracy.

[0004] Meanwhile, when faced with niche sample specifications newly received by drug testing institutions or drug specifications newly put into production by enterprises, only a very small number of standard samples can often be obtained in the early stages of actual operation. There is a fundamental conflict between the extreme sparsity of the samples and the high dependence of deep learning models on the variance of feature distribution. Directly using a small number of samples for fine-tuning can easily lead to the model overfitting to the accidental features of specific samples, such as specific lighting angles, thus failing to build robust classification boundaries and causing serious compliance risks and sorting errors. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, the objective of this invention is to propose a multi-specification drug classification and sorting method and computing device based on AI visual recognition, so as to minimize error rate and drug loss while ensuring sorting efficiency.

[0006] To achieve the above objectives, a first aspect of the present invention proposes a method for classifying and sorting multi-specification drugs based on AI visual recognition, comprising the following steps:

[0007] In response to a trigger signal indicating that the drug has entered the sorting area, the current multimodal technical characteristic data of the drug is collected;

[0008] The current multimodal technology feature data is compared with a pre-set drug packaging benchmark feature library to determine whether the drug packaging has been changed.

[0009] If so, the incremental model adaptation process is initiated, the classification model is updated based on the current multimodal technology feature data, and the updated classification model is used to classify the drug by specification, generating classification results and confidence scores.

[0010] If not, the current classification model is used directly to classify the drug by specification, and the classification results and confidence scores are generated.

[0011] Based on the classification results and confidence levels, a target sorting instruction for the drug is determined to drive the sorting execution mechanism to complete the physical sorting.

[0012] The process of determining whether the drug has undergone packaging changes includes: calculating the multi-dimensional similarity between the current multimodal technical feature data and the corresponding benchmark features in the drug packaging benchmark feature library; if the similarity in any dimension is lower than a preset change determination threshold, a packaging change signal is generated.

[0013] To achieve the above objectives, a second aspect of the present invention proposes a multi-specification drug classification and sorting system based on AI visual recognition, comprising:

[0014] The data acquisition module is used to collect the current multimodal technical feature data of the drug in response to the trigger signal that the drug enters the sorting area;

[0015] The change perception module is used to compare the current multimodal technology feature data with a preset drug packaging benchmark feature library to determine whether the drug packaging has changed.

[0016] The model adaptation and classification module is used to initiate an incremental model adaptation process to update the classification model and perform classification when a packaging change is detected; or to directly use the current classification model to perform classification when no packaging change is detected, and output the classification result and confidence score.

[0017] The sorting control module is used to determine the target sorting instruction for the drug based on the classification result and confidence level, so as to drive the sorting execution mechanism to complete the physical sorting.

[0018] The process of determining packaging changes in the change perception module includes: calculating the multi-dimensional similarity between the current multimodal technical feature data and the corresponding benchmark features in the drug packaging benchmark feature library; if the similarity in any dimension is lower than the preset change judgment threshold, a packaging change signal is generated.

[0019] To achieve the above objectives, a third aspect of the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, it implements the above-described method for classifying and sorting multi-specification drugs based on AI visual recognition.

[0020] The multi-specification drug classification and sorting method and computing device based on AI visual recognition in this invention introduces a dynamic elastic registration mechanism based on edge anchor points and uses a thin plate spline interpolation model to perform non-rigid mesh deformation on multimodal data. Without the need for re-physical calibration, the algorithm adaptively eliminates the data alignment error caused by soft packaging deformation during high-speed transmission, ensuring the purity of feature extraction.

[0021] To address the issue of sparse samples of new packaging, this solution utilizes feature illusion technology based on covariance transfer to extract feature distribution patterns from historical data of similar drugs and infer the variance of new samples. Without relying on massive amounts of data, it successfully constructs a classification model with robust boundaries, significantly improving the system's cold-start adaptability to new packaging. Finally, by combining digital twin damage prediction with blockchain secondary verification, a closed-loop guarantee is achieved for the entire process from physical sorting to logical compliance, minimizing error rates and drug losses while ensuring sorting efficiency. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the multi-specification drug classification and sorting method based on AI visual recognition provided by the present invention.

[0023] Figure 2 This is a comparison of image grayscale profiles and signal-to-noise ratios before and after multi-scale residual enhancement in the multi-specification drug classification and sorting method based on AI visual recognition provided by this invention.

[0024] Figure 3 This is a simulation curve showing the correlation between the gripping force of a robotic arm based on digital twins and the risk of drug breakage in the multi-specification drug classification and sorting method based on AI visual recognition provided by this invention.

[0025] Figure 4 This is a visualization diagram of the local position deviation vector field of multimodal data in the multi-specification drug classification and sorting method based on AI visual recognition provided by the present invention;

[0026] Figure 5 This is a simulation diagram of the elastic mesh deformation based on thin plate spline interpolation (TPS) in the multi-specification drug classification and sorting method based on AI visual recognition provided by the present invention.

[0027] Figure 6 This is the convergence curve of the normalized mutual information (NMI) value in the iterative registration process of the multi-specification drug classification and sorting method based on AI visual recognition provided by the present invention;

[0028] Figure 7 This is a comparison diagram of the virtual feature distribution cloud based on covariance transfer and the feature space distribution of real samples in the multi-specification drug classification and sorting method based on AI visual recognition provided by this invention.

[0029] Figure 8 This is a simulation diagram of the convergence trend of the training loss function containing orthogonal constraints in the multi-specification drug classification and sorting method based on AI visual recognition provided by the present invention.

[0030] Figure 9 This is a schematic diagram illustrating the implementation of the multi-specification drug classification and sorting system based on AI visual recognition provided by the present invention;

[0031] Figure 10 This is a schematic diagram of the structure of the computing device provided by the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0033] The following description, with reference to the accompanying drawings, outlines an embodiment of the present invention for a multi-specification drug classification and sorting method, system, and computing device based on AI visual recognition.

[0034] Example 1:

[0035] This embodiment details a multi-specification drug classification and sorting method based on AI visual recognition. This method constructs a dynamic closed loop of perception-adaptation-execution, combining multimodal data acquisition, dynamic packaging change perception, incremental model adaptation, hybrid architecture classification reasoning, digital twin simulation, and blockchain secondary verification to achieve accurate classification and compliant sorting of multi-specification drugs. This method is primarily implemented in the intelligent sorting control system of drug testing institutions or drug manufacturing enterprises.

[0036] like Figure 1 As shown, the method in this embodiment specifically includes the following:

[0037] S1, Drug entry triggering and multimodal data acquisition and preprocessing.

[0038] Specifically, the method flow of this embodiment begins with the sensing of the sorting object. In response to a trigger signal indicating that the medicine has entered the sorting area, the system first performs a data acquisition step. This trigger signal can be generated by a photoelectric switch, light curtain sensor, or weight sensor installed at the conveyor belt entrance. Once the medicine is detected to be in place, the system immediately controls the multimodal sensor group to synchronously acquire data.

[0039] It is important to note that, in order to acquire sufficiently rich physical and chemical characteristics to handle complex pharmaceutical packaging, the multimodal sensor group employs a specific hardware combination, including a color camera, a depth camera, a near-infrared sensor, and a terahertz sensor. The color camera is preferably a high-resolution industrial camera combined with a polarized light source to acquire visible light images of the pharmaceutical product, thereby obtaining appearance and specification markings, such as text, patterns, colors, and barcodes on the packaging box. The depth camera is preferably a 3D camera based on structured light or Time-of-Flight (ToF) technology to acquire the three-dimensional shape features of the pharmaceutical product, including its length, width, height, and surface texture. The near-infrared sensor is used to acquire the spectral characteristics of the pharmaceutical packaging material, identifying the packaging material by analyzing the absorption and reflection characteristics of near-infrared light, such as distinguishing between cardboard boxes, plastic bags, or aluminum foil. The terahertz sensor utilizes the penetrating power of terahertz waves to acquire the internal structural features of the pharmaceutical product, such as detecting the arrangement of internal blister packs or the liquid level of the medicine without damaging the packaging.

[0040] After acquiring the raw data, in order to eliminate environmental noise and highlight key features, the system also includes a data preprocessing step after acquiring the current multimodal technical feature data of the drug.

[0041] For example, this preprocessing step employs an image processing algorithm based on multi-scale residual enhancement. Specifically, the system first constructs a multi-scale Gaussian blurred image. Let the original image be... ,in and Represents pixel coordinates. The system utilizes two-dimensional Gaussian kernel functions at different scales. Perform a convolution operation on the original image to generate multiple blurred images. This process can be expressed by the formula:

[0042] ;

[0043] In the formula, The representative standard deviation is The two-dimensional Gaussian kernel function; A scale parameter used to control the degree of ambiguity; Values ,in This represents the total number of blur scale levels, for example, 3 or 5; * represents a two-dimensional convolution operation. Through this step, high-frequency noise in the original image is smoothed to varying degrees, while low-frequency structural information is preserved.

[0044] Next, the system extracts high-frequency detail residual features between the original image and the multi-scale Gaussian blurred image. This step aims to separate subtle texture and edge information in the image, which is crucial for recognizing tiny text on pharmaceutical packaging. Among these features, the detail residuals... The calculation formula is as follows:

[0045] ;

[0046] Subsequently, the system weights and fuses the high-frequency detail residual features with the original image based on residual weights to generate enhanced multimodal technical feature data. The fused enhanced image. The calculation is as follows:

[0047] ;

[0048] In the formula, For the corresponding number The residual weights for each scale typically range from 0 to 1. This weight parameter... The system can be dynamically adjusted based on the ambient light intensity or the sensor's signal-to-noise ratio. Noise can be further suppressed by a spatial-pixel joint guided filter, ultimately yielding high-quality augmented data for subsequent comparison and classification.

[0049] like Figure 2 The figure shows a comparison of the image grayscale profile and signal-to-noise ratio before and after multi-scale residual enhancement. This figure intuitively demonstrates the significant effect of the preprocessing algorithm described in this embodiment on improving image quality.

[0050] Figure 2The horizontal axis represents the spatial coordinates of the pixel position across the tiny text or barcode texture on the surface of the drug packaging, and the vertical axis represents the corresponding pixel grayscale intensity value.

[0051] Figure 2 The gray fluctuating curve represents the original acquired image signal. It can be seen that the curve has violent random oscillations in the flat area, which corresponds to the high-frequency environmental noise introduced during the sensor acquisition process. This noise reduces the signal-to-noise ratio and obscures weak edge features.

[0052] In comparison, Figure 2 The smooth red curve represents the image signal after residual enhancement following multi-scale Gaussian blurring and weighted fusion of detail residuals. A comparison clearly shows that the fluctuation amplitude of the red curve in the low-grayscale background area and the high-grayscale text area is significantly compressed. This indicates that the multi-scale Gaussian kernel function effectively smooths background noise and significantly improves the image's signal-to-noise ratio.

[0053] More importantly, at the edges where grayscale values ​​transition, the red curve not only doesn't become blurred, but instead exhibits a steeper upward and downward slope than the gray curve. This waveform transformation characteristic directly reflects the algorithm's extraction and backfilling of high-frequency detail residuals. This effect of suppressing background noise while directionally enhancing edge gradients strongly demonstrates that the present invention can more easily and accurately identify minute specification markings on pharmaceutical packaging in subsequent feature comparison steps, thereby ensuring high purity and robustness in extracting blurred or subtle features in high-speed sorting environments.

[0054] It is also important to note that since different sensors are installed in different locations in the physical space and their acquisition frequencies may have slight differences, strict spatiotemporal alignment is necessary. According to this embodiment, the system uses an affine transformation matrix to spatially align the aforementioned feature data based on a pre-constructed unified spatial coordinate system, and completes timing synchronization based on a unified timestamp.

[0055] Specifically, during the system initialization phase, a unified spatial coordinate system is established using a checkerboard calibration method. By utilizing the coordinate correspondence of checkerboard corner points in different sensor fields of view, the rotation and translation matrices of each sensor relative to the principal coordinate system are calculated, and then an affine transformation matrix is ​​synthesized. In actual operation, the system uses this affine transformation matrix to map depth and spectral data onto the coordinate plane of the color image, ensuring that the same pixel corresponds to the color, depth, and spectral information of the same physical location on the drug. The spatial alignment meets a preset alignment precision. For example, the reprojection error is less than 0.1 mm to ensure precise correspondence between minute text and depth contours.

[0056] Meanwhile, the system uses a programmable logic controller (PLC) to trigger hardware synchronous acquisition, giving all data streams a unified nanosecond-level timestamp, ensuring that the timing synchronization meets the preset time synchronization threshold, for example, the synchronization error is less than 1 millisecond, thereby avoiding motion blur or data misalignment caused by the movement of the conveyor belt.

[0057] S2. Packaging change perception and judgment mechanism.

[0058] After acquiring high-quality current multimodal technology feature data, the system enters the core perception and judgment stage. The system compares the current multimodal technology feature data with a pre-set drug packaging benchmark feature library to determine whether the drug packaging has changed. This step is the key difference between this embodiment and traditional static matching technology, as it introduces dynamic perception logic.

[0059] Specifically, the process of determining whether the drug has undergone packaging changes includes multi-dimensional similarity calculations. The system calculates visual feature similarity, material spectral similarity, and three-dimensional size similarity, respectively.

[0060] For example, visual feature similarity can be obtained by calculating the cosine similarity or structural similarity (SSIM) between the current image feature vector and the corresponding feature vector in the benchmark library; material spectral similarity can be obtained by calculating the spectral angular distance (SAM) between the current spectral curve and the benchmark spectral curve; and three-dimensional size similarity can be obtained by calculating the Euclidean distance deviation between the current drug point cloud bounding box size and the benchmark size.

[0061] In addition to physical feature comparison, this embodiment also introduces semantic-level verification. The system calls a semantic text recognition model, such as a large OCR model based on the Transformer architecture, to parse the packaging text in the current multimodal technology feature data, verifying the consistency of specification keywords, numbers, and unit combinations. For example, the system will extract key information such as 500mg, 0.25g, and 10ml from the packaging and check whether their arrangement and values ​​completely match the information recorded in the benchmark library.

[0062] Based on the above calculation results, the system executes a strict change judgment logic: when the visual feature similarity is lower than the first preset similarity threshold (e.g., 95%), or the material spectral similarity is lower than the second preset similarity threshold (e.g., 98%), or the three-dimensional size similarity deviation exceeds the preset tolerance range (e.g., the length, width, and height deviation exceeds 0.5mm), or the packaging text combination verification fails, the system will determine that a packaging change has occurred and generate a packaging change signal.

[0063] S3, Model Incremental Adaptation and Classification Inference.

[0064] Based on the results of the above determination, the system will execute different processing branches.

[0065] Branch 1: If the determination result is yes, meaning a packaging change has occurred, the system initiates the model incremental adaptation process. This process aims to address the pain point of traditional deep learning models needing to be fully retrained when faced with new data.

[0066] Specifically, the process of updating the classification model based on the current multimodal technology feature data is as follows: First, the system collects a new packaging sample set of the drug. In an actual production line, this may mean that the system temporarily caches and continuously collects image data of the first few drugs in the current batch, or prompts manual assistance to collect a small number of multi-angle images. Next, the system extracts close-up features of the changed areas in the new packaging sample set. For example, if only the logo color on the packaging has changed, or the position of the dosage label has moved, the system will use an attention mechanism to focus on extracting feature maps of these changed areas.

[0067] Subsequently, to prevent catastrophic forgetting—that is, learning the new packaging but forgetting the old packaging—the system employs a training freeze strategy. Specifically, the system freezes the parameters of the classification model's backbone network (such as the convolutional layer parameters of ResNet or EfficientNet). These layers are responsible for extracting common texture and edge features, which are typically shared between the new and old packaging. Simultaneously, the system unfreezes the parameters of the fully-connected layers that are relevant to specification features.

[0068] Finally, the system utilizes the close-up features of the changed region and the common features of similar drugs in the drug packaging benchmark feature library to perform local fine-tuning training on the fully connected layer, generating an adapted updated model. Here, the common features of similar drugs refer to the feature distribution patterns stored in the benchmark library that belong to the same major category as the current drug, such as both being aluminum-plastic capsules. Then, by mapping the features of new samples to this distribution pattern, the system can quickly adjust the weight boundaries of the fully connected layer with a very small sample size, achieving accurate adaptation to the new packaging.

[0069] Branch 2: If the determination result is negative, that is, no packaging change has occurred, or after the above incremental adaptation has been completed, the system will use the updated classification model or directly use the current classification model to classify the drug by specification, and generate classification results and confidence levels.

[0070] It should be noted that the classification model described in this embodiment adopts a robust hybrid architecture, which integrates attention mechanism and graph neural network (GNN) in its processing.

[0071] First, the attention mechanism module focuses on the specification identification region in the current multimodal technical feature data. Since there is a large amount of interfering information on drug packaging, such as advertising slogans and manufacturer addresses, the attention module generates a spatial weight map and assigns a higher weight coefficient to regions combining numbers and units (e.g., 0.5g) than to non-specification regions. The weights typically range from 0 to 1, but the weights of key regions may be dynamically increased to 0.6 to 0.7 or higher to dominate the classification decision.

[0072] Secondly, to uncover the topological relationships between features, the system utilizes a graph neural network module to construct a drug region graph structure. Let the graph structure be... ,in For a set of nodes, This is the set of edges. The system treats each sub-region after image segmentation as a node and extracts the statistical attributes of the region as node feature vectors. .

[0073] Specifically, node features These include components such as the region's average disturbance intensity, standard deviation, skewness, kurtosis, profile compactness, and region's centroid coordinates. These statistics can effectively describe the texture roughness and shape regularity of a region.

[0074] Simultaneously, the system establishes graph edge relationships based on spatial and similarity thresholds. For any two nodes... and If their spatial distance is less than the spatial threshold Furthermore, the feature similarity, such as cosine similarity, is greater than the similarity threshold. Then an edge is established between them. This can be expressed as:

[0075] ;

[0076] Graph neural networks combine local specification features with global packaging structure features by passing messages and aggregating them on these edges, outputting a final high-dimensional feature vector.

[0077] Finally, the feature vector is input into the classification decision layer, usually the Softmax layer, and the classification result (e.g., amoxicillin capsules 0.25g x 24 capsules) and the corresponding confidence level (e.g., 80.9%) are output.

[0078] S4. Generation of sorting instructions based on digital twins.

[0079] After obtaining the classification results, the system does not blindly perform sorting directly, but introduces a digital twin simulation process to ensure the safety of physical execution, especially for the protection of fragile medicines.

[0080] Specifically, the system determines the target sorting instruction for the drug based on the classification results and confidence levels. This process includes: first, loading the classification results and the three-dimensional model parameters of the drug into a digital twin simulation environment. This simulation environment is a virtual mirror of the physical sorting workstation, containing the kinematic model of the robotic arm, the physical properties of the suction cups or grippers, and the real-time speed of the conveyor belt.

[0081] Next, the gripping path and force of the sorting actuator are simulated in the digital twin simulation environment. The system attempts to plan an optimal trajectory from the conveyor belt to the target medicine box and simulates the force at the moment the robotic arm's end effector contacts the medicine. The system calculates the breakage risk value of the medicine. This calculation takes into account the brittleness of the medicine material (from a benchmark library), the deformability of the packaging (from depth camera data), and the simulated gripping pressure.

[0082] Subsequently, the system makes decisions based on the risk value: if the damage risk value is lower than a preset risk threshold, such as 0.1%, the grasping scheme is deemed safe, and the system generates the target sorting instruction containing the target storage location and grasping parameters. This instruction not only tells the robot where to go but also how much force to apply and at what angle to grip. If the damage risk value is higher than or equal to the preset risk threshold, it indicates that the current sorting strategy may damage the medicine, and the system generates an abnormal alarm instruction, prompting manual intervention or moving the medicine to an abnormal buffer area.

[0083] like Figure 3 The diagram shows a simulation of the correlation between the gripping force of a robotic arm based on digital twins and the risk of drug breakage. This diagram vividly illustrates the core decision-making logic of the sorting control module in this embodiment, which uses a digital twin engine to ensure the safety of fragile drugs.

[0084] Figure 3 The horizontal axis represents the gripping force applied by the end effector of the robotic arm to the surface of the medicine, and the vertical axis represents the predicted probability of medicine breakage calculated by the system based on the brittleness coefficient of the medicine material and the three-dimensional deformation capability of the packaging.

[0085] Figure 3 The blue nonlinear rising curve is the correlation curve between gripping force and breakage risk. Its trend shows that the risk of breakage of the drug is extremely low and remains stable when subjected to low gripping force. However, once the force exceeds a certain critical physical range, the risk of breakage will increase exponentially. This is highly consistent with the stress failure law of brittle materials in the real physical world.

[0086] Figure 3 The horizontal dashed red line represents the system's preset safety threshold for damage risk, which is 0.1 percentage points. This threshold line visually divides the chart space into a safe operating area below and a danger warning area above.

[0087] In actual operation, the digital twin system calculates the risk points corresponding to the current planned force on the blue curve in real time. If the risk point is below the red dotted line, it means that the probability of damage is within an acceptable safety range. The system will generate sorting instructions containing the target storage location and grasping parameters to drive the physical mechanism to execute. Conversely, if the risk point crosses the red dotted line and enters the upper area, the system will immediately determine that the current grasping strategy has a high risk, thereby triggering an abnormal alarm instruction and refusing to execute physical grasping, thus achieving preventive protection for high-value or fragile medicines.

[0088] S5, physical execution and blockchain secondary verification.

[0089] After receiving the target sorting instruction, the sorting execution mechanism, such as an industrial robot or a pneumatic push rod, drives the mechanical structure to complete the physical sorting action and move the medicine to the designated logistics channel or storage location.

[0090] However, in order to meet the extremely high compliance requirements (GMP / GSP) of the pharmaceutical industry, this embodiment also sets up a rigorous secondary verification step after the physical sorting action to form a closed loop.

[0091] Specifically, after the drive sorting actuator completes the physical sorting, the system controls a microscope camera and a terahertz sensor to collect the characteristics of the sorted medicines. The microscope camera is used to reconfirm the specifications and printing details of the medicines at a microscopic level, verifying their match with the sorting results; the terahertz sensor can then reconfirm whether the contents of the packaging were damaged during the sorting process.

[0092] Simultaneously, the system performs compliance verification. The system reads the drug's unique traceability identifier, such as an electronic regulatory code or RFID tag, and queries the blockchain's evidence data. The blockchain stores all the drug's flow records from factory to the current stage, as well as its pre-assigned compliant storage location. The system verifies the consistency between the current physical sorting location, i.e., the location where the robot just placed the drug, and the compliant storage location recorded on the blockchain.

[0093] Finally, the system takes action based on the verification results: if the matching degree is lower than a preset verification pass threshold (e.g., set to 99.5%) or the consistency verification fails (e.g., medicine that should have been sent to the cold chain area was sorted into the room temperature area), the system triggers an anomaly handling process and generates an anomaly traceability report. This report includes information such as the time and location of the anomaly, the drug ID involved, and the hash value of the original sensor data, and is automatically uploaded to the blockchain for evidence storage, so as to facilitate subsequent compliance audits and root cause analysis.

[0094] Through the above steps, this embodiment not only realizes a technically automated classification method, but also constructs an intelligent sorting system with adaptive capabilities that meets the strict regulatory requirements of the pharmaceutical industry.

[0095] Example 2:

[0096] This embodiment, based on Embodiment 1, further elaborates on a particularly challenging special case in drug sorting scenarios. In actual drug testing and sorting operations, a large number of drugs are not packaged in rigid cardboard boxes, but rather in non-rigid packaging such as aluminum-plastic blister packs, infusion bags, or composite film bags. When these non-rigid drug packages are transported at high speeds of several meters per second on conveyor belts, they are easily subjected to minute twists, bulges, or vibrations that are difficult to detect with the naked eye due to mechanical vibrations, sudden stops, or acceleration inertia. This physical phenomenon is called non-rigid deformation.

[0097] Specifically, in this case, the global static affine transformation matrix based on checkerboard calibration mentioned in Example 1 can often only solve the alignment problem of rigid displacements, but cannot describe local, nonlinear geometric deformations.

[0098] For example, the edge of a medicine box may bend at the micrometer level due to vibration, causing the texture information captured by the color camera and the contour information captured by the depth camera to not precisely overlap spatially. This discrepancy directly leads to subsequent spectral analysis potentially extracting the material features of the background conveyor belt instead of the features of the medicine itself. Therefore, the method in this embodiment, after initial alignment using an affine transformation matrix, further includes a dynamic elastic registration correction step to ensure high-precision fusion of multimodal data at the pixel level. This embodiment specifically includes the following:

[0099] Step 1: Extraction and definition of dynamic geometric anchor points.

[0100] It is worth noting that among multimodal sensor arrays, color cameras typically have the highest resolution and the richest high-frequency texture information, making them the best choice for establishing dynamic benchmarks.

[0101] Specifically, the system first extracts high-frequency edge contour features from the images captured by the color camera. The system uses the Difference of Gaussian (DoG) operator or the Canny edge detection operator to convolve the pre-aligned color image, extracting the edge lines with the strongest contrast on the drug packaging, such as the rectangular outline of the drug, the circular edge of the blister pack, or the stroke boundaries of large fonts. The system defines these features, extracted in real-time in the current frame and reflecting the drug's current instantaneous physical form, as dynamic geometric anchor points. These anchor points form the reference skeleton for deformation of all subsequent modal data.

[0102] Step 2: Construction of the local position deviation vector field.

[0103] After establishing the reference skeleton, the system needs to calculate the offset of other modal data relative to the skeleton.

[0104] For example, the system maps the depth data acquired by the depth camera and the spectral data acquired by the near-infrared sensor to the unified spatial coordinate system. Since only a rigid affine transformation is performed at this point, there is often a slight misalignment between the mapped depth data contour and the dynamic geometric anchor points of the color image. The system searches and matches between the depth edge image and the color edge image using a contour-based Iterative Closest Point (ICP) variant algorithm or dense optical flow.

[0105] Specifically, for each feature point on the edge of the depth image, the system searches for the nearest corresponding point near the dynamic geometric anchor point of the color image and calculates the displacement vector between them. The displacement vectors of all feature points are then aggregated to calculate the local positional deviation vector field between the mapped data contour and the dynamic geometric anchor point. This vector field is not a single numerical value, but a two-dimensional vector matrix covering the entire image region, describing how much and in which direction each local region in the image needs to be moved to perfectly match the color image.

[0106] like Figure 4 The diagram shown is a visualization of the local positional deviation vector field of multimodal data. This diagram intuitively illustrates how the system in this embodiment accurately quantifies the minute deformations of the surface of non-rigid pharmaceutical packaging.

[0107] Figure 4 The horizontal and vertical axes represent the horizontal and vertical pixel spatial coordinates in a unified spatial coordinate system, covering a typical local area of ​​the pharmaceutical packaging.

[0108] Figure 4 The variations in the background color visually reflect the strength of the positional deviation at each point within the area. Dark blue areas indicate smaller deviations, while bright yellow areas indicate significant physical deformation caused by mechanical vibration or packaging bulging.

[0109] Figure 4 The array of white arrows superimposed on the background represents the calculated local positional deviation displacement vector. The direction of each arrow precisely indicates the misalignment direction of the current modal data relative to the dynamic geometric anchor point of the color image, while the length of the arrow corresponds to the magnitude of the misalignment distance.

[0110] For example, in the image edge region, the arrows exhibit a clear rotational divergence trend and a brighter background color, vividly revealing the nonlinear distortion phenomenon that occurs at the edges of flexible packaging during high-speed transmission. By constructing this refined vector field, the system successfully decomposes complex non-rigid deformation into computable pixel-by-pixel displacement data, providing a solid mathematical basis for subsequently solving the minimum bending energy transformation parameters using a thin-plate spline interpolation model, and ensuring accurate pixel-level alignment of multimodal data after elastic correction.

[0111] Step 3: Constructing a non-rigid model based on thin plate spline interpolation.

[0112] After obtaining the deviation vector field, the system needs to construct a mathematical model to describe this complex torsion transformation. This embodiment adopts a non-rigid transformation model based on thin-plate-spline interpolation (TPS).

[0113] It is also important to note that thin-plate spline interpolation is a physical analogy method that simulates the physical state of an infinitely large thin metal plate under the traction of several control points, namely the aforementioned edge feature points, during bending deformation. This deformation ensures both precise alignment at the control points and a smooth transition in the non-control point areas, making it very suitable for describing the continuous flexible deformation of pharmaceutical packaging.

[0114] Specifically, the core of this step lies in using the local position deviation vector field as a constraint to solve for the transformation parameters that minimize the non-rigid bending energy. The system defines the coordinates from the source image. to target image coordinates mapping function The mapping function consists of two parts: an affine component describing the global trend and a non-affine component describing the local deformation. Its mathematical expression is shown below:

[0115] ;

[0116] In the formula, Represents the pixel coordinates in the mode to be corrected (such as a depth map); Represents the affine transformation coefficients, used to maintain the global linear correspondence; This represents the number of control points selected, i.e., the number of feature points in the vector field mentioned above; Representing the The coordinates of the control points; Representing the The weight coefficient of each control point determines the influence of that point on the deformation of the surrounding area. Represents Euclidean distance.

[0117] In the formula, Defined as a radial basis function, its specific mathematical form in the thin-plate spline model is as follows:

[0118] ;

[0119] In the formula, Represents the current pixel to the 1st pixel. Control points The distance. The physical meaning of this function is that it minimizes the bending energy generated by deformation. To solve for the unknown parameters in the above formula... as well as The system constructs a system of linear equations that includes interpolation conditions and bending energy minimization conditions. Solving this system of equations yields the unique transformation parameters describing the current deformation state of the drug.

[0120] Step 4: Execution of elastic mesh deformation.

[0121] After calculating the transformation parameters, the system performs actual image correction.

[0122] Specifically, the system uses the transformation parameters to perform pixel-by-pixel elastic mesh deformation on the depth data and the spectral data. The system treats the depth image and the spectral image as an elastic rubber mesh, according to the aforementioned mapping function. The system calculates the new coordinates of each pixel in the original image within the corrected image. Since the calculated new coordinates are usually not integers, the system uses bilinear or bicubic interpolation algorithms to calculate the pixel value at the new coordinates. Through this process, the depth and spectral information, which were originally misaligned with the color texture due to packaging bulges or bends, are forcibly and semi-slippery stretched and distorted until the edge contours perfectly coincide with the dynamic geometric anchor points of the color image, thereby eliminating local nonlinear deformation errors.

[0123] like Figure 5 The figure shown is a simulation result of elastic mesh deformation based on thin plate spline interpolation TPS. This figure intuitively demonstrates the geometric principle and effect of using a non-rigid transformation model to perform spatial correction on multimodal data in this embodiment.

[0124] Figure 5 The gray dashed grid in the background represents a standard unified spatial coordinate system, corresponding to a rigid geometric structure that has not undergone deformation under ideal conditions.

[0125] Figure 5 The superimposed blue solid line grid represents the elastic deformation grid calculated by the thin plate spline interpolation model. This grid simulates the nonlinear transformation relationship used when depth data and spectral data are mapped to a unified coordinate system.

[0126] from Figure 5 It can be clearly observed that the central area of ​​the grid exhibits a distinct outward expansion and smooth bending shape. This waveform transformation trajectory realistically simulates the physical bulging deformation of soft-packaged medicines caused by internal compression or mechanical vibration during high-speed transportation. Figure 5 The scattered red dots represent the extracted dynamic geometric anchor points, which act as strong constraints to firmly lock the blue mesh at specific feature locations.

[0127] By comparing the spatial differences between the gray dashed-line nodes and the blue solid-line nodes, the amount of displacement compensation performed by the algorithm on each pixel can be intuitively quantified. This continuous and smooth mesh distortion effect proves that the present invention can precisely pull back and reconstruct multimodal data undergoing non-rigid deformation pixel by pixel, just like stretching elastic skin, thereby ensuring that extremely high alignment accuracy can still be maintained between different modal data under complex working conditions.

[0128] Step 5: Verification of normalized mutual information values ​​and closed-loop control.

[0129] While the above algorithm can theoretically correct deformation, in industrial settings, extreme deformation or lighting interference can cause the registration algorithm to fail, for example, misidentifying shadows as edges. Therefore, an independent verification metric must be introduced.

[0130] For example, the system calculates the normalized mutual information (NMI) value between multimodal data after elastic mesh deformation. Mutual information is a metric that measures the statistical dependence between two random variables. It does not require the two images to have the same grayscale distribution and is well-suited for measuring the alignment between color images (reflecting reflectivity) and depth images (reflecting distance) or spectral images (reflecting material).

[0131] Specifically, the system first calculates the corrected color image. With depth images joint probability distribution and their respective marginal probability distributions and .in, and Representing images respectively and images The system calculates the Shannon entropy of the pixels based on their grayscale values ​​and probability distributions. , and joint entropy .

[0132] The formula for calculating Shannon entropy is:

[0133] ;

[0134] Based on this, normalize the mutual information value The calculation formula is defined as follows:

[0135] ;

[0136] The physical implication of this formula is that if two images are perfectly aligned, they contain the most shared information and have a relatively small joint entropy. The value reaches its peak; if alignment fails, the pixel distribution of the two images becomes disordered, the shared information decreases, and the joint entropy increases. The value decreased.

[0137] Finally, the system will calculate the results. The value is compared with a preset standard. Only when the normalized mutual information value is greater than a preset registration confidence threshold, such as 1.15, does the system determine that spatial alignment is complete and outputs feature data to the subsequent classification module. This indicates that the multimodal data has successfully overcome non-rigid deformation and achieved pixel-level fusion accuracy.

[0138] like Figure 6 The figure shows a schematic diagram of the convergence curve of the normalized mutual information (NMI) value during the iterative registration process. This figure quantitatively demonstrates how the closed-loop control system in this embodiment determines the effectiveness of spatial alignment based on statistical indicators.

[0139] Figure 6 The horizontal axis represents the number of iterations and corrections performed by the non-rigid transformation model when solving for the minimum bending energy parameter, and the vertical axis represents the normalized mutual information value between the calculated color image and the depth or spectral image.

[0140] Figure 6 The blue rising curve depicts how, as the elastic grid deformation is gradually implemented, the spatial overlap between multimodal data continuously increases, resulting in a decrease in the joint entropy of the two images and a rapid increase in mutual information values, which eventually tends to stabilize and converge. Figure 6 The horizontal dashed red line represents the preset registration confidence threshold set by the system, which has a value of 1.15.

[0141] The trend of the curve in the figure clearly shows that the mutual information value is lower than the red dotted line in the early stage of iteration, indicating that the correction has not yet been completed. As the number of iterations increases, the blue curve successfully crosses and stabilizes above the red dotted line at a certain moment. This crossing action constitutes the mathematical trigger condition for the system to determine that spatial alignment has been completed and output feature data. This proves that the verification mechanism can effectively prevent data that fails to align due to excessive deformation from entering the subsequent classification process, ensuring the reliability of the system under extreme conditions.

[0142] Conversely, if the normalized mutual information value is lower than the preset registration confidence threshold, it indicates that the current deformation is too drastic and exceeds the algorithm's correction capability, or that the sensor has undergone physical displacement, causing the static calibration parameters to fail severely. In this case, the system generates a calibration parameter failure signal and triggers a hardware recalibration process. This process may include pausing the sorting line, controlling the robotic arm to grasp the standard checkerboard calibration plate into the field of view, or prompting maintenance personnel to check the tightness of the sensor bracket, thus forming a closed-loop control system with self-diagnosis and self-recovery capabilities.

[0143] Through the detailed description of this embodiment, the present invention not only solves the problem of drug classification under normal conditions, but also overcomes the identification problem caused by the deformation of soft-packaged drugs on high-speed production lines by introducing dynamic elastic registration technology based on physical models, thus ensuring the robustness and reliability of the system under extreme conditions.

[0144] Example 3:

[0145] This embodiment, building upon Embodiments 1 and 2, provides a detailed explanation of the high-frequency and highly challenging scenario of introducing new specifications or making temporary packaging changes during drug production and sorting. In actual drug testing facilities, when a new drug specification is introduced, or when there are minor adjustments to the material or printing of existing drug packaging, the production line often cannot immediately provide thousands of training images covering various lighting angles and placement postures. Typically, only a handful of standard sample images can be collected on-site.

[0146] It's also important to note that existing deep learning classification algorithms typically rely on massive amounts of data to fit the spatial distribution of features. If these few samples are used directly for fine-tuning and training, the model will only learn the point features of these specific samples, failing to grasp the volumetric characteristics of the drug in the actual complex sorting environment, i.e., the variance of the feature distribution. This can lead to severe overfitting; if subsequent drugs experience slight rotation, reflection, or positional shifts on the conveyor belt, the model may misclassify them because the feature has fallen outside the range of those isolated points.

[0147] To address the problem that the sample size in the new packaging sample set is insufficient to cover the variance of the physical characteristics of the new packaging in the actual production environment, resulting in insufficient model generalization ability, the method in this embodiment specifically executes a feature distribution correction process based on covariance transfer when starting the model incremental adaptation process.

[0148] Step 1: Calculation and definition of the initial feature prototype.

[0149] When the change perception module determines that a packaging change has occurred and triggers the adaptation process, the system first performs the step of calculating the mean vector of features of the changed areas in the new packaging sample set. Assume the system has collected... A sample image of the new packaging ( Typically small integers, such as 3 to 5, are used to extract a set of feature vectors after passing through the backbone network. Although these feature vectors are few in number, their high-dimensional coordinates accurately reflect the core visual attributes of the new packaging under the current specific acquisition conditions.

[0150] The system then aggregates these discrete feature vectors into a central vector through mathematical averaging. This central vector eliminates random noise from a single sampling and represents the geometric center of the new class in the feature space. The system defines this mean vector as the initial feature prototype of the new class, denoted as . The calculation formula is expressed as follows:

[0151] ;

[0152] The initial feature prototype Although it accurately locates the position of the new drug's features, it lacks information on volume and shape, meaning it does not know in which direction the features will diverge under different environmental disturbances.

[0153] Step 2: Retrieval and transfer of feature covariance matrix.

[0154] In order to endow this isolated feature prototype with environmental adaptability, this embodiment introduces the core idea of ​​covariance transfer.

[0155] For example, the system performs the step of retrieving similar drugs from the drug packaging benchmark feature library that match the material and shape attributes of the new packaging sample set. The underlying physical logic is that although the printed patterns of the old and new drugs are different, if their physical materials are the same (e.g., both are polyethylene plastic bottles) and their geometric shapes are similar (e.g., both are cylinders), then under the same conveyor belt environment and the same light source, their reflective properties, shadow variation patterns, and blurring patterns caused by vibration will be highly consistent. Mathematically, this variation pattern is represented by the second-order statistics of the eigenvectors, namely the covariance matrix.

[0156] Specifically, the system queries the metadata of the benchmark database to find an old category (Base-Class) with sufficient data and similar physical attributes. The system extracts the feature covariance matrix of the aforementioned drugs in the historical full dataset, denoted as... This matrix is ​​a matrix with dimension 1. Symmetric matrix ( The feature covariance matrix (where is the feature dimension) has diagonal elements representing the variance of each feature dimension, i.e., the degree of change, while off-diagonal elements represent the correlation between different feature dimensions, i.e., the pattern of linked changes. Therefore, the feature covariance matrix characterizes the distribution pattern of feature changes of this type of drug in the sorting environment.

[0157] The system assumes that the feature distribution shape of the new category is approximately the same as the distribution shape of the baseline category, thus directly applying... The covariance matrix serves as a new category. This operation, mathematically known as the transfer of statistics, is equivalent to directly knowing the range of possible visual variations of a new drug in the future based on prior knowledge, without any new data.

[0158] Step 3: Generation of Virtual Feature Distribution Clouds

[0159] The center position (initial feature prototype) was obtained. ) and shape range (transferred covariance matrix) After that, the system has the ability to reason about more samples.

[0160] Specifically, the system constructs a feature distribution generator that does not generate pixel-level images but directly generates data in a high-dimensional feature space. The system then performs a step of generating virtual enhanced feature vectors using multi-dimensional Gaussian sampling, centered on the initial feature prototype and constrained by the feature covariance matrix.

[0161] In this process, the system assumes that the features of the new category follow a multidimensional Gaussian distribution. In order to sample and generate virtual feature vectors. The system first processes the covariance matrix. Perform Cholesky Decomposition or Singular Value Decomposition (SVD) to decompose it into a lower triangular matrix. The product of its transpose, i.e. , where the matrix It can be understood as a transformation basis of the feature space.

[0162] Subsequently, the system generates a random noise vector that follows a standard normal distribution. The dimension of this vector is the same as the feature dimension. The generator uses the following transformation formula to generate virtual feature vectors:

[0163] ;

[0164] In the formula, This is a scaling factor used to control the generation range, typically set to 1.0 or fine-tuned according to the desired level of discreteness. Random noise is generated through repeated iterations. The system can continuously generate hundreds or thousands of [items / items]. These vectors surround The direction and extent of its spread are entirely determined by the surrounding environment. The decision was made. These massive sets of virtual vectors came together to form a virtual feature distribution cloud. This cloud filled the gaps between real samples in the feature space, simulating the characteristic performance of the new drug under various unseen lighting and angles, thus achieving feature-hallucination.

[0165] like Figure 7 The diagram shows a comparison between the virtual feature distribution cloud based on covariance transfer and the feature space distribution of real samples. This diagram intuitively reveals the core principle of the few-shot learning reinforcement algorithm in this embodiment, which uses statistical prior knowledge to expand the data distribution.

[0166] Figure 7 The horizontal and vertical axes represent the two principal component dimensions of the high-dimensional feature space after dimensionality reduction and projection.

[0167] Figure 7 The large solid blue dots clustered at the geometric center represent a very small number of real samples of new packaging collected by the system during the cold start phase. As can be seen from the figure, these samples are extremely sparse and compact, and in fact, they are isolated points in the vast feature space. They cannot cover the range of complex feature changes caused by light fluctuations or angle tilts in the real environment.

[0168] The dense red dots surrounding the blue dots constitute the virtual feature distribution cloud generated by the system. The distribution range of this cloud and the specific elliptical diffusion direction it presents are entirely determined by the covariance matrix transferred from historical data of similar drugs. This signifies that the algorithm has successfully used prior knowledge to fill in the potential changes that new samples may undergo under different physical conditions.

[0169] Figure 7 The gray dashed line marks the theoretical statistical boundary of the distribution. By using this virtual feature cloud, which has both volume and a specific directional evolution trend, to replace the single mean point to train the classification model, the system can extend the decision boundary from a narrow overfit point estimate to a robust probability space coverage. This strongly proves that the method can still construct a recognition boundary with high generalization ability under conditions of very few samples.

[0170] Step 4: Loss function and model training based on orthogonal constraints.

[0171] After acquiring sufficient (albeit dummy) training data, the system needs to update the weights of the fully connected layers in the classification model. However, direct training may cause the model to compromise its ability to recognize old classes in order to adapt to new classes, a phenomenon known as catastrophic forgetting. To address this issue, this embodiment introduces an orthogonal constraint mechanism.

[0172] Specifically, the system constructs a loss function that includes orthogonal constraints, and uses the virtual feature distribution cloud to train the fully connected layer. This loss function... It consists of two parts: classification loss and orthogonal constraint loss .

[0173] The classification loss typically employs the cross-entropy loss function, which aims to maximize the model's classification accuracy for samples in a virtual feature distribution cloud, forcing the classifier's decision boundary to encompass the entire cloud.

[0174] The orthogonality constraint is used to force the weight vector of the updated model to maintain orthogonal separation from the original drug classification weight vectors in the baseline feature library in the feature space. Let the weight vector of the new category be... The set of weight vectors for the old category is Orthogonal constraint loss Aiming to minimize With all The cosine similarity between them. Its mathematical expression is as follows:

[0175] ;

[0176] The system minimizes the total loss function:

[0177] ;

[0178] In the formula, This is the balance coefficient.

[0179] This training method allows the model to find a direction in the feature space that is as perpendicular or as far away from all old categories as possible to place the decision hyperplane for the new category. This not only ensures that the model can accurately identify new drugs with rich variations through training on virtual clouds, but also minimizes interference with the original category feature space through orthogonal constraints, thereby expanding the ability to recognize new categories while suppressing catastrophic forgetting of old categories.

[0180] like Figure 8 The figure shown is a simulation diagram of the convergence trend of the training loss function with orthogonal constraints. This figure intuitively reflects the dynamic convergence process of the fully connected layer of the classification model during local fine-tuning training in this embodiment.

[0181] Figure 8 The horizontal axis represents the number of training iterations for local fine-tuning of the model, the vertical axis on the left represents the magnitude of the training loss function, and the vertical axis on the right represents the percentage of accuracy in identifying new categories.

[0182] Figure 8 The solid blue curve represents the change curve of the classification loss value of the new category. It can be seen that its value decreases exponentially and rapidly with the increase of the number of iterations. This indicates that the model is rapidly fitting and mastering the new packaging feature patterns provided by the virtual feature distribution cloud.

[0183] Figure 8 The green dashed curve represents the curve of the feature orthogonal constraint loss value change. The synchronous downward trend of this curve proves that the orthogonal constraint mechanism proposed in this invention is working. That is, during the training process, the weight vector of the new class is forced to continuously adjust its direction in the high-dimensional feature space, so that it gradually maintains a vertical separation from the weight vector of the old class, thereby minimizing the overlap of the feature space.

[0184] Figure 8 The dotted-line curve in orange-red represents the curve of the new category recognition accuracy. Thanks to the joint optimization of the dual loss function, the recognition accuracy rapidly climbed from the initial level of random guessing and stabilized at a high level of over 99.5% after about eighty iterations.

[0185] These curves strongly demonstrate that after introducing orthogonal constraints, the model not only did not experience training divergence or oscillation due to additional geometric restrictions, but also achieved accurate identification of new categories and non-interference with the feature spaces of old categories, verifying the convergence and robustness of the algorithm in incremental learning scenarios.

[0186] Through the illustration of this embodiment, the present invention demonstrates how to construct a drug classification model with both high generalization ability and high stability in industrial cold start scenarios with extremely scarce data by utilizing statistical transfer and generative techniques, thus completely solving the robustness problem of industrial-grade AI model updates under small sample conditions.

[0187] Example 4:

[0188] like Figure 9As shown, this embodiment provides a multi-specification drug classification and sorting system based on AI visual recognition. This system adopts a modular and intelligent architecture design, materializing each logical step in the aforementioned method embodiment into specific hardware execution units or software functional components. This system effectively overcomes the high downtime and maintenance costs caused by frequent changes in drug packaging in existing technologies, as well as the decrease in recognition accuracy caused by non-rigid deformation of flexible packaging during high-speed transportation. It also solves the problem of model overfitting caused by sparse samples in the initial stage of introducing new specifications.

[0189] Specifically, the system mainly includes a data acquisition module, a change sensing module, a model adaptation and classification module, and a sorting control module. These modules interact via high-speed industrial Ethernet or an internal bus to collaboratively complete the entire process from sensing to execution.

[0190] First, the data acquisition module serves as the system's entry point for perceiving the physical world. It primarily responds to trigger signals indicating that medicines are entering the sorting area and collects the current multimodal technical characteristic data of the medicines. In actual hardware deployment, this module integrates high-performance sensing devices such as color cameras, depth cameras, near-infrared sensors, and terahertz sensors. To address the issue mentioned in the background section regarding the slight bending or vibration of soft-packaged medicines on high-speed conveyor belts, the data acquisition module integrates an advanced data preprocessing unit. This unit not only performs basic time synchronization and spatial coordinate system calibration but also incorporates the dynamic elastic registration algorithm described in Example 2. It extracts high-frequency edges from color images as dynamic geometric anchor points and uses a thin-plate spline interpolation model to perform non-rigid mesh deformation correction on depth and spectral data. This mechanism ensures that even if the medicine undergoes physical deformation, the collected multimodal data remains precisely aligned at the pixel level, thus laying a high-quality data foundation for subsequent accurate identification.

[0191] Secondly, the change perception module, as the intelligent monitoring center of the system, compares the current multimodal technology feature data with a pre-set drug packaging benchmark feature library to determine whether the drug packaging has changed. This module is designed to address the pain point of traditional vision systems' inability to automatically adapt to packaging updates. The process of determining packaging changes in the change perception module specifically includes:

[0192] The system calculates the multi-dimensional similarity between the current multimodal technical feature data and the corresponding benchmark features in the drug packaging benchmark feature library. This multi-dimensional similarity includes not only visual texture similarity but also consistency of material spectral features and matching degree of three-dimensional geometric dimensions. Simultaneously, this module incorporates semantic text recognition technology to logically verify the specification text on the packaging. The system sets stringent judgment thresholds; if the similarity in any dimension falls below the preset change judgment threshold, or if the text logic verification fails, the module immediately generates a packaging change signal. This multi-dimensional cross-validation mechanism significantly improves the system's ability to detect minor packaging changes, effectively preventing compliance incidents caused by packaging version confusion.

[0193] Secondly, the model adaptation and classification module is the computational core of the system, possessing adaptive learning and inference capabilities. This module is used to initiate an incremental model adaptation process to update the classification model and perform classification when a packaging change is detected; or, if no packaging change is detected, it directly uses the current classification model to perform classification, outputting the classification result and confidence score. Addressing the technical challenge of extremely scarce samples in the initial stage of introducing new drug specifications, this module, upon receiving a change signal, invokes the feature distribution correction function based on covariance transfer as described in Example 3. It utilizes the historical covariance matrix of similar drugs in the benchmark database to simulate the distribution pattern of new packaging features in complex environments, generating massive virtual feature clouds to train the fully connected layer of the classifier, and introducing orthogonal constraints during training to prevent catastrophic forgetting. In the conventional classification mode, this module utilizes a hybrid architecture model including attention mechanisms and graph neural networks to perform deep fusion inference on multimodal features, thereby outputting high-precision classification results and confidence scores, ensuring that the system maintains robust recognition performance in dynamically changing environments.

[0194] Finally, the sorting control module, the system's execution brain, determines the target sorting instructions for the medicine based on the classification results and confidence levels, driving the sorting execution mechanism to complete the physical sorting. To ensure medicine safety and data compliance during the physical execution process, this module integrates a digital twin simulation engine and a blockchain verification interface. Before generating the target sorting instructions, the simulation engine pre-simulates the robotic arm's grasping path in a virtual environment based on the medicine's material properties and 3D model, assessing potential damage risks. Only after the risk assessment is passed will the instructions be issued to the physical execution mechanism. Simultaneously, this module binds the sorting results to the medicine's unique traceability identifier and stores the evidence through a blockchain interface, achieving closed-loop management from physical sorting actions to digital compliance records. This fully demonstrates the beneficial effects of this invention in improving sorting efficiency while ensuring medicine quality and traceability.

[0195] In summary, the system in this embodiment, through the close collaboration and technological integration of its various modules, has successfully built an intelligent sorting platform with dynamic perception, incremental learning, and compliance execution capabilities, providing strong technical support for the automation upgrade of the pharmaceutical industry.

[0196] Example 5:

[0197] Corresponding to the above embodiments, the present invention also proposes a computing device.

[0198] like Figure 10 The diagram shows a structural schematic of a computing device according to the present invention. The computing device 100 includes a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, for example, via a bus 102. Optionally, the computing device 100 may further include a transceiver 104. It should be noted that in practical applications, the transceiver 104 is not limited to one type, and the structure of this computing device 100 does not constitute a limitation on the embodiments of the present invention.

[0199] Processor 101 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0200] Bus 102 may include a pathway for transmitting information between the aforementioned components. Bus 102 may be a PCI bus or an EISA bus, etc. Bus 102 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0201] The memory 103 stores a computer program corresponding to the AI-based visual recognition method for classifying and sorting multi-specification drugs according to the above embodiments of the present invention. This computer program is controlled and executed by the processor 101. The processor 101 executes the computer program stored in the memory 103 to implement the content shown in the aforementioned method embodiments.

[0202] The computing device 100 includes, but is not limited to, mobile terminals such as laptops and tablets, as well as fixed terminals such as desktop computers. Figure 10 The computing device 100 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0203] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for classifying and sorting multi-specification drugs based on AI visual recognition, characterized in that, Includes the following steps: In response to a trigger signal indicating that the drug has entered the sorting area, the current multimodal technical characteristic data of the drug is collected; The current multimodal technology feature data is compared with a pre-set drug packaging benchmark feature library to determine whether the drug packaging has been changed. If so, the incremental model adaptation process is initiated, the classification model is updated based on the current multimodal technology feature data, and the updated classification model is used to classify the drug by specification, generating classification results and confidence scores. If not, the current classification model is used directly to classify the drug by specification, and the classification results and confidence scores are generated. Based on the classification results and confidence levels, a target sorting instruction for the drug is determined to drive the sorting execution mechanism to complete the physical sorting. The process of determining whether the drug has undergone packaging changes includes: calculating the multi-dimensional similarity between the current multimodal technical feature data and the corresponding benchmark features in the drug packaging benchmark feature library; if the similarity in any dimension is lower than a preset change determination threshold, a packaging change signal is generated.

2. The method according to claim 1, characterized in that, The collection of current multimodal technical feature data of the drug includes: The system controls a multimodal sensor group to synchronously acquire data, the multimodal sensor group including a color camera, a depth camera, a near-infrared sensor, and a terahertz sensor; The appearance and specification features captured by the color camera, the three-dimensional shape features captured by the depth camera, the material spectral features captured by the near-infrared sensor, and the internal structural features captured by the terahertz sensor are obtained. Based on a pre-constructed unified spatial coordinate system, the above feature data are spatially aligned using an affine transformation matrix, and time synchronization is completed based on a unified timestamp, wherein the spatial alignment meets a preset alignment accuracy, and the time synchronization meets a preset time synchronization threshold.

3. The method according to claim 2, characterized in that, The spatial alignment step, after initial alignment using an affine transformation matrix, further includes a dynamic elastic registration and correction step: High-frequency edge contour features are extracted from the images captured by the color camera and defined as dynamic geometric anchor points. The depth data acquired by the depth camera and the spectral data acquired by the near-infrared sensor are mapped to the unified spatial coordinate system, and the local position deviation vector field between the mapped data contour and the dynamic geometric anchor point is calculated. A non-rigid transformation model based on thin plate spline interpolation is constructed. The local position deviation vector field is used as a constraint condition to solve the transformation parameters that minimize the non-rigid bending energy. The transformation parameters are then used to perform pixel-by-pixel elastic mesh deformation on the depth data and the spectral data. Calculate the normalized mutual information value between the multimodal data after the elastic mesh deformation. Only when the normalized mutual information value is greater than the preset registration confidence threshold is spatial alignment determined and feature data is output. If the normalized mutual information value is lower than the preset registration confidence threshold, a calibration parameter failure signal is generated.

4. The method according to claim 1, characterized in that, The step of comparing the current multimodal technology feature data with a pre-set drug packaging benchmark feature library to determine whether the drug packaging has changed includes: Calculate visual feature similarity, material spectral similarity, and three-dimensional size similarity respectively; The semantic text recognition model is invoked to parse the packaging text in the current multimodal technology feature data, and to verify the consistency of the combination of specification keywords, numbers and units; When the visual feature similarity is lower than the first preset similarity threshold, or the material spectral similarity is lower than the second preset similarity threshold, or the three-dimensional size similarity deviation exceeds the preset tolerance range, or the packaging text combination verification fails, it is determined that a packaging change has occurred.

5. The method according to claim 1, characterized in that, The incremental adaptation process for initiating the model updates the classification model based on the current multimodal technology feature data, including: Collect a new packaging sample set of the drug and extract close-up features of the changed areas in the new packaging sample set; Freeze the backbone network parameters of the classification model and unfreeze the parameters of the specification feature-related fully connected layers; By utilizing the close-up features of the changed region and the common features of similar drugs in the drug packaging benchmark feature library, the fully connected layer is locally fine-tuned and trained to generate an adapted updated model.

6. The method according to claim 5, characterized in that, The local fine-tuning training step specifically includes a feature distribution correction process based on covariance transfer: Calculate the feature mean vector of the close-up features of the changed region in the new packaging sample set, and define it as the initial feature prototype of the new category; Retrieve similar drugs from the drug packaging benchmark feature library that match the material and shape attributes of the new packaging sample set, and extract the feature covariance matrix of the similar drugs in the historical full data. The feature covariance matrix characterizes the feature change distribution law of the drug in the sorting environment. A feature distribution generator is constructed, centered on the initial feature prototype and constrained by the feature covariance matrix, to generate virtual enhanced feature vectors through multidimensional Gaussian sampling, forming a virtual feature distribution cloud. A loss function containing orthogonal constraints is constructed, and the fully connected layer is trained using the virtual feature distribution cloud. The orthogonal constraints are used to force the weight vector of the updated model to maintain orthogonal separation in the feature space from the classification weight vector of the original drugs in the benchmark feature library.

7. The method according to claim 1, characterized in that, The process of classifying the drug by specification using the updated classification model or directly using the current classification model, wherein the classification model adopts a hybrid architecture, includes: The attention mechanism module is used to focus on the specification identification region in the current multimodal technology feature data, and the region combining numbers and units is assigned a higher weight coefficient than the non-specification region. A drug region graph structure is constructed using a graph neural network module. The average perturbation intensity, standard deviation, and contour compactness of the region are used as node features. The graph edge relationship is established based on spatial threshold and similarity threshold, and the feature vector is output. The feature vector is input into the classification decision layer, and the classification result is output.

8. The method according to claim 1, characterized in that, After collecting the current multimodal technical feature data of the drug, a data preprocessing step is also included: Construct a multi-scale Gaussian blurred image and extract high-frequency detail residual features between the original image and the multi-scale Gaussian blurred image; The high-frequency detail residual features are weighted and fused with the original image based on residual weights to generate enhanced multimodal technical feature data for subsequent comparison and classification.

9. The method according to claim 1, characterized in that, The step of determining the target sorting instruction for the drug based on the classification results and confidence levels includes: The classification results and the three-dimensional model parameters of the drug are loaded into the digital twin simulation environment; The grasping path and force of the sorting actuator are simulated in the digital twin simulation environment to calculate the damage risk value of the medicine; If the damage risk value is lower than the preset risk threshold, the target sorting instruction containing the target storage location and grabbing parameters is generated; If the damage risk value is higher than or equal to the preset risk threshold, an abnormal alarm command is generated.

10. The method according to claim 1, characterized in that, After the drive sorting actuator completes the physical sorting, a secondary verification step is also included: The microscopic camera and terahertz sensor are used to collect the characteristics of the sorted medicines and verify their matching degree with the classification results. Read the unique traceability identifier of the medicine and query the blockchain evidence data to verify the consistency between the current physical sorting location and the compliant storage location; If the matching degree is lower than the preset verification pass threshold or the consistency verification fails, the exception handling process is triggered and an exception tracing report is generated and uploaded to the blockchain.

11. A multi-specification drug classification and sorting system based on AI visual recognition, characterized in that, include: The data acquisition module is used to collect the current multimodal technical feature data of the drug in response to the trigger signal that the drug enters the sorting area; The change perception module is used to compare the current multimodal technology feature data with a preset drug packaging benchmark feature library to determine whether the drug packaging has changed. The model adaptation and classification module is used to initiate an incremental model adaptation process to update the classification model and perform classification when a packaging change is detected; or to directly use the current classification model to perform classification when no packaging change is detected, and output the classification result and confidence score. The sorting control module is used to determine the target sorting instruction for the drug based on the classification result and confidence level, so as to drive the sorting execution mechanism to complete the physical sorting. The process of determining packaging changes in the change perception module includes: calculating the multi-dimensional similarity between the current multimodal technical feature data and the corresponding benchmark features in the drug packaging benchmark feature library; if the similarity in any dimension is lower than the preset change judgment threshold, a packaging change signal is generated.

12. A computing device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, it implements the multi-specification drug classification and sorting method based on AI visual recognition as described in any one of claims 1-10.

Images