Drug identification method, device and equipment and storage medium

Abstract

The embodiment of the invention discloses a medicine identification method and device, equipment and a storage medium, and relates to the technical field of image identification, and the method comprises the steps: obtaining a two-dimensional surface image and three-dimensional contour data of a medicine; and analyzing the two-dimensional surface image based on a feature extraction rule to extract surface structure features of the medicine. And converting the three-dimensional contour data into spatial structure characteristics. And fusing the surface structure features, the spatial structure features and the obtained physical features and text features of the medicine to obtain joint features. And performing multi-label classification on the joint features by using a drug classification model to determine category information corresponding to the drugs. By performing feature extraction on the two-dimensional surface image and the three-dimensional contour data, the structural features of the medicine can be obtained more comprehensively and meticulously. By combining the physical characteristics and the text characteristics of the medicine, the combined characteristics covering the information of multiple dimensions can be obtained. And the accuracy of medicine identification is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, device, and storage medium for drug identification. Background Technology

[0002] In the medical field, accurate identification of drugs is crucial for drug management, patient medication safety, and drug quality control.

[0003] Traditional drug identification methods primarily rely on manual visual recognition and simple image matching techniques. This reliance on manual identification by staff is prone to errors due to staff fatigue or negligence. Current image recognition technologies are sensitive to interference from lighting conditions and drug wear, resulting in insufficient accuracy. Furthermore, image recognition relies on single image features, which can lead to misidentification of similar drugs.

[0004] It is evident that improving the accuracy of drug identification is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a drug identification method, apparatus, device, and storage medium.

[0006] To address the aforementioned technical problems, embodiments of this application provide a drug identification method, comprising:

[0007] Acquire two-dimensional surface images and three-dimensional contour data of the drug;

[0008] Two-dimensional surface images are analyzed based on feature extraction rules to extract the surface structure features of pharmaceuticals. These surface structure features include color features, texture features, semantic features, and local features. Semantic features include the pharmaceutical traceability code.

[0009] Convert 3D contour data into spatial structural features;

[0010] Surface structure features, spatial structure features, and acquired drug physical and textual features are fused to obtain joint features.

[0011] A drug classification model is used to perform multi-label classification on joint features to determine the category information of the drug; the category information includes drug specifications, quantity and / or use.

[0012] On the one hand, based on feature extraction rules, two-dimensional surface images are analyzed to extract the surface structural features of the pharmaceutical product, including:

[0013] Deep learning models are used to extract color features, texture features, semantic features, and local features contained in two-dimensional surface images; among them, semantic features include drug dosage form classification features and authenticity identification features;

[0014] The text information recognition tool is used to identify the text information contained in two-dimensional surface images in order to extract character features containing drug specifications, quantities and uses.

[0015] On the one hand, before using deep learning models to extract the color features, texture features, semantic features, and local features contained in two-dimensional surface images, it also includes:

[0016] Obtain the application scenarios corresponding to drug identification;

[0017] A deep learning model matching the application scenario is selected from a pre-trained deep learning model library; the channel weights of the deep learning models corresponding to different application scenarios are different.

[0018] On the one hand, before analyzing two-dimensional surface images based on feature extraction rules to extract the surface structural features of pharmaceuticals, the following steps are also included:

[0019] An adaptive illumination compensation algorithm is used to eliminate ambient light interference in the two-dimensional surface image to obtain a standard image;

[0020] Accordingly, based on feature extraction rules, the two-dimensional surface image is analyzed to extract the surface structural features of the drug, including:

[0021] The standard image is analyzed based on feature extraction rules to extract the surface structural features of the drug.

[0022] On the one hand, ambient light interference in the two-dimensional surface image is eliminated according to the adaptive illumination compensation algorithm to obtain a standard image, including:

[0023] The two-dimensional surface image is decomposed into reflection and illumination components;

[0024] Based on the established nonlinear mapping relationship between ambient light and image brightness, the brightness of the illumination component in the two-dimensional surface image is adjusted to obtain a standard image.

[0025] On the one hand, after adjusting the brightness of the illumination components in the two-dimensional surface image based on the established nonlinear mapping relationship between ambient light and image brightness to obtain a standard image, the process also includes:

[0026] Identifying the actual irradiance of two-dimensional surface images based on regression networks;

[0027] The actual irradiance is calibrated based on the single-point correction expected irradiance to obtain the calibrated actual irradiance.

[0028] Based on the set standard irradiance, brightness adjustment speed, system gain coefficient, and calibrated actual irradiance, the supplementary power of the light source is determined to facilitate the adjustment of the light source power of the image acquisition equipment.

[0029] On the one hand, after using a drug classification model to perform multi-label classification on joint features to determine the corresponding category information of the drug, it also includes:

[0030] In the current application scenario of drug loading, determine whether the drug specifications, quantity and purpose included in the category information match the drug information recorded on all drug orders;

[0031] If the drug specifications, quantity, and purpose contained in the category information match the drug information recorded on all drug orders, a dispensing instruction is issued to the dispensing machine so that the dispensing machine can perform the dispensing operation.

[0032] In the current application scenario of dispensing medicine, it is determined whether the medicine specifications and quantities included in the category information of the medicine after dispensing match the medicine information recorded on the corresponding medication order of the user picking up the medicine.

[0033] If the drug category information, drug specifications, and quantity match the drug information recorded on the corresponding user's drug order after dispensing, a bagging instruction is sent to the dispensing machine so that the dispensing machine can perform the bagging operation.

[0034] In the current application scenario of bagging, determine whether the drug category information, drug specifications, and quantity contained in each medicine bag are consistent with the drug information on the corresponding user's medicine order.

[0035] If the drug category information, drug specifications, and quantity in each medicine bag match the drug information on the corresponding user's medicine order, a medicine dispensing instruction is issued to the dispensing machine so that the dispensing machine can transfer the medicine to the dispensing area.

[0036] This application also provides a drug identification device, including an acquisition unit, an extraction unit, a conversion unit, a fusion unit, and a classification unit;

[0037] The acquisition unit is used to acquire two-dimensional surface images and three-dimensional contour data of the drug.

[0038] An extraction unit is used to analyze a two-dimensional surface image based on feature extraction rules to extract the surface structure features of the drug; wherein, the surface structure features include color features, texture features, semantic features, and local features; the semantic features include the drug traceability code;

[0039] The conversion unit is used to convert three-dimensional contour data into spatial structural features;

[0040] The fusion unit is used to fuse surface structure features, spatial structure features, and acquired drug physical and textual features to obtain joint features.

[0041] The classification unit is used to perform multi-label classification on joint features using a drug classification model to determine the category information corresponding to the drug; the category information includes drug specifications, quantity and / or use.

[0042] This application also provides an electronic device, including:

[0043] Memory, used to store computer programs;

[0044] A processor for executing computer programs to implement the steps of any of the drug identification methods described above.

[0045] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described drug identification methods.

[0046] As can be seen from the above technical solution, the process involves acquiring two-dimensional surface images and three-dimensional contour data of the drug; analyzing the two-dimensional surface images based on feature extraction rules to extract the surface structural features of the drug; these surface structural features can include color features, texture features, semantic features, and local features. The three-dimensional contour data is then converted into spatial structural features. Both surface and spatial structural features are structural features obtained from image analysis. To more accurately identify the drug, physical and textual features of the drug can be acquired, and these features are then fused with the surface and spatial structural features, along with the acquired physical and textual features, to obtain joint features. A drug classification model is used to perform multi-label classification on the joint features to determine the corresponding category information of the drug; this category information can include drug specifications, quantity, and / or usage. In this technical solution, feature extraction from two-dimensional surface images and three-dimensional contour data allows for a more comprehensive and detailed acquisition of the drug's structural features. Based on this, combining the drug's physical and textual features yields joint features. These joint features encompass information from multiple dimensions, resulting in more accurate drug category information. The drug identification scheme provided in this solution effectively improves the accuracy of drug identification. Attached Figure Description

[0047] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a drug identification method provided in this application embodiment;

[0049] Figure 2 This application provides an overall system architecture diagram for applying a drug identification method to a dispensing machine.

[0050] Figure 3 This is a schematic diagram of the structure of a drug identification device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0052] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0053] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Traditional drug identification methods primarily rely on manual visual recognition and simple image matching techniques. However, interference from factors such as lighting conditions or drug wear can lead to low accuracy in drug identification. Furthermore, traditional visual recognition analyzes single image features, making it prone to misidentification of similar drugs.

[0055] Therefore, embodiments of this application provide a drug identification method, apparatus, device, and storage medium. By extracting features from two-dimensional surface images and three-dimensional contour data, the structural features of drugs can be obtained more comprehensively and meticulously. Based on this, by combining drug physical features and textual features, joint features can be obtained, achieving multimodal information fusion. The joint features cover information from multiple dimensions, enabling more precise differentiation of similar drugs; therefore, the drug category information identified based on the joint features is more accurate. The drug identification scheme provided in this application effectively improves the accuracy of drug identification, reduces the labor costs of drug identification, and minimizes human error. Simultaneously, this scheme has high anti-interference capability and accuracy. Utilizing this scheme to implement refined management of drug information is of great significance.

[0056] The drug identification solution provided in this application is applicable to drug management, patient medication safety, and medical information scenarios, including drug receiving and acceptance, outpatient and inpatient drug dispensing windows, intravenous solution preparation verification, drug counting, and patient self-medication. It is particularly effective in accurately identifying drugs at different stages of the tablet dispensing machine's workflow. Applying this drug identification solution to tablet dispensing machines can improve the efficiency of drug dispensing windows, increase dispensing accuracy, and reduce medication accidents caused by dispensing errors.

[0057] Next, a drug identification method provided by the embodiments of this application will be described in detail. Figure 1 A flowchart of a drug identification method provided in this application embodiment, the method including:

[0058] S101: Acquire two-dimensional surface images and three-dimensional contour data of the drug.

[0059] In this embodiment of the application, in order to ensure the integrity of the drug image, a smart camera can be used to acquire two-dimensional surface images and three-dimensional contour data of the drug.

[0060] Considering the diverse presentation states of pharmaceuticals, a database of images representing various states can be established, such as images of entire boxes, blister packs, tubes, bags, tablets, and granules, using multiple dimensions. Based on this state database, the three-dimensional contour data of the currently acquired pharmaceutical images can be determined.

[0061] S102: Analyze two-dimensional surface images based on feature extraction rules to extract the surface structural features of pharmaceuticals.

[0062] Surface structure features can include color features, texture features, semantic features, and local features.

[0063] In the embodiments of this application, deep learning models can be used to extract color features, texture features, semantic features, and local features contained in two-dimensional surface images.

[0064] Color features can be used to distinguish the colors of medicines and identify anti-counterfeiting marks.

[0065] Texture features can be used to detect surface damage and identify the surface texture of pharmaceuticals. Texture features are extracted through traditional image processing or shallow convolutional layers, focusing on local physical details.

[0066] In this embodiment, grouped convolutions can be used to replace standard convolutional layers, splitting the original ResNet-50 3x3 convolutions into a multi-branch structure, reducing the number of parameters by 30%-50% while maintaining the receptive field, and improving parallel computing efficiency by combining channel rearrangement techniques.

[0067] Texture features can include microstructure information, statistical quantification indicators, and optical response characteristics.

[0068] Microstructure information can include surface grain size (such as pharmaceutical roughness), the geometry of scratches / stains, and the edge sharpness of printed characters on packaging. Edge sharpness can be quantified using a histogram of oriented gradients (HOG).

[0069] Statistical quantification indicators can include multiple parameters of the Gray Level Co-Occurrence Matrix (GLCM): contrast (reflecting scratch depth), entropy (characterizing randomness), and inverse moment (local uniformity). Local binary patterns (LBP) generate binary-coded histograms that describe the microscopic texture patterns on the drug surface.

[0070] Optical response characteristics can include the light reflection intensity distribution of reflective materials (such as aluminum foil), and fluorescence / transmission characteristics under multispectral imaging, such as anti-counterfeiting marks in the near-infrared range.

[0071] Semantic features can include functional semantic features, character features, contextual associations, etc.

[0072] Functional semantic features include dosage form classification features and authenticity identification features for pharmaceuticals. Dosage form classification features include oral tablets, injections, ointments, etc. Authenticity identification features may include abstract patterns of specific trademarks (LOGOs).

[0073] The text information contained in the two-dimensional surface image is identified by an Optical Character Recognition (OCR) tool to extract character features such as drug specifications, quantity, and drug traceability code.

[0074] Contextual association can include the topological relationship between the drug and its packaging, and multimodal association. The topological relationship between the drug and its packaging is such as the position of the drug in the blister pack, while multimodal association includes cross-modal alignment between the image and the OCR text.

[0075] S103: Convert 3D contour data into spatial structural features.

[0076] Spatial structure features are program codes based on three-dimensional contour data. Spatial structure features belong to a deeper level of coding language. They can be seen as converting three-dimensional contour data into program codes that are easy for computers to process, thus facilitating the processing of intelligent algorithms.

[0077] Spatial structural features can be used to identify the shape of medicines and correct their placement.

[0078] S104: The surface structure features, spatial structure features, and acquired drug physical features and text features are fused to obtain joint features.

[0079] The input layer of the model can be integrated with a drug instruction text encoder. Text features are extracted through the BERT model and concatenated with surface structure features, spatial structure features, and drug physical features in the channel dimension to establish a text-image association feature space.

[0080] Building upon the existing ResNet-50 architecture, a Bidirectional Feature Pyramid Network (BiFPN) is introduced to fuse surface structure features, spatial structure features, drug physical features, and text features across multiple levels. Learnable weight coefficients adaptively adjust the contribution of features at each level, enhancing the ability to capture details such as drug packaging text and drug shape.

[0081] The fusion formula is as follows:

[0082] F out =∑ i w i ⋅F i with∑w i =1;

[0083] Among them, F i These represent feature maps at different levels, including surface structure features, spatial structure features, drug physical features, and text features; w i F represents the learnable weight coefficients, satisfying the normalization constraint ∑wi = 1, ensuring the stability of the feature values ​​after fusion; out It represents the combined features after fusion, which includes both local details and global semantic information.

[0084] In this embodiment, a bidirectional feature pyramid network is introduced to fuse features from different levels, thereby enhancing the ability to preserve local details.

[0085] S105: Use a drug classification model to perform multi-label classification on the joint features to determine the category information corresponding to the drug.

[0086] The category information may include drug specifications, quantity, and / or intended use. Drug specifications may include information such as drug type, dosage, and formulation. Depending on actual needs, the category information may include more types of information, which are not limited here.

[0087] In this embodiment, a random forest model can be used to perform multi-label classification on joint features, outputting drug specifications, quantity, and usage. Category information may also include the drug's production batch number, and the expiration date information on the drug packaging can be identified and compared, with reminders issued for drugs exceeding the time limit in the database.

[0088] It connects to a cloud database, dynamically updates the drug database, and sends the identification results back to the user's terminal.

[0089] As can be seen from the above technical solution, the process involves acquiring two-dimensional surface images and three-dimensional contour data of the drug; analyzing the two-dimensional surface images based on feature extraction rules to extract the surface structural features of the drug; these surface structural features can include color features, texture features, semantic features, and local features. The three-dimensional contour data is then converted into spatial structural features. Both surface and spatial structural features are structural features obtained from image analysis. To more accurately identify the drug, physical and textual features of the drug can be acquired, and these features are then fused with the surface and spatial structural features, along with the acquired physical and textual features, to obtain joint features. A drug classification model is used to perform multi-label classification on the joint features to determine the corresponding category information of the drug; this category information can include drug specifications, quantity, and / or usage. In this technical solution, feature extraction from two-dimensional surface images and three-dimensional contour data allows for a more comprehensive and detailed acquisition of the drug's structural features. Based on this, combining physical and textual features yields joint features. These joint features encompass information from multiple dimensions, resulting in more accurate drug category information. The drug identification scheme provided in this solution effectively improves the accuracy of drug identification.

[0090] Considering the varying accuracy requirements for different image features across different application scenarios, different deep learning models can be trained for each scenario. Before extracting color, texture, semantic, and local features from a two-dimensional surface image using a deep learning model, the application scenario corresponding to drug recognition can be determined. A deep learning model matching the application scenario can then be selected from a pre-trained deep learning model library. The channel weights of the deep learning models corresponding to different application scenarios will vary.

[0091] In this embodiment, the deep learning model can use an improved ResNet-50 as its basic architecture, embedding an Efficient Channel Attention (ECA) module to enhance the capture of subtle drug features such as engravings and anti-counterfeiting labels on packaging. During training, a local microscopic feature extraction branch is introduced, performing 128×128 local region sampling. A multi-task supervision mechanism is introduced to simultaneously train drug classification and instruction manual text matching tasks, constructing a cross-modal feature space. An Online Hard Sample Mining (OHEM) strategy is used to focus on optimizing the feature discrimination of easily confused drugs, such as drugs with the same name from different manufacturers.

[0092] By embedding the ECA module, the feature responses of different channels are dynamically weighted, which can effectively improve the focus on salient features. The ECA module achieves inter-channel dependency modeling through lightweight one-dimensional convolution, increasing the computational cost by only 0.03%, but improving feature discriminative power.

[0093] A channel attention module is embedded after a 3×3 convolution: global average pooling is used to obtain channel feature vectors, and channel weights are generated through two fully connected layers to perform channel weighting on the feature maps.

[0094] During model training, the importance of parameters is quantified and regularization constraints are applied to combat catastrophic forgetting while retaining key drug feature memory. The main implementation steps are as follows:

[0095] Step 1: Initial Task Training. Train the base model to identify the first batch of drugs, such as 100 common drugs. Save the optimal parameters θ∗ and the Fisher matrix F.

[0096] The optimal parameter θ∗ and the high / low Fi parameters in the Fisher matrix essentially work together: θ∗ provides the baseline values ​​for the parameters, and Fi quantifies their importance weights; together, they control the regularization strength, achieving a balance between combating forgetting (protecting high Fi parameters) and adapting to new tasks (relaxing low Fi parameters). Therefore, the optimal parameter θ∗ is the foundation for Fi calculation, while Fi determines which parameters in θ∗ should be prioritized for regularization. High Fi parameters may include drug name OCR layer weights and notch edge detection convolution kernels. Low Fi parameters may include background illumination adaptation layer parameters.

[0097] Step 2: Incremental task learning.

[0098] Loading old parameters and constraints: Initialize the parameters of the new task model and inject regularization terms. The old parameters loaded here are the optimal parameters θ∗ saved in step 1.

[0099] The core of the constraint mechanism is to limit the range of parameter updates through regularization terms, which specifically includes the following two parts: parameter importance constraints (based on the Fisher matrix F) and elastic regularization penalty terms.

[0100] Training on new tasks: Using data from new drugs for training, during backpropagation: impose strong constraints on high Fi parameters (e.g., variation in notch feature weights < 0.1%). Low Fi parameters can be freely adjusted to accommodate differences in new drug packaging.

[0101] Update the Fisher matrix: merge the F matrices of the old and new tasks and dynamically adjust the importance of parameters.

[0102] Step 3: Visual verification of feature memory.

[0103] Channel weighting enhances feature discriminative power. Traditional convolution treats all channels equally, while the SE module in this application learns complex relationships between channels through fully connected layers. For example, in recognizing drug packaging, text and color channels have a collaborative relationship; for instance, red warning text requires higher weights, and the SE module explicitly models this dependency. For similar drugs (e.g., drugs with the same name from different manufacturers), channels with subtle differences (e.g., trademark font, scratch depth) are assigned high weights, amplifying the classification boundary. Global average pooling weakens local brightness differences caused by uneven lighting, focusing weights on content rather than brightness. When drug packaging is partially occluded, the SE module maintains recognition capability by preserving channel weights in the unoccluded areas. Many channels in drug images carry repetitive information such as background texture; the SE module suppresses redundant channels, reducing subsequent computation. The SE module adds only a few parameters (two fully connected layers) but significantly improves feature quality.

[0104] The channels involved in the model can include color space channels, multimodal data channels, and deep learning feature map channels.

[0105] Color space channels: By using the color component channels of the color space (RGB / HSV / HSI), combined with the RGB / HSV / HSI color space model and multispectral imaging technology, the colors of medicines can be distinguished, reflective interference can be suppressed, and the response of anti-counterfeiting marks or internal components can be enhanced.

[0106] Multimodal data channels: By using visual-text cross-modal channels and node channel technology in heterogeneous graph models, image and text information are fused to assist classification.

[0107] Deep learning feature map channels: Through multi-head attention channel technology, focus on local details of the drug such as text and damage.

[0108] In this embodiment, a progressive feature distillation strategy is adopted to guide deep feature learning through the teacher network and reduce feature redundancy. Progressive feature distillation compresses the high-dimensional features of the teacher network into the student network through phased and multi-level knowledge transfer, mainly including the following three stages: (1) Basic feature imitation stage: The student network directly learns the final output of the teacher network, such as the classification probability distribution, and establishes preliminary feature representation capabilities. KL divergence loss is used to align the output distribution of the teacher and student networks. For example, the teacher network transfers soft labels of drug texture and color. In drug packaging OCR recognition, the student network quickly grasps the overall distribution pattern of label text. (2) Intermediate layer feature progressive alignment stage: A hierarchical distillation mechanism is adopted to divide the network into multiple hierarchical modules, such as the convolutional blocks of ResNet. The deep modules act as teachers and the shallow modules act as students, transferring abstract features layer by layer. Kernel function mapping: For the differences in feature distribution between teachers and students, such as the Gaussian distribution of CNN and the Laplacian distribution of ANN, the Gaussian kernel / Laplacian kernel is used to map the features to a high-dimensional space to eliminate the distribution differences. Adaptive channel weighting: The channel attention module (ACAM) dynamically allocates the feature channel weights, suppresses redundant channels such as those corresponding to background noise, and enhances key channels such as those corresponding to drug edges. (3) Fine-tuning and attention fusion stage: Multi-loss joint optimization, combining probability distribution alignment (KL Loss), feature map similarity (L2 Loss), and cross-entropy loss for end-to-end training.

[0109] By leveraging intermediate features from the teacher network, such as the target query vector, to generate a spatial attention mask, the student network is guided to focus on key regions of the drug, such as the location of the batch number. In drug image recognition, the guidance provided by the teacher network essentially transforms high-dimensional features into structured knowledge that a lightweight model can digest. Through this mechanism, the student network approximates the recognition performance of the teacher network while maintaining low computational overhead, making it particularly suitable for scenarios in pharmaceutical quality inspection with stringent requirements for accuracy and real-time performance, such as drug defect detection and authenticity verification.

[0110] In order to improve the robustness of the model during the training phase, the training images can be preprocessed, including operations such as grayscale conversion, edge detection, and normalization.

[0111] In practical applications, poor image quality can arise due to lighting conditions, thus affecting the accuracy of drug identification. Common scenarios where lighting factors impact image quality include: overexposure in reflective areas and loss of detail in shadow areas due to uneven ambient light on the drug surface; insufficient contrast of text in the grooves of the blister pack under low light; and color shift in the drug bottle label under mixed light sources. Therefore, in this embodiment, to improve image quality, an adaptive lighting compensation algorithm can be used to eliminate ambient light interference and enhance texture features.

[0112] In practice, an adaptive illumination compensation algorithm can be used to eliminate ambient light interference in the two-dimensional surface image to obtain a standard image; the standard image is then analyzed based on feature extraction rules to extract the surface structure features of the drug.

[0113] The adaptive illumination compensation algorithm decomposes a two-dimensional surface image into a reflection component and an illumination component. Based on the established nonlinear mapping relationship between ambient light and image brightness, the brightness of the illumination component in the two-dimensional surface image is adjusted to obtain a standard image.

[0114] The essence of nonlinear mapping is to achieve dynamic transformation from ambient light to image brightness through parameterized learnable functions. The functional forms of nonlinear mapping include piecewise adjustment (hyperbolic tangent), probability weighting (Sigmoid), and linear combination (gating factor), etc. Parameters are generated in real-time by the CNN based on the input lighting conditions. Lighting correction and multiple tasks such as drug segmentation / text recognition share backbone features, which can improve system efficiency.

[0115] After obtaining the standard image, the actual irradiance of the two-dimensional surface image can be identified based on the regression network; the actual irradiance is calibrated according to the single-point correction expected irradiance to obtain the calibrated actual irradiance; the supplementary power of the light source is determined according to the set standard irradiance, brightness adjustment speed, system gain coefficient and calibrated actual irradiance, so as to adjust the light source power of the image acquisition device.

[0116] In this embodiment, a projector-camera system linkage model can be established using three-dimensional coordinate mapping technology, and multi-source iterative compensation can be achieved through the HDD algorithm. Combining a temperature coefficient compensation model to eliminate environmental interference errors is a step in the HDD algorithm's implementation of multi-source iterative compensation.

[0117] By establishing a projector-camera system linkage model, the core problems of drug recognition, such as illumination interference, pose deformation, and segmentation ambiguity, were systematically solved. At the same time, it provided a physical basis for multimodal feature fusion and deep learning optimization, ultimately achieving high-precision and high-robust drug recognition.

[0118] Three-dimensional coordinate mapping technology constructs the spatial geometric framework of the projector-camera system, while the irradiance-power correction formula and temperature compensation ensure the stability of signal transmission and environmental robustness. The two form a closed loop through real-time data interaction (such as temperature feedback to adjust projection power and injection of correction parameters into coordinate calculation), ultimately achieving micron-level precision in three-dimensional reconstruction in complex environments.

[0119] The irradiance-power correction formula is primarily used to address image quality instability caused by light source fluctuations, environmental interference, or the reflective properties of pharmaceutical surfaces. Its core principle is to ensure image data consistency by dynamically calibrating the relationship between irradiance and sensor response. The irradiance mentioned here refers to the radiant power received per unit area.

[0120] In drug identification scenarios, irradiance directly affects the brightness value captured by the image sensor, thus impacting the accuracy of feature extraction. Irradiance (E) is defined as the radiant flux received per unit area: E=dSdΦ (unit: W / m2);

[0121] Where Φ is the luminous flux, i.e., the radiant power, and the unit is watt (W); S is the light-receiving area (m²).

[0122] In practical implementation, actual irradiance can be predicted using regression networks:

[0123] Epredict=(Emax−Emin)*predict(net, G, t)+Emin;

[0124] Where Emin represents the minimum value of the theoretical range of pixel irradiance, and Emax represents the maximum value of the theoretical range of pixel irradiance; predict(net, G, t) represents the output of the trained regression network, G represents the normalized grayscale response, and t represents the integration time.

[0125] Normalized grayscale response is an intermediate variable that maps the raw grayscale values ​​output by the camera / sensor to a standard irradiance range. It can be obtained through calibration and algorithm processing.

[0126] Integration time is the exposure time of the sensor, which directly affects signal strength and noise level.

[0127] For infrared systems or scenarios susceptible to temperature effects, calibration with a known blackbody radiation source is required: Ltarget = f2*τtπ*(Eexpects−Epredict);

[0128] Where Ltarget represents the actual radiance after calibration; Eexpect_s represents the single-point rectified expected radiance; f represents the F-number of the optical system; and τt represents the transmittance from the target to the focal plane.

[0129] In the drug identification system, the light source power is dynamically adjusted through feedback control: ΔP=k*|Estd-Ecurrent|*vadjust;

[0130] Where Estd represents the irradiance required by standard image attributes, i.e., standard irradiance; vadjust represents the brightness adjustment speed, which is positively correlated with the imaging brightness difference; and k represents the system gain coefficient.

[0131] For unlabeled injectable drugs, the power of the bottom light source should be adjusted first to compensate for differences in drug light transmittance.

[0132] In this embodiment, a closed-loop feedback system for brightness adjustment effect is constructed to continuously optimize the local brightness adjustment strategy, thereby improving image quality and ultimately achieving high-precision and robust drug recognition.

[0133] In practical applications, drug identification methods can be deployed on intelligent identification terminals. These terminals can be used in dispensing machines. A dispensing machine includes a filling layer, a dispensing layer, and a bagging layer. Different layers of the dispensing machine correspond to different application scenarios. A corresponding intelligent identification terminal can be deployed on each layer of the dispensing machine.

[0134] In the current application scenario of drug loading, the intelligent identification terminal deployed in the drug loading layer can determine whether the drug specifications, quantity and purpose contained in the category information match the drug information recorded on all drug orders; if the drug specifications, quantity and purpose contained in the category information match the drug information recorded on all drug orders, it issues a drug dispensing instruction to the drug dispensing machine so that the drug dispensing machine can perform the drug dispensing operation.

[0135] In the current application scenario of dispensing medicine, the intelligent identification terminal deployed in the dispensing layer can determine whether the medicine specifications and quantities included in the category information of the medicine after dispensing match the medicine information recorded on the corresponding medication order of the user. If the medicine specifications and quantities included in the category information of the medicine after dispensing match the medicine information recorded on the corresponding medication order of the user, a bagging instruction is issued to the dispensing machine so that the dispensing machine can perform the bagging operation.

[0136] In the current application scenario of bagging, the intelligent identification terminal deployed in the bagging layer can determine whether the drug category information, drug specifications, and quantity in each medicine bag are consistent with the drug information on the corresponding user's medicine order. If the drug category information, drug specifications, and quantity in each medicine bag are consistent with the drug information on the corresponding user's medicine order, a medicine dispensing instruction is issued to the dispensing machine so that the dispensing machine can transfer the medicine to the dispensing area.

[0137] By using multimodal data fusion and deep learning algorithms to identify medicines, accurate identification of medicines is achieved at different stages of the dispensing machine's workflow, improving the efficiency of medicine window dispensing, increasing the accuracy of dispensing, and reducing medication accidents caused by dispensing errors.

[0138] Figure 2 This application provides a system architecture diagram for applying a drug identification method to a dispensing machine. Figure 2Taking a tablet dispensing machine as an example, this machine consists of three layers: a filling layer, a dispensing layer, and a filling layer. Intelligent recognition terminals are installed at both the automatic and manual dispensing windows in the filling layer. The intelligent recognition terminal at the automatic dispensing window can identify the medication during the dispensing process. The intelligent recognition terminal at the manual dispensing window can identify the medication in the manual dispensing window. An intelligent recognition terminal is installed near the dispensing device in the dispensing layer to identify the medication after dispensing. Similarly, an intelligent recognition terminal is installed in the automatic bagging area of ​​the bagging layer to identify the medication in the bagged medication. Throughout the entire dispensing process, the tablet dispensing machine enables full-process identification and tracking of the medication. Multimodal image recognition technology is used to classify and identify the target medication, and the identification results are fed back to the human-machine interface in real time.

[0139] For the automated dispensing stage, an intelligent identification terminal is installed on the automatic dispensing device of the tablet dispensing machine to achieve precise replenishment of medication at the front end of the dispensing process. For the manual dispensing stage, an intelligent identification terminal is installed on the manual dispensing device of the tablet dispensing machine to achieve precise replenishment of medication in special circumstances where manual dispensing is required.

[0140] For the dispensing process, an intelligent identification terminal is installed on the dispensing device inside the tablet dispensing machine to identify the medicines after dispensing but before bagging, thus eliminating dispensing errors caused by malfunctions in the dispensing machine.

[0141] For automated bagging, an intelligent identification terminal is installed near the bag outlet at the rear end of the tablet dispensing machine's bagging process to identify the bagged medicines and eliminate bagging errors caused by the dispensing machine's bagging error.

[0142] In this embodiment, the aforementioned drug identification technology is applied to the entire workflow of the tablet dispensing machine, systematically solving the problem of refined drug management throughout the entire process, including drug entry, exit, management, dispensing, compounding, finished product verification, and return. This improves work efficiency, reduces labor costs, and significantly reduces errors in drug management and dispensing, thereby enhancing the patient experience.

[0143] Figure 3 A schematic diagram of a drug identification device provided in an embodiment of this application includes an acquisition unit 31, an extraction unit 32, a conversion unit 33, a fusion unit 34, and a classification unit 35;

[0144] Acquisition unit 31 is used to acquire two-dimensional surface images and three-dimensional contour data of the drug.

[0145] Extraction unit 32 is used to analyze a two-dimensional surface image based on feature extraction rules to extract the surface structure features of the drug; wherein, the surface structure features include color features, texture features, semantic features, and local features; the semantic features include the drug traceability code;

[0146] The conversion unit 33 is used to convert three-dimensional contour data into spatial structural features;

[0147] The fusion unit 34 is used to fuse surface structure features, spatial structure features, and acquired drug physical features and text features to obtain joint features;

[0148] Classification unit 35 is used to perform multi-label classification on joint features using a drug classification model to determine the category information corresponding to the drug; wherein, the category information includes drug specifications, quantity and / or use.

[0149] In some embodiments, the extraction unit is used to extract color features, texture features, semantic features, and local features contained in a two-dimensional surface image using a deep learning model; wherein, the semantic features include dosage form classification features and authenticity identification features of the drug.

[0150] The text information recognition tool is used to identify the text information contained in two-dimensional surface images in order to extract character features containing drug specifications, quantities and uses.

[0151] In some embodiments, it further includes: a scene acquisition unit and a model selection unit;

[0152] The scene acquisition unit is used to acquire the application scene corresponding to drug identification;

[0153] The model selection unit is used to select a deep learning model that matches the application scenario from a pre-trained deep learning model library; the channel weights of the deep learning models corresponding to different application scenarios are different.

[0154] In some embodiments, an elimination unit is also included;

[0155] An elimination unit is used to eliminate ambient light interference in a two-dimensional surface image according to an adaptive illumination compensation algorithm to obtain a standard image;

[0156] Correspondingly, the extraction unit is used to analyze the standard image based on feature extraction rules to extract the surface structural features of the drug.

[0157] In some embodiments, the elimination unit is used to decompose a two-dimensional surface image into a reflection component and an illumination component;

[0158] Based on the established nonlinear mapping relationship between ambient light and image brightness, the brightness of the illumination component in the two-dimensional surface image is adjusted to obtain a standard image.

[0159] In some embodiments, the system further includes an identification unit, a calibration unit, and a power determination unit;

[0160] A recognition unit is used to identify the actual irradiance of a two-dimensional surface image based on a regression network.

[0161] The calibration unit is used to calibrate the actual irradiance based on the single-point correction desired irradiance to obtain the calibrated actual irradiance.

[0162] The power determination unit is used to determine the supplementary power of the light source based on the set standard irradiance, brightness adjustment speed, system gain coefficient, and calibrated actual irradiance, so as to adjust the light source power of the image acquisition device.

[0163] In some embodiments, the system further includes a first determination unit, a first execution unit, a second determination unit, a second execution unit, a third determination unit, and a third execution unit;

[0164] The first judgment unit is used to determine whether the drug specifications, quantity and purpose included in the category information match the drug information recorded on all drug orders when the current application scenario is a drug loading scenario.

[0165] The first execution unit is used to issue a dispensing instruction to the dispensing machine when the drug specifications, quantity and purpose contained in the category information match the drug information recorded on all drug orders, so that the dispensing machine can perform the dispensing operation.

[0166] The second judgment unit is used to determine whether the drug specifications and quantity included in the drug category information after drug distribution match the drug information recorded on the corresponding drug order of the user when the current application scenario is a drug distribution scenario.

[0167] The second execution unit is used to issue a bagging instruction to the dispensing machine when the drug category information and drug specifications and quantity contained in the dispensing drug match the drug information recorded on the corresponding drug order of the user, so that the dispensing machine can perform the bagging operation.

[0168] The third judgment unit is used to determine whether the drug category information, drug specifications and quantity contained in each medicine bag are consistent with the drug information on the corresponding drug order of the user picking up the medicine when the current application scenario is a bagging scenario.

[0169] The third execution unit is used to issue a medicine dispensing instruction to the medicine dispensing machine when the medicine category information, medicine specifications, and quantity contained in each medicine bag are consistent with the medicine information on the corresponding medicine dispensing user's medicine order, so that the medicine dispensing machine can transfer the medicine to the medicine dispensing area.

[0170] Figure 3 For a description of the features in the corresponding embodiments, please refer to Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0171] As can be seen from the above technical solution, the process involves acquiring two-dimensional surface images and three-dimensional contour data of the drug; analyzing the two-dimensional surface images based on feature extraction rules to extract the surface structural features of the drug; these surface structural features can include color features, texture features, semantic features, and local features. The three-dimensional contour data is then converted into spatial structural features. Both surface and spatial structural features are structural features obtained from image analysis. To more accurately identify the drug, physical and textual features of the drug can be acquired, and these features are then fused with the surface and spatial structural features, along with the acquired physical and textual features, to obtain joint features. A drug classification model is used to perform multi-label classification on the joint features to determine the corresponding category information of the drug; this category information can include drug specifications, quantity, and / or usage. In this technical solution, feature extraction from two-dimensional surface images and three-dimensional contour data allows for a more comprehensive and detailed acquisition of the drug's structural features. Based on this, combining physical and textual features yields joint features. These joint features encompass information from multiple dimensions, resulting in more accurate drug category information. The drug identification scheme provided in this solution effectively improves the accuracy of drug identification.

[0172] This application also provides an electronic device, which includes: a memory for storing computer programs;

[0173] A processor is used to execute a computer program to implement the steps of the drug identification method as described in the above embodiments.

[0174] The electronic device provided in this embodiment can be a terminal device for image recognition. The terminal device is equipped with a smart camera. The terminal device includes, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0175] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may also include a main processor and coprocessors. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0176] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory is used to store at least the following computer program, which, after being loaded and executed by a processor, is capable of implementing the relevant steps of the drug identification method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory may also include an operating system and data, and the storage method may be temporary or permanent storage. The operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, two-dimensional surface images and three-dimensional contour data, drug category information, etc.

[0177] In some embodiments, the electronic device may further include a display screen, input / output interfaces, communication interfaces, a power supply, and a communication bus.

[0178] Those skilled in the art will understand that the structures mentioned above do not constitute a limitation on electronic devices and may include more or fewer components than those shown in the figures.

[0179] It is understood that if the drug identification method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0180] Based on this, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the drug identification method described above.

[0181] The foregoing has provided a detailed description of a drug identification method, apparatus, device, and storage medium provided in the embodiments of this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0182] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] The foregoing has provided a detailed description of a drug identification method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for identifying drugs, characterized in that, include: Acquire two-dimensional surface images and three-dimensional contour data of the drug; The two-dimensional surface image is analyzed based on feature extraction rules to extract the surface structure features of the drug; wherein, the surface structure features include color features, texture features, semantic features, and local features; the semantic features include the drug traceability code; The three-dimensional contour data is converted into spatial structural features; The surface structure features, the spatial structure features, and the acquired drug physical features and text features are fused to obtain joint features; The joint features are classified using a drug classification model to determine the category information corresponding to the drug; wherein, the category information includes drug specifications, quantity and / or use.

2. The drug identification method according to claim 1, characterized in that, The two-dimensional surface image is analyzed based on feature extraction rules to extract the surface structural features of the drug, including: The color features, texture features, semantic features, and local features contained in the two-dimensional surface image are extracted using a deep learning model; wherein, the semantic features include the dosage form classification features and authenticity identification features of the drug. The text information in the two-dimensional surface image is identified using a text information recognition tool to extract character features containing drug specifications, quantity, and purpose.

3. The drug identification method according to claim 2, characterized in that, Before using a deep learning model to extract the color features, texture features, semantic features, and local features contained in the two-dimensional surface image, the process also includes: Obtain the application scenarios corresponding to drug identification; A deep learning model matching the application scenario is selected from a pre-trained deep learning model library; wherein the channel weights of the deep learning models corresponding to different application scenarios are different.

4. The drug identification method according to claim 1, characterized in that, Before analyzing the two-dimensional surface image based on feature extraction rules to extract the surface structure features of the drug, the method further includes: An adaptive illumination compensation algorithm is used to eliminate ambient light interference in the two-dimensional surface image to obtain a standard image; Accordingly, the two-dimensional surface image is analyzed based on feature extraction rules to extract the surface structural features of the drug, including: The standard image is analyzed based on feature extraction rules to extract the surface structural features of the drug.

5. The drug identification method according to claim 4, characterized in that, An adaptive illumination compensation algorithm is used to eliminate ambient light interference in the two-dimensional surface image to obtain a standard image, including: The two-dimensional surface image is decomposed into a reflection component and an illumination component; Based on the established nonlinear mapping relationship between ambient light and image brightness, the brightness of the illumination component in the two-dimensional surface image is adjusted to obtain a standard image.

6. The drug identification method according to claim 5, characterized in that, After adjusting the brightness of the illumination components in the two-dimensional surface image based on the established nonlinear mapping relationship between ambient light and image brightness to obtain a standard image, the process further includes: The actual irradiance of the two-dimensional surface image is identified based on a regression network. The actual irradiance is calibrated based on the single-point correction expected irradiance to obtain the calibrated actual irradiance. Based on the set standard irradiance, brightness adjustment speed, system gain coefficient, and the calibrated actual irradiance, the supplementary power of the light source is determined to facilitate the adjustment of the light source power of the image acquisition device.

7. The drug identification method according to any one of claims 1 to 6, characterized in that, After using a drug classification model to perform multi-label classification on the joint features to determine the category information corresponding to the drug, the method further includes: In the current application scenario of drug loading, determine whether the drug specifications, quantity and purpose included in the category information match the drug information recorded on all drug orders; If the drug specifications, quantity, and purpose contained in the category information match the drug information recorded on all drug orders, a dispensing instruction is issued to the dispensing machine so that the dispensing machine can perform the dispensing operation. In the current application scenario of dispensing medicine, it is determined whether the medicine specifications and quantities included in the category information of the medicine after dispensing match the medicine information recorded on the corresponding medication order of the user picking up the medicine. If the drug category information, drug specifications, and quantity match the drug information recorded on the corresponding user's drug order after dispensing, a bagging instruction is sent to the dispensing machine so that the dispensing machine can perform the bagging operation. In the current application scenario of bagging, determine whether the drug category information, drug specifications, and quantity contained in each medicine bag are consistent with the drug information on the corresponding user's medicine order. If the drug category information, drug specifications, and quantity in each medicine bag match the drug information on the corresponding user's medicine order, a medicine dispensing instruction is issued to the dispensing machine so that the dispensing machine can transfer the medicine to the medicine dispensing area.

8. A drug identification device, characterized in that, It includes acquisition units, extraction units, transformation units, fusion units, and classification units; The acquisition unit is used to acquire two-dimensional surface images and three-dimensional contour data of the drug. The extraction unit is used to analyze the two-dimensional surface image based on feature extraction rules to extract the surface structure features of the drug; wherein, the surface structure features include color features, texture features, semantic features, and local features; the semantic features include the drug traceability code; The conversion unit is used to convert the three-dimensional contour data into spatial structural features; The fusion unit is used to fuse the surface structure features, the spatial structure features, and the acquired drug physical features and text features to obtain joint features; The classification unit is used to perform multi-label classification on the joint features using a drug classification model to determine the category information corresponding to the drug; wherein, the category information includes drug specifications, quantity and / or use.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the drug identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the drug identification method as described in any one of claims 1 to 7.

Images