Intelligent medicine labeling method, system and equipment based on depth perception and multi-modal feature fusion and medium

Abstract

The invention relates to the technical field of medicine circulation and automation, in particular to an intelligent medicine labeling method, system and equipment based on depth perception and multi-modal feature fusion and a medium. The method comprises the steps that a visible light image and three-dimensional depth information of a medicine box are collected; carrying out multi-modal feature fusion on the visible light image and the three-dimensional depth information, carrying out drug identity verification based on a fusion result, extracting drug identification information, and carrying out comparison verification on the drug identification information and to-be-labeled information; the medicine in-place state and labeling feasibility verification is executed based on the three-dimensional depth information; medicine features are extracted based on the visible light image and the three-dimensional depth information, labeling tracks and parameters are generated based on the medicine features, and a mechanical arm is controlled to execute labeling operation; and labeling result quality detection is carried out based on the visible light image, and the mechanical arm is controlled to execute correction operation when defects are detected. The accuracy, reliability and efficiency of labeling operation can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medicine circulation and automation technology, in particular to an intelligent medicine labeling method, system, device and medium based on deep perception and multi-modal feature fusion. BACKGROUND

[0002] In the field of medicine circulation and automation, medicine labeling is a key link to ensure the accuracy and traceability of medicine information. With the popularization of intelligent pharmacies, automatic medicine vending machines and other devices, higher requirements for automation, precision and efficiency of medicine labeling are put forward to ensure medication safety and operational efficiency.

[0003] In the prior art, during medicine labeling, a recognition method based on visible light images is usually used for medicine identity verification, such as capturing images of medicine boxes through a camera and extracting features for comparison using image processing algorithms. After the medicine identity verification, a mechanical arm is used to perform the labeling operation, and a sensor is used to detect the position of the medicine to realize the automatic labeling process.

[0004] However, the existing medicine labeling technology has the following problems: medicine identity authentication mainly relies on visible light images, and the recognition accuracy is insufficient when medicines are densely placed or partially blocked; the existing method cannot accurately determine the positioning state and labeling feasibility of the medicine, which may cause misplacement or label falling off; when a mechanical arm is used for labeling operation, the labeling trajectory and parameters are usually fixed, which cannot adapt to the size and material of different medicine boxes, affecting the labeling effect; and the quality detection after labeling mainly relies on manual operation, which is low in efficiency and easy to miss. SUMMARY

[0005] To solve the technical problem that the existing medicine labeling method relies on a single visible light image, resulting in insufficient accuracy and adaptability in the aspects of medicine recognition, positioning determination, trajectory planning and quality detection, the present application provides an intelligent medicine labeling method, system, device and medium based on deep perception and multi-modal feature fusion, which realizes the intelligentization and precision of the whole process from medicine identity verification, labeling feasibility determination, adaptive trajectory planning to quality detection and correction by fusing deep perception and multi-modal features, significantly improving the accuracy, reliability and efficiency of labeling operation.

[0006] In the first aspect, the present application provides an intelligent medicine labeling method based on deep perception and multi-modal feature fusion, comprising the following steps: acquiring visible light images and three-dimensional depth information of a medicine box; performing multi-modal feature fusion on the visible light images and three-dimensional depth information, performing medicine identity verification based on the fusion results, and extracting medicine identification information for comparison and verification with the information of the label to be attached; performing medicine positioning state and labeling feasibility verification based on three-dimensional depth information; extracting drug features based on visible light images and three-dimensional depth information, generating labeling trajectories and parameters based on drug features, and controlling a mechanical arm to perform labeling operations; detecting labeling result quality based on visible light images, and controlling a mechanical arm to perform correction operations when defects are detected.

[0007] It should be further explained that the three-dimensional depth information includes spatial coordinates, plane angles, inclination angles, and minimum edge lengths. The spatial coordinates include X, Y, and Z coordinate values of the drug box in three-dimensional space, the X and Y coordinates are used to determine whether the drug box is within the working range of the mechanical arm, and the Z coordinate is used to determine the height of the drug box. The plane angle and the inclination angle are used to evaluate the orientation of the labeling surface. The minimum edge length is used to identify the upright state of the drug box.

[0008] It should be further explained that the visible light image of the drug box is collected in real time using an RGB camera. The three-dimensional depth information of the drug box is collected using a depth camera.

[0009] It should be further explained that the visible light image and the three-dimensional depth information are subjected to multi-modal feature fusion, and the drug identity verification is based on the fusion result, including: The visible light image and the three-dimensional depth information are input into a pre-trained dual-branch deep learning model to output a fusion feature vector. The fusion feature vector is compared with the standard drug feature template in the pre-set drug feature library through a SURF feature point comparison algorithm to calculate the similarity, and when the similarity score is greater than a pre-set similarity threshold, it is determined that the drug identity verification is passed. The dual-branch deep learning model includes: A super-resolution reconstruction branch is used to perform super-resolution reconstruction on the visible light image to output high-resolution image features with enhanced texture details. A target segmentation branch is used to process the visible light image and the three-dimensional depth information to output segmentation features of the drug region. A feature fusion layer is used to fuse the high-resolution image features and the segmentation features to obtain a fusion feature vector.

[0010] It should be further explained that the super-resolution reconstruction branch is based on a lightweight cross-scale residual super-resolution convolutional neural network, and its architecture includes: An input layer is used to receive a low-resolution visible light image, which is initially enlarged through adaptive bicubic interpolation, and outputs the enlarged image. a core feature extraction module, configured to receive the amplified image, perform multi-scale feature extraction through four serially connected lightweight cross-scale residual blocks, and output multi-scale feature maps, each of the lightweight cross-scale residual blocks comprising a 3*3 convolution layer, a batch normalization layer, a Swish activation function, a 1*1 convolution dimension reduction layer, and a cross-scale connection; a texture-guided enhancement module, configured to receive the multi-scale feature maps and the normalized three-dimensional depth information, multiply the normalized depth map as a texture weight map with the multi-scale feature maps pixel by pixel, and output texture-enhanced feature maps; an output layer, configured to receive the texture-enhanced feature maps, perform processing through a 3*3 convolution layer and a Tanh activation function, and output a high-resolution visible light image.

[0011] It should be further explained that the target segmentation branch is constructed based on a depth-enhanced YOLO model, and the architecture thereof comprises: an input layer, configured to receive a visible light image and three-dimensional depth information, splice a three-channel visible light image and a single-channel depth image into four-channel input data, and output a four-channel fusion tensor; a depth-enhanced backbone network, configured to receive the four-channel fusion tensor, perform feature extraction through a plurality of depth-enhanced C2f modules, and output multi-scale depth-enhanced feature maps, each of the depth-enhanced C2f modules comprising a depth-guided spatial attention mechanism; a neck network, configured to receive the multi-scale depth-enhanced feature maps, perform feature fusion through a feature pyramid structure and a path aggregation network, and output optimized multi-scale feature maps; a detection head, configured to receive the optimized multi-scale feature maps, perform anchor box prediction and depth consistency verification processing, and output segmentation features of a medicine region.

[0012] It should be further explained that the feature fusion of the high-resolution image features and the segmentation features comprises: performing three-level down-sampling processing on the high-resolution image features to match the scale of the segmentation features, and outputting aligned multi-scale image features; calculating a cosine similarity matrix of the multi-scale image features and the segmentation features, performing enhancement processing on the multi-scale image features and the segmentation features based on the cosine similarity matrix respectively, and outputting calibrated image features and calibrated segmentation features; calculating response entropy of the calibrated image features and the calibrated segmentation features respectively, calculating adaptive fusion weights based on the response entropy values, and outputting image feature weights and segmentation feature weights; performing weighted summation on the calibrated features according to the image feature weights and the segmentation feature weights, and performing dimension reduction processing through a 1*1 convolution, and finally outputting a fusion feature vector.

[0013] It should be further explained that the training steps of the dual-branch deep learning model comprise: Prepare a training data set, the training data set contains a plurality of training samples, each training sample includes a set of low-resolution visible light images, high-resolution visible light images, three-dimensional depth information and drug segmentation annotation data of a drug box; Input the low-resolution visible light image and three-dimensional depth information of the training sample into the double-branch deep learning model, perform forward propagation calculation, and output the predicted fusion feature vector; The high-resolution visible light image of the training sample is used as a true label to calculate the loss function of the super-resolution reconstruction branch, and the loss function includes L1 loss, perception loss based on VGG19 network and texture similarity loss; The drug segmentation annotation data of the training sample is used as a true label to calculate the loss function of the target segmentation branch, and the loss function includes the cross-entropy loss and Dice loss between the predicted segmentation result and the drug segmentation annotation data; The loss functions of the two branches are weighted and summed according to the preset weight to obtain the total loss of the model; According to the model total loss, perform back propagation calculation, and update all parameters of the double-branch deep learning model; Repeat the forward propagation and back propagation steps until the model total loss converges to a preset threshold.

[0014] Further need to be explained is that in the training process of the double-branch deep learning model, the specific parameters set include: In the model parameter optimization process, the AdamW optimizer is used, the initial learning rate is set to 1e-4, and the weight decay coefficient is set to 5e-6; In the training data loading process, the batch size is set to 16, and the total training round is set to 800,000 rounds; In the loss function calculation process, the loss weight of the super-resolution reconstruction branch is set to 0.4, and the loss weight of the target segmentation branch is set to 0.6; In the loss calculation of the super-resolution reconstruction branch, the L1 loss weight is set to 0.6, the perception loss weight is set to 0.3, and the texture similarity loss weight is set to 0.1; In the loss calculation of the target segmentation branch, the cross-entropy loss weight is set to 0.7, and the Dice loss weight is set to 0.3; In the learning rate adjustment process, the cosine annealing decay strategy is used, and the minimum learning rate is set to 1e-6; In the gradient back propagation process, the gradient clipping threshold is set to 1.0.

[0015] Further need to be explained is that the similarity calculation between the fusion feature vector and the standard drug feature template in the preset drug feature library through the SURF feature point comparison algorithm specifically includes: Extract key feature points from the fusion feature vector, use Hessian matrix to detect feature point position, dynamically adjust Hessian threshold according to image clarity, output key point set and corresponding 64-dimensional key point descriptor; Perform K-neighbor matching on the extracted key point descriptor, find the two nearest neighbor feature points of each key point in the preset drug feature library, and output the initial matching pair set; Perform improved Lowe ratio test on the initial matching pair set, calculate the ratio of the optimal matching distance to the suboptimal matching distance, and retain the matching pair when the ratio is less than 0.65, and output the screened matching pair set; Perform depth consistency check on the screened matching pair, calculate the depth measurement difference of the matching point pair, and eliminate the matching pair when the difference exceeds 3mm, and output the final effective matching pair set; Calculate the similarity score based on the final effective matching pair set, calculate the ratio of the number of effective matching pairs to the total number of key points, and output the final similarity score.

[0016] Further need to be explained is that in the process of extracting key feature points, the dynamic adjustment range of Hessian threshold is set to 300-400; In the K-neighbor matching process, the number of nearest neighbor search K is set to 2; In the Lowe ratio test process, the distance ratio threshold of optimal matching and suboptimal matching is set to 0.65; In the depth consistency check process, the depth tolerance threshold of the matching point pair is set to 3mm.

[0017] Further need to be explained is that extracting drug identification information and comparing and verifying with the information to be labeled includes: Collect the identification image on the surface of the medicine box, and the identification image contains a barcode or a two-dimensional code; Perform optical character recognition processing on the identification image to extract drug identification information, including drug name, batch number and expiration date; Compare the extracted drug identification information with the information to be labeled at the field level, and if the comparison result is consistent, it is determined that the comparison and verification is passed, otherwise it is determined that the comparison and verification is not passed.

[0018] Further need to be explained is that when the comparison and verification is not passed, the labeling process is terminated, and the medicine box is recycled.

[0019] Further need to be explained is that the optical character recognition processing on the identification image includes: Input the barcode or two-dimensional code image into the multi-scale feature fusion CRNN model; Extract features through MobileNetV3 backbone network and add channel attention mechanism; The sequence features are processed by bidirectional attention LSTM; The recognition result is decoded by the improved beam search.

[0020] It should be further explained that the field-level comparison includes: The complete matching verification is performed on the drug name; The format and validity period verification is performed on the batch number.

[0021] It should be further explained that the drug in-place state and labeling feasibility verification based on three-dimensional depth information includes: The time-ordered depth information sequence is generated based on the continuously collected three-dimensional depth information; The time difference algorithm is used to process the depth information sequence, calculate the depth difference between adjacent frames, and determine that the drug box reaches the labeling area when the depth difference changes from greater than the preset depth threshold to less than the depth threshold; The three-dimensional depth information of the drug box in the labeling area is obtained; Based on the three-dimensional depth information of the drug box in the labeling area, the feasibility score is calculated by the labeling feasibility evaluation model, and the following judgments are performed: When the feasibility score is greater than or equal to the preset feasibility judgment threshold, it is determined that the labeling feasibility verification is passed; When the feasibility score is less than the preset feasibility judgment threshold, it is determined that the labeling feasibility verification is not passed.

[0022] It should be further explained that the time difference algorithm is used to process the depth information sequence, which includes: The continuous frame data is extracted from the depth information sequence at a preset sampling frequency; The depth difference of the corresponding pixel points of adjacent frames is calculated to generate a depth difference matrix; The depth difference matrix is subjected to median filtering to eliminate noise interference; The proportion of pixels less than the dynamic threshold in the depth difference matrix is counted; When more than 85% of the pixel depth difference in the continuous two frames is less than the dynamic threshold, and the drug center coordinates fall within the preset range of the labeling area, it is determined that the drug reaches the labeling area.

[0023] It should be further explained that the dynamic threshold in the time difference algorithm The following formula is used to calculate:

[0024] Wherein is the size factor, which is calculated as the ratio of the actual diameter of the drug to the standard diameter; is the distance from the drug to the collection device; For the depth confidence factor, take 1.0 when the depth measurement confidence is greater than 0.9, take 1.2 when the confidence is between 0.7-0.9, and take 1.5 when the confidence is less than 0.7.

[0025] It should be further explained that the calculation of the feasibility score by the labeling feasibility evaluation model includes the following steps: An evaluation index system is established, including position accuracy, attitude stability, surface state and environmental adaptation as four first-level indexes, and each first-level index includes two second-level indexes; Each second-level index is fuzzed, and the membership score of each index is calculated using a triangular membership function; The weight coefficients of each index are calculated by a random forest regression model; An environmental correction factor is introduced to dynamically adjust the weight coefficients; The comprehensive score is calculated according to the weighted sum formula, which is:

[0026] Wherein, is the adjusted weight coefficient of the th index; is the membership score of the th index.

[0027] It should be further explained that the evaluation index system includes the following eight second-level indexes: Center offset, calculated as the ratio of actual offset distance to maximum allowed offset; Angle offset, calculated as the ratio of actual offset angle to maximum allowed angle; Angle variance, calculated based on the variance of angle data of five consecutive frames; Position variance, calculated based on the variance of position data of five consecutive frames; Surface flatness, represented by the depth variance of the labeling area; Surface roughness, calculated by surface texture features; Occlusion rate, calculated as the ratio of occluded pixels to total pixels; Light uniformity, represented by the standard deviation of light intensity.

[0028] It should be further explained that the setting of the environmental correction factor includes: When the environmental humidity exceeds 60%, the weight coefficient of surface flatness is multiplied by 1.2; When the light non-uniformity exceeds the standard value, the weight coefficient of light uniformity is multiplied by 1.3; When the environmental temperature exceeds the working range, all weight coefficients are uniformly multiplied by 0.8.

[0029] It should be further explained that when it is determined that the labeling feasibility verification fails, the following processing is performed: When the three-dimensional coordinates of the medicine box exceed the working range of the mechanical arm, triggering the mechanical arm to perform a medicine recovery operation; When the medicine box is in an upright state, triggering the mechanical arm to perform a push-down operation; When the inclination angle of the medicine box exceeds the allowed range, triggering the mechanical arm to perform a posture adjustment operation.

[0030] It should be further explained that based on the visible light image and the three-dimensional depth information, the medicine features are extracted, the labeling trajectory and parameters are generated based on the medicine features, and the mechanical arm is controlled to perform the labeling operation, which specifically includes: Establishing a dynamic mapping relationship among the mechanical arm coordinate system, the depth camera coordinate system and the RGB camera coordinate system; Based on the visible light image and the three-dimensional depth information, the size data and the spatial coordinates of the medicine box are extracted, and based on the size data and the spatial coordinates of the medicine box, the optimal labeling trajectory is generated by using a fusion ant colony algorithm and a 3-5-3 interval planning strategy; Based on historical labeling records and medicine box material characteristics, the labeling parameters are optimized by a reinforcement learning algorithm model, including the labeling pressure, the motion speed and the label preheating temperature parameters of the mechanical arm; According to the optimal labeling trajectory and the optimized labeling parameters, the mechanical arm is controlled to perform the labeling operation, and the motion state of the mechanical arm is monitored in real time during the execution process.

[0031] It should be further explained that establishing a dynamic mapping relationship among the mechanical arm coordinate system, the depth camera coordinate system and the RGB camera coordinate system specifically includes: The conversion matrix among the coordinate systems is calculated by a hand-eye calibration algorithm, and the Tsai-Lenz method is used to solve the rotation and translation parameters; The conversion matrix is calibrated in real time by a Kalman filter to eliminate the coordinate drift error caused by equipment vibration; The coordinate conversion error tolerance is set to ±0.1mm, and when it is detected that the error exceeds the limit, automatic recalibration is performed.

[0032] It should be further explained that generating the optimal labeling trajectory includes: The particle initialization process of the ant colony algorithm is optimized by using a double chaotic Logit-Tent mapping; The optimal path is searched in the solution space through a pheromone concentration updating mechanism; A 3-5-3 interval planning strategy is used to divide the complete trajectory into an acceleration segment containing 3 sampling points, a uniform speed segment containing 5 sampling points and a deceleration segment containing 3 sampling points; The trajectory adaptation error threshold is set to 0.3mm, and the trajectory is re-planned when the trajectory error exceeds the threshold.

[0033] Further, the label parameter optimization by the reinforcement learning algorithm comprises: The state space is defined to include 8 dimensions of drug material, environmental temperature, humidity, historical parameters, surface roughness, label viscosity, surface reflectivity, and label material; The action space is defined to include 3 dimensions of label pressure increment, motion speed increment, and preheating temperature increment; The reward function is designed to include label quality score, label efficiency score, running stability score, and defect penalty term; The transfer deep deterministic policy gradient algorithm is used for incremental learning based on the pre-trained model.

[0034] Further, the specific calculation of the reward function comprises: The label quality score is calculated based on label offset and bubble number, 10 points for offset≤1mm, 5 points for 1mm<offset≤2mm, 15 points deducted for offset>2mm, 5 points for no bubbles, and 5 points deducted for more than 3 bubbles; The label efficiency score is calculated based on label time, 3 points for label time≤0.4s, 1 point for 0.4s<label time≤0.6s, and 2 points deducted for label time>0.6s; The running stability score is calculated based on consecutive qualified times, 3 points for 3 consecutive qualified times, and 0 points otherwise; The defect penalty term includes 50 points deducted for damage to the medicine box and 20 points deducted for label creases.

[0035] Further, the control of the robot arm to perform the labeling operation comprises: According to the optimal label trajectory, joint motion instructions are generated, and each joint angle is accurately controlled through a PID controller; The joint torque and end effector position are monitored in real time, and the motion is immediately paused when abnormal vibration or position deviation is detected; The actual parameters and results of each labeling are recorded and updated to the historical labeling database for optimization of the reinforcement learning algorithm model.

[0036] Further, the label result quality detection based on visible light images comprises: Visible light images of the labeled medicine box are collected, and the images are preprocessed to output standardized images; The standardized images are input into a rotation-invariant feature extraction network based on ResNet-50 to extract multi-angle features and output feature vectors; The cosine similarity between the feature vectors and the standard templates in the preset qualified feature library is calculated, and the similarity score is output. determine the quality level of the labeling according to the similarity score: when the similarity is greater than or equal to 0.95, it is determined to be qualified, when 0.9 is less than the similarity and less than 0.95, it is determined to be a slight defect, and when the similarity is less than 0.9, it is determined to be a serious defect; When it is determined that there is a defect, trigger the corresponding mechanical arm correction action according to the defect level.

[0037] It should be further explained that when a defect is detected, the mechanical arm is controlled to perform a correction operation, which includes: When it is determined to be a slight defect, the mechanical arm is controlled to drive the labeling head to translate in the opposite direction of the offset, with a translation accuracy of 0.1 mm, and at the same time, a rotation correction is performed, with a rotation accuracy of 0.1°; When it is determined to be a serious defect, the end of the mechanical arm is controlled to adsorb the edge of the label with a suction force of 5N, and the label is peeled off at a speed of 5mm / s; The labeling parameters optimized by reinforcement learning are called to re-perform the labeling operation on the peeled-off medicine; After the correction is completed, visible light images are re-acquired for quality detection, and if they are still unqualified, warning information is generated and pushed to the user.

[0038] It should be further explained that the multi-angle feature extraction specifically includes: The standardized image is respectively rotated by 0°, 90°, 180° and 310° to generate four rotated version images; The four rotated version images are respectively input into the ResNet-50 network, and the rotated consistent attention module is processed at the conv3_4 layer; The rotation consistency weight of the four rotation angle feature maps is calculated to strengthen the rotation invariant feature expression; The weighted feature map is input into the subsequent network layer, and a 2048-dimensional feature vector is obtained through global average pooling.

[0039] It should be further explained that the qualified feature library is constructed by the following method: 10000 groups of qualified labeling sample images are collected, including multiple angle qualified labeling images of each medicine box; Use Triplet Loss metric learning to train the feature encoder to ensure that the intra-class distance is less than 0.3 and the inter-class distance is greater than 0.8; Cluster analysis is performed on the feature vectors of each medicine box to generate 10 cluster centers as the qualified feature templates of the medicine box of this type; The qualified feature templates are stored in the vector database to support real-time retrieval based on cosine similarity.

[0040] It should be further explained that the image preprocessing specifically includes: Gaussian filter denoising is performed on the collected visible light image, and the filter kernel size is set to 3x3; Histogram equalization processing is performed to enhance the contrast of the image. Color correction is performed, and the image color temperature is adjusted using a white balance algorithm. The image is cropped to a fixed size, and the labeling area is retained as the detection area.

[0041] In a second aspect, the present application provides an intelligent medicine labeling system based on deep perception and multi-modal feature fusion, which is used to implement the above-mentioned intelligent medicine labeling method, comprising: A medicine box information acquisition module is used to acquire visible light images and three-dimensional depth information of the medicine box. A medicine identity verification and comparison verification module is used to perform multi-modal feature fusion on the visible light images and three-dimensional depth information, perform medicine identity verification based on the fusion results, and extract medicine identification information for comparison verification with the label information to be labeled. A labeling feasibility verification module is used to perform medicine in-place state and labeling feasibility verification based on three-dimensional depth information. A labeling execution module is used to extract medicine features based on visible light images and three-dimensional depth information, generate labeling trajectories and parameters based on the medicine features, and control the mechanical arm to perform labeling operations. A labeling result quality detection and correction module is used to detect the quality of the labeling result based on the visible light image, and control the mechanical arm to perform correction operations when defects are detected.

[0042] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor is used to implement the steps of the above-mentioned intelligent medicine labeling method when the computer program is executed.

[0043] In a fourth aspect, the present application provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned intelligent medicine labeling method.

[0044] From the above technical solutions, the present application has the following advantages: 1. The present application continuously acquires visible light images and three-dimensional depth information of the medicine box, and performs medicine identity verification based on multi-modal features, which effectively improves the accuracy and robustness of medicine identification, especially in dense placement or occlusion scenarios, reducing the risk of misjudgment and missed judgment.

[0045] 2. The present application performs medicine in-place state and labeling feasibility verification based on three-dimensional depth information, which can detect position deviation or posture problems in advance, avoid invalid labeling, reduce label waste and operation errors, and improve the reliability of the labeling process.

[0046] 3. The application extracts drug features based on visible light images and three-dimensional depth information, generates adaptive labeling trajectories and parameters, and controls the robot arm to perform labeling operations, thereby adapting to the size and material of different drug boxes, improving labeling accuracy and pass rate, and reducing defects caused by trajectory mismatch.

[0047] 4. The application detects labeling result quality based on visible light images, and controls the robot arm to perform correction operations when defects are detected, thereby realizing closed-loop quality control of the labeling process, reducing manual reinspection costs, and improving overall efficiency and labeling consistency. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description, obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 is a flow chart of an intelligent drug labeling method based on depth perception and multi-modal feature fusion in an embodiment of the present application.

[0050] Figure 2 is a schematic block diagram of an intelligent drug labeling system based on depth perception and multi-modal feature fusion in an embodiment of the present application.

[0051] Figure 3 is a schematic diagram of the hardware structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the application purposes, features, advantages of the present application more obvious and easy to understand, the following will use specific embodiments and drawings to clearly and completely describe the technical solutions protected by the present application. Obviously, the following described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] The intelligent drug labeling method related to the present application will be described in detail below. In order to illustrate but not to limit, specific details such as specific system structure, technology are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details.

[0054] In the intelligent medicine labeling method related in the present application, the term "comprising" indicates the presence of the described features, whole, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0055] In order to clearly describe the technical solutions of the present application, the terms "first", "second" and the like are used to distinguish the same or similar items or components with basically the same function and role. Those skilled in the art can understand that the terms "first", "second" and the like do not limit the quantity and execution order, and the terms "first", "second" and the like do not necessarily mean different.

[0056] The phrases "one embodiment", "some embodiments" and the like appearing in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in other some embodiments" and the like appearing in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application.

[0058] The intelligent medicine labeling method provided by the embodiments of the present application is executed by a computer device, and accordingly, the intelligent medicine labeling system based on depth perception and multi-modal feature fusion runs in the computer device.

[0059] Figure 1 is a flowchart of the intelligent medicine labeling method based on depth perception and multi-modal feature fusion according to one embodiment of the present application. In the flowchart, Figure 1 The execution subject can be an intelligent medicine labeling system. According to different needs, the order of steps in the flowchart can be changed, and some can be omitted.

[0060] As Figure 1 shown, the intelligent medicine labeling method based on depth perception and multi-modal feature fusion includes: Step S1, acquiring visible light images and three-dimensional depth information of a medicine box.

[0061] By collecting the visible light image and three-dimensional depth information of the medicine box, a comprehensive and reliable multi-modal data basis is provided for subsequent accurate verification of medicine identity, accurate judgment of in-place state, and intelligent generation of labeling trajectory and parameters, which fundamentally overcomes the lack of sensing ability of single visible light sensor in complex scenes.

[0062] In some embodiments, in step S1, the three-dimensional depth information includes spatial coordinates, plane angle, inclination angle, and minimum side length. The spatial coordinates include X, Y, Z coordinate values of the medicine box in three-dimensional space, X, Y coordinates are used to determine whether the medicine box is within the working range of the mechanical arm, and Z coordinate is used to determine the height of the medicine box. The plane angle and inclination angle are used to evaluate the orientation of the labeling surface. The minimum side length is used to identify the upright state of the medicine box.

[0063] By specifying the three-dimensional depth information to include specific parameters such as spatial coordinates, plane angle, inclination angle, and minimum side length, the three-dimensional perception data can be accurately used to determine whether the medicine is within the working range of the mechanical arm, evaluate the orientation of the labeling surface, and identify the upright state, etc. Key pre-judgment links, providing clear and quantifiable decision-making basis for subsequent feasibility evaluation and mechanical arm control.

[0064] In some embodiments, in step S1, an RGB camera is used to collect real-time visible light images of the medicine box. A depth camera is used to collect three-dimensional depth information of the medicine box.

[0065] By using an RGB camera to collect real-time visible light images and a depth camera to collect three-dimensional depth information, a low-cost, easy-to-deploy, and information-complementary dual-modal sensing system is constructed, providing stable and high-quality two-dimensional texture and three-dimensional geometric data sources for the entire intelligent labeling device, ensuring that the input information for subsequent algorithm processing is rich and reliable.

[0066] In step S2, the visible light image and three-dimensional depth information are subjected to multi-modal feature fusion, the medicine identity is verified based on the fusion result, and the medicine identification information is extracted and compared with the label information to be pasted.

[0067] By multi-modal feature fusion of the visible light image and three-dimensional depth information, the medicine identity is verified based on the fusion result, and the medicine identification information is extracted and compared with the label information to be pasted, which can accurately verify the medicine identity and label matching at the front end, effectively preventing serious errors such as medicine mislabeling and label mislabeling, and ensuring the safety of medication and the accuracy of labeling operation from the source.

[0068] In some specific embodiments, the visible light image and the three-dimensional depth information are subjected to multi-modal feature fusion, and the drug identity verification based on the fusion result comprises: inputting the visible light image and the three-dimensional depth information into a pre-trained double-branch deep learning model to output a fusion feature vector; performing similarity calculation on the fusion feature vector and a standard drug feature template in a preset drug feature library through a SURF feature point comparison algorithm, and determining that the drug identity verification is passed when a similarity score is greater than a preset similarity threshold; the double-branch deep learning model comprises: a super-resolution reconstruction branch for super-resolution reconstruction of the visible light image to output a high-resolution image feature with enhanced texture details; a target segmentation branch for processing the visible light image and the three-dimensional depth information to output a segmentation feature of a drug region; a feature fusion layer for performing feature fusion on the high-resolution image feature and the segmentation feature to obtain the fusion feature vector.

[0069] By using the pre-trained double-branch deep learning model to process the visible light image and the three-dimensional depth information to output the fusion feature vector, and using the SURF feature point comparison algorithm to perform similarity calculation with the drug feature library, the powerful feature extraction capability of deep learning and the robustness of traditional algorithms in feature matching are fully utilized, thereby ensuring high precision and high reliability of the drug identity verification link.

[0070] In some specific embodiments, the super-resolution reconstruction branch is constructed based on a lightweight cross-scale residual super-resolution convolutional neural network, and the architecture thereof comprises: an input layer for receiving a low-resolution visible light image, performing preliminary enlargement through adaptive bicubic interpolation, and outputting an enlarged image; a core feature extraction module for receiving the enlarged image, performing multi-scale feature extraction through four serial lightweight cross-scale residual blocks, and outputting multi-scale feature maps, each lightweight cross-scale residual block comprising a 3x3 convolution layer, a batch normalization layer, a Swish activation function, a 1x1 convolution dimension reduction layer, and a cross-scale connection; a texture-guided enhancement module for receiving the multi-scale feature maps and normalized three-dimensional depth information, multiplying the normalized depth map as a texture weight map with the multi-scale feature maps pixel by pixel to output texture-enhanced feature maps; an output layer for receiving the texture-enhanced feature maps, performing processing through a 3x3 convolution layer and a Tanh activation function, and outputting a high-resolution visible light image.

[0071] By constructing a lightweight cross-scale residual super-resolution convolutional neural network including an input layer, a core feature extraction module, a texture-guided enhancement module, and an output layer as a super-resolution reconstruction branch, the detail clarity and texture richness of the low-resolution input image can be effectively improved while keeping low computational complexity, providing a higher quality image basis for subsequent drug identification and feature comparison.

[0072] In some embodiments, the target segmentation branch is constructed based on a depth-enhanced YOLO model, and the architecture thereof includes: An input layer for receiving a visible light image and three-dimensional depth information, concatenating a three-channel visible light image and a single-channel depth image into four-channel input data, and outputting a four-channel fusion tensor; A depth-enhanced backbone network for receiving the four-channel fusion tensor, performing feature extraction through multiple depth-enhanced C2f modules, and outputting multi-scale depth-enhanced feature maps, each depth-enhanced C2f module including a depth-guided spatial attention mechanism; A neck network for receiving the multi-scale depth-enhanced feature maps, performing feature fusion through a feature pyramid structure and a path aggregation network, and outputting optimized multi-scale feature maps; A detection head for receiving the optimized multi-scale feature maps, performing anchor box prediction and depth consistency verification processing, and outputting segmentation features of the drug region.

[0073] By constructing a depth-enhanced YOLO model including an input layer, a depth-enhanced backbone network, a neck network, and a detection head as a target segmentation branch, and using the concatenated four-channel input data and the depth-guided spatial attention mechanism, the segmentation accuracy and robustness of the model for the drug region, especially for overlapping or occluded drugs, in complex backgrounds are significantly improved.

[0074] In some embodiments, the feature fusion of the high-resolution image features and the segmentation features includes: Performing three-level down-sampling processing on the high-resolution image features to match their scale with that of the segmentation features, and outputting aligned multi-scale image features; Calculating a cosine similarity matrix of the multi-scale image features and the segmentation features, enhancing the multi-scale image features and the segmentation features based on the cosine similarity matrix, and outputting calibrated image features and calibrated segmentation features; Calculating the response entropy of the calibrated image features and the calibrated segmentation features, respectively, calculating adaptive fusion weights based on the response entropy values, and outputting image feature weights and segmentation feature weights; Performing weighted summation on the calibrated features according to the image feature weights and the segmentation feature weights, and performing dimension reduction processing through 1x1 convolution, and finally outputting a fusion feature vector.

[0075] By adopting a multi-step fusion strategy including down-sampling alignment, cosine similarity-based feature calibration, response entropy-based adaptive weight calculation, and weighted summation and dimension reduction, deep complementarity and effective integration between the high-resolution image features of the super-resolution reconstruction branch and the segmentation features of the target segmentation branch are achieved, and a more discriminative fusion feature vector is generated.

[0076] In some embodiments, the training step of the dual-branch deep learning model comprises: A training data set is prepared, which contains a plurality of training samples, each training sample including a set of low-resolution visible light images, high-resolution visible light images, three-dimensional depth information, and drug segmentation annotation data of a drug box; The low-resolution visible light image and three-dimensional depth information of the training sample are input into the dual-branch deep learning model, and forward propagation calculation is performed to output the predicted fusion feature vector; The high-resolution visible light image of the training sample is used as the true label to calculate the loss function of the super-resolution reconstruction branch, which includes L1 loss, perception loss based on VGG19 network, and texture similarity loss; The drug segmentation annotation data of the training sample is used as the true label to calculate the loss function of the target segmentation branch, which includes cross-entropy loss and Dice loss between the predicted segmentation result and the drug segmentation annotation data; The loss functions of the two branches are weighted and summed according to the preset weight to obtain the total loss of the model; According to the total loss of the model, back propagation calculation is performed to update all parameters of the dual-branch deep learning model; The forward propagation and back propagation steps are repeated until the total loss of the model converges to a preset threshold.

[0077] By training the super-resolution reconstruction branch using a multi-task loss function combination including L1 loss, perception loss, and texture similarity loss, and training the target segmentation branch using cross-entropy loss and Dice loss, and finally weighting and summing the losses of the two branches to jointly optimize the entire model, it is ensured that the dual-branch deep learning model can simultaneously consider image reconstruction quality and target segmentation accuracy.

[0078] In some embodiments, during the training process of the dual-branch deep learning model, the specific parameters set include: During the model parameter optimization process, the AdamW optimizer is used, with an initial learning rate of 1e-4 and a weight decay coefficient of 5e-6; During the training data loading process, the batch size is set to 16 and the total number of training rounds is set to 800,000 rounds; During the loss function calculation process, the loss weight of the super-resolution reconstruction branch is set to 0.4, and the loss weight of the target segmentation branch is set to 0.6. In the loss calculation of the super-resolution reconstruction branch, the L1 loss weight is set to 0.6, the perceptual loss weight is set to 0.3, and the texture similarity loss weight is set to 0.1; In the loss calculation of the target segmentation branch, the cross-entropy loss weight is set to 0.7, and the Dice loss weight is set to 0.3; In the learning rate adjustment process, the cosine annealing decay strategy is adopted, and the minimum learning rate is set to 1e-6; In the gradient backpropagation process, the gradient clipping threshold is set to 1.0.

[0079] By specifically setting the AdamW optimizer, initial learning rate, weight decay, batch size, training rounds, loss function weights, cosine annealing decay strategy, and gradient clipping threshold during model training, a stable and efficient training environment is provided for the dual-branch deep learning model, ensuring that the model can fully learn the data features and achieve the expected performance convergence.

[0080] In some specific embodiments, the similarity calculation between the fusion feature vector and the standard drug feature template in the preset drug feature library through the SURF feature point comparison algorithm specifically includes: Extract key feature points from the fusion feature vector, detect feature point positions using the Hessian matrix, dynamically adjust the Hessian threshold according to image clarity, output the key point set and corresponding 64-dimensional key point descriptors; Perform K-nearest neighbor matching on the extracted key point descriptors, find the two nearest neighbor feature points of each key point in the preset drug feature library, and output the initial matching pair set; Perform improved Lowe ratio test on the initial matching pair set, calculate the ratio of the optimal matching distance to the suboptimal matching distance, and retain the matching pair when the ratio is less than 0.65, output the filtered matching pair set; Perform depth consistency verification on the filtered matching pairs, calculate the depth measurement difference of the matching point pairs, and eliminate the matching pair when the difference exceeds 3mm, output the final valid matching pair set; Calculate the similarity score based on the final valid matching pair set, calculate the ratio of the number of valid matching pairs to the total number of key points, and output the final similarity score.

[0081] By adopting a series of refined steps such as dynamically adjusting the Hessian threshold to extract key points, K-nearest neighbor matching, improved Lowe ratio test, and depth consistency verification, SURF feature point comparison and similarity calculation are completed, significantly improving the accuracy and anti-interference ability of feature matching, effectively reducing the risk of false matching caused by image noise, deformation, or partial occlusion.

[0082] In some embodiments, during the extraction of key feature points, the dynamic adjustment range of the Hessian threshold is set to 300-400. In the K nearest neighbor matching process, the number of nearest neighbor searches K is set to 2. In the Lowe ratio test process, the distance ratio threshold of the optimal match and the suboptimal match is set to 0.65. In the depth consistency verification process, the depth tolerance threshold of the matched point pair is set to 3mm.

[0083] By setting a specific numerical range or fixed value for the key parameters in the SURF feature point extraction and matching process, the feature comparison algorithm provides reliable parameter support for stable operation and consistent performance under different working conditions, enhancing the repeatability and robustness of the entire identification process.

[0084] In some embodiments, the extraction of the drug identification information and the comparison and verification with the information to be labeled include: Collecting an identification image on the surface of the medicine box, the identification image containing a barcode or a two-dimensional code; Performing optical character recognition processing on the identification image to extract drug identification information, the drug identification information including the drug name, batch number and expiration date; Comparing and verifying the extracted drug identification information with the information to be labeled at the field level, and determining that the comparison and verification is passed when the comparison result is consistent, otherwise determining that the comparison and verification is not passed.

[0085] By collecting an identification image on the surface of the medicine box and performing optical character recognition processing to extract key identification information such as the drug name, batch number and expiration date, and comparing and verifying the information to be labeled at the field level, the automatic and high-precision reading and verification of the drug identification information is realized, providing a reliable technical means for the secondary confirmation of the drug identity.

[0086] In some embodiments, when the comparison and verification is not passed, the labeling process is terminated and the medicine box is recycled.

[0087] By immediately terminating the labeling process and recycling the medicine box when the comparison and verification is not passed, a quick response safety interception mechanism is established, which can effectively prevent the wrong label from being attached to the mismatched medicine, prevent the wrong flow, and effectively ensure the safety of the end user and the standardization of the production process.

[0088] In some embodiments, the optical character recognition processing on the identification image includes: Inputting the barcode or two-dimensional code image into a multi-scale feature fusion CRNN model; Extracting features through a MobileNetV3 backbone network and adding a channel attention mechanism; The sequence features are processed by bidirectional attention LSTM; The recognition result is output by improved beam search decoding.

[0089] By adopting the multi-scale feature fusion CRNN model of the integrated MobileNetV3 backbone network, the channel attention mechanism, the bidirectional attention LSTM, and the improved beam search decoding, the identification image is subjected to optical character recognition, the recognition speed is greatly improved while ensuring the recognition accuracy, the adaptability to complex scenes such as inclination and blur is improved, and the demand for online real-time processing is met.

[0090] In some specific embodiments, the field-level comparison includes: Performing complete matching verification on the drug name; Performing format and expiration date verification on the batch number.

[0091] By performing complete matching verification on the drug name and performing format and expiration date verification on the batch number in the field-level comparison, a differentiated verification strictness level is set, which not only ensures the absolute accuracy of the most critical information, but also retains certain fault tolerance flexibility for secondary information, thereby improving the practicability and pass rate of the system while ensuring safety.

[0092] Step S3, performing drug in-place state and labeling feasibility verification based on three-dimensional depth information.

[0093] By performing drug in-place state and labeling feasibility verification based on three-dimensional depth information, it can be determined in advance whether the drug has accurately arrived at the station and whether its position and posture meet the labeling requirements before the labeling action is performed, thereby avoiding invalid or incorrect labeling operations when the conditions are not met, reducing label waste and improving the smoothness and success rate of the process.

[0094] In some specific embodiments, in step S3, performing drug in-place state and labeling feasibility verification based on three-dimensional depth information includes: Generating a depth information sequence sorted in time sequence based on the continuously collected three-dimensional depth information; Using a time series difference algorithm to process the depth information sequence, calculating the depth difference between adjacent frames, and determining that the drug box has arrived at the labeling area when the depth difference changes from greater than a preset depth threshold to less than the depth threshold; Obtaining three-dimensional depth information of the drug box in the labeling area; Based on the three-dimensional depth information of the drug box in the labeling area, calculating a feasibility score by a labeling feasibility evaluation model, and performing the following judgment: When the feasibility score is greater than or equal to a preset feasibility judgment threshold, it is determined that the labeling feasibility verification is passed; When the feasibility score is less than the preset feasibility judgment threshold, it is determined that the labeling feasibility verification fails.

[0095] By using the time series difference algorithm based on the continuously collected three-dimensional depth information sequence to determine the arrival state of the medicine, and combining the labeling feasibility evaluation model to calculate and determine the feasibility score of the arrived medicine, a complete prediction chain from perception to decision is constructed, the intelligent and quantitative judgment of the labeling time and conditions is realized, and blind operation is avoided.

[0096] In some embodiments, processing the depth information sequence using the time series difference algorithm includes: extracting continuous frame data from the depth information sequence at a preset sampling frequency; calculating the depth difference value of the corresponding pixel points of adjacent frames to generate a depth difference value matrix; median filtering the depth difference value matrix to eliminate noise interference; statistically analyzing the proportion of pixels less than the dynamic threshold in the depth difference value matrix; When more than 85% of the pixel depth difference values in the two continuous frames are less than the dynamic threshold, and the center coordinates of the medicine fall within the preset range of the labeling area, it is determined that the medicine has arrived in the labeling area.

[0097] By using the time series difference algorithm to process the depth information sequence including extracting continuous frames, calculating the depth difference value matrix, median filtering to remove noise, statistically analyzing the proportion of pixels less than the dynamic threshold, and combining the center coordinates of the medicine falling within the preset range, etc. Multiple steps are realized. The stable, reliable and low delay detection of whether the medicine has accurately arrived in the labeling area provides a key signal for the accurate triggering of the subsequent process.

[0098] In some embodiments, the dynamic threshold in the time series difference algorithm The following formula is used to calculate:

[0099] wherein is a size factor, calculated as the ratio of the actual diameter of the medicine to the standard diameter; is the distance from the medicine to the collection device; is a depth confidence factor, which is 1.0 when the depth measurement confidence is greater than 0.9, 1.2 when the confidence is between 0.7 and 0.9, and 1.5 when the confidence is less than 0.7.

[0100] By employing a formula that combines size factor, distance from the drug to the acquisition device, and depth confidence factor to calculate the dynamic threshold in the time-series difference algorithm, the threshold can adapt to changes in the actual physical size of the drug, distance, and depth measurement quality, thereby maintaining high detection sensitivity and accuracy under different working conditions.

[0101] In some specific embodiments, calculating the feasibility score using the labeling feasibility assessment model includes the following steps: An evaluation index system was established, which includes four primary indicators: positional accuracy, attitude stability, surface condition, and environmental adaptation. Each primary indicator contains two secondary indicators. Each secondary indicator is fuzzified, and the membership score of each indicator is calculated using a triangular membership function. The weight coefficients of each indicator are calculated using a random forest regression model; Introduce an environmental correction factor to dynamically adjust the weighting coefficients; The overall score is calculated using the weighted summation formula, which is as follows:

[0102] in, For the first The adjusted weighting coefficients of each indicator; For the first The membership score of each indicator.

[0103] By establishing an evaluation system with multi-level indicators, fuzzifying each secondary indicator, and then dynamically weighting and summing the weight coefficients calculated by the random forest regression model and the environmental correction factor to obtain a comprehensive feasibility score, an objective and quantitative labeling feasibility assessment method that can comprehensively consider multiple factors and adapt to environmental changes has been realized.

[0104] In some specific embodiments, the evaluation index system includes the following eight secondary indicators: The center offset is calculated as the ratio of the actual offset distance to the maximum allowable offset. Angular offset is calculated as the ratio of the actual offset angle to the maximum allowable angle. Angle variance is calculated based on angle data from five consecutive frames. Location variance is calculated based on location data from five consecutive frames. Surface smoothness is characterized by the depth variance of the labeled area; Surface roughness is calculated using surface texture features; Occlusion rate is calculated as the ratio of occluded pixels to the total number of pixels. Illumination uniformity is characterized by the standard deviation of illumination intensity.

[0105] By explicitly setting eight specific secondary indicators, a comprehensive and quantifiable evaluation dimension is provided for the labeling feasibility evaluation model, ensuring that the evaluation results can truly and comprehensively reflect the state and environmental conditions of the drug.

[0106] In some specific embodiments, the setting of the environmental correction factor includes: When the environmental humidity exceeds 60%, the weight coefficient of the surface flatness is multiplied by 1.2; When the light unevenness exceeds the standard value, the weight coefficient of the light uniformity is multiplied by 1.3; When the environmental temperature exceeds the working range, all weight coefficients are uniformly multiplied by 0.8.

[0107] By dynamically adjusting the weight coefficients of indicators such as surface flatness and light uniformity according to actual working conditions such as environmental humidity, light uniformity, and environmental temperature, and even performing global weight correction, the labeling feasibility evaluation model can flexibly adapt to changes in the external environment, maintain the accuracy of the evaluation results and the effectiveness of the decision-making, and enhance the environmental robustness of the system.

[0108] In some specific embodiments, when it is determined that the labeling feasibility verification fails, the following processing is performed: When the three-dimensional coordinates of the drug box exceed the working range of the mechanical arm, the mechanical arm is triggered to perform a drug recovery operation; When the drug box is in an upright state, the mechanical arm is triggered to perform a push-down operation; When the inclination angle of the drug box exceeds the allowed range, the mechanical arm is triggered to perform a posture adjustment operation.

[0109] By triggering the mechanical arm to perform differentiated processing operations such as drug recovery, push-down, or posture adjustment according to the specific abnormal conditions of the drug (such as coordinate overrun, upright state, and inclination overrun) when it is determined that the labeling feasibility verification fails, the automatic and targeted disposal of abnormal drugs is realized, and the continuous and stable operation of the production line is ensured.

[0110] Step S4: Extracting drug features based on visible light images and three-dimensional depth information, generating labeling trajectories and parameters based on drug features, and controlling the mechanical arm to perform labeling operations.

[0111] By extracting drug features based on visible light images and three-dimensional depth information, and generating labeling trajectories and parameters to control the mechanical arm to perform labeling operations, the labeling process can adapt to different drug sizes, shapes, and spatial postures, achieving precise and personalized labeling actions, and significantly improving the position accuracy and fitting quality of label attachment.

[0112] In some specific embodiments, step S4 specifically includes: establishing a dynamic mapping relationship among the robot arm coordinate system, the depth camera coordinate system, and the RGB camera coordinate system; Based on the visible light image and the three-dimensional depth information, the size data and the spatial coordinates of the medicine box are extracted, and based on the size data and the spatial coordinates of the medicine box, an optimal labeling trajectory is generated by using a fusion ant colony algorithm and a 3-5-3 interval planning strategy. Based on the historical labeling records and the material characteristics of the medicine box, the labeling parameters are optimized by using a reinforcement learning algorithm model, including the labeling pressure, the motion speed, and the label preheating temperature parameters of the robot arm. According to the optimal labeling trajectory and the optimized labeling parameters, the robot arm is controlled to perform the labeling operation, and the motion state of the robot arm is monitored in real time during the execution.

[0113] By establishing a dynamic mapping relationship among the robot arm coordinate system, the depth camera coordinate system, and the RGB camera coordinate system, using a fusion optimization algorithm to generate an optimal labeling trajectory, and based on historical data and drug characteristics, optimizing the labeling parameters by reinforcement learning, the robot arm is finally controlled to perform labeling and state monitoring, realizing a highly adaptive, high-precision, and closed-loop optimized labeling execution control scheme, which significantly improves the labeling success rate and quality.

[0114] In some specific embodiments, establishing a dynamic mapping relationship among the robot arm coordinate system, the depth camera coordinate system, and the RGB camera coordinate system specifically includes: The conversion matrix between the coordinate systems is calculated by using a hand-eye calibration algorithm, and the Tsai-Lenz method is used to solve the rotation and translation parameters. The conversion matrix is calibrated in real time by using a Kalman filter to eliminate the coordinate drift error caused by device vibration. The coordinate conversion error tolerance is set to ±0.1 mm, and when the error is detected to be out of limit, automatic re-calibration is performed.

[0115] By using a hand-eye calibration algorithm to calculate the conversion matrix, using a Kalman filter for real-time calibration, and setting the error tolerance and automatic re-calibration mechanism, the long-term accuracy and stability of the mapping relationship among the robot arm coordinate system, the depth camera coordinate system, and the RGB camera coordinate system are ensured, laying a solid foundation for precise motion control of the robot arm.

[0116] In some specific embodiments, generating an optimal labeling trajectory includes: The particle initialization process of the ant colony algorithm is optimized by using a double chaotic Logit-Tent mapping. The optimal path is searched in the solution space by using an information concentration updating mechanism. A 3-5-3 interval planning strategy is used to divide the complete trajectory into an acceleration segment containing 3 sampling points, a uniform speed segment containing 5 sampling points, and a deceleration segment containing 3 sampling points. Set the trajectory adaptation error threshold to 0.3mm, and re-plan when the trajectory error exceeds the threshold.

[0117] By adopting the initialization process of the double chaotic mapping optimization ant colony algorithm and combining the 3-5-3 interval planning strategy to generate the mechanical arm labeling trajectory, the convergence speed and global optimization ability of the trajectory planning are improved, the stability and precision of the mechanical arm movement process are ensured, and the vibration, impact and position deviation in the labeling process are effectively reduced.

[0118] In some specific embodiments, optimizing the labeling parameters by the reinforcement learning algorithm comprises: Defining the state space to include 8 dimensions of drug material, environmental temperature, humidity, historical parameters, surface roughness, label viscosity, surface reflectivity, and label material; Defining the action space to include 3 dimensions of labeling pressure increment, movement speed increment, and preheating temperature increment; Designing the reward function to include labeling quality score, labeling efficiency score, running stability score, and defect penalty term; Using the transfer deep deterministic policy gradient algorithm to perform incremental learning based on the pre-trained model.

[0119] By defining a multi-dimensional state space including drug material, environmental parameters, historical parameters, surface properties, and label properties, and an action space of labeling pressure, movement speed, and preheating temperature increment, and designing a hybrid reward function including quality, efficiency, stability, and penalty, and using the transfer deep deterministic policy gradient algorithm to optimize the labeling parameters, an intelligent parameter decision mechanism capable of self-learning and continuous improvement is formed.

[0120] In some specific embodiments, the specific calculation of the reward function comprises: The labeling quality score is calculated based on label offset and bubble number, 10 points for offset ≤1mm, 5 points for 1mm<offset≤2mm, minus 15 points for offset>2mm, 5 points for no bubbles, and minus 5 points for bubbles>3; The labeling efficiency score is calculated based on labeling time, 3 points for labeling time≤0.4s, 1 point for 0.4s<labelling time≤0.6s, and minus 2 points for labeling time>0.6s; The running stability score is calculated based on consecutive qualified times, 3 points for 3 consecutive qualified times, and 0 points otherwise; The defect penalty term includes 50 points for drug box damage and 20 points for label crease.

[0121] By setting specific and quantifiable calculation rules for the labeling quality score, labeling efficiency score, operational stability score, and defect penalty term in the reward function, a clear and explicit optimization goal and guide is provided for the reinforcement learning algorithm, effectively guiding the agent to learn a high-quality, high-efficiency, and stable labeling parameter strategy.

[0122] In some specific embodiments, controlling the robotic arm to perform the labeling operation includes: The joint motion commands are generated based on the optimal labeling trajectory, and the angles of each joint are precisely controlled by a PID controller. Real-time monitoring of joint torque and end effector position; immediate suspension of movement when abnormal vibration or positional deviation is detected. Record the actual parameters and results of each labeling session and update the historical labeling database for optimization of the reinforcement learning algorithm model.

[0123] By generating joint commands based on the optimal trajectory and using a PID controller for precise control, while simultaneously monitoring joint torque and end-effector position in real time and pausing immediately in case of abnormalities, and recording actual parameters to update the database, the robot arm's labeling action is executed accurately and safely, and operational data is continuously accumulated, providing data support for process traceability and further algorithm optimization.

[0124] Step S5: Perform labeling result quality inspection based on visible light images, and control the robotic arm to perform correction operations when defects are detected.

[0125] In some specific embodiments, the quality detection of labeling results based on visible light images specifically includes: Collect visible light images of the labeled medicine boxes, preprocess the images, and output standardized images; The standardized image is input into a rotation-invariant feature extraction network based on ResNet-50 to extract multi-angle features and output a feature vector. Calculate the cosine similarity between the feature vector and the standard template in the preset qualified feature library, and output the similarity score; The labeling quality level is determined based on the similarity score: when the similarity is ≥0.95, it is considered qualified; when the similarity is ≤0.95 and <0.95, it is considered a minor defect; when the similarity is <0.9, it is considered a serious defect. When a defect is detected, the corresponding robotic arm correction action is triggered according to the defect level.

[0126] By acquiring and preprocessing images after labeling, and then using a rotation-invariant feature extraction network to extract feature vectors and calculate similarity with a qualified feature library to determine the quality level and trigger corresponding corrections, an automated, high-precision online labeling quality detection and feedback correction process capable of identifying multiple defects has been formed, significantly improving the pass rate of the final product.

[0127] In some specific embodiments, controlling the robotic arm to perform corrective operations when a defect is detected includes: When a minor defect is identified, the robotic arm is controlled to move the labeling head in the opposite direction of the offset, with a translation accuracy of 0.1mm. At the same time, rotational correction is performed with a rotation accuracy of 0.1°. When a serious defect is identified, the negative pressure suction cup at the end of the control robot arm is used to adsorb the edge of the label with a suction force of 5N and peel off the label at a speed of 5mm / s. The labeling operation is re-performed on the stripped medicine by calling the labeling parameters optimized by reinforcement learning. After the correction is completed, a new visible light image is acquired for quality inspection. If the image is still not up to standard, an early warning message is generated and pushed to the user.

[0128] By controlling the robotic arm to perform differentiated correction operations such as translational rotation correction or label peeling and re-application with different precision according to the defect level (minor or severe), and re-inspecting after correction until qualified or alarm, a refined, efficient and thorough automated defect handling process has been formed, which minimizes human intervention and ensures the quality of products leaving the factory.

[0129] In some specific embodiments, extracting multi-angle features specifically includes: The standardized image was rotated by 0°, 90°, 180° and 310° respectively to generate four rotated versions of the image; The four rotated versions of the image are input into the ResNet-50 network and processed by the rotation consistent attention module in the conv3_4 layer. Calculate the rotation consistency weights of the four rotation angle feature maps to enhance the representation of rotation-invariant features; The weighted feature map is input into subsequent network layers, and a 2048-dimensional feature vector is obtained through global average pooling.

[0130] By performing multi-angle rotation processing on standardized images and extracting features using a network with an integrated rotation-consistent attention module, and then calculating rotation consistency weights to strengthen rotation-invariant features, the extracted feature vectors are highly insensitive to the placement angle of the medicines, thereby significantly improving the recognition accuracy and robustness of the quality inspection model under different orientations.

[0131] In some specific embodiments, the qualified feature library is constructed using the following methods: Collect 10,000 sets of sample labels, including qualified label images from multiple angles for each type of medicine box; The feature encoder is trained using Triplet Loss metric learning, ensuring that the intra-class distance is less than 0.3 and the inter-class distance is greater than 0.8. Cluster analysis was performed on the feature vectors of each type of medicine box to generate 10 cluster centers as qualified feature templates for that type of medicine box; Qualified feature templates are stored in a vector database, supporting real-time retrieval based on cosine similarity.

[0132] By collecting a large number of qualified samples and using Triplet Loss metric learning to train the feature encoder to ensure intra-class compactness and inter-class separation, and then generating cluster centers for each drug as qualified templates and storing them in a vector database, a high-quality, high-coverage, and fast-retrieval standardized qualified feature reference system was established for the quality inspection process.

[0133] In some specific embodiments, image preprocessing specifically includes: Gaussian filtering was applied to the acquired visible light images to remove noise, with the filter kernel size set to 3×3; Perform histogram equalization to enhance image contrast; Perform color correction and adjust the image color temperature using a white balance algorithm; The image is cropped to a fixed size, and the labeled area is retained as the detection area.

[0134] By performing Gaussian filtering for noise reduction, histogram equalization for contrast enhancement, white balance color correction, and fixed-size cropping on the acquired visible light images, interference factors such as noise, uneven illumination, and color cast in the images are effectively eliminated, and the area of ​​interest is unified to the labeling area, providing a standardized and high-quality input image for subsequent quality detection algorithms.

[0135] The following are embodiments of the intelligent drug labeling system based on depth perception and multimodal feature fusion provided in this application. This intelligent drug labeling system based on depth perception and multimodal feature fusion belongs to the same inventive concept as the intelligent drug labeling methods in the above embodiments. For details not described in detail in the embodiments of the intelligent drug labeling system, please refer to the embodiments of the intelligent drug labeling methods based on depth perception and multimodal feature fusion described above.

[0136] like Figure 2 As shown, the intelligent drug labeling system based on deep perception and multimodal feature fusion includes: The medicine box information acquisition module is used to acquire visible light images and three-dimensional depth information of the medicine box; The drug identity verification and comparison module is used to perform multimodal feature fusion of visible light images and three-dimensional depth information, verify drug identity based on the fusion results, extract drug identification information, and compare and verify it with the information of the label to be affixed. The labeling feasibility verification module is used to verify the drug's arrival status and labeling feasibility based on three-dimensional depth information. The labeling execution module is used to extract drug features based on visible light images and three-dimensional depth information, generate labeling trajectories and parameters based on drug features, and control the robotic arm to perform labeling operations. The labeling result quality inspection and correction module is used to inspect the labeling result quality based on visible light images and control the robotic arm to perform correction operations when defects are detected.

[0137] The intelligent drug labeling system in this embodiment is used to implement an intelligent drug labeling method based on depth perception and multimodal feature fusion.

[0138] This application also provides an electronic device for implementing the various embodiments of this application. Figure 3 To illustrate the hardware structure of an electronic device according to various embodiments of this application, as shown in the following diagram... Figure 3 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor.

[0139] Those skilled in the art will understand that the electronic device structure involved in the embodiments of this application does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0140] In embodiments of this application, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0141] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0142] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0143] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0144] This application also provides a storage medium storing a program product capable of implementing an intelligent drug labeling method based on depth perception and multimodal feature fusion. In some possible implementations, various aspects of this application can also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0145] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A smart drug labeling method based on depth perception and multimodal feature fusion, characterized in that, include: Acquire visible light images and three-dimensional depth information of the medicine box; Multimodal feature fusion is performed on visible light images and 3D depth information. Based on the fusion results, drug identity verification is performed, and drug identification information is extracted and compared with the information of the label to be affixed for verification. Verification of drug delivery status and labeling feasibility based on 3D depth information; Drug features are extracted based on visible light images and 3D depth information. Labeling trajectories and parameters are generated based on drug features, and the robotic arm is controlled to perform labeling operations. The labeling results are inspected based on visible light images, and a robotic arm is controlled to perform correction operations when defects are detected.

2. The intelligent drug labeling method as described in claim 1, characterized in that, Use an RGB camera to capture visible light images of the medicine box in real time; A depth camera was used to capture the 3D depth information of the medicine box.

3. The intelligent drug labeling method as described in claim 1, characterized in that, Multimodal feature fusion of visible light images and 3D depth information, and drug authentication based on the fusion results, include: The visible light image and 3D depth information are input into a pre-trained dual-branch deep learning model, which outputs a fused feature vector. The SURF feature point comparison algorithm is used to calculate the similarity between the fused feature vector and the standard drug feature template in the preset drug feature library. When the similarity score is greater than the preset similarity threshold, the drug identity verification is deemed successful. Two-branch deep learning models include: The super-resolution reconstruction branch is used to perform super-resolution reconstruction on visible light images, outputting high-resolution image features that enhance texture details; The target segmentation branch is used to process the visible light image and 3D depth information to output the segmentation features of the drug region; The feature fusion layer is used to fuse high-resolution image features and segmentation features to obtain a fused feature vector.

4. The intelligent drug labeling method as described in claim 1, characterized in that, Feasibility verification of drug delivery status and labeling based on 3D depth information includes: A sequence of depth information ordered in chronological order is generated based on continuously acquired 3D depth information. The depth information sequence is processed using a temporal difference algorithm to calculate the depth difference between adjacent frames. When the depth difference changes from being greater than a preset depth threshold to being less than the preset depth threshold, it is determined that the medicine box has reached the labeling area. Obtain the three-dimensional depth information of the medicine box in the labeling area; Based on the three-dimensional depth information of the medicine box in the labeling area, a feasibility score is calculated using a labeling feasibility assessment model, and the following judgments are performed: When the feasibility score is greater than or equal to the preset feasibility judgment threshold, the labeling feasibility verification is deemed to have passed. When the feasibility score is less than the preset feasibility judgment threshold, the labeling feasibility verification is deemed unsuccessful.

5. The intelligent drug labeling method as described in claim 4, characterized in that, Calculating the feasibility score using the labeling feasibility assessment model includes the following steps: An evaluation index system was established, which includes four primary indicators: positional accuracy, attitude stability, surface condition, and environmental adaptation. Each primary indicator contains two secondary indicators. Each secondary indicator is fuzzified, and the membership score of each indicator is calculated using a triangular membership function. The weight coefficients of each indicator are calculated using a random forest regression model; Introduce an environmental correction factor to dynamically adjust the weighting coefficients; The overall score is calculated using the weighted summation formula, which is as follows: in, For the first The adjusted weighting coefficients of each indicator; For the first The membership score of each indicator.

6. The intelligent drug labeling method as described in claim 2, characterized in that, Drug features are extracted based on visible light images and 3D depth information. Labeling trajectories and parameters are generated based on these features, and a robotic arm is controlled to perform the labeling operation. Specifically, this includes: Establish a dynamic mapping relationship between the robotic arm coordinate system, the depth camera coordinate system, and the RGB camera coordinate system; The size data and spatial coordinates of the medicine box are extracted based on visible light images and three-dimensional depth information. Based on the size data and spatial coordinates of the medicine box, the optimal labeling trajectory is generated by fusion of ant colony algorithm and 3-5-3 interval planning strategy. Based on historical labeling records and the material characteristics of medicine boxes, the labeling parameters, including the labeling pressure, movement speed, and label preheating temperature parameters of the robotic arm, are optimized through a reinforcement learning algorithm model. Based on the optimal labeling trajectory and optimized labeling parameters, the robotic arm is controlled to perform the labeling operation, and the movement status of the robotic arm is monitored in real time during the execution.

7. The intelligent drug labeling method as described in claim 1, characterized in that, The quality inspection of labeling results based on visible light images specifically includes: Collect visible light images of the labeled medicine boxes, preprocess the images, and output standardized images; The standardized image is input into a rotation-invariant feature extraction network based on ResNet-50 to extract multi-angle features and output a feature vector. Calculate the cosine similarity between the feature vector and the standard template in the preset qualified feature library, and output the similarity score; The labeling quality level is determined based on the similarity score: when the similarity is ≥0.95, it is considered qualified; when the similarity is ≤0.95 and <0.95, it is considered a minor defect; when the similarity is <0.9, it is considered a serious defect. When a defect is detected, the corresponding robotic arm correction action is triggered according to the defect level.

8. An intelligent drug labeling system based on deep perception and multimodal feature fusion, characterized in that, To implement the intelligent drug labeling method as described in any one of claims 1-7, the method includes: The medicine box information acquisition module is used to acquire visible light images and three-dimensional depth information of the medicine box; The drug identity verification and comparison module is used to perform multimodal feature fusion of visible light images and three-dimensional depth information, verify drug identity based on the fusion results, extract drug identification information, and compare and verify it with the information of the label to be affixed. The labeling feasibility verification module is used to verify the drug's arrival status and labeling feasibility based on three-dimensional depth information. The labeling execution module is used to extract drug features based on visible light images and three-dimensional depth information, generate labeling trajectories and parameters based on drug features, and control the robotic arm to perform labeling operations. The labeling result quality inspection and correction module is used to inspect the labeling result quality based on visible light images and control the robotic arm to perform correction operations when defects are detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the steps of the intelligent drug labeling method as described in any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent drug labeling method as described in any one of claims 1-7.

Images