Vision-based medicine bag sorting method and system, electronic equipment and storable medium

Abstract

The invention provides a medicine bag sorting method and system based on vision, electronic equipment and a storage medium, and belongs to the field of image recognition. The method comprises the steps that surface overall pictures of a plurality of to-be-sorted medicine bags are collected through a calibrated visual system; preprocessing the whole surface picture to obtain a preprocessed picture; detecting the preprocessed picture based on an ORB feature algorithm to obtain a corresponding feature vector; identifying the feature vectors by adopting a decoder in combination with a display point technology to obtain corresponding medicine bag texts; fitting the outer contour of the overall surface picture to obtain a polygonal contour; determining grabbing points of the to-be-sorted medicine bags based on the polygonal contour; and the multiple to-be-sorted medicine bags are sorted according to the grabbing points and the medicine bag texts. According to the application, the defects of low recognition efficiency, high error recognition rate, low sorting efficiency and the like of an existing automatic sorting system during sorting of small traditional Chinese medicine packages can be effectively overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data reading and recognition, and particularly relates to a vision-based medicine bag sorting method, system, electronic device, and storage medium. Background Art

[0002] Small-packaged traditional Chinese medicine decoction pieces have gradually replaced the original bulk traditional Chinese medicine decoction pieces. However, major medical institutions still use traditional manual methods for dispensing them. Pharmacists still repeat the processes of searching for medicines, counting medicines, and packing them among various medicine cabinets, which is not only time-consuming but also error-prone. Implementing the automatic sorting of small-packaged traditional Chinese medicine can improve the prescription dispensing speed and accuracy, reduce the labor cost in the pharmacy, shorten the waiting time for patients to pick up medicines, and accelerate the pace of automation and informatization construction in the traditional Chinese medicine pharmacy.

[0003] At present, although the emergence of an automatic sorting system can achieve the automatic grasping of small-packaged traditional Chinese medicine bags, thus releasing human labor and improving the work efficiency of medicine grabbing in the pharmacy. However, due to the diverse types and various shapes of small-packaged traditional Chinese medicine bags, as well as the characteristics of high variability and non-linearity of the text forms on the medicine bags, it often leads to low accuracy, poor stability in the sorting and grasping process of small-packaged medicine bags, and low accuracy in recognizing the text on the medicine bags during the sorting and grasping process. As a result, there are drawbacks such as low recognition efficiency, high misrecognition rate, and low sorting efficiency in the medicine grabbing process by the sorting manipulator.

[0004] Therefore, how to solve the drawbacks such as low recognition efficiency, high misrecognition rate, and low sorting efficiency existing in the current automatic sorting system for small-packaged traditional Chinese medicine bags is an urgent issue for those skilled in the art to solve. Summary of the Invention

[0005] To solve at least one of the above technical problems, the present invention provides a vision-based medicine bag sorting method, system, electronic device, and storage medium.

[0006] In a first aspect, the invention provides a vision-based medicine bag sorting method, including:

[0007] Collecting overall surface pictures of a plurality of medicine bags to be sorted through a calibrated vision system;

[0008] Preprocessing the overall surface pictures to obtain preprocessed pictures, where the preprocessing includes grayscale conversion, filtering, binarization, and morphological operations;

[0009] Detecting the preprocessed pictures based on the ORB feature algorithm to obtain corresponding feature vectors;

[0010] Using a decoder combined with a display point technique to recognize the feature vectors to obtain the corresponding medicine bag text;

[0011] Fit the outer contour of the overall surface image to obtain a polygonal contour;

[0012] Determine the grasping points of the medicine bags to be sorted based on the polygonal contour;

[0013] Sort a number of medicine bags to be sorted according to the grasping points and the medicine bag text.

[0014] Preferably, the step of collecting the overall surface images of a number of medicine bags to be sorted by the calibrated vision system specifically includes:

[0015] Calculate the internal and external parameter matrices and distortion coefficients of the shooting camera to calibrate the shooting camera;

[0016] Perform nine-point calibration according to the self-made calibration card to obtain the transformation matrix between the shooting camera and the sorting manipulator;

[0017] Correct the position coordinates of the sorting manipulator according to the transformation matrix;

[0018] Collect the overall surface images of a number of medicine bags to be sorted within the line of sight of the calibrated shooting camera.

[0019] Preferably, the step of preprocessing the overall surface image to obtain a preprocessed image specifically includes:

[0020] Grayscale the overall surface image by using the weighted average method;

[0021] Filter the grayscaled overall surface image by using the median filtering method;

[0022] Binarize the filtered overall surface image by using the Otsu method;

[0023] Obtain a preprocessed image by performing dilation and erosion processing on the binarized overall surface image.

[0024] Preferably, the step of detecting the preprocessed image based on the ORB feature algorithm to obtain the corresponding feature vector specifically includes:

[0025] Perform convolution on the preprocessed image through a Gaussian function to obtain a scale image, and extract the feature points of the scale image;

[0026] Calculate the centroid of the average gray level of the scale image to obtain the main direction of the feature points relative to the centroid;

[0027] Describe the feature points by using the RBRIEF descriptor;

[0028] Add rotation information to the RBRIEF descriptor by using the main direction to obtain the feature vector of the preprocessed image.

[0029] Preferably, the step of using a decoder combined with a display point technique to identify the feature vector to obtain the corresponding medicine bag text specifically includes:

[0030] Obtain a set of four control points of the feature vector to jointly define the corresponding Bézier curve;

[0031] Uniformly sample a series of points representing text from the Bézier curve;

[0032] Combine the position information of the points and the content information queried by the explicit points to form a composite query;

[0033] Input the composite query into the Transformer decoder, and aggregate multi-scale text features through the deformable cross-attention mechanism to obtain the medicine bag text.

[0034] Preferably, the step of fitting the outer contour of the overall surface picture to obtain a polygon contour specifically includes:

[0035] Classify the vector graph edges of the overall surface picture using the large law algorithm;

[0036] Use the 3σ law to delete discrete points with large deviations, and then fit them into a straight line using the least squares method;

[0037] Determine the positional relationship between two straight lines based on the slope of the straight line, and use the intersection point of the reconstructed line segments as the vertex of the vector graph;

[0038] Connect the vertices in sequence to form the polygon contour of the overall surface picture.

[0039] Preferably, the step of determining the grasping point of the medicine bag to be sorted based on the polygon contour specifically includes:

[0040] Segment the stacked medicine bags based on the principle that one medicine bag is segmented from every two concave folding points among the convex and concave folding points in the polygon contour;

[0041] Determine the target medicine bag that is most suitable for grasping among the current several medicine bags to be sorted according to the minimum overlapping area method;

[0042] Obtain the longest side in the contour fitting graph of the target medicine bag;

[0043] Based on the midpoint of the longest side, translate a preset distance along the direction perpendicular to the longest side to obtain the grasping point of the target medicine bag.

[0044] In a second aspect, a vision-based medicine bag sorting system includes:

[0045] A calibration module for collecting overall surface pictures of several medicine bags to be sorted through a calibrated vision system;

[0046] A preprocessing module for preprocessing the overall surface image to obtain a preprocessed image;

[0047] An extraction module for detecting the preprocessed image based on the ORB feature algorithm to obtain corresponding feature vectors;

[0048] An identification module for using a decoder combined with a display point technique to identify the feature vectors to obtain the corresponding medicine bag text;

[0049] A fitting module for fitting the outer contour of the overall surface image to obtain a polygon contour;

[0050] A fixed-point module for determining the grasping points of the medicine bags to be sorted based on the polygon contour;

[0051] A sorting module for sorting a number of medicine bags to be sorted according to the grasping points and the medicine bag text.

[0052] Preferably, the calibration module specifically includes:

[0053] A first calibration unit for calculating the internal and external parameter matrices and distortion coefficients of the shooting camera to calibrate the shooting camera;

[0054] A second calibration unit for performing nine-point calibration according to a self-made calibration card to obtain the transformation matrix between the shooting camera and the sorting manipulator;

[0055] A correction unit for correcting the position coordinates of the sorting manipulator according to the transformation matrix;

[0056] An acquisition unit for acquiring the overall surface images of a number of medicine bags to be sorted within its line of sight through the calibrated shooting camera.

[0057] Preferably, the preprocessing module specifically includes:

[0058] A grayscale unit for grayscale processing the overall surface image by using the weighted average method;

[0059] A filtering unit for filtering the grayscale processed overall surface image by using the median filtering method;

[0060] A binarization unit for binarizing the filtered overall surface image by using the Otsu method;

[0061] A morphological unit for obtaining a preprocessed image by performing dilation and erosion processing on the binarized overall surface image.

[0062] Preferably, the extraction module specifically includes:

[0063] A convolution unit, configured to perform convolution on the preprocessed image through a Gaussian function to obtain a scaled image and extract feature points of the scaled image;

[0064] A centroid finding unit, configured to calculate the centroid of the average gray level of the scaled image to obtain the principal direction of the feature points relative to the centroid;

[0065] A description unit, configured to describe the feature points using an RBRIEF descriptor;

[0066] An addition unit, configured to add rotation information to the RBRIEF descriptor using the principal direction to obtain a feature vector of the preprocessed image.

[0067] Preferably, the recognition module specifically includes:

[0068] A definition unit, configured to obtain a set of four control points of the feature vector to jointly define a corresponding Bezier curve;

[0069] A sampling unit, configured to uniformly sample a series of points representing text from the Bezier curve;

[0070] A query unit, configured to form a composite query by combining the position information of the points and the content information of the explicit point query;

[0071] An aggregation unit, configured to input the composite query into a Transformer decoder, and aggregate multi-scale text features through a deformable cross-attention mechanism to obtain the medicine bag text.

[0072] Preferably, the fitting module specifically includes:

[0073] A classification unit, configured to classify the vector graph edges of the overall surface image using a large law algorithm;

[0074] A fitting unit, configured to use the 3σ law to delete discrete points with large deviations, and then fit them into a straight line using the least squares method;

[0075] A vertex determination unit, configured to determine the positional relationship between two straight lines based on the slope of the straight line, and use the reconstructed intersection point of the line segment as the vertex of the vector graph;

[0076] A sequential connection unit, configured to sequentially connect the vertices to form a polygon contour of the overall surface image.

[0077] Preferably, the fixed-point module specifically includes:

[0078] A segmentation unit, configured to perform segmentation on the stacked medicine bags based on the principle of dividing one medicine bag for every two concave folding points among the convex and concave folding points in the polygon contour;

[0079] A screening unit for determining a target medicine bag most suitable for grasping among a number of current medicine bags to be sorted according to the minimum overlapping area method;

[0080] A searching unit for obtaining the longest side in the contour fitting diagram of the target medicine bag;

[0081] A translation unit for translating a preset distance along a direction perpendicular to the longest side based on the midpoint of the longest side to obtain a grasping point of the target medicine bag.

[0082] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the vision-based medicine bag sorting method described in the first aspect is implemented.

[0083] In a fourth aspect, the present application provides a storage medium, on which a computer program is stored. When the program is executed by a processor, the vision-based medicine bag sorting method described in the first aspect is implemented.

[0084] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0085] 1. The Zhang-Zhengyou calibration method is used to solve the error risks caused by unknown internal and external parameters of the shooting camera and distortion, and fully consider the difference in the positions of the medicine bag in the image coordinate system and the manipulator coordinate system. A nine-point calibration is completed through a self-made calibration card to obtain the transformation matrix between the two, so that the error between the two is within the allowable range, thereby improving the grasping accuracy of the manipulator.

[0086] 2. In order to facilitate the subsequent processing of the captured pictures, the captured pictures are processed through grayscale conversion, filtering, binarization, and morphological processing, which can reduce the noise in the image, restore and amplify the required information, and reduce the amount of information to be processed, thereby improving the subsequent processing efficiency of the pictures.

[0087] 3. The problem that the feature points do not have scale information is solved by convolving and solving the preprocessed pictures with a Gaussian function, and the problem that the feature points extracted by conventional algorithms do not have directionality is solved by combining with the ORB algorithm. The two are combined to quickly create a comprehensive and deep feature vector for the feature points in the image, providing a strong foundation for subsequent image analysis and recognition tasks.

[0088] 4. The decoder is combined with the display point technology for text recognition, which can not only accurately locate the text area, but also accurately identify the text content, and maintain a high consistency between the predicted shape and the true shape; and simplify the text detection and recognition process, improving the training efficiency and generalization ability of the recognition model.

[0089] 5. By combining the use of the large law algorithm, the 3σ law, and the least squares method, the deletion of redundant nodes and the reconstruction of key nodes can be achieved, enabling the fitting graph to closely conform to the outer contour of the captured image, improving the simulation degree of the polygon contour, and the sorting success rate of medicine bag sorting and grasping.

[0090] 6. Based on the polygon contour characteristics of the captured image and the shape characteristics of the medicine bag itself, and by means of the minimum overlapping area method, the medicine bag that is most suitable for grasping currently and its corresponding grasping points can be quickly determined, improving the efficiency of medicine bag sorting and grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0092] Figure 1 It is a flowchart of a vision-based medicine bag sorting method provided in Embodiment 1 of the present invention;

[0093] Figure 2 It is a structural block diagram of a vision-based medicine bag sorting system corresponding to the method in Embodiment 1 provided in Embodiment 2 of the present invention;

[0094] Figure 3 It is a schematic diagram of the hardware structure of a computer provided in Embodiment 3 of the present invention.

[0095] Description of the reference numerals:

[0096] 10 - Calibration module, 11 - First calibration unit, 12 - Second calibration unit, 13 - Correction unit, 64 - Acquisition unit.

[0097] 20 - Preprocessing module, 21 - Grayscale unit, 22 - Filtering unit, 23 - Thresholding unit, 24 - Morphological unit.

[0098] 30 - Extraction module, 31 - Convolution unit, 32 - Quality finding unit, 33 - Description unit, 34 - Adding unit.

[0099] 40 - Recognition module, 41 - Definition unit, 42 - Sampling unit, 43 - Query unit, 44 - Aggregation unit.

[0100] 50 - Fitting module, 51 - Classification unit, 52 - Fitting unit, 53 - Point determination unit, 54 - Sequential connection unit.

[0101] 60 - Fixed point module, 61 - Segmentation unit, 62 - Screening unit, 63 - Searching unit, 64 - Translation unit.

[0102] 70 - Sorting module.

[0103] 80 - Bus, 81 - Processor, 82 - Memory, 83 - Communication interface. Detailed implementation manners

[0104] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art.

[0105] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well - known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0106] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0107] The flowcharts shown in the drawings are only illustrative and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0108] As automated equipment is increasingly widely used in pharmacies. Currently, most automated medicine - picking systems are studied for boxed medicines with regular packaging, and the grasping of small - packaged traditional Chinese medicines still needs to be completed manually, which greatly affects the working efficiency of traditional Chinese medicine pharmacies. The automatic sorting system for small - packaged traditional Chinese medicine medicine bags designed in this application completes the automatic identification and grasping of small - packaged traditional Chinese medicine medicine bags, basically realizing the automatic sorting of small - packaged traditional Chinese medicine medicine bags, replacing manual medicine dispensing, and can effectively improve the working efficiency of pharmacies.

[0109] Embodiment 1

[0110] Specifically, Figure 1The figure shows a schematic flow chart of the vision-based medicine bag sorting method provided in this embodiment.

[0111] As Figure 1 shown, the vision-based medicine bag sorting method in this embodiment includes the following steps:

[0112] S101. Collect overall surface pictures of a number of medicine bags to be sorted through a calibrated vision system.

[0113] Specifically, the vision system for sorting and the corresponding detection and positioning algorithms are the core of the sorting equipment. Currently, the internal and external parameters of the shooting cameras equipped with the sorting equipment are unknown and there are distortions, etc. Therefore, to complete the sorting task, it is necessary to calibrate the shooting cameras first. In addition, since the positions of traditional Chinese medicine medicine bags are different in the image coordinate system and the coordinate system of the grasping manipulator, it is necessary to find the transformation matrix to complete the coordinate conversion.

[0114] Furthermore, the specific steps of step S101 include:

[0115] S1011. Calculate the internal and external parameter matrices and distortion coefficients of the shooting camera to calibrate the shooting camera.

[0116] Specifically, the camera is the core component in the vision system. The image sensor converts the optical signal into an electrical signal, and its performance is related to the quality of the camera imaging picture. The image sensor used in this embodiment is CMOS, and the specifically selected shooting camera is a Microvision industrial camera. In this step, the Zhang Zhengyou calibration method is used to calibrate the shooting camera. Its advantage is that the experimental equipment is easy to obtain, the experimental process has fewer constraints, the camera and the calibration board can be freely placed, and the relative error can be controlled within 0.3%. The Zhang Zhengyou calibration method in this embodiment specifically includes fixed plate image input, corner coordinate extraction, equation construction for parameter solving, least squares parameter estimation, maximum likelihood parameter optimization, distortion coefficient calculation, distortion correction, and output of the corrected image. Among them, in the solution of the distortion coefficient, the most important is the radial distortion, and the influence of the tangential distortion is very small. Therefore, generally only the radial distortion needs to be calculated using the expansion formula of the binary Taylor series. The specific mathematical expression is as follows:

[0117] υ = υ + (υ - υ0){k1(x 2 + y 2 ) + k2(x 2 + y 2 ) 2};

[0118]

[0119] In the formula, Let \((\upsilon, u)\) denote the actual pixel coordinates, \((\upsilon_0, u_0)\) denote the principal point, \((x, y)\) denote the ideal continuous image coordinates, and \(k_1\), \(k_2\) denote the distortion coefficients.

[0120] S1012. Perform nine-point calibration based on a self-made calibration card to obtain the transformation matrix between the shooting camera and the sorting manipulator.

[0121] Specifically, in the sorting system, an industrial robot is mainly used to grasp and place the target object. Since this application is for processing small packaging medicine bags as a specific target, the load performance requirements for the robotic arm are very low. Mainly, indicators such as speed, accuracy, and cost should be considered. Therefore, a synchronous belt linear module is used as the XY axis, and a vertical cylinder is used as the Z axis to form a three-axis degree-of-freedom manipulator. The nine-point calibration adopted in this implementation is completed on the Windows system. To conveniently obtain the calibration image and record the coordinate information of each calibration point, a corresponding MFC program is written. The specific calibration steps are as follows: First, use a white rectangular card and a black circular sticker to make a calibration card; Second, fix the camera and the three-axis degree-of-freedom manipulator of XYZ to ensure that the coordinate origin of the two does not change during the calibration process; Third, place the calibration card successively at 9 evenly distributed positions on the sorting platform, collect one image at each position, identify the center coordinates of the calibration card, then move the calibration needle above the center of the calibration card, read and record the coordinates in the robotic arm coordinate system at this time; Finally, calculate the 6 parameters of the transformation matrix through the nine groups of coordinates obtained in the third step.

[0122] S1013. Correct the position coordinates of the sorting manipulator according to the transformation matrix.

[0123] Specifically, to reduce errors, the manipulator is placed three times respectively, and it is ensured that each placement does not cause any offset to the calibration card. The average value of the three times is taken as the actual coordinate. Finally, the parameters of the transformation matrix are obtained as follows: \(a = -34.4183\), \(b = 0.856417\), \(c = 38529.6\), \(d = -21496.5\), \(e = -22625.1\), \(f = 306703\). The coordinates of the shooting camera are used to correct the position coordinates of the sorting manipulator through the transformation matrix with error-adjusted parameters, laying a calibration foundation for the subsequent grasping of small packaging medicine bags.

[0124] S1014. Collect overall surface pictures of several medicine bags to be sorted within the line of sight of the calibrated shooting camera.

[0125] Specifically, during the sorting process of small-packaged medicine bags, the overall image acquisition hardware system of the camera is in a sealed state and is only illuminated by the built-in light source. The light intensity can be adjusted in real time to ensure the quality of the captured images of small-packaged medicine bags, greatly reducing the interference of ambient light. The system adsorbs the small-packaged medicine bags onto the conveyor belt through negative pressure. The color of the conveyor belt is black, which can better avoid misjudgment of defects due to the similar colors of the conveyor belt and the small-packaged medicine bags.

[0126] S102. Preprocess the overall surface picture to obtain a preprocessed picture.

[0127] Specifically, the preprocessing of the picture taken of the small-packaged medicine bag is the basis for subsequent upper-layer operations. Its main functions are: reducing noise in the image; restoring and magnifying the required information; and reducing the amount of information to be processed. The image preprocessing process adopted in this application includes: grayscale conversion, filtering, binarization, and morphological operations.

[0128] Further, the specific steps of step S102 include:

[0129] S1021. Convert the overall surface picture to grayscale using the weighted average method.

[0130] Specifically, most cameras capture color images. The RGB image reflects the color characteristics of the target object, and the amount of information to be processed is three times that of a grayscale image. Therefore, in order to perform subsequent upper-layer operations such as efficient segmentation, detection, and recognition of the image, the color image needs to be converted to a black-and-white image. In this embodiment, the weighted average method is used to convert the image to grayscale, and the calculation formula is as follows:

[0131] Gray(i,j) = 0.229 * R(i,j) + 0.578 * G(i,j) + 0.114 * B(i,j);

[0132] In the formula, Gray(i,j) represents the grayscale value of each pixel after processing, and R(i,j), G(i,j), and B(i,j) respectively correspond to the values of the red, green, and blue channels in the original color image. The coefficients of the three are determined by the human eye's perception ability of each light. This method can better retain the morphological features.

[0133] S1022. Filter the grayscale-converted overall surface picture using the median filtering method.

[0134] Specifically, affected by the shooting environment (such as light, dust, etc.), or due to defects in the manufacturing process of the camera itself, or interference during the encoding, transmission, and decoding of the picture digital signal, the obtained image is mixed with noise to a certain extent. Therefore, the grayscale-converted picture is filtered. In this embodiment, the median filtering method is adopted, and its specific representation formula is as follows:

[0135] y ij =Med{x (i+r)(j+s) , (r, s) ∈ A, (i, j) ∈ I 2};

[0136] Among them, y ij represents the processed median value, A represents the filtering window, and (i, j) ∈ I 2 represents the points in the image. Comparing median filtering and Gaussian filtering, the results show that both filtering methods have good noise suppression effects, but median filtering better preserves the edge information of the image.

[0137] S1023. Perform binarization processing on the overall surface picture obtained by filtering using the Otsu method.

[0138] Specifically, after the grayscale image is binarized, the number of pixel levels suddenly decreases to 2, and the processing becomes simple. Threshold determination is the key to image binarization processing. In this embodiment, the Otsu method is used for binarization processing. It divides the grayscale image into two regions, the background and the target. The greater the variance between the two regions, the greater the difference. The pixel values of the grayscale image range from 0 to 255, a total of 255 levels. Count how many pixels there are at each level on the image and make a chart. The abscissa is the pixel level, and the ordinate is the number of pixels, then the pixel statistical histogram can be drawn; obtaining the histogram is a key step in target extraction of the grayscale image.

[0139] S1024. Obtain a preprocessed picture by performing dilation and erosion processing on the overall surface picture after binarization.

[0140] Specifically, in order to achieve an ideal effect in subsequent upper-layer operations, morphological operations are also required. In morphological operations, a structuring element is first set. This element has obvious processing effects on some information and not obvious on others. Therefore, it can perform operations on specific information to obtain an ideal output. The morphological processing in this embodiment specifically refers to dilation and erosion. Among them, dilation is similar to the convolution operation. Given an image A and a structuring element B, the dilation process is as follows: Use the anchor point of the structuring element B to scan each pixel point in the image A; calculate the maximum pixel value of the image A covered by the structuring element B; assign this value to the anchor point. It should be noted that the erosion operation is similar to the dilation operation, and the difference is only that the minimum pixel value is assigned to the anchor point.

[0141] S103. Detect the preprocessed picture based on the ORB feature algorithm to obtain the corresponding feature vector.

[0142] Specifically, when the ORB algorithm detects the overall surface image, the detected feature points are relatively concentrated and mostly scattered in the area of the small packaging medicine bags of traditional Chinese medicine. In terms of the algorithm operation efficiency, the ORB algorithm has an advantage in efficiency and a relatively fast matching speed. Therefore, considering comprehensively that the feature points detected by the ORB algorithm are more concentrated near the small packaging medicine bags of traditional Chinese medicine and the ORB algorithm has higher real-time performance, the detection algorithm of this embodiment adopts the ORB algorithm.

[0143] Further, the specific steps of step S103 include:

[0144] S1031, perform convolution on the preprocessed image through a Gaussian function to obtain a scale image, and extract the feature points of the scale image.

[0145] Specifically, the ORB algorithm decomposes the preprocessed image into 8 layers of different scale images, performs convolution on the image through the Gaussian function of the Gaussian kernel, and the preprocessed image passes through different Gaussian functions to obtain the scale image of each layer. The Gaussian convolution formula is as follows:

[0146] L(x, y, σ) = G(x, y, σ) ⊕ I(x, y);

[0147] In the formula, I(x, y) represents the original image, σ represents the scale factor, and the larger the value, the more blurred the image. The ORB algorithm constructs an image pyramid. After creating the image pyramid, it will use the FAST algorithm to quickly find feature points from images of different sizes at each level.

[0148] S1032, calculate the centroid of the average gray level of the scale image to obtain the main direction of the feature point relative to the centroid.

[0149] Specifically, aiming at the problem that the feature points extracted by the conventional algorithm do not have directionality, the ORB algorithm calculates the gray centroid of the image to obtain the main direction of the feature point relative to the centroid. First, calculate the intensity centroid of the box centered on the feature point, that is, the centroid position of the average gray level of the given image area, and its calculation formula is as follows:

[0150]

[0151] In the formula, C represents the centroid and m represents the moment of the image. Further, the main direction calculation formula is as follows:

[0152] θ = tan -1 (m 01 / m 10 ).

[0153] S1033, describe the feature points using the RBRIEF descriptor.

[0154] Specifically, RBRIEF is an image feature descriptor that represents and describes the detected feature points. Different from the traditional method of extracting features using the gray - level histogram or gradient histogram of the local neighborhood of the image, RBRIEF is a binary - coded feature description, which can not only reduce the storage space requirements, improve the speed of generating feature descriptors, but also reduce the time required for feature matching. Specifically, in this embodiment: First, assume that P is a feature point, and a square neighborhood of size S*S is selected centered on it for Gaussian smoothing. Second, N point pairs are generated using a random algorithm within the neighborhood, and a decision function T is defined. The formula of the function T is as follows: In the formula, p(x) and p(y) represent the gray - level values of x and y. Third, select a pair (x, y) from the randomly generated point pairs and compare the magnitudes of their gray - level values; repeat the above step N times to obtain an N - dimensional descriptor composed of binary codes.

[0155] S1034, use the main direction to add rotation information to the RBRIEF descriptor to obtain the feature vector of the pre - processed picture.

[0156] Specifically, when extracting the features of a picture, it is necessary to retain the position information of the features in the original image. In this embodiment, adding rotation information to the RBRIEF descriptor using the main direction realizes the feature of retaining position information, which can better understand the spatial relationship between text and image content. The sequence of feature vectors with position information is then fed into the Transformer encoder. In the encoder, the feature vectors are processed through the multi - head self - attention mechanism. Each "head" is responsible for paying attention to different subsets of the feature sequence, enabling the model to capture the multi - dimensional relationships between features in parallel and construct a comprehensive and deep feature representation.

[0157] S104, use a decoder combined with the explicit point technique to identify the feature vector to obtain its corresponding medicine bag text.

[0158] Specifically, in this embodiment, by combining the explicit point with the decoder, the accuracy of text recognition is effectively improved. First, the features of the text region are extracted through pre - determined processing operations, and then the Transformer decoder combined with the explicit point is used to decode the text content word by word. Introducing the explicit point can enhance the model's understanding of the text shape and layout, thereby improving the recognition performance in complex backgrounds. This method of use significantly improves the recognition accuracy and robustness while maintaining high efficiency.

[0159] Furthermore, the specific steps of step S104 include:

[0160] S1041, obtain a set of four control points of the feature vector to jointly define the corresponding Bezier curve.

[0161] Specifically, before defining the Bézier curve with the obtained feature vectors, the feature vectors are first flattened into one-dimensional feature vectors through a flattening operation. A Bézier curve is defined by a set of control points, and these control points determine the shape of the curve; therefore, for an nth-degree Bézier curve, n + 1 control points are required. Specifically, four Bézier control points are given to the top and bottom of each text instance respectively, and then the Bézier control points of the central curve are calculated by taking the average of the corresponding control points at the top and bottom. The Bézier curve accurately depicts the outline of the text, which not only reflects the position of the text but also its shape, including any possible bends or tilts, and can adapt to various deformations of the text, such as bending or twisting, which is difficult to achieve in the traditional rectangular bounding box method.

[0162] S1042, Uniformly sample a series of points representing the text from the Bézier curve.

[0163] Specifically, for each text instance in the image, after giving four Bézier control points to its top and bottom respectively and then calculating the average of the corresponding control points at the top and bottom, before starting the query, based on the existing Bézier central curve, the model uniformly samples N points, and uses the Bernstein polynomial to obtain the normalized coordinates of the points. Based on these coordinates, a position query for the points is generated, and its specific formula is Pq = MLP(PE(Coords)), where PE represents the sine position encoding function.

[0164] S1043, Combine the position information of the points and the content information of the explicit point query to form a composite query.

[0165] Specifically, the introduction of explicit points enhances the model's understanding of the text shape and layout, and can improve the recognition performance in complex backgrounds. The explicit point query models the text instance, realizes the effective integration of scene text detection and recognition, and learns a learnable content query through the model. After determining the Bézier central curve, the model uniformly samples a certain number of points on the curve. The coordinates of these points are then encoded into position queries and combined with the learnable content queries to form a composite query for the Transformer decoder. These queries are expected to encode sufficient text semantic and position information to provide a basis for subsequent text recognition and detection.

[0166] S1044, Input the composite query into the Transformer decoder, and aggregate multi-scale text features through the deformable cross-attention mechanism to obtain the medicine bag text.

[0167] Specifically, the composite query is fed into the Transformer decoder, which updates the query by considering the relationships between text instances and the relationships with the text features output by the encoder. The decoder output performs multi-task prediction based on the updated query, including text instance classification, character classification, and prediction of the positions of the center line and boundary points. In addition, considering that traditional methods often encounter greater challenges when dealing with text having complex shapes and layouts, a classification loss function, a text recognition loss function, a shape loss function, and a decoder loss function are proposed. By comprehensively considering the weight factors to adjust the proportion of each loss in the total loss, in the form of a weighted sum, the total loss function can comprehensively consider multiple aspects of text detection and recognition, guiding the model to optimize these aspects simultaneously during the training process. Such a design enables the model to not only accurately locate the text region but also accurately identify the text content and maintain a high degree of consistency between the predicted shape and the true shape.

[0168] S105. Fit the outer contour of the overall surface picture to obtain a polygonal contour.

[0169] Specifically, aiming at problems such as key nodes being smoothly expressed and node redundancy existing in the vectorized picture outer contour, the present application proposes a contour fitting algorithm. This fitting algorithm first classifies the vector graph edges using the Otsu algorithm, deletes discrete points with large deviations using the 3σ law, then fits them into straight lines using the least squares method, then determines the positional relationship between two straight lines using the slope of the fitted straight lines, and finally takes the intersection points of the reconstructed line segments as the vertices of the vector graph, and connects the vertices in sequence to finally complete the contour fitting.

[0170] Further, the specific steps of step S105 include:

[0171] S1051. Classify the vector graph edges of the overall surface picture using the Otsu algorithm.

[0172] Specifically, adaptive binarization can automatically adjust the threshold according to the local features of the image, thus better adapting to local changes in the image. This local adaptability makes adaptive binarization perform excellently when processing images with different lighting conditions, contrasts, and textures. By adaptively selecting the threshold, adaptive binarization can enhance the contrast of the image, making the details in the image more clearly visible. In adaptive binarization, the Otsu method is generally used to solve for the threshold. The implementation steps of the Otsu algorithm in this embodiment mainly include: First, calculate the histogram of the image, that is, count the number of pixel points falling in each of the 256 bins from 0 to 255 for all pixel points in the image; Second, normalize the histogram, that is, divide the number of pixel points in each bin by the total number of pixel points; Third, traverse each possible threshold from 0 to 255 and calculate the between-class variance of the foreground and background; Finally, select the threshold that maximizes the between-class variance as the optimal threshold. Based on the obtained optimal threshold, the vector map edges of the overall surface image are divided into two categories, namely the main direction and the secondary direction perpendicular to the main direction.

[0173] S1052, Use the 3σ rule to delete discrete points with large deviations, and then fit them into a straight line using the least squares method.

[0174] Specifically, the 3σ rule first assumes that a set of detection data only contains random errors, calculates and processes it to obtain the standard deviation, determines an interval with a certain probability, and believes that any error exceeding this interval does not belong to random error but gross error, and the data containing this error should be excluded. The fitting of a straight line using the least squares method in this embodiment is directly implemented through code. For example, the sample code is as follows:

[0175] % Given data

[0176] x = [0.1; 0.3; 0.4; 0.75; 0.9];

[0177] y = [1.7805; 2.2285; 2.3941; 3.2226; 3.5697];

[0178] % Calculate k and b

[0179] N = length(x);

[0180] k = (sum(y.*x) - N * mean(y) * mean(x)) / (sum(x.^2) - N * mean(x)^2);

[0181] b = mean(y) - k * mean(x);

[0182] % Plot the fitted straight line

[0183] x_line = linspace(min(x), max(x), 100);

[0184] y_line = k * x_line + b;

[0185] plot(x, y, 'o', x_line, y_line, '-r');

[0186] legend('Original data', 'Fitted line');

[0187] xlabel('x');

[0188] ylabel('y');

[0189] title('Least Squares Method for Fitting a Line');

[0190] S1053. Determine the positional relationship between two lines based on the slope of the line, and use the reconstructed intersection point of the line segment as the vertex of the vector graph.

[0191] Specifically, generally, the positional relationship between two lines is determined based on the slope of the line. Currently, there are three cases for determination. First, when it is known that the slopes of the two lines must exist: two lines are parallel if and only if the slopes of the two lines are equal and the intercepts on the corresponding coordinate axes are not equal; two lines are perpendicular if and only if the product of the slopes of the two lines is -1. Second, the case where the slopes of the two lines may not exist: if the slopes of both lines do not exist, when the intercepts of the two lines on the x-axis are not equal, the two lines are parallel; when the intercepts of the two lines on the x-axis are equal, the two lines coincide; if the slopes of the two lines exist, the same as the first case, if the slope of one of the two lines does not exist and the slope of the other line is 0, then the two lines are perpendicular. Third, when the general equations of the two lines are known: directly apply the conclusion for determination; or the slope can be obtained from the line equation (or it can be determined that the slope does not exist), and it is transformed into the first or second case for determination.

[0192] S1054. Connect the vertices in sequence to form the polygonal contour of the overall surface picture.

[0193] Specifically, the polygonal contour is formed by sequentially connecting the vertices to trace the contour boundary. For the determined vertex results, the vertices of the polygonal contour are recorded starting from 0 in the order of 0, 1,.... Using the above algorithm, the contour of the medicine bag picture can be obtained. It should be noted that in the case of stacked medicine bags, in order to reduce the probability that the suction head falls into the overlapping seam between the medicine bags, a method of grasping from the outside to the inside is adopted, so the inner contour can be discarded.

[0194] S106. Determine the grasping points of the medicine bags to be sorted based on the polygonal contour.

[0195] Specifically, the small-package medicine bag of this embodiment has wrinkled areas at the top, bottom and middle positions. If the gripping point is positioned in the above areas, the success rate of suction will be greatly reduced. In addition to avoiding the wrinkled area and the middle seam, the gripping point should also be avoided in the middle area of the medicine bag as much as possible, because the closer the gripping point is to the edge, the greater the difference in the gravity of the small-package medicine bag at each position, resulting in a greater possibility of the medicine bag falling during the movement of the gripping manipulator. Therefore, determining the gripping point of the medicine bag to be sorted is particularly important for the stability of sorting.

[0196] Furthermore, the specific steps of step S106 include:

[0197] S1061, segmenting the stacked medicine bags based on the principle that one medicine bag is segmented for every two concave inflection points of the convex and concave inflection points in the polygonal contour.

[0198] Specifically, after contour extraction and fitting operations, a single bag of medicine is generally rectangular, convex quadrilateral, or convex pentagonal due to folded corners. The final result of processing a single medicine bag image is a convex polygon, and when medicine bags are stacked, the angle where the two medicine bags overlap will be a concave angle. Therefore, the angle where the concave inflection point is located must be the intersection of the two medicine bags, that is, every two concave inflection points are a segmented medicine bag. Based on this, the stacked medicine bags can be segmented to achieve the segmentation of medicine bags in the stacked state. Determining which medicine bag is most suitable for grabbing at the moment is the key to improving the success rate of grabbing.

[0199] S1062, determining the most suitable target medicine bag to be grasped among the current plurality of medicine bags to be sorted according to the minimum overlapping area method.

[0200] Specifically, when medicine bags are stacked, the grasping point should try to avoid the gap at the junction of the stacked medicine bags to ensure successful grasping. Therefore, the medicine bag with the smallest overlapping area should be selected for grasping. Specifically, the medicine bag with the largest side length is regarded as the one with the smallest overlapping area with other medicine bags.

[0201] S1063, obtaining the longest side in the target medicine bag contour fitting graph.

[0202] Specifically, after the current grabbed medicine bag is determined, it is only necessary to obtain the length and width of the medicine bag from the contour fitting diagram of the medicine bag, restore the medicine bag to a rectangle, and then locate the grabbing point to the center of the suitable grabbing area according to the grabbing position of the medicine bag determined in the previous step.

[0203] S1064, obtaining a grasping point of the target medicine bag by translating the midpoint of the longest side by a preset distance in a direction perpendicular to the longest side.

[0204] Specifically, based on the shape characteristics of the small-packaged medicine bags, the landing points of the grasping points need to avoid the wrinkled areas, the middle seams, and the middle areas of the medicine bags. Therefore, when determining the landing points of the grasping points, it is not necessarily the middle position, but the grasping points should be preset according to the actual situation of the small-packaged medicine bags to be grasped and sorted. Therefore, the translation pre-distance is not fixed.

[0205] S107. Sort a number of medicine bags to be sorted according to the grasping points and the medicine bag text.

[0206] Specifically, the above steps are combined to calculate the specific positions of the grasping points and accurately identify the text on the surface of the small-packaged medicine bags. Through the external three-axis manipulator, the small-packaged medicine bags to be sorted can be accurately grasped and sorted.

[0207] To sum up: First, the nine-point calibration is completed through the Zhang-Zhengyou calibration method and the self-made calibration card, so that the error between the camera and the grasping manipulator is within the allowable range; and the captured images are processed by grayscale conversion, filtering, binarization, and morphology to facilitate the subsequent processing of the captured images. Second, the Gaussian function and the ORB algorithm are combined to quickly create comprehensive and deep feature vectors for the feature points in the image. Third, the decoder and the display point technology are used for text recognition, which can simplify the process of text detection and recognition. Third, the use of the great law algorithm, the 3σ law, and the least square method makes the fitting graph closely fit the outer contour of the captured image. Finally, based on the polygon contour characteristics of the captured image and the shape characteristics of the medicine bags themselves, and by means of the minimum overlapping area method, the medicine bag that is most suitable for grasping currently and its corresponding grasping points can be quickly determined. Through the above steps, the disadvantages of the existing automatic sorting system in sorting small-packaged traditional Chinese medicine bags, such as low recognition efficiency, high misrecognition rate, and low sorting efficiency, are solved.

[0208] Embodiment 2

[0209] This embodiment provides a structural block diagram of a system corresponding to the method described in Embodiment 1. Figure 2 It is a structural block diagram of a vision-based medicine bag sorting system according to this embodiment, as Figure 2 shown. The system includes:

[0210] A calibration module 10 for collecting overall surface pictures of a number of medicine bags to be sorted through a calibrated vision system;

[0211] A preprocessing module 20 for preprocessing the overall surface pictures to obtain preprocessed pictures;

[0212] An extraction module 30 for detecting the preprocessed pictures based on the ORB feature algorithm to obtain corresponding feature vectors;

[0213] An identification module 40, configured to identify the corresponding medicine bag text of the feature vector by using a decoder in combination with a display point technique;

[0214] A fitting module 50, configured to fit the outer contour of the overall surface picture to obtain a polygon contour;

[0215] A fixed-point module 60, configured to determine the grasping points of the medicine bags to be sorted based on the polygon contour;

[0216] A sorting module 70, configured to sort a plurality of medicine bags to be sorted according to the grasping points and the medicine bag text.

[0217] Further, the calibration module 10 specifically includes:

[0218] A first calibration unit 11, configured to calculate the internal and external parameter matrices and distortion coefficients of the shooting camera to calibrate the shooting camera;

[0219] A second calibration unit 12, configured to perform nine-point calibration according to a self-made calibration card to obtain the transformation matrix between the shooting camera and the sorting manipulator;

[0220] A correction unit 13, configured to correct the position coordinates of the sorting manipulator according to the transformation matrix;

[0221] An acquisition unit 14, configured to acquire the overall surface pictures of a plurality of medicine bags to be sorted within its line of sight through the calibrated shooting camera.

[0222] Further, the preprocessing module 20 specifically includes:

[0223] A grayscale unit 21, configured to perform grayscale processing on the overall surface picture by using a weighted average method;

[0224] A filtering unit 22, configured to perform filtering processing on the grayscale processed overall surface picture by using a median filtering method;

[0225] A binarization unit 23, configured to perform binarization processing on the filtered overall surface picture by using the Otsu method;

[0226] A morphological unit 24, configured to perform dilation and erosion processing on the binarized overall surface picture to obtain a preprocessed picture.

[0227] Further, the extraction module 30 specifically includes:

[0228] A convolution unit 31, configured to perform convolution solution on the preprocessed picture through a Gaussian function to obtain a scale picture, and extract the feature points of the scale picture;

[0229] The centroid finding unit 32 is used to calculate the centroid of the average grayscale of the scale picture, and obtain the main direction of the feature point relative to the centroid;

[0230] The description unit 33 is used to describe the feature point by using the RBRIEF descriptor;

[0231] The adding unit 34 is used to add rotation information to the RBRIEF descriptor by using the main direction to obtain the feature vector of the preprocessed picture.

[0232] Further, the recognition module 40 specifically includes:

[0233] The definition unit 41 is used to obtain a set of four control points of the feature vector to jointly define the corresponding Bezier curve;

[0234] The sampling unit 42 is used to uniformly sample a series of points representing the text from the Bezier curve;

[0235] The query unit 43 is used to form a composite query by combining the position information of the points and the content information of the explicit point query;

[0236] The aggregation unit 44 is used to input the composite query into the Transformer decoder, and aggregate multi-scale text features through the deformable cross-attention mechanism to obtain the medicine bag text.

[0237] Further, the fitting module 50 specifically includes:

[0238] The classification unit 51 is used to classify the vector graph edges of the overall surface picture by using the large law algorithm;

[0239] The fitting unit 52 is used to delete the discrete points with large deviations by using the 3σ law, and then fit them into a straight line by using the least square method;

[0240] The vertex determination unit 53 is used to determine the positional relationship between two straight lines based on the slope of the straight line, and reconstruct the intersection point of the line segment as the vertex of the vector graph;

[0241] The sequential connection unit 54 is used to sequentially connect the vertices to form the polygon contour of the overall surface picture.

[0242] Further, the fixed-point module 60 specifically includes:

[0243] The segmentation unit 61 is used to segment the stacked medicine bags based on the principle of dividing one medicine bag for every two concave folding points among the convex and concave folding points in the polygon contour;

[0244] The screening unit 62 is used to determine the target medicine bag most suitable for grasping among the current several medicine bags to be sorted according to the minimum overlapping area method;

[0245] A searching unit 63 for obtaining the longest side in the contour fitting diagram of the target medicine bag;

[0246] A translation unit 64 for translating a preset distance along a direction perpendicular to the longest side based on the midpoint of the longest side to obtain the grasping point of the target medicine bag.

[0247] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combined form.

[0248] Embodiment 3

[0249] Combined with Figure 1 The vision-based medicine bag sorting method described above can be implemented by a computer. Figure 3 It is a schematic diagram of the hardware structure of a computer according to this embodiment.

[0250] The computer may include a processor 80 and a memory 82 storing computer program instructions.

[0251] Specifically, the above-mentioned processor 80 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present application.

[0252] Among them, the memory 82 may include a mass storage for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 82 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 82 may be internal or external to the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). In a suitable case, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. In a suitable case, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0253] The memory 82 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 80.

[0254] The processor 80 reads and executes the computer program instructions stored in the memory 82 to implement the vision-based medicine bag sorting method of Embodiment 1 above.

[0255] In some of these embodiments, the computer may further include a communication interface 83 and a bus 80. Among them, as Figure 3 shown, the processor 80, the memory 82, and the communication interface 83 are connected through the bus 80 to complete communication with each other.

[0256] The communication interface 83 is used to implement communication between the various modules, devices, units, and / or devices in this application. The communication interface 83 can also implement data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0257] Bus 80 includes hardware, software, or both, and couples components of a computer to each other. Bus 80 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, Local Bus. By way of example and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable bus or a combination of two or more of these. In a suitable case, Bus 80 may include one or more buses. Although the present application describes and shows specific buses, the present application contemplates any suitable bus or interconnect.

[0258] The computer can obtain a vision-based medicine bag sorting system and execute the vision-based medicine bag sorting method of Embodiment 1.

[0259] In addition, in combination with the vision-based medicine bag sorting method in Embodiment 1 above, the present application can be implemented by providing a storage medium. Computer program instructions are stored on the storage medium; when the computer program instructions are executed by a processor, the vision-based medicine bag sorting method of Embodiment 1 above is implemented.

[0260] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0261] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. Claims A vision-based medicine bag sorting method, characterized in that, Including: Collecting overall surface images of several medicine bags to be sorted through a calibrated vision system; Preprocessing the overall surface image to obtain a preprocessed image, where the preprocessing includes grayscale conversion, filtering, binarization, and morphological operations; Detecting the preprocessed image based on the ORB feature algorithm to obtain corresponding feature vectors; Using a decoder combined with a display point technique to identify the feature vectors to obtain the corresponding medicine bag text; Fitting the outer contour of the overall surface image to obtain a polygon contour; Determining the grasping points of the medicine bags to be sorted based on the polygon contour; Sorting several medicine bags to be sorted according to the grasping points and the medicine bag text.

2. The vision-based medicine bag sorting method according to claim 1, wherein The step of collecting overall surface images of several medicine bags to be sorted through a calibrated vision system specifically includes: Calculating the internal and external parameter matrices and distortion coefficients of the shooting camera to calibrate the shooting camera; Performing nine-point calibration according to a self-made calibration card to obtain the transformation matrix between the shooting camera and the sorting manipulator; Correcting the position coordinates of the sorting manipulator according to the transformation matrix; Collecting overall surface images of several medicine bags to be sorted within the line of sight through the calibrated shooting camera.

3. The vision-based medicine bag sorting method according to claim 1, characterized in that The step of preprocessing the overall surface image to obtain a preprocessed image specifically includes: Performing grayscale conversion on the overall surface image using the weighted average method; Performing filtering processing on the grayscale-converted overall surface image using the median filtering method; Performing binarization processing on the filtered overall surface image using the Otsu method; Performing dilation and erosion processing on the binarized overall surface image to obtain a preprocessed image.

4. The vision-based medicine bag sorting method according to claim 1, wherein The step of detecting the preprocessed image based on the ORB feature algorithm to obtain corresponding feature vectors specifically includes: Performing convolution on the preprocessed image through a Gaussian function to obtain a scale image, and extracting the feature points of the scale image; Calculating the centroid of the average grayscale of the scale image to obtain the main direction of the feature points relative to the centroid; Describing the feature points using the RBRIEF descriptor; Adding rotation information to the RBRIEF descriptor using the main direction to obtain the feature vectors of the preprocessed image.

5. The vision-based medicine bag sorting method according to claim 1, wherein The step of using a decoder combined with a display point technique to identify the feature vectors to obtain the corresponding medicine bag text specifically includes: Obtaining a set of four control points of the feature vectors to jointly define a corresponding Bezier curve; Uniformly sampling from the Bezier curve to obtain a series of points representing the text; Combining the position information of the points and the content information queried by the explicit points to form a composite query; Inputting the composite query into a Transformer decoder, and aggregating multi-scale text features through a deformable cross-attention mechanism to obtain the medicine bag text.

6. The vision-based medicine bag sorting method according to claim 1, wherein The step of fitting the outer contour of the overall surface image to obtain a polygon contour specifically includes: Classifying the vector map edges of the overall surface image using the Otsu algorithm; Deleting discrete points with large deviations using the 3σ law, and then fitting them into a straight line using the least squares method; Determining the positional relationship between two straight lines based on the slope of the straight lines, and using the reconstructed intersection points of the line segments as the vertices of the vector map; Sequentially connecting the vertices to form the polygon contour of the overall surface image.

7. The vision-based medicine bag sorting method according to claim 1, characterized in that, The steps of determining the grasping points of the medicine bags to be sorted based on the polygon contour specifically include: Dividing the stacked medicine bags according to the principle of dividing one medicine bag for every two concave folding points among the convex and concave folding points of the polygon contour; Determining the target medicine bag most suitable for grasping among the current several medicine bags to be sorted according to the minimum overlapping area method; Obtaining the longest side in the contour fitting diagram of the target medicine bag; Obtaining the grasping points of the target medicine bag by translating a preset distance along the direction perpendicular to the longest side based on the midpoint of the longest side.

8. A vision-based medicine bag sorting system, characterized in that Including: A calibration module for collecting overall surface pictures of several medicine bags to be sorted through a calibrated vision system; A preprocessing module for preprocessing the overall surface pictures to obtain preprocessed pictures; An extraction module for detecting the preprocessed pictures based on the ORB feature algorithm to obtain corresponding feature vectors; An identification module for identifying the feature vectors by using a decoder in combination with the display point technology to obtain the corresponding medicine bag texts; A fitting module for fitting the outer contour of the overall surface pictures to obtain a polygon contour; A fixed-point module for determining the grasping points of the medicine bags to be sorted based on the polygon contour; A sorting module for sorting several medicine bags to be sorted according to the grasping points and the medicine bag texts.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the vision-based medicine bag sorting method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the vision-based medicine bag sorting method according to any one of claims 1 to 7.

Images