Method and device for determining medicine feeding parameters of flat groove dispensing machine

Abstract

The invention discloses a method and a device for determining medicine feeding parameters of a flat groove medicine dispensing machine, and the method comprises the steps: obtaining a 2D image and a point cloud picture of each medicine box through a 3D camera, and obtaining a high-resolution 2D image of each medicine box through a 2D camera; grouping all the medicine boxes based on the 2D image corresponding to each medicine box; matching the high-resolution 2D image of each medicine box in the medicine box set with the point cloud image to obtain a matched point cloud image of each medicine box; based on the matched point cloud atlas corresponding to each medicine box in the medicine box set, first positioning information under the camera coordinates is obtained, and second positioning information under the conveyor belt coordinates is obtained in combination with a pre-established coordinate system conversion matrix; calculating positioning rectangular frame information corresponding to the medicine box set based on the second positioning information of each medicine box in the medicine box set; and feature information is extracted based on the high-resolution 2D image of each medicine box in the medicine box set to obtain the category of each medicine box. The medicine feeding parameters can be automatically determined, and the intelligent level and the operation reliability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical device manufacturing, and in particular to a method and device for determining medicine feeding parameters of a flat slot medicine dispensing machine. Background Art

[0002] As a type of automated equipment in hospital pharmacies, the flat-trough medicine dispensing machine is used and functions mainly to improve the efficiency, accuracy and safety of medicine dispensing.

[0003] Flat-slot dispensers typically utilize multi-layered tracks or storage troughs, where medications are stored by size and type. Operators neatly place medications in designated locations through the dispenser port. The device automatically associates the medication information with the storage trough by scanning barcodes or using an identification system. A robotic arm or pusher quickly delivers medications to the trough, reducing manual operation time. Dispensed medications are then transported via conveyor belts or slides to intelligent medicine baskets or dispenser windows, significantly improving pharmacy efficiency. Summary of the Invention

[0004] In order to automatically and effectively determine the medicine feeding parameters, and thereby effectively improve the intelligence level and operation reliability of the flat-trough medicine dispensing machine, the present invention provides a method and device for determining the medicine feeding parameters of the flat-trough medicine dispensing machine.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining medicine feeding parameters of a flat slot medicine dispenser, comprising:

[0006] The 3D camera acquires a 2D image and point cloud of each medicine box on the conveyor belt, and the 2D camera acquires a high-resolution 2D image of each medicine box on the conveyor belt;

[0007] grouping all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets;

[0008] For each set of medicine boxes, the high-resolution 2D image of each medicine box in the set is matched with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the internal reference information of the 2D camera, to obtain the matched point cloud image of each medicine box.

[0009] For each set of medicine boxes, the first positioning information in camera coordinates is obtained based on the matched point cloud image corresponding to each medicine box in the set, and the second positioning information in conveyor coordinates is obtained by combining the pre-established coordinate system transformation matrix;

[0010] For each medicine box set, based on the second positioning information of each medicine box in the medicine box set, calculate the positioning rectangular frame information corresponding to the medicine box set to obtain the medicine feeding position, wherein the medicine feeding position includes the center point coordinates, width, and distance from adjacent medicine box sets corresponding to the medicine box set;

[0011] For each medicine box set, feature information is extracted based on a high-resolution 2D image of each medicine box in the medicine box set, and the feature information is matched with a pre-established feature library to obtain the category of each medicine box.

[0012] Optionally, for each medicine box set, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera and the internal reference information of the 2D camera, the high-resolution 2D image of each medicine box in the medicine box set is matched with the point cloud map to obtain a matched point cloud map of each medicine box, including:

[0013] For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the internal reference information of the 2D camera, and the point cloud image corresponding to each medicine box in the medicine box set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotated rectangular frame of each medicine box is obtained;

[0014] For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the medicine box set, the second pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the second pixel rotated rectangular frame of each medicine box is obtained;

[0015] For each set of medicine boxes, the second pixel rotated rectangular frame corresponding to each medicine box in the medicine box set is matched one by one with the first pixel rotated rectangular frames of all the medicine boxes in the medicine box set to obtain the first pixel rotated rectangular frame that matches the second pixel rotated rectangular frame corresponding to each medicine box, and obtain the matched point cloud image of each medicine box.

[0016] Optionally, for each medicine box set, obtaining the first pixel coordinates of the four endpoints and the center point of each medicine box based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the internal reference information of the 2D camera, and the point cloud image corresponding to each medicine box in the medicine box set, and obtaining the first pixel rotated rectangular frame includes:

[0017] For each set of medicine boxes, determine the coordinates of the center point and the four endpoints in the 3D camera coordinate system based on the point cloud image corresponding to each medicine box in the medicine box set. Combined with the rotation matrix and translation vector between the 3D camera and the 2D camera, the coordinates of the center point and the four endpoints of each medicine box in the 2D camera coordinate system are obtained;

[0018] Based on the internal reference information of the 2D camera, the center point coordinates of each medicine box in the 2D camera coordinate system and the coordinates of the four endpoints, the first pixel coordinates of the center point of each medicine box and the first pixel coordinates of the four endpoints are obtained, and the first pixel rotated rectangular frame of each medicine box is obtained.

[0019] Optionally, for each medicine box set, the second pixel-rotated rectangular frame corresponding to each medicine box in the medicine box set is matched one by one with the first pixel-rotated rectangular frames of all medicine boxes in the medicine box set to obtain a first pixel-rotated rectangular frame that matches the second pixel-rotated rectangular frame corresponding to each medicine box, and obtain a matched point cloud image of each medicine box, including:

[0020] For each medicine box in each medicine box set, the intersection-and-union ratio between the second pixel rotated rectangular box corresponding to each medicine box and the first pixel rotated rectangular boxes of all medicine boxes is calculated using the following formula:

[0021]

[0022] in, Represents the i-th second pixel rotated rectangle; Indicates the jth first pixel rotated rectangular box; ROI ij represents the intersection-over-union ratio between the i-th second pixel rotated rectangle and the j-th first pixel rotated rectangle;

[0023] Based on all interaction ratios, the first pixel-rotated rectangular box corresponding to the largest intersection-and-union ratio is obtained as the first pixel-rotated rectangular box that matches the second pixel-rotated rectangular box corresponding to each medicine box, and the point cloud image corresponding to the matched first pixel-rotated rectangular box is used as the matched point cloud image of each medicine box.

[0024] Optionally, grouping all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets includes:

[0025] Based on the 2D image corresponding to each medicine box, a pixel rectangular envelope of each medicine box is obtained, and parameters of each pixel rectangular envelope are obtained, wherein the parameters include a horizontal coordinate of an upper left corner, a width, a height, and a horizontal coordinate of a lower right corner;

[0026] Sort all the pixel rectangular envelopes according to the order of the upper left corner horizontal coordinate from small to large to obtain a pixel rectangular envelope sequence;

[0027] Traversing all pixel rectangular envelopes in the pixel rectangular envelope sequence, and determining whether the horizontal coordinate of the lower right corner of any pixel rectangular envelope is greater than or equal to the horizontal coordinate of the upper left corner of the next pixel rectangular envelope;

[0028] If so, the two pixel rectangular envelopes are merged into the same medicine box set, and the range of the merged rectangular frame is updated;

[0029] If not, the two pixel rectangular envelopes cannot be merged into the same medicine box set, and the judgment process is re-executed for the next pixel rectangular envelope.

[0030] Optionally, for each medicine box set, extracting feature information based on a high-resolution 2D image of each medicine box in the medicine box set, matching the feature information with a pre-established feature library to obtain the category of each medicine box includes:

[0031] For each set of medicine boxes, based on the high-resolution 2D image of each medicine box in the set, a pre-set visual feature extraction algorithm is used to extract the feature information of each medicine box;

[0032] For the feature information of each medicine box, the cosine similarity with each template in the pre-established feature library is calculated, and the template corresponding to the maximum cosine similarity is screened out, and the category corresponding to the template is used as the category of each medicine box.

[0033] Optionally, the method for constructing the coordinate system conversion matrix includes:

[0034] Use a 3D camera to obtain a point cloud image of the conveyor belt when no medicine boxes are placed on the conveyor belt;

[0035] Based on the conveyor belt point cloud image, obtaining the normal vector of the conveyor belt and the coordinates of any reference point;

[0036] Acquire a first point cloud image of the medicine box on the conveyor belt when it is at a first reference position using a 3D camera, and determine a first coordinate point projected onto the conveyor belt plane by the center point of the medicine box at the first reference position using the conveyor belt normal vector as a projection direction, based on the normal vector of the conveyor belt and the coordinates of any reference point;

[0037] Acquire a second point cloud image of the medicine box on the conveyor belt when it is at a second reference position using a 3D camera, and determine a second coordinate point on the conveyor belt plane where the center point of the medicine box at the second reference position is projected using the conveyor belt normal vector as a projection direction, by combining the normal vector of the conveyor belt and the coordinates of any reference point;

[0038] Obtaining a conveyor belt motion direction vector based on the first coordinate point and the second coordinate point;

[0039] Obtaining a conveyor belt transverse direction vector based on the conveyor belt motion direction vector and the conveyor belt normal vector;

[0040] Based on the first coordinate point, determining a distance from the first coordinate point to a predetermined reference point, and determining the coordinates of the reference point in a 3D camera coordinate system in combination with the conveyor belt motion direction vector;

[0041] Based on the lateral direction vector of the conveyor belt, the movement direction vector of the conveyor belt and the normal vector of the conveyor belt, the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system is obtained, and the coordinate system conversion matrix is ​​obtained by combining the coordinates of the reference point in the 3D camera coordinate system.

[0042] Optionally, the step of acquiring a 2D image and a point cloud image of each medicine box on the conveyor belt by a 3D camera, and acquiring a high-resolution 2D image of each medicine box on the conveyor belt by a 2D camera, includes:

[0043] Performing image segmentation on the original 2D images of all medicine boxes on the conveyor belt acquired by the 3D camera to obtain a first segmentation mask corresponding to each medicine box, and obtaining a 2D image and a point cloud map corresponding to each medicine box based on the first segmentation mask;

[0044] Image segmentation is performed based on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask corresponding to each medicine box, and a high-resolution 2D image corresponding to each medicine box is obtained based on the second segmentation mask.

[0045] In a second aspect, an embodiment of the present invention provides a device for determining medicine feeding parameters of a flat slot medicine dispenser, comprising:

[0046] An image collection module is used to obtain a 2D image and point cloud image of each medicine box on the conveyor belt through a 3D camera, and to obtain a high-resolution 2D image of each medicine box on the conveyor belt through a 2D camera;

[0047] a grouping module, configured to group all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets;

[0048] A data matching module is used to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the internal reference information of the 2D camera, for each medicine box set, to obtain a matched point cloud image of each medicine box;

[0049] A positioning information confirmation module is used to obtain, for each set of medicine boxes, first positioning information in camera coordinates based on the matched point cloud image corresponding to each medicine box in the set, and to obtain second positioning information in conveyor coordinates by combining a pre-established coordinate system conversion matrix;

[0050] a medicine feeding position confirmation module, configured to calculate, for each medicine box set, the positioning rectangular frame information corresponding to the medicine box set based on the second positioning information of each medicine box in the medicine box set, and obtain the medicine feeding position, wherein the medicine feeding position includes the coordinates of the center point corresponding to the medicine box set, the width, and the distance from the adjacent medicine box set;

[0051] The category recognition module is used to extract feature information from each medicine box set based on the high-resolution 2D image of each medicine box in the medicine box set, match the feature information with a pre-established feature library, and obtain the category of each medicine box.

[0052] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the medicine feeding parameters of the flat-slot medicine dispenser as described in the first aspect.

[0053] In a fourth aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining the medicine feeding parameters of the flat-slot medicine dispenser as described in the first aspect is implemented.

[0054] In a fifth aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the method for determining the medicine feeding parameters of the flat slot medicine dispenser as described in the first aspect.

[0055] The beneficial effects of the above technical solutions provided in the embodiments of the present invention include at least:

[0056] In an embodiment of the present invention, a method for determining the drug feeding parameters of a flat-slot medicine dispenser is provided. A 3D camera is used to obtain a 2D image and point cloud map of each medicine box on a conveyor belt, and a 2D camera is used to obtain a high-resolution 2D image of each medicine box on the conveyor belt. Drug feeding parameters may include drug feeding location and medicine box category. By grouping the medicine boxes and calculating the group positioning parameters (i.e., second positioning information), the drug feeding location of the flat-slot medicine dispenser's feed port can be determined. This can be used as a lateral positioning reference for the robotic arm and can adapt to the random placement of different medicines. Based on the high-resolution 2D image of each medicine box in each group, high-resolution feature information of the medicine box can be obtained, thereby enabling the classification of the medicine box to be identified, facilitating medicine sorting and verification, and avoiding the limited ability of traditional manual verification or single visual recognition methods to distinguish complex textures and similar packaging. By fusing image data from 3D cameras and 2D cameras, full process automation is achieved, from drug feeding location and medicine box identification to grasping operations, effectively improving the intelligence level and operational reliability of the flat-slot medicine dispenser.

[0057] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 This is a schematic structural diagram of a flat slot medicine dispensing machine provided in an embodiment of the present invention;

[0061] Figure 2 This is a flow chart of a method for determining medicine feeding parameters of a flat-slot medicine dispenser provided in an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a starting position calibrated during the construction of a coordinate system conversion matrix provided in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a first reference position calibrated during the construction of a coordinate system conversion matrix provided in an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of a second reference position calibrated during the construction of a coordinate system conversion matrix provided in an embodiment of the present invention;

[0065] Figure 6 Schematic diagram of each medicine kit set provided in an embodiment of the present invention;

[0066] Figure 7 This is a schematic diagram of a device for determining medicine feeding parameters of a flat-grooved medicine dispenser provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," "far," "near," "front," and "back" and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0069] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0070] The inventors discovered that in the prior art, the positioning of medicines in flat-slot medicine dispensers mainly relies on fixed card slots or mechanical limit devices, which cannot dynamically adapt to the random placement of different medicines; the identification of drug categories mostly uses manual verification or single visual recognition methods, and has limited ability to distinguish complex textures and similar packaging. Existing visual recognition systems have functional fragmentation problems in obtaining drug information: positioning technology based on three-dimensional point clouds has difficulty in synchronously obtaining high-resolution surface texture features, while two-dimensional imaging systems lack depth information, resulting in insufficient dimensional measurement accuracy. In addition, barcode recognition modules usually require separate configuration of dedicated scanning equipment, which makes the system structure complex and increases maintenance costs.

[0071] Therefore, in order to solve the above problems, the inventors have proposed a method and device for determining the medicine feeding parameters of a flat-trough medicine dispensing machine through research and development. Through multimodal data fusion technology, the fully automatic and high-precision recognition of the medicine feeding position and the category of the medicine box is realized to accurately obtain the medicine feeding parameters, which can effectively improve the intelligence level and operational reliability of the flat-trough medicine dispensing machine.

[0072] First, some nouns or terms that appear in the description of the embodiments of this application are subject to the following interpretations:

[0073] Segmentation mask: is an image representation method used to mark each pixel in the image as belonging to a specific category or target. It is a two-dimensional matrix with the same size as the original image, and each element in the matrix corresponds to a pixel in the original image. For example, for a color image with a resolution of 640×480, its segmentation mask is also a 640×480 matrix. The value of each element represents the category label to which the pixel belongs, such as 0 for background and 1 for medicine box. The segmentation mask can accurately define the position and shape of the medicine box in the image, allowing the system to accurately identify the medicine box and provide a clear target area for subsequent processing.

[0074] Point Cloud: A representation of three-dimensional spatial data consisting of a large number of discrete points, each of which records its position in space (typically including X, Y, and Z coordinates) and may also include other attributes (such as color, reflection intensity, and normal vector). Point Clouds are a core data type in 3D reconstruction, computer vision, and robotic perception, used to describe the geometry and surface features of objects or scenes.

[0075] A pixel bounding box (PixelBoundingBox) is a fundamental data structure used in computer vision and image processing to describe the location and extent of a target object in a two-dimensional image. It completely encloses the pixel region of the target object in the image using a rectangular box and records key parameters of the rectangle (such as coordinates and dimensions), providing location information for subsequent tasks such as object detection, recognition, segmentation, or tracking.

[0076] The intrinsic parameters of a 2D camera are parameters that describe the camera's internal optical and geometric properties and are used to project points in the 3D camera coordinate system onto the 2D pixel coordinate system. This intrinsic parameter information is the core output of camera calibration and directly determines the image model and geometric distortion correction. Multiple sets of images are captured using a calibration plate (such as a checkerboard), and the intrinsic parameter information is calculated using an optimization algorithm (such as the Zhang Zhengyou calibration method). This intrinsic parameter information can include focal length, principal point, distortion coefficient, pixel size, and more.

[0077] Focal length: The optical focal length of the camera lens. The horizontal focal length can be expressed as f x , the vertical focal length can be expressed as f y .

[0078] Principal point: The coordinates of the intersection of the camera optical axis and the image plane, in pixels. The horizontal principal point coordinates can be expressed as c x , the vertical principal point coordinates can be expressed as c y .

[0079] Example 1

[0080] This embodiment proposes a method for determining the feeding parameters of a flat slot medicine dispensing machine, which can be applied to the flat slot medicine dispensing machine. Figure 1The flat-trough medicine dispensing machine mainly includes a 3D camera, a 2D camera and a conveyor belt. The medicine feeding parameters may include the medicine feeding position, the size of the medicine box, the number of medicine boxes and the category of the medicine box. The 3D camera can locate the position of the medicine box and measure the size of the medicine box. The 2D camera can obtain high-resolution feature information of the medicine box, thereby identifying the category of the medicine box. If there is a QR code or barcode on the medicine box, the barcode or QR code can also be scanned. By segmenting the relevant images of the 2D camera or the 3D camera, the quantity information of the relevant feeding medicines can also be obtained. By fusing and analyzing the images obtained by the 2D camera and the 3D camera, the medicine feeding position and category of the medicine box can be automatically and effectively identified. It can also be used to identify the number and size of the medicine boxes to accurately obtain the medicine feeding parameters and effectively improve the intelligence level of the flat-trough medicine dispensing machine.

[0081] See Figure 2 The method for determining the medicine feeding parameters of the flat slot medicine dispenser can specifically include the following steps:

[0082] Step S1: Acquire a 2D image and a point cloud image of each medicine box on the conveyor belt through a 3D camera, and acquire a high-resolution 2D image of each medicine box on the conveyor belt through a 2D camera;

[0083] Step S2: grouping all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple groups of medicine box sets;

[0084] Step S3: For each set of medicine boxes, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera and the internal reference information of the 2D camera, the high-resolution 2D image of each medicine box in the medicine box set is matched with the point cloud image to obtain a matched point cloud image of each medicine box;

[0085] Step S4: for each set of medicine boxes, based on the matched point cloud corresponding to each medicine box in the medicine box set, obtain first positioning information in camera coordinates, and combine it with the pre-established coordinate system conversion matrix to obtain second positioning information in conveyor belt coordinates;

[0086] Step S5: For each medicine box set, based on the second positioning information of each medicine box in the medicine box set, calculate the positioning rectangular frame information corresponding to the medicine box set to obtain the medicine inlet position, wherein the medicine inlet position includes the center point coordinates, width, and distance from adjacent medicine box sets corresponding to the medicine box set;

[0087] Step S6: For each medicine box set, extract feature information based on the high-resolution 2D image of each medicine box in the medicine box set, match the feature information with a pre-established feature library, and obtain the category of each medicine box.

[0088] In order to explain more clearly the method for determining the medicine feeding parameters of the above-mentioned flat-trough medicine dispenser, each step will be described in detail below.

[0089] In the above step S1, the specific process of acquiring a 2D image and a point cloud image of each medicine box on the conveyor belt by a 3D camera, and acquiring a high-resolution 2D image of each medicine box on the conveyor belt by a 2D camera may include the following steps:

[0090] Step S101: performing image segmentation based on the original 2D images of all medicine boxes on the conveyor belt acquired by the 3D camera to obtain a first segmentation mask corresponding to each medicine box, and obtaining a 2D image and point cloud map corresponding to each medicine box based on the first segmentation mask.

[0091] In the above step S101, the 3D camera can be installed directly above the conveyor belt, with the field of view covering the entire medicine feeding area, and the 3D camera (such as an RGB-D camera, a lidar or a structured light camera) is used to obtain the original 2D image and the original point cloud map of the medicine box on the conveyor belt for three-dimensional spatial positioning and size measurement. After the 3D camera is installed, it is necessary to align the point cloud map of the 3D camera to output the aligned 2D image and 3D information matrix. The so-called alignment means that the length and width of the 2D image are equal to the length and width of the 3D information matrix. The 3D information matrix will store the depth information or 3D coordinate information of each pixel point in the 2D image in sequence according to the coordinates of each pixel point. Therefore, after the original 2D image is segmented to obtain the first segmentation mask of the target, the 3D information matrix can be filtered with the first segmentation mask to obtain the 3D information of the desired target, that is, the point cloud map.

[0092] On a conveyor belt, if two medicine boxes of the same height are placed close together, it would be difficult to distinguish them using point cloud segmentation methods. However, a 3D camera that has aligned the original point cloud images using the above method can segment each medicine box. The 3D camera can obtain the original 2D image and the original point cloud image that corresponds pixel by pixel to the original 2D image.

[0093] Before performing image segmentation on the original 2D image, the original 2D image can be denoised (such as Gaussian filtering), normalized (mapping the depth value to a fixed range) and background separated (removing the conveyor belt background by the threshold method) to improve image quality, reduce noise interference, remove irrelevant background, and simplify the segmentation task. After completing the above preprocessing, the original 2D image can be segmented using a deep learning algorithm (such as the YOLO series, Mask R-CNN, SOLO and other instance segmentation algorithms) or a traditional method (such as template matching), and the first segmentation mask for each medicine box can be output. According to the first segmentation mask, the 2D image corresponding to each medicine box is cropped out from the original 2D image. Since the original 2D image output by the 3D camera corresponds to the original point cloud map pixel by pixel, the point cloud map corresponding to each medicine box can be extracted through the first segmentation mask.

[0094] Step S102: performing image segmentation based on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask corresponding to each medicine box, and obtaining a high-resolution 2D image corresponding to each medicine box based on the second segmentation mask.

[0095] In the above step S102, a 2D camera (such as an industrial CCD camera) shoots the conveyor belt from directly above or obliquely above to obtain an original high-resolution 2D image, which contains characteristic information such as the color, texture, text, and two-dimensional outline of the medicine box. Before the original high-resolution 2D image is segmented, the original high-resolution 2D image can be denoised (median filtering), contrast enhanced (histogram equalization), and distortion corrected (eliminating lens distortion) to improve image quality, enhance target features, and highlight segmentation boundaries. After completing the above preprocessing, a deep learning algorithm (such as the YOLO series, Mask R-CNN, SOLO, and other instance segmentation algorithms) or a traditional method (such as template matching) can be used to segment the original high-resolution 2D image to obtain a second segmentation mask corresponding to each medicine box. According to the second segmentation mask, a high-resolution 2D image of each medicine box is cropped from the original high-resolution 2D image, retaining the characteristic information of the medicine box for subsequent identification of the medicine box category.

[0096] In step S2 above, all medicine boxes are grouped based on the 2D image corresponding to each medicine box. By analyzing the spatial relationship of the rectangular envelope of the pixels of the medicine boxes, medicine boxes that are physically adjacent or overlapping are merged into the same group, thereby optimizing subsequent processing (such as robotic arm grasping or batch sorting). Figure 1 The telescopic direction of the mechanical arm of the flat slot dispenser is the transverse direction of the conveyor belt, so the incoming medicine boxes are grouped in the direction of the column on the conveyor belt. Specifically, the process of obtaining multiple groups of medicine boxes may include the following steps:

[0097] Step S201: Based on the 2D image corresponding to each medicine box, obtain the pixel rectangular envelope of each medicine box and obtain the parameters of each pixel rectangular envelope, including the upper left corner horizontal coordinate, width, height and lower right corner horizontal coordinate.

[0098] In the above step S201, a target detection model (such as YOLO, Faster R-CNN) or a traditional method (such as contour detection + minimum bounding rectangle) can be used to detect the 2D image of each medicine box, generate a pixel rectangular envelope, and obtain the parameters of each pixel rectangular envelope, which include the upper left corner horizontal coordinate, width, height and lower right corner horizontal coordinate. Of course, the parameters can also include the upper left corner vertical coordinate. In this embodiment, since the pixel rectangular envelope is not a rotated rectangular frame, the upper left corner horizontal coordinate can also represent the horizontal coordinate of the left side, and the upper left corner vertical coordinate can also represent the vertical coordinate of the upper side.

[0099] The parameter list corresponding to each medicine box can be expressed as the following formula (1):

[0100] rect i :x i ,y i ,w i ,h i ,1≤i≤N (1)

[0101] In the above formula (1), rect i represents the i-th pixel rectangular envelope of the segmentation; w i and h i Respectively represent the width and height of the pixel rectangular envelope; x i and y i Represents the upper left corner horizontal coordinate and the upper left corner vertical coordinate respectively.

[0102] The horizontal coordinate x' of the lower right corner i It can be expressed as the following formula (2):

[0103] x' i =x i +w i (2)

[0104] The vertical coordinate y' of the lower right corner i It can be expressed as the following formula (3):

[0105] y' i =y i +h i (3)

[0106] Step S202 : Sort all pixel rectangular envelopes in ascending order of the upper left corner horizontal coordinate to obtain a pixel rectangular envelope sequence.

[0107] In the above step S202, by sorting all pixel rectangular envelopes, it is possible to ensure that adjacent medicine boxes are arranged continuously in the sequence, which is convenient for subsequent merging and judgment.

[0108] Step S203, traverse all pixel rectangular envelopes in the pixel rectangular envelope sequence, and determine whether the horizontal coordinate of the lower right corner of any pixel rectangular envelope is greater than or equal to the horizontal coordinate of the upper left corner of the next pixel rectangular envelope; if so, execute step S204; if not, execute step S205;

[0109] In step S203, all pixel rectangular envelopes in the pixel rectangular envelope sequence are traversed, and the intersection of the two preceding and succeeding pixel rectangular envelopes is determined. The specific process of traversal and merging can be: starting from the i-th pixel rectangular envelope in the sequence, it is sequentially compared with the i+1-th pixel rectangular envelope. The grouping rule can be expressed by the following formula (4):

[0110]

[0111] If the current pixel rectangle envelope and the next pixel rectangle envelope intersect in the x-axis direction, that is, x i +w i ≥x i+1 , then the pixel rectangular envelope and the next pixel rectangular envelope have an intersection and can be grouped together, and the two are then assigned to the current pixel rectangular envelope (i.e., the merged rectangular frame range is updated), and the following step S204 is executed; if the current pixel rectangular envelope and the next pixel rectangular envelope do not intersect in the x-axis direction, that is, the two pixel rectangular envelopes are not in the same group, the current pixel rectangular envelope is treated as an independent group, and the judgment of the next pixel rectangular envelope is continued, i=i+1, 1≤i≤N-1, that is, the following step S205 is executed. After completing the grouping of all the medicine boxes, the index value of each group of medicine box sets can be set. The index value includes the positioning information of each medicine box in the current group, which facilitates the positioning and identification of the medicine boxes.

[0112] Step S204: merge the two pixel rectangular envelopes into the same medicine box set, and update the range of the merged rectangular frame.

[0113] Step S205: The two pixel rectangular envelopes cannot be combined into the same medicine box set, and the judgment process is re-executed for the next pixel rectangular envelope.

[0114] In this embodiment, by grouping all pill boxes and merging physically adjacent or overlapping pill boxes into the same group, efficient physical grouping of pill boxes is achieved. This allows for guidance of a robotic arm to grasp multiple adjacent pill boxes at once, improving efficiency and providing critical spatial layout information for automated pharmacies or industrial sorting systems. Furthermore, this system facilitates batch counting, counting the number of pill boxes in each group and reducing repetitive operations.

[0115] In the above step S3, when the 3D camera and the 2D camera are installed, the 3D camera and the 2D camera need to be aligned. By aligning the original 2D image output by the 3D camera and the original high-resolution 2D image output by the 2D camera, the positional relationship between the two cameras can be solved, thereby obtaining the rotation matrix and translation vector between the 3D camera and the 2D camera. For each set of medicine boxes, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera and the internal reference information of the 2D camera, the high-resolution 2D image of each medicine box in the medicine box set is matched with the point cloud map. The specific process of obtaining the matched point cloud map of each medicine box may include the following steps:

[0116] Step S301: For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the internal reference information of the 2D camera, and the point cloud image corresponding to each medicine box in the medicine box set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotated rectangular frame of each medicine box is obtained.

[0117] In the above step S301, for each medicine box set, the specific process of obtaining the first pixel rotated rectangular frame of each medicine box may include the following steps:

[0118] Step S3011: For each set of medicine boxes, determine the coordinates of the center point and the coordinates of the four endpoints in the 3D camera coordinate system based on the point cloud image corresponding to each medicine box in the medicine box set, and combine the rotation matrix and translation vector between the 3D camera and the 2D camera to obtain the coordinates of the center point and the coordinates of the four endpoints in the 2D camera coordinate system of each medicine box.

[0119] In the above step S3011, for each medicine box, the pixel coordinates of its center point in the high-resolution 2D image are (q x ,q y By performing radius filtering, downsampling, plane extraction and other operations on the point cloud of each medicine box, the coordinates of the center point of each medicine box in the 3D camera coordinate system, the normal vector of the medicine box plane, the size of the medicine box (length, width, height), the coordinates of the four endpoints of the medicine box and the pixel rectangular envelope of the medicine box can be output. The coordinates of the center point of each medicine box in the 3D camera coordinate system are marked as p 3d =(x 3d ,y 3d ,z 3d ) t Combining the rotation matrix R' and translation vector t' between the 3D camera and the 2D camera, the coordinates p of the center point of each medicine box in the 2D camera coordinate system are 2d It can be expressed as the following formula (5):

[0120] p 2d =R'p 3d +t' (5)

[0121] In the above formula (5), the coordinates of the center point of each medicine box in the 2D camera coordinate system are: p 2d =(x 2d ,y 2d ,z 2d ) t .

[0122] Step S3012: Based on the internal reference information of the 2D camera, the center point coordinates of each medicine box in the 2D camera coordinate system, and the coordinates of the four endpoints, obtain the first pixel coordinates of the center point of each medicine box and the first pixel coordinates of the four endpoints, and obtain the first pixel rotated rectangular frame of each medicine box.

[0123] In step S3012 above, the first pixel coordinates of the center point of each medicine box are used as an example. Based on the intrinsic reference information of the 2D camera (focal length and principal point), combined with the center point coordinates of each medicine box in the 2D camera coordinate system, the pixel coordinates of the center point of each medicine box in the 2D camera image are obtained, i.e., the first pixel coordinates. The first pixel coordinates can be expressed as follows (6):

[0124]

[0125] In the above formula (6), f x is the horizontal focal length; f y is the vertical focal length; c x is the horizontal coordinate of the principal point; c y is the coordinate of the principal point in the vertical direction; (p x ,p y ) is the coordinate p of the center point of the medicine box in the 3D camera coordinate system 3d Convert to the first pixel coordinate of the 2D camera image.

[0126] Similarly, the four endpoints of the medicine box can also be transformed using the above method to obtain the first pixel coordinates of the four endpoints. Combined with the first pixel coordinates of the center point of each medicine box, the first pixel rotation rectangle rotRect of each medicine box is obtained. 3d .

[0127] Step S302: For each medicine box set, based on the high-resolution 2D image corresponding to each medicine box in the medicine box set, obtain the second pixel coordinates of the four endpoints and the center point of each medicine box, and obtain the second pixel rotated rectangle rotRect of each medicine box. 2d .

[0128] Step S303: For each group of medicine box sets, the second pixel rotated rectangular frame corresponding to each medicine box in the medicine box set is matched one by one with the first pixel rotated rectangular frames of all the medicine boxes in the medicine box set to obtain a first pixel rotated rectangular frame that matches the second pixel rotated rectangular frame corresponding to each medicine box, and obtain a matched point cloud image of each medicine box.

[0129] In the above step S303, for each set of medicine boxes, the specific process of obtaining the matched point cloud image of each medicine box may include the following steps:

[0130] Step S3031: For each medicine box in each medicine box set, the intersection-and-union ratio between the second pixel-rotated rectangular frame corresponding to each medicine box and the first pixel-rotated rectangular frames of all medicine boxes is calculated using the following formula (7):

[0131]

[0132] In the above formula (7), Represents the i-th second pixel rotated rectangle; Indicates the jth first pixel rotated rectangular box; ROI ij It represents the intersection-over-union ratio between the i-th second-pixel rotated rectangle and the j-th first-pixel rotated rectangle.

[0133] Therefore, for the i-th second pixel rotated rectangular box corresponding to the 2D camera, M different ROI values ​​can be obtained by calculating the M first pixel rotated rectangular boxes corresponding to the 3D camera.

[0134] Step S3032: Based on all interaction ratios, obtain the first pixel rotated rectangular box corresponding to the largest intersection-and-union ratio as the first pixel rotated rectangular box that matches the second pixel rotated rectangular box corresponding to each medicine box, and use the point cloud image corresponding to the matched first pixel rotated rectangular box as the matched point cloud image of each medicine box.

[0135] In step S3032, the largest ROI value among the M ROI values ​​is taken to obtain the first pixel rotated rectangular frame corresponding to the i-th first pixel rotated rectangular frame, thereby matching the second pixel rotated rectangular frame of each medicine box with the corresponding first pixel rotated rectangular frame. The point cloud corresponding to the matched first pixel rotated rectangular frame serves as the matched point cloud for each medicine box. Thus, an association can be established between the 3D camera and the 2D camera, achieving a precise match between the 2D image of the medicine box and the point cloud. This allows the positioning information of each medicine box from the 3D camera to be subsequently matched with the identification information and code scanning information of each medicine box from the 2D camera, providing multimodal perception capabilities for the automation system of the flat-slot medicine dispenser.

[0136] In the above step S4, refer to Figure 3-Figure 5, is a schematic diagram of conveyor belt calibration during the construction of the coordinate system transformation matrix, where Figure 3 For the calibration starting position, the leftmost side of the conveyor belt is aligned with the left side of the medicine box, and the center point of the left side of the conveyor belt is used as the reference point O. Figure 4 is the first reference position for calibration. At this time, the moving distance of the conveyor belt relative to the starting position is scale1. Figure 5 is the second reference position for calibration. At this time, the conveyor belt moves a distance of scale2 relative to the starting position. Obviously, scale2>scale1. Specifically, the method for constructing the coordinate system transformation matrix may include the following steps:

[0137] Step S401, refer to Figure 3 ,The 3D camera is used to obtain the point cloud image of the conveyor belt when no medicine box is set on the conveyor belt.

[0138] Step S402: Based on the conveyor belt point cloud image, obtain the normal vector of the conveyor belt and the coordinates of any reference point.

[0139] In the above step S402, point cloud preprocessing and plane extraction operations are performed on the conveyor belt point cloud image to obtain the normal vector of the conveyor belt and the coordinates of any reference point, and the plane equation of the conveyor belt shown in the following formula (8) can be established:

[0140] n x (xp x )+n y (yp y )+n z (zp z )=0 (8)

[0141] In the above formula (8), n=(n x ,n y ,n z ) t is the normal vector of the conveyor belt, p v =(p x ,p y ,p z ) t is the coordinate of any reference point on the conveyor belt.

[0142] In this embodiment, since the 3D camera is installed directly above the conveyor belt, the normal vector of the 3D camera is vertically downward, while the normal vectors of the conveyor belt and the medicine box are vertically upward. z Set to n z <0. The constraint equation for the medicine box above the conveyor belt can be expressed as:

[0143] n x (xp x )+ny (yp y )+n z (zp z )>0(9)

[0144] Step S403: Obtain a first point cloud image of the medicine box on the conveyor belt when it is at a first reference position through a 3D camera, and determine the first coordinate point on the conveyor belt plane projected by the center point of the medicine box when it is at the first reference position using the normal vector of the conveyor belt as the projection direction, by combining the normal vector of the conveyor belt and the coordinates of any reference point.

[0145] In the above step S403, refer to Figure 4 , the first point cloud image of the medicine box on the conveyor belt is obtained by a 3D camera when the medicine box is at the first reference position, and the point cloud preprocessing and plane extraction operations are performed on the first point cloud image to obtain the normal vector n1=(n x1 ,n y1 ,n z1 ) t and the coordinates of the center point of the upper surface of the medicine box at the first reference position p1=(p x1 ,p y1 ,p z1 ) t , combined with the above formula (9), the plane equation of the upper surface point cloud of the medicine box at the first reference position can be established:

[0146] n x1 (xp x1 )+n y1 (yp y1 )+n z1 (zp z1 )=0 (10)

[0147] According to the normal vector n of the conveyor belt = (n x ,n y ,n z ) t and the coordinates p of any reference point on the conveyor belt v =(p x ,p y ,p z ) t , the center point of the upper surface of the medicine box when it is in the first reference position can be calculated as p1=(p x1 ,p y1 ,p z1 ) t In the projection direction n=(n x ,n y ,n z ) t The first coordinate point projected onto the conveyor plane is p'1 = (p' x1 ,p'y1 ,p' z1 ) t .

[0148] Step S404: Obtain a second point cloud image of the medicine box on the conveyor belt when it is at a second reference position through a 3D camera. Combine the normal vector of the conveyor belt and the coordinates of any reference point to determine the center point of the medicine box when it is at the second reference position, projecting the center point onto the conveyor belt plane with the normal vector of the conveyor belt as the projection direction.

[0149] In the above step S404, refer to Figure 5 , obtain the second point cloud image of the medicine box on the conveyor belt at the second reference position through the 3D camera, perform point cloud preprocessing and plane extraction operations on the second point cloud image, and obtain the normal vector n2=(n x2 ,n y2 ,n z2 ) t and the coordinates of the center point of the upper surface of the medicine box at the second reference position p2=(p x2 ,p y2 ,p z2 ) t , combined with the above formula (9), the plane equation of the upper surface point cloud of the medicine box at the second reference position can be established:

[0150] n 2x (xp 2x )+n 2y (yp 2y )+n z1 (zp 1z )=0 (11)

[0151] According to the normal vector n of the conveyor belt = (n x ,n y ,n z ) t and the coordinates p of any reference point on the conveyor belt v =(p x ,p y ,p z ) t , the center point of the upper surface of the medicine box when it is in the second reference position can be calculated as p2 = (p x2 ,p y2 ,p z2 ) t In the projection direction n=(n x ,n y ,n z ) t The second coordinate point projected onto the conveyor plane is p'2 = (p' x2 ,p' y2 ,p' z2) t .

[0152] Step S405: Obtain a conveyor belt motion direction vector based on the first coordinate point and the second coordinate point.

[0153] In the above step S405, based on the first coordinate point and the second coordinate point, the conveyor belt motion direction vector v shown in the following formula (12) is obtained:

[0154] v=p'2-p'1=(p' x2 -p' x1 ,p' y2 -p' y1 ,p' z2 -p' z1 ) t (12)

[0155] Step S406: Obtain the lateral direction vector of the conveyor belt based on the conveyor belt movement direction vector and the conveyor belt normal vector.

[0156] In the above step S406, since the conveyor belt motion direction vector v and the conveyor belt transverse direction vector n are perpendicular to each other, that is, v⊥n. According to the conveyor belt motion direction vector v and the conveyor belt normal vector n, the conveyor belt transverse direction vector u can be calculated as:

[0157] u=n×v (13)

[0158] Step S407: Based on the first coordinate point, determine the distance from the first coordinate point to a predetermined reference point, and determine the coordinates of the reference point in the 3D camera coordinate system in combination with the conveyor belt motion direction vector.

[0159] In the above step S407, refer to Figure 3 and Figure 4 , based on the first coordinate point, the distance of the projection point p'1 can be calculated Figure 3 The distance d from the reference point O on the left side of the conveyor belt is:

[0160] d=scale1+0.5w (14)

[0161] In the above formula (14), w is the width of the medicine box.

[0162] The conveyor coordinate system is established with the leftmost reference point O on the conveyor plane. The x-axis direction of the coordinate system is the conveyor moving direction, the y-axis direction is the conveyor lateral direction, and the z-axis direction is the conveyor normal direction. The coordinates of the reference point O in the 3D camera coordinate system are p o It can be expressed as:

[0163] p o =p'1-d*v (15)

[0164] Step S408: Based on the conveyor belt's transverse direction vector, the conveyor belt's motion direction vector, and the conveyor belt's normal vector, obtain the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system, and combine the coordinates of the reference point in the 3D camera coordinate system to obtain the coordinate system transformation matrix.

[0165] In the above step S408, based on the conveyor belt transverse direction vector, the conveyor belt motion direction vector and the conveyor belt normal vector, the rotation matrix R between the 3D camera coordinate system and the conveyor belt coordinate system is expressed by the following equation (16):

[0166]

[0167] Combined with the coordinates of the reference point in the 3D camera coordinate system, the coordinate system transformation matrix T is expressed as follows (17):

[0168]

[0169] Therefore, for any point p in the 3D camera coordinate system, it can be converted to the coordinate p in the conveyor coordinate system through the coordinate system transformation matrix b :

[0170] p b =R inv (pp o ) (18)

[0171] In the above formula (18), p b =(x b ,y b ,z b ) t , x b is relative to the reference point p o The component of movement in the conveyor belt moving direction v; R inv is the inverse of the rotation matrix R.

[0172] Any direction vector t in the 3D camera coordinate system can be converted into a vector in the conveyor coordinate system through the coordinate system transformation matrix:

[0173] t b =R inv t (19)

[0174] In this embodiment, based on the matched point cloud image of each medicine box, by performing radius filtering, downsampling, plane extraction and other operations on the matched point cloud image of each medicine box, the first positioning information of each medicine box in the 3D camera coordinate system (including the center point coordinates, the normal vector of the medicine box plane, the size of the medicine box, the coordinates of the four endpoints of the medicine box, and the pixel rectangular envelope of the medicine box) can be output. The first positioning information is obtained based on the 3D camera coordinate system, and the corresponding first positioning information (the center point coordinates of each medicine box in the 3D camera coordinate system, the normal vector of the medicine box plane, and the coordinates of the four endpoints of the medicine box) can be converted into the second positioning information (the center point coordinates of each medicine box in the 3D camera coordinate system, the normal vector of the medicine box plane, and the coordinates of the four endpoints of the medicine box in the data in the conveyor coordinate system) through the above formula (18) and formula (19).

[0175] In the above step S5, refer to Figure 6 For each medicine box set, based on the second positioning information of each medicine box in the medicine box set, the positioning rectangular frame information corresponding to the medicine box set is calculated, wherein the positioning rectangular frame information includes the coordinates p of the center point corresponding to the medicine box set c =(x c ,y c ,z c ) t , width W, and distances Wl and Wr from adjacent medicine box sets, and locate the rectangular box information as the medicine feeding position. Figure 6 The dotted frame shown in FIG is the boundary of the positioning rectangular frame of a set of medicine boxes.

[0176] In the above step S6, for each medicine box set, the specific process of obtaining the category of each medicine box may include the following steps:

[0177] Step S601: For each medicine box set, based on the high-resolution 2D image of each medicine box in the medicine box set, a preset visual feature extraction algorithm is used to extract feature information of each medicine box.

[0178] In the above step S601, the visual feature extraction algorithm can adopt a traditional feature extraction algorithm (such as ORB feature extraction algorithm) or a deep learning algorithm (such as ResNet-50 / 101, EfficientNet-B4 or CNN+Transformer hybrid model, etc.).

[0179] Step S602: Calculate the cosine similarity between the feature information of each medicine box and each template in the pre-established feature library, select the template corresponding to the maximum cosine similarity, and use the category corresponding to the template as the category of each medicine box.

[0180] In this embodiment, the drug feed parameters may include the drug feed location, the size of the medicine box, the number of medicine boxes, and the type of medicine box. By grouping the medicine boxes and calculating the group positioning parameters (i.e., the second positioning information), the drug feed position of the flat-slot medicine dispenser's feed port can be determined. This can be used as a lateral positioning reference for the robotic arm, adapting to the random placement of different medicines. By grouping all the medicine boxes using the 2D image corresponding to each medicine box, the number of medicine boxes in each group can be determined by segmenting and positioning the medicine boxes, which can be used to plan the number of grasping times for the robotic arm. Based on the high-resolution 2D image of each medicine box in each group, high-resolution feature information of the medicine box can be obtained, thereby enabling the classification of the medicine boxes, facilitating drug sorting and verification, and avoiding the limited ability of traditional manual verification or single visual recognition methods to distinguish complex textures and similar packaging. Furthermore, if the medicine boxes have a QR code or barcode, a 2D camera can be used to scan the barcode or QR code, eliminating the need for a separate dedicated scanning device, effectively simplifying the structure and saving costs. At the same time, based on the point cloud image of each medicine box in each set, the dimensions of each medicine box can be output, thereby determining the degree of opening and closing of the robotic arm and abnormality detection. By fusing image data from 3D cameras with image data from 2D cameras, the entire process from medicine inlet position and medicine box identification to grasping operation is automated, effectively improving the intelligence level and operational reliability of the flat-trough medicine dispenser.

[0181] Example 2

[0182] Based on the same inventive concept, see Figure 7 The embodiment of the present application further proposes a device for determining medicine feeding parameters of a flat slot medicine dispenser, comprising:

[0183] Image collection module 101, used to obtain a 2D image and point cloud image of each medicine box on the conveyor belt through a 3D camera, and obtain a high-resolution 2D image of each medicine box on the conveyor belt through a 2D camera;

[0184] A grouping module 102 is configured to group all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets;

[0185] The data matching module 103 is configured to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera and the internal reference information of the 2D camera for each medicine box set, thereby obtaining a matched point cloud image of each medicine box.

[0186] Positioning information confirmation module 104 is used to obtain first positioning information in camera coordinates for each set of medicine boxes based on the matched point cloud corresponding to each medicine box in the set, and obtain second positioning information in conveyor coordinates by combining the pre-established coordinate system conversion matrix;

[0187] The medicine feeding position confirmation module 105 is configured to calculate, for each medicine box set, the positioning rectangular frame information corresponding to the medicine box set based on the second positioning information of each medicine box in the medicine box set, and obtain the medicine feeding position, wherein the medicine feeding position includes the center point coordinates, width, and distance from adjacent medicine box sets corresponding to the medicine box set;

[0188] The category recognition module 106 is configured to extract feature information from each medicine box set based on a high-resolution 2D image of each medicine box in the medicine box set, and match the feature information with a pre-established feature library to obtain the category of each medicine box.

[0189] The implementation principle and technical effects of the device for determining the medicine feeding parameters of the flat-trough medicine dispenser provided in the embodiment of the present invention are similar to those of the first embodiment and will not be repeated here.

[0190] Example 3

[0191] Based on the same inventive concept, an embodiment of the present application also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for determining the medicine feeding parameters of the flat-slot medicine dispenser as in Example 1 is implemented.

[0192] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the first embodiment of the present invention.

[0193] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] Example 4

[0195] Based on the same inventive concept, an embodiment of the present application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements a method for determining the medicine feeding parameters of the flat-slot medicine dispenser as in Example 1.

[0196] Example 5

[0197] Based on the same inventive concept, an embodiment of the present application proposes a computer program product containing instructions. When the computer program product is run on a computer device, the computer device executes the method for determining the medicine feeding parameters of the flat-slot medicine dispenser in Example 1.

[0198] The principles of solving the problems described above by the apparatus, client, medium, and related equipment in the embodiments of the present invention are similar to those of the aforementioned methods, so their implementation can refer to the implementation of the aforementioned methods, and repeated details will not be repeated.

[0199] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0203] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. The present disclosure is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and variations may be made without departing from the scope of the present disclosure. The scope of the present disclosure is limited solely by the appended claims. Thus, to the extent such modifications and variations fall within the scope of the claims and their equivalents, the present disclosure is intended to include such modifications and variations.

Claims

1. A method for determining the feeding parameters of a flat slot medicine dispenser, characterized in that: include: The 3D camera acquires a 2D image and point cloud of each medicine box on the conveyor belt, and the 2D camera acquires a high-resolution 2D image of each medicine box on the conveyor belt; grouping all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets; For each set of medicine boxes, the high-resolution 2D image of each medicine box in the set is matched with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the internal reference information of the 2D camera, to obtain the matched point cloud image of each medicine box. For each set of medicine boxes, the first positioning information in camera coordinates is obtained based on the matched point cloud image corresponding to each medicine box in the set, and the second positioning information in conveyor coordinates is obtained by combining the pre-established coordinate system transformation matrix; For each medicine box set, based on the second positioning information of each medicine box in the medicine box set, calculate the positioning rectangular frame information corresponding to the medicine box set to obtain the medicine feeding position, wherein the medicine feeding position includes the center point coordinates, width, and distance from adjacent medicine box sets corresponding to the medicine box set; For each medicine box set, feature information is extracted based on a high-resolution 2D image of each medicine box in the medicine box set, and the feature information is matched with a pre-established feature library to obtain the category of each medicine box.

2. The method for determining the feeding parameters of the flat slot medicine dispenser according to claim 1, characterized in that: For each set of medicine boxes, the high-resolution 2D image of each medicine box in the medicine box set is matched with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera and the internal reference information of the 2D camera to obtain a matched point cloud image of each medicine box, including: For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the internal reference information of the 2D camera, and the point cloud image corresponding to each medicine box in the medicine box set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotated rectangular frame of each medicine box is obtained; For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the medicine box set, the second pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the second pixel rotated rectangular frame of each medicine box is obtained; For each set of medicine boxes, the second pixel rotated rectangular frame corresponding to each medicine box in the medicine box set is matched one by one with the first pixel rotated rectangular frames of all the medicine boxes in the medicine box set to obtain the first pixel rotated rectangular frame that matches the second pixel rotated rectangular frame corresponding to each medicine box, and obtain the matched point cloud image of each medicine box.

3. The method for determining the feeding parameters of the flat slot medicine dispenser according to claim 2, characterized in that: For each medicine box set, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the internal reference information of the 2D camera, and the point cloud image corresponding to each medicine box in the medicine box set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotated rectangular frame is obtained, including: For each set of medicine boxes, determine the coordinates of the center point and the four endpoints in the 3D camera coordinate system based on the point cloud image corresponding to each medicine box in the medicine box set. Combined with the rotation matrix and translation vector between the 3D camera and the 2D camera, the coordinates of the center point and the four endpoints of each medicine box in the 2D camera coordinate system are obtained; Based on the internal reference information of the 2D camera, the center point coordinates of each medicine box in the 2D camera coordinate system and the coordinates of the four endpoints, the first pixel coordinates of the center point of each medicine box and the first pixel coordinates of the four endpoints are obtained, and the first pixel rotated rectangular frame of each medicine box is obtained.

4. The method for determining the feeding parameters of the flat slot medicine dispenser according to claim 2, characterized in that: For each medicine box set, the second pixel rotated rectangular frame corresponding to each medicine box in the medicine box set is matched one by one with the first pixel rotated rectangular frames of all medicine boxes in the medicine box set to obtain a first pixel rotated rectangular frame that matches the second pixel rotated rectangular frame corresponding to each medicine box, and obtain a matched point cloud image of each medicine box, including: For each medicine box in each medicine box set, the intersection-and-union ratio between the second pixel rotated rectangular box corresponding to each medicine box and the first pixel rotated rectangular boxes of all medicine boxes is calculated using the following formula: in, Represents the i-th second pixel rotated rectangle; Indicates the jth first pixel rotated rectangular box; ROI ij represents the intersection-over-union ratio between the i-th second pixel rotated rectangle and the j-th first pixel rotated rectangle; Based on all interaction ratios, the first pixel-rotated rectangular box corresponding to the largest intersection-and-union ratio is obtained as the first pixel-rotated rectangular box that matches the second pixel-rotated rectangular box corresponding to each medicine box, and the point cloud image corresponding to the matched first pixel-rotated rectangular box is used as the matched point cloud image of each medicine box.

5. The method for determining the medicine feeding parameters of the flat slot medicine dispenser according to claim 1, characterized in that: The step of grouping all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets includes: Based on the 2D image corresponding to each medicine box, a pixel rectangular envelope of each medicine box is obtained, and parameters of each pixel rectangular envelope are obtained, wherein the parameters include a horizontal coordinate of an upper left corner, a width, a height, and a horizontal coordinate of a lower right corner; Sort all the pixel rectangular envelopes according to the order of the upper left corner horizontal coordinate from small to large to obtain a pixel rectangular envelope sequence; Traversing all pixel rectangular envelopes in the pixel rectangular envelope sequence, and determining whether the horizontal coordinate of the lower right corner of any pixel rectangular envelope is greater than or equal to the horizontal coordinate of the upper left corner of the next pixel rectangular envelope; If so, the two pixel rectangular envelopes are merged into the same medicine box set, and the range of the merged rectangular frame is updated; If not, the two pixel rectangular envelopes cannot be merged into the same medicine box set, and the judgment process is re-executed for the next pixel rectangular envelope.

6. The method for determining the medicine feeding parameters of the flat slot medicine dispenser according to claim 1, characterized in that: For each medicine box set, feature information is extracted based on a high-resolution 2D image of each medicine box in the medicine box set, and the feature information is matched with a pre-established feature library to obtain the category of each medicine box, including: For each set of medicine boxes, based on the high-resolution 2D image of each medicine box in the set, a pre-set visual feature extraction algorithm is used to extract the feature information of each medicine box; For the feature information of each medicine box, the cosine similarity with each template in the pre-established feature library is calculated, and the template corresponding to the maximum cosine similarity is screened out, and the category corresponding to the template is used as the category of each medicine box.

7. The method for determining the feeding parameters of a flat slot medicine dispenser according to claim 1, characterized in that: The method for constructing the coordinate system conversion matrix includes: Use a 3D camera to obtain a point cloud image of the conveyor belt when no medicine boxes are placed on the conveyor belt; Based on the conveyor belt point cloud image, obtaining the normal vector of the conveyor belt and the coordinates of any reference point; Acquire a first point cloud image of the medicine box on the conveyor belt when it is at a first reference position using a 3D camera, and determine a first coordinate point projected onto the conveyor belt plane by the center point of the medicine box at the first reference position using the conveyor belt normal vector as a projection direction, based on the normal vector of the conveyor belt and the coordinates of any reference point; Acquire a second point cloud image of the medicine box on the conveyor belt when it is at a second reference position using a 3D camera, and determine a second coordinate point on the conveyor belt plane where the center point of the medicine box at the second reference position is projected using the conveyor belt normal vector as a projection direction, by combining the normal vector of the conveyor belt and the coordinates of any reference point; Obtaining a conveyor belt motion direction vector based on the first coordinate point and the second coordinate point; Obtaining a conveyor belt transverse direction vector based on the conveyor belt motion direction vector and the conveyor belt normal vector; Based on the first coordinate point, determining a distance from the first coordinate point to a predetermined reference point, and determining the coordinates of the reference point in a 3D camera coordinate system in combination with the conveyor belt motion direction vector; Based on the lateral direction vector of the conveyor belt, the movement direction vector of the conveyor belt and the normal vector of the conveyor belt, the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system is obtained, and the coordinate system conversion matrix is ​​obtained by combining the coordinates of the reference point in the 3D camera coordinate system.

8. The method for determining the feeding parameters of a flat slot medicine dispenser according to claim 1, characterized in that: The method of obtaining a 2D image and a point cloud image of each medicine box on the conveyor belt by a 3D camera and obtaining a high-resolution 2D image of each medicine box on the conveyor belt by a 2D camera includes: Performing image segmentation on the original 2D images of all medicine boxes on the conveyor belt acquired by the 3D camera to obtain a first segmentation mask corresponding to each medicine box, and obtaining a 2D image and a point cloud map corresponding to each medicine box based on the first segmentation mask; Image segmentation is performed based on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask corresponding to each medicine box, and a high-resolution 2D image corresponding to each medicine box is obtained based on the second segmentation mask.

9. A device for determining the feeding parameters of a flat slot medicine dispenser, characterized in that: include: An image collection module is used to obtain a 2D image and point cloud image of each medicine box on the conveyor belt through a 3D camera, and to obtain a high-resolution 2D image of each medicine box on the conveyor belt through a 2D camera; a grouping module, configured to group all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple medicine box sets; A data matching module is used to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud image based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the internal reference information of the 2D camera, for each medicine box set, to obtain a matched point cloud image of each medicine box; A positioning information confirmation module is used to obtain, for each set of medicine boxes, first positioning information in camera coordinates based on the matched point cloud image corresponding to each medicine box in the set, and to obtain second positioning information in conveyor coordinates by combining a pre-established coordinate system conversion matrix; a medicine feeding position confirmation module, configured to calculate, for each medicine box set, the positioning rectangular frame information corresponding to the medicine box set based on the second positioning information of each medicine box in the medicine box set, and obtain the medicine feeding position, wherein the medicine feeding position includes the coordinates of the center point corresponding to the medicine box set, the width, and the distance from the adjacent medicine box set; The category recognition module is used to extract feature information from each medicine box set based on the high-resolution 2D image of each medicine box in the medicine box set, match the feature information with a pre-established feature library, and obtain the category of each medicine box.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining the medicine feeding parameters of the flat-slot medicine dispenser as described in any one of claims 1 to 8 is implemented.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for determining the medicine feeding parameters of the flat-slot medicine dispenser according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising instructions, which, when run on a computer device, causes the computer device to execute the method for determining medicine feeding parameters of a flat-slot medicine dispenser according to any one of claims 1 to 8.

Images