A method and apparatus for determining the feeding parameters of a flat-groove dispensing machine.

Abstract

This invention discloses a method and apparatus for determining the feeding parameters of a flat-groove dispensing machine. The method includes: acquiring 2D images and point cloud maps of each medicine box using a 3D camera; acquiring high-resolution 2D images of each medicine box using a 2D camera; grouping all medicine boxes based on the 2D images corresponding to each medicine box; matching the high-resolution 2D images of each medicine box in the group with the point cloud maps to obtain matched point cloud maps for each medicine box; obtaining first positioning information in camera coordinates based on the matched point cloud maps of each medicine box in the group, and obtaining second positioning information in conveyor belt coordinates by combining a pre-established coordinate system transformation matrix; calculating the positioning rectangle information corresponding to the medicine box group based on the second positioning information of each medicine box in the group; and extracting feature information from the high-resolution 2D images of each medicine box in the group to obtain the category of each medicine box. This method can automatically determine the feeding parameters, improving the level of intelligence and operational reliability.

Description

Technical Field

[0001] This invention relates to the field of medical device manufacturing technology, and in particular to a method and apparatus for determining the feeding parameters of a flat-groove dispensing machine. Background Technology

[0002] As a type of automated equipment in hospital pharmacies, the flat-slot dispensing machine is mainly used to improve the efficiency, accuracy, and safety of drug dispensing.

[0003] Flat-slot dispensing machines typically employ a multi-layer track or storage tank design, with medications categorized and stored within the tanks according to specifications and type. Operators neatly arrange medications in designated locations through the dispensing port, and the equipment automatically links medication information to the storage tank via barcode scanning or an identification system. A robotic arm or pushing device quickly delivers the medications into the storage tanks, reducing manual operation time. Dispensed medications are then transported via conveyor belts or chutes to intelligent medicine baskets or dispensing windows, significantly improving pharmacy efficiency. Summary of the Invention

[0004] In order to automatically and effectively determine the feeding parameters, thereby effectively improving the intelligence level and operational reliability of the flat-groove dispensing machine, this invention provides a method and apparatus for determining the feeding parameters of the flat-groove dispensing machine.

[0005] In a first aspect, embodiments of the present invention provide a method for determining the feeding parameters of a flat-groove dispensing machine, comprising:

[0006] 2D images and point cloud maps of each medicine box on the conveyor belt are obtained using a 3D camera, and high-resolution 2D images of each medicine box on the conveyor belt are obtained using a 2D camera.

[0007] Based on the 2D image corresponding to each medicine box, all medicine boxes are grouped to obtain multiple sets of medicine boxes;

[0008] For each set of medicine boxes, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, the high-resolution 2D image of each medicine box in the set is matched with the point cloud map to obtain the matched point cloud map of each medicine box.

[0009] For each set of medicine boxes, based on the matched point cloud map corresponding to each medicine box in the set, the first positioning information in the camera coordinates is obtained, and combined with the pre-established coordinate system transformation matrix, the second positioning information in the conveyor belt coordinates is obtained.

[0010] For each set of medicine boxes, based on the second positioning information of each medicine box in the set, the positioning rectangle information corresponding to the set of medicine boxes is calculated to obtain the medicine delivery position. The medicine delivery position includes the center point coordinates, width, and distance to adjacent sets of medicine boxes.

[0011] For each set of medicine boxes, feature information is extracted based on the high-resolution 2D image of each medicine box in the set. The feature information is then matched with a pre-established feature library to obtain the category of each medicine box.

[0012] Optionally, for each set of pillboxes, based on the pre-calibrated rotation matrix and translation vector between the 3D and 2D cameras, as well as the intrinsic parameter information of the 2D camera, the high-resolution 2D image of each pillbox in the set is matched with the point cloud map to obtain the matched point cloud map for each pillbox, including:

[0013] For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotation rectangle of each medicine box is obtained.

[0014] For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the set, the second pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the second pixel rotated rectangle of each medicine box is obtained.

[0015] For each set of medicine boxes, the second pixel rotation rectangle corresponding to each medicine box in the set is matched one by one with the first pixel rotation rectangle of all medicine boxes in the set to obtain the first pixel rotation rectangle that matches the second pixel rotation rectangle corresponding to each medicine box, thus obtaining the matched point cloud map of each medicine box.

[0016] Optionally, for each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, resulting in a first pixel rotation rectangle, including:

[0017] For each set of medicine boxes, the coordinates of the center point and the four endpoints in the 3D camera coordinate system are determined based on the point cloud map corresponding to each medicine box in the set. Combined with the rotation matrix and translation vector between the 3D camera and the 2D camera, the coordinates of the center point and the four endpoints of each medicine box in the 2D camera coordinate system are obtained.

[0018] Based on the intrinsic parameters of the 2D camera, the coordinates of the center point of each medicine box in the 2D camera coordinate system, and the coordinates of the four endpoints, the first pixel coordinates of the center point and the first pixel coordinates of the four endpoints of each medicine box are obtained, and the first pixel rotated rectangle of each medicine box is obtained.

[0019] Optionally, for each set of medicine boxes, the second pixel rotated rectangle corresponding to each medicine box in the set is matched one by one with the first pixel rotated rectangles of all medicine boxes in the set to obtain the first pixel rotated rectangle that matches the second pixel rotated rectangle corresponding to each medicine box, thus obtaining the matched point cloud map of each medicine box, including:

[0020] For each medicine box in each medicine box set, the intersection-union ratio (IUU) between the second pixel rotated rectangle of each medicine box and the first pixel rotated rectangles of all medicine boxes is calculated using the following formula:

[0021]

[0022] in, This represents the i-th second-pixel rotated rectangle; Represents the first pixel of the j-th rotated rectangle; ROI ij This represents the intersection-union ratio between the i-th second-pixel rotated rectangle and the j-th first-pixel rotated rectangle;

[0023] Based on all interaction ratios, the first pixel rotation rectangle corresponding to the largest intersection-union ratio is obtained, which is used as the first pixel rotation rectangle that matches the second pixel rotation rectangle corresponding to each medicine box. The point cloud map corresponding to the matched first pixel rotation rectangle is used as the matched point cloud map for each medicine box.

[0024] Optionally, the step of grouping all the medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple sets of medicine boxes includes:

[0025] Based on the 2D image corresponding to each medicine box, a pixel rectangle envelope of each medicine box is obtained, and the parameters of each pixel rectangle envelope are obtained, including the horizontal coordinate of the upper left corner, the width, the height, and the horizontal coordinate of the lower right corner;

[0026] Sort all the pixel rectangle envelopes in ascending order of their top-left horizontal coordinates to obtain a pixel rectangle envelope sequence.

[0027] Traverse all pixel rectangles in the pixel rectangle envelope sequence and determine whether the lower right corner x-coordinate of any pixel rectangle envelope is greater than or equal to the upper left corner x-coordinate of the next pixel rectangle envelope.

[0028] If so, merge the two pixel rectangular envelopes into the same set of medicine boxes, and update the range of the merged rectangle.

[0029] If not, the two pixel rectangular envelopes cannot be merged into the same set of medicine boxes, and the judgment process is re-executed for the next pixel rectangular envelope.

[0030] Optionally, for each set of pillboxes, feature information is extracted based on a high-resolution 2D image of each pillbox in the set, and the feature information is matched with a pre-established feature library to obtain the category of each pillbox, including:

[0031] For each set of medicine boxes, feature information of each medicine box is extracted using a pre-set visual feature extraction algorithm based on the high-resolution 2D image of each medicine box in the set.

[0032] For each medicine box, the cosine similarity with each template in the pre-established feature library is calculated based on the feature information. The template with the highest cosine similarity is then selected, and the category corresponding to the template is used as the category of each medicine box.

[0033] Optionally, the method for constructing the coordinate system transformation matrix includes:

[0034] A point cloud image of the conveyor belt without medicine boxes is obtained using a 3D camera.

[0035] Based on the point cloud map of the conveyor belt, the normal vector of the conveyor belt and the coordinates of any reference point are obtained;

[0036] The first point cloud image of the medicine box on the conveyor belt when it is in the first reference position is obtained by a 3D camera. Combined with the normal vector of the conveyor belt and the coordinates of any reference point, the center point of the medicine box when it is in the first reference position is determined to be the first coordinate point projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction.

[0037] The second point cloud map of the medicine box on the conveyor belt when it is in the second reference position is obtained by a 3D camera. Combined with the normal vector of the conveyor belt and the coordinates of any reference point, the center point of the medicine box when it is in the second reference position is determined to be the second coordinate point projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction.

[0038] Based on the first coordinate point and the second coordinate point, the conveyor belt movement direction vector is obtained;

[0039] Based on the conveyor belt's motion direction vector and the conveyor belt's normal vector, the conveyor belt's lateral direction vector is obtained;

[0040] Based on the first coordinate point, the distance from the first coordinate point to a predetermined reference point is determined, and combined with the conveyor belt movement direction vector, the coordinates of the reference point in the 3D camera coordinate system are determined;

[0041] Based on the lateral direction vector of the conveyor belt, the motion direction vector of the conveyor belt, and the normal vector of the conveyor belt, the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system is obtained. Combined with the coordinates of the reference point in the 3D camera coordinate system, the coordinate system transformation matrix is ​​obtained.

[0042] Optionally, the step of acquiring 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and acquiring high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera, includes:

[0043] Image segmentation is performed on the original 2D images of all medicine boxes on the conveyor belt obtained by the 3D camera to obtain the first segmentation mask for each medicine box, and the 2D image and point cloud map corresponding to each medicine box are obtained based on the first segmentation mask.

[0044] Image segmentation is performed on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask for each medicine box, and a high-resolution 2D image for each medicine box is obtained based on the second segmentation mask.

[0045] Secondly, embodiments of the present invention provide a device for determining the feeding parameters of a flat-groove dispensing machine, comprising:

[0046] The image acquisition module is used to acquire 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and to acquire high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera.

[0047] The grouping module is used to group all the medicine boxes based on the 2D image corresponding to each medicine box, so as to obtain multiple sets of medicine boxes;

[0048] The data matching module is used to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud map based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, to obtain the matched point cloud map of each medicine box.

[0049] The positioning information confirmation module is used to obtain the first positioning information in camera coordinates based on the matched point cloud map corresponding to each medicine box in each medicine box set, and to obtain the second positioning information in conveyor belt coordinates by combining the pre-established coordinate system transformation matrix.

[0050] The medication placement location confirmation module is used to calculate the positioning rectangle information corresponding to each set of medicine boxes based on the second positioning information of each medicine box in the set, and to obtain the medication placement location. The medication placement location includes the center point coordinates, width, and distance to adjacent sets of medicine boxes corresponding to the set.

[0051] The category recognition module is used to extract feature information based on the high-resolution 2D image of each medicine box in each medicine box set, and match the feature information with a pre-established feature library to obtain the category of each medicine box.

[0052] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the feeding parameters of a flat-groove dispensing machine as described in the first aspect.

[0053] Fourthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for determining the feeding parameters of a flat-groove dispensing machine as described in the first aspect.

[0054] Fifthly, embodiments of the present invention provide a computer program product containing instructions that, when the computer program product is run on a computer device, cause the computer device to execute the method for determining the feeding parameters of a flat-groove dispensing machine as described in the first aspect.

[0055] The beneficial effects of the above-mentioned technical solutions provided in the embodiments of the present invention include at least the following:

[0056] This invention provides a method for determining the feeding parameters of a flat-slot dispensing machine. A 3D camera acquires 2D images and point cloud maps of each medicine box on the conveyor belt, and a 2D camera acquires high-resolution 2D images of each medicine box on the conveyor belt. The feeding parameters can include the feeding position and the medicine box category. By grouping the medicine boxes and calculating group positioning parameters (i.e., second positioning information), the feeding position of the flat-slot dispensing machine's feed inlet can be determined. This can be used as a lateral positioning reference for the robotic arm and can adapt to the random placement of different medicines. Based on the high-resolution 2D images of each medicine box in each group, high-resolution feature information of the medicine boxes can be obtained, thereby enabling medicine box category identification, facilitating medicine sorting and verification, and avoiding the limited ability of traditional manual verification or single visual recognition methods to distinguish complex textures and similar packaging. By fusing image data from 3D and 2D cameras, the entire process from feeding position and medicine box recognition to grasping operation is automated, effectively improving the intelligence level and operational reliability of the flat-slot dispensing machine.

[0057] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0058] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0059] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0060] Figure 1 This is a schematic diagram of the structure of the flat-groove dispensing machine provided in an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of the method for determining the feeding parameters of the flat-groove dispensing machine provided in the embodiments of the present invention;

[0062] Figure 3 This is a schematic diagram of the starting position calibrated during the construction of the coordinate system transformation matrix provided in this embodiment of the invention;

[0063] Figure 4 This is a schematic diagram of the first reference position calibrated during the construction of the coordinate system transformation matrix provided in this embodiment of the invention;

[0064] Figure 5 This is a schematic diagram of the second reference position calibrated during the construction of the coordinate system transformation matrix provided in this embodiment of the invention;

[0065] Figure 6 This is a schematic diagram of each set of medicine boxes provided in the embodiments of the present invention;

[0066] Figure 7 This is a schematic diagram of the device for determining the feeding parameters of the flat-groove dispensing machine provided in an embodiment of the present invention. Detailed Implementation

[0067] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0068] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," "far," "near," "front," and "rear," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0069] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0070] The inventors discovered that in existing technologies, the positioning of medicines in flat-slot dispensing machines mainly relies on fixed slots or mechanical limiting devices, which cannot dynamically adapt to the random placement of different medicines. Medicine category identification often relies on manual verification or single visual recognition methods, which have limited ability to distinguish complex textures and similar packaging. Existing visual recognition systems suffer from functional fragmentation in medicine information acquisition: 3D point cloud-based positioning technology struggles to simultaneously acquire high-resolution surface texture features, while 2D imaging systems lack depth information, resulting in insufficient dimensional measurement accuracy. Furthermore, barcode recognition modules typically require separate dedicated scanning equipment, complicating system structure and increasing maintenance costs.

[0071] Therefore, in order to solve the above problems, the inventors have developed a method and device for determining the feeding parameters of a flat-groove dispensing machine. Through multimodal data fusion technology, the fully automatic and high-precision identification of the feeding position and the type of medicine box is realized, so as to accurately obtain the feeding parameters and effectively improve the intelligence level and operational reliability of the flat-groove dispensing machine.

[0072] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0073] A segmentation mask is an image representation used to label each pixel in an image as belonging to a specific category or target. It is a two-dimensional matrix of the same size as the original image, with each element corresponding to a pixel in the original image. For example, for a color image with a resolution of 640×480, its segmentation mask is also a 640×480 matrix. The value of each element represents the category label to which the pixel belongs, such as 0 for background and 1 for a medicine box. The segmentation mask accurately defines the position and shape of the medicine box in the image, enabling the system to accurately identify the medicine box and providing a clear target area for subsequent processing.

[0074] Point cloud is a representation of three-dimensional spatial data, composed of a large number of discrete points. Each point records its position information in space (usually including X, Y, and Z coordinates) and may have additional attributes (such as color, reflection intensity, and normal vector). Point cloud is a core data type in 3D reconstruction, computer vision, and robot perception, used to describe the geometric shape and surface features of objects or scenes.

[0075] Pixel bounding box (PixelBoundingBox): A fundamental data structure in computer vision and image processing used to describe the position and extent of a target object in a two-dimensional image. It completely encloses the pixel region of the target object in the image with a rectangle and records the key parameters of the rectangle (such as coordinates and dimensions), thus providing positional information for subsequent tasks such as target detection, recognition, segmentation, or tracking.

[0076] Intrinsic parameters of a 2D camera describe its internal optical and geometric characteristics, used to project points from the 3D camera coordinate system to the 2D pixel coordinate system. These intrinsic parameters are the core output of camera calibration, directly determining the image's imaging model and geometric distortion correction. Multiple images are acquired using a calibration board (such as a checkerboard), and intrinsic parameters are calculated using optimization algorithms (such as the Zhang Zhengyou calibration method). Intrinsic parameters may include focal length, principal point, distortion coefficients, and pixel size.

[0077] Focal length: The optical focal length of a camera lens. The horizontal focal length can be expressed as f0. x The focal length in the vertical direction can be expressed as f. y .

[0078] Principal point: The coordinates of the intersection of the camera's optical axis and the image plane, expressed in pixels. The horizontal principal point coordinates can be represented as c. x The coordinates of the principal point in the vertical direction can be represented as c. y .

[0079] Example 1

[0080] This embodiment proposes a method for determining the feeding parameters of a flat-groove dispensing machine, which can be applied to flat-groove dispensing machines. (See reference...) Figure 1This flat-slot dispensing machine mainly includes a 3D camera, a 2D camera, and a conveyor belt. Feeding parameters can include the feeding position, the size of the medicine box, the quantity of medicine boxes, and the type of medicine box. The 3D camera can locate the position of the medicine box and measure its size. The 2D camera can acquire high-resolution feature information of the medicine box, thereby enabling medicine box type identification. If the medicine box has a QR code or barcode, it can also scan the barcode or QR code. Segmenting the images from the 2D or 3D cameras can also obtain the quantity information of the fed medicine. By fusing and analyzing the images obtained from the 2D and 3D cameras, the feeding position and type of medicine box can be automatically and effectively identified, and the quantity and size of the medicine boxes can also be identified, thus accurately obtaining feeding parameters and effectively improving the intelligence level of the flat-slot dispensing machine.

[0081] See Figure 2 The method for determining the feeding parameters of the flat-groove dispensing machine may specifically include the following steps:

[0082] Step S1: Obtain 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and obtain high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera.

[0083] Step S2: Group all the medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple sets of medicine boxes;

[0084] Step S3: For each set of medicine boxes, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, the high-resolution 2D image of each medicine box in the set is matched with the point cloud map to obtain the matched point cloud map of each medicine box.

[0085] Step S4: For each set of medicine boxes, based on the matched point cloud map corresponding to each medicine box in the set, obtain the first positioning information in the camera coordinates, and combine it with the pre-established coordinate system transformation matrix to obtain the second positioning information in the conveyor belt coordinates.

[0086] Step S5: For each set of medicine boxes, based on the second positioning information of each medicine box in the set, calculate the positioning rectangle information corresponding to the set of medicine boxes to obtain the medicine delivery position. The medicine delivery position includes the center point coordinates, width and distance to adjacent sets of medicine boxes.

[0087] Step S6: For each set of medicine boxes, extract feature information based on the high-resolution 2D image of each medicine box in the set, match the feature information with the pre-established feature library, and obtain the category of each medicine box.

[0088] To provide a clearer explanation of the method for determining the feeding parameters of the flat-groove dispensing machine, a detailed explanation of each step is provided below.

[0089] In step S1 above, the specific process of acquiring 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and acquiring high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera, may include the following steps:

[0090] Step S101: Perform image segmentation on the original 2D images of all medicine boxes on the conveyor belt obtained by the 3D camera to obtain the first segmentation mask corresponding to each medicine box, and obtain the 2D image and point cloud map corresponding to each medicine box based on the first segmentation mask.

[0091] In step S101 above, a 3D camera can be installed directly above the conveyor belt, with a field of view covering the entire drug delivery area. The 3D camera (such as an RGB-D camera, LiDAR, or structured light camera) acquires the original 2D image and point cloud map of the medicine box on the conveyor belt for 3D spatial positioning and dimensional measurement. After installing the 3D camera, the point cloud map needs to be aligned to output an aligned 2D image and a 3D information matrix. Alignment means that the dimensions of the 2D image and the 3D information matrix are equal. The 3D information matrix stores the depth information or 3D coordinate information of each pixel in the 2D image, corresponding to the pixel coordinates. Therefore, after performing image segmentation on the original 2D image to obtain the first segmentation mask of the target, the 3D information matrix can be filtered using the first segmentation mask to obtain the desired 3D information of the target, i.e., the point cloud map.

[0092] On a conveyor belt, if two medicine boxes of the same height are placed close together, it is difficult to distinguish them using point cloud segmentation methods. However, a 3D camera that aligns the original point cloud image using the method described above can segment each medicine box. The 3D camera can acquire the original 2D image and the original point cloud image corresponding to each pixel of the original 2D image.

[0093] Before segmenting the original 2D image, denoising (e.g., Gaussian filtering), normalization (mapping depth values ​​to a fixed range), and background separation (removing the conveyor belt background using thresholding) can be performed to improve image quality, reduce noise interference, remove irrelevant background, and simplify the segmentation task. After the above preprocessing, deep learning algorithms (e.g., YOLO series, Mask R-CNN, SOLO, etc.) or traditional methods (e.g., template matching) can be used to segment the original 2D image, outputting a first segmentation mask for each medicine box. Based on the first segmentation mask, the 2D image corresponding to each medicine box is cropped from the original 2D image. Since the original 2D image output by the 3D camera corresponds pixel-wise to the original point cloud image, the point cloud image corresponding to each medicine box can be extracted using the first segmentation mask.

[0094] Step S102: Perform image segmentation on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask for each medicine box, and obtain a high-resolution 2D image for each medicine box based on the second segmentation mask.

[0095] In step S102 above, a 2D camera (such as an industrial CCD camera) captures an image of the conveyor belt from directly above or diagonally above, obtaining a raw high-resolution 2D image containing feature information such as the color, texture, text, and two-dimensional contours of the medicine boxes. Before image segmentation of the raw high-resolution 2D image, denoising (median filtering), contrast enhancement (histogram equalization), and distortion correction (lens distortion elimination) can be performed to improve image quality, enhance target features, and highlight segmentation boundaries. After the above preprocessing, deep learning algorithms (such as YOLO series, Mask R-CNN, SOLO, etc.) or traditional methods (such as template matching) can be used to segment the raw high-resolution 2D image, obtaining a second segmentation mask corresponding to each medicine box. Based on the second segmentation mask, a high-resolution 2D image of each medicine box is cropped from the raw high-resolution 2D image, retaining the feature information of the medicine box for subsequent medicine box category recognition.

[0096] In step S2 above, all medicine boxes are grouped based on the 2D image corresponding to each box. By analyzing the spatial relationship of the pixel rectangular envelopes of the medicine boxes, physically adjacent or overlapping medicine boxes are merged into the same group, thereby optimizing subsequent processing (such as robotic arm grasping or batch sorting). See also Figure 1 The extension and retraction direction of the robotic arm of the flat-groove dispensing machine is the transverse direction of the conveyor belt. Therefore, the feeding medicine boxes are grouped according to the column direction on the conveyor belt. Specifically, the process of obtaining multiple sets of medicine boxes may include the following steps:

[0097] Step S201: Based on the 2D image corresponding to each medicine box, obtain the pixel rectangular envelope of each medicine box, and obtain the parameters of each pixel rectangular envelope, including the horizontal coordinate of the upper left corner, the width, the height, and the horizontal coordinate of the lower right corner.

[0098] In step S201 above, an object detection model (such as YOLO, Faster R-CNN) or a traditional method (such as contour detection + minimum bounding rectangle) can be used to detect the 2D image of each medicine box, generating a pixel rectangular envelope and obtaining the parameters of each pixel rectangular envelope. The parameters include the x-coordinate of the upper left corner, width, height, and the x-coordinate of the lower right corner. Of course, the parameters can also include the y-coordinate of the upper left corner. In this embodiment, since the pixel rectangular envelope is not a rotated rectangle, the x-coordinate of the upper left corner can also represent the x-coordinate of the left side, and the y-coordinate of the upper left corner can also represent the y-coordinate of the upper side.

[0099] The parameter list for each medicine box can be represented by the following formula (1):

[0100] rect i :x i ,y i ,w i ,h i ,1≤i≤N (1)

[0101] In equation (1) above, rect i w represents the rectangular bounding box of the i-th pixel segment. i and h i These represent the width and height of the pixel rectangle's envelope, respectively; x i and y i These represent the x-coordinate and y-coordinate of the top left corner, respectively.

[0102] The bottom right x-coordinate i It can be expressed as the following formula (2):

[0103] x' i =x i +w i (2)

[0104] The bottom right ordinate y' i It can be expressed as the following formula (3):

[0105] y' i =y i +h i (3)

[0106] Step S202: Sort all pixel rectangle envelopes in ascending order of their top-left horizontal coordinates to obtain a pixel rectangle envelope sequence.

[0107] In step S202 above, sorting all the pixel rectangular envelopes ensures that adjacent medicine boxes are arranged continuously in the sequence, which facilitates subsequent merging and judgment.

[0108] Step S203: Traverse all pixel rectangles in the pixel rectangle envelope sequence and determine whether the lower right corner x-coordinate of any pixel rectangle envelope is greater than or equal to the upper left corner x-coordinate of the next pixel rectangle envelope; if yes, proceed to step S204; if no, proceed to step S205.

[0109] In step S203 above, all pixel rectangle envelopes in the pixel rectangle envelope sequence are traversed, and the intersection of consecutive pixel rectangle envelopes is determined. The specific process of traversal and merging can be as follows: starting from the i-th pixel rectangle envelope in the sequence, it is compared with the (i+1)-th pixel rectangle envelope in turn. The grouping rule can be expressed by the following formula (4):

[0110]

[0111] If the current pixel rectangle and the next pixel rectangle intersect in the x-axis direction, i.e., x i +w i ≥x i+1 If the current pixel rectangle envelope and the next pixel rectangle envelope intersect and can be grouped together, then the union of the two is assigned to the current pixel rectangle envelope (i.e., updating the range of the merged rectangle), i.e., proceeding to step S204 below; if the current pixel rectangle envelope and the next pixel rectangle envelope do not intersect in the x-axis direction, i.e., these two pixel rectangle envelopes are not in the same group, then the current pixel rectangle envelope is treated as an independent group, and the judgment of the next pixel rectangle envelope continues, i = i + 1, 1 ≤ i ≤ N - 1, i.e., proceeding to step S205 below. After completing the grouping of all medicine boxes, an index value can be set for each group of medicine boxes. The index value includes the positioning information of each medicine box in the current group, which facilitates the positioning and identification of medicine boxes.

[0112] Step S204: Merge the two pixel rectangular envelopes into the same set of medicine boxes, and update the range of the merged rectangle.

[0113] Step S205: Two pixel rectangular envelopes cannot be merged into the same set of medicine boxes. The judgment process is re-executed for the next pixel rectangular envelope.

[0114] In this embodiment, by grouping all the medicine boxes, merging physically adjacent or overlapping boxes into the same group, efficient physical grouping of medicine boxes can be achieved. This guides the robotic arm to grasp multiple adjacent medicine boxes at once, improving efficiency. Simultaneously, it provides crucial spatial layout information for automated pharmacies or industrial sorting systems. Furthermore, it facilitates batch counting, allowing for the tallying of medicine boxes in each group and reducing repetitive operations.

[0115] In step S3 above, registration of the 3D and 2D cameras is required during installation. By registering the original 2D image output by the 3D camera and the original high-resolution 2D image output by the 2D camera, the positional relationship between the two cameras can be calculated, thereby obtaining the rotation matrix and translation vector between the 3D and 2D cameras. For each set of pillboxes, based on the pre-calibrated rotation matrix and translation vector between the 3D and 2D cameras, as well as the intrinsic parameter information of the 2D camera, the high-resolution 2D image of each pillbox in the set is matched with the point cloud map to obtain the matched point cloud map of each pillbox. The specific process may include the following steps:

[0116] Step S301: For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the set, obtain the first pixel coordinates of the four endpoints and the center point of each medicine box, and obtain the first pixel rotation rectangle of each medicine box.

[0117] In step S301 above, the specific process of obtaining the first pixel rotated rectangle of each medicine box for each set of medicine boxes may include the following steps:

[0118] Step S3011: For each set of medicine boxes, determine the coordinates of the center point and the coordinates of the four endpoints in the 3D camera coordinate system based on the point cloud map corresponding to each medicine box in the set. Combine the rotation matrix and translation vector between the 3D camera and the 2D camera to obtain the coordinates of the center point and the coordinates of the four endpoints in the 2D camera coordinate system of each medicine box.

[0119] In step S3011 above, for each medicine box, the center pixel coordinates in the high-resolution 2D image are (q x ,q y By performing radius filtering, downsampling, and plane extraction on the point cloud image of each medicine box, the coordinates of the center point of each medicine box in the 3D camera coordinate system, the normal vector of the medicine box plane, the dimensions (length, width, height) of the medicine box, the coordinates of the four endpoints of the medicine box, and the pixel rectangular envelope of the medicine box can be output. The coordinates of the center point of each medicine box in the 3D camera coordinate system are denoted as p. 3d =(x 3d ,y 3d ,z 3d ) t Combining the rotation matrix R' and translation vector t' between the 3D and 2D cameras, the coordinates p of the center point of each pillbox in the 2D camera coordinate system are... 2d It can be expressed as the following formula (5):

[0120] p 2d =R'p 3d +t' (5)

[0121] In equation (5) above, the coordinates of the center point of each medicine box in the 2D camera coordinate system are: p 2d =(x 2d ,y 2d ,z 2d ) t .

[0122] Step S3012: Based on the intrinsic parameter information of the 2D camera, the coordinates of the center point of each medicine box in the 2D camera coordinate system, and the coordinates of the four endpoints, the first pixel coordinates of the center point and the first pixel coordinates of the four endpoints of each medicine box are obtained, and the first pixel rotated rectangle of each medicine box is obtained.

[0123] In step S3012 above, the first pixel coordinate of the center point of each medicine box is used as an example for explanation. Based on the intrinsic parameters of the 2D camera (focal length and principal point), combined with the center point coordinates of each medicine box in the 2D camera coordinate system, the pixel coordinates of the center point of each medicine box on the 2D camera image are obtained, that is, the first pixel coordinates. The first pixel coordinates can be represented by the following formula (6):

[0124]

[0125] In equation (6) above, f x f is the focal length in the horizontal direction. y c is the focal length in the vertical direction; x c represents the coordinates of the principal point in the horizontal direction. y The coordinates of the principal point in the vertical direction; (p x ,p y Let p be the coordinates of the center point of the medicine box in the 3D camera coordinate system. 3d Transform to the first pixel coordinates of the 2D camera image.

[0126] Similarly, the four endpoints of the medicine box can be transformed using the same method to obtain the first pixel coordinates of the four endpoints. Combined with the first pixel coordinates of the center point of each medicine box, the first pixel rotation rectangle (rotRect) of each medicine box is obtained. 3d .

[0127] Step S302: For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the set, obtain the second pixel coordinates of the four endpoints and the center point of each medicine box, and obtain the second pixel rotation rectangle rotRect for each medicine box. 2d .

[0128] Step S303: For each set of medicine boxes, for the second pixel rotation rectangle corresponding to each medicine box in the set of medicine boxes, match it one by one with the first pixel rotation rectangle of all medicine boxes in the set of medicine boxes to obtain the first pixel rotation rectangle that matches the second pixel rotation rectangle corresponding to each medicine box, and obtain the matched point cloud map of each medicine box.

[0129] In step S303 above, the specific process of obtaining the matched point cloud map for each medicine box for each set of medicine boxes may include the following steps:

[0130] Step S3031: For each medicine box in each medicine box set, calculate the intersection-union ratio between the second pixel rotated rectangle corresponding to each medicine box and the first pixel rotated rectangle of all medicine boxes using the following formula (7):

[0131]

[0132] In equation (7) above, This represents the i-th second-pixel rotated rectangle; Represents the first pixel of the j-th rotated rectangle; ROI ij This represents the intersection-union ratio between the i-th second-pixel rotated rectangle and the j-th first-pixel rotated rectangle.

[0133] Therefore, for the i-th second pixel rotation rectangle corresponding to the 2D camera, M different ROI values ​​can be obtained by calculating its M first pixel rotation rectangles corresponding to the 3D camera.

[0134] Step S3032: Based on all interaction ratios, obtain the first pixel rotation rectangle corresponding to the largest intersection-union ratio, and use it as the first pixel rotation rectangle that matches the second pixel rotation rectangle corresponding to each medicine box. The point cloud map corresponding to the matched first pixel rotation rectangle is used as the matched point cloud map of each medicine box.

[0135] In step S3032 above, the largest ROI value among the M ROI values ​​is taken to obtain the first pixel rotation rectangle corresponding to the i-th first pixel rotation rectangle. This allows matching the second pixel rotation rectangle of each medicine box with its corresponding first pixel rotation rectangle, and the point cloud image corresponding to the matched first pixel rotation rectangle becomes the matched point cloud image for each medicine box. Thus, a correlation can be established between the 3D camera and the 2D camera, achieving accurate matching between the 2D image of the medicine box and the point cloud image. This enables subsequent matching of the positioning information of each medicine box from the 3D camera with the recognition and scanning information of each medicine box from the 2D camera, providing multimodal perception capabilities for the automated system of the flat-slot dispensing machine.

[0136] In step S4 above, see [reference] Figures 3-5This is a schematic diagram of the conveyor belt calibration during the construction of the coordinate system transformation matrix, where... Figure 3 To define the starting position, the leftmost side of the conveyor belt is aligned with the left side of the medicine box, with the center point of the left side of the conveyor belt as the reference point O. Figure 4 The first reference position is defined, and the distance the conveyor belt has moved relative to the starting position is scale1. Figure 5 Let the second reference position be calibrated. At this point, the conveyor belt has moved a distance of scale2 relative to the starting position, and clearly scale2 > scale1. Specifically, the method for constructing the coordinate system transformation matrix can include the following steps:

[0137] Step S401, see Figure 3 A point cloud map of the conveyor belt without medicine boxes is obtained using a 3D camera.

[0138] Step S402: Based on the point cloud map of the conveyor belt, obtain the normal vector of the conveyor belt and the coordinates of any reference point.

[0139] In step S402 above, point cloud preprocessing and plane extraction operations are performed on the point cloud map of the conveyor belt to obtain the normal vector of the conveyor belt and the coordinates of any reference point, and the plane equation of the conveyor belt shown in equation (8) can be established:

[0140] n x (xp x )+n y (yp y )+n z (zp z )=0 (8)

[0141] In equation (8) above, n = (n x ,n y ,n z ) t Let p be the normal vector of the conveyor belt. v =(p x ,p y ,p z ) t Let be the coordinates of any reference point on the conveyor belt.

[0142] In this embodiment, since the 3D camera is mounted directly above the conveyor belt, its normal vector is vertically downwards, while the normal vectors of the conveyor belt and the medicine box are vertically upwards. The z-direction component n in the conveyor belt normal vector... z Set to n z <0. The constraint equation for the medicine box above the conveyor belt can be expressed as:

[0143] n x (xp x )+ny (yp y )+n z (zp z )>0(9)

[0144] Step S403: Obtain the first point cloud image of the medicine box on the conveyor belt when it is in the first reference position using a 3D camera. Combine the normal vector of the conveyor belt and the coordinates of any reference point to determine the first coordinate point of the center point of the medicine box when it is in the first reference position, which is projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction.

[0145] In step S403 above, see [reference] Figure 4 The first point cloud image of the medicine box on the conveyor belt at the first reference position is obtained using a 3D camera. Point cloud preprocessing and plane extraction operations are performed on the first point cloud image to obtain the normal vector n1 of the medicine box at the first reference position. x1 ,n y1 ,n z1 ) t The coordinates of the center point p1 of the upper surface of the medicine box at the first reference position are p1 = (p x1 ,p y1 ,p z1 ) t Combining the above equation (9), we can establish the plane equation of the point cloud on the upper surface of the medicine box at the first reference position:

[0146] n x1 (xp x1 )+n y1 (yp y1 )+n z1 (zp z1 )=0 (10)

[0147] According to the normal vector of the conveyor belt n=(n x ,n y ,n z ) t and the coordinates p of any reference point on the conveyor belt v =(p x ,p y ,p z ) t The center point p1 of the upper surface of the medicine box when it is in the first reference position can be calculated as p1 = (p x1 ,p y1 ,p z1 ) t With projection direction n = (n x ,n y ,n z ) t The first coordinate point projected onto the conveyor belt plane is p'1 = (p' x1 ,p'y1 ,p' z1 ) t .

[0148] Step S404: Obtain the second point cloud image of the medicine box on the conveyor belt when it is in the second reference position using a 3D camera. Combine the normal vector of the conveyor belt and the coordinates of any reference point to determine the second coordinate point of the center point of the medicine box when it is in the second reference position, which is projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction.

[0149] In step S404 above, see [reference] Figure 5 A second point cloud image is acquired using a 3D camera when the medicine box on the conveyor belt is at the second reference position. Point cloud preprocessing and plane extraction operations are performed on the second point cloud image to obtain the normal vector n2 of the medicine box at the second reference position. x2 ,n y2 ,n z2 ) t The coordinates of the center point p2 of the upper surface of the medicine box at the second reference position are p2 = (p x2 ,p y2 ,p z2 ) t Combining equation (9) above, we can establish the plane equation of the point cloud on the upper surface of the medicine box at the second reference position:

[0150] n 2x (xp 2x )+n 2y (yp 2y )+n z1 (zp 1z )=0 (11)

[0151] According to the normal vector of the conveyor belt n=(n x ,n y ,n z ) t and the coordinates p of any reference point on the conveyor belt v =(p x ,p y ,p z ) t The center point p2 of the upper surface of the medicine box when it is in the second reference position can be calculated as p2 = (p x2 ,p y2 ,p z2 ) t With projection direction n = (n x ,n y ,n z ) t The second coordinate point projected onto the conveyor belt plane is p'2 = (p' x2 ,p' y2 ,p' z2) t .

[0152] Step S405: Based on the first coordinate point and the second coordinate point, obtain the conveyor belt motion direction vector.

[0153] In step S405 above, based on the first coordinate point and the second coordinate point, the conveyor belt motion direction vector v shown in equation (12) is obtained:

[0154] v = p'2 - p'1 = (p' x2 -p' x1 ,p' y2 -p' y1 ,p' z2 -p' z1 ) t (12)

[0155] Step S406: Based on the conveyor belt motion direction vector and the conveyor belt normal vector, obtain the conveyor belt lateral direction vector.

[0156] In step S406 above, since the conveyor belt's motion direction vector v and its lateral direction vector n are perpendicular (v⊥n), the lateral direction vector u can be calculated based on the conveyor belt's motion direction vector v and its normal vector n:

[0157] u=n×v (13)

[0158] Step S407: Based on the first coordinate point, determine the distance from the first coordinate point to the predetermined reference point, and combine the conveyor belt movement direction vector to determine the coordinates of the reference point in the 3D camera coordinate system.

[0159] In step S407 above, see [reference] Figure 3 and Figure 4 The distance to the projection point p'1 can be calculated based on the first coordinate point. Figure 3 The distance d from the reference point O on the left side of the conveyor belt is:

[0160] d = scale1 + 0.5w (14)

[0161] In the above formula (14), w is the width of the medicine box.

[0162] A conveyor belt coordinate system is established using the leftmost reference point O on the conveyor belt plane. The x-axis represents the direction of conveyor belt movement, the y-axis represents the lateral direction, and the z-axis represents the normal direction. The coordinates p of reference point O in the 3D camera coordinate system are... o It can be represented as:

[0163] p o =p'1-d*v (15)

[0164] Step S408: Based on the lateral direction vector of the conveyor belt, the motion direction vector of the conveyor belt, and the normal vector of the conveyor belt, obtain the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system. Combined with the coordinates of the reference point in the 3D camera coordinate system, obtain the coordinate system transformation matrix.

[0165] In step S408 above, based on the lateral direction vector of the conveyor belt, the motion direction vector of the conveyor belt, and the normal vector of the conveyor belt, the rotation matrix R between the 3D camera coordinate system and the conveyor belt coordinate system is expressed by the following equation (16):

[0166]

[0167] Combining the coordinates of the reference point in the 3D camera coordinate system, the coordinate transformation matrix T is expressed by the following equation (17):

[0168]

[0169] Therefore, for any point p in the 3D camera coordinate system, it can be transformed into coordinate p in the conveyor belt coordinate system using a coordinate transformation matrix. b :

[0170] p b =R inv (pp o (18)

[0171] In equation (18) above, p b =(x b ,y b ,z b ) t x b For relative to reference point p o The component of movement in the direction of movement v of the conveyor belt; R inv Let R be the inverse of the rotation matrix.

[0172] Any directional vector t in the 3D camera coordinate system can be converted into a vector in the conveyor belt coordinate system using a coordinate transformation matrix:

[0173] t b =R inv t (19)

[0174] In this embodiment, based on the matched point cloud map of each medicine box, by performing operations such as radius filtering, downsampling, and plane extraction on the matched point cloud map of each medicine box, the first positioning information of each medicine box in the 3D camera coordinate system (including the center point coordinates, the normal vector of the medicine box plane, the size of the medicine box, the coordinates of the four endpoints of the medicine box, and the pixel rectangle envelope of the medicine box) can be output. The first positioning information is obtained based on the 3D camera coordinate system. The corresponding first positioning information (the center point coordinates of each medicine box in the 3D camera coordinate system, the normal vector of the medicine box plane, and the coordinates of the four endpoints of the medicine box) can be transformed into the second positioning information (the center point coordinates of each medicine box in the data 3D camera coordinate system, the normal vector of the medicine box plane, and the coordinates of the four endpoints of the medicine box) through the above equations (18) and (19).

[0175] In step S5 above, see [reference] Figure 6 For each set of medicine boxes, based on the second positioning information of each medicine box in the set, the positioning rectangle information corresponding to the set of medicine boxes is calculated. The positioning rectangle information includes the coordinates p of the center point of the set of medicine boxes. c =(x c ,y c ,z c ) t The width W and the distances Wl and Wr from the adjacent medicine box sets are used to locate the rectangular frame information as the medicine delivery position. Figure 6 The dashed box shown is the boundary of the positioning rectangle of a set of medicine boxes.

[0176] In step S6 above, the specific process of obtaining the category of each medicine box for each set of medicine boxes may include the following steps:

[0177] Step S601: For each set of medicine boxes, based on the high-resolution 2D image of each medicine box in the set, extract the feature information of each medicine box using a pre-set visual feature extraction algorithm.

[0178] In step S601 above, the visual feature extraction algorithm can be a traditional feature extraction algorithm (such as ORB feature extraction algorithm) or a deep learning algorithm (such as ResNet-50 / 101, EfficientNet-B4 or CNN+Transformer hybrid model, etc.).

[0179] Step S602: For the feature information of each medicine box, calculate the cosine similarity with each template in the pre-established feature library, filter out the template corresponding to the maximum cosine similarity, and use the category corresponding to the template as the category of each medicine box.

[0180] In this embodiment, the drug feeding parameters may include the drug feeding position, the size of the medicine box, the number of medicine boxes, and the category of the medicine box. By grouping the medicine boxes and calculating the group positioning parameters (i.e., the second positioning information), the drug feeding position of the feed inlet of the flat-groove dispensing machine can be determined, which can be used as the lateral positioning reference of the robotic arm and can adapt to the random placement of different medicines. Since when all medicine boxes are grouped using the 2D image corresponding to each medicine box, the number of medicine boxes in each group can be obtained by segmenting and positioning the medicine boxes, which can be used for planning the number of grasping operations of the robotic arm. Based on the high-resolution 2D image of each medicine box in each group, high-resolution feature information of the medicine box can be obtained, thereby enabling the identification of the category of the medicine box, facilitating the sorting and verification of medicines, and avoiding the problem that traditional manual verification or single visual recognition methods have limited ability to distinguish complex textures and similar packaging. At the same time, if there is a QR code or barcode on the medicine box, the barcode or QR code can be scanned using a 2D camera, without the need for a separate dedicated scanning device, which can effectively simplify the structure and save costs. Meanwhile, based on the point cloud map of each medicine box in each group, the dimensions of each medicine box can be output, thereby determining the opening and closing degree of the robotic arm and anomaly detection. By fusing image data from 3D and 2D cameras, the entire process from medicine feeding position and medicine box recognition to grasping operation is automated, which can effectively improve the intelligence level and operational reliability of the flat-slot dispensing machine.

[0181] Example 2

[0182] Based on the same inventive concept, see [reference] Figure 7 This application also proposes a device for determining the feeding parameters of a flat-groove dispensing machine, comprising:

[0183] The image collection module 101 is used to acquire 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and to acquire high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera.

[0184] Grouping module 102 is used to group all medicine boxes based on the 2D image corresponding to each medicine box to obtain multiple sets of medicine boxes;

[0185] The data matching module 103 is used to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud map based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, for each medicine box set, to obtain the matched point cloud map of each medicine box.

[0186] The positioning information confirmation module 104 is used to obtain the first positioning information in camera coordinates based on the matched point cloud map corresponding to each medicine box in each medicine box set, and to obtain the second positioning information in conveyor belt coordinates by combining the pre-established coordinate system transformation matrix.

[0187] The medication placement location confirmation module 105 is used to calculate the positioning rectangle information corresponding to each set of medicine boxes based on the second positioning information of each medicine box in the set, and to obtain the medication placement location. The medication placement location includes the center point coordinates, width and distance to adjacent sets of medicine boxes.

[0188] The category recognition module 106 is used to extract feature information based on the high-resolution 2D image of each medicine box in each medicine box set, and match the feature information with a pre-established feature library to obtain the category of each medicine box.

[0189] The device for determining the feeding parameters of the flat-groove dispensing machine provided in this embodiment of the invention has a similar implementation principle and technical effect to that of Embodiment 1, and will not be described again here.

[0190] Example 3

[0191] Based on the same inventive concept, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for determining the feeding parameters of the flat-groove dispensing machine as described in Embodiment 1.

[0192] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to Embodiment 1 of the present invention.

[0193] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] Example 4

[0195] Based on the same inventive concept, this application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for determining the feeding parameters of the flat-groove dispensing machine as described in Embodiment 1.

[0196] Example 5

[0197] Based on the same inventive concept, this application proposes a computer program product containing instructions. When the computer program product is run on a computer device, the computer device executes the method for determining the feeding parameters of the flat-groove dispensing machine in Embodiment 1.

[0198] The principles by which the above-mentioned devices, clients, media, and related equipment in this embodiment of the invention solve the problem are similar to those of the aforementioned methods. Therefore, their implementation can refer to the implementation of the aforementioned methods, and repeated details will not be repeated.

[0199] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0200] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A method for determining the feeding parameters of a flat-groove dispensing machine, characterized in that, include: 2D images and point cloud maps of each medicine box on the conveyor belt are obtained using a 3D camera, and high-resolution 2D images of each medicine box on the conveyor belt are obtained using a 2D camera. Based on the 2D image corresponding to each medicine box, all medicine boxes are grouped to obtain multiple sets of medicine boxes; For each set of medicine boxes, based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, the high-resolution 2D image of each medicine box in the set is matched with the point cloud map to obtain the matched point cloud map of each medicine box. This includes: for each set of medicine boxes, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the first pixel rotated rectangle of each medicine box is obtained. For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the set, the second pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the second pixel rotated rectangle of each medicine box is obtained. For each set of medicine boxes, for the second pixel rotated rectangle corresponding to each medicine box in the set, it is matched one by one with the first pixel rotated rectangles of all medicine boxes in the set to obtain the first pixel rotated rectangle that matches the second pixel rotated rectangle corresponding to each medicine box. This results in the matched point cloud image for each medicine box, including: for each medicine box in each set, the intersection-union ratio (IUU) between the second pixel rotated rectangle corresponding to each medicine box and the first pixel rotated rectangles of all medicine boxes is calculated using the following formula: ； in, This represents the i-th second-pixel rotated rectangle; This represents the first pixel of the j-th rotated rectangle; The intersection-union ratio (IU) is represented between the i-th second-pixel rotated rectangle and the j-th first-pixel rotated rectangle. Based on all IU ratios, the first-pixel rotated rectangle corresponding to the largest IU ratio is obtained, which is used as the first-pixel rotated rectangle that matches the second-pixel rotated rectangle corresponding to each medicine box. The point cloud map corresponding to the matched first-pixel rotated rectangle is used as the matched point cloud map for each medicine box. For each set of medicine boxes, based on the matched point cloud map corresponding to each medicine box in the set, the first positioning information in the camera coordinates is obtained, and combined with the pre-established coordinate system transformation matrix, the second positioning information in the conveyor belt coordinates is obtained. For each set of medicine boxes, based on the second positioning information of each medicine box in the set, the positioning rectangle information corresponding to the set of medicine boxes is calculated to obtain the medicine delivery position. The medicine delivery position includes the center point coordinates, width, and distance to adjacent sets of medicine boxes. For each set of medicine boxes, feature information is extracted based on the high-resolution 2D image of each medicine box in the set. The feature information is then matched with a pre-established feature library to obtain the category of each medicine box.

2. The method for determining the feeding parameters of the flat-groove dispensing machine according to claim 1, characterized in that, For each set of medicine boxes, based on the rotation matrix and translation vector between the 3D and 2D cameras, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the set, the first pixel coordinates of the four endpoints and the center point of each medicine box are obtained, resulting in a first pixel rotation rectangle, including: For each set of medicine boxes, the coordinates of the center point and the four endpoints in the 3D camera coordinate system are determined based on the point cloud map corresponding to each medicine box in the set. Combined with the rotation matrix and translation vector between the 3D camera and the 2D camera, the coordinates of the center point and the four endpoints of each medicine box in the 2D camera coordinate system are obtained. Based on the intrinsic parameters of the 2D camera, the coordinates of the center point of each medicine box in the 2D camera coordinate system, and the coordinates of the four endpoints, the first pixel coordinates of the center point and the first pixel coordinates of the four endpoints of each medicine box are obtained, and the first pixel rotated rectangle of each medicine box is obtained.

3. The method for determining the feeding parameters of the flat-groove dispensing machine according to claim 1, characterized in that, The process involves grouping all the medicine boxes based on the 2D image corresponding to each medicine box, resulting in multiple sets of medicine boxes, including: Based on the 2D image corresponding to each medicine box, a pixel rectangle envelope of each medicine box is obtained, and the parameters of each pixel rectangle envelope are obtained, including the horizontal coordinate of the upper left corner, the width, the height, and the horizontal coordinate of the lower right corner; Sort all the pixel rectangle envelopes in ascending order of their top-left horizontal coordinates to obtain a pixel rectangle envelope sequence. Traverse all pixel rectangles in the pixel rectangle envelope sequence and determine whether the lower right corner x-coordinate of any pixel rectangle envelope is greater than or equal to the upper left corner x-coordinate of the next pixel rectangle envelope. If so, merge the two pixel rectangular envelopes into the same set of medicine boxes, and update the range of the merged rectangle. If not, the two pixel rectangular envelopes cannot be merged into the same set of medicine boxes, and the judgment process is re-executed for the next pixel rectangular envelope.

4. The method for determining the feeding parameters of the flat-groove dispensing machine according to claim 1, characterized in that, For each set of medicine boxes, feature information is extracted based on a high-resolution 2D image of each medicine box in the set. This feature information is then matched against a pre-established feature library to determine the category of each medicine box, including: For each set of medicine boxes, feature information of each medicine box is extracted using a pre-set visual feature extraction algorithm based on the high-resolution 2D image of each medicine box in the set. For each medicine box, the cosine similarity with each template in the pre-established feature library is calculated based on the feature information. The template with the highest cosine similarity is then selected, and the category corresponding to the template is used as the category of each medicine box.

5. The method for determining the feeding parameters of the flat-groove dispensing machine according to claim 1, characterized in that, The method for constructing the coordinate system transformation matrix includes: A point cloud image of the conveyor belt without medicine boxes is obtained using a 3D camera. Based on the point cloud map of the conveyor belt, the normal vector of the conveyor belt and the coordinates of any reference point are obtained; The first point cloud image of the medicine box on the conveyor belt when it is in the first reference position is obtained by a 3D camera. Combined with the normal vector of the conveyor belt and the coordinates of any reference point, the center point of the medicine box when it is in the first reference position is determined to be the first coordinate point projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction. The second point cloud map of the medicine box on the conveyor belt when it is in the second reference position is obtained by a 3D camera. Combined with the normal vector of the conveyor belt and the coordinates of any reference point, the center point of the medicine box when it is in the second reference position is determined to be the second coordinate point projected onto the plane of the conveyor belt with the normal vector of the conveyor belt as the projection direction. Based on the first coordinate point and the second coordinate point, the conveyor belt movement direction vector is obtained; Based on the conveyor belt's motion direction vector and the conveyor belt's normal vector, the conveyor belt's lateral direction vector is obtained; Based on the first coordinate point, the distance from the first coordinate point to a predetermined reference point is determined, and combined with the conveyor belt movement direction vector, the coordinates of the reference point in the 3D camera coordinate system are determined; Based on the lateral direction vector of the conveyor belt, the motion direction vector of the conveyor belt, and the normal vector of the conveyor belt, the rotation matrix between the 3D camera coordinate system and the conveyor belt coordinate system is obtained. Combined with the coordinates of the reference point in the 3D camera coordinate system, the coordinate system transformation matrix is ​​obtained.

6. The method for determining the feeding parameters of the flat-groove dispensing machine according to claim 1, characterized in that, The process of acquiring 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and acquiring high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera, includes: Image segmentation is performed on the original 2D images of all medicine boxes on the conveyor belt obtained by the 3D camera to obtain the first segmentation mask for each medicine box, and the 2D image and point cloud map corresponding to each medicine box are obtained based on the first segmentation mask. Image segmentation is performed on the original high-resolution 2D images of all medicine boxes on the conveyor belt acquired by the 2D camera to obtain a second segmentation mask for each medicine box, and a high-resolution 2D image for each medicine box is obtained based on the second segmentation mask.

7. A device for determining the feeding parameters of a flat-groove dispensing machine, characterized in that, include: The image acquisition module is used to acquire 2D images and point cloud maps of each medicine box on the conveyor belt using a 3D camera, and to acquire high-resolution 2D images of each medicine box on the conveyor belt using a 2D camera. The grouping module is used to group all the medicine boxes based on the 2D image corresponding to each medicine box, so as to obtain multiple sets of medicine boxes; The data matching module is used to match the high-resolution 2D image of each medicine box in the medicine box set with the point cloud map based on the pre-calibrated rotation matrix and translation vector between the 3D camera and the 2D camera, as well as the intrinsic parameter information of the 2D camera, for each medicine box set, to obtain the matched point cloud map of each medicine box. This includes: for each medicine box set, based on the rotation matrix and translation vector between the 3D camera and the 2D camera, the intrinsic parameter information of the 2D camera, and the point cloud map corresponding to each medicine box in the medicine box set, obtaining the first pixel coordinates of the four endpoints and the center point of each medicine box, and obtaining the first pixel rotated rectangle of each medicine box; For each set of medicine boxes, based on the high-resolution 2D image corresponding to each medicine box in the set, the second pixel coordinates of the four endpoints and the center point of each medicine box are obtained, and the second pixel rotated rectangle of each medicine box is obtained. For each set of medicine boxes, for the second pixel rotated rectangle corresponding to each medicine box in the set, it is matched one by one with the first pixel rotated rectangles of all medicine boxes in the set to obtain the first pixel rotated rectangle that matches the second pixel rotated rectangle corresponding to each medicine box. This results in the matched point cloud image for each medicine box, including: for each medicine box in each set, the intersection-union ratio (IUU) between the second pixel rotated rectangle corresponding to each medicine box and the first pixel rotated rectangles of all medicine boxes is calculated using the following formula: ； in, This represents the i-th second-pixel rotated rectangle; This represents the first pixel of the j-th rotated rectangle; The intersection-union ratio (IU) is represented between the i-th second-pixel rotated rectangle and the j-th first-pixel rotated rectangle. Based on all IU ratios, the first-pixel rotated rectangle corresponding to the largest IU ratio is obtained, which is used as the first-pixel rotated rectangle that matches the second-pixel rotated rectangle corresponding to each medicine box. The point cloud map corresponding to the matched first-pixel rotated rectangle is used as the matched point cloud map for each medicine box. The positioning information confirmation module is used to obtain the first positioning information in camera coordinates based on the matched point cloud map corresponding to each medicine box in each medicine box set, and to obtain the second positioning information in conveyor belt coordinates by combining the pre-established coordinate system transformation matrix. The medication placement location confirmation module is used to calculate the positioning rectangle information corresponding to each set of medicine boxes based on the second positioning information of each medicine box in the set, and to obtain the medication placement location. The medication placement location includes the center point coordinates, width, and distance to adjacent sets of medicine boxes corresponding to the set. The category recognition module is used to extract feature information based on the high-resolution 2D image of each medicine box in each medicine box set, and match the feature information with a pre-established feature library to obtain the category of each medicine box.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the method for determining the feeding parameters of the flat-groove dispensing machine as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for determining the feeding parameters of the flat-groove dispensing machine as described in any one of claims 1-6.

10. A computer program product containing instructions, which, when run on a computer device, causes the computer device to perform the method for determining the feeding parameters of a flat-groove dispensing machine as described in any one of claims 1-6.

Images