Tray grid pose acquisition method, device, equipment and program product

Abstract

The application relates to the field of automation, in particular to a tray grid pose acquisition method, device, equipment and program product. The method comprises the following steps: acquiring a first tray image of a to-be-operated tray through a camera, and determining camera parameters of the camera; detecting first feature point information in the first tray image; matching the first feature point information with second feature point information of a preset tray template, estimating a homography matrix of the first feature point information and the second feature point information, and generating the second feature point information according to size data and calibration data of a second tray image collected in advance; and determining a grid pose of the to-be-operated tray according to the homography matrix and the camera parameters. Compared with a CAD modeling method, the method greatly reduces the adaptation cycle and maintenance cost, compared with a deep learning method, the method can greatly reduce the data preparation cost, and the generalization ability of the pose determined based on the homography matrix is less likely to be limited.

Description

Technical Field

[0001] This application relates to the field of automation, and in particular to methods, apparatus, equipment and program products for obtaining the pose of a material tray grid. Background Technology

[0002] In industrial scenarios such as automated assembly, sorting, and loading / unloading, customized trays are widely used to orderly carry and position parts. In order to guide robots to accurately pick up or place parts from the tray slots, it is necessary to obtain the pose of each slot in three-dimensional space in real time and accurately.

[0003] Currently, methods for obtaining grid pose mainly include CAD modeling / offline programming and keypoint detection or direct pose estimation based on deep learning. However, modeling each type of material tray individually consumes a significant amount of engineering time and cost, has a long adaptation cycle, and high maintenance costs. Deep learning-based methods, on the other hand, have high data preparation costs and limited generalization ability. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, device and program product for obtaining the pose of a material tray grid, in order to solve the problems in the prior art that require a lot of engineering time and cost for modeling the material tray separately, have a long adaptation cycle and high maintenance cost, and have high data preparation cost and limited generalization ability based on deep learning methods.

[0005] A first aspect of this application provides a method for obtaining the pose of a material tray grid, the method comprising: The camera captures an image of the first material tray to be operated using a camera, and the camera parameters of the camera are determined. Detect the first feature point information in the first material tray image; The first feature point information is matched with the second feature point information of the preset material tray template to estimate the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second material tray image. The grid pose of the tray to be operated is determined based on the homography matrix and the camera parameters.

[0006] In conjunction with the first aspect, in a first possible implementation of the first aspect, before matching the first feature point information with the second feature point information of the corresponding material tray template, the method further includes: The second material tray image of the calibration tray is acquired using a camera; Receive calibration data and dimension data, wherein the calibration data includes the position information of at least four key geometric points in the second tray image; Based on the size data and the calibration data, a tray template corresponding to the second tray image is determined, and the tray template includes the second feature point information in the calibration tray.

[0007] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the second feature point information includes a template point set and a lattice-vertex index table; Based on the size data and the calibration data, the tray template corresponding to the second tray image is determined, including: Based on the dimensional data, determine the ideal location and category of the key geometric points included in the template plane; Based on the ideal location and category of the key geometric points, and in conjunction with the calibration data in the second tray image, determine the set of template points included in the second tray image; Based on the correspondence between the grid and the template point set, determine the grid-vertex index table.

[0008] In conjunction with the first aspect, in a third possible implementation of the first aspect, the calibration data includes: At least one of the following: the corner position in the second tray image and the intersection position of the grid dividing lines in the second tray image; The dimensional data includes: The length and width of the grid, the distance between the grid and the edge of the tray, and the spacing between the grids.

[0009] In conjunction with the first aspect, in the fourth possible implementation of the first aspect, the first feature point information is matched with the second feature point information of a preset material tray template to estimate the homography matrix of the first feature point information and the second feature point information, including: Enumerate the first subset of points that meet the predetermined requirements from the set of template points of the material tray template; Multiple candidate subsets are determined by enumerating geometric feature points from the first feature point information; Based on the matching of the first subset of points and the multiple candidate subsets, the homography matrix of the first feature point information and the second feature point information is estimated.

[0010] In conjunction with the fourth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, matching is performed based on the first subset of points and the multiple candidate subsets to estimate the homography matrix of the first feature point information and the second feature point information, including: Robust estimation is performed on multiple initial corresponding point pairs composed of the first subset of points and the multiple candidate subsets to obtain the initial homography matrix; Starting with the initial homography matrix, iterative matching and optimization under category constraints are performed according to the category of the feature points in the first feature point information until convergence is obtained to obtain the final homography matrix.

[0011] In conjunction with the first aspect, in the sixth possible implementation of the first aspect, determining the grid pose of the tray to be operated based on the homography matrix and the camera parameters includes: The homography matrix is ​​normalized according to the camera parameters to obtain a normalized matrix; The rotation matrix in the pose is obtained by scaling the first and second column vectors of the normalized matrix using a size factor. The translation matrix in the pose is determined based on the first column vector and the second column vector.

[0012] A second aspect of this application provides a tray grid pose acquisition device, the device comprising: An image acquisition unit is used to acquire a first image of the material tray to be operated via a camera and to determine the camera parameters of the camera. The information detection unit is used to detect the first feature point information in the first material tray image; The matching unit is used to match the first feature point information with the second feature point information of the preset tray template, and estimate the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second tray image. The pose determination unit is used to determine the grid pose of the tray to be operated based on the homography matrix and the camera parameters.

[0013] A third aspect of this application provides a tray grid pose acquisition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the tray grid pose acquisition device performs the method described in any of the first aspects.

[0014] A fourth aspect of this application provides a computer program product that, when run on a computer, causes the computer to execute the methods described in the first aspect or its various implementations.

[0015] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0016] A sixth aspect of this application provides a chip for implementing the methods in the various implementations of the first aspect described above. Specifically, the chip includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the methods as described in the first aspect or its various implementations.

[0017] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires a first tray image of the tray to be operated by a camera, determines the camera parameters, extracts the first feature point information in the first tray image, matches the first feature point information with the second feature point information in the preset tray template, and estimates the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second tray image. Based on the determined homography matrix and camera parameters, the grid pose of the tray to be operated can be determined. Since this method only requires a simple data calibration of the second tray image of the tray to be operated, i.e., generating the second feature point information for matching calculation, it greatly reduces the adaptation cycle and maintenance cost compared with CAD modeling. Compared with deep learning methods, the method in this application embodiment can greatly reduce the data preparation cost, and the generalization ability of the pose determined based on the homography matrix is ​​less likely to be limited. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an implementation scenario of a method for obtaining the pose of a material tray grid provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation process of a method for obtaining the pose of a material tray grid provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the ideal location of key geometric points on a template plane based on dimensional data, provided in an embodiment of this application. Figure 4 This is a schematic diagram illustrating the implementation process of a method for determining a homography matrix provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the implementation process of determining a material tray template according to an embodiment of this application; Figure 6 This is a schematic diagram of a tray grid pose acquisition device provided in an embodiment of this application; Figure 7 This is a schematic diagram of a tray grid pose acquisition device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0022] In industrial automation processes such as assembly, sorting, and loading / unloading, customized trays are often used for the orderly carrying and positioning of parts. To enable robots to accurately pick up and place parts in each compartment of the tray, it is necessary to obtain the position and orientation of each compartment in three-dimensional space in real time and accurately.

[0023] Currently, technologies for obtaining grid pose mainly fall into two categories: one is based on CAD modeling and offline programming, and the other is based on keypoint detection or end-to-end pose estimation methods using deep learning. However, methods based on CAD modeling and offline programming require modeling and programming for each material tray of different specifications, resulting in long implementation cycles, high engineering costs, and heavy subsequent maintenance and adaptation burdens. On the other hand, methods based on keypoint detection or end-to-end pose estimation rely on a large amount of labeled data to train the model, which suffers from high data preparation costs, limited model generalization ability, and insufficient adaptability to new types or long-tailed material trays.

[0024] To address the aforementioned problems, this application proposes a method for obtaining the pose of a material tray grid, such as... Figure 1The diagram illustrates an implementation scenario for this method. This scenario includes a camera 1, a tray grid pose acquisition device 2, a calibration tray 3, and multiple trays 4 to be operated. The tray grid pose acquisition device 2 can pre-capture a second tray image, including the calibration tray 3, via the camera 1, and receive calibration data based on the second tray image. This data includes the positions of at least four corner points in the second tray image, such as tray boundary corner points, intersections of tray grid dividing lines, and grid corner points, determining the pixel positions of these corner points in the second tray image. Based on the tray's size parameters, including grid length and width, side margins, and row / column spacing, the ideal positions of the tray's key geometric points in the template plane coordinate system are determined. By establishing the correspondence between pixel positions and ideal positions, the second feature point information, including the pixel position of each corner point of the tray in the second tray image, can be determined. When the tray to be operated 4 and the calibration tray are of the same specifications, the first tray image of the tray to be operated 4 can be acquired, the first feature point information in the first tray image can be detected, the homography matrix can be determined according to the matching relationship between the first feature point information and the second feature point information, and the grid pose of the tray to be operated 4 can be obtained according to the homography matrix and the camera parameters.

[0025] Figure 2 A schematic diagram illustrating the implementation process of a method for obtaining the pose of a material tray grid, provided in an embodiment of this application, is described in detail below: In S201, an image of the first material tray to be operated is acquired by a camera, and the camera parameters of the camera are determined.

[0026] In this embodiment, the tray to be operated is a tray for picking up or placing parts. In order to accurately and reliably pick up or place parts, it is necessary to accurately obtain the position and orientation of the grid.

[0027] In the embodiments of this application, when the camera is in a fixed position, the camera parameters may include camera intrinsic parameters, so that the pose of the current material to be operated and the pose of each cell in the material to be operated can be identified by combining the camera intrinsic parameters with the first feature point information in the identified first material tray image.

[0028] When a camera can change its pose, including altering external parameters such as rotation angle and translation distance, camera parameters can include both internal and external parameters. The first tray image captured under different poses is transformed using the camera's external parameters, and the first feature point information in the transformed first tray image is matched and calculated with the second feature point information.

[0029] In S202, the first feature point information in the first material tray image is detected.

[0030] The first feature point information in this embodiment may include the position information of the feature points in the first material tray image, or it may also include the type information of the feature points. The type information of the feature points may include corner points, intersection points, etc.

[0031] Edge detection and line segment detection can be used to transform raw pixels into "lines" with clear geometric meaning. Based on the obtained edges and line segments, corner detection algorithms (such as the Harris corner detection algorithm) can be run to extract corners in the image, or the intersections of horizontal and vertical line segments can be calculated to extract intersection points. After extracting corners and intersection points, morphological analysis and / or connected component analysis can be used to optimize and validate the extracted corners and intersection points, improving data quality and robustness.

[0032] In S203, the first feature point information is matched with the second feature point information of the preset material tray template to estimate the homography matrix of the first feature point information and the second feature point information.

[0033] The second feature point information is generated based on the size data and the calibration data of the pre-acquired second tray image.

[0034] The calibration data in this embodiment may include data used by a user or worker to calibrate positions based on at least one of the corner points in the second tray image and the intersection points of the grid dividing lines in the second tray image. Based on the calibration data, the pixel coordinates of the corner points or intersection points in the second tray image can be determined.

[0035] In general, to facilitate subsequent calculations and simplify calibration operations, the number of calibrated corner points or intersection points can be four, or more. Among the calibrated corner points or intersection points, at least four points must satisfy rectangular constraints or convex constraints.

[0036] The dimensional data in this embodiment can be used to generate a template point set. The template point set includes the positions and categories of key geometric points in the calibration tray that serves as the template. In one possible implementation, the dimensional data may include the length and width of the grid, the edge distance between the grid and the edge of the tray, and the spacing between the grids. Based on the above dimensional data, the coordinate positions of the key geometric points of the tray in the template plane coordinate system can be effectively determined.

[0037] like Figure 3As shown, the calibration tray is assumed to be a 2x2 square tray with 4 grids. The dimensions include: grid length 30mm, grid width 30mm, and the distance from the outermost grid edge to the tray border is 5mm on the top, bottom, left, and right. The spacing between grids is the horizontal and vertical distance (i.e., row and column spacing) between the center points of adjacent grids. Since the grids are closely arranged, the grid spacing is equal to the length and width of the grid itself.

[0038] A template plane coordinate system is constructed with the top-left corner of the physical outer edge of the calibration tray as the origin (0, 0). The positive directions of the x-axis and y-axis are right and down, respectively. The theoretical coordinates of the top-left vertex P1 of the first cell (top-left cell) are obtained by moving the vertex one left margin (5mm) to the right and then one top margin (5mm) down from the origin. That is, the coordinates of point P1 are (5, 5), and its type is a corner point. The coordinates and types of other key geometric points can be determined in the same way. For example, the coordinates of the bottom-right vertex P9 of the fourth cell (bottom-right cell) are (65, 65), and its type is a corner point.

[0039] In this embodiment, the process of estimating the homography matrix by matching the first feature point information and the second feature point information can be as follows: Figure 4 As shown, it includes: In S401, a first subset of points that meet predetermined requirements is enumerated from the template point set of the material tray template.

[0040] The template point set of the material tray template includes the pixel coordinates of multiple key geometric points in the second material tray image. A first subset of points that meets predetermined requirements can be selected. The predetermined requirements can be that at least four points satisfy rectangular or convex constraints. This ensures that the selected at least four points can form a convex quadrilateral (such as a rectangle, parallelogram, etc.), thus providing good and stable constraints for planar perspective transformations (homography).

[0041] In possible implementations, the first subset can be directly based on the key geometric points determined by the calibration data, or based on a subset of the key geometric points in the first subset.

[0042] In S402, multiple candidate subsets are determined by enumerating geometric feature points from the first feature point information.

[0043] From the "first feature point information" (including the set of all candidate key geometric points in the first material tray image) extracted from the real-time acquired first material tray image, multiple candidate subsets consisting of at least four points satisfying predetermined requirements (such as rectangular or convex constraints) are also enumerated. Since the first material tray image may contain noise, missing points, or false detections, the enumeration process can be carried out by random sampling or combination attempts to cover various possible point combinations.

[0044] In S403, the homography matrix of the first feature point information and the second feature point information is estimated by matching the first sub-set and the multiple candidate subsets.

[0045] When matching the first subset of points with multiple candidate subsets, the first subset of points can be robustly estimated by constructing initial corresponding points with the candidate subsets to obtain the initial homography matrix. Starting from the initial homography matrix, iterative matching and optimization under category constraints are performed according to the category of the feature points in the first feature point information until convergence is obtained to obtain the final homography matrix.

[0046] During the robust estimation phase, the pre-established tray template contains the coordinates and category information of all key geometric points (including corner points and intersections of dividing lines) on the ideal template plane. The first subset of points is a set of points enumerated from this template point set that satisfy predetermined geometric requirements (such as forming a convex quadrilateral), representing a reference local structure on the template.

[0047] Accordingly, feature points such as corner points and intersection points can be extracted from the first material tray image, and multiple sets of points that can form similar geometric structures can be enumerated. Since the image may contain noise, occlusion, or false detections, there will be multiple candidate subsets.

[0048] The first subset of points is matched with the first group of candidate subsets to perform matching calculations and determine an initial homography matrix. After the homography matrices of multiple candidate subsets are determined, the template point set can be projected onto the first tray image using these homography matrices for matching optimization. The matching score can be determined by the average matching distance of the matching results corresponding to each homography matrix; the shorter the average matching distance, the higher the matching score. Based on the ranking of matching scores, homography matrices with higher matching scores are selected for matching optimization.

[0049] A bidirectional nearest neighbor search is performed between the projected template points (template projection points) and all detected feature points (image points) in the first tray image (i.e., image points find the nearest template projection point, and template projection points also find the nearest image point). A class consistency constraint is applied to find more reliable matching point pairs. If mismatches exist for key geometric points, other homography matrices can be selected based on the matching score. If no mismatches for key geometric points occur after bidirectional nearest neighbor matching and class consistency matching, then the homography matrix can be determined as the homography matrix matching the tray template and the first tray image.

[0050] In S204, the grid pose of the tray to be operated is determined based on the homography matrix and the camera parameters.

[0051] When determining lattice pose based on homography matrix and camera parameters, the rotation matrix can be obtained by decomposing the homography matrix and camera parameters, and the lattice pose can then be determined using the rotation matrix. Alternatively, the PnP (Perspective n-Point) algorithm can also be used to determine the lattice pose.

[0052] The rotation matrix is ​​obtained by decomposing the homography matrix and camera parameters. When determining the grid pose using the rotation matrix, the homography matrix can be normalized according to the camera parameters to obtain a normalized matrix. The first and second column vectors of the normalized matrix are scaled using a size factor to obtain the rotation matrix in the pose. The translation matrix in the pose is determined based on the first and second column vectors.

[0053] Normalizing the homography matrix H based on the camera parameters (intrinsic parameter matrix K) yields a normalized matrix. Essentially, this involves transforming H to remove intrinsic parameters, resulting in a matrix that only reflects rotation and translation in the three-dimensional plane. Mathematically, this can be represented as: .in, This represents the normalized matrix.

[0054] Scaling the first two column vectors of the normalized matrix (i.e., the first and second column vectors of the normalized matrix) yields the first two columns of the rotation matrix R describing the orientation of the tray plane, which are expressed as follows: .in, , Represents the first two columns of the rotation matrix R. As a scaling factor, it can be expressed as .

[0055] pass and The cross product is used to calculate the third column of the rotation matrix. The rotation matrix R is obtained as follows: Here, orth() represents orthogonalization.

[0056] The scale factor λ and the third column of the normalized matrix can directly determine the translation vector.

[0057] In this embodiment of the application, before feature matching, the tray template corresponding to the current tray to be operated can be predetermined. That is, the operator only needs to perform a simple calibration operation to calibrate the key geometric points of the second tray image of the calibrated tray and input the size data to quickly and effectively generate the tray template. Specifically, it can be as follows: Figure 5 As shown, it includes: In S501, the second tray image of the calibration tray is acquired by a camera.

[0058] The calibration tray can be selected as the calibration tray when the tray to be operated cannot match the tray template. An image of the second tray, including the calibration tray, is then captured by aiming the camera at the calibration tray.

[0059] In S502, calibration data and dimension data are received, wherein the calibration data includes the position information of at least four key geometric points in the second tray image.

[0060] The calibration data in this embodiment may include data used by a user or worker to calibrate positions based on at least one of the corner points in the second tray image and the intersection points of the grid dividing lines in the second tray image. Based on the calibration data, the pixel coordinates of the corner points or intersection points in the second tray image can be determined.

[0061] In general, to facilitate subsequent calculations and simplify calibration operations, the number of calibrated corner points or intersection points can be four, or more. Among the calibrated corner points or intersection points, at least four points must satisfy rectangular constraints or convex constraints.

[0062] The dimensional data in this embodiment can be used to generate a template point set. The template point set includes the positions and categories of key geometric points in the calibration tray that serves as the template. In one possible implementation, the dimensional data may include the length and width of the grid, the edge distance between the grid and the edge of the tray, and the spacing between the grids. Based on the above dimensional data, the coordinate positions of the key geometric points of the tray in the template plane coordinate system can be effectively determined.

[0063] In S503, the tray template corresponding to the second tray image is determined based on the size data and the calibration data.

[0064] The material tray template includes the second feature point information in the calibration material tray.

[0065] In possible implementations, the second feature point information may include a template point set and a grid-vertex index table. The positions and categories of key geometric points included in the template plane, such as corner points and intersections, can be determined based on the size data. Based on the calibration data in the second tray image, matching key geometric points in the template plane can be determined. Based on the positional relationships between the key geometric points in the template plane, the pixel coordinates of the key geometric points in the second tray image can be determined, thus defining the template point set included in the second tray image. Here, key geometric points include positions such as corner points and intersections determined based on detected line segments. The template point set includes the key geometric points in the second tray image and their corresponding pixel coordinates.

[0066] Once the template point set is determined, the correspondence between each cell and vertex can be further established, and a cell-vertex index table can be constructed. This facilitates the rapid retrieval of relevant vertices for the cells to be operated on. The second feature information includes the constructed cell-vertex index table and the template point set.

[0067] Based on the vertices (i1, i2, i3, i4) corresponding to each grid, the grid center point and grid normal vector can be calculated according to the three-dimensional coordinates of the corresponding template point set. The actual grid pose can be calculated according to the rotation matrix and translation matrix.

[0068] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0069] Figure 6 This application provides a schematic diagram of a tray grid pose acquisition device, which includes: The image acquisition unit 601 is used to acquire a first image of the material tray to be operated through a camera and to determine the camera parameters of the camera.

[0070] The information detection unit 602 is used to detect the first feature point information in the first material tray image.

[0071] The matching unit 603 is used to match the first feature point information with the second feature point information of the preset tray template, and estimate the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second tray image.

[0072] The pose determination unit 604 is used to determine the grid pose of the tray to be operated based on the homography matrix and the camera parameters.

[0073] Figure 6 The tray grid pose acquisition device shown is related to... Figure 2 The method for obtaining the pose of the tray grid shown corresponds to this.

[0074] Figure 7 This is a schematic diagram of a tray grid pose acquisition device provided in an embodiment of this application. Figure 7As shown, the tray grid pose acquisition device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a tray grid pose acquisition program. When the processor 70 executes the computer program 72, it implements the steps in the above-described tray grid pose acquisition method embodiments. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments.

[0075] For example, the computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the tray grid pose acquisition device 7.

[0076] The tray grid pose acquisition device 7 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The tray grid pose acquisition device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 This is merely an example of a tray grid pose acquisition device 7 and does not constitute a limitation on the tray grid pose acquisition device 7. It may include more or fewer components than shown, or combine certain components, or different components. For example, the tray grid pose acquisition device may also include input / output devices, network access devices, buses, etc.

[0077] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0078] The memory 71 can be an internal storage unit of the tray grid pose acquisition device 7, such as a hard disk or memory of the tray grid pose acquisition device 7. The memory 71 can also be an external storage device of the tray grid pose acquisition device 7, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the tray grid pose acquisition device 7. Furthermore, the memory 71 can include both internal storage units and external storage devices of the tray grid pose acquisition device 7. The memory 71 is used to store the computer program and other programs and data required by the tray grid pose acquisition device. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0080] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0081] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the intended application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each intended application, but such implementation should not be considered beyond the scope of this application.

[0082] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0086] In addition, this application also provides a computer program product that, when run on a computer, causes the computer to execute the methods in the above-described implementations.

[0087] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for obtaining the pose of a material tray grid, characterized in that, The method includes: The camera captures an image of the first material tray to be operated using a camera, and the camera parameters of the camera are determined. Detect the first feature point information in the first material tray image; The first feature point information is matched with the second feature point information of the preset material tray template to estimate the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second material tray image. The grid pose of the tray to be operated is determined based on the homography matrix and the camera parameters.

2. The method according to claim 1, characterized in that, Before matching the first feature point information with the second feature point information of the corresponding material tray template, the method further includes: The second material tray image of the calibration tray is acquired using a camera; Receive calibration data and dimension data, wherein the calibration data includes the position information of at least four key geometric points in the second tray image; Based on the size data and the calibration data, a tray template corresponding to the second tray image is determined, and the tray template includes the second feature point information in the calibration tray.

3. The method according to claim 2, characterized in that, The second feature point information includes a template point set and a lattice-vertex index table; Based on the size data and the calibration data, the tray template corresponding to the second tray image is determined, including: Based on the dimensional data, determine the ideal location and category of the key geometric points included in the template plane; Based on the ideal location and category of the key geometric points, and in conjunction with the calibration data in the second tray image, determine the set of template points included in the second tray image; Based on the correspondence between the grid and the template point set, determine the grid-vertex index table.

4. The method according to claim 1, characterized in that, The calibration data includes: At least one of the following: the corner position in the second tray image and the intersection position of the grid dividing lines in the second tray image; The dimensional data includes: The length and width of the grid, the distance between the grid and the edge of the tray, and the spacing between the grids.

5. The method according to claim 1, characterized in that, Matching the first feature point information with the second feature point information of a preset material tray template, and estimating the homography matrix of the first feature point information and the second feature point information, including: Enumerate the first subset of points that meet the predetermined requirements from the set of template points of the material tray template; Multiple candidate subsets are determined by enumerating geometric feature points from the first feature point information; Based on the matching of the first subset of points and the multiple candidate subsets, the homography matrix of the first feature point information and the second feature point information is estimated.

6. The method according to claim 5, characterized in that, Matching the first subset of points with the multiple candidate subsets, and estimating the homography matrix of the first feature point information and the second feature point information, including: Robust estimation is performed on multiple initial corresponding point pairs composed of the first subset of points and the multiple candidate subsets to obtain the initial homography matrix; Starting with the initial homography matrix, iterative matching and optimization under category constraints are performed according to the category of the feature points in the first feature point information until convergence is obtained to obtain the final homography matrix.

7. The method according to claim 1, characterized in that, Determining the grid pose of the tray to be operated based on the homography matrix and the camera parameters includes: The homography matrix is ​​normalized according to the camera parameters to obtain a normalized matrix; The rotation matrix in the pose is obtained by scaling the first and second column vectors of the normalized matrix using a size factor. The translation matrix in the pose is determined based on the first column vector and the second column vector.

8. A device for acquiring the position and orientation of a material tray grid, characterized in that, The device includes: An image acquisition unit is used to acquire a first image of the material tray to be operated via a camera and to determine the camera parameters of the camera. The information detection unit is used to detect the first feature point information in the first material tray image; The matching unit is used to match the first feature point information with the second feature point information of the preset tray template, and estimate the homography matrix of the first feature point information and the second feature point information. The second feature point information is generated based on the size data and the calibration data of the pre-acquired second tray image. The pose determination unit is used to determine the grid pose of the tray to be operated based on the homography matrix and the camera parameters.

9. A tray grid pose acquisition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the tray grid pose acquisition device to implement the method as described in any one of claims 1-7.

10. A computer program product comprising computer program instructions, characterized in that, When the computer program is run, the method as described in any one of claims 1-7 is performed.

Images