Object grabbing method and system based on LightGlue algorithm

Abstract

The invention provides an object grabbing method and system based on a LightGlue algorithm, and relates to the technical field of computer vision and robots, and the method comprises the steps: obtaining an RGB image, a depth image and a reference image; performing feature point matching on the RGB image and the reference image through a LightGlue algorithm to obtain a matching point pair set; judging whether the number of matching point pairs in the matching point pair set is smaller than a preset number of matching point pairs; if yes, re-acquiring the image; otherwise, according to the matching point pair set, calculating a homography matrix through an RANSAC algorithm; based on the homography matrix, mapping the bounding box of the reference image into the RGB image to obtain an object bounding box; smoothing the bounding box of the object; according to the depth image and the smoothed object bounding box, the six-dimensional attitude of the to-be-grabbed object is determined through a geometric algorithm; and according to the six-dimensional posture, the robotic arm is controlled to grab the to-be-grabbed object.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and robotics, and particularly to an object grasping method and system based on the LightGlue algorithm. Background Art

[0002] In robot grasping tasks, object recognition and six-dimensional pose estimation are core technologies. With the wide application of intelligent service robots in fields such as healthcare, aging society, and personalized home services, how to achieve high-precision and real-time object recognition and grasping in complex and dynamic environments has become a major challenge in robotics.

[0003] However, existing feature matching methods, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features), although they can perform well under stable conditions by extracting local feature points of images and performing matching, their robustness is poor and the matching accuracy drops significantly when facing complex scenarios such as object occlusion, illumination changes, and large viewing angle differences.

[0004] In addition, existing six-dimensional pose estimation methods (such as PoseCNN) can provide relatively accurate pose estimation in static environments, but due to their complex network structures and high computational requirements, they are easily affected by noise and environmental changes in dynamic environments and are difficult to meet the requirements of real-time applications. Summary of the Invention

[0005] In view of the above deficiencies of the prior art, the purpose of the embodiments of the present invention is to provide an object grasping method based on the LightGlue algorithm, which can solve the technical problems that existing feature matching methods have poor robustness and significantly reduced matching accuracy when facing complex scenarios such as object occlusion, illumination changes, and large viewing angle differences, and existing six-dimensional pose estimation methods are easily affected by noise and environmental changes in dynamic environments and are difficult to meet the requirements of real-time applications.

[0006] In the first aspect of the embodiments of the present invention, an object grasping method based on the LightGlue algorithm is proposed, including:

[0007] S1: Obtain the RGB image, depth image in the scene, and the reference image of the object to be grasped;

[0008] S2: Perform feature point matching on the RGB image and the reference image through the LightGlue algorithm to obtain a set of matching point pairs;

[0009] S3: Determine whether the number of matching point pairs in the set of matching point pairs is less than the preset number of matching point pairs; if so, return to step S1; otherwise, proceed to step S4;

[0010] S4: Calculate the homography matrix according to the set of matching point pairs by using the RANSAC algorithm;

[0011] S5: Based on the homography matrix, map the bounding box of the reference image to the RGB image to obtain the object bounding box;

[0012] S6: Combine the Kalman filter algorithm and the exponential moving average algorithm to smooth the object bounding box;

[0013] S7: According to the depth image and the smoothed object bounding box, determine the six-dimensional pose of the object to be grasped by using a geometric algorithm;

[0014] S8: According to the six-dimensional pose, control the robotic arm to grasp the object to be grasped.

[0015] In the second aspect of the embodiments of the present invention, an object grasping system based on the LightGlue algorithm is proposed, including: a processor and a memory;

[0016] The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, the steps of the object grasping method based on the LightGlue algorithm as described in the first aspect are implemented.

[0017] In the third aspect of the embodiments of the present invention, a readable storage medium is proposed. A program or instructions are stored on the readable storage medium. When the program or instructions are executed by a processor, the steps of the object grasping method based on the LightGlue algorithm as described in the first aspect are implemented.

[0018] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0019] In the embodiments of the present invention, the LightGlue algorithm is used to perform feature point matching on the RGB image and the reference image to obtain a set of matching point pairs, and it is judged whether the number of matching point pairs in the set of matching point pairs is less than the preset number of matching point pairs. In the face of complex scenarios such as object occlusion, illumination change, and large viewing angle difference, it has strong robustness and significantly improved matching accuracy. By using a geometric algorithm to determine the six-dimensional pose of the object to be grasped according to the depth image and the smoothed object bounding box, it is not easily affected by noise and environmental changes in a dynamic environment and can well meet the requirements of real-time applications. Description of the Drawings

[0020] The accompanying drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference signs denote the same components. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 is a schematic flowchart of an object grasping method based on the LightGlue algorithm provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic structural diagram of an object grasping system based on the LightGlue algorithm provided by an embodiment of the present invention. Detailed Description of the Invention

[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0024] The object grasping method based on the LightGlue algorithm provided by the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, through specific embodiments and their application scenarios.

[0025] Refer to the attached drawings of the specification Figure 1 which shows a schematic flowchart of an object grasping method based on the LightGlue algorithm provided by an embodiment of the present invention.

[0026] The embodiments of the present invention provide an object grasping method based on the LightGlue algorithm, which may include the following steps:

[0027] S1: Obtain the RGB image, depth image, and reference image of the object to be grasped in the scene.

[0028] Optionally, by using the Jetson Nano main control system, start the Intel RealSense D435 camera to obtain the RGB image and depth image in the scene.

[0029] Furthermore, by calibrating the Intel RealSense D435 camera, the internal and external parameters of the camera are obtained. Using the depth information of each pixel in the depth image and the internal parameters of the RGB image, coordinate transformation and projection are performed to map the 3D coordinates in the depth image to the RGB image, thus achieving precise alignment of the RGB image and the depth image, and obtaining the aligned RGB image and depth image.

[0030] It should be noted that the Jetson Nano main control system is a small and efficient embedded computing platform launched by NVIDIA, designed specifically for AI applications and robot development. It has powerful GPU computing capabilities and low power consumption characteristics, and supports a variety of deep learning, computer vision, and artificial intelligence tasks. The Jetson Nano main control system is widely used in fields such as intelligent robots, Internet of Things devices, and automation control, and is an ideal choice for embedded developers and robot enthusiasts.

[0031] It should be noted that the Intel RealSense D435 camera is a depth camera launched by Intel, which can simultaneously capture RGB (color) images and depth images. Based on infrared structured light technology and stereo vision technology, this camera can provide high-precision depth perception and is widely used in scenarios such as robots, augmented reality (AR), virtual reality (VR), and object recognition. The D435 camera has a wide field of view and high depth accuracy, making it suitable for 3D perception and scene modeling in complex environments.

[0032] In the embodiment of the present invention, by combining the Jetson Nano main control system and the Intel RealSense D435 camera, and using the calibration of the camera and the precise alignment of the depth image and the RGB image, the present invention realizes high-precision object recognition and grasping. The depth image provides precise 3D perception capabilities, while the RGB image provides support for the visual features of the object. The powerful computing capabilities of the Jetson Nano ensure that the system can process image data in real time and perform deep learning and vision tasks, thereby enhancing the adaptability and object positioning accuracy of the robot in complex environments and improving the accuracy and success rate of grasping. This technical solution effectively improves the performance of the robot in dynamic and irregular environments.

[0033] In a possible implementation manner, after S1 and before S2, it further includes steps S1A to S1D:

[0034] S1A: Encode the RGB image using the Webp encoding format.

[0035] It should be noted that the WebP encoding format is an image compression format developed by Google, aiming to provide higher compression efficiency than JPEG and PNG while maintaining good image quality. WebP supports both lossy and lossless compression, which can significantly reduce the size of image files, especially suitable for optimizing the loading speed of web images.

[0036] In the embodiment of the present invention, the RGB image is encoded using the WebP encoding format. The WebP format is developed by Google, which can provide higher compression efficiency than JPEG and PNG, and supports both lossy and lossless compression. It is suitable for reducing the size of image files while maintaining good image quality, especially suitable for optimizing the loading speed of web images.

[0037] S1B: Use the Png encoding format to encode the depth image.

[0038] It should be noted that the PNG encoding format (Portable Network Graphics format) is a lossless compressed image format, widely used for storing images that require a transparent background. The PNG format can retain all details of the image and is suitable for high-quality image storage, especially in images that require high contrast or transparency.

[0039] In the embodiment of the present invention, the depth image is encoded using the PNG encoding format. PNG is a lossless compressed format that can retain all details of the image and is suitable for storing high-quality images, especially in scenarios that require a transparent background or high contrast.

[0040] S1C: Use the TCP protocol to transmit the encoded RGB image and depth image to the host computer.

[0041] It should be noted that the TCP protocol (Transmission Control Protocol) is a connection-oriented communication protocol that ensures the reliable transmission of data in a computer network. The TCP protocol guarantees the integrity and order of data through packet serialization, acknowledgment mechanism, and retransmission mechanism, and is widely used in services such as file transfer, email, and web browsing on the Internet.

[0042] Specifically, the Jetson Nano main control system uses a dual-thread parallel design (one thread is specifically responsible for encoding the RGB image, and the other thread is responsible for data transmission) to transmit the aligned RGB image and depth image to the host computer at a rate of 10 frames per second through the TCP protocol. To optimize the transmission efficiency, the RGB image is encoded using the Webp encoding format (set to lossless mode), while the depth image is encoded using the Png encoding format.

[0043] In the embodiments of the present invention, the encoded RGB image and depth image are transmitted to the host computer through the TCP protocol. The TCP protocol ensures the reliability of data transmission and guarantees that the image data can be completely transmitted to the host computer. The Jetson Nano main control system is designed with dual-thread parallelism. One thread is responsible for RGB image encoding, and the other thread is responsible for data transmission, transmitting 10 frames of images per second.

[0044] S1D: When the host computer receives the encoded RGB image and depth image, it decodes the encoded RGB image and depth image to obtain the RGB image and depth image.

[0045] S2: Use the LightGlue algorithm to perform feature point matching on the RGB image and the reference image to obtain a set of matching point pairs.

[0046] It should be noted that the LightGlue algorithm is a deep learning-based feature matching algorithm used to efficiently match local feature points between two images. Compared with traditional feature matching algorithms (such as SIFT or SURF), LightGlue enhances the accuracy and robustness of feature matching between images by utilizing self-attention and cross-attention mechanisms. It can still maintain a high matching effect in complex environments such as illumination changes, perspective differences, or occlusions, and is widely used in computer vision tasks such as object recognition, image registration, and visual localization.

[0047] Specifically, the LightGlue algorithm efficiently matches local feature points in the image through deep learning technology. First, key points are extracted from the image. These feature points are significant and unique in the image, which helps in object recognition. The algorithm adopts self-attention and cross-attention mechanisms to improve the matching accuracy from the global context and the feature point correlation between images respectively. At the same time, the matching process is optimized through a soft partial assignment matrix, and false matches are reduced according to the feature similarity and matching scores, further improving the accuracy. Through these steps, a set of matching point pairs is obtained.

[0048] In the embodiments of the present invention, by using the LightGlue algorithm and combining deep learning technology with self-attention and cross-attention mechanisms, the accuracy and robustness of feature matching between images are significantly improved. This algorithm can still maintain high-precision matching in complex environments such as illumination changes, perspective differences, or occlusions, reduce false matches, and optimize feature point associations. The matching accuracy is further improved through a soft partial assignment matrix, ensuring stability and adaptability in a dynamic environment, thus providing high-quality data support for subsequent object localization, pose estimation, and robotic arm control, and improving the success rate and efficiency of the object grasping task.

[0049] In a possible implementation, S2 specifically includes sub-steps S201 to S206:

[0050] S201: Detect M and N key points and their descriptors for the RGB image (A) and the reference image (B) respectively, denoted as where, is the position coordinate of the i-th key point in the RGB image is the descriptor of the i-th key point in the RGB image, M represents the total number of key points in the RGB image, is the position coordinate of the i-th key point in the reference image is the descriptor of the i-th key point in the reference image, N represents the total number of key points in the reference image.

[0051] S202: Initialize these descriptors as "state vectors" in LightGlue, that is, let These state vectors will be iteratively updated in the multi-layer Transformer to fuse global and local information to optimize the feature representation required for matching.

[0052] S203: In the multi-layer structure, first perform the self-attention mechanism on each image. Taking the RGB image as an example, its calculation method can be expressed as:

[0053]

[0054] where, a ij represents the self-attention score between the i-th key point and the j-th key point in the RGB image, q i and k j represent the state vectors of the i-th key point and the j-th key point in the RGB image respectively and are obtained through different linear transformations: q i represents the query vector of the i-th key point in the RGB image, W Q represents the weight matrix of the query vector, k j represents the key vector of the j-th key point in the RGB image, W K represents the weight matrix of the key vector, T represents the transpose operation, represents the rotational encoding, which is used to model the relative position relationship between key points.

[0055] Furthermore, similar operations are performed on the reference image.

[0056] S204: Execute the Cross-Attention mechanism. In the cross-attention, calculate the similarity between the $i$-th key point of the RGB image and all key points of the reference image in the following form:

[0057]

[0058] where and both represent the cross-attention scores between the $i$-th key point in the RGB image and the $j$-th key point in the reference image. represents the key vector of the $i$-th key point in the RGB image, represents the key vector of the $j$-th key point in the reference image:

[0059]

[0060] After calculating the attention scores, use Softmax normalization: represents the context vector of the $i$-th key point in the RGB image. Finally, use MLP for state update:

[0061] Furthermore, for the reference image, also execute the cross-attention mechanism.

[0062] It should be noted that MLP (Multilayer Perceptron, Chinese name: Multilayer Perceptron) is a feedforward neural network, usually used to process non-linear problems. MLP consists of multiple layers, including at least three layers: the input layer, the hidden layer, and the output layer. The neurons (or nodes) in each layer are connected to all neurons in the previous layer, and information is transmitted through weights. The input layer receives data and passes it to the hidden layer. The hidden layer performs non-linear transformation through the activation function, and then passes the processed data to the output layer to finally give the prediction result.

[0063] S205: Calculate the matching similarity (Similarity) and the matchability probability (Matchability Probability). In the final output part, calculate the matching similarity $S$ ij and the match probability $P$ ij , and the formulas are as follows:

[0064] Calculate the matching similarity $S$ ij : Calculate the matching score and Calculate the match probability $P$ ij :

[0065] Where: S ij represents the matching similarity between key points and the matching similarity between. and represents the matching score of key points, which is used to judge whether key points are likely to match. P ij represents the final matching probability, which is used to select matching point pairs.

[0066] S206: Combine the matching score and similarity to calculate the final matching probability:

[0067]

[0068] If P ij is the maximum in a certain row and column and Pi j > τ (τ is the matching threshold), then (i, j) is considered the final matching point. If P ij ≤ τ, then the point is considered non-matchable and is excluded. Finally, multiple matching point pairs between the RGB image and the reference image are output, and a set of matching point pairs is constructed.

[0069] S3: Determine whether the number of matching point pairs in the set of matching point pairs is less than the preset number of matching point pairs. If so, return to step S1. Otherwise, go to step S4.

[0070] It should be noted that those skilled in the art can set the size of the preset number of matching point pairs according to actual needs, and the present invention does not limit this here.

[0071] Optionally, the preset number of matching point pairs is 40.

[0072] Specifically, when the number of matching point pairs in the set of matching point pairs is less than the preset number of matching point pairs, it is determined that the object to be grasped exists in the scene. When the number of matching point pairs in the set of matching point pairs is greater than or equal to the preset number of matching point pairs, it is determined that the object to be grasped does not exist in the scene.

[0073] In the embodiment of the present invention, by judging whether the number of matching point pairs in the set of matching point pairs reaches the preset number, the accuracy of object recognition and the reliability of grasping can be ensured. When the number of matching point pairs is less than the preset value, the system will re-collect images to avoid misrecognition. When the number of matching point pairs meets the requirements, the system considers that the object exists and enters the subsequent steps. This mechanism ensures that in a complex environment, object recognition and grasping operations are based on stable and reliable data, improving the accuracy, robustness and efficiency of the system.

[0074] S4: Calculate the homography matrix according to the set of matching point pairs by using the RANSAC algorithm.

[0075] It should be noted that the RANSAC algorithm (Random Sample Consensus algorithm) is an iterative method used to estimate the parameters of a mathematical model from data and improve the estimation accuracy by eliminating outliers. It is particularly suitable for situations where the data contains noise and outliers. By randomly selecting samples, estimating the model, evaluating inliers, and optimizing the model, it gradually finds the model that best fits the data.

[0076] Specifically, use the cv2.findHomography function built into the computer vision library OpenCV, and specify the use of the RANSAC algorithm by setting the parameter cv2.RANSAC. At the same time, set the reprojection error threshold to obtain the final homography matrix.

[0077] It should be noted that those skilled in the art can set the size of the reprojection error threshold according to actual needs, and the present invention does not make any limitations here.

[0078] It should be noted that OpenCV (Open Source Computer Vision Library) is a widely used computer vision and image processing library that provides a rich set of tools and functions to help developers implement tasks such as image analysis, computer vision, and machine learning. It supports multiple programming languages such as C++, Python, Java, etc., and is widely used in industrial, academic, and personal projects.

[0079] It should be noted that the cv2.findHomography function is a function in the OpenCV library used to calculate the homography matrix between two images. This matrix describes the perspective transformation relationship from one image to another and is commonly used in tasks such as image registration, image stitching, and feature matching.

[0080] It should be noted that cv2.RANSAC is one of the functional parameters in OpenCV for performing the RANSAC algorithm. When using the cv2.findHomography function, cv2.RANSAC is passed as a parameter, indicating that when calculating the homography matrix, the RANSAC algorithm is used to eliminate outliers and select the best model, thereby improving the matching accuracy.

[0081] In the embodiments of the present invention, by using the RANSAC algorithm, it is possible to effectively calculate the homography matrix from the matching point pairs. The RANSAC algorithm ensures that in the presence of noise and outliers, it can accurately estimate the perspective transformation relationship between images. This improves the accuracy and robustness of feature matching and ensures the reliability of subsequent steps.

[0082] In a possible implementation manner, S4 specifically includes sub-steps S401 to S406:

[0083] S401: Randomly select a preset number of matching point pairs from the set of matching point pairs.

[0084] It should be noted that those skilled in the art can set the size of the preset number according to actual needs, and the present invention does not limit it here.

[0085] Optionally, randomly select 4 matching point pairs from the set of matching point pairs, that is, N = 4.

[0086] S402: Calculate the candidate homography matrix according to each matching point pair by the least squares method:

[0087]

[0088] Among them, H s represents the candidate homography matrix, arg min represents minimization, H represents the homography matrix, N represents the number of randomly selected matching point pairs, src_pts i represents the coordinates of the matching point corresponding to the reference image in the i-th matching point pair, T represents the transpose operation, dst_pts i represents the coordinates of the matching point corresponding to the RGB image in the i-th matching point pair, |||| 2 represents the square of the Euclidean distance, (H · [src_pts i , 1] T ) z represents the z-axis component after the homography matrix is transformed by homogeneous coordinates.

[0089] It should be noted that the least squares method is a mathematical optimization method used to fit a data model by minimizing the sum of the squares of errors. It finds the optimal model parameters by solving for the parameter values that minimize the sum of the squares of the differences (usually residuals) between the observed values and the model predictions. The least squares method is widely used in fields such as regression analysis, curve fitting, and solving linear systems, and is a commonly used optimization technique in data analysis and machine learning.

[0090] S403: Calculate the reprojection error of each matching point pair according to the candidate homography matrix:

[0091]

[0092] Among them, ε i represents the reprojection error of the i-th matching point pair, ||||2 represents the Euclidean norm, (H s · [src_pts i , 1] T ) z represents the z-axis component after the candidate homography matrix is transformed by homogeneous coordinates.

[0093] S404: Determine whether the reprojection error of each pair of matching points is less than the reprojection error threshold. If so, determine the pair of matching points as inliers. Otherwise, determine the pair of matching points as outliers.

[0094] S405: Repeat steps S401 to S404 to obtain multiple candidate homography matrices.

[0095] S406: Select the candidate homography matrix with the largest number of inliers as the homography matrix.

[0096] In the embodiments of the present invention, by calculating the homography matrix, the matching accuracy and robustness of the object recognition and grasping system can be effectively improved. By randomly selecting pairs of matching points, using the least squares method to calculate candidate matrices, removing outliers, and selecting the matrix with the most inliers, the system can optimize the matching results in a complex environment and reduce the impact of incorrect matches. Setting the reprojection error threshold and evaluating multiple candidate matrices makes the algorithm highly adaptable, ensuring the accuracy of the final homography matrix, thereby providing reliable data support for subsequent object pose estimation and grasping tasks and enhancing the stability and grasping success rate of the system.

[0097] S5: Based on the homography matrix, map the bounding box of the reference image to the RGB image to obtain the object bounding box.

[0098] In the embodiments of the present invention, by mapping the bounding box of the reference image to the RGB image, precise alignment between images can be achieved. This enables the positions of the object in different images to correspond accurately, providing reliable data support for subsequent object recognition, pose estimation, and grasping tasks, and ensuring that the system can perform tasks efficiently and precisely in a complex environment.

[0099] In a possible implementation manner, S5 specifically includes sub-steps S501 to S504:

[0100] S501: Obtain the coordinates of multiple corner points of the bounding box of the reference image.

[0101] It should be noted that the bounding box of the reference image includes four corner points: the upper left corner, the lower left corner, the upper right corner, and the lower right corner. The coordinates of the four corner points are: (0,0), (0, h'-1), (w'-1, 0), and (w'-1, h'-1), where w' and h' respectively represent the width and height of the reference image.

[0102] S502: Based on the homography matrix, map each corner point coordinate to the RGB image to obtain the homogeneous coordinates of the RGB image:

[0103]

[0104] Among them, Denote the homogeneous coordinates of the point (p, q) in the RGB image, where p and q represent the abscissa and ordinate of the corner point mapped to the RGB image respectively, r represents the third component in the homogeneous coordinates of the RGB image, and H represents the homography matrix. Denote the homogeneous coordinates of the point (x, y) in the reference image, where x and y represent the abscissa and ordinate of the corner point of the bounding box of the reference image respectively, and h bd Denote the element in the b-th row and d-th column of the homography matrix, where b = 1, 2, 3 and d = 1, 2, 3.

[0105] S503: Normalize the homogeneous coordinates of the RGB image to determine the coordinates of each corner point of the object bounding box:

[0106]

[0107] Among them, x′ represents the abscissa of the corner point of the object bounding box, and y′ represents the ordinate of the corner point of the object bounding box.

[0108] S504: Map the bounding box of the reference image to the RGB image according to the coordinates of each corner point of the object bounding box to obtain the object bounding box.

[0109] In the embodiments of the present invention, by accurately mapping the bounding box of the reference image to the RGB image and using the homography matrix, the accurate alignment of the object position and size is ensured. This process can adapt to perspective changes, reduce mis-matching, ensure the accurate alignment of the object bounding box between different images, and provide reliable object positioning information. Through homogeneous coordinates and normalization processing, the accuracy and stability of the object grasping task are further improved, providing stable data support for the accurate grasping of the robotic arm.

[0110] S6: Smooth the object bounding box by combining the Kalman filter algorithm and the exponential moving average algorithm.

[0111] It should be noted that the Kalman filter algorithm is a recursive filtering algorithm designed to estimate the true state of the system from noisy data. It combines prediction and observation information, uses state equations and observation equations, and gradually optimizes the estimation results. The Kalman filter is widely used in fields such as navigation, positioning, and signal processing, and can effectively handle uncertainties and noises in dynamic systems, providing accurate state estimations.

[0112] It should be noted that the exponential moving average algorithm (EMA) is a weighted average method used to smooth data sequences and reduce the influence of short-term fluctuations. It assigns decreasing weights to historical data points, making the latest data point have a greater impact on the result. EMA is commonly used in financial data analysis, time series prediction, etc., and can better capture trends and respond to data changes.

[0113] In the embodiments of the present invention, by smoothing the object bounding box, noise and short-term fluctuations can be effectively reduced, and the positioning accuracy can be improved. The Kalman filter optimizes the state estimation, enhancing the robustness of the dynamic system, while the EMA reduces instability by smoothing historical data, ensuring the stability of the object bounding box and providing more reliable data support for the object grasping task.

[0114] In a possible implementation manner, S6 specifically includes sub-steps S601 to S606:

[0115] S601: Define a ten-dimensional state vector of the Kalman filter algorithm:

[0116] X t =[x′1,y′1,x'2,y'2,x'3,y'3,x'4,y'4,v x' ,v y' T

[0117] Wherein, X t represents the ten-dimensional state vector at time t, x′ c represents the abscissa of the c-th corner point of the object bounding box, y′ c represents the ordinate of the c-th corner point of the object bounding box, c = 1, 2, 3, 4, v x ' represents the velocity component of the object bounding box in the abscissa direction, v y ' represents the velocity component of the object bounding box in the ordinate direction, and T represents the transpose operation.

[0118] S602: Construct a state transition equation according to the ten-dimensional state vector:

[0119] X t|t-1 =AX t-1 +w t

[0120]

[0121] Wherein, X t|t-1 represents the state vector at time t predicted based on the state vector at time t - 1, represents the state transition matrix, represents the set of real numbers, X t-1 represents the ten-dimensional state vector at time t - 1, and w t represents the process noise at time t.

[0122] It should be noted that in the state transition matrix, the first 8 columns of each row correspond to the corner coordinates, and the 9th and 10th columns correspond to the velocity components v x' and v y' ​The update of each corner point coordinate is \(x''_1 = x'_1+v\) c +v x' and \(y''_1\) c = \(y'_1\) c +v y' The velocity components \(v_x\) x' and \(v_y\) y' remain constant during the prediction phase (the last two rows of the matrix), but are indirectly adjusted through the Kalman gain during the update phase. For example, the first row is \(x''_1 = x'_1+v_x\) x' The second row is \(y''_1 = y'_1+v_y\) y' The last two rows are \(v_x\) x' = \(v_x\) x' , \(v_y\) y' = \(v_y\) y' .

[0123] It should be noted that the covariance matrix of \(w\) t is: \(Q = 10\cdot I\) -4 ·I 10 , where \(Q\) represents the covariance matrix of \(w\) t , and \(I\) 10 represents a 10×10 identity matrix, which is used to characterize the uncertainty of the prediction model.

[0124] S603: Define the observation vector of the Kalman filter algorithm:

[0125] \(Z\) t = \([x'_1,y'_1,x'_2,y'_2,x'_3,y'_3,x'_4,y'_4]\) T

[0126] where \(Z\) t represents the observation vector at time \(t\).

[0127] S604: Construct the observation equation based on the ten-dimensional state vector and the observation vector:

[0128] \(Z\) t = \(H'X\) t + \(V\) t

[0129]

[0130] where represents the observation matrix, and \(V\) t represents the observation noise at time \(t\).

[0131] It should be noted that in the observation matrix, only one corner point coordinate is selected as the observation value for each row. For example, the first row observes \(x'_1\). The second row observes \(y'_1\). And so on.

[0132] It should be noted that \(V\) tThe covariance matrix is: Q' = 10 -2 ·I8, where Q' represents the covariance matrix of V t and I8 represents an 8×8 identity matrix used to characterize measurement noise.

[0133] S605: According to the state transition equation and the observation equation, through the Kalman filter algorithm, output the estimated values of the corner coordinates of the object bounding box.

[0134] S606: Perform time series smoothing on the estimated values through the exponential moving average algorithm:

[0135] EMA t = α·filter_dst t +(1 - α)·EMA t-1

[0136] where EMA t represents the smoothed estimated value at time t, α represents the smoothing factor, and filter_dst t represents the estimated value output by the Kalman filter algorithm at time t, and EMA t-1 represents the smoothed estimated value at time t - 1.

[0137] It should be noted that those skilled in the art can set the size of the smoothing factor according to actual needs, and the present invention does not make any limitations here.

[0138] Optionally, the smoothing factor is 0.3.

[0139] In the embodiments of the present invention, by smoothing the object bounding box, the present invention can provide accurate and stable object positioning. The Kalman filter estimates the object position and velocity through the state vector, effectively coping with noise and uncertainty in the dynamic environment to ensure the accuracy of the object bounding box. EMA further smooths the estimated value output by the Kalman filter, reducing short-term fluctuations and measurement errors, and improving the stability of the system. By flexibly adjusting the smoothing factor, the system can adapt to different environments, ultimately improving the accuracy and success rate of object grasping, and significantly enhancing the robustness of the system in complex environments.

[0140] S7: According to the depth image and the smoothed object bounding box, determine the six-dimensional pose of the object to be grasped through a geometric algorithm.

[0141] It should be noted that geometric algorithms are a class of algorithms used to process geometric shapes and spatial relationships, and are widely applied in fields such as computer graphics, computer vision, and robotics. These algorithms solve problems related to geometric elements such as points, lines, planes, curves, angles, distances, etc. through mathematical models and geometric principles. Common geometric algorithms include calculating the distance, angle, intersection point, area, volume of an object, as well as tasks such as shape matching, object recognition, and path planning in images. Geometric algorithms play a crucial role in applications such as 3D modeling, navigation, and map drawing.

[0142] In the embodiment of the present invention, by using geometric algorithms, the six-dimensional pose of the object to be grasped can be accurately calculated. This method is based on geometric principles and mathematical models, ensuring the accuracy of object positioning. Especially in complex environments, it provides efficient processing of spatial relationships, enhancing the accuracy and robustness of object recognition, pose estimation, and subsequent grasping operations.

[0143] In a possible implementation manner, S7 specifically includes sub-steps S701 to S709:

[0144] S701: According to the smoothed object bounding box, extract multiple key points of the object to be grasped from the depth image. The key points include the object center point, the center point of the right border, and the center point of the lower border.

[0145] S702: Taking the object center point as the origin, calculate the coordinate difference between the center point of the right border and the object center point to obtain the first direction vector:

[0146]

[0147] Among them, represents the first direction vector, O' x represents the center point of the right border, O' o represents the object center point.

[0148] S703: Taking the object center point as the origin, calculate the coordinate difference between the center point of the lower border and the object center point to obtain the second direction vector:

[0149]

[0150] Among them, represents the second direction vector, O' y represents the center point of the lower border.

[0151] S704: Perform a cross product operation on the first direction vector and the second direction vector to obtain the normal vector:

[0152]

[0153] Among them, Indicates the normal vector, z X , z Y and z Z respectively represent the components of the normal vector in the X-axis direction, Y-axis direction, and Z-axis direction. X, Y, and Z respectively represent the X-axis, Y-axis, and Z-axis of the world coordinate system.

[0154] S705: According to the second direction vector and the normal vector, the first direction vector is corrected through cross product operation to obtain the third direction vector:

[0155]

[0156] where represents the third direction vector.

[0157] It should be noted that since may not be completely orthogonal to the other two vectors and , therefore, we need to ensure its complete orthogonality with and and by recalculating the direction of

[0158] S706: Normalize the second direction vector, the normal vector, and the third direction vector into unit vectors:

[0159]

[0160] where represents the first unit vector obtained by normalizing the third direction vector, i X , i Y and i Z respectively represent the components of the first unit vector in the X-axis direction, Y-axis direction, and Z-axis direction, represents the second unit vector obtained by normalizing the second direction vector, j X , j Y and j Z respectively represent the components of the second unit vector in the X-axis direction, Y-axis direction, and Z-axis direction, represents the third unit vector obtained by normalizing the normal vector, k X , k Y and k Z respectively represent the components of the third unit vector in the X-axis direction, Y-axis direction, and Z-axis direction.

[0161] Specifically, the second direction vector, the normal vector, and the third direction vector are normalized to unit vectors, which are respectively represented as the X-axis, Y-axis, and Z-axis directions to construct an object coordinate system.

[0162] S707: Construct a rotation matrix based on the unit vectors:

[0163]

[0164] where R represents the rotation matrix.

[0165] S708: Orthogonalize the rotation matrix through the singular value decomposition algorithm.

[0166] It should be noted that the singular value decomposition algorithm (SVD) is a matrix decomposition method that decomposes a matrix into the product of three matrices. Specifically, for an arbitrary matrix, SVD decomposes it into three matrices: an orthogonal matrix, a diagonal matrix, and another orthogonal matrix. Through this decomposition, SVD can extract the principal components of the matrix and is widely used in tasks such as data dimensionality reduction, feature extraction, and noise removal. SVD has important application values in fields such as signal processing, image compression, and recommendation systems.

[0167] In a possible implementation, S708 specifically includes sub-steps S7081 and S7085:

[0168] S7081: Perform singular value decomposition on the rotation matrix to obtain the left singular vector matrix, the diagonal matrix, and the right singular vector matrix:

[0169] R = UΣV T

[0170] where R represents the rotation matrix, U represents the left singular vector matrix, Σ represents the diagonal matrix, V represents the right singular vector matrix, and T represents the transpose operation.

[0171] S7082: Obtain the orthogonalization matrix through multiplication based on the left singular vector matrix and the right singular vector matrix:

[0172] R ortho = UV T

[0173] where R ortho represents the orthogonalization matrix.

[0174] S7083: Calculate the determinant of the orthogonalization matrix:

[0175] det(R ortho ) = det(U)·det(V T )

[0176] Among them, det() represents the determinant operation.

[0177] S7084: Adjust the left singular vector matrix according to the determinant of the orthogonalization matrix.

[0178] Specifically, since both the left singular vector matrix and the right singular vector matrix are orthogonal matrices, their determinant values are ±1. When det(R ortho ) = 1, adjust the sign of the last column of the left singular vector matrix:

[0179]

[0180] Among them, Uadjusted represents the adjusted left singular vector matrix.

[0181] S7085: Recalculate the orthogonal matrix according to the adjusted left singular vector matrix to complete the orthogonalization process of the rotation matrix.

[0182] Specifically, recalculate the orthogonal matrix according to the adjusted left singular vector matrix, and finally ensure that the determinant of the orthogonalization matrix is +1 to complete the orthogonalization process of the rotation matrix:

[0183] R' ortho = U adjusted V T

[0184] Among them, R′ ortho represents the orthogonal matrix after the orthogonalization process.

[0185] In the embodiment of the present invention, the SVD decomposition decomposes the rotation matrix into a left singular vector matrix, a diagonal matrix, and a right singular vector matrix. By adjusting the matrix determinant, numerical errors are eliminated, ensuring the accuracy of the rotation matrix. Finally, it is ensured that the determinant of the rotation matrix is +1, improving the accuracy and robustness of the algorithm, ensuring the accuracy and stability of the object pose estimation, and providing reliable data support for the grasping task.

[0186] S709: Convert the rotation matrix after the orthogonalization process into a quaternion to determine the six-dimensional pose of the object to be grasped.

[0187] In a possible implementation manner, S709 is specifically:

[0188] According to the following formula, convert the rotation matrix after the orthogonalization process into a quaternion to determine the six-dimensional pose of the object to be grasped:

[0189]

[0190]

[0191] Among them, represents the first unit vector after orthogonalization, i' X , i' Y and i' Z respectively represent the components of the first unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. represents the second unit vector after orthogonalization, j' X , j' Y and j' Z respectively represent the components of the second unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. represents the third unit vector after orthogonalization, k' X , k' Y and k' Z respectively represent the components of the third unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. R′ ortho represents the orthogonal matrix after orthogonalization, (q w , q x , q y , q z ) represents a quaternion.

[0192] Furthermore, if q w is close to zero (i.e., 1 + i' X + j' Y + k' Z is close to zero), then other paths need to be selected to avoid numerical instability. For example, if when i′ X is the largest, then the quaternion is calculated through the following formula:

[0193]

[0194] For the case where j' Y or k' Z is the largest, a similar processing method can also be adopted. This multi-path selection strategy ensures that when q w is close to zero, the quaternion can still be calculated stably, thus avoiding problems caused by numerical instability.

[0195] In the embodiments of the present invention, a geometric algorithm and depth image data are used to accurately calculate the six - dimensional pose of an object, ensuring an accurate estimation of the object's position and rotation. By extracting key points and calculating direction vectors and normal vectors, the orthogonal directions of the object coordinate system are corrected and standardized. Then, the rotation matrix is orthonormalized through singular value decomposition (SVD) and converted into a quaternion, finally obtaining a stable representation of the object's pose. These steps effectively eliminate errors, improve the accuracy and robustness of pose estimation, and ensure that the object grasping task in a dynamic and complex environment can be executed more precisely and efficiently.

[0196] S8: According to the six - dimensional pose, control the robotic arm to grasp the object to be grasped.

[0197] Specifically, the six - dimensional pose of the object to be grasped is transmitted to the Jetson Nano main control system through the ROS asynchronous communication mechanism, and then control the Realmen RM65 robotic arm to perform the grasping operation. The system will wait for stable six - dimensional pose data to ensure that the robotic arm can accurately adjust its pose and position, thus precisely completing the grasping task of the object to be grasped.

[0198] It should be noted that the ROS asynchronous communication mechanism means that in the Robot Operating System (ROS), the transmission of data and messages does not require waiting for the confirmation or response of the receiving party. The sending party can independently send data without blocking and waiting. ROS realizes asynchronous communication through the publish - subscribe mode and the service call mechanism. The publisher publishes data to a certain topic, and the subscriber asynchronously receives this data without real - time response, thus improving the flexibility and real - time performance of the system. This mechanism is particularly suitable for a dynamically changing environment, enabling different components to efficiently exchange information without causing system performance degradation due to waiting for responses.

[0199] In the embodiments of the present invention, using the ROS asynchronous communication mechanism to transmit six - dimensional pose data can improve the flexibility and real - time performance of the system. This mechanism enables data transmission without waiting for the response of the receiving party, reduces latency, and ensures that the robotic arm can receive stable pose data in real - time, accurately adjusting the grasping position and pose, thereby improving the success rate and efficiency of the grasping task.

[0200] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0201] In an embodiment of the present invention, the LightGlue algorithm is used to perform feature point matching on an RGB image and a reference image to obtain a set of matching point pairs, and it is determined whether the number of matching point pairs in the set of matching point pairs is less than a preset number of matching point pairs. In the face of complex scenarios such as object occlusion, lighting changes, and large viewing angle differences, it has strong robustness and significantly improved matching accuracy. By using the depth image and the smoothed object bounding box, the six-dimensional pose of the object to be grasped is determined through a geometric algorithm, which is not easily affected by noise and environmental changes in a dynamic environment and can well meet the requirements of real-time applications.

[0202] Refer to the attached drawings of the specification Figure 2 , which shows a schematic structural diagram of an object grasping system based on the LightGlue algorithm provided by an embodiment of the present invention.

[0203] An embodiment of the present invention provides an object grasping system 20 based on the LightGlue algorithm, including: a processor 201 and a memory 202;

[0204] The memory 202 stores programs or instructions that can run on the processor 201. When the programs or instructions are executed by the processor 201, the steps of the above-mentioned object grasping method based on the LightGlue algorithm are implemented, and the same technical effects can be achieved. To avoid repetition, the present invention will not elaborate further.

[0205] It should be understood that the processor 201 in the embodiment of the present invention may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0206] It should also be understood that the memory 202 in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0207] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains a collection of one or more available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0208] It should be understood that in various embodiments of the present invention, the order numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0209] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0210] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0211] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0212] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0214] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0215] An embodiment of the present invention provides a readable storage medium including: programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by a processor, the steps of the above-mentioned object grasping method based on the LightGlue algorithm are implemented, and the same technical effects can be achieved. To avoid repetition, the present invention will not be described in detail again.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An object grasping method based on the LightGlue algorithm, characterized in that, Including: S1: Obtain the RGB image, depth image in the scene, and the reference image of the object to be grasped; S2: Perform feature point matching on the RGB image and the reference image through the LightGlue algorithm to obtain a set of matching point pairs; S3: Determine whether the number of matching point pairs in the set of matching point pairs is less than a preset number of matching point pairs; if so, return to step S1; otherwise, proceed to step S4; S4: Calculate the homography matrix through the RANSAC algorithm according to the set of matching point pairs; S5: Based on the homography matrix, map the bounding box of the reference image to the RGB image to obtain the object bounding box; S6: Combine the Kalman filter algorithm and the exponential moving average algorithm to smooth the object bounding box; S7: Determine the six-dimensional pose of the object to be grasped through a geometric algorithm according to the depth image and the smoothed object bounding box; S8: Control the robotic arm to grasp the object to be grasped according to the six-dimensional pose.

2. The object grasping method based on the LightGlue algorithm according to claim 1, wherein, After S1 and before S2, it further includes: S1A: Encode the RGB image using the Webp encoding format; S1B: Encode the depth image using the Png encoding format; S1C: Transmit the encoded RGB image and depth image to the host computer using the TCP protocol; S1D: When the host computer receives the encoded RGB image and depth image, decode the encoded RGB image and depth image to obtain the RGB image and the depth image.

3. The object grasping method based on the LightGlue algorithm according to claim 1, characterized in that, S4 specifically includes: S401: Randomly select a preset number of matching point pairs from the set of matching point pairs; S402: Calculate the candidate homography matrix through the least squares method according to each matching point pair: Among them, H s represents the candidate homography matrix, arg min represents minimization, H represents the homography matrix, N represents the number of randomly selected matching point pairs, and src_pts i represents the coordinates of the matching point corresponding to the reference image in the i-th matching point pair, T represents the transpose operation, and dst_pts i represents the coordinates of the matching point corresponding to the RGB image in the i-th matching point pair, |||| 2 represents the square of the Euclidean distance, (H · [src_pts i , 1] T ) z represents the z-axis component after the homography matrix undergoes homogeneous coordinate transformation; S403: Calculate the reprojection error of each matching point pair according to the candidate homography matrix; where, ε i represents the reprojection error of the i-th matching point pair, ||||2 represents the Euclidean norm, (H s · [src_pts i , 1] T ) z represents the z-axis component after homogeneous coordinate transformation of the candidate homography matrix; S404: Determine whether the reprojection error of each matching point pair is less than the reprojection error threshold; if so, determine the matching point pair as an inlier; otherwise, determine the matching point pair as an outlier; S405: Repeat steps S401 to S404 to obtain multiple candidate homography matrices; S406: Select the candidate homography matrix with the largest number of inliers as the homography matrix.

4. The object grasping method based on the LightGlue algorithm according to claim 1, characterized in that S5 specifically includes: S501: Obtain the coordinates of multiple corner points of the bounding box of the reference image; S502: Based on the homography matrix, map each corner point coordinate to the RGB image to obtain the homogeneous coordinates of the RGB image: Among them, represents the homogeneous coordinates of the point (p, q) in the RGB image. p and q respectively represent the abscissa and ordinate of the corner point mapped to the RGB image, r represents the third component in the homogeneous coordinates of the RGB image, H represents the homography matrix, represents the homogeneous coordinates of the point (x, y) in the reference image. x and y respectively represent the abscissa and ordinate of the corner point of the bounding box of the reference image, h bd represents the element in the b-th row and d-th column of the homography matrix, where b = 1, 2, 3 and d = 1, 2, 3; S503: Normalize the homogeneous coordinates of the RGB image to determine the coordinates of each corner point of the object bounding box: Where x' represents the abscissa of the corner point of the object bounding box, and y' represents the ordinate of the corner point of the object bounding box; S504: Map the bounding box of the reference image to the RGB image according to the coordinates of each corner point of the object bounding box to obtain the object bounding box.

5. The object grasping method based on the LightGlue algorithm according to claim 1, characterized in that S6 specifically includes: S601: Define the ten-dimensional state vector of the Kalman filter algorithm: X t = [x1', y1', x'2, y'2, x'3, y'3, x'4, y'4, v x' , v y' T ;​ Among them, X t represents the ten-dimensional state vector at time t, x′ c represents the abscissa of the c-th corner point of the object bounding box, y′ c represents the ordinate of the c-th corner point of the object bounding box, c = 1, 2, 3, 4, v x' represents the velocity component of the object bounding box in the abscissa direction, v y' represents the velocity component of the object bounding box in the ordinate direction, T represents the transpose operation; S602: Construct a state transition equation according to the ten-dimensional state vector: X t|t-1 = AX t-1 + w t ; Among them, X t|t-1 represents the state vector at time t predicted based on the state vector at time t - 1, represents the state transition matrix, represents the set of real numbers, X t-1 represents the ten-dimensional state vector at time t - 1, w t represents the process noise at time t; S603: Define the observation vector of the Kalman filter algorithm: Z t = [x1', y1', x'2, y'2, x'3, y'3, x'4, y'4] T ; Among them, Z t represents the observation vector at time t; S604: Construct an observation equation according to the ten-dimensional state vector and the observation vector: Z t = H′X t + V t ; Among them, represents the observation matrix, and V t represents the observation noise at time t; S605: According to the state transition equation and the observation equation, output the estimated values of the coordinates of each corner point of the object bounding box through the Kalman filter algorithm; S606: Perform time series smoothing processing on the estimated values through the exponential moving average algorithm: EMA t = α · filter_dst t + (1 - α) · EMA t-1 ; Among them, EMA t represents the estimated value after smoothing at time t, α represents the smoothing factor, and filter_dst t represents the estimated value output by the Kalman filter algorithm at time t, and EMA t-1 represents the estimated value after smoothing at time t-1.

6. The object grasping method based on the LightGlue algorithm according to claim 1, characterized in that, The specific steps of S7 include: S701: Extract multiple key points of the object to be grasped from the depth image according to the smoothed object bounding box, where the key points include the object center point, the center point of the right border, and the center point of the lower border; S702: Taking the object center point as the origin, calculate the coordinate difference between the center point of the right border and the object center point to obtain the first direction vector: Among them, represents the first direction vector, O' x represents the center point of the right frame, O' o represents the center point of the object; S703: Taking the object center point as the origin, calculate the coordinate difference between the center point of the lower border and the object center point to obtain the second direction vector: Among them, represents the second direction vector, and O' y represents the center point of the lower border; S704: Perform a cross product operation on the first direction vector and the second direction vector to obtain the normal vector: Among them, represents the normal vector, and z X , z Y and z Z respectively represent the components of the normal vector in the X-axis direction, Y-axis direction, and Z-axis direction, and X, Y, and Z respectively represent the X-axis, Y-axis, and Z-axis of the world coordinate system; S705: According to the second direction vector and the normal vector, correct the first direction vector through the cross product operation to obtain the third direction vector: Among them, represents the third direction vector; S706: Normalize the second direction vector, the normal vector, and the third direction vector into unit vectors: Among them, represents the first unit vector obtained by normalizing the third direction vector, i X , i Y and i Z respectively represent the components of the first unit vector in the X-axis direction, Y-axis direction, and Z-axis direction, represents the second unit vector obtained by normalizing the second direction vector, j X , j Y and j Z respectively represent the components of the second unit vector in the X-axis direction, Y-axis direction, and Z-axis direction, represents the third unit vector obtained by normalizing the normal vector, k X , k Y and k Z respectively represent the components of the third unit vector in the X-axis direction, Y-axis direction, and Z-axis direction; S707: Construct a rotation matrix according to the unit vectors: where R represents the rotation matrix; S708: Perform orthogonalization processing on the rotation matrix through the singular value decomposition algorithm; S709: Convert the orthonormalized rotation matrix into a quaternion to determine the six-dimensional pose of the object to be grasped.

7. The object grasping method based on the LightGlue algorithm according to claim 6, characterized in that, The specific steps of S708 include: S7081: Perform singular value decomposition on the rotation matrix to obtain a left singular vector matrix, a diagonal matrix, and a right singular vector matrix: R = UΣV T ; where R represents the rotation matrix, U represents the left singular vector matrix, Σ represents the diagonal matrix, V represents the right singular vector matrix, and T represents the transpose operation; S7082: According to the left singular vector matrix and the right singular vector matrix, obtain an orthonormalized matrix through multiplication operation: R ortho = UV T ; wherein, R ortho represents an orthogonalization matrix; S7083: Calculate the determinant of the orthonormalized matrix: det(R ortho ) = det(U)·det(V T ); where det() represents the determinant operation; S7084: Adjust the left singular vector matrix according to the determinant of the orthonormalized matrix; S7085: Recalculate the orthogonal matrix according to the adjusted left singular vector matrix to complete the orthogonalization processing of the rotation matrix.

8. The object grasping method based on the LightGlue algorithm according to claim 6, characterized in that, The specific content of S709 is: Convert the orthonormalized rotation matrix into a quaternion according to the following formula to determine the six-dimensional pose of the object to be grasped: Among them, represents the first unit vector after orthogonalization, i' X , i' Y and i' Z respectively represent the components of the first unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. represents the second unit vector after orthogonalization, j' X , j' Y and j' Z respectively represent the components of the second unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. represents the third unit vector after orthogonalization, k' X , k' Y and k' Z respectively represent the components of the third unit vector after orthogonalization in the X-axis direction, Y-axis direction, and Z-axis direction. R' ortho represents the orthogonal matrix after orthogonalization, (q w , q x , q y , q z ) represents a quaternion.

9. An object grasping system based on the LightGlue algorithm, characterized in that, including: a processor and a memory; The memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, the steps of the object grasping method based on the LightGlue algorithm as described in any one of claims 1 to 8 are implemented.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, the steps of the object grasping method based on the LightGlue algorithm described in any one of claims 1 to 8 are implemented.

Images