A transparent glass pose detection method and system based on multi-modal data and a glass grabbing method

Abstract

The application discloses a transparent glass pose detection method and system based on multi-modal data and a glass grabbing method, relates to the technical field of glass pose detection, and discloses the following technical scheme: a camera is used to capture the geometric distortion features of a glass contour, the relative pose relationship between the geometric distortion features and a camera plane is analyzed, a mechanical arm is driven to be roughly adjusted to a parallel pose, distance information collected by an ultrasonic sensor array is used to fit a glass plane equation through spatial geometric solving, and the normal vector direction and distance parameters of the glass are accurately obtained, so that closed-loop fine adjustment of the pose of the mechanical arm is realized. The application breaks through the detection bottleneck that transparent media are difficult to reflect optical signals, realizes high-precision pose detection without the need of installing auxiliary equipment on the surface of the glass, and avoids damage to the surface of the glass caused by a contact type mark. Multi-modal data complement each other, the detection reliability is significantly improved, low-cost sensors are used to replace high-precision laser equipment, and industrial-grade precision and economy are combined.

Description

Technical Field

[0001] This invention relates to the field of glass pose detection technology, and in particular to a transparent glass pose detection method, system, and glass gripping method based on multimodal data. Background Technology

[0002] Glass pose detection is a key technology that aims to determine the position and normal vector direction of a target glass in space. It is widely used in automated production processes such as glass manufacturing and processing to guide robotic arms and suction cups in picking up glass. In modern glass manufacturing plants and processing sites, glass pose detection has become a core technology for ensuring glass processing quality and improving operational efficiency. By using data acquired by sensors and calculating relevant parameters, the detection system can quickly and accurately determine the pose state of the glass, providing a reliable basis for subsequent glass processing, handling, and installation operations, ensuring that all tasks are completed smoothly and accurately.

[0003] As a transparent medium, glass cannot generate effective point cloud data on its surface, making it impossible to obtain angle and distance information between the glass and the sensor. Furthermore, glass is a fragile material; if pose information such as angle and distance cannot be obtained during the robotic arm's grasping process, the glass is prone to breakage. Therefore, obtaining the pose information between the glass and the sensor is both the key and the challenge in guiding the robotic arm to pick up or grasp glass.

[0004] Chinese patent CN2024107312205 discloses a device and method for measuring and positioning ultra-large glass, which uses a mounting plate to position ultra-large glass. This method requires auxiliary equipment on the glass, is inefficient, and cannot be used in automated production lines. Another Chinese patent CN2024119376655 uses a camera for visual inspection to calculate the glass's pose information. This method addresses the pose detection of the glass on the machine tool, knowing the depth information between the glass and the camera, and the angle between the glass's normal vector and the camera's optical axis. However, when dealing with stacked glass, information such as the distance between the glass to be inspected and the camera is missing, rendering this method unsuitable.

[0005] In summary, existing methods are not flexible and adaptable enough to meet the complex and ever-changing working conditions such as glass stacking in automated production sites. Summary of the Invention

[0006] One of the objectives of this invention is to provide a transparent glass pose detection method based on multimodal data with better detection performance. This method eliminates the need to install related equipment on the glass in advance, thereby improving convenience and reducing costs.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for detecting the pose of transparent glass based on multimodal data, comprising the following steps:

[0008] S1. Install a data acquisition device at the end of the robotic arm. The data acquisition device includes a monocular camera and multiple non-collinearly arranged ultrasonic sensors.

[0009] S2. The contour image data of the target glass is acquired by the monocular camera. The geometric center and imaging distortion parameters of the glass are extracted based on the image features. The first relative pose relationship between the glass plane and the camera plane is calculated. The robotic arm is driven to perform coarse pose adjustment so that the flange plane at the end of the robotic arm is initially parallel to the glass plane.

[0010] S3. The distance data of the glass is collected synchronously by multiple ultrasonic sensors. Based on the distance information of multiple sensors, the second relative pose relationship of the glass plane in the flange coordinate system is calculated. The plane normal vector deviation and distance parameters are calculated, and the robotic arm is driven to fine-tune the pose until the preset parallel threshold is met.

[0011] S4. Output the pose parameters of the glass plane relative to the end of the robotic arm. The pose parameters include the normal vector direction and distance information, so that the robotic arm can perform a grasping operation in the future.

[0012] Preferably, step S1 further includes a device installation and calibration process: determining the spatial transformation relationship between the monocular camera and the coordinate system of the end flange of the robotic arm through a calibration object, and recording the installation coordinate parameters of each ultrasonic sensor in the flange plane.

[0013] More preferably, the step S2 of driving the robotic arm to perform coarse pose adjustment specifically includes the following steps:

[0014] S2.1 Data Acquisition: Acquire image data and extract the contour features of the glass;

[0015] S2.2 Extracting the orientation information of the glass: Using the contour information of the glass, calculate the geometric center point of the glass and the distortion of the glass in the camera image, including the geometric structure of the glass imaging. Specific parameters include: bottom angle, tilt angle, horizontal offset, ratio of the length of the upper and lower bottom edges, and tilt direction of the inclined side.

[0016] S2.3 Threshold Comparison: Estimate the difference between the geometric center of the glass and the center of the flange plane by using the position of the glass geometric center in the image; estimate the angle between the glass normal vector and the flange plane normal vector by using the distortion of the glass image; if both are less than the set threshold, terminate the pose coarse adjustment operation; otherwise, proceed to step S2.4.

[0017] S2.4 Adjusting the pose of the robotic arm: Send a command to adjust the pose of the end effector of the robotic arm. Through rotation and translation operations, make the normal vector of the flange plane approximately parallel to the normal vector of the glass, and make the normal vector at the center of the flange plane point to the geometric center of the glass; and continue the data acquisition in step S2.1.

[0018] More preferably, step S3, which involves driving the robotic arm to fine-tune its pose, specifically includes the following steps:

[0019] S3.1 In a stationary state, the distances to the glass are simultaneously detected by three coplanar ultrasonic sensors and recorded as follows: , and ;

[0020] S3.2, after After the time interval, continue with step S3.1, and save the new distance information as follows. , and This process is repeated N times. Mean filtering is used to remove sensor noise. The specific operation method is as follows:

[0021]

[0022] in, Let be the distance value after filtering by the i-th ultrasonic sensor. This is the window length for mean filtering. Let be the distance value of the i-th ultrasonic sensor at time t;

[0023] S3.3. Using the differences between the three distances, fit the spatial attitude information of the glass plane in the sensor coordinates;

[0024] S3.4 Calculate the angle between the two planes and determine whether fine-tuning is needed; compare the calculated rotation angle with a pre-set threshold. If the rotation angle is less than the threshold, proceed to step S3.6; otherwise, proceed to step S3.5.

[0025] S3.5 Guide the robotic arm to perform a reverse rotation operation so that the detection plane is parallel to the glass plane in space;

[0026] S3.6 Calculate the distance from the end effector of the robotic arm to the glass.

[0027] More preferably, in step S3.3, let the plane on which the ultrasonic sensor is installed be the reference plane S, and the target plane be T. The equation of the target plane T is: The distances measured by the three ultrasonic sensors are respectively , , The coordinates of the three ultrasonic sensors , and Substitute into the target plane equation By combining the distance measurement values, the following system of equations can be obtained:

[0028]

[0029] Its matrix form is ,in

[0030] , ,

[0031] Because the three ultrasonic sensors are not collinear, matrix X is invertible. By solving, we can obtain The value is Therefore, the normal vector of the target plane T is .

[0032] More preferably, in step S3.4, let the normal vector of the reference plane be... The normal vector of the target plane is When the two normal vectors are not parallel, the axis of rotation is the cross product of the two normal vectors, that is:

[0033]

[0034] Normalizing it yields the unit axis of rotation. ,

[0035] in .

[0036] The rotation angle is calculated using the dot product of the two normal vectors:

[0037]

[0038] For the calculated With a pre-set threshold When a comparison is performed, Less than If the distance is too short, calculate the distance from the end of the robotic arm to the glass; otherwise, proceed to step S3.5.

[0039] More preferably, in step S3.5, the Rodriguez rotation formula is used to construct a rotation along any axis by means of the rotation axis and rotation angle. Rotation angle Rotation matrix:

[0040]

[0041] in, It is the identity matrix. .

[0042] The reference plane is multiplied by a rotation matrix to adjust the posture of the robotic arm's end effector. Then, the process returns to step S3.1 to continue the next round of data acquisition and comparison.

[0043] More preferably, in step S3.6, calculating the distance from the robotic arm's end effector to the glass specifically includes: taking the arithmetic mean of the filtered distances from the three ultrasonic sensors, i.e.

[0044]

[0045] In the formula, This is the distance that the end effector of the robotic arm needs to move along the z-axis of the coordinate system.

[0046] In addition, this invention also provides a glass grasping method based on multimodal data-based transparent glass pose detection: multiple suction cups are installed at the end of a robotic arm, and the aforementioned multimodal data-based transparent glass pose detection method is used to guide the end of the robotic arm to move along the z-axis of the end-effector coordinate system. They also pick up the glass for handling.

[0047] Furthermore, the present invention also provides a system for implementing the above-described transparent glass pose detection method based on multimodal data, comprising: a robotic arm execution unit, a monocular camera mounted at the end of the robotic arm, at least three non-collinearly arranged ultrasonic sensors, and a data processing unit; the data processing unit is configured to: execute an image feature extraction algorithm, a plane equation solving algorithm, and a pose control command generation module.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] 1. By using a monocular camera in conjunction with multiple non-collinear ultrasonic sensors, the limitations of single sensors in transparent media detection are overcome. The camera's vision quickly captures the glass contour features for coarse positioning, while the ultrasonic ranging accurately calculates spatial pose parameters for fine-tuning. The complementary data from the two modes effectively solves problems such as weak reflection signals caused by the transparency of glass and the failure of traditional sensors (such as LiDAR), significantly improving the environmental adaptability and reliability of the detection system.

[0050] 2. Detection is accomplished using only a single bullet screen camera and a conventional ultrasonic sensor, which significantly reduces hardware costs compared to high-precision lidar or structured light equipment.

[0051] 3. No auxiliary markings or equipment need to be installed on the glass surface. Pure non-contact detection is achieved through a combination of monocular camera visual recognition and ultrasonic ranging. This completely avoids scratches, contamination or structural stress damage to the glass surface caused by traditional contact positioning (such as fixture installation) or semi-contact detection (such as affixing reflective markings). It is especially suitable for precision scenarios such as electronic glass and optical lenses with high cleanliness requirements. Attached Figure Description

[0052] Figure 1 This is a schematic diagram showing the position layout of the end-effector acquisition device in the embodiment;

[0053] Figure 2 This is a schematic diagram of the camera calibration process in the embodiment;

[0054] Figure 3 This is a schematic diagram illustrating the acquisition of image data and extraction of the glass contour features in this embodiment.

[0055] Figure 4 This is a schematic diagram of the glass attitude information extracted in the embodiment;

[0056] Figure 5 This is a schematic diagram illustrating the process principle of coarse pose adjustment of the robotic arm in the embodiment.

[0057] Figure 6 This is a schematic diagram of three non-collinear ultrasonic sensors collecting glass information during fine-tuning of the robotic arm's end-effector pose in the embodiment.

[0058] Figure 7 This is a schematic diagram illustrating the process principle of the robotic arm fine-tuning its pose in the embodiment. Detailed Implementation

[0059] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0060] A method for detecting the pose of transparent glass based on multimodal data includes the following steps:

[0061] S1. Install a data acquisition device at the end of the robotic arm. The data acquisition device includes a monocular camera and multiple non-collinearly arranged ordinary ultrasonic sensors.

[0062] Specifically, such as Figure 1As shown, a pre-designed fixture is first installed on the flange at the end of the robotic arm. The fixture is a rectangular plane, with the flange positioned at the center of the rectangle. The normal vector of the fixture plane is parallel to the flange direction, and the edge of the fixture plane is parallel to the flange coordinate axes. Both the camera and ultrasonic sensors are mounted next to the flange at the end of the robotic arm. The camera's optical axis direction vector and the ultrasonic detection direction vector are parallel to the normal vector of the flange plane at the end of the robotic arm. There are three ultrasonic sensors, and their installation positions on the fixture plane are not collinear.

[0063] Simultaneously, the coordinate information of the three ultrasonic sensors on the end flange plane needs to be recorded for subsequent attitude calculation. The coordinate information is based on the flange position as the origin, with the horizontal direction as the x-axis and the vertical direction as the y-axis. The coordinates of the three ultrasonic sensors are as follows: , and .

[0064] The camera and ultrasonic sensor serve as data acquisition functions in the entire system. By analyzing the data collected by the camera and ultrasonic sensor, the posture of the robotic arm is adjusted so that it can correctly adsorb the target glass.

[0065] Step S1 also includes the equipment installation and calibration process, specifically: such as Figure 2 As shown, the goal of camera calibration is to accurately determine the transformation relationship between the camera coordinate system and the coordinate system of the robotic arm's end flange. This is achieved by placing a calibration object with known features, such as a checkerboard pattern, in the world coordinate system, then guiding the robotic arm to move the camera to different positions and capturing multiple images of the calibration object. The pixel coordinates of feature points in the images are then detected. And the coordinates of known feature points of the calibration object in the world coordinate system. The pose of the robotic arm's end effector in the world coordinate system is obtained by combining the kinematic model of the robotic arm, and a system of equations is established using the aforementioned coordinate transformation relationships and the camera imaging model. By solving multiple sets of such equations, the transformation matrix between the camera coordinate system and the robotic arm's end effector coordinate system can be obtained. Commonly used solution algorithms include the Tsai-Lenz method and the Park method.

[0066] S2. Coarse adjustment of the robotic arm's pose.

[0067] By detecting the orientation of the glass, the position and orientation of the robotic arm's end effector in space are adjusted in reverse, so that the robotic arm flange is at the geometric center of the glass, and the flange plane normal vector is approximately parallel to the glass normal vector.

[0068] Specifically, it includes the following steps:

[0069] S2.1 Data Acquisition. For example... Figure 3As shown, image data was acquired and the contour features of the glass were extracted.

[0070] S2.2 Extract the glass's attitude information. For example... Figure 4 As shown, the geometric center point of the glass and the distortion of the glass in the camera image are calculated using the glass contour information, including the geometric structure of the glass imaging (parallelogram and trapezoid, and related parameters such as interior angles).

[0071] Specific parameters include the bottom angle. Top and bottom edges and left and right sides By using these parameters, the relative positional parameters between the glass and the camera can be roughly determined. The specific method is as follows:

[0072] (1) Parallelograms form an image:

[0073] Tilt Angle: The size of the acute or obtuse angle of a parallelogram reflects the degree of tilt between the camera's optical axis and the object's plane. By measuring the interior angles of the parallelogram, the angle between the camera's optical axis and the normal to the object's plane can be calculated. Assume one interior angle of the parallelogram is... Then the angle between the camera's optical axis and the object's plane .

[0074] Horizontal offset: The extensions of the two hypotenuses of a parallelogram intersect at a point. The direction of the line connecting this point to the center of the rectangular object indicates the direction of the camera's horizontal offset relative to the object. If the intersection point is to the right of the object's center, the camera is diagonally above or below the object's left side; conversely, if the intersection point is to the left, the camera is diagonally above or below the object's right side.

[0075] (2) Trapezoidal imaging:

[0076] The ratio of the upper and lower base lengths: The ratio of the upper and lower base lengths of a trapezoid is related to the angle between the camera's optical axis and the object's plane, as well as the camera's height. A greater difference in base lengths indicates a larger angle between the camera's optical axis and the object's plane, and a relatively lower camera position. Let the upper base of the trapezoid be *a*, and the lower base be *b*. This can be determined by... To estimate the angle between the camera's optical axis and the object's plane. , where h is the vertical distance from the camera to the center of the object. If the distance is unknown, the relevant angle can be roughly estimated.

[0077] Hypotenuse inclination direction: The inclination direction of the two hypotenuses of a trapezoid reflects the horizontal offset direction of the camera. If the two hypotenuses tilt to the right, the camera is diagonally above or below the left side of the object; if they tilt to the left, the camera is diagonally above or below the right side.

[0078] S2.3 Threshold Comparison. Estimate the difference between the glass geometric center and the flange plane center by using the position of the glass geometric center in the image. Estimate the angle between the glass normal vector and the flange plane normal vector by considering the distortion of the glass image. If both are less than the set threshold, terminate the coarse pose adjustment operation; otherwise, proceed to step S2.4.

[0079] S2.4 Adjust the pose of the robotic arm. Send a command to adjust the pose of the robotic arm end effector. Through rotation and translation operations, make the normal vector of the flange plane approximately parallel to the normal vector of the glass, and make the normal vector at the center of the flange plane point to the geometric center of the glass; and continue with the information extraction in step S2.1.

[0080] First, the robotic arm's end effector is translated so that the geometric center of the glass in the camera coincides with the center of the camera image. The mathematical model for this translation operation is:

[0081]

[0082] in, The coordinates of the moved flange. The flange coordinates before movement. These are the estimated movement parameters.

[0083] Secondly, the robotic arm's end effector is rotated so that the normal vector of the end flange is parallel to the glass plane. The mathematical model for this rotation operation is:

[0084]

[0085] in, This provides the attitude information of the end flange before it rotates. This provides the attitude information of the end flange after rotation. Here is the rotation matrix. The rotation matrix formed by rotating the end flange along three axes is as follows:

[0086]

[0087] Let x be the rotation matrix of the end flange along the x-axis. The rotation matrix is:

[0088]

[0089] The rotation matrix for rotating the end flange along the y-axis is:

[0090]

[0091] The rotation matrix for rotating the end flange along the z-axis is:

[0092]

[0093] Because of the positional information between the camera and the glass, the adjustment is made iteratively. That is, after determining the approximate relative relationship between the camera and the glass, the end of the robotic arm is rotated using the minimum angle parameter, so that it gradually rotates towards the direction of the glass normal vector.

[0094] The above describes the method for coarse adjustment of the robotic arm's end effector pose. The entire process is as follows: Figure 5 As shown. This method uses a regular camera to quickly adjust the pose of the robotic arm's end effector, so that the end flange is roughly above the geometric center of the glass and its direction is roughly parallel to the glass's normal vector, thus providing a better state for subsequent fine-tuning.

[0095] S3, fine-tuning of robotic arm posture (precise adjustment).

[0096] Because the coarse adjustment of the robotic arm's pose relies solely on the distortion of the glass image to correct the posture of the robotic arm's end effector, this method has inherent errors. Furthermore, computer vision methods cannot obtain the distance information between the robotic arm's end effector and the glass. Therefore, fine adjustment of the robotic arm's pose is introduced here, such as... Figure 6 As shown, the robot arm's posture is calibrated using three non-collinear ultrasonic sensors, and the precise distance from the robot arm's end effector to the glass is obtained.

[0097] The process for fine-tuning the pose of the robotic arm is as follows:

[0098] S3.1 In a stationary state, the distances to the glass are simultaneously detected by three coplanar ultrasonic sensors and recorded as follows: , and .

[0099] S3.2, after After the time interval, continue with step 1, and save the new distance information as follows. , and This process is repeated N times (where N is a hyperparameter representing the width of the filtering window set by the system). Mean filtering removes sensor noise; the specific operation method is as follows:

[0100]

[0101] in, Let be the distance value after filtering by the i-th ultrasonic sensor. This is the window length for mean filtering. Let be the distance value of the i-th ultrasonic sensor at time t.

[0102] S3.3. Using the differences between the three distances, fit the spatial attitude information of the glass plane in the sensor coordinates;

[0103] Let the plane on which the ultrasonic sensor is installed be the reference plane S, and the target plane be T. The equation of the target plane T is: The distances measured by the three ultrasonic sensors are as follows: , , The coordinates of the three ultrasonic sensors , and Substitute into the target plane equation By combining the distance measurement values, the following system of equations can be obtained:

[0104]

[0105] Its matrix form is ,in

[0106] , ,

[0107] Because the three ultrasonic sensors are not collinear, matrix X is invertible. By solving, we can obtain The value is Therefore, the normal vector of the target plane T is .

[0108] S3.4 Calculate the angle between the two planes and determine whether fine-tuning is needed;

[0109] Let the normal vector of the reference plane be... The normal vector of the target plane is .

[0110] When the two normal vectors are not parallel, the axis of rotation is the cross product of the two normal vectors, i.e.

[0111]

[0112] Normalizing it yields the unit axis of rotation. ,

[0113] in .

[0114] The rotation angle can be calculated using the dot product of the two normal vectors:

[0115]

[0116] Here is the calculated With a pre-set threshold When a comparison is performed, Less than If the condition is met, proceed to step S3.6; otherwise, continue to step S3.5.

[0117] S3.5 Guide the robotic arm forearm to rotate in the opposite direction so that the detection plane is parallel to the glass plane in space;

[0118] By using the axis of rotation and the rotation angle, the Rodrigues rotation formula can be used to construct rotations along any axis. Rotation angle The rotation matrix.

[0119]

[0120] in, It is the identity matrix. .

[0121] The reference plane is multiplied by a rotation matrix to adjust the posture of the robotic arm's end effector. Then, the process returns to step S3.1 to continue the next round of data acquisition and comparison.

[0122] S3.6 Calculate the distance from the end effector of the robotic arm to the glass.

[0123] Take the arithmetic mean of the filtered distances from the three ultrasonic sensors, i.e.

[0124]

[0125] In the formula, This is the distance that the end effector of the robotic arm needs to move along the z-axis of the coordinate system.

[0126] At this point, the glass pose detection is basically complete. For subsequent glass grasping, it is only necessary to guide the robotic arm's end effector to move along the z-axis of the end effector coordinate system. The robot arm uses multiple suction cups at its end to pick up glass for transport. The entire process of finely adjusting the robot arm's posture and picking up the glass is as follows: Figure 7 As shown.

[0127] The transparent glass pose detection method based on multimodal data provided in the above embodiments constructs a multimodal detection system by fusing visual data from a monocular camera with ranging data from multiple non-collinear ultrasonic sensors. First, the camera captures the geometric distortion features of the glass contour, analyzes its relative pose relationship with the camera plane, and drives the robotic arm to coarsely adjust to a roughly parallel pose. Then, based on the distance information collected by the ultrasonic sensor array, the glass plane equation is fitted through spatial geometric calculations to accurately obtain the glass's normal vector direction and distance parameters, achieving closed-loop fine-tuning of the robotic arm's pose. The entire method, through non-contact data acquisition and algorithmic collaboration, completes the pose detection of transparent glass in three-dimensional space.

[0128] This invention overcomes the detection bottleneck caused by the difficulty of transparent media reflecting optical signals, achieving high-precision pose detection without the need for auxiliary equipment on the glass surface, thus avoiding damage to the glass surface caused by contact marking. Multimodal data complementarity significantly improves detection reliability, while the use of low-cost sensors to replace high-precision laser equipment combines industrial-grade accuracy with cost-effectiveness. The fully automated process is compatible with high-speed production lines, and the standardized pose parameters output can directly drive a robotic arm to complete the grasping process, making it widely applicable in glass manufacturing, electronic assembly, and other fields.

[0129] To facilitate understanding by those skilled in the art of the improvements of this invention over the prior art, some of the accompanying drawings and descriptions have been simplified. The above embodiments are preferred implementations of this invention. In addition, this invention can be implemented in other ways. Any obvious substitutions without departing from the concept of this technical solution are within the protection scope of this invention.

Claims

1. A method for detecting the pose of transparent glass based on multimodal data, characterized in that, Includes the following steps: S1. Install a data acquisition device at the end of the robotic arm, the data acquisition device including a monocular camera and multiple non-collinearly arranged ultrasonic sensors; S2. The contour image data of the target glass is acquired by the monocular camera. The geometric center and imaging distortion parameters of the glass are extracted based on the image features. The first relative pose relationship between the glass plane and the camera plane is calculated. The robotic arm is driven to perform coarse pose adjustment so that the flange plane at the end of the robotic arm is initially parallel to the glass plane. S3. The distance data of the glass is collected synchronously by multiple ultrasonic sensors. Based on the distance information of multiple sensors, the second relative pose relationship of the glass plane in the flange coordinate system is calculated. The plane normal vector deviation and distance parameters are calculated. The robotic arm is driven to fine-tune the pose until the preset parallel threshold is met. S4. Output the pose parameters of the glass plane relative to the end of the robotic arm. The pose parameters include the normal vector direction and distance information, so that the robotic arm can perform a grasping operation later. Step S3, which involves driving the robotic arm to fine-tune its pose, specifically includes the following steps: S3.1 In a stationary state, the distances to the glass are simultaneously detected by three coplanar ultrasonic sensors and recorded as follows: , and ; S3.2, after After the time interval, continue with step S3.1, and save the new distance information as follows. , and This process is repeated N times in total. Sensor noise is removed through mean filtering. The specific operation method is as follows: ； in, Let be the distance value after filtering by the i-th ultrasonic sensor. This is the window length for mean filtering. Let be the distance value of the i-th ultrasonic sensor at time t; S3.

3. Using the differences between the three distances, fit the spatial attitude information of the glass plane in the sensor coordinates; S3.4 Calculate the angle between the two planes and determine whether fine-tuning is needed; compare the calculated rotation angle with a pre-set threshold. If the rotation angle is less than the threshold, proceed to step S3.6; otherwise, proceed to step S3.

5. S3.5 Guide the robotic arm to perform a reverse rotation operation so that the detection plane is parallel to the glass plane in space; S3.6 Calculate the distance from the end effector of the robotic arm to the glass; In step S3.4, let the normal vector of the reference plane be... The normal vector of the target plane is When the two normal vectors are not parallel, the axis of rotation is the cross product of the two normal vectors, that is: ； Normalizing it yields the unit axis of rotation. , in ; The rotation angle is calculated using the dot product of the two normal vectors: ； For the calculated With a pre-set threshold When a comparison is performed, Less than If the distance is too short, calculate the distance from the end of the robotic arm to the glass; otherwise, proceed to step S3.

5.

2. The transparent glass pose detection method based on multimodal data according to claim 1, characterized in that: Step S1 also includes the equipment installation and calibration process: determining the spatial transformation relationship between the monocular camera and the coordinate system of the end flange of the robotic arm through the calibration object, and recording the installation coordinate parameters of each ultrasonic sensor in the flange plane.

3. The method for detecting the pose of transparent glass based on multimodal data according to claim 1, characterized in that, Step S2, which involves driving the robotic arm to perform coarse pose adjustment, specifically includes the following steps: S2.1 Data Acquisition: Acquire image data and extract the contour features of the glass; S2.2 Extracting the orientation information of the glass: Using the contour information of the glass, calculate the geometric center point of the glass and the distortion of the glass in the camera image, including the geometric structure of the glass imaging. Specific parameters include: bottom angle, tilt angle, horizontal offset, ratio of the length of the upper and lower bottom edges, and tilt direction of the inclined side. S2.3 Threshold Comparison: Estimate the difference between the geometric center of the glass and the center of the flange plane by using the position of the glass geometric center in the image; estimate the angle between the glass normal vector and the flange plane normal vector by using the distortion of the glass image; if both are less than the set threshold, terminate the pose coarse adjustment operation; otherwise, proceed to step S2.

4. S2.4 Adjusting the pose of the robotic arm: Send a command to adjust the pose of the end effector of the robotic arm. Through rotation and translation operations, make the normal vector of the flange plane approximately parallel to the normal vector of the glass, and make the normal vector at the center of the flange plane point to the geometric center of the glass; and continue the data acquisition in step S2.

1.

4. The transparent glass pose detection method based on multimodal data according to claim 1, characterized in that: In step S3.3, let the plane where the ultrasonic sensor is installed be the reference plane S, and the target plane be T. The equation of the target plane T is: ; The distances measured by the three ultrasonic sensors are respectively , , The coordinates of the three ultrasonic sensors , and Substitute into the target plane equation By combining the distance measurement values, the following system of equations can be obtained: ； Its matrix form is ,in ， ， ； Because the three ultrasonic sensors are not collinear, matrix X is invertible. By solving, we can obtain The value is ; Therefore, the normal vector of the target plane T is .

5. The transparent glass pose detection method based on multimodal data according to claim 4, characterized in that: In step S3.5, the Rodriguez rotation formula is used to construct a rotation along any axis by using the rotation axis and rotation angle. Rotation angle Rotation matrix: ； in, It is the identity matrix. ; The reference plane is multiplied by a rotation matrix to adjust the posture of the robotic arm's end effector. Then, the process returns to step S3.1 to continue the next round of data acquisition and comparison.

6. The transparent glass pose detection method based on multimodal data according to claim 5, characterized in that: In step S3.6, calculating the distance from the robotic arm's end effector to the glass specifically includes: taking the arithmetic mean of the filtered distances from the three ultrasonic sensors, i.e. ； In the formula, This is the distance that the end effector of the robotic arm needs to move along the z-axis of the coordinate system.

7. A glass grasping method based on transparent glass pose detection using multimodal data, characterized in that: Multiple suction cups are installed at the end effector of the robotic arm, and the transparent glass pose detection method based on multimodal data as described in claim 6 is used to guide the end effector to move along the z-axis of the end effector coordinate system. They also pick up the glass for handling.

8. A system for implementing the transparent glass pose detection method based on multimodal data according to any one of claims 1-6, characterized in that, include: The robotic arm execution unit, a monocular camera mounted at the end of the robotic arm, at least three non-collinearly arranged ultrasonic sensors, and a data processing unit; The data processing unit is configured to execute an image feature extraction algorithm, a plane equation solving algorithm, and a pose control command generation module.

Images