Transparent glass pose detection method and system based on multi-modal data and glass grabbing method

Abstract

The invention discloses a transparent glass pose detection method and system based on multi-modal data and a glass grabbing method, and relates to the technical field of glass pose detection. Geometric distortion characteristics of a glass contour are captured by using a camera, a relative pose relationship between the geometric distortion characteristics and a camera plane is analyzed, and a mechanical arm is driven to be roughly adjusted to a roughly parallel pose; based on distance information collected by an ultrasonic sensor array, a glass plane equation is fitted through space geometric solution, the normal vector direction and distance parameters of glass are accurately obtained, and closed-loop fine adjustment of the pose of the mechanical arm is achieved. The detection bottleneck that a transparent medium is difficult to reflect an optical signal is broken through, high-precision pose detection can be realized without installing auxiliary equipment on the surface of the glass, and the damage of a contact mark to the surface of the glass is avoided. The detection reliability is remarkably improved through multi-modal data complementation, meanwhile, a low-cost sensor is adopted to replace high-precision laser equipment, and industrial-grade precision and economical efficiency are both achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of glass posture detection, and in particular to a transparent glass posture detection method and system based on multimodal data, and a glass grasping method. Background Art

[0002] Glass posture detection is a key detection technology that aims to detect the position and normal vector direction of the target glass in space. Glass posture detection is widely used in glass manufacturing, glass processing and other related automated production processes to guide the robotic arm suction cup to complete the glass suction work. In modern glass production plants and glass processing production sites, glass posture detection has become a core technical means to ensure glass processing quality and improve work efficiency. With the data obtained by the sensor and the calculation of relevant parameters, the detection system can quickly and accurately determine the posture state of the glass, providing a reliable basis for subsequent glass processing, transportation and installation operations, ensuring that each task can be completed smoothly and accurately.

[0003] Glass is a transparent medium, so commonly used LiDAR sensors cannot generate valid point cloud data on its surface. Consequently, they are unable to obtain angle and distance information between the glass and the sensor. Glass is also a fragile material, so if a robotic arm cannot obtain positional information such as angle and distance during grasping, it can easily break the glass. Therefore, obtaining positional information between the glass and the sensor is both a key and challenging aspect of guiding the robotic arm to grasp or grip the glass.

[0004] Chinese patent CN2024107312205 discloses a device and method for measuring, positioning and controlling oversized glass, which uses a related mounting plate to complete the positioning of oversized glass. This method requires the installation of auxiliary facilities on the glass, which is inefficient and cannot be used on automated production lines. Another Chinese patent CN2024119376655 uses a camera to achieve visual inspection and calculate the posture information of the glass. This method solves the problem of posture detection of glass on a machine tool. The depth information between the glass and the camera, and the angle between the glass normal vector and the camera optical axis are known. However, when dealing with stacked glass, information such as the distance between the glass to be detected and the camera is missing, and this method cannot be applied.

[0005] In summary, the existing methods have poor flexibility and adaptability and cannot meet the complex and changeable working conditions such as glass stacking in automated production sites. Summary of the Invention

[0006] One of the purposes of the present invention is to provide a transparent glass posture detection method based on multimodal data with better detection effect. Through this method, there is no need to install relevant equipment on the glass in advance, so as to improve convenience and reduce costs.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: a transparent glass posture detection method based on multimodal data, comprising the following steps:

[0008] S1. Install a data acquisition device at the end of the robotic arm, wherein the data acquisition device includes a monocular camera and multiple non-collinearly arranged ultrasonic sensors.

[0009] S2. Collect contour image data of the target glass through the monocular camera, extract the geometric center and imaging distortion parameters of the glass based on image features, calculate the first relative posture relationship between the glass plane and the camera plane, and drive the robotic arm to roughly adjust the posture so that the flange plane at the end of the robotic arm and the glass plane are initially parallel.

[0010] S3. Synchronously collect the distance data of the glass through multiple ultrasonic sensors, solve the second relative posture relationship of the glass plane in the flange coordinate system based on the multi-sensor distance information, calculate the plane normal vector deviation and distance parameters, and drive the robotic arm to fine-tune the posture until the preset parallel threshold is met.

[0011] S4. Output the posture parameters of the glass plane relative to the end of the robotic arm, where the posture parameters include normal vector direction and distance information, so that the robotic arm can perform a subsequent grasping operation.

[0012] Preferably, step S1 also includes an equipment installation calibration process: determining the spatial transformation relationship between the monocular camera and the flange coordinate system of the end of the robotic arm through a calibration object, and recording the installation coordinate parameters of each ultrasonic sensor in the flange plane.

[0013] More preferably, driving the robotic arm to perform coarse posture adjustment in step S2 specifically includes the following steps:

[0014] S2.1. Data acquisition: Collect image data and extract the contour features of the glass;

[0015] S2.2. Extracting Glass Posture Information: Using the glass's contour information, calculate the geometric center point of the glass and the distortion of the glass in the camera image, including the geometric structure of the glass image. Specific parameters include: bottom angle, tilt angle, horizontal offset, upper and lower bottom edge length ratio, and hypotenuse tilt direction.

[0016] S2.3, threshold comparison: Estimate the difference between the geometric center of the glass and the center of the flange plane through the position of the geometric center of the glass in the image; Estimate the angle between the normal vector of the glass and the normal vector of the flange plane through the distortion of the glass imaging; If both are less than the set threshold, terminate the coarse adjustment of the posture, otherwise proceed to step S2.4.

[0017] S2.4. Adjust the position of the robotic arm: Send a command to adjust the position of the end of the robotic arm. Through rotation and translation operations, make the normal vector of the flange plane roughly parallel to the normal vector of the glass, and the normal vector at the center of the flange plane points to the geometric center of the glass; and continue with data collection in step S2.1.

[0018] More preferably, driving the robotic arm to fine-tune the posture in step S3 specifically includes the following steps:

[0019] S3.1. In a stationary state, three coplanar ultrasonic sensors simultaneously detect the distance to the glass and record it as D 1_t , D 2_t and D 3_t ;

[0020] S3.2, after a time interval of Δt, continue with step S3.1, and save the new distance information as D 1_t+1 , D 2_t+1 and D 3_t+1 , repeat N times in total. Remove the noise of the sensor through mean filtering. The specific operation method is as follows:

[0021]

[0022] in, is the distance value after filtering of the i-th ultrasonic sensor, N is the window length of the mean filter, D i_t is the distance value of the i-th ultrasonic sensor at time t;

[0023] S3.3. Using the differences between the three distances, fit the spatial posture information of the glass plane in the sensor coordinates;

[0024] S3.4. Calculate the angle between the two planes to determine whether fine-tuning is required; compare the calculated rotation angle with a pre-set threshold. If the rotation angle is less than the threshold, proceed to step S3.6; otherwise, proceed to step S3.5.

[0025] S3.5. Guide the robotic arm to perform a reverse rotation operation so that the detection plane and the glass plane are parallel in space;

[0026] S3.6. Calculate the distance from the end of the robotic arm to the glass.

[0027] More preferably, in step S3.3, the plane on which the laser rangefinder is installed is set as the reference plane S, the target plane is set as T, and the equation of the target plane T is z=ax+by+c; the distances measured by the three laser rangefinders are respectively Substituting the coordinates of the three rangefinders (x1, y1), (x2, y2), and (x3, y3) into the target plane equation z = ax + by + c and combining them with the range values, we can obtain the following equations:

[0028]

[0029] Its matrix form is Y=Xβ, where

[0030]

[0031] Since the three ultrasonic sensors are not collinear, the matrix X is invertible, so β ​​= X -1 Y; by solving, we can get the values ​​of a, b, and c Therefore, the normal vector of the target plane T is

[0032] More preferably, in step S3.4, the normal vector of the reference plane is The normal vector of the target plane is When the two normal vectors are not parallel, the rotation axis is the cross product of the two normal vectors, that is:

[0033]

[0034] Normalizing it yields the unit rotation axis

[0035] in

[0036] The rotation angle is calculated by taking the dot product of the two normal vectors:

[0037]

[0038] Compare the calculated θ with the pre-set threshold ∈. When θ is less than ∈, calculate the distance from the end of the robotic arm to the glass; otherwise, go to step S3.5.

[0039] More preferably, in step S3.5, the Rodrigues rotation formula is used to construct the rotation angle along any axis by using the rotation axis and the rotation angle. The rotation matrix for the rotation angle θ:

[0040]

[0041] Where I is the identity matrix,

[0042] The reference plane is multiplied by the rotation matrix to adjust the posture of the end of the robotic arm. After that, it returns to step S3.1 to continue the next round of data collection and comparison.

[0043] More preferably, in step S3.6, calculating the distance from the end of the robotic arm to the glass specifically includes: taking the arithmetic mean of the distances after filtering of the three ultrasonic sensors, that is,

[0044]

[0045] Where, This is the distance the end of the robotic arm needs to move along the z-axis of the coordinate system.

[0046] In addition, the present invention also provides a glass grasping method based on transparent glass posture detection of multimodal data: multiple suction cups are installed at the end of the robot arm, and the transparent glass posture detection method based on multimodal data is used to guide the end of the robot arm to move along the z-axis direction of the end coordinate system. And suck the glass for handling work.

[0047] In addition, the present invention also provides a system for implementing the above-mentioned transparent glass posture detection method based on multimodal data, which includes: a robotic arm execution unit, a monocular camera installed at the end of the robotic arm, at least three non-collinearly arranged ultrasonic sensors, and a data processing unit; the data processing unit is configured to: execute an image feature extraction algorithm, a plane equation solving algorithm and a posture control instruction generation module.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. By combining a monocular camera with multiple non-collinear ultrasonic sensors, the limitations of single sensors in transparent media detection are overcome. Camera vision rapidly captures the glass's contours for coarse positioning, while ultrasonic ranging accurately calculates spatial pose parameters for fine-tuning. The complementary data from these two modalities effectively addresses issues such as weak reflection signals caused by glass's transparency and the failure of traditional sensors (such as LiDAR), significantly improving the detection system's environmental adaptability and reliability.

[0050] 2. Detection is completed using only a bullet screen camera and a conventional ultrasonic sensor, which greatly reduces hardware costs compared to high-precision lidar or structured light equipment.

[0051] 3. No auxiliary marks or equipment need to be installed on the glass surface. Pure non-contact detection is achieved through the combination of monocular camera visual recognition and ultrasonic ranging, which completely avoids scratches, contamination or structural stress damage to the glass surface caused by traditional contact positioning (such as fixture installation) or semi-contact detection (such as affixing reflective marks). It is especially suitable for precision scenes such as electronic glass and optical lenses with high cleanliness requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the position layout of the collection device at the end of the robotic arm in the embodiment;

[0053] Figure 2 Schematic diagram of the camera calibration process in an embodiment;

[0054] Figure 3 A schematic diagram of collecting image data and extracting contour features of glass in an embodiment;

[0055] Figure 4 Schematic diagram of glass posture information extracted in the embodiment;

[0056] Figure 5 Schematic diagram of the process principle of coarse adjustment of the robot arm posture in the embodiment;

[0057] Figure 6 Schematic diagram of three non-collinear ultrasonic sensors collecting glass information during fine adjustment of the end-arm posture in an embodiment;

[0058] Figure 7 Schematic diagram of the process principle of fine-tuning the posture of the robotic arm in the embodiment. DETAILED DESCRIPTION

[0059] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0060] A method for detecting the position and posture of transparent glass based on multimodal data comprises the following steps:

[0061] S1. Install a data acquisition device at the end of the robotic arm. The data acquisition device includes a monocular camera and multiple ordinary ultrasonic sensors arranged non-collinearly.

[0062] Specifically, if Figure 1 As shown, a designed fixture is first installed on the flange at the end of the robot arm. The fixture is a rectangular plane with the flange mounted at the center of the rectangle. The fixture plane normal is parallel to the flange direction, and the fixture plane border is parallel to the flange coordinate axis. A camera and ultrasonic sensor are both mounted next to the flange at the end of the robot arm. The camera optical axis direction vector and the ultrasonic detection direction vector are parallel to the normal vector of the flange plane at the end of the robot arm. There are three ultrasonic sensors, and their installation positions on the fixture plane are non-collinear.

[0063] At the same time, the coordinates of the three ultrasonic sensors on the flange plane must be recorded for later posture calculation. The coordinates are based on the flange position as the origin, with the horizontal x-axis and the vertical y-axis. The coordinates of the three ultrasonic sensors are P1 = (x1, y1, 0), P2 = (x2, y2, 0), and P3 = (x3, y3, 0).

[0064] The camera and ultrasonic sensor act as data acquisition functions in the entire system. By analyzing the data collected by the camera and ultrasonic sensor, the posture of the robotic arm is adjusted so that it can correctly adsorb the target glass.

[0065] Step S1 also includes the equipment installation and calibration process, specifically: Figure 2 As shown in the figure, the goal of camera calibration is to accurately determine the transformation relationship between the camera coordinate system and the robot arm end flange coordinate system. This is accomplished by placing a calibration object with known features, such as a checkerboard, in the world coordinate system. The robot arm is then guided to move the camera to different positions, capturing multiple images of the calibration object. By detecting the pixel coordinates (u, v) of feature points in the image and the known coordinates (X_w, Y_w, Z_w) of the calibration object feature points in the world coordinate system, the position of the robot arm end effector in the world coordinate system is determined by combining the robot arm kinematic model. Using this coordinate transformation relationship and the camera imaging model, a set of equations is established. By solving multiple sets of these equations, the transformation matrix between the camera coordinate system and the robot arm end effector coordinate system can be obtained. Commonly used solution algorithms include the Tsai-Lenz method and the Park method.

[0066] S2. Rough adjustment of the robot arm posture.

[0067] By detecting the posture of the glass, the position and posture of the end of the robot arm in space are adjusted in the opposite direction so that the robot arm flange is at the geometric center of the glass and the flange plane normal vector is roughly parallel to the glass normal vector.

[0068] Specifically, the following steps are included:

[0069] S2.1, data collection. Figure 3 As shown, image data is collected and the contour features of the glass are extracted.

[0070] S2.2, extract the posture information of the glass. Figure 4 As shown, the geometric center point of the glass and the distortion of the glass in the camera image are calculated using the contour information of the glass, including the geometric structure of the glass imaging (parallelogram and trapezoid, and related parameters such as internal angles).

[0071] The specific parameters include the bottom angle θ, top and bottom, and left and right. Through these parameters, we can roughly determine the relative position parameters between the glass and the camera. The specific method is:

[0072] (1) Parallelogram imaging:

[0073] Tilt Angle: The acute or obtuse angles of a parallelogram reflect the degree of tilt between the camera's optical axis and the object plane. By measuring the parallelogram's interior angles, the angle between the camera's optical axis and the normal to the object plane can be calculated. Assuming one interior angle of the parallelogram is θ, the angle between the camera's optical axis and the object plane is α = 90° - θ.

[0074] Horizontal offset: The extended lines of the two hypotenuses of a parallelogram intersect at a point. The direction of the line connecting this point and the center of the rectangular object indicates the horizontal offset of the camera relative to the object. If the intersection is to the right of the object's center, the camera is diagonally above or below the left side of the object; otherwise, the camera is diagonally above or below the right side.

[0075] (2) Trapezoid imaging:

[0076] The ratio of the lengths of the upper and lower bases of the trapezoid is related to the angle between the camera's optical axis and the object plane, as well as the camera's height. The greater the difference in base lengths, the greater the angle between the camera's optical axis and the object plane, and the lower the camera position. Let the upper base of the trapezoid be a and the lower base be b, which can be calculated by To estimate the angle β between the camera optical axis and the object plane, where h is the vertical distance from the camera to the center of the object. If the distance is unknown, the relevant angle can be roughly estimated.

[0077] Hypotenuse Tilt: The tilt of the two hypotenuses of a trapezoid reflects the horizontal offset of the camera. If the two hypotenuses tilt to the right, the camera is diagonally above or below the left side of the object; if they tilt to the left, the camera is diagonally above or below the right side.

[0078] S2.3. Threshold comparison. Estimate the difference between the glass's geometric center and the flange's plane center based on the position of the glass's geometric center in the image. Estimate the angle between the glass's normal vector and the flange's plane normal vector based on the glass's image distortion. If both are less than the set threshold, terminate the coarse pose adjustment. Otherwise, proceed to step S2.4.

[0079] S2.4. Adjust the robotic arm's posture. Send a command to adjust the robotic arm's end-of-arm posture. Through rotation and translation, make the flange plane normal vector roughly parallel to the glass normal vector, and the normal vector at the center of the flange plane point to the geometric center of the glass. Continue extracting information from step S2.1.

[0080] First, the end of the robotic arm is translated so that the geometric center of the glass in the camera coincides with the center of the camera image. The mathematical model of the translation operation is:

[0081] P d =P0+ΔP

[0082] Among them, P dis the flange coordinate after movement, P0 is the flange coordinate before movement, ΔP=[Δx,Δy,Δz] T are the estimated movement parameters.

[0083] Secondly, the end of the robot arm is rotated so that the normal vector of the end flange is parallel to the glass plane. The mathematical model of the rotation operation is:

[0084] R new =R rot R0

[0085] Among them, R0 is the posture information of the end flange before rotation, R new is the attitude information of the end flange after rotation, R rot is the rotation matrix. Here R rot is the rotation matrix formed by rotating the end flange along the three axes, namely:

[0086] R rot =R z (γ)R y (β)R x (α)

[0087] R x (α) is the rotation matrix of the end flange along the x-axis. The rotation matrix is:

[0088]

[0089] R y (β) is the rotation matrix of the end flange along the y-axis. The rotation matrix is:

[0090]

[0091] R z (γ) is the rotation matrix of the end flange along the z-axis. The rotation matrix is:

[0092]

[0093] Because of the position information between the camera and the glass, only an iterative method is used here for adjustment. That is, after determining the approximate relative relationship between the camera and the glass, the minimum angle parameter is used to rotate the end of the robotic arm so that it gradually rotates toward the direction of the glass normal vector.

[0094] The above is the method for coarse adjustment of the end-arm posture. The whole process is as follows: Figure 5 This method uses a common camera to quickly adjust the position of the end of the robotic arm so that the end flange is roughly located directly above the geometric center of the glass and its direction is roughly parallel to the normal vector of the glass, thus providing a relatively good state for subsequent fine-tuning.

[0095] S3. Fine-tune the robotic arm's posture (fine adjustment).

[0096] Since the rough adjustment of the robot arm posture only corrects the posture of the end of the robot arm by the distortion of the glass imaging, there is a certain error in this method, and the distance information between the end of the robot arm and the glass cannot be obtained by the computer vision method. Therefore, the fine adjustment of the robot arm posture is introduced here, such as Figure 6 As shown, the robot arm posture is calibrated by three non-collinear ultrasonic sensors, and the precise distance between the end of the robot arm and the glass is obtained.

[0097] The process of fine adjustment of the robot arm posture is as follows:

[0098] S3.1. In a stationary state, three coplanar ultrasonic sensors simultaneously detect the distance to the glass and record it as D 1_t , D 2_t and D 3_t .

[0099] S3.2, after the time interval of Δt, continue with step 1 and save the new distance information as D 1_t+1 , D 2_t+1 and D 3_t+1 , repeated N times in total (where N is a hyperparameter, which is the filter window width set by the system). The noise of the sensor is removed by mean filtering. The specific operation method is as follows:

[0100]

[0101] in, is the distance value after filtering of the i-th ultrasonic sensor, N is the window length of the mean filter, D i_t is the distance value of the i-th ultrasonic sensor at time t.

[0102] S3.3. Using the differences between the three distances, fit the spatial posture information of the glass plane in the sensor coordinates;

[0103] Assume that the plane where the laser rangefinder is installed is the reference plane S, the target plane is T, and the equation of the target plane T is z=ax+by+c. The distances measured by the three laser rangefinders are Substituting the coordinates of the three rangefinders (x1, y1), (x2, y2), and (x3, y3) into the target plane equation z = ax + by + c and combining them with the range values, we can obtain the following equations:

[0104]

[0105] Its matrix form is Y=Xβ, where

[0106]

[0107] Since the three ultrasonic sensors are not collinear, the matrix X is invertible, so β ​​= X -1 Y. By solving, we can get the values ​​of a, b, and c Therefore, the normal vector of the target plane T is

[0108] S3.4. Calculate the angle between the two planes and determine whether fine-tuning is required.

[0109] Let the normal vector of the reference plane be The normal vector of the target plane is

[0110] When the two normal vectors are not parallel, the rotation axis is the cross product of the two normal vectors, that is,

[0111]

[0112] Normalizing it yields the unit rotation axis

[0113] in

[0114] The rotation angle can be calculated by taking the dot product of the two normal vectors:

[0115]

[0116] Here, the calculated θ is compared with the pre-set threshold ∈. When θ is less than ∈, proceed to step S3.6; otherwise, proceed to step S3.5.

[0117] S3.5. Guide the forearm of the robotic arm to rotate in the opposite direction so that the detection plane and the glass plane are parallel in space.

[0118] By using the rotation axis and the rotation angle, the Rodriguez rotation formula can be used to construct a rotation along any axis. The rotation matrix for the rotation angle θ.

[0119]

[0120] Where I is the identity matrix,

[0121] The reference plane is multiplied by the rotation matrix to adjust the posture of the end-arm. After that, the process returns to step S3.1 to continue the next round of data collection and comparison.

[0122] S3.6. Calculate the distance from the end of the robotic arm to the glass.

[0123] Take the arithmetic mean of the filtered distances of the three ultrasonic sensors, that is,

[0124]

[0125] Where, This is the distance the end of the robotic arm needs to move along the z-axis of the coordinate system.

[0126] At this point, the glass posture detection has been basically completed. When the glass is subsequently grasped, it is only necessary to guide the end of the robot arm to move along the z-axis direction of the end coordinate system on this basis. The glass is picked up by multiple suction cups installed at the end of the robot arm to carry out the handling work. The whole process of fine adjustment of the robot arm posture and picking up the glass is as follows Figure 7 shown.

[0127] The transparent glass posture detection method based on multimodal data provided in the above embodiment constructs a multimodal detection system by fusing visual data from a monocular camera with ranging data from multiple non-collinear ultrasonic sensors. First, the camera captures the geometric distortion characteristics of the glass outline, analyzes its relative posture relationship with the camera plane, and drives the robotic arm to coarsely adjust to a roughly parallel posture. Then, based on the distance information collected by the ultrasonic sensor array, the glass plane equation is fitted through spatial geometry calculations to accurately obtain the normal vector direction and distance parameters of the glass, achieving closed-loop fine-tuning of the robotic arm's posture. The entire method achieves posture detection of transparent glass in three-dimensional space through the collaboration of non-contact data acquisition and algorithms.

[0128] This invention overcomes the detection bottleneck of transparent media's difficulty reflecting optical signals. High-precision position detection can be achieved without installing auxiliary equipment on the glass surface, thus avoiding damage to the glass surface caused by contact marking. Multimodal data complementation significantly improves detection reliability, while low-cost sensors replace high-precision laser equipment, achieving both industrial-grade accuracy and affordability. The fully automated process is compatible with high-speed production lines, and the output of standardized position parameters can directly drive a robotic arm to perform grasping, offering broad application value in fields such as glass manufacturing and electronic assembly.

[0129] In order to make it easier for ordinary technicians in this field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and the above-mentioned embodiments are preferred implementation schemes of the present invention. In addition, the present invention can also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the scope of protection of the present invention.

Claims

1. A transparent glass posture detection method based on multimodal data, characterized in that: The following steps are involved: S1. Install a data acquisition device at the end of the robotic arm, wherein the data acquisition device includes a monocular camera and multiple non-collinearly arranged ultrasonic sensors; S2. Collecting contour image data of the target glass using the monocular camera, extracting the geometric center and imaging distortion parameters of the glass based on image features, calculating a first relative pose relationship between the glass plane and the camera plane, and driving the robotic arm to perform coarse pose adjustment so that the flange plane at the end of the robotic arm and the glass plane are initially parallel; S3, synchronously collecting distance data of the glass using multiple ultrasonic sensors, solving a second relative posture relationship of the glass plane in the flange coordinate system based on the multi-sensor distance information, calculating the plane normal vector deviation and distance parameters, and driving the robotic arm to fine-tune the posture until a preset parallel threshold is met; S4. Output the posture parameters of the glass plane relative to the end of the robotic arm, where the posture parameters include normal vector direction and distance information, so that the robotic arm can perform a subsequent grasping operation.

2. The transparent glass posture detection method based on multimodal data according to claim 1, characterized in that: Step S1 also includes an equipment installation calibration process: determining the spatial transformation relationship between the monocular camera and the flange coordinate system of the end of the robotic arm through the calibration object, and recording the installation coordinate parameters of each ultrasonic sensor in the flange plane.

3. The transparent glass posture detection method based on multimodal data according to claim 1, characterized in that: Driving the robotic arm to perform rough posture adjustment in step S2 specifically includes the following steps: S2.1, data acquisition: collecting image data and extracting the contour features of the glass; S2.

2. Extracting Glass Posture Information: Using the glass's contour information, calculate the geometric center point of the glass and the distortion of the glass in the camera image, including the geometric structure of the glass image. Specific parameters include: bottom angle, tilt angle, horizontal offset, upper and lower bottom edge length ratio, and hypotenuse tilt direction. S2.3, threshold comparison: Estimate the difference between the geometric center of the glass and the center of the flange plane through the position of the geometric center of the glass in the image; Estimate the angle between the normal vector of the glass and the normal vector of the flange plane through the distortion of the glass imaging; If both are less than the set threshold, terminate the coarse adjustment of the posture, otherwise proceed to step S2.

4. S2.

4. Adjust the position of the robotic arm: Send a command to adjust the position of the end of the robotic arm. Through rotation and translation operations, make the normal vector of the flange plane roughly parallel to the normal vector of the glass, and the normal vector at the center of the flange plane points to the geometric center of the glass; and continue with data collection in step S2.

1.

4. The transparent glass posture detection method based on multimodal data according to claim 1, characterized in that: In step S3, the robot arm is driven to fine-tune its posture, which specifically includes the following steps: S3.

1. In a stationary state, three coplanar ultrasonic sensors are used to simultaneously detect the distance to the glass, and record it as D 1_t , D 2_t and D 3_t ; S3.2, after a time interval of Δt, continue with step S3.1, and save the new distance information as D 1_t+1 , D 2_t+1 and D 3_t+1 , repeat N times in total. Remove the noise of the sensor through mean filtering. The specific operation method is as follows: in, is the distance value after filtering of the i-th ultrasonic sensor, N is the window length of the mean filter, D i_t is the distance value of the i-th ultrasonic sensor at time t; S3.

3. Using the differences between the three distances, fit the spatial posture information of the glass plane in the sensor coordinates; S3.

4. Calculate the angle between the two planes to determine whether fine-tuning is required; compare the calculated rotation angle with a pre-set threshold. If the rotation angle is less than the threshold, proceed to step S3.6; otherwise, proceed to step S3.

5. S3.

5. Guide the robotic arm to perform a reverse rotation operation so that the detection plane and the glass plane are parallel in space; S3.

6. Calculate the distance from the end of the robotic arm to the glass.

5. The transparent glass posture detection method based on multimodal data according to claim 4, characterized in that: In step S3.3, let the plane where the laser rangefinder is installed be the reference plane S, the target plane be T, and the equation of the target plane T be z=ax+by+c; the distances measured by the three laser rangefinders are Substituting the coordinates of the three rangefinders (x1, y1), (x2, y2), and (x3, y3) into the target plane equation z = ax + by + c and combining them with the range values, we can obtain the following equations: Its matrix form is Y=Xβ, where Since the three ultrasonic sensors are not collinear, the matrix X is invertible, so β ​​= X -1 Y; by solving, we can get the values ​​of a, b, and c Therefore, the normal vector of the target plane T is 6. The transparent glass posture detection method based on multimodal data according to claim 5, characterized in that: In step S3.4, let the normal vector of the reference plane be The normal vector of the target plane is When the two normal vectors are not parallel, the rotation axis is the cross product of the two normal vectors, that is: Normalizing it yields the unit rotation axis in The rotation angle is calculated by taking the dot product of the two normal vectors: Compare the calculated θ with the pre-set threshold ∈. When θ is less than ∈, calculate the distance from the end of the robotic arm to the glass; otherwise, go to step S3.

5.

7. The transparent glass posture detection method based on multimodal data according to claim 6, characterized in that: In step S3.5, the Rodriguez rotation formula is used to construct the rotation angle along any axis by using the rotation axis and the rotation angle. The rotation matrix for the rotation angle θ: Where I is the identity matrix, The reference plane is multiplied by the rotation matrix to adjust the posture of the end of the robotic arm. After that, it returns to step S3.1 to continue the next round of data collection and comparison.

8. The transparent glass posture detection method based on multimodal data according to claim 7, characterized in that: In step S3.6, the distance between the end of the robot arm and the glass is calculated by taking the arithmetic mean of the filtered distances of the three ultrasonic sensors, i.e. Where, This is the distance the end of the robotic arm needs to move along the z-axis of the coordinate system.

9. A glass grasping method based on transparent glass posture detection using multimodal data, characterized by: A plurality of suction cups are installed at the end of the robotic arm, and the transparent glass posture detection method based on multimodal data as described in claim 8 is used to guide the end of the robotic arm to move along the z-axis direction of the end coordinate system. And suck the glass for handling work.

10. A system for implementing the transparent glass posture detection method based on multimodal data according to any one of claims 1 to 8, characterized in that: include: A robotic arm execution unit, a monocular camera mounted on the end of the robotic arm, at least three non-collinearly arranged ultrasonic sensors, and a data processing unit; The data processing unit is configured to execute an image feature extraction algorithm, a plane equation solving algorithm, and a posture control instruction generation module.

Images