Vibrating mechanical arm normal insertion posture self-adaption method and system facing inclined bin face and medium

Abstract

The invention discloses an adaptive method and system for a normal insertion attitude of a vibrating mechanical arm facing an inclined bin surface and a medium, and the method comprises the steps: collecting a color image and a depth image through a depth camera carried at the tail end of the vibrating mechanical arm, carrying out the spatial reconstruction, and generating a point cloud; generating a surface normal vector of a local point cloud area by combining internal parameters of a camera, pixel coordinates of a vibrating point and depth information, uniformly reversing the surface normal vector downwards as an insertion direction of a vibrating rod, mapping the normal vector to a base coordinate system of the mechanical arm through coordinate transformation, solving a tail end attitude angle, and obtaining a target point cloud area; and posture self-adaptive adjustment and accurate insertion of the vibrating rod in the normal direction of the local area of the insertion point are achieved. According to the method, the problem that the insertion direction of the vibrating rod under the bin surface with any dip angle is inconsistent with the normal direction of the local area of the insertion point is solved, the construction specification requirement that the vibrating rod needs to be vertically inserted in the normal direction of the bin surface in the vibrating operation is met, and the compactness, uniformity and intelligent level of concrete vibrating are remarkably improved; and the method has good robustness and engineering adaptation value.

Description

TECHNICAL FIELD

[0001] The application relates to the field of construction robots, in particular to a normal insertion posture self-adaptive method, system and medium for a vibrating mechanical arm facing an inclined warehouse surface. BACKGROUND

[0002] Concrete vibration is a key process for improving structure compactness and durability. The insertion type vibrating rod is inserted vertically into the concrete and vibrated at high frequency, effectively eliminating air bubbles and pores, so as to ensure the full compactness of the concrete. At present, the construction site mainly relies on manual operation, and the insertion angle and depth of the vibrating rod are mainly judged by the experience of the operator, so it is difficult to realize the standardization and intelligent control of the operation, and the vibration quality has obvious uncertainty and risk.

[0003] With the development of intelligent construction technology, the insertion type vibrating robot is gradually applied to the field of concrete construction, and has shown obvious advantages in improving operation accuracy and reducing human error. However, "vertical insertion" is still a problem that the current intelligent vibrating system has not effectively solved. According to the concrete construction specification, the vibrating rod should be inserted along the normal direction of the formwork surface to ensure that the insertion force is perpendicular to the pouring surface, so as to maximize the elimination of the air area between the reinforcement and the formwork, improve the overall compactness and durability of the concrete structure, and avoid quality defects such as interlayer, honeycomb and rough surface.

[0004] However, in the actual construction site, the formwork surface often has an inclined, curved or irregular geometry, and the insertion is mainly based on experience in the manual construction process, so it is difficult to ensure that the insertion direction is consistent with the local normal direction of the formwork, which leads to unstable vibration effect. The existing robot vibrating system also generally adopts a fixed vertical insertion strategy, lacks the ability to perceive the spatial posture of the vibrating point surface, and cannot adaptively adjust according to the actual inclination angle of the target point. Such deviated insertion path not only affects the vibration depth and contact angle, but also may cause vibration force deviation, insufficient regional vibration and other problems, which seriously affect the construction quality. At present, there is no mature method to realize the normal self-adaptive adjustment of the insertion posture of the vibrating mechanical arm in the complex inclined formwork environment. SUMMARY

[0005] The purpose of the present application is to provide a vibrating mechanical arm normal insertion posture self-adaptive method capable of realizing the normal self-adaptive adjustment of the insertion posture of the vibrating mechanical arm in the complex inclined formwork environment; another purpose of the present application is to provide a vibrating mechanical arm normal insertion posture self-adaptive system and medium facing an inclined warehouse surface.

[0006] Technical scheme: the vibrating mechanical arm normal insertion posture self-adaptive method facing an inclined warehouse surface provided by the present application comprises the following steps:

[0007] (1) Collect color image and depth image of the construction surface to be constructed by the depth camera installed at the end of the vibrating mechanical arm, and construct a three-dimensional point cloud model under the camera coordinate system based on the color image and the depth image;

[0008] (2) Perform instance segmentation on the color image to extract the candidate region of the concrete region, determine the target vibrating point meeting the spacing constraint based on mask optimization, connected region analysis and point layout algorithm, and obtain the pixel coordinate set of the target vibrating point in the pixel space.

[0009] (3) Calibrate the depth camera to obtain the intrinsic matrix of the depth camera, convert the pixel coordinate set of the target vibrating point obtained in step (2) into a three-dimensional space coordinate set of the target vibrating point under the camera coordinate system combined with the depth information of the target vibrating point, and associate the three-dimensional space coordinate set of the target vibrating point with the three-dimensional point cloud model in step (1) to determine the point cloud position corresponding to the target vibrating point.

[0010] (4) For each target vibrating point, extract the local point cloud in its neighborhood, and determine the normal vector of the target vibrating point by principal component analysis method.

[0011] (5) Obtain the transformation matrix between the depth camera coordinate system and the mechanical arm base coordinate system through hand-eye calibration, and convert the normal vector of the target vibrating point obtained in step (4) from the camera coordinate system to the insertion normal vector under the mechanical arm base coordinate system.

[0012] (6) Combine the initial pose of the current mechanical arm end tool to calculate the minimum rotation path when the Z axis of the mechanical arm end is aligned with the insertion normal vector, and obtain the target pose angle (rx, ry, rz).

[0013] (7) Determine the vibrating point pose based on the target pose angle and the three-dimensional space coordinate of the target vibrating point under the camera coordinate system, calculate the vibrating path through the mechanical arm control system, and realize adaptive inclined insertion in the direction of the insertion normal vector.

[0014] (8) If the target vibrating point is converted, execute the vibration; otherwise, return to step (4).

[0015] Further, in step (1), the three-dimensional point cloud model under the camera coordinate system is constructed based on the color image and the depth image, specifically as follows: by combining the color image coordinates of each pixel point with its corresponding depth value, using the depth camera intrinsic matrix for spatial projection, calculating the three-dimensional point cloud coordinates (x, y, z) of each pixel point, and obtaining the three-dimensional point cloud model under the camera coordinate system.

[0016] Further, in step (2), the color image is instance segmented to extract the candidate region of the concrete region, and based on mask optimization, connected region analysis and point layout algorithm, the target vibrating point satisfying the spacing constraint is determined to obtain the pixel coordinate set of the target vibrating point in the pixel space, which is specifically as follows:

[0017] (21) A YOLO concrete instance segmentation model integrated with an attention mechanism is constructed, and the training of the model is completed based on the training set;

[0018] (22) The color image is input into the trained YOLO concrete instance segmentation model to perform instance segmentation, extract the candidate region of the concrete region and mask optimization, and generate a vibrating region instance mask graph;

[0019] (23) The vibrating region instance mask graph is subjected to connected region analysis and adhesion segmentation, and a point layout algorithm based on geometric centroid calculation and point optimization strategy is combined to obtain all target vibrating points satisfying the spacing constraint.

[0020] Further, step (4) is specifically as follows:

[0021] Taking the three-dimensional coordinates of each target vibrating point in the camera coordinate system as the center, the local point cloud data is searched in the spherical neighborhood with a radius of R, the point set based on KD tree index is constructed, and the eigenvalue decomposition of the covariance matrix of the neighborhood points is performed by principal component analysis method, the feature vector corresponding to the minimum eigenvalue is selected, and it is judged whether the Z-axis direction of the feature vector is upward or not. If it is upward, the direction of the feature vector is reversed to ensure that the feature vector is uniformly directed downward along the normal direction of the concrete surface as the normal vector of the target vibrating point.

[0022] Further, in step (5), the hand-eye calibration is specifically as follows:

[0023] A depth camera is fixedly installed at the end of the mechanical arm, a nine-point calibration board is placed in the working area of the mechanical arm, a plurality of point pairs are established by image acquisition and actual end pose, and the rigid transformation matrix from the camera coordinate system to the mechanical arm base coordinate system is solved by using the registration of the three groups of data of image points, depth information and mechanical arm end pose, which is used for coordinate conversion between the camera coordinate system and the mechanical arm base coordinate system.

[0024] Further, step (6) is specifically as follows:

[0025] The minimum rotation vector between the Z-axis direction of the initial pose coordinate system of the current mechanical arm end tool and the normal vector of the target vibrating point is calculated, and a rotation increment matrix is constructed. The rotation increment matrix and the initial pose matrix of the mechanical arm end tool are multiplied to obtain the target pose matrix. The target pose angle (rx, ry, rz) is obtained through Euler angle transformation, wherein the Euler angle rotation sequence is x, y and z, and the unit is degree.

[0026] Further, the hand-eye calibration method adopts a three-dimensional point correspondence calibration method based on a nine-point calibration board, and specifically as follows:

[0027] Place the nine-point calibration board in the depth camera field of view, complete one image acquisition under the condition of no distortion image, and extract the pixel coordinates of the nine calibration points in the image;

[0028] Control the end effector of the mechanical arm, i.e., the end of the vibrating rod, to accurately contact the nine physical points on the nine-point calibration board in turn, record the space pose coordinates of each point in the mechanical arm base coordinate system through the mechanical arm feedback, and construct a corresponding three-dimensional point set in the mechanical arm coordinate system;

[0029] Based on the nine-point calibration board coordinate system, three sets of point sets are constructed: the point set in the camera coordinate system obtained through camera image calibration, the point set in the mechanical arm base coordinate system obtained through the end of the mechanical arm, and the point set in the workboard coordinate system known on the nine-point calibration board;

[0030] A least square rigid registration algorithm based on singular value decomposition or iterative closest point method is used to calculate the transformation matrix of the mechanical arm base coordinate system to the workboard coordinate system, and a PnP-RANSAC algorithm is used to solve the transformation matrix of the camera coordinate system to the workboard coordinate system;

[0031] Based on the above two transformation matrices, the spatial transformation relationship between the camera coordinate system and the mechanical arm base coordinate system is solved, and is saved as a hand-eye calibration matrix.

[0032] The normal insertion posture adaptive system of the vibrating mechanical arm facing the inclined bin surface, comprises:

[0033] A three-dimensional point cloud model construction module is configured to acquire color images and depth images of a bin surface to be constructed by a depth camera installed at the end of the vibrating mechanical arm, and construct a three-dimensional point cloud model in the camera coordinate system based on the color images and the depth images;

[0034] A target vibrating point positioning module is configured to perform instance segmentation on the color images, extract candidate regions of the concrete area, determine target vibrating points satisfying the spacing constraint based on mask optimization, connected region analysis and point layout algorithm, and obtain a pixel coordinate set of the target vibrating points in the pixel space;

[0035] A conversion and association module is configured to calibrate the depth camera to obtain an intrinsic matrix of the depth camera, convert the pixel coordinate set of the target vibrating points into a three-dimensional space coordinate set of the target vibrating points in the camera coordinate system in combination with the depth information of the target vibrating points, associate the three-dimensional space coordinate set of the target vibrating points with the three-dimensional point cloud model of step (1), and determine the point cloud position corresponding to the target vibrating points;

[0036] A normal vector estimation module is configured to extract a local point cloud in a neighborhood of each target vibrating point, and determine a normal vector of the target vibrating point by using a principal component analysis method.

[0037] A vector conversion module is configured to convert the normal vector of the target vibrating point from a camera coordinate system to an insertion normal vector in a base coordinate system of a robot arm by using a transformation matrix between the camera coordinate system and the base coordinate system of the robot arm obtained by hand-eye calibration.

[0038] A pose angle solving module is configured to calculate a minimum rotation path for aligning a Z-axis of an end tool of the robot arm with the insertion normal vector, and obtain a target pose angle (rx, ry, rz) in combination with an initial pose of the end tool of the robot arm.

[0039] A vibrating path calculation module is configured to determine a vibrating point pose based on the target pose angle and three-dimensional space coordinates of the target vibrating point in the camera coordinate system, and calculate a vibrating path by using a robot arm control system, so that the vibrating rod is inserted in a direction of the insertion normal vector to achieve adaptive tilt insertion.

[0040] A checking module is configured to determine whether the target vibrating point is converted, and perform vibrating if the target vibrating point is converted; otherwise, the next vibrating point is converted.

[0041] The computer device comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the method when executing the computer program.

[0042] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method.

[0043] Advantages: Compared with the prior art, the method has the following advantages: 1. The method solves the problem that the insertion direction of the vibrating rod is inconsistent with the local area normal of the insertion point at any inclination angle, and meets the construction specification requirement that the vibrating rod needs to be inserted vertically along the normal of the surface; 2. The method is suitable for the concrete formwork environment with inclination, curved surface and irregular structure, has the functions of adaptive adjustment of the pose, controllability of the path and matching of the construction specification, and can significantly improve the standardization and intelligent level of the vibrating operation. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the method is shown in the figure;

[0045] Figure 2 The three-dimensional point cloud data generated by superimposing the color image and the depth image collected by the depth camera is shown in the figure;

[0046] Figure 3 The visualization diagram of the local point cloud neighborhood range extracted for the target vibrating point is shown in the figure;

[0047] Figure 4 The schematic diagram of the normal vector estimation result of the target vibration point method; Figure 4 (a) in the figure is a rear view, Figure 4 (b) in the figure is a front view, Figure 4 (c) in the figure is a right view, Figure 4 (d) in the figure is a left view. DETAILED DESCRIPTION

[0048] The application will be further described below in combination with the drawings.

[0049] The normal insertion posture adaptive method of the vibration mechanical arm facing the inclined bin surface, comprising the following steps:

[0050] (1) Hand-eye calibration: in the system initialization stage, first complete the coordinate transformation relationship establishment of the camera and the mechanical arm. Fix the depth camera at the end of the vibration mechanical arm, and set a nine-point calibration board in the working area. Extract the pixel coordinates of the nine calibration points in the image in turn. Combined with the camera internal parameter and the depth value, the three-dimensional point set in the camera coordinate system is obtained by back projection. At the same time, control the mechanical arm to make its end tool contact the known coordinate points on the nine-point board one by one, and obtain the corresponding three-dimensional coordinate points in the mechanical arm base coordinate system. The SVD method is used to rigidly register the two groups of point sets, and the 4x4 transformation matrix from the camera coordinate system to the mechanical arm coordinate system is solved, which is saved as the hand-eye calibration result.

[0051] The hand-eye calibration method adopts a three-dimensional point corresponding calibration method based on a nine-point calibration board, which specifically comprises the following steps:

[0052] Place the nine-point calibration board with nine known spatial coordinate points in the field of view of the depth camera, complete an image acquisition under the condition of no distortion image, and extract the pixel coordinates of the nine calibration points in the image;

[0053] Control the end effector (vibration rod end) of the mechanical arm to accurately contact the nine physical points on the nine-point calibration board in turn, record the spatial pose coordinates of each point in the mechanical arm base coordinate system through the mechanical arm feedback, and construct the corresponding three-dimensional point set in the mechanical arm coordinate system;

[0054] Based on the nine-point calibration board coordinate system, three groups of point sets are constructed: the point set in the camera coordinate system obtained by camera image calibration; the point set in the mechanical arm base coordinate system obtained by the mechanical arm end contact; and the point set in the working board coordinate system known on the nine-point calibration board.

[0055] The least square rigid registration algorithm based on singular value decomposition (SVD) or iterative closest point method (ICP) is used to calculate the transformation matrix from the mechanical arm base coordinate system to the working board coordinate system, and the transformation matrix from the camera coordinate system to the working board coordinate system is solved by using the PnP-RANSAC algorithm subsequently.

[0056] Based on the above two transformation matrices, the spatial transformation relationship between the camera coordinate system and the robot base coordinate system is solved, and is saved as a hand-eye calibration matrix, which is used to accurately convert subsequent image data, point cloud data, and normal vector information from the camera coordinate system to the base coordinate system used by the robot control system.

[0057] (2) Image acquisition and point cloud reconstruction: Before the formal vibration operation, the color image and depth image of the construction area are collected by using the depth camera. Combined with the two-dimensional coordinates and depth values of each pixel point, the camera intrinsic matrix is used for back projection to construct a dense point cloud model in the camera coordinate system. The point cloud is used for subsequent spatial feature extraction and normal estimation.

[0058] (3) Vibration area segmentation and candidate point extraction: The color image is input into the pre-trained concrete instance segmentation model YOLO11n-seg, and the vibration area instance mask graph is output. The mask is analyzed for connected regions to identify each independent concrete region. The adhesion region is analyzed and processed: if the minimum bounding rectangle of the mask exceeds the dynamic threshold, it is judged as an adhesion region; the shallow adhesion region is segmented by using the midpoint connection method, and the deep adhesion region is segmented by using the concave point segmentation method, and the main shaft of the concave point is extracted for region segmentation. For each independent region, the geometric centroid is calculated as a candidate vibration point. After obtaining the candidate point set, based on the geometric constraint and coverage efficiency, a double-standard greedy algorithm is used for point selection and layout optimization. The algorithm starts from the top-left target point, preferentially selects the farthest point in the uncovered region, and selects the farthest point from the selected point set in the point with the highest coverage efficiency, and finally obtains the target vibration point set that meets the spacing specification.

[0059] (4) Vibration point space positioning: The vibration point pixel coordinates and its depth value are combined to obtain the three-dimensional space position in the camera coordinate system through back projection. The coordinates are converted to the robot base coordinate system through the hand-eye calibration matrix, which is used as the target position for subsequent insertion control.

[0060] (5) Surface normal estimation: Taking the three-dimensional coordinates of each vibration point as the center, a spherical neighborhood with a radius of 20 mm is constructed in its point cloud. The neighborhood point set is extracted through KD tree indexing, and the covariance matrix is calculated. The eigenvalue decomposition is performed by using the PCA method, and the feature vector corresponding to the minimum eigenvalue is extracted as the normal vector of the region. The normal vector Z direction is judged, and if it is upward, it is reversed to make it point to the inside of the template.

[0061] (6) Attitude angle solution: after obtaining the current attitude of the end, the rotation relationship between the Z-axis direction vector and the normal vector is constructed. The rotation matrix is constructed by using the axis-angle method, and is multiplied by the current end pose matrix to obtain the target attitude matrix. Finally, the target attitude matrix is converted into Euler angles (rx, ry, rz), and the rotation order is xyz.

[0062] (7) Insertion path planning and execution: in the base coordinate system of the robot arm, the insertion and extraction path is planned based on the spatial position of the vibrating point and its normal vector. The approach point is a point 200mm offset from the vibrating point along the negative normal direction, the insertion point is a point 300mm offset along the positive direction, and the retreat point is the same as the approach point. The path points and target attitude angles are input into the robot arm control system to execute the automatic approach, insertion and retreat path, guiding the vibrating rod to complete the adaptive insertion along the normal direction.

[0063] The method can effectively adapt to any inclined template, significantly improve the compactness, uniformity and intelligent level of concrete vibration, and has good robustness and engineering adaptability value

[0064] The normal insertion attitude adaptive system of the vibrating robot arm facing the inclined bin surface comprises:

[0065] A three-dimensional point cloud model construction module is used to collect color images and depth images of the to-be-constructed bin surface through a depth camera installed at the end of the vibrating robot arm, and a three-dimensional point cloud model in the camera coordinate system is constructed based on the color images and the depth images;

[0066] A target vibrating point positioning module is used to perform instance segmentation on the color images, extract candidate regions of the concrete region, determine target vibrating points meeting the spacing constraint based on mask optimization, connected region analysis and point layout algorithm, and obtain a pixel coordinate set of the target vibrating points in the pixel space;

[0067] A conversion association module is used to calibrate the depth camera to obtain an intrinsic matrix of the depth camera, convert the pixel coordinate set of the target vibrating points into a three-dimensional space coordinate set of the target vibrating points in the camera coordinate system in combination with the depth information of the target vibrating points, associate the three-dimensional space coordinate set of the target vibrating points with the three-dimensional point cloud model of step (1) in point position, and determine the point cloud position corresponding to the target vibrating point;

[0068] A normal vector estimation module is used to extract local point clouds in the neighborhood of each target vibrating point, and determine the normal vector of the target vibrating point by using principal component analysis method;

[0069] A vector conversion module is used to obtain a transformation matrix between the depth camera coordinate system and the robot arm base coordinate system through hand-eye calibration, and convert the normal vector of the target vibrating point from the camera coordinate system into an insertion normal vector in the robot arm base coordinate system;

[0070] A pose angle solving module is configured to calculate a minimum rotation path for aligning the Z axis of the robot arm end tool with the insertion normal vector based on an initial pose of the robot arm end tool, and obtain a target pose angle (rx, ry, rz);

[0071] A vibrating path calculating module is configured to determine a vibrating point pose based on the target pose angle and the three-dimensional space coordinates of the target vibrating point in the camera coordinate system, calculate a vibrating path through the robot arm control system, and realize adaptive tilting insertion of the vibrating rod in the direction of the insertion normal vector.

[0072] A checking module is configured to determine whether the target vibrating point is converted, and execute vibrating if yes, or convert the next vibrating point if no.

[0073] The computer device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the above method when executing the computer program.

[0074] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the above method.

Claims

1. A method for self-adapting normal insertion posture of a vibrating mechanical arm facing a tilted bin surface, characterized in that, The method comprises the following steps: (1) Collecting color images and depth images of a construction surface to be constructed by a depth camera installed at the end of a vibrating mechanical arm, and constructing a three-dimensional point cloud model under the camera coordinate system based on the color images and the depth images; (2) Performing instance segmentation on the color images to extract candidate regions of the concrete area, determining target vibrating points meeting the spacing constraint based on mask optimization, connected region analysis and point layout algorithm, and obtaining a pixel coordinate set of the target vibrating points in the pixel space; (3) Calibrating the depth camera to obtain an intrinsic matrix of the depth camera, converting the pixel coordinate set of the target vibrating points obtained in step (2) into a three-dimensional spatial coordinate set of the target vibrating points in the camera coordinate system in combination with the depth information of the target vibrating points, associating the three-dimensional spatial coordinate set of the target vibrating points with the three-dimensional point cloud model in step (1) in point position, and determining the point cloud position corresponding to the target vibrating points; (4) For each target vibrating point, extracting local point clouds in its neighborhood, and determining the normal vector of the target vibrating point by principal component analysis; (5) Obtaining a transformation matrix between the depth camera coordinate system and the mechanical arm base coordinate system through hand-eye calibration, and converting the normal vector of the target vibrating point obtained in step (4) from the camera coordinate system into an insertion normal vector in the mechanical arm base coordinate system; (6) Calculating a minimum rotation path for aligning the Z-axis of the mechanical arm end with the insertion normal vector in combination with the initial pose of the current mechanical arm end tool to obtain target pose angles (rx, ry, rz); (7) Determining the vibrating point pose based on the target pose angles and the three-dimensional spatial coordinates of the target vibrating points in the camera coordinate system, calculating the vibrating path through the mechanical arm control system, and realizing adaptive inclined insertion of the vibrating rod in the direction of the insertion normal vector; (8) If the target vibrating point conversion is completed, vibrating is performed; otherwise, returning to step (4).

2. The method of claim 1, wherein the method is characterized by, In step (1), the three-dimensional point cloud model under the camera coordinate system is constructed based on the color images and the depth images, specifically as follows: the color image coordinates and the corresponding depth values of each pixel point are combined, the spatial projection is performed using the intrinsic matrix of the depth camera, the three-dimensional point cloud coordinates (x, y, z) of each pixel point are calculated, and the three-dimensional point cloud model under the camera coordinate system is obtained.

3. The method of claim 1, wherein the method further comprises: determining a normal direction of the inclined surface of the bin; and determining the normal insertion direction of the robotic arm based on the normal direction of the inclined surface of the bin. In step (2), the color images are subjected to instance segmentation to extract candidate regions of the concrete area, the target vibrating points meeting the spacing constraint are determined based on mask optimization, connected region analysis and point layout algorithm, and the pixel coordinate set of the target vibrating points in the pixel space is obtained, specifically as follows: (21) Constructing a YOLO concrete instance segmentation model integrating attention mechanism, and completing the training of the model based on a training set; (22) Inputting the color images into the trained YOLO concrete instance segmentation model to perform instance segmentation, extracting candidate regions of the concrete area and mask optimization, and generating a vibrating region instance mask graph; (23) Performing connected region analysis and adhesion segmentation on the vibrating region instance mask graph, and obtaining all target vibrating points meeting the spacing constraint in combination with the point layout algorithm based on geometric centroid calculation and point optimization strategy.

4. The method of claim 1, wherein the method further comprises: Step (4) is specifically as follows: With the three-dimensional coordinates of each target vibration point in the camera coordinate system as the center, local point cloud data is searched in a spherical neighborhood with a radius of R, a point set based on KD tree indexing is constructed, and the eigenvalue decomposition of the covariance matrix of the neighborhood points is performed by principal component analysis. The characteristic vector corresponding to the minimum eigenvalue is selected, and it is judged whether the Z-axis direction of the characteristic vector is upward. If it is upward, the direction of the characteristic vector is reversed to ensure that the characteristic vector points uniformly downward to the normal of the concrete surface, as the normal vector of the target vibration point.

5. The method of claim 1, wherein the method further comprises: determining a normal insertion pose of the robotic arm based on the determined orientation of the tilt-up wall panel. In step (5), the hand-eye calibration is as follows: ​ A depth camera is fixedly installed at the end of the mechanical arm, a nine-point calibration board is placed in the working area of the mechanical arm, a plurality of point pairs are established by collecting images and actual end poses, and the rigidity transformation matrix from the camera coordinate system to the mechanical arm base coordinate system is solved by using image points, depth information and mechanical arm end pose data for registration, which is used for coordinate conversion between the camera coordinate system and the mechanical arm base coordinate system.

6. The method of claim 1, wherein, Step (6) is as follows: For the initial pose coordinate system Z-axis direction of the current mechanical arm end tool and the normal vector of the target vibration point, the minimum rotation vector therebetween is calculated, and a rotation increment matrix is constructed; The rotation increment matrix is multiplied by the initial pose matrix of the mechanical arm end tool to obtain a target pose matrix; the target pose angle (rx, ry, rz) is obtained through Euler angle transformation, wherein the Euler angle rotation sequence is x, y and z, and the unit is degree.

7. The method of claim 5, wherein the method further comprises: determining a normal direction of the inclined surface of the bin; and determining the normal insertion direction of the robotic arm based on the normal direction of the inclined surface of the bin. The hand-eye calibration method adopts a three-dimensional point correspondence calibration method based on a nine-point calibration board, and is as follows: The nine-point calibration board is placed in the field of view of the depth camera, and one image acquisition is completed under the condition of no distortion image, and the pixel coordinates of the nine calibration points in the image are extracted; The mechanical arm end effector, i.e. the end of the vibrating rod, is controlled to accurately contact the nine physical points on the nine-point calibration board in turn, and the space pose coordinates of each point in the mechanical arm base coordinate system are recorded through the mechanical arm feedback to construct a three-dimensional point set in the mechanical arm coordinate system. Three sets of point sets are constructed based on the nine-point calibration board coordinate system: the point set in the camera coordinate system obtained by camera image calibration, the point set in the mechanical arm base coordinate system obtained by the mechanical arm end contact, and the point set in the workboard coordinate system known on the nine-point calibration board. A least square rigid registration algorithm based on singular value decomposition or iterative closest point method is used to calculate the transformation matrix from the mechanical arm base coordinate system to the workboard coordinate system, and a PnP-RANSAC algorithm is used to solve the transformation matrix from the camera coordinate system to the workboard coordinate system. The spatial transformation relationship between the camera coordinate system and the mechanical arm base coordinate system is solved based on the above two transformation matrices, and is saved as a hand-eye calibration matrix.

8. A system for self-adapting normal insertion pose of a vibrating mechanical arm facing a tilted bin face, characterized in that, It comprises: A three-dimensional point cloud model construction module is used to collect color images and depth images of the to-be-constructed warehouse surface by a depth camera installed at the end of the vibrating mechanical arm, and construct a three-dimensional point cloud model in the camera coordinate system based on the color images and the depth images; The target vibration point positioning module is configured to perform instance segmentation on the color image, extract a candidate region of the concrete region, determine a target vibration point meeting a spacing constraint based on mask optimization, connected region analysis, and point layout algorithm, and obtain a pixel coordinate set of the target vibration point in a pixel space; The conversion association module is configured to calibrate the depth camera to obtain an intrinsic matrix of the depth camera, convert the pixel coordinate set of the target vibration point into a three-dimensional space coordinate set of the target vibration point in a camera coordinate system in combination with depth information of the target vibration point, associate the three-dimensional space coordinate set of the target vibration point with the three-dimensional point cloud model in step (1) to determine a point cloud position corresponding to the target vibration point, and determine the target vibration point. The normal vector estimation module is configured to extract local point clouds in a neighborhood of each target vibration point, and determine a normal vector of the target vibration point by using a principal component analysis method. The vector conversion module is configured to obtain a transformation matrix between the depth camera coordinate system and a robot base coordinate system by hand-eye calibration, and convert the normal vector of the target vibration point from the camera coordinate system into an insertion normal vector in the robot base coordinate system. The pose angle solving module is configured to calculate a minimum rotation path for aligning a Z-axis of a robot end tool with the insertion normal vector in combination with an initial pose of the robot end tool to obtain target pose angles (rx, ry, rz). The vibration path calculation module is configured to determine a vibration point pose based on the target pose angles and the three-dimensional space coordinates of the target vibration point in the camera coordinate system, calculate a vibration path by a robot control system, and realize adaptive tilt insertion of a vibration rod in a direction of the insertion normal vector. The checking module is configured to determine whether the target vibration point is converted, and perform vibration if the target vibration point is converted. Otherwise, the next vibration point is converted.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images