A 3d vision-based pose recognition method for excavators

Abstract

The present application is suitable for the technical field of pose recognition, and provides a 3D vision-based excavator pose recognition method.The present application can realize automatic pose recognition of the excavator during work, without manual monitoring, thereby reducing labor cost and improving the working efficiency of the excavator.Only one monocular camera is needed to predict the pose of the excavator, without the need to install visual markers, which is convenient to install and has low use and maintenance cost.Meanwhile, by recognizing and optimizing additional points, even in the case that some key points of the excavator are blocked, the problem of accuracy reduction can be effectively alleviated.According to the physical size constraint and kinematic characteristics of the excavator, an optimization pre-processing method and execution logic are designed, and a new target function is designed based on the projection relationship, and a global and local optimization combined method is used to obtain the 3D pose, thereby improving the recognition accuracy and speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pose recognition, and particularly relates to a method for recognizing the pose of a excavator based on 3D vision. BACKGROUND

[0002] As a widely used engineering machinery, the operation safety and efficiency of the excavator are crucial to the progress of the entire project. However, the working environment of the excavator is complex and often harsh, and the traditional monitoring method mainly relies on manual on-site monitoring, which combines manual visual inspection and simple equipment with personnel experience to provide danger warning. However, this method is not only time-consuming and labor-intensive, but also may fail to provide timely warning due to human negligence, and cannot accurately assess the work efficiency.

[0003] Although there have been many related technical researches aimed at solving these problems, the current excavator monitoring technology still has many shortcomings. Although the contact sensor monitoring method is effective, it is costly and inconvenient to maintain. The visual monitoring technology is easily affected by the visual angle obstruction, resulting in a significant reduction in recognition effect, and the use of multi-camera fusion or depth camera is limited by the scene, the calibration process is complex, and the installation and operation requirements are high. In addition, some monitoring methods that rely on visual markers also face the problem of marker obstruction, damage or falling off on the construction site. Although the method based on convolutional neural network (CNN) has potential, when facing excavator models not included in the training data, it needs to collect data and train the model, which is time-consuming and labor-intensive. The current synthetic image dataset method also has the problems of insufficient model accuracy and unrealistic background, which may result in the recognition accuracy of the trained model not meeting the requirements in actual application.

[0004] In summary, although many excavator monitoring technologies have been proposed, so far there is no mature technical solution that can better monitor the state of the excavator, so manual on-site monitoring is still the main monitoring method. In view of this, the present application proposes a method for recognizing the pose of a excavator based on 3D vision. SUMMARY

[0005] The purpose of the present application is to provide a method for recognizing the pose of a excavator based on 3D vision, which aims to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] A method for recognizing the pose of a excavator based on 3D vision, comprising the following steps:

[0008] Step S1, selecting key points, and establishing an actual excavator dataset and a synthetic image dataset according to the key points;

[0009] Step S2, training using YOLOv8x-pose-p6 as a pre-training model;

[0010] Step S3, kinematic modeling of the excavator to obtain projected 3d coordinates;

[0011] Step S4, coordinate transformation of the projected 3d coordinates to obtain the projected 2d coordinates of the key points;

[0012] Step S5, preprocessing of the estimated 2d coordinates;

[0013] Step S6, establishing an optimization objective function;

[0014] Step S7, combining global optimization algorithms and local optimization algorithms to calculate the best 3d pose of the excavator;

[0015] Step S8, smoothing the 3d pose to make the excavator pose change continuous.

[0016] Further, the specific steps of step S1 are as follows:

[0017] Step S11, selecting key points and designing labels;

[0018] Seven key points are selected: boom hinge point, stick hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, stick cylinder hinge point, and bucket cylinder hinge point;

[0019] Design the key point label [x, y, visible], which includes an excavator bounding box and seven key points, where x and y represent the horizontal and vertical coordinates respectively; visible represents visibility, 0 for out of view, 1 for occlusion, and 2 for visibility;

[0020] Step S12, establishing an actual excavator dataset and a synthetic image dataset;

[0021] Establishing an actual excavator dataset: take photos of excavators in earthwork sites and manually label the labels, then perform offline enhancement;

[0022] Establishing a synthetic image dataset: use Blender to establish a virtual model of the excavator, simulate the environment and randomize parameters to synthesize different pictures, while extracting the 3d coordinates of the key points of the excavator and projecting them as 2d coordinates to generate corresponding label files.

[0023] Further, the specific process of step S2 is as follows: using YOLOv8x-pose-p6 as a pre-training model, training the actual excavator dataset and the synthetic image dataset to identify the key points and bounding boxes of the excavator.

[0024] Further, the specific process of step S3 is as follows:

[0025] The kinematic model of the excavator is established by the standard DH method, and the coordinate transformation matrix between adjacent links i-1 to i is obtained:

[0026]

[0027] in, represents the transformation matrix from the coordinate system of the i-1th link to the coordinate system of the i-th link; a i For along X i Axis direction, Z i Axis to Z i+1 Axis distance; α i For X i Axis, Z i Axis to Z i+1 The rotation angle of the axis; i is the rotation angle along the Z i Axis direction, X i Axis to X i+1 Axis distance; θ i For Z i Axis, X i Axis to X i+1 The rotation angle of the axis, θ0 is the rotation angle; θ1 is the boom joint angle; θ2 is the arm joint angle, and θ3 is the bucket joint angle; is the rotation matrix, which represents the rotation of the i-th joint relative to the i-1-th joint; is the translation matrix, which represents the translation from the i-th joint relative to the i-1-th joint;

[0028] Use the coordinate transformation matrix to calculate the 3D coordinates of the seven selected key points in the excavator coordinate system The subscript is the key point number, and the superscript is the coordinate system number.

[0029] Multiply the Cartesian coordinates of each point by the rotation matrix R related to the tilt angle θ4 to obtain the kinematic model of the excavator considering the uneven ground;

[0030]

[0031] Among them, p' i is the new projected 3D coordinate of the key point after adding the tilt angle θ4; p'0 is the new projected 3D coordinate of the intersection of the rotation center axis and the ground after adding the tilt angle θ4; is the coordinate of the intersection of the rotation center axis and the ground;

[0032] The pose variables involved in establishing the excavator kinematic model and calculating the 3D coordinates of its key points The eight variables include the X, Y, and Z coordinates of the intersection of the center axis of the slewing to the ground in the camera coordinate system, the slewing angle θ0, the boom joint rotation angle θ1, the stick joint rotation angle θ2, the bucket joint rotation angle θ3, and the tilt angle θ4.

[0033] Further, the specific process of the step S4 is as follows:

[0034] Convert the 3d coordinates (p i , i = 1, 2, 3, …, 7) of the projection of the key points into 2d coordinates (P I , i = 1, 2, 3, …, 7) in the pixel coordinate system, and the conversion is represented as:

[0035]

[0036] wherein P I is the 2d coordinate in the pixel coordinate system; T is the transpose operation; K is the intrinsic matrix; E is the extrinsic matrix; Z c,i is the Z coordinate of the i-th key point in the camera coordinate system;

[0037] Perform distortion correction on the key point coordinates, and after the distortion correction, the projection 2d coordinates of the key points in the pixel coordinate system are:

[0038]

[0039] wherein P I is the projection 2d coordinate of the key point in the pixel coordinate system after the distortion correction; K c = 1 + k1r i 2 +k2r i 4 , k1, k2 are distortion parameters determined by camera calibration, and r i is the distance from the original coordinate value of the i-th key point to the center of the image.

[0040] Further, the specific process of the step S5 is as follows:

[0041] Select the position of the excavator in the image in the first frame, and sequentially detect the excavator in the next frame that is closest to the position to eliminate other excavators, while using the area size of the surrounding frame of the excavator as an auxiliary criterion to exclude other excavators; when occlusion occurs, perform outlier detection and fitting repair on each 2d point coordinate in each frame.

[0042] Further, the specific process of the step S6 is as follows:

[0043] Minimize the error between the projection 2d coordinate P i of the key point and the estimated 2d coordinate as the target, and optimize to obtain Pi Pose variables in Establish the projection error function and convert the variable solving problem into an optimization problem:

[0044] First, With P i The weighted sum of the distance differences between them is used as the objective function Obj1:

[0045]

[0046] Among them, w i is the value corresponding to the i-th key point and P i The weight of the absolute value of the coordinate difference always maintains

[0047] Then use P es and P i Calculate the estimated length between the boom hinge point and the stick hinge point Projected length l1, estimated length between the arm hinge point and the bucket hinge point Projected length l2, estimated length between bucket hinge point and bucket tip The projection length l3 is calculated, and the sum of the differences between the estimated lengths and the projection length is used as the objective function Obj2:

[0048]

[0049] in, is the estimated length between adjacent points; l j is the projection length between adjacent points;

[0050] The final objective function Obj is:

[0051] Obj=Obj1+k×Obj2

[0052] Finally, Obj1 and Obj2 are normalized and dimensionless, and k is set as the dynamic weight.

[0053] Furthermore, the specific process of step S7 is as follows:

[0054] Use the global optimization algorithm to optimize the objective function and obtain the best projected 2D coordinates and 3D pose variables;

[0055] An early stopping mechanism is introduced. When the number of optimization iterations exceeds 50 and the fitness function change is less than 1, the optimization stops. At the same time, the excavator's continuous movement and close position in the previous and next frames of the video are used to take advantage of the temporal information. The optimization result of the previous frame is used as the initial value, and a local optimization algorithm is used to narrow the search range and speed up the calculation.

[0056] I. Design the execution conditions of global optimization and local optimization based on the kinematic characteristics and physical constraints of the excavator

[0057] a. Perform global optimization once every 60 frames;

[0058] b. When the 2d coordinate value difference of the key points in t-1 frame and t-2 frame is greater than the threshold value, it is considered that the local optimization is wrong, and the global optimization is performed;

[0059] c. When the coordinate value difference between the optimized point P i and the real point in t-1 frame is greater than the threshold value, the global optimization is called;

[0060] In other cases, local optimization is selected.

[0061] Further, the specific process of the step S8 is as follows:

[0062] After reconstructing the complete excavator pose, the different frequencies of the pose curve are decomposed by using fast Fourier transform; the high frequency part in the pose curve is removed by using a low pass filter, and the low frequency part is reserved; finally, the curve is smoothed by using a moving window method, so that the excavator pose change is continuous.

[0063] Compared with the prior art, the beneficial effects of the present application are:

[0064] 1. The present application can realize automatic pose recognition of the excavator during work, without manual monitoring, thereby reducing the labor cost and improving the working efficiency of the excavator.

[0065] 2. The present application only needs a monocular camera to predict the excavator pose, without the need to install visual markers, and is convenient to install and has low use and maintenance cost. At the same time, by recognizing and optimizing the additional points, even in the case that part of the key points of the excavator are blocked, the problem of accuracy decline can be effectively alleviated.

[0066] 3. The present application designs the optimization pre-processing method and execution logic according to the physical size constraints and kinematic characteristics of the excavator, and designs a new objective function based on the projection relationship, and uses the combination method of global and local optimization to obtain the 3d pose, thereby improving the recognition accuracy and speed. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is the flowchart of the method of the present application.

[0068] Figure 2 is the schematic diagram of the blocking of the base points and the key points of the excavator, wherein (a) is the schematic diagram of the blocking of the base points; (b) is the schematic diagram of the key points.

[0069] Figure 3A D-H coordinate system is established for the working device of the excavator, wherein (a) is a 0-4 point coordinate system; and (b) is a 5-8 point coordinate system.

[0070] Figure 4 Eight variables related to the pose of the excavator are provided.

[0071] Figure 5 A monocular vision system coordinate system is provided. DETAILED DESCRIPTION

[0072] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0073] The specific implementation of the present application will be described in detail below in combination with specific embodiments.

[0074] An embodiment of the present application provides a 3D vision-based excavator pose recognition method, a flowchart of which is shown in Figure 1 The method comprises the following steps:

[0075] Step S1, selecting key points, and establishing an actual excavator data set and a synthetic image data set according to the key points;

[0076] Step S2, using YOLOv8x-pose-p6 as a pre-training model for training;

[0077] Step S3, kinematically modeling the excavator to obtain projected 3D coordinates;

[0078] Step S4, performing coordinate transformation on the projected 3D coordinates to obtain projected 2D coordinates of the key points;

[0079] Step S5, preprocessing the estimated 2D coordinates;

[0080] Step S6, establishing an optimization objective function;

[0081] Step S7, combining a global optimization algorithm and a local optimization algorithm to calculate the best 3D pose of the excavator;

[0082] Step S8, smoothing the 3D pose to make the excavator pose change continuous.

[0083] Specifically as follows:

[0084] (1) Seven key points are selected on the excavator, and key point labels are designed.

[0085] (1) Seven key points are selected on the excavator, and key point labels are designed.

[0086] Since the present application does not install visual markers on the excavator, there are higher requirements for the selection of key points. We select the following seven key points: boom hinge point, arm hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, arm cylinder hinge point and bucket cylinder hinge point;

[0087] Design a key point label [x, y, visible], which includes an excavator bounding box and seven key points, where x and y represent the horizontal and vertical coordinates respectively; visible represents visibility, 0 for out of view, 1 for occlusion, and 2 for visibility.

[0088] Figure 2 In (a), the base point occlusion diagram shows the position of the key points in the case of possible occlusion; (b) is a key point diagram that clearly marks the seven key points on the excavator.

[0089] (2) Establish actual excavator dataset and synthetic image dataset.

[0090] Establish the actual excavator dataset: By taking pictures of excavators on the site of earthwork operations, a total of 9342 pictures were obtained, and manual label annotation was performed. Finally, offline enhancement was performed based on flipping, rotation, translation, scaling, contrast transformation, adding Gaussian noise, color transformation, etc. to simulate real-life situations, and finally 37368 pictures were obtained, of which 70% were used as the training set and 30% were used as the validation set.

[0091] Establish the synthetic image dataset: Use Blender to establish a virtual model of the excavator, simulate the environment and randomize the parameters to synthesize different pictures, while extracting the 3D coordinates of the key points of the excavator and projecting them as 2D coordinates to generate the corresponding label file.

[0092] (2) Step S2, use YOLOv8x-pose-p6 as a pre-trained model for training.

[0093] YOLO v8-pose is an improved version based on the previous versions of the YOLO series, still using the ideas of CSP, feature fusion method (PANFPN) and SPPF module in YOLO v5. On this basis, YOLO v8-pose also makes some optimizations, so that it can recognize the 2D coordinates of the key points in the picture and video, i.e. get 2D pose. In order to save training time, improve recognition accuracy and reduce the risk of overfitting, the present application uses a pre-trained model to initialize the weights of the network structure. Specifically, we choose YOLOv8x-pose-p6 as the pre-trained model, and use the model to train the actual excavator dataset and synthetic image dataset, so that the network can accurately recognize the key points and bounding box of the excavator.

[0094] (3) Excavator 3D pose reconstruction based on projection pose optimization.

[0095] (1) Step S3: Perform kinematic modeling on the excavator to obtain the projected 3D coordinates.

[0096] The kinematic model of the excavator is established by the standard DH method. Each part of the excavator is regarded as a rod and an associated coordinate system is established. The coordinate transformation matrix between adjacent links i-1 to i can be obtained. The DH coordinate system of the excavator working device is as follows: Figure 3 As shown, O i X i Y i Z i is the coordinate system, Figure 3 (a) is the 0-4 point coordinate system; (b) is the 5-8 point coordinate system;

[0097]

[0098] in Represents the transformation matrix from the coordinate system of the i-1th link to the coordinate system of the i-th link, which contains all the necessary rotation and translation information so that we can transform the coordinates in one coordinate system into the coordinates in another coordinate system. i For along X i Axis direction, Z i Axis to Z i+1 Axis distance; α i For X i Axis, Z i Axis to Z i+1 The rotation angle of the axis; d i Along Z i Axis direction, X i Axis to X i+1 Axis distance; θ i For Z i Axis, X i Axis to X i+1 The rotation angle of the axis, θ0 is the rotation angle; θ1 is the boom joint angle; θ2 is the arm joint angle, and θ3 is the bucket joint angle; is the rotation matrix, which represents the rotation of the i-th joint relative to the i-1-th joint; is the translation matrix, which represents the translation from the i-th joint relative to the i-1-th joint;

[0099] Through the coordinate transformation matrix, the 3D coordinates of the seven selected key points in the excavator coordinate system are calculated. The subscript is the key point number, and the superscript is the coordinate system number.

[0100] The Cartesian coordinates of each point are multiplied by a rotation matrix R related to the tilt angle θ4, to obtain the excavator kinematics model considering the uneven ground;

[0101]

[0102] where p' i is the new projection 3d coordinates of the key points after adding the tilt angle θ4; p'0 is the new projection 3d coordinates of the intersection of the center axis of rotation and the ground after adding the tilt angle θ4; is the intersection coordinates of the center axis of rotation and the ground;

[0103] The pose variables involved in the establishment of the excavator kinematics model and the calculation of the 3d coordinates of the key points include the X, Y, Z coordinates of the intersection of the center axis of rotation and the ground in the camera coordinate system, the rotation angle θ0, the boom joint rotation angle θ1, the arm joint rotation angle θ2, the bucket joint rotation angle θ3 and the tilt angle θ4, as shown in Figure 4 .

[0104] At this point, the relationship between the projection 3d coordinates of the seven key points is established.

[0105] (2) Step S4, coordinate transformation is performed on the projection 3d coordinates to obtain the projection 2d coordinates of the key points. The monocular vision system coordinate system is shown in Figure 5 .

[0106] The projection 3d coordinates (p' i , i = 1, 2, 3, …, 7) of the key points are converted to 2d coordinates (P' I , i = 1, 2, 3, …, 7) in the pixel coordinate system, which can be represented as:

[0107]

[0108] where P' I is the 2d coordinates in the pixel coordinate system; T is the inverse operation; K is the intrinsic matrix, which includes the internal parameters of the camera, such as the focal length, the principal point coordinates, etc.; E is the extrinsic matrix, which includes the external parameters of the camera, i.e. the position and pose of the camera relative to the world coordinate system; Z c,i is the Z coordinate of the i-th key point in the camera coordinate system.

[0109] In addition to the intrinsic parameters, distortion also needs to be considered, and the present application mainly considers radial distortion. After distortion correction, the projection 2d coordinates of the key points in the pixel coordinate system are:

[0110]

[0111] where P​I K is the projected 2D coordinate of the key point in the pixel coordinate system after distortion correction; c =1+k1r i 2 +k2r i 4 , k1, k2 are the distortion parameters determined by camera calibration, r i is the distance from the original coordinate value of the i-th key point to the center of the image.

[0112] (3) Step S5: Preprocess the estimated 2D coordinates to exclude other excavators in the actual photographed image and repair outliers and fitting repairs.

[0113] In actual images, there may be multiple excavators. Therefore, after selecting the position of the excavator in the first frame, the closest excavator in the subsequent frame is detected to eliminate the other excavators. Furthermore, the excavator of interest is often closest to the camera and occupies the largest area in the image. The area of ​​each excavator's bounding box in the image can be used as an auxiliary criterion for excluding other excavators. The captured video may also have some parts of the excavator occluded. When occlusion occurs, the excavator's 2D coordinates cannot be successfully identified. Therefore, outlier detection and fitting repair are performed on the coordinates of each 2D point in each frame.

[0114] (4) Step S6: Establish an optimization objective function.

[0115] The projected 2D coordinates P of the key points i and estimate the 2d coordinates As the goal, the error between the two can be minimized, and the optimization can obtain P i Pose variables in Establish the projection error function and convert the variable solving problem into an optimization problem:

[0116] First, With P i The weighted sum of the distance differences between them is used as the objective function Obj1:

[0117]

[0118] Among them, w i is the value corresponding to the i-th key point and P i The weight of the absolute value of the coordinate difference can be adjusted dynamically, but always maintain By adding additional points and weights w i Dynamic adjustment can reduce the occlusion in the image. es Inaccuracy leads to inaccurate optimized 3D pose of the excavator.

[0119] In order to avoid the problem that the turning angle of the excavator is opposite to the real turning angle direction during optimization, P es and P i Calculate the estimated length between the boom hinge point and the arm hinge point The projection length l1 is the estimated length between the arm hinge point and the bucket hinge point The projection length l2 is the estimated length between the bucket hinge point and the bucket tip point The projection length l3, and the sum of the differences between the estimated lengths and the projection lengths is taken as the objective function Obj2:

[0120]

[0121] Wherein, is the estimated length between adjacent points; l j is the projection length between adjacent points;

[0122] The final objective function Obj is:

[0123] Obj = OBj1 + k x Obj2

[0124] Finally, Obj1 and Obj2 are normalized and dimensionless, and k is set as a dynamic weight, so that Obj1 and Obj2 always maintain a certain proportion.

[0125] (5) Step S7, the global optimization algorithm and the local optimization algorithm are combined to calculate the best 3d pose of the excavator.

[0126] The global optimization algorithm such as the enhanced whale optimization algorithm (E-WOA) is used to optimize the objective function, and the best projection 2d coordinates and 3d pose variables are obtained.

[0127] The early stop mechanism is added in the algorithm, and when the optimization iteration number is greater than 50 times and the fitness function change is less than 1, the optimization stops. At the same time, by using the characteristics that the excavator is continuously moving and the position is close in the front and rear frames of the video, i.e. the time information, the results of the last frame optimization are used as the initial value, and the local optimization algorithm is used to reduce the search range and speed up the operation speed.

[0128] Based on the kinematic characteristics and physical constraints of the excavator, the execution conditions I of the global optimization and the local optimization are designed:

[0129] a. The actual running speed of the excavator is relatively low, such as the walking speed is generally 3-6 km / h, the turning speed is 7-10 rpm, and the frame rate of the shooting video is 60 fps, so the pose change of two adjacent time points is not large, and therefore the global optimization is executed once every 60 frames to correct the situation that the local optimization gradually deviates from the real value due to the input value deviation;

[0130] b. According to the historical information and the motion ability of the excavator, the speed of the 2d coordinate change of the excavator 2d point is determined, and the threshold is updated. When the difference between the 2d coordinate values of the key points in the t-1 frame and the t-2 frame is greater than the threshold, it is considered that the local optimization is wrong, and the global optimization is performed;

[0131] c. When the coordinate value difference between the optimized point P i and the real point in the t-1 frame is greater than the threshold, the global optimization is called;

[0132] In other cases, local optimization is selected.

[0133] (6) Step S8, the 3d pose is smoothed to make the excavator pose change continuous.

[0134] After reconstructing the complete excavator pose, the different frequencies of the pose curve are decomposed using fast Fourier transform, the high frequency part of the pose curve is removed using a low pass filter, and only the low frequency part is retained; finally, the curve is smoothed using the moving window method to make the excavator pose change continuous.

[0135] Example 1, experimental verification;

[0136] (1) Model training and data collection;

[0137] The YOLOv8x-pose-p6 model is used for training, the optimizer is selected as SGD, the batch size is set to 8, the training rounds are set to 100 rounds, and the rest of the hyperparameters are selected according to experience. In terms of data sets, the data set Dataset#1 used by the first model contains 37368 actual excavator pictures taken on site, of which 70% is used as the training set and 30% is used as the verification set. The data set Dataset#2 used by the second model consists of 10000 pictures randomly selected from Dataset#1 and 14000 synthetic pictures generated by Blender randomization, a total of 24000 pictures, of which 70% is used as the training set and 30% is used as the verification set.

[0138] The collection of real 3d coordinates is calculated by installing displacement sensors on the excavator working device. The real 2d coordinates are obtained by manually adding coordinate labels after extracting pictures and sensor data from the video.

[0139] (2) Test index;

[0140] The normalized error (NE) and the percentage of correct key points (PCK) are used as evaluation indices. NE is the estimated 2d coordinate and the real 2d coordinate distance difference between the estimated 2d coordinates and the real 2d coordinates is within a set threshold a PCK .

[0141]

[0142] where NE i represents the NE value of the ith key point, and represents the distance difference between the estimated 2d coordinates and the real 2d coordinates normalized by the diagonal length; PCK i (a) represents the proportion of the detected key points whose Euclidean distance between the estimated 2d coordinates and the real 2d coordinates is less than the set threshold a PCK ; f represents the total number of frames in the test set; n is the frame number; represents the Euclidean distance between the real 2d coordinates of the ith key point in the nth frame and the estimated 2d coordinates; a PCK represents the threshold, which is set to 0.02 according to the actual size of the key point in the image; l n is the original diagonal length of the image in the nth frame.

[0143] In addition, in order to evaluate the accuracy of the 3d pose reconstruction algorithm, the mean per joint position error (MPJPE) is used to measure the accuracy of the projected 3d coordinates p' i,n after optimization, that is:

[0144]

[0145] where MPJPE i represents the MPJPE value of the ith key point;

[0146] Since the projected 3d coordinates p' 1,n of the first key point after optimization are in the excavator coordinate system, p' 1,n coincides with p' , so the value of MPJPE1 is not considered.

[0147] In actual earthwork operations, it is necessary to monitor the joint motion information of the excavator to help project managers identify potential risks, optimize planning and operating parameters, accurately price and budget future projects, and the commonly used effective monitoring information is generally the extension length of each oil cylinder of the working device. Therefore, the mean length error (MLE) is used to measure the real oil cylinder length and the reconstructed oil cylinder length l n,jthe error.

[0148]

[0149] wherein subscript j = 1, 2, 3 respectively represent the boom cylinder, stick cylinder and bucket cylinder cylinder extension length; max(L j ) represents the maximum extension length of the cylinder without the barrel; is the real cylinder extension length of the nth frame; λ n,j is the reconstructed cylinder extension length of the nth frame.

[0150] (3) Experimental implementation and results;

[0151] 1. Key point recognition experiment;

[0152] The PCK value and NE value of key point recognition are shown in Table 1 and Table 2 respectively:

[0153] Table 1 PCK value (%) of key point recognition

[0154]

[0155] Table 2 NE value of key point recognition

[0156]

[0157] From the data in the table, it can be seen that the model trained using Dataset#1 achieves an average PCK value of 96.28% and an average NE value of 0.0059 on all key points, indicating that the predicted 2d point coordinates are close to the true values, and the model has good accuracy. The model trained using Dataset#2 achieves an average PCK value of 97.64% and an average NE value of 0.0056 on all key points. This shows that training the model using synthetic image dataset can also obtain high-precision recognition results, proving the effectiveness of the synthetic image dataset.

[0158] Compared with existing researches (Tian et al. identified the above four points and the tail point of the machine body, only the accuracy of the common 2d points was compared, Mahmood et al. used an end-to-end CNN model, Assadzadeh et al. used an HRNet, and Wen et al. used an improved R-CNN), the method proposed in the present application performs better in key point identification. In addition to the four basic points (boom hinge point point_1, arm hinge point point_2, bucket hinge point point_3, and bucket tip point point_4), three additional points (boom cylinder hinge point point_5, arm cylinder hinge point point_6, and bucket cylinder hinge point point_7) are added, which makes the average PCK value and the average NE value of the key point identification result of the present application better than the results of existing researches. At the same time, the method of the present application also does not require visual markers, further improving the reliability of the algorithm.

[0159] 2. Pose reconstruction experiment;

[0160] First, the 2d point information identified by the model trained using Dataset#1 and Dataset#2 is input into the 3d pose reconstruction algorithm, and the MPJPE and MLE of the obtained results are tested. In order to make a comparison, according to the method proposed by Wen et al., the objective function of the 3d pose reconstruction part is changed to use only the information of point_1 to point_4, without using the additional points point_5 to point_7 and the difference between the real length and the projection length of the working device mechanism as the objective function, and the 2d key point identification part remains unchanged, to obtain a comparative method. Then the same test data is verified to obtain new MPJPE and MLE.

[0161] Four groups of cross experiments are carried out, which are: Dataset#1 + the method of the present application, Dataset#1 + the comparative method, Dataset#2 + the method of the present application, and Dataset#2 + the comparative method.

[0162] The results of the comparison are shown in Tables 3 and 4:

[0163] Table 3 Key point MPJPE (mm) of 3d pose reconstruction

[0164]

[0165] Table 4 Cylinder MLE (%) of 3d pose reconstruction

[0166]

[0167] From the data in the table, it can be seen that the accuracy of the method of the present application is higher than that of the comparative method whether Dataset #1 or Dataset #2 is used. The MPJPE average value obtained by training the model using Dataset #1 is reduced by 14.39% compared with the comparative method, and the cylinder MLE average value obtained is also smaller than that of the comparative method, indicating that the method of the present application can accurately estimate the 3D pose of the excavator in the real working scene. In addition, the results of 3D pose reconstruction using Dataset #2 to train the model also show that the MPJPE average value and the cylinder MLE average value of the method of the present application are lower than those of the model trained by Dataset #1. Although the accuracy of individual points decreases (such as Point_7), the overall accuracy of the reconstructed 3D pose is higher. Similarly, when the 3D pose reconstruction is performed using the comparative method, the MPJPE average value obtained by using the recognition result of the model trained by Dataset #2 as input is not much different from that of the model trained by Dataset #1, but the cylinder MLE average value is reduced by 0.98%, which further proves that the synthetic picture dataset can be used as a substitute for the real photo dataset to train a high-precision model.

[0168] The above is only a preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which should also be considered as the protection scope of the present application, and these will not affect the effect and practicality of the patent.

Claims

1. A 3D vision-based pose recognition method for a shovel, characterized in that, The method comprises the following steps: Step S1, selecting key points, and establishing an actual excavator dataset and a synthetic image dataset according to the key points; Step S2, using YOLOv8x-pose-p6 as a pre-training model for training; Step S3, kinematically modeling the excavator to obtain projected 3d coordinates; Step S4, performing coordinate transformation on the projected 3d coordinates to obtain projected 2d coordinates of the key points; Step S5, preprocessing the estimated 2d coordinates; Step S6, establishing an optimization objective function; Step S7, combining a global optimization algorithm and a local optimization algorithm to calculate the optimal 3d pose of the excavator; Step S8, performing smoothing processing on the 3d pose to make the excavator pose change continuously; The specific process of step S6 is as follows: Projected 2D coordinates of the key points and estimate the 2d coordinates The goal is to minimize the error between Pose variables in ; The key points include the boom hinge point, the bucket hinge point, the bucket tip point, the boom cylinder hinge point, the bucket cylinder hinge point and the bucket cylinder hinge point, and the posture variables Including the intersection of the rotation center axis and the ground in the camera coordinate system Coordinates, rotation angle , boom joint angle , arm joint angle , bucket joint angle and tilt angle These eight variables; establish the projection error function and convert the variable solving problem into an optimization problem: First, the weighted sum of the distance differences between and is taken as the objective function : ； Wherein, As the first Key point corresponding And The absolute value of the coordinate difference weight, always keep ; Then, the estimated length between the boom hinge point and the arm hinge point is calculated and the estimated length between the arm hinge point and the bucket hinge point is calculated The estimated length between the boom hinge point and the arm hinge point is calculated , the projected length The estimated length between the arm hinge point and the bucket hinge point is calculated , the projected length The estimated length between the bucket hinge point and the bucket tip point is calculated , the projected length The sum of the differences between the estimated lengths and the projected lengths is used as the objective function : ； in, is the estimated length between adjacent points; is the projection length between adjacent points; Objective function is: ； Finally, the normalized and dimensionless values of and are set as dynamic weights ; The specific process of step S7 is as follows: The global optimization algorithm is used to optimize the objective function to obtain the optimal projected 2d coordinates and 3d pose variables; An early stop mechanism is introduced, and when the optimization iteration number is greater than 50 times and the fitness function change is less than 1, the optimization stops; at the same time, by using the characteristics of continuous movement and close position of the excavator in the front and rear frames of the video, that is, the time information, the optimized result of the last frame is used as the initial value, and the local optimization algorithm is used to reduce the search range and speed up the operation speed; Execution conditions for global optimization and local optimization are designed based on kinematic characteristics and physical constraints of the excavator : a. Global optimization is performed once every 60 frames; b. when frames are If the difference between the 2D coordinate values of the key points in the frames is greater than a threshold value, it is considered that the local optimization is wrong, and the global optimization is performed. c. When optimized points in frame and real points when coordinate value difference is greater than threshold, call global optimization; In other cases, local optimization is selected. 2.The 3D vision based excavator pose identification method of claim 1, wherein, The specific steps of step S1 are as follows: Step S11, selecting key points and designing labels; Seven key points are selected: boom hinge point, arm hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, arm cylinder hinge point, and bucket cylinder hinge point; Design key point labels , contains an excavator bounding box and seven key points, among which 、 Represent the horizontal and vertical coordinates respectively; Represents visibility, 0 is out of view, 1 is blocked, and 2 is visible; Step S12, establishing an actual excavator dataset and a synthetic image dataset; The actual excavator dataset is established: the excavator is photographed on the earthwork site, and manual label annotation is performed, and then offline enhancement is performed; The synthetic image dataset is established: a virtual model of the excavator is established by using Blender, different pictures are synthesized by simulating the environment and randomizing parameters, and 3d coordinates of key points of the excavator are extracted and projected into 2d coordinates to generate corresponding label files. 3.The 3D vision based excavator pose identification method of claim 2, wherein, The specific process of step S2 is: YOLOv8x-pose-p6 is used as a pre-training model to train the actual excavator dataset and the synthetic image dataset to identify the key points and bounding boxes of the excavator.

4. The 3D vision-based excavator pose recognition method of claim 3, wherein, The specific process of step S3 is as follows: The kinematic model of the excavator is established by the standard D-H method to obtain the coordinate transformation matrix between adjacent links i-1 and i: ； wherein, represents a transformation matrix from the coordinate system of the i-1th link to the coordinate system of the ith link; is a distance along the axis to the axis; is a distance along the axis to the axis; is a rotation angle about the axis to the axis; is a distance along the axis to the axis; is a rotation angle about the axis to the axis; is a rotation angle about the axis to the is a swing angle; is a stick angle; is a bucket angle; is a rotation matrix representing a rotation of the ith joint with respect to the i-1th joint; is a translation matrix representing a translation from the ith joint with respect to the i-1th joint; The 3D coordinates of the selected seven key points in the coordinate system of the excavator are solved by using a coordinate transformation matrix wherein the subscript is the key point serial number, and the superscript is the serial number of the coordinate system Multiplying the Cartesian coordinates of the points by the tilt angle The relevant rotation matrix , obtaining the excavator kinematic model considering ground unevenness; ； ； wherein, for adding the tilt angle new projected 3d coordinates of the intersection of the rear pivotal center axis with the ground; for adding the tilt angle new projected 3d coordinates of the intersection of the rear pivotal center axis with the ground; for the intersection coordinates of the pivotal center axis with the ground; The pose variables involved in establishing the kinematic model of the excavator and calculating the 3D coordinates of its key points including the intersection of the center axis of rotation to the ground in the camera coordinate system coordinates, rotation angle , boom joint rotation angle , arm joint rotation angle , bucket joint rotation angle and tilt angle These eight variables.

5. The 3D vision-based excavator pose identification method of claim 4, wherein, The specific process of step S4 is as follows: projected 3d coordinates of the key points 2d coordinates in the pixel coordinate system , are represented as ； in, is the 2D coordinate in the pixel coordinate system; T is the transpose operation; is the internal parameter matrix; is the external parameter matrix; For the The key points in the camera coordinate system coordinate; The key point coordinates are corrected for distortion, and the projected 2d coordinates of the key points in the pixel coordinate system after distortion correction are: ； wherein, is the projection 2d coordinate of the key point in the pixel coordinate system after distortion correction; , is the distortion parameter determined by camera calibration, is the distance from the original coordinate value of the th key point to the image center. 6.The 3D vision based excavator pose identification method of claim 5, wherein, The specific process of step S5 is as follows: The position of the excavator in the image in the first frame is selected, and the excavator with the closest position in the subsequent frame is detected in sequence to eliminate other excavators, and the area size of the excavator bounding box is used as an auxiliary criterion to exclude other excavators; when occlusion occurs, outlier detection and fitting repair are performed on each 2d point coordinate in each frame. 7.The 3D vision based excavator pose identification method of claim 1, wherein, The specific process of step S8 is as follows: After the complete pose of the excavator is reconstructed, the different frequencies of the pose curve are decomposed by using fast Fourier transform. The high frequency part of the pose curve is removed and the low frequency part is reserved by using low-pass filter. Finally, the curve is smoothed by using moving window method, so that the pose change of the excavator is continuous.

Images