Excavator pose recognition method based on 3d vision

Abstract

The invention is suitable for the technical field of pose recognition, and provides an excavator pose recognition method based on 3d vision. According to the invention, automatic pose recognition during working of the excavator can be realized without manual monitoring, so that the labor cost is reduced, and the working efficiency of the excavator is improved. Only one monocular camera is needed to predict the position and posture of the excavator, visual marks do not need to be installed, installation is convenient, and use and maintenance cost is low. And meanwhile, by identifying and optimizing the additional points, the problem that the accuracy is reduced can be effectively relieved even if part of key points of the excavator are shielded. According to physical dimension constraints and kinematics characteristics of the excavator, an optimization pre-and-post processing method and execution logic are designed, a new objective function is designed based on a projection relation, a 3d pose is obtained by using a global and local optimization combined method, and the recognition precision and speed are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of posture recognition, and in particular relates to an excavator posture recognition method based on 3D vision. Background Art

[0002] As a widely used construction machinery, the operation safety and work efficiency of excavators are crucial to the progress of the entire project. However, the working environment of excavators is complex, changeable and often harsh. Traditional monitoring methods mainly rely on manual on-site monitoring, using manual visual inspection and simple equipment combined with personnel experience to provide early warning of dangers. However, this method is not only time-consuming and labor-intensive, but may also fail to warn of dangers in a timely manner due to human negligence, and it is also impossible to accurately evaluate work efficiency.

[0003] Although there have been many related technical studies aimed at solving these problems, current excavator monitoring technology still has many shortcomings. Although contact sensor monitoring methods are effective, they are expensive and inconvenient to maintain. Visual monitoring technology is easily affected by view occlusion, which greatly reduces the recognition effect. The use of multi-camera fusion or depth cameras is limited by the scene, the calibration process is complicated, and the installation and operation requirements are high. In addition, some monitoring methods that rely on visual markers also face the problem of markers being easily occluded, damaged, or falling off on construction sites. Although methods based on convolutional neural networks (CNNs) have potential, when faced with excavator models that are not included in the training data, it is time-consuming and laborious to collect data again and train the model. The current synthetic image dataset method has the problems of insufficient model accuracy and unrealistic background, which may result in the trained model's recognition accuracy failing to meet the requirements in practical applications.

[0004] In summary, although a variety of excavator monitoring technologies have been proposed, so far, there is no mature technical solution that can better monitor the status of the excavator, so manual on-site monitoring is still the main monitoring method. In view of this, the present invention proposes an excavator posture recognition method based on 3D vision. Summary of the invention

[0005] The purpose of the present invention is to provide an excavator posture recognition method based on 3D vision, aiming to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for recognizing an excavator's posture based on 3D vision comprises the following steps:

[0008] Step S1, selecting key points, and establishing an actual excavator dataset and a synthetic image dataset based on the key points;

[0009] Step S2: Use YOLOv8x-pose-p6 as a pre-trained model for training;

[0010] Step S3, kinematic modeling of the excavator to obtain projection 3D coordinates;

[0011] Step S4, transform the projected 3D coordinates to obtain the projected 2D coordinates of the key points;

[0012] Step S5, preprocessing the estimated 2D coordinates;

[0013] Step S6, establishing an optimization objective function;

[0014] Step S7, combining the global optimization algorithm and the local optimization algorithm to calculate the optimal 3D posture of the excavator;

[0015] Step S8: Smooth the 3D posture to make the excavator posture change continuously.

[0016] Furthermore, the specific steps of step S1 are as follows:

[0017] Step S11, select key points and design labels;

[0018] Select seven key points: boom hinge point, arm hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, arm cylinder hinge point and bucket cylinder hinge point;

[0019] Design key point labels [x, y, visible], including an excavator bounding box and seven key points, where x and y represent the horizontal and vertical coordinates respectively; visible represents visibility, 0 is out of view, 1 is blocked, and 2 is visible;

[0020] Step S12, establishing an actual excavator data set and a synthetic image data set;

[0021] Create a real excavator dataset by photographing excavators at earthmoving sites, manually labeling them, and then performing offline enhancement.

[0022] Establish a synthetic image dataset: Use Blender to build a virtual model of the excavator, simulate the environment and randomize parameters to synthesize different pictures. At the same time, extract the 3D coordinates of the key points of the excavator and project them into 2D coordinates to generate the corresponding label files.

[0023] Furthermore, the specific process of step S2 is: using YOLOv8x-pose-p6 as a pre-trained model, training the actual excavator data set and the synthetic image data set to identify the key points and bounding boxes of the excavator.

[0024] Furthermore, the specific process of step S3 is as follows:

[0025] The kinematic model of the excavator is established by the standard DH method, and the coordinate transformation matrix between adjacent connecting rods i-1 to i is obtained:

[0026]

[0027] in, represents the transformation matrix from the coordinate system of the i-1th link to the coordinate system of the i-th link; a i For along X i Axis direction, Z i Axis to Z i+1 Axis distance; α i For around X i Axis, Z i Axis to Z i+1 The rotation angle of the axis; i is the rotation angle along the Z i Axis direction, X i Axis to X i+1 The distance between the axes; θ i For Z i Axis, X i Axis to X i+1 The rotation angle of the axis, θ0 is the rotation angle; θ1 is the boom joint angle; θ2 is the arm joint angle, and θ3 is the bucket joint angle; is the rotation matrix, which represents the rotation of the i-th joint relative to the i-1-th joint; is the translation matrix, which represents the translation from the i-th joint relative to the i-1-th joint;

[0028] Use the coordinate transformation matrix to find the 3D coordinates of the seven selected key points in the excavator coordinate system. The subscript is the key point number, and the superscript is the coordinate system number;

[0029] The Cartesian coordinates of each point are multiplied by the rotation matrix R associated with the tilt angle θ4 to obtain the kinematic model of the excavator considering the uneven ground surface;

[0030]

[0031] Among them, p' i is the new projected 3D coordinate of the key point after adding the inclination angle θ4; p'0 is the new projected 3D coordinate of the intersection of the rotation center axis and the ground after adding the inclination angle θ4; is the coordinate of the intersection of the rotation center axis and the ground;

[0032] The pose variables involved in establishing the kinematic model of the excavator and calculating the 3D coordinates of its key points It includes eight variables: X, Y, Z coordinates of the intersection of the rotation center axis and the ground in the camera coordinate system, rotation angle θ0, boom joint angle θ1, arm joint angle θ2, bucket joint angle θ3 and tilt angle θ4.

[0033] Furthermore, the specific process of step S4 is as follows:

[0034] The projected 3D coordinates of the key points (p' i , i = 1, 2, 3, ..., 7) to the 2D coordinates in the pixel coordinate system (P' I , i=1,2,3,…,7), expressed as:

[0035]

[0036] Among them, P' I is the 2D coordinate in the pixel coordinate system; T is the transposition operation; K is the internal parameter matrix; E is the external parameter matrix; Z c,i is the Z coordinate of the i-th key point in the camera coordinate system;

[0037] The coordinates of the key points are corrected for distortion. After distortion correction, the projected 2D coordinates of the key points in the pixel coordinate system are:

[0038]

[0039] Among them, P I K is the projected 2D coordinate of the key point in the pixel coordinate system after distortion correction; c =1+k1r i 2 +k2r i 4 , k1, k2 are distortion parameters determined by camera calibration, r i is the distance from the original coordinate value of the i-th key point to the center of the image.

[0040] Furthermore, the specific process of step S5 is as follows:

[0041] The position of the excavator in the first frame is selected, and the excavator closest to its position in the next frame is detected in turn to eliminate other excavators. At the same time, the area of ​​the excavator's bounding box is used as an auxiliary criterion to exclude other excavators. When occlusion occurs, outlier detection and fitting repair are performed on the coordinates of each 2D point in each frame.

[0042] Furthermore, the specific process of step S6 is as follows:

[0043] The projected 2D coordinates P of the key points i and estimate the 2d coordinates The goal is to minimize the error betweeni Pose variables in Establish the projection error function and convert the variable solving problem into an optimization problem:

[0044] First, With P i The weighted sum of the distance differences between them is used as the objective function Obj1:

[0045]

[0046] Among them, w i is the value corresponding to the i-th key point and P i The weight of the absolute value of the coordinate difference is always maintained

[0047] Then use P es and P i Calculate the estimated length between the boom hinge point and the stick hinge point Projected length l1, estimated length between the arm hinge point and the bucket hinge point Projected length l2, estimated length between bucket hinge point and bucket tip The projection length l3, and the sum of the differences between each estimated length and the projection length is used as the objective function Obj2:

[0048]

[0049] in, is the estimated length between adjacent points; l j is the projection length between adjacent points;

[0050] The final objective function Obj is:

[0051] Obj=Obj1+k×Obj2

[0052] Finally, Obj1 and Obj2 are normalized and dimensionless, and k is set as the dynamic weight.

[0053] Furthermore, the specific process of step S7 is as follows:

[0054] Use the global optimization algorithm to optimize the objective function and obtain the best projected 2D coordinates and 3D pose variables;

[0055] An early stopping mechanism is introduced. When the number of optimization iterations is greater than 50 and the fitness function changes less than 1, the optimization stops. At the same time, the excavator moves continuously and its position is close in the previous and next frames of the video, that is, time information, and the optimization result of the previous frame is used as the initial value. The local optimization algorithm is used to narrow the search range and speed up the calculation.

[0056] Design execution conditions of global optimization and local optimization based on kinematic characteristics and physical constraints of the excavator I:

[0057] a. Perform global optimization once every 60 frames;

[0058] b. When the difference between the 2D coordinate values ​​of the key points in the t-1 frame and the t-2 frame is greater than the threshold, it is considered that the local optimization is wrong and global optimization is performed;

[0059] c. When the optimized point P in frame t-1 i And be real When the coordinate value difference is greater than the threshold, global optimization is called;

[0060] In other cases local optimization is chosen.

[0061] Furthermore, the specific process of step S8 is as follows:

[0062] After reconstructing the complete excavator posture, the fast Fourier transform is used to decompose the different frequencies of the posture curve; a low-pass filter is used to remove the high-frequency part of the posture curve and retain the low-frequency part; finally, the moving window method is used to smooth the curve to make the excavator posture change continuous.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. The present invention can realize automatic posture recognition when the excavator is working, without the need for manual monitoring, thereby reducing labor costs and improving the working efficiency of the excavator.

[0065] 2. The present invention only needs a monocular camera to predict the excavator's position and posture, and does not require the installation of visual markers. It is easy to install and has low use and maintenance costs. At the same time, by identifying and optimizing additional points, the problem of reduced accuracy can be effectively alleviated even when some key points of the excavator are blocked.

[0066] 3. The present invention designs the pre- and post-processing methods and execution logic of optimization based on the physical size constraints and kinematic characteristics of the excavator, designs a new objective function based on the projection relationship, and uses a combination of global and local optimization methods to obtain the 3D pose, thereby improving the recognition accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Flow chart of the method of the present invention.

[0068] Figure 2 Schematic diagram of basic point occlusion and key points of excavator, where (a) is the schematic diagram of basic point occlusion; (b) is the schematic diagram of key points.

[0069] Figure 3It is the DH coordinate system of the excavator working device, where (a) is the 0-4 point coordinate system and (b) is the 5-8 point coordinate system.

[0070] Figure 4 There are eight variables related to the excavator posture.

[0071] Figure 5 is the coordinate system of the monocular vision system. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0073] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0074] An embodiment of the present invention provides a method for recognizing an excavator posture based on 3D vision, and its flow chart is as follows: Figure 1 As shown, the following steps are included:

[0075] Step S1, selecting key points, and establishing an actual excavator dataset and a synthetic image dataset based on the key points;

[0076] Step S2: Use YOLOv8x-pose-p6 as a pre-trained model for training;

[0077] Step S3, kinematic modeling of the excavator to obtain projection 3D coordinates;

[0078] Step S4, transform the projected 3D coordinates to obtain the projected 2D coordinates of the key points;

[0079] Step S5, preprocessing the estimated 2D coordinates;

[0080] Step S6, establishing an optimization objective function;

[0081] Step S7, combining the global optimization algorithm and the local optimization algorithm to calculate the optimal 3D posture of the excavator;

[0082] Step S8: Smooth the 3D posture to make the excavator posture change continuously.

[0083] The details are as follows:

[0084] (i) Step S1: Select key points and establish an actual excavator dataset and a synthetic image dataset based on the key points.

[0085] (1) Select seven key points on the excavator and design key point labels.

[0086] Since the present invention does not install visual markers on the excavator, the selection of key points has higher requirements. We select the following seven key points: boom hinge point, dipper arm hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, dipper arm cylinder hinge point and bucket cylinder hinge point;

[0087] The key point label [x, y, visible] is designed, which contains an excavator bounding box and seven key points, where x and y represent the horizontal and vertical coordinates respectively; visible represents visibility, 0 is out of view, 1 is blocked, and 2 is visible.

[0088] Figure 2 In the figure, (a) is a basic point occlusion diagram, showing the position of key points when they may be occluded; (b) is a key point diagram, clearly marking the seven key points on the excavator.

[0089] (2) Establish actual excavator dataset and synthetic image dataset.

[0090] Establishing a real excavator dataset: A total of 9,342 images were obtained by photographing excavators at earthwork sites and manually annotated. Finally, offline enhancement was performed to simulate real-life situations based on operations such as flipping, rotation, translation, scaling, contrast transformation, adding Gaussian noise, and color transformation. Finally, 37,368 images were obtained, of which 70% were used as training sets and 30% as validation sets.

[0091] Establish a synthetic image dataset: Use Blender to build a virtual model of the excavator, simulate the environment and randomize parameters to synthesize different pictures. At the same time, extract the 3D coordinates of the key points of the excavator, project them into 2D coordinates, and generate the corresponding label files.

[0092] (ii) Step S2: Use YOLOv8x-pose-p6 as the pre-trained model for training.

[0093] YOLO v8-pose is an improved version based on the previous version of the YOLO series, and still uses the ideas of CSP, feature fusion method (PANFPN) and SPPF module in YOLO v5. On this basis, YOLO v8-pose has also been partially optimized so that it can recognize the 2d coordinates of key points in pictures and videos, that is, obtain 2d poses. In order to save training time, improve recognition accuracy and reduce the risk of overfitting, the present invention uses a pre-training model to initialize the weights of the network structure. Specifically, we selected YOLOv8x-pose-p6 as the pre-training model, and used the model to train the actual excavator data set and the synthetic image data set, so that the network can accurately identify the key points and bounding boxes of the excavator.

[0094] (III) Excavator 3D pose reconstruction based on projection pose optimization.

[0095] (1) Step S3: Perform kinematic modeling on the excavator to obtain projected 3D coordinates.

[0096] The kinematic model of the excavator is established by the standard DH method. Each part of the excavator is regarded as a rod and an associated coordinate system is established. The coordinate transformation matrix between adjacent connecting rods i-1 to i can be obtained. The DH coordinate system of the excavator working device is as follows: Figure 3 As shown, O i X i Y i Z i is the coordinate system, Figure 3 (a) is the 0-4 point coordinate system; (b) is the 5-8 point coordinate system;

[0097]

[0098] in represents the transformation matrix from the coordinate system of the i-1th link to the coordinate system of the i-th link, which contains all the necessary rotation and translation information so that we can transform the coordinates in one coordinate system into the coordinates in another coordinate system. i For along X i Axis direction, Z i Axis to Z i+1 Axis distance; α i For around X i Axis, Z i Axis to Z i+1 The rotation angle of the axis; d i Along Z i Axis direction, X i Axis to X i+1 The distance between the axes; θ i For Z i Axis, X i Axis to X i+1 The rotation angle of the axis, θ0 is the rotation angle; θ1 is the boom joint angle; θ2 is the arm joint angle, and θ3 is the bucket joint angle; is the rotation matrix, which represents the rotation of the i-th joint relative to the i-1-th joint; is the translation matrix, which represents the translation from the i-th joint relative to the i-1-th joint;

[0099] Through the coordinate transformation matrix, the 3D coordinates of the selected seven key points in the excavator coordinate system are calculated. The subscript is the key point number, and the superscript is the coordinate system number;

[0100] By multiplying the Cartesian coordinates of each point by the rotation matrix R associated with the tilt angle θ4, the kinematic model of the excavator considering the uneven ground surface can be obtained;

[0101]

[0102] Among them, p' i is the new projected 3D coordinate of the key point after adding the inclination angle θ4; p'0 is the new projected 3D coordinate of the intersection of the rotation center axis and the ground after adding the inclination angle θ4; is the coordinate of the intersection of the rotation center axis and the ground;

[0103] The pose variables involved in establishing the kinematic model of the excavator and calculating the 3D coordinates of its key points It includes eight variables: X, Y, Z coordinates of the intersection of the rotation center axis and the ground in the camera coordinate system, rotation angle θ0, boom joint angle θ1, arm joint angle θ2, bucket joint angle θ3 and tilt angle θ4, such as Figure 4 shown.

[0104] So far, the The relationship between the projected 3D coordinates of the seven key points.

[0105] (2) Step S4: transform the projected 3D coordinates to obtain the projected 2D coordinates of the key points. The coordinate system of the monocular vision system is as follows: Figure 5 shown.

[0106] The projected 3D coordinates of the key points (p' i , i = 1, 2, 3, ..., 7) to the 2D coordinates in the pixel coordinate system (P' I , i=1,2,3,…,7), can be expressed as:

[0107]

[0108] Among them, P' I is the 2D coordinate in the pixel coordinate system; T is the inversion operation; K is the internal parameter matrix, which contains the internal parameters of the camera, such as focal length, principal point coordinates, etc.; E is the external parameter matrix, which contains the external parameters of the camera, that is, the position and posture of the camera relative to the world coordinate system; Z c,i is the Z coordinate of the i-th key point in the camera coordinate system.

[0109] In addition to the internal parameters, distortion also needs to be considered. The present invention mainly considers radial distortion. The coordinates of the key points are corrected for distortion. After distortion correction, the projected 2D coordinates of the key points in the pixel coordinate system are:

[0110]

[0111] Among them, PI K is the projected 2D coordinate of the key point in the pixel coordinate system after distortion correction; c =1+k1r i 2 +k2r i 4 , k1, k2 are distortion parameters determined by camera calibration, r i is the distance from the original coordinate value of the i-th key point to the center of the image.

[0112] (3) Step S5: pre-process the estimated 2D coordinates to exclude other excavators in the actual photographed image and to repair outliers and fit patches.

[0113] There may be multiple excavators in the actual image. Therefore, after selecting the position of the excavator in the first frame, the excavator with the closest position in the next frame is detected in turn, and other excavators can be eliminated. In addition, the excavator of interest is often the closest to the camera and occupies the largest area in the image. The area of ​​each excavator's bounding box in the image can be used as an auxiliary criterion for excluding other excavators. It is also possible that some parts of the excavator are blocked in the captured video. When occlusion occurs, the excavator cannot successfully identify the 2D coordinates. Therefore, outlier detection and fitting repair are performed on the coordinates of each 2D point in each frame.

[0114] (4) Step S6: Establish an optimization objective function.

[0115] The projected 2D coordinates P of the key points i and estimate the 2d coordinates As the goal, the error between the two can be minimized, and the optimization can obtain P i Pose variables in Establish the projection error function and convert the variable solving problem into an optimization problem:

[0116] First, With P i The weighted sum of the distance differences between them is used as the objective function Obj1:

[0117]

[0118] Among them, w i is the value corresponding to the i-th key point and P i The weight of the absolute value of the coordinate difference can be adjusted dynamically, but always maintain By adding additional points and weights w i Dynamic adjustment can reduce the occlusion in the image. es Inaccuracy leads to inaccurate optimized 3D pose of the excavator.

[0119] In order to avoid the problem that the excavator rotation angle is opposite to the actual rotation angle during optimization, P es and P i Calculate the estimated length between the boom hinge point and the stick hinge point Projected length l1, estimated length between the arm hinge point and the bucket hinge point Projected length l2, estimated length between bucket hinge point and bucket tip The projection length l3, and the sum of the differences between each estimated length and the projection length is used as the objective function Obj2:

[0120]

[0121] in, is the estimated length between adjacent points; l j is the projection length between adjacent points;

[0122] The final objective function Obj is:

[0123] Obj=OBj1+k×Obj2

[0124] Finally, Obj1 and Obj2 are normalized and dimensionless, and k is set as a dynamic weight to always keep Obj1 and Obj2 in a certain ratio.

[0125] (5) Step S7: Calculate the optimal 3D posture of the excavator by combining the global optimization algorithm and the local optimization algorithm.

[0126] The objective function is optimized using a global optimization algorithm such as the enhanced whale optimization algorithm (E-WOA) to obtain the best projected 2D coordinates and 3D pose variables.

[0127] The present invention adds an early stopping mechanism to the algorithm. When the number of optimization iterations is greater than 50 and the fitness function changes less than 1, the optimization stops. At the same time, the characteristics of the excavator's continuous movement and close position in the previous and next frames of the video, that is, time information, are used to take the optimization result of the previous frame as the initial value, use the local optimization algorithm, narrow the search range, and speed up the calculation speed.

[0128] The execution conditions of global optimization and local optimization are designed based on the kinematic characteristics and physical constraints of the excavator:

[0129] a. The actual operating speed of the excavator is relatively low. For example, the walking speed is generally 3-6 km / h, the rotation speed is 7-10 rpm, and the frame rate of the video is 60 fps. The posture change between two adjacent time points is not large. Therefore, a global optimization is performed every 60 frames to correct the situation where the local optimization gradually deviates from the true value due to the deviation of the input value;

[0130] b. Determine the change speed of the 2D point coordinates of the excavator based on historical information and the excavator's motion capability, and update the threshold. When the difference between the 2D coordinate values ​​of the key points in the t-1 frame and the t-2 frame is greater than the threshold, it is considered that the local optimization is wrong and global optimization is performed;

[0131] c. When the optimized point P in frame t-1 i And be real When the coordinate value difference is greater than the threshold, global optimization is called;

[0132] In other cases local optimization is chosen.

[0133] (6) Step S8: Smooth the 3D posture to make the excavator posture change continuously.

[0134] After reconstructing the complete excavator posture, the fast Fourier transform is used to decompose the different frequencies of the posture curve, and a low-pass filter is used to remove the high-frequency part of the posture curve, leaving only the low-frequency part; finally, the moving window method is used to smooth the curve to make the excavator posture change continuous.

[0135] Embodiment 1, experimental verification;

[0136] (1) Model training and data collection;

[0137] The YOLOv8x-pose-p6 model is used for training, where the optimizer is selected as SGD, the batch size is set to 8, the number of training rounds is set to 100 rounds, and the remaining hyperparameters are selected based on experience. In terms of data sets, the first model uses the data set Dataset#1, which contains 37,368 actual excavator pictures taken on site, of which 70% are used as training sets and 30% are used as validation sets. The data set Dataset#2 used by the second model consists of 10,000 randomly selected pictures from Dataset#1 and 14,000 synthetic photos generated by Blender randomization, totaling 24,000 pictures, of which 70% are used as training sets and 30% are used as validation sets.

[0138] Real 3D coordinates The collection is done by installing a displacement sensor on the excavator working device, which obtains the displacement of the cylinder and calculates the real 2D coordinates. After extracting the images and sensor data from the captured video, the coordinate labels are manually added.

[0139] (2) Test indicators;

[0140] The normalized error (NE) and the percentage of correct key points (PCK) are used as evaluation An indicator of accuracy. NE is the estimated 2D coordinates with real 2D coordinates The distance difference between them is normalized by the length of the image diagonal. PCK depends on the true 2D coordinates of the key points. To estimate the 2d coordinates Is the Euclidean distance within the set threshold α? PCK Within.

[0141]

[0142] Among them, NE i Represents the NE value of the i-th key point, indicating the estimated 2D coordinates with real 2D coordinates The dimensionless error index obtained by normalizing the distance difference between and the diagonal length; PCK i (α) represents the true 2D coordinates of the detected key points To estimate the 2d coordinates The Euclidean distance is less than the set threshold α PCK The ratio of; f represents the total number of frames in the test set; n is the frame number; represents the Euclidean distance from the real 2D coordinate to the estimated 2D coordinate of the i-th key point in the n-th frame; α PCK represents the threshold value, which is set to 0.02 according to the actual size of the excavator key point in the image; l n is the original diagonal length of the image in the nth frame.

[0143] In addition, in order to evaluate the accuracy of the 3D pose reconstruction algorithm, the mean per-joint position error (MPJPE) is used to measure the projected 3D coordinates p' obtained after optimization. i,n The accuracy is:

[0144]

[0145] Among them, MPJPE i Represents the MPJPE value of the i-th key point;

[0146] Since the projection 3D coordinate p' of the first key point obtained after optimization 1,n is in the excavator coordinate system, so With p' 1,n The values ​​of MPJPE1 are not considered because they overlap.

[0147] In actual earthwork operations, it is necessary to monitor the joint motion information of the excavator to help project managers identify potential risks, optimize planning and operating parameters, and accurately price and budget future projects. The commonly used effective monitoring information is generally the extension length of each cylinder of the working device. Therefore, the mean length error (MLE) is used to measure the actual cylinder length. and the reconstruction cylinder length λ n,jof error.

[0148]

[0149] Wherein, the subscripts j = 1, 2, and 3 represent the extension lengths of the boom cylinder, the arm cylinder, and the bucket cylinder, respectively; max(L j ) indicates the maximum extension length of the cylinder excluding the cylinder body; is the actual extension length of the oil cylinder in the nth frame; n,j The extension length of the cylinder reconstructed at the nth frame.

[0150] (3) Experimental implementation and results;

[0151] 1. Key point identification experiment;

[0152] The PCK values ​​and NE values ​​of key point recognition are shown in Table 1 and Table 2 respectively:

[0153] Table 1 PCK value of key point recognition (%)

[0154]

[0155] Table 2 NE values ​​of key point recognition

[0156]

[0157] From the data in the table, we can see that the model trained with Dataset#1 achieved an average PCK value of 96.28% and an average NE value of 0.0059 at all key points, indicating that the predicted 2D point coordinates are close to the true values ​​and the model has good accuracy. The model trained with Dataset#2 achieved an average PCK value of 97.64% and an average NE value of 0.0056 at all key points. This shows that using synthetic image datasets to train models can also obtain high-precision recognition results, proving the effectiveness of synthetic image datasets.

[0158] Compared with existing studies (Tian et al. identified the first 4 points and the tail point of the body, and only compared the accuracy of common 2D points, Mahmood et al. used an end-to-end CNN model, Assadzadeh et al. used HRNet, and Wen et al. used improved R-CNN), the method proposed in the present invention performs better in key point recognition. In addition to the four commonly used basic points (boom hinge point_1, arm hinge point point_2, bucket hinge point point_3, bucket tip point_4), we also added three additional points (boom cylinder hinge point_5, arm cylinder hinge point point_6, and bucket cylinder hinge point point_7), which makes the average PCK value and average NE value of the key point recognition results of the present invention better than the existing research results. At the same time, the method of the present invention does not require visual marking, which further improves the reliability of the algorithm.

[0159] 2. Posture reconstruction experiment;

[0160] First, the 2D point information identified by the models trained with Dataset#1 and Dataset#2 was input into the 3D pose reconstruction algorithm, and the MPJPE and MLE of the obtained results were tested. For comparison, according to the method proposed by Wen et al., the objective function of the 3D pose reconstruction part was changed to use only the information from point_1 to point_4, without using the additional points from point_5 to point_7 and the difference between the actual length and the projected length of the working device mechanism as the objective function, and the 2D key point recognition part remained unchanged, and the comparison method was obtained. Then, the same test data was verified to obtain the new MPJPE and MLE.

[0161] Four groups of crossover experiments were conducted, namely: Dataset #1 + the method of the present invention, Dataset #1 + the comparison method, Dataset #2 + the method of the present invention, and Dataset #2 + the comparison method.

[0162] The comparison results are shown in Table 3 and Table 4:

[0163] Table 3 Key point MPJPE (mm) of 3D pose reconstruction

[0164]

[0165] Table 4 3d pose reconstruction of the cylinder MLE (%)

[0166]

[0167] It can be seen from the data in the table that the accuracy of the method of the present invention is higher than that of the comparison method regardless of whether Dataset#1 or Dataset#2 is used. The average MPJPE obtained by using the Dataset#1 training model is reduced by 14.39% compared with the comparison method, and the average MLE of the cylinder obtained is also smaller than that of the comparison method, indicating that the method of the present invention can accurately estimate the 3D pose of the excavator in the real working scene. In addition, the results of 3D pose reconstruction using the Dataset#2 training model also show that the average MPJPE and the average MLE of the cylinder of the method of the present invention are lower than those of the Dataset#1 training model. Although the accuracy of individual points has decreased (such as Point_7), the overall accuracy of the reconstructed 3D pose is higher. Similarly, when the comparison method is used for 3D pose reconstruction, compared with the recognition result of the Dataset#1 model as input, the average MPJPE obtained by using the recognition result of the Dataset#2 training model as input is not much different, while the average MLE of the cylinder is reduced by 0.98%, which further proves that the synthetic picture dataset can be used as a substitute for the real photo dataset for training high-precision models.

[0168] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for recognizing the posture of an excavator based on 3D vision, characterized in that: The following steps are involved: Step S1, selecting key points, and establishing an actual excavator dataset and a synthetic image dataset based on the key points; Step S2: Use YOLOv8x-pose-p6 as a pre-trained model for training; Step S3, performing kinematic modeling on the excavator to obtain projection 3D coordinates; Step S4, transform the projected 3D coordinates to obtain the projected 2D coordinates of the key points; Step S5, preprocessing the estimated 2D coordinates; Step S6, establishing an optimization objective function; Step S7, combining the global optimization algorithm and the local optimization algorithm to calculate the optimal 3D posture of the excavator; Step S8: Smooth the 3D posture to make the excavator posture change continuously.

2. The method for excavator posture recognition based on 3D vision according to claim 1 is characterized in that: The specific steps of step S1 are as follows: Step S11, select key points and design labels; Select seven key points: boom hinge point, arm hinge point, bucket hinge point, bucket tip point, boom cylinder hinge point, arm cylinder hinge point and bucket cylinder hinge point; Design key point labels [x, y, visible], including an excavator bounding box and seven key points, where x and y represent the horizontal and vertical coordinates respectively; Visible represents visibility, 0 is out of view, 1 is blocked, and 2 is visible; Step S12, establishing an actual excavator data set and a synthetic image data set; Create a real excavator dataset by photographing excavators at earthmoving sites, manually labeling them, and then performing offline enhancement. Establish a synthetic image dataset: Use Blender to build a virtual model of the excavator, simulate the environment and randomize parameters to synthesize different pictures. At the same time, extract the 3D coordinates of the key points of the excavator and project them into 2D coordinates to generate the corresponding label files.

3. The method for excavator posture recognition based on 3D vision according to claim 2 is characterized in that: The specific process of step S2 is: using YOLOv8x-pose-p6 as a pre-trained model, training the actual excavator dataset and the synthetic image dataset to identify the key points and bounding boxes of the excavator.

4. The method for excavator posture recognition based on 3D vision according to claim 3 is characterized in that: The specific process of step S3 is as follows: The kinematic model of the excavator is established by the standard DH method, and the coordinate transformation matrix between adjacent connecting rods i-1 to i is obtained: in, represents the transformation matrix from the coordinate system of the i-1th link to the coordinate system of the i-th link; a i For along X i Axis direction, Z i Axis to Z i+1 Axis distance; α i For around X i Axis, Z i Axis to Z i+1 The rotation angle of the axis; d i Along Z i Axis direction, X i Axis to X i+1 The distance between the axes; θ i For Z i Axis, X i Axis to X i+1 The rotation angle of the axis, θ0 is the rotation angle; θ1 is the boom joint angle; θ2 is the arm joint angle, and θ3 is the bucket joint angle; is the rotation matrix, which represents the rotation of the i-th joint relative to the i-1-th joint; is the translation matrix, which represents the translation from the i-th joint relative to the i-1-th joint; Use the coordinate transformation matrix to find the 3D coordinates of the seven selected key points in the excavator coordinate system. (i=1,2,…,7), where the subscript is the key point number and the superscript is the coordinate system number; The Cartesian coordinates of each point are multiplied by the rotation matrix R associated with the tilt angle θ4 to obtain the kinematic model of the excavator considering the uneven ground surface; Among them, p' i is the new projected 3D coordinate of the key point after adding the inclination angle θ4; p'0 is the new projected 3D coordinate of the intersection of the rotation center axis and the ground after adding the inclination angle θ4; is the coordinate of the intersection of the rotation center axis and the ground; The pose variables involved in establishing the kinematic model of the excavator and calculating the 3D coordinates of its key points It includes eight variables: X, Y, Z coordinates of the intersection of the rotation center axis and the ground in the camera coordinate system, rotation angle θ0, boom joint angle θ1, arm joint angle θ2, bucket joint angle θ3 and tilt angle θ4.

5. The method for excavator posture recognition based on 3D vision according to claim 4 is characterized in that: The specific process of step S4 is as follows: The projected 3D coordinates of the key points (p' i , i = 1, 2, 3, ..., 7) to the 2D coordinates in the pixel coordinate system (P' I , i=1,2,3,…,7), expressed as: Among them, P' I is the 2D coordinate in the pixel coordinate system; T is the transposition operation; K is the internal parameter matrix; E is the external parameter matrix; Z c,i is the Z coordinate of the i-th key point in the camera coordinate system; The coordinates of the key points are corrected for distortion. After distortion correction, the projected 2D coordinates of the key points in the pixel coordinate system are: Among them, P I K is the projected 2D coordinate of the key point in the pixel coordinate system after distortion correction; c =1+k1r i 2 +k2r i 4 , k1, k2 are distortion parameters determined by camera calibration, r i is the distance from the original coordinate value of the i-th key point to the center of the image.

6. The method for excavator posture recognition based on 3D vision according to claim 5 is characterized in that: The specific process of step S5 is as follows: The position of the excavator in the first frame is selected, and the excavator closest to its position in the next frame is detected in turn to eliminate other excavators. At the same time, the area of ​​the excavator's bounding box is used as an auxiliary criterion to exclude other excavators. When occlusion occurs, outlier detection and fitting repair are performed on the coordinates of each 2D point in each frame.

7. The method for excavator posture recognition based on 3D vision according to claim 6 is characterized in that: The specific process of step S6 is as follows: The projected 2D coordinates P of the key points i and estimate the 2d coordinates The goal is to minimize the error between i Pose variables in Establish the projection error function and convert the variable solving problem into an optimization problem: First, With P i The weighted sum of the distance differences between them is used as the objective function Obj1: Among them, w i is the value corresponding to the i-th key point and P i The weight of the absolute value of the coordinate difference is always maintained Then use P es and P i Calculate the estimated length between the boom hinge point and the stick hinge point Projected length l1, estimated length between the arm hinge point and the bucket hinge point Projected length l2, estimated length between bucket hinge point and bucket tip The projection length l3, and the sum of the differences between each estimated length and the projection length is used as the objective function Obj2: in, is the estimated length between adjacent points; l j is the projection length between adjacent points; The final objective function Obj is: Obj=Obj1+k×Obj2 Finally, Obj1 and Obj2 are normalized and dimensionless, and k is set as the dynamic weight.

8. The method for excavator posture recognition based on 3D vision according to claim 7 is characterized in that: The specific process of step S7 is as follows: Use the global optimization algorithm to optimize the objective function and obtain the best projected 2D coordinates and 3D pose variables; An early stopping mechanism is introduced. When the number of optimization iterations is greater than 50 and the fitness function changes less than 1, the optimization stops. At the same time, the excavator moves continuously and its position is close in the previous and next frames of the video, that is, time information, and the optimization result of the previous frame is used as the initial value. The local optimization algorithm is used to narrow the search range and speed up the calculation. Design execution conditions of global optimization and local optimization based on kinematic characteristics and physical constraints of the excavator I: a. Perform global optimization once every 60 frames; b. When the difference between the 2D coordinate values ​​of the key points in the t-1 frame and the t-2 frame is greater than the threshold, it is considered that the local optimization is wrong and global optimization is performed; c. When the optimized point P in frame t-1 i And be real When the coordinate value difference is greater than the threshold, global optimization is called; In other cases local optimization is chosen.

9. The method for excavator posture recognition based on 3D vision according to claim 8, characterized in that: The specific process of step S8 is as follows: After reconstructing the complete excavator posture, the fast Fourier transform is used to decompose the different frequencies of the posture curve; a low-pass filter is used to remove the high-frequency part of the posture curve and retain the low-frequency part; finally, the moving window method is used to smooth the curve to make the excavator posture change continuous.

Images