Method and apparatus for determining position of excavator bucket, and excavator

Abstract

The present application relates to the technical field of engineering machinery, and discloses a method and apparatus for determining the position of an excavator bucket, and an excavator. The method comprises: acquiring a bucket image and an excavator body rotation angle, wherein the bucket image is captured by a monocular camera arranged on an excavator body; detecting feature points of an excavator bucket in the bucket image to obtain a plurality of key point positions; acquiring coordinates of the monocular camera relative to an excavator coordinate system; and on the basis of the coordinates of the monocular camera relative to the excavator coordinate system, the excavator body rotation angle, the plurality of key point positions, and a preset conversion relationship between an image pixel coordinate system and a camera coordinate system, converting pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket. A position in a bucket image can be accurately converted into a position in an excavator coordinate system, thereby improving the accuracy of excavator bucket position detection, making it convenient for an operator to operate, and improving the operation efficiency.

Description

Methods and devices for determining the position of the excavator bucket and the excavator itself.

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Chinese Patent Application No. 202410663337.4, filed on May 27, 2024, the contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of engineering machinery technology, specifically to a method for determining the position of an excavator bucket, a device for determining the position of an excavator bucket, an excavator, a machine-readable storage medium, and an electronic device. Background Technology

[0004] Excavators are one of the important tools in modern engineering construction and production. They are widely used in earthwork infrastructure, water conservancy and transportation, mining and other fields. Their main functions include excavation, loading, transportation and leveling.

[0005] With existing excavators, operators can only visually determine the approximate position of the bucket, not its precise location, reducing operational efficiency. While existing wireless long-range remote-controlled excavators can transmit images from the site to the remote control via cameras, operators still cannot determine the bucket's exact position from these images.

[0006] Therefore, existing methods for determining the position of excavator buckets cannot determine the precise location of the bucket. Summary of the Invention

[0007] The purpose of this application is to provide a method for determining the position of an excavator bucket, a device for determining the position of an excavator bucket, an excavator, a machine-readable storage medium, and an electronic device. This addresses the problem in the prior art that the precise position of the bucket cannot be determined.

[0008] To achieve the above objectives, the first aspect of this application provides a method for determining the position of an excavator bucket, comprising:

[0009] The bucket image and the vehicle body rotation angle are acquired. The bucket image is obtained by a monocular camera installed on the excavator body.

[0010] Feature points of the excavator bucket are detected in the bucket image to obtain the positions of multiple key points;

[0011] Obtain the coordinates of the monocular camera relative to the excavator coordinate system;

[0012] Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket.

[0013] Wherein, the image pixel coordinate system is the coordinate system used to describe the position of a pixel in an image when the monocular camera captures an image, the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera, and the excavator coordinate system is the coordinate system established with a specific position or component of the excavator as the origin.

[0014] In this embodiment of the application, the step of converting the pixel coordinates of the bucket in the bucket image into the coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system includes:

[0015] Based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the multiple key points, the depth of field distance from the bucket to the monocular camera is calculated.

[0016] Based on the depth of field distance from the bucket to the monocular camera and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the camera coordinate system to obtain the first coordinates of the bucket position point.

[0017] Based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

[0018] In this embodiment of the application, the multiple key point locations include the vertex locations at both ends of the bottom edge of the bucket;

[0019] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0020] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0021] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0022] Based on the first depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0023] In this embodiment of the application, the step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes:

[0024] Based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

[0025] In this embodiment of the application, the multiple key point locations include the vertex locations at both ends of the bottom edge of the bucket;

[0026] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0027] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0028] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0029] Based on the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0030] In this embodiment of the application, the step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes:

[0031] Based on the second depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

[0032] In this embodiment of the application, the multiple key point locations include the vertex locations at both ends of the upper bottom edge of the bucket and the vertex locations at both ends of the lower bottom edge of the bucket;

[0033] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0034] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0035] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0036] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0037] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0038] Based on the first depth of field distance and the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0039] In this embodiment of the application, the method further includes:

[0040] The bucket size is calculated based on the first depth of field distance, the second depth of field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system.

[0041] In this embodiment of the application, the calculation of the bucket size based on the first depth-of-field distance, the second depth-of-field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system includes:

[0042] Based on the vertex positions located at both ends of the bottom edge of the bucket, the length of the bottom edge of the bucket in the image pixel coordinate system is obtained;

[0043] The width of the bucket is calculated based on the first depth of field distance and the length of the bottom edge of the bucket in the image pixel coordinate system;

[0044] The position of the first vertex is calculated based on the first depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the bottom edge of the bucket.

[0045] Based on the second depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the upper bottom edge of the bucket, the second vertex position is calculated, and the vertex position located at one end of the upper bottom edge of the bucket corresponds to the vertex position located at one end of the lower bottom edge of the bucket.

[0046] The length of the bucket is calculated based on the positions of the first and second vertices;

[0047] The bucket dimensions are obtained based on the width and length of the bucket.

[0048] In this embodiment of the application, the method further includes:

[0049] Based on the first depth of field distance, the second depth of field distance, and the bucket size, the angle between the bucket and the horizontal plane is calculated.

[0050] In this embodiment of the application, the method further includes:

[0051] Based on the excavator bucket position and the first coordinates of the bucket position point, the coordinates of the projection point are determined. The projection point is the projection of the bucket position point onto the horizontal plane, and the coordinates of the projection point are the coordinates of the projection point in the camera coordinate system.

[0052] Based on the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the coordinates of the projection point are converted into the coordinates of the projection point in the image pixel coordinate system to obtain the pixel coordinates of the projection point.

[0053] Based on the pixel coordinates of the projection points, annotations are made in the bucket image to show the vertical landing position of the bucket.

[0054] In this embodiment of the application, the step of detecting feature points of the excavator bucket in the bucket image to obtain the locations of multiple key points includes:

[0055] A preset bucket feature point detection model is used to perform target detection on the bucket image to obtain the locations of multiple key points.

[0056] In this embodiment of the application, the construction process of the preset bucket feature point detection model includes:

[0057] Images of bucket movement are continuously acquired, and key points are annotated for each acquired image of bucket movement to obtain multiple sample images;

[0058] Based on the multiple sample images, the YOLOv8 model is trained to obtain the bucket feature point detection model.

[0059] A second aspect of this application provides an excavator bucket position determination device, comprising:

[0060] The first acquisition module is used to acquire bucket images and vehicle body rotation angles. The bucket images are acquired by a monocular camera set on the excavator body.

[0061] The detection module is used to detect feature points of the excavator bucket in the bucket image and obtain the positions of multiple key points;

[0062] The second acquisition module is used to acquire the coordinates of the monocular camera relative to the excavator coordinate system, wherein the excavator coordinate system is a coordinate system established with a specific position or component of the excavator as the origin;

[0063] The conversion module is used to convert the pixel coordinates of the bucket in the bucket image into the coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset conversion relationship between the image pixel coordinate system and the camera coordinate system, thereby obtaining the position of the excavator bucket; wherein, the image pixel coordinate system is the coordinate system used when the monocular camera captures the image to describe the position of the pixel in the image, and the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera.

[0064] The third aspect of this application discloses an excavator that uses the aforementioned method for determining the position of the excavator bucket to determine the position of the excavator bucket.

[0065] A fourth aspect of this application provides an electronic device, the electronic device comprising:

[0066] At least one processor;

[0067] A memory connected to the at least one processor;

[0068] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the above-described method for determining the position of the excavator bucket by executing the instructions stored in the memory.

[0069] A fifth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the excavator bucket position determination method described above.

[0070] The above technical solution acquires the bucket image and the vehicle body rotation angle. The bucket image is obtained by a monocular camera mounted on the excavator body. Feature points of the excavator bucket are detected in the bucket image to obtain the positions of multiple key points. The coordinates of the monocular camera relative to the excavator coordinate system are obtained. Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted to the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the excavator bucket position. By detecting the feature points of the excavator bucket from the bucket image, and based on the feature points of the excavator bucket and the vehicle body rotation angle, the position in the bucket image can be accurately converted to the position in the excavator coordinate system, improving the accuracy of excavator bucket position detection, assisting the operator in operation, and improving work efficiency.

[0071] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0072] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0073] Figure 1 schematically illustrates a flowchart of a method for determining the position of an excavator bucket according to an embodiment of this application;

[0074] Figure 2 schematically illustrates the four vertices of an excavator bucket according to an embodiment of this application;

[0075] Figure 3 schematically illustrates the deep learning steps according to an embodiment of this application;

[0076] Figure 4 schematically illustrates a camera imaging diagram according to an embodiment of this application;

[0077] Figure 5 schematically illustrates an excavator according to an embodiment of this application;

[0078] Figure 6 schematically illustrates one implementation step diagram according to an embodiment of this application;

[0079] Figure 7 schematically illustrates the structure of an excavator bucket position determination device according to an embodiment of this application;

[0080] Figure 8 schematically illustrates the internal structure of a computer device according to an embodiment of this application.

[0081] Explanation of reference numerals in the attached drawings: 410 - First acquisition module; 420 - Detection module; 430 - Second acquisition module; 440 - Conversion module; A01 - Processor; A02 - Network interface; A03 - Internal memory; A04 - Display screen; A05 - Input device; A06 - Non-volatile storage medium; B01 - Operating system; B02 - Computer program. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0083] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0084] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0085] Figure 1 schematically illustrates a flowchart of a method for determining the position of an excavator bucket according to an embodiment of this application. As shown in Figure 1, this embodiment provides a method for determining the position of an excavator bucket by detecting key points of the bucket. Based on the pixel coordinates of the key points and the vehicle body rotation angle, the three-dimensional position and horizontal tilt angle of the bucket relative to the excavator coordinate system can be obtained. Based on the position of the bucket in the camera coordinate system and its height in the excavator coordinate system, the bucket landing point can be located in the image, facilitating operator work and improving efficiency. It should be noted that, in this embodiment, the upper bottom edge of the bucket refers to the edge near the connecting rod, as shown in CD in Figure 2; the lower bottom edge of the bucket refers to the edge where the outermost edge of the bucket teeth is located, i.e., the end of the bucket, as shown in AB in Figure 2. The method may include the following steps:

[0086] Step 210: Acquire the bucket image and vehicle body rotation angle. The bucket image is acquired by a monocular camera mounted on the excavator body.

[0087] In this embodiment, the monocular camera is mounted at a suitable position on the vehicle body to capture all positions during the bucket's movement. The bucket can be raised or lowered, and the monocular camera can capture all positions during both raising and lowering. For example, the monocular camera can be mounted above the excavator's cab, facing directly forward with its optical axis horizontally forward; "directly forward" refers to the area directly in front of the cab. The vehicle body rotation angle can be obtained in real time by installing an angle sensor on the vehicle body.

[0088] Step 220: Detect feature points of the excavator bucket in the bucket image to obtain the positions of multiple key points;

[0089] In this embodiment, the feature points of the excavator bucket are located on the excavator bucket, and there can be multiple feature points, which can be set according to actual needs. The multiple key point positions include the vertex positions at both ends of the bottom edge of the bucket and / or the vertex positions at both ends of the top edge of the bucket.

[0090] Referring to Figure 2, the feature points of the excavator bucket can be the vertices at both ends of the bottom edge of the bucket or the vertices at both ends of the top edge of the bucket, or they can be the four vertices on the bucket plane, namely the vertices at both ends of the bottom edge of the bucket and the vertices at both ends of the top edge of the bucket. As shown in Figure 2, the feature points of the excavator bucket are points A, B, C, and D. The aforementioned feature points of the excavator bucket can be obtained through manual annotation or through automatic detection using a target detection model.

[0091] In some embodiments, detecting feature points of the excavator bucket in the bucket image to obtain multiple key point locations includes: performing target detection on the bucket image using a preset bucket feature point detection model to obtain multiple key point locations.

[0092] In this embodiment, the aforementioned pre-set bucket feature point detection model can be obtained by pre-training an object detection model, thereby enabling the detection of bucket feature points based on deep learning, and thus quickly and accurately detecting the locations of multiple key points. The aforementioned object detection model includes, but is not limited to, the YOLOv8 model, the YOLOv7 model, and the R-CNN model.

[0093] The construction process of the preset bucket feature point detection model includes:

[0094] First, images of bucket movement are continuously acquired, and key points are labeled for each acquired image of bucket movement to obtain multiple sample images;

[0095] In this embodiment, a monocular camera can be installed above the cab, with its optical axis pointing horizontally forward, continuously acquiring images of the bucket's movement. The aforementioned key point annotation can be performed using Labelme image annotation software to annotate the bucket's feature points in each image. For example, if the bucket's feature points are four vertices, then the positions of these four vertices can be annotated for each acquired image of the bucket's movement.

[0096] Then, based on the multiple sample images, the YOLOv8 model is trained to obtain the bucket feature point detection model.

[0097] In this embodiment, the aforementioned YOLOv8 model employs advanced object detection algorithms and technologies, achieving high accuracy in various scenarios. Compared to other object detection models, YOLOv8 has lower computational complexity, better meeting real-time requirements. Furthermore, the YOLOv8 model supports training with custom datasets, allowing for adjustments and optimizations based on actual needs. This makes the bucket feature point detection model faster and more accurate, improving detection flexibility. Training can be achieved by dividing multiple sample images into training, validation, and test sets in a specific ratio, such as an 8:1:1 ratio. The YOLOv8 model is then trained using these sets to obtain optimal weight parameters, which are saved locally. These optimal weight parameters are then loaded to obtain the bucket feature point detection model for real-time detection of bucket key points.

[0098] It should be noted that when using a point on the bucket to determine the position of the excavator bucket, it is necessary to determine the corresponding key point positions. Therefore, it is necessary to train a corresponding bucket feature point detection model.

[0099] The above-mentioned deep learning detection of bucket key points can be achieved by using a pre-set bucket feature point detection model. Please refer to Figure 3. The steps of the above-mentioned deep learning detection of bucket key points are shown in Figure 3. Image data is collected, feature points are labeled on the image data, the dataset is divided, and finally the divided dataset is used to train the model to obtain the bucket feature point detection model. Finally, the bucket feature point detection model is used to detect the model and obtain the positions of multiple key points.

[0100] Step 230: Obtain the coordinates of the monocular camera relative to the excavator coordinate system, wherein the excavator coordinate system is a coordinate system established with a specific position or component of the excavator as the origin;

[0101] In this embodiment, the excavator coordinate system can be a coordinate system established with the projection of the excavator's rotation center on the ground as the origin, along the excavator's forward movement direction, with the X-axis being horizontal, the Y-axis perpendicular to the X-axis and pointing to the left, and the Z-axis perpendicular to the X-axis and pointing upwards, as shown in Figure 7. After the monocular camera is installed, the coordinates of the monocular camera relative to the excavator coordinate system can be obtained, that is, the offset of the monocular camera's installation position relative to the origin of the excavator coordinate system on each axis, which can be denoted as x0, y0, and z0 respectively.

[0102] Step 240: Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket; wherein, the image pixel coordinate system is the coordinate system used when the monocular camera captures the image to describe the position of the pixel in the image, and the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera.

[0103] In this embodiment, the conversion of the pixel coordinates of the bucket in the bucket image to the coordinates of the excavator bucket relative to the excavator coordinate system can be achieved by first converting the pixel coordinates of the bucket in the bucket image to the coordinates of the camera coordinate system, and then, based on the camera imaging principle, converting the coordinates of the camera coordinate system to the coordinates of the excavator coordinate system to obtain the position of the excavator bucket. The aforementioned preset conversion relationship between the image pixel coordinate system and the camera coordinate system refers to:

[0104] Please refer to Figure 4. In the figure, o is the origin of the pixel plane. If the coordinates of point Q in the camera coordinate system are (X... C ,Y C ZC Given that the coordinates of the camera in the image pixel coordinate system are (u, v), the offset of the optical center of the monocular camera relative to the center of the pixel plane is (u0, v0), the camera focal length is f, dx and dy represent the number of length units occupied by one pixel in the x and y directions, respectively, and the intrinsic parameter matrix of the camera is K, then...

[0105] The coordinates of point Q in the camera coordinate system can be obtained using the above formula. Then, the coordinates in the camera coordinate system can be converted to the coordinates in the excavator coordinate system to obtain the position of the excavator bucket.

[0106] It should be noted that since the bucket is a rigid body, once the position of one point on the bucket is determined, the position of the entire bucket can be determined. Therefore, the pixel coordinates of the bucket in the above bucket image can be the pixel coordinates of one point on the bucket in the bucket image, or it can be the pixel coordinates of multiple points.

[0107] In some embodiments, converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the plurality of key points, and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system includes the following steps:

[0108] First, based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the multiple key points, the depth of field distance from the bucket to the monocular camera is calculated.

[0109] Then, based on the depth of field distance from the bucket to the monocular camera and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the camera coordinate system to obtain the first coordinates of the bucket position point.

[0110] Finally, based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

[0111] In some embodiments, to facilitate determining the position of the excavator bucket, the position of the bucket's end can be used to determine the excavator bucket's position, which is the position point on the bottom edge of the bucket. Accordingly, the multiple key point positions include the vertex positions located at both ends of the bottom edge of the bucket, and the pixel coordinates of the bucket in the bucket image are the pixel coordinates of the position point on the bottom edge of the bucket.

[0112] Accordingly, the calculation of the depth-of-field distance from the bucket to the monocular camera based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the plurality of key points includes:

[0113] First, based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0114] In this embodiment, as shown in Figure 4, the vertices located at both ends of the bottom edge of the bucket are points A and B, respectively. The positions of points A and B can be represented as A(u A ,v A ),B(u B ,v B ), E(u E ,v E () is the midpoint of AB. Since the bottom edge and top edge of the bucket are always parallel to the horizontal plane of the cab, when the monocular camera is located in the cab, the bottom edge and top edge of the bucket are parallel to the x-axis of the pixel plane. Therefore, v A =v B =v E Point E is at the midpoint of AB, therefore, From this, we can obtain the pixel coordinates (u) of the midpoint of the bottom edge of the bucket. E ,v E ).

[0115] Then, based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0116] In this embodiment, x0, y0, and z0 are the offsets of the monocular camera's mounting position relative to the origin of the excavator's coordinate system on each axis; that is, the coordinates of the monocular camera relative to the excavator's coordinate system are (x0, y0, z0). The offset of the optical center relative to the pixel coordinate system is (u0, v0). The monocular camera is located to the left of the excavator's origin. y0 can be considered as the distance between the center of the bucket end and the camera's optical axis. Therefore, the depth-of-field distance from AB to the camera can be calculated as follows:

[0117] Where, d AB The depth of field distance from AB to the camera is the distance from the bottom edge of the bucket to the monocular camera.

[0118] Then, based on the first depth-of-field distance, the depth-of-field distance from the bucket to the monocular camera is obtained.

[0119] In this embodiment, the first depth of field distance can be used as the depth of field distance from the bucket to the monocular camera.

[0120] Then, based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, so as to obtain the first coordinates of the bucket position point.

[0121] In this embodiment, the pixel coordinates of the point on the bottom edge of the bucket can be any point on the bottom edge of the bucket. For ease of explanation, this embodiment mainly uses the midpoint of the bottom edge of the bucket as the example. Based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the position of the center point E at the end of the bucket in the camera coordinate system can be obtained, that is, the coordinates of the midpoint of the bottom edge of the bucket relative to the camera coordinate system are:

[0122] in, Let E be the coordinates of the center point E at the end of the bucket in the camera coordinate system.

[0123] Finally, based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

[0124] In this embodiment, the position of the center point E at the end of the bucket in the excavator coordinate system can be represented as:

[0125] Where θ is the vehicle body rotation angle;

[0126] (x E y E , z E () represents the coordinates of the midpoint of the bottom edge of the bucket relative to the excavator's coordinate system;

[0127] The formula for rotation about the z-axis in the excavator coordinate system;

[0128] This describes the transformation relationship between the camera coordinate system and the excavator coordinate system when the excavator is not rotating and the camera optical axis is horizontally forward.

[0129] In some embodiments, the plurality of key point locations may further include vertex locations at both ends of the bottom edge of the bucket; the pixel coordinates of the bucket in the bucket image are the pixel coordinates of the points on the bottom edge of the bucket;

[0130] Accordingly, the calculation of the depth-of-field distance from the bucket to the monocular camera based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the plurality of key points includes:

[0131] First, based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0132] In this embodiment, as shown in Figure 4, the vertices located at both ends of the bottom edge of the bucket are points C and D, respectively. The positions of points C and D can be represented as C(u) C ,v C ),D(u D ,v D ), F(u F ,v F () is the midpoint of CD. Since the bottom edge and top edge of the bucket are always parallel to the horizontal plane of the cab, when the monocular camera is located in the cab, the bottom edge and top edge of the bucket are parallel to the x-axis of the pixel plane. Therefore, v C =v D =v F Point F is at the midpoint of CD, therefore, From this, we can obtain the pixel coordinates (u) of the midpoint of the bottom edge of the bucket. F ,v F ).

[0133] Then, based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0134] In this embodiment, x0, y0, and z0 are the offsets of the monocular camera's mounting position relative to the origin of the excavator's coordinate system on each axis, i.e., the coordinates of the monocular camera relative to the excavator's coordinate system are (x0, y0, z0). The offset of the optical center relative to the pixel coordinate system is (u0, v0). The monocular camera is located to the left of the excavator's origin. y0 can be considered as the distance between the center of the bottom edge of the bucket and the camera's optical axis. Therefore, the depth-of-field distance from the CD to the camera can be calculated as follows:

[0135] Where, d CD The depth of field distance from the CD to the camera is the distance from the bottom edge of the bucket to the monocular camera.

[0136] Then, based on the second depth-of-field distance, the depth-of-field distance from the bucket to the monocular camera is obtained.

[0137] In this embodiment, the second depth of field distance can be used as the depth of field distance from the bucket to the monocular camera.

[0138] Then, based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, so as to obtain the first coordinates of the bucket position point.

[0139] Finally, based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

[0140] In this embodiment, the pixel coordinates of the position point on the bottom edge of the bucket can be any point on the bottom edge of the bucket. For ease of explanation, this embodiment mainly uses the midpoint of the bottom edge of the bucket as the position point as an example. As shown in Figure 4, the vertices at both ends of the bottom edge of the bucket are points C and D, and the coordinates of points C and D are respectively: C(u C ,v C ),D(u D ,v D ), F(u F ,v F Point F is the midpoint of CD. Accordingly, we can first find the position of point F in the camera coordinate system, and then, based on the camera imaging principle, convert the coordinates of point F in the camera coordinate system to the coordinates in the excavator coordinate system to obtain the position of the excavator bucket. The above conversion process of the coordinates of point F in the excavator coordinate system is the same as the conversion process of the coordinates of point E in the excavator coordinate system, and will not be repeated here.

[0141] In some embodiments, the plurality of key point locations include vertex locations at both ends of the upper bottom edge of the bucket and vertex locations at both ends of the lower bottom edge of the bucket;

[0142] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0143] First, based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0144] Then, based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0145] Then, based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0146] Then, based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0147] Finally, based on the first depth-of-field distance and the second depth-of-field distance, the depth-of-field distance from the bucket to the monocular camera is obtained.

[0148] In this embodiment, the calculation process for the first depth of field distance and the second depth of field distance is the same as the aforementioned calculation process, and will not be repeated here. By calculating the first depth of field distance and the second depth of field distance, the position of the excavator bucket can be determined by points on the bucket other than the top and bottom edges. For example, as shown in Figure 4, the position of the excavator bucket can be determined by the position of any point on AD. Specifically, after determining the positions and depth of field distances of points A and D, the ratio of the distance from that point to points A and D can be calculated, and then the position of that point can be obtained through the proportional relationship.

[0149] It should be noted that the above-mentioned determination of the excavator bucket position can also be made by multiple points, and this embodiment does not limit it.

[0150] In the above implementation process, the bucket image and the vehicle body rotation angle are acquired. The bucket image is acquired by a monocular camera set on the excavator body. Feature points of the excavator bucket are detected in the bucket image to obtain the positions of multiple key points. The coordinates of the monocular camera relative to the excavator coordinate system are obtained. Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted to the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket. Herein, the image pixel coordinate system is the coordinate system used to describe the position of pixels in the image when the monocular camera captures the image, the camera coordinate system is the coordinate system describing the position and orientation of the monocular camera, and the excavator coordinate system is the coordinate system established with a specific position or component of the excavator as the origin. By detecting feature points of the excavator bucket from the bucket image, and based on the feature points of the excavator bucket and the rotation angle of the excavator body, the position in the bucket image can be accurately converted into the position in the excavator coordinate system. This improves the accuracy of excavator bucket position detection, helps operators perform their work, and increases work efficiency.

[0151] In some embodiments, the bucket size can be further determined, and the method further includes: calculating the bucket size based on the first depth of field distance, the second depth of field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system.

[0152] In this embodiment, the bucket dimensions include the length and width of the bucket.

[0153] In some embodiments, calculating the bucket size based on the first depth-of-field distance, the second depth-of-field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system includes the following steps:

[0154] The first step is to obtain the length of the bottom edge of the bucket in the image pixel coordinate system based on the vertex positions located at both ends of the bottom edge of the bucket;

[0155] In this embodiment, based on the ordinates of the vertex positions located at both ends of the bottom edge of the bucket, the length of the bottom edge of the bucket in the pixel plane can be obtained, and the distance AB is: l AB =u B -u A , where l AB This represents the length of the bottom edge of the bucket in the image pixel coordinate system.

[0156] The second step is to calculate the width of the bucket based on the first depth of field distance and the length of the bottom edge of the bucket in the image pixel coordinate system.

[0157] In this embodiment, the width of the bucket, i.e., the length of AB, is:

[0158] Where W is the width of the bucket.

[0159] It should be noted that the above calculation refers to the length of AB. In practice, the length of CD can also be calculated. The calculation process is the same as that for AB: first, calculate the length of the bottom edge of the bucket in the image pixel coordinate system, i.e., l. CD =u C -u D , l CD Let be the length of the bottom edge of the bucket in the image pixel coordinate system, and then based on the second depth of field distance d. CD and l CD Substitute the values ​​into the formula above to calculate the width of the bucket.

[0160] The third step is to calculate the position of the first vertex based on the first depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the position of the vertex located at one end of the bottom edge of the bucket.

[0161] The fourth step is to calculate the second vertex position based on the second depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the upper bottom edge of the bucket. The vertex position located at one end of the upper bottom edge of the bucket corresponds to the vertex position located at one end of the lower bottom edge of the bucket.

[0162] In this embodiment, the aforementioned first vertex position and second vertex position refer to the positions of the first vertex and the second vertex in the camera coordinate system. Taking point A as the first vertex and point D as the second vertex as an example, the position of point A in the camera coordinate system is:

[0163] The position of point D in the camera coordinate system is:

[0164] in, Let A be the coordinates of point A in the camera coordinate system, i.e., the position of the first vertex; Let D be the coordinates of point D in the camera coordinate system, i.e., the position of the second vertex.

[0165] It should be noted that the first vertex mentioned above can also be point B, and the corresponding second vertex is point C. The positions of points B and C in the camera coordinate system can be calculated according to the above calculation process, which will not be repeated here.

[0166] Fifth step: Calculate the length of the bucket based on the positions of the first and second vertices;

[0167] In this embodiment, after obtaining the positions of the first and second vertices, the length of the bucket can be calculated using the distance calculation formula. Taking point A as the first vertex and point D as the second vertex as an example, the length of the bucket, which is the length of AD, can be expressed as:

[0168] Where L is the length of AD in the camera coordinate system.

[0169] It should be noted that when the first vertex is point B and the corresponding second vertex is point C, the length of BC can be calculated in the same way, which will not be elaborated here.

[0170] Step 6: Based on the width and length of the bucket, obtain the bucket dimensions.

[0171] The length and width of the bucket can be accurately calculated using the first depth of field distance, the second depth of field distance, the vertex positions at both ends of the top bottom edge of the bucket, and the vertex positions at both ends of the bottom bottom edge of the bucket, which helps the operator to guide the bucket operation more accurately.

[0172] In some embodiments, after determining the position of the excavator bucket, the angle between the bucket and the horizontal plane can be further determined to determine the bucket pose. After calculating the length of the bucket, the method further includes: calculating the angle between the bucket and the horizontal plane based on the first depth-of-field distance, the second depth-of-field distance, and the bucket size.

[0173] In this embodiment, the angle between the bucket and the horizontal plane can be calculated using geometric relationships and can be expressed as:

[0174] in, The angle between the bucket and the horizontal plane.

[0175] By using the first depth of field distance, the second depth of field distance, and the length of the bucket, the angle between the bucket and the horizontal plane can be accurately calculated, thereby determining the angle information of the excavator bucket. This further improves the accuracy of excavator bucket position detection, helps operators perform their work, and increases work efficiency.

[0176] In some embodiments, the landing point of the bucket can also be displayed in the bucket image. Specifically, after obtaining the position of the excavator bucket, the following steps are also included:

[0177] First, based on the position of the excavator bucket and the first coordinates of the bucket position point, the coordinates of the projection point are determined. The projection point is the projection point of the position point on the bottom edge of the bucket on the horizontal plane, and the coordinates of the projection point are the coordinates of the projection point in the camera coordinate system.

[0178] In this embodiment, the x and z coordinates of the projection point are the same as the x and z coordinates of the first coordinate of the point on the bottom edge of the bucket. The y coordinate of the projection point is the sum of the ordinate of the first coordinate of the point on the bottom edge of the bucket and the z coordinate of the excavator bucket position. Taking the point on the bottom edge of the bucket as the midpoint of the bottom edge of the bucket as an example, the coordinates of the projection point P of the midpoint E of the bucket end on the horizontal plane in the camera coordinate system can be expressed as:

[0179] Then, based on the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the coordinates of the projection point are converted into the coordinates of the projection point in the image pixel coordinate system to obtain the pixel coordinates of the projection point;

[0180] In this embodiment, the coordinates of the projection point P in the image pixel coordinate system are (u p ,v p u can be calculated using the following formula. p ,v p ,

[0181] Finally, based on the pixel coordinates of the projection points, annotations are made in the bucket image to show the vertical landing position of the bucket.

[0182] In this embodiment, point P can be marked on the bucket image to display the vertical landing point of the bucket, thus facilitating the operator's work. For example, when remotely controlling an excavator to dig and load ore, the operator can observe whether the bucket landing point is on the ore pile during digging and whether it is inside the ore clamp during unloading, which can enhance the operator's confidence and improve work efficiency.

[0183] It should be noted that the above projection point is described using the projection of the point on the bottom edge of the bucket. In actual implementation, it can also be the projection of other points. The calculation process of the pixel coordinates of the projection point is the same as the calculation process of the projection of the point on the bottom edge of the bucket, so it will not be repeated here.

[0184] The following example illustrates the solution. Please refer to Figure 8. A monocular camera is installed above the excavator's cab, and an angle sensor is installed on the vehicle body to measure the length and width of the bucket. The four vertices of the bucket are detected using a deep learning YOLOv8 model. Then, the position and tilt angle of the bucket relative to the excavator's coordinate system are calculated based on the pixel coordinates of the four vertices of the excavator and the vehicle body's rotation angle. Finally, the location of the bucket's landing point is displayed in the image.

[0185] Figure 1 is a flowchart illustrating the method for determining the excavator bucket position in this embodiment. It should be understood that although the steps in the flowchart of Figure 1 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps; they can be executed in other orders. Furthermore, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0186] This embodiment provides an excavator, which uses the above-described method for determining the excavator bucket position to determine the excavator bucket position.

[0187] In this embodiment, the excavator can be a wireless medium-to-long-range remote-controlled excavator, an intelligent excavator, or a short-range remote-controlled excavator, etc. By using the above-mentioned excavator bucket position determination method to determine the excavator bucket position, the accuracy of excavator bucket position detection is improved, which helps the operator to perform the work and improves the work efficiency.

[0188] Please refer to Figure 7, which schematically illustrates the structure of an excavator bucket position determination device according to an embodiment of this application. This embodiment provides an excavator bucket position determination device, including a first acquisition module 410, a detection module 420, a second acquisition module 430, and a conversion module 440, wherein:

[0189] The first acquisition module 410 is used to acquire bucket images and vehicle body rotation angles. The bucket images are acquired by a monocular camera set on the excavator body.

[0190] The detection module 420 is used to detect feature points of the excavator bucket in the bucket image and obtain the positions of multiple key points;

[0191] The second acquisition module 430 is used to acquire the coordinates of the monocular camera relative to the excavator coordinate system, wherein the excavator coordinate system is a coordinate system established with a specific position or component of the excavator as the origin;

[0192] The conversion module 440 is used to convert the pixel coordinates of the bucket in the bucket image into the coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset conversion relationship between the image pixel coordinate system and the camera coordinate system, thereby obtaining the position of the excavator bucket; wherein, the image pixel coordinate system is the coordinate system used when the monocular camera captures the image to describe the position of the pixel in the image, and the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera.

[0193] The excavator bucket position determination device includes a processor and a memory. The first acquisition module 410, the detection module 420, the second acquisition module 430, and the conversion module 440 are all stored in the memory as program units. The processor executes the program units stored in the memory to achieve the corresponding functions.

[0194] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and the precise position of the bucket can be determined by adjusting the kernel parameters.

[0195] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0196] This invention provides a machine-readable storage medium storing a program that, when executed by a processor, implements the excavator bucket position determination method.

[0197] This invention provides a processor for running a program, wherein the program executes the excavator bucket position determination method during runtime.

[0198] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 8. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor A01, it implements a method for determining the position of an excavator bucket. The display screen A04 of the computer device can be an LCD screen or an e-ink screen. The input device A05 of the computer device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0199] Those skilled in the art will understand that the structure shown in Figure 8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0200] In one embodiment, the excavator bucket position determination device provided in this application can be implemented as a computer program, which can run on a computer device as shown in FIG8. The memory of the computer device can store various program modules constituting the excavator bucket position determination device, such as the first acquisition module 410, detection module 420, second acquisition module 430, and conversion module 440 shown in FIG7. The computer program composed of these program modules causes the processor to execute the steps in the excavator bucket position determination method of the various embodiments of this application described in this specification.

[0201] The computer device shown in Figure 8 can execute step 210 through the first acquisition module 410 in the excavator bucket position determination device shown in Figure 7. The computer device can execute step 220 through the detection module 420. The computer device can execute step 230 through the second acquisition module 430. The computer device can execute step 240 through the conversion module 440.

[0202] This application provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-described excavator bucket position determination method by executing the instructions stored in the memory. When the processor executes the instructions, it performs the following steps:

[0203] The bucket image and the vehicle body rotation angle are acquired. The bucket image is obtained by a monocular camera installed on the excavator body.

[0204] Feature points of the excavator bucket are detected in the bucket image to obtain the positions of multiple key points;

[0205] Obtain the coordinates of the monocular camera relative to the excavator coordinate system;

[0206] Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket.

[0207] Wherein, the image pixel coordinate system is the coordinate system used to describe the position of a pixel in an image when the monocular camera captures an image, the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera, and the excavator coordinate system is the coordinate system established with a specific position or component of the excavator as the origin.

[0208] In one embodiment, converting the pixel coordinates of the bucket in the bucket image to coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle rotation angle, the positions of the multiple key points, and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system includes:

[0209] Based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the multiple key points, the depth of field distance from the bucket to the monocular camera is calculated.

[0210] Based on the depth of field distance from the bucket to the monocular camera and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the camera coordinate system to obtain the first coordinates of the bucket position point.

[0211] Based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

[0212] In one embodiment, the plurality of key point locations include the vertex locations at both ends of the bottom edge of the bucket;

[0213] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0214] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0215] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0216] Based on the first depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0217] In one embodiment, the step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes:

[0218] Based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

[0219] In one embodiment, the plurality of key point locations include the vertex locations at both ends of the bottom edge of the bucket;

[0220] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0221] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0222] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0223] Based on the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0224] In one embodiment, the step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes:

[0225] Based on the second depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

[0226] In one embodiment, the plurality of key point locations include vertex locations at both ends of the upper bottom edge of the bucket and vertex locations at both ends of the lower bottom edge of the bucket;

[0227] The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes:

[0228] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0229] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0230] Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained;

[0231] Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera.

[0232] Based on the first depth of field distance and the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

[0233] In one embodiment, the method further includes:

[0234] The bucket size is calculated based on the first depth of field distance, the second depth of field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system.

[0235] In one embodiment, calculating the bucket size based on the first depth-of-field distance, the second depth-of-field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system includes:

[0236] Based on the vertex positions located at both ends of the bottom edge of the bucket, the length of the bottom edge of the bucket in the image pixel coordinate system is obtained;

[0237] The width of the bucket is calculated based on the first depth of field distance and the length of the bottom edge of the bucket in the image pixel coordinate system;

[0238] The position of the first vertex is calculated based on the first depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the bottom edge of the bucket.

[0239] Based on the second depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the upper bottom edge of the bucket, the second vertex position is calculated, and the vertex position located at one end of the upper bottom edge of the bucket corresponds to the vertex position located at one end of the lower bottom edge of the bucket.

[0240] The length of the bucket is calculated based on the positions of the first and second vertices;

[0241] The bucket dimensions are obtained based on the width and length of the bucket.

[0242] In one embodiment, the method further includes:

[0243] Based on the first depth of field distance, the second depth of field distance, and the bucket size, the angle between the bucket and the horizontal plane is calculated.

[0244] In one embodiment, the method further includes:

[0245] Based on the excavator bucket position and the first coordinates of the bucket position point, the coordinates of the projection point are determined. The projection point is the projection of the bucket position point onto the horizontal plane, and the coordinates of the projection point are the coordinates of the projection point in the camera coordinate system.

[0246] Based on the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the coordinates of the projection point are converted into the coordinates of the projection point in the image pixel coordinate system to obtain the pixel coordinates of the projection point.

[0247] Based on the pixel coordinates of the projection points, annotations are made in the bucket image to show the vertical landing position of the bucket.

[0248] In one embodiment, detecting feature points of the excavator bucket in the bucket image to obtain the locations of multiple key points includes:

[0249] A preset bucket feature point detection model is used to perform target detection on the bucket image to obtain the locations of multiple key points.

[0250] In one embodiment, the construction process of the preset bucket feature point detection model includes:

[0251] Images of bucket movement are continuously acquired, and key points are annotated for each acquired image of bucket movement to obtain multiple sample images;

[0252] Based on the multiple sample images, the YOLOv8 model is trained to obtain the bucket feature point detection model.

[0253] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0254] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0255] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0256] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0257] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0258] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0259] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0260] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0261] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining the position of an excavator bucket, characterized in that, include: The bucket image and the vehicle body rotation angle are acquired. The bucket image is obtained by a monocular camera installed on the excavator body. Feature points of the excavator bucket are detected in the bucket image to obtain the positions of multiple key points; Obtain the coordinates of the monocular camera relative to the excavator coordinate system; Based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the excavator coordinate system to obtain the position of the excavator bucket. Wherein, the image pixel coordinate system is the coordinate system used to describe the position of a pixel in an image when the monocular camera captures an image, the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera, and the excavator coordinate system is the coordinate system established with a specific position or component of the excavator as the origin.

2. The method according to claim 1, characterized in that, The step of converting the pixel coordinates of the bucket in the bucket image to the coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle rotation angle, the positions of multiple key points, and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system includes: Based on the coordinates of the monocular camera relative to the excavator coordinate system and the positions of the multiple key points, the depth of field distance from the bucket to the monocular camera is calculated. Based on the depth of field distance from the bucket to the monocular camera and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the bucket in the bucket image are converted into the coordinates of the excavator bucket relative to the camera coordinate system to obtain the first coordinates of the bucket position point. Based on the vehicle body rotation angle and the coordinates of the monocular camera relative to the excavator coordinate system, the first coordinates of the bucket position point are converted into the coordinates of the bucket position point relative to the excavator coordinate system to obtain the excavator bucket position.

3. The method according to claim 2, characterized in that, The locations of the multiple key points include the apex positions located at both ends of the bottom edge of the bucket; The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes: Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained; Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera. Based on the first depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

4. The method according to claim 3, characterized in that, The step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system, based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes: Based on the first depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

5. The method according to claim 2, characterized in that, The locations of the multiple key points include the apex positions located at both ends of the bottom edge of the bucket; The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes: Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained; Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera. Based on the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

6. The method according to claim 5, characterized in that, The step of converting the pixel coordinates of the bucket in the bucket image into coordinates of the excavator bucket relative to the camera coordinate system, based on the depth-of-field distance from the bucket to the monocular camera and a preset transformation relationship between the image pixel coordinate system and the camera coordinate system, to obtain the first coordinates of the bucket position point, includes: Based on the second depth of field distance and the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the pixel coordinates of the point on the bottom edge of the bucket are converted into the coordinates of the point on the bottom edge of the bucket relative to the camera coordinate system, thus obtaining the first coordinates of the bucket position point.

7. The method according to claim 2, characterized in that, The multiple key point locations include the vertex locations at both ends of the upper bottom edge of the bucket and the vertex locations at both ends of the lower bottom edge of the bucket; The calculation of the depth of field distance from the bucket to the monocular camera, based on the coordinates of the monocular camera relative to the excavator's coordinate system and the positions of the multiple key points, includes: Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained; Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the first depth of field distance is calculated. The first depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera. Based on the vertex positions located at both ends of the bottom edge of the bucket, the pixel coordinates of the midpoint of the bottom edge of the bucket are obtained; Based on the pixel coordinates of the midpoint of the bottom edge of the bucket and the coordinates of the monocular camera relative to the excavator coordinate system, the second depth of field distance is calculated. The second depth of field distance is the depth of field distance from the bottom edge of the bucket to the monocular camera. Based on the first depth of field distance and the second depth of field distance, the depth of field distance from the bucket to the monocular camera is obtained.

8. The method according to claim 7, characterized in that, The method further includes: The bucket size is calculated based on the first depth of field distance, the second depth of field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system.

9. The method according to claim 8, characterized in that, The bucket size is calculated based on the first depth-of-field distance, the second depth-of-field distance, the vertex positions at both ends of the upper bottom edge of the bucket, the vertex positions at both ends of the lower bottom edge of the bucket, and the coordinates of the monocular camera relative to the excavator coordinate system, including: Based on the vertex positions located at both ends of the bottom edge of the bucket, the length of the bottom edge of the bucket in the image pixel coordinate system is obtained; The width of the bucket is calculated based on the first depth of field distance and the length of the bottom edge of the bucket in the image pixel coordinate system; The position of the first vertex is calculated based on the first depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the bottom edge of the bucket. Based on the second depth of field distance, the coordinates of the monocular camera relative to the excavator coordinate system, and the vertex position located at one end of the upper bottom edge of the bucket, the second vertex position is calculated, and the vertex position located at one end of the upper bottom edge of the bucket corresponds to the vertex position located at one end of the lower bottom edge of the bucket. The length of the bucket is calculated based on the positions of the first and second vertices; The bucket dimensions are obtained based on the width and length of the bucket.

10. The method according to claim 9, characterized in that, The method further includes: Based on the first depth of field distance, the second depth of field distance, and the bucket size, the angle between the bucket and the horizontal plane is calculated.

11. The method according to claim 2, characterized in that, The method further includes: Based on the excavator bucket position and the first coordinates of the bucket position point, the coordinates of the projection point are determined. The projection point is the projection of the bucket position point onto the horizontal plane, and the coordinates of the projection point are the coordinates of the projection point in the camera coordinate system. Based on the preset transformation relationship between the image pixel coordinate system and the camera coordinate system, the coordinates of the projection point are converted into the coordinates of the projection point in the image pixel coordinate system to obtain the pixel coordinates of the projection point. Based on the pixel coordinates of the projection points, annotations are made in the bucket image to show the vertical landing position of the bucket.

12. The method according to claim 1, characterized in that, The process involves detecting feature points of the excavator bucket in the bucket image to obtain the locations of multiple key points, including: A preset bucket feature point detection model is used to perform target detection on the bucket image to obtain the locations of multiple key points.

13. The method according to claim 12, characterized in that, The construction process of the pre-set bucket feature point detection model includes: Images of bucket movement are continuously acquired, and key points are annotated for each acquired image of bucket movement to obtain multiple sample images; Based on the multiple sample images, the YOLOv8 model is trained to obtain the bucket feature point detection model.

14. A device for determining the position of an excavator bucket, characterized in that, include: The first acquisition module is used to acquire bucket images and vehicle body rotation angles. The bucket images are acquired by a monocular camera installed on the excavator body. The detection module is used to detect feature points of the excavator bucket in the bucket image and obtain the positions of multiple key points; The second acquisition module is used to acquire the coordinates of the monocular camera relative to the excavator coordinate system, wherein the excavator coordinate system is a coordinate system established with a specific position or component of the excavator as the origin; The conversion module is used to convert the pixel coordinates of the bucket in the bucket image into the coordinates of the excavator bucket relative to the excavator coordinate system based on the coordinates of the monocular camera relative to the excavator coordinate system, the vehicle body rotation angle, the positions of the multiple key points, and the preset conversion relationship between the image pixel coordinate system and the camera coordinate system, thereby obtaining the position of the excavator bucket; wherein, the image pixel coordinate system is the coordinate system used when the monocular camera captures the image to describe the position of the pixel in the image, and the camera coordinate system is the coordinate system used to describe the position and orientation of the monocular camera.

15. An excavator, characterized in that, The excavator bucket position is determined using the method for determining the excavator bucket position as described in any one of claims 1-13.

16. An electronic device, characterized in that, The electronic device includes: At least one processor; A memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the excavator bucket position determination method according to any one of claims 1 to 13 by executing the instructions stored in the memory.

17. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the excavator bucket position determination method according to any one of claims 1 to 13.

Images