Mine truck hopper corner point positioning method and system based on key points and storage medium

Abstract

The invention discloses a mine truck bucket angular point positioning method and system based on key points and a storage medium, and belongs to the technical field of unmanned excavator, and the method comprises the steps: obtaining a multi-view mine truck image through an image collection device installed on an excavator truck; performing target detection on the multi-view mine card image to obtain a detected mine card bucket of a corresponding category; key point detection and speculation of corresponding categories are carried out on the mine truck hopper, two-dimensional coordinates of all corner points of the mine truck hopper in a coordinate system of an image acquisition device are obtained, and the key points are mark points on all edges of the mine truck hopper and comprise the corner points; obtaining a transformation matrix between a mine truck bucket coordinate system and an excavator coordinate system based on the two-dimensional coordinates; and according to the transformation matrix, obtaining the three-dimensional coordinate of the angular point of the mine truck bucket in the excavator coordinate system, thereby accurately sensing the position and attitude of the mine truck bucket and providing safety guarantee for the work of the unmanned excavator.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned excavators, and in particular relates to a method, system and storage medium for positioning bucket corners of a mining truck based on key points. Background Art

[0002] Smart mines are developing in full swing. The design and development of unmanned excavators are an important part of the ultimate realization of smart mines, which can promote the intelligent development of the mining industry. By developing unmanned excavators, the intelligence level of the entire mine can be improved and the transformation and upgrading of the mining industry can be promoted. Unmanned excavators can achieve 24-hour uninterrupted operation and improve production efficiency. Unmanned excavators can avoid accidents caused by human factors during operation and improve operation safety. Unmanned excavators can reduce labor costs and equipment maintenance costs. Unmanned excavators can adapt to various complex environments, such as high temperature, low temperature, high altitude, etc., to increase the scope of use of equipment.

[0003] When loading materials into a mining truck, an unmanned excavator needs to accurately sense the position and posture of the mining truck bucket. Therefore, identifying the mining truck bucket and detecting the 3D coordinates of the bucket corners is one of the important research contents of the unmanned excavator system. The existing scheme combines the target detection and key point detection model based on deep learning to detect the mining truck and identify the key points of the bucket, and then calculates the 3D coordinates of the corner points of the mining truck bucket by combining the shape and size of the mining truck bucket as prior information. Since the shape and size of the bucket are required as a priori, when there are multiple types of mining truck buckets in the working environment, the robustness of this type of key point-based scheme is generally not good enough, and the scalability of the method is low; and when the camera field of view cannot completely cover the detected object, only a part of the mining truck bucket appears in the image. Because the preset key points are blocked or do not appear in the image at all, the number of detections is insufficient, and the posture calculation algorithm will also fail. Summary of the invention

[0004] The present invention provides a method, system and storage medium for locating the corner points of a mining truck bucket based on key points. By acquiring mining truck images from multiple perspectives and performing bucket detection on different types of mining trucks, a bucket detection frame of the corresponding category is obtained, and based on the bucket detection frame, the complete two-dimensional coordinates of the bucket corner points are obtained, and then the two-dimensional coordinates are transformed into three-dimensional coordinates of the excavator, so as to accurately perceive the position and posture of the mining truck bucket, providing safety protection for the operation of the unmanned excavator.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for locating a bucket corner of a mining truck based on key points, comprising:

[0007] The image acquisition device installed on the excavator obtains images of the mining truck from multiple perspectives;

[0008] Performing target detection on the multi-view mining truck images to obtain a detected mining truck bucket of a corresponding category;

[0009] Detect and infer key points of corresponding categories on the mining truck bucket, and obtain the two-dimensional coordinates of each corner point of the mining truck bucket in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the mining truck bucket, including the corner points;

[0010] Acquire a transformation matrix between the mining truck bucket coordinate system and the excavator coordinate system based on the two-dimensional coordinates;

[0011] The three-dimensional coordinates of the bucket corner points of the mining truck in the excavator coordinate system are obtained according to the transformation matrix.

[0012] Optionally, the acquiring of multi-perspective working environment images by an image acquisition device installed on the excavator includes: respectively acquiring working environment images by multi-field-of-view cameras pre-installed on the excavator, wherein the working environment includes an unmanned mining truck for loading materials.

[0013] Optionally, performing target detection on the multi-view mining truck images to obtain a detected mining truck bucket of a corresponding category includes:

[0014] The mining truck image is input into a pre-trained target detection model to obtain a mining truck bucket detection frame of the corresponding category;

[0015] The method for obtaining the pre-trained target detection model includes:

[0016] Obtain historical samples of mining truck images containing the truck bed detection frame;

[0017] The historical samples are classified according to the types of mining truck buckets and the corresponding mining truck category labels are marked, wherein the mining truck with a rectangular bucket longitudinal section is labeled Truck1, the mining truck with a rectangular bucket longitudinal section is labeled Truck2, and the mining truck with a trapezoidal bucket longitudinal section is labeled Truck3;

[0018] The classified and labeled historical samples are input into the pre-built target detection model to obtain a trained target detection model.

[0019] Optionally, the detecting and inferring of key points of corresponding categories on the mining truck bucket to obtain the two-dimensional coordinates of each corner point of the mining truck bucket in the coordinate system of the image acquisition device includes:

[0020] Establishing a rectangular coordinate system in the mining truck bucket detection frame;

[0021] The mining truck bucket detection frame after the coordinate system is established is input into the pre-trained key point detection model to obtain the key points covered by the mining truck image and their two-dimensional coordinates in the camera coordinate system;

[0022] Based on the key points covered by the mining card image, the corner points not covered by the mining card image are inferred, and the two-dimensional coordinates of the uncovered corner points in the camera coordinate system are obtained.

[0023] Optionally, the inferring key points not covered by the mining card image based on the key points covered by the mining card image to obtain two-dimensional coordinates of the uncovered corner points in the camera coordinate system includes:

[0024] Obtain two edges that should intersect at the corner point not covered by the mining card image, fit corresponding straight line equations according to the coordinates of the key points covered by the mining card image on the two edges, and obtain the coordinates of the uncovered corner point through the straight line equations of the two edges;

[0025] Or, obtain an edge that should extend to the corner point not covered by the mining card image, fit a straight line equation according to the coordinates of the key points covered by the mining card image on the edge, and obtain the distance between two key points covered by the mining card image on the edge according to the straight line equation, obtain the scale of the edge according to the distance, and obtain the coordinates of the corner point not covered by the mining card image according to the scale.

[0026] Optionally, acquiring a transformation matrix between a mining truck bucket coordinate system and an excavator coordinate system based on the two-dimensional coordinates includes:

[0027] According to the pre-calibration of the camera installed on the excavator, a first transformation matrix T1 of the camera coordinate system relative to the coordinate system of the excavator is obtained;

[0028] According to the rotation angle between the excavator onboard and the excavator offboard during operation, a second transformation matrix T2 of the coordinate system of the excavator onboard relative to the coordinate system of the excavator offboard is obtained;

[0029] Acquire a third transformation matrix T3 of the mining truck bucket coordinate system relative to the camera coordinate system according to the two-dimensional coordinates;

[0030] According to the first, second and third transformation matrices, the transformation matrix T of the mining truck bucket coordinate system relative to the excavator undercarriage coordinate system is obtained, T = T1 T2 T3.

[0031] Optionally, obtaining a third transformation matrix T3 of the mining truck bucket coordinate system relative to the camera coordinate system according to the two-dimensional coordinates includes:

[0032] Construct a three-dimensional coordinate system based on the actual contour of the mining truck bucket;

[0033] Use parameters W, H1, H2, L1, L2 to describe the x-coordinate, y-coordinate and z-coordinate of each corner point of the truck bucket, respectively, and obtain the three-dimensional coordinates of each corner point in the truck bucket coordinate system, where L1 represents the rectangular length of the longitudinal section of Truck1, Truck2 and Truck3 buckets, H1 represents the rectangular height of the longitudinal section of Truck1 and Truck2 buckets and the length of the longer bottom side of the trapezoid of the longitudinal section of Truck3 bucket, H2 represents the rectangular height of the longitudinal section of Truck1 and Truck2 buckets and the length of the shorter bottom side of the trapezoid of the longitudinal section of Truck3 bucket, L2 represents the perpendicular distance from the vertex of the triangle of the longitudinal section of Truck2 and Truck3 buckets as the corner point of the bucket to the side where H2 is located, W represents the width of the truck bucket, W is the x-coordinate of the corner point of the bucket in the three-dimensional coordinate system, H1 and H2 are the y-coordinates of the corner point of the bucket in the three-dimensional coordinate system, and L1 and L2 are the z-coordinates of the corner point of the bucket in the three-dimensional coordinate system;

[0034] The two-dimensional coordinates of each corner point in the camera coordinate system are matched with the three-dimensional coordinates of each corner point in the truck bucket coordinate system in sequence to obtain a two-dimensional-three-dimensional point pair, and the two-dimensional-three-dimensional point pair is input into the PNP solution module to obtain the third transformation matrix T3 of the truck bucket coordinate system relative to the camera coordinate system.

[0035] Optionally, obtaining the three-dimensional coordinates of the corner points of the mine truck bucket in the excavator coordinate system according to the transformation matrix includes: substituting the three-dimensional coordinates of each corner point in the mine truck bucket coordinate system into the transformation matrix T to obtain the coordinates of each corner point of the mine truck in space relative to the excavator.

[0036] In a second aspect, the present invention provides a mining truck bucket corner positioning system based on key points, comprising:

[0037] Mining truck image acquisition module: used to acquire multi-viewing mining truck images through an image acquisition device installed on the excavator;

[0038] A mine truck bucket detection module is used to perform target detection on the multi-view mine truck image to obtain a detected mine truck bucket of a corresponding category;

[0039] The bucket key point detection module is used to detect the key points of the corresponding category of the bucket of the mining truck, and obtain the two-dimensional coordinates of the corner points of the bucket of the mining truck in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the bucket of the mining truck, including the corner points;

[0040] A bucket-excavator coordinate transformation matrix acquisition module is used to acquire a transformation matrix between a bucket coordinate system of a mining truck and a coordinate system of an excavator based on the two-dimensional coordinates;

[0041] Corner point three-dimensional coordinate acquisition module: used to obtain the three-dimensional coordinates of the corner points of the mining truck bucket in the excavator coordinate system according to the transformation matrix.

[0042] In a third aspect, the present invention provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for locating the bucket corner of a mining truck based on key points as described in any step of the first aspect.

[0043] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: by performing bucket target detection on mining truck images under multiple viewing angles according to different mining truck types, an accurate bucket detection frame of the corresponding category can be obtained, and key point detection and inference are performed on the bucket detection frame, so that all corner points of the bucket detection frame can be obtained, thereby constructing a two-dimensional and three-dimensional coordinate system and calculating the coordinate transformation matrix between each coordinate system, and finally obtaining the three-dimensional coordinates of all corner points in the coordinate system of the excavator, thereby realizing accurate perception of the bucket posture, thereby ensuring the safety and efficiency of unmanned excavation operations and improving the intelligence level of mines; the present invention classifies mining trucks in advance according to the bucket cross-sectional type, and trains a target detection model according to the classified samples, so as to adapt to various types of bucket targets that may appear in the operation scene, and has a wide range of applications. In addition, considering that the camera field of view cannot cover the entire bucket, the position of all corner points of the bucket can be provided by combining increasing the camera field of view and inferring invisible key points based on visible key points, thereby greatly improving the accuracy of bucket posture detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a flow chart of a method for positioning a corner point of a mining truck bucket based on key points in an embodiment of the present invention;

[0045] Figure 2 The figure shows a schematic diagram of an unmanned excavation operation scene in an embodiment of the present invention;

[0046] Figure 3 The figure shows a schematic diagram of the structure of a multi-camera field of view in one embodiment of the present invention;

[0047] Figure 4 The figure shows an architecture diagram of a corner point positioning entity system in one embodiment of the present invention;

[0048] Figure 5 The figure is a flow chart of a target detection module in a positioning system in an embodiment of the present invention;

[0049] Figure 6 Shown is a schematic diagram of bucket sample classification in one embodiment of the present invention;

[0050] Figure 7The figure is a flow chart of a key point detection module in a positioning system in an embodiment of the present invention;

[0051] Figure 8 Shown is a diagram showing a calculation principle for estimating invisible key points in an embodiment of the present invention;

[0052] Fig. 9 Shown is another calculation principle diagram for estimating invisible key points in one embodiment of the present invention;

[0053] Fig.10 The figure is a flow chart of a corner point 3D coordinate calculation module in a positioning system in an embodiment of the present invention;

[0054] Fig.11 Shown is a schematic diagram of a vehicle bucket coordinate system in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0056] Example 1

[0057] like Figure 1 As shown, this embodiment provides a method for locating the bucket corner of a mining truck based on key points, including:

[0058] The image acquisition device installed on the excavator obtains images of the mining truck from multiple perspectives;

[0059] Performing target detection on the multi-view mining truck images to obtain a detected mining truck bucket of a corresponding category;

[0060] Detect and infer key points of corresponding categories on the mining truck bucket, and obtain the two-dimensional coordinates of each corner point of the mining truck bucket in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the mining truck bucket, including the corner points;

[0061] Acquire a transformation matrix between the mining truck bucket coordinate system and the excavator coordinate system based on the two-dimensional coordinates;

[0062] The three-dimensional coordinates of the bucket corner points of the mining truck in the excavator coordinate system are obtained according to the transformation matrix.

[0063] By acquiring images of mining trucks from multiple perspectives, the shortcomings of the existing technology in which a single camera has insufficient field of view and collects too few corner points of mining trucks are overcome. Mining trucks are classified in advance, so that the bucket detection frames of corresponding categories are obtained based on the mining truck images, which can improve the robustness of detection of multiple types of mining truck buckets. In addition, based on key point detection, invisible key points are inferred based on the detected visible key points, which improves the stability of posture detection when the mining truck bucket in the image is incomplete.

[0064] Example 2

[0065] Based on Example 1, this example also makes the following design.

[0066] like Figure 2 As shown, the system operation scenario described in this embodiment is a scenario where an excavator and a mining truck cooperate to load materials. During operation, the mining truck enters the designated area, and the unmanned excavator detects the position and posture of the mining truck bucket for subsequent planning and control of loading materials into the bucket.

[0067] like Figure 3 As shown, a forward-looking camera, a left-looking camera, and a right-looking camera are installed on the top, left, and right sides of the unmanned excavator, respectively, and the three cameras are adjusted to appropriate pitch angles to ensure that the cameras are not blocked by other objects when collecting image data of the working environment. In addition, the cameras need to be calibrated in advance to obtain the first transformation matrix T1 (the first transformation matrix of the forward-looking camera, the first transformation matrix of the left-looking camera, and the first transformation matrix of the right-looking camera) between the camera coordinate system and the coordinate system of the excavator.

[0068] A rotary encoder is installed at the turntable between the upper and lower vehicles of the excavator to measure the rotation angle between the upper and lower vehicles during operation and to calculate the second transformation matrix T2 from the upper and lower vehicles to the lower and upper vehicles.

[0069] like Figure 4 As shown, the corner point positioning method is realized by a corner point positioning entity system integrated on the excavator. The three modules in the corner point positioning entity system are described in detail below.

[0070] 1. Object Detection Module

[0071] like Figure 5 As shown, the image data of the working environment collected by the three cameras at the same time are input into the target detection module to determine which camera has detected the mining truck, and record the first transformation matrix T1 corresponding to the camera, and the result is the detection frame of the target object area.

[0072] In this embodiment, the mining truck bucket target detection module is based on the target detection network model and sample classification. Sample classification means that the target detection module classifies trucks according to their types, and different truck types output different types of detection box categories. Figure 6 As shown in the figure, after accumulating a large number of samples, it is found that the cross-sections of existing mining truck bodies can be basically summarized into three types: rectangle, rectangle + triangle, trapezoid + triangle. After classification and sorting, in the sample labeling stage of the target detection module, the truck bodies are classified according to their types, and different types of trucks are labeled with corresponding labels such as: Truck1, Truck 2, Truck3, and then the target detection network model is trained based on the classified labels.

[0073] 2. Key point detection module

[0074] Key points are pixels in an image that have semantic information. The key points set in this embodiment include the corner points of the truck bed. Three key points may also be set on each edge of the truck bed. The number of key points on the edge of the bed may be increased or decreased as needed.

[0075] like Figure 7 As shown, the detection frame and the original image data of the camera are input into the key point detection module, and the invisible key points are inferred based on the detected visible key points, and the output is the 2D pixel coordinates P_2d_image of the corner points of the mining truck bucket in the image: .

[0076] When the parking position of the mining truck cannot ensure that the mining truck appears completely in the image field of view, the mathematical equation of the edge can be fitted according to the existing key points on the edge of the truck, and then the coordinates of the corner points on the edge can be calculated according to the size of the truck or other prior information, and the invisible key points can be inferred:

[0077] (1) If Figure 8 As shown, the straight line equation 1 is fitted according to the four key points on edge 1, and then the straight line equation 2 is fitted according to the three key points on edge 2. The 2D image coordinates (u, v) of the corner point can be calculated by intersecting the two straight lines.

[0078] (2) If Fig. 9 As shown, the straight line equation 1 is fitted according to the two key points on edge 1, and then the ratio of the pixel distance d between the two key points and the actual distance D is calculated. Finally, according to the actual distance L between the two corner points on edge 1, the 2D image coordinates (u, v) of another corner point on edge 1 that is not in the image can be calculated.

[0079] 3. Corner point 3D coordinate calculation module

[0080] like Fig.10As shown, the 2D pixel coordinates are input into the corner point 3D coordinate calculation module, and the coordinates of the corner points of the truck bucket relative to the camera in space are calculated based on the pnp algorithm. The angle data of the rotary encoder is read, and the second transformation matrix T2 from the excavator to the excavator is calculated. Combined with the first transformation matrix T1 from the camera to the excavator, the three-dimensional coordinates P_3d_truck of the corner points of the truck bucket in the excavator coordinate system are obtained: .

[0081] (1) Definition of the truck bed coordinate system and generation of key points

[0082] like Fig.11 As shown, the upper left corner of the truck bed is defined as the coordinate origin, the z-axis is forward along the edge of the bed, the x-axis is to the right along the edge of the bed, and the y-axis is downward along the edge of the bed.

[0083] Five parameters (L1, L2, H1, H2, W) are used to mathematically describe the three types of trucks. When the detection result is Truck3, these five parameters are used; when the detection result is Truck2, H1=H2; when the detection result is Truck1, H1=H2 and L2=0. Through this representation, the 3D coordinates of the eight corner points of the mining truck bucket can be calculated using a unified formula. You only need to enter the corresponding parameters under different truck types. The 3D coordinates of the eight corner points are expressed as: (0,0,0), (0,0,L1), (W,0,L1), (W,0,0), (0,H2,-L2), (0,H1,L1), (W,H1,L1), (W,H2,-L2).

[0084] (2) PNP calculation transformation matrix

[0085] The P_2d_image and P_3d_truck calculated by the previous module are matched one by one to form a 2D-3D point pair, which is input into the PNP solution module to calculate the transformation matrix T3 between the truck bucket coordinate system and the camera coordinate system on the excavator, and then the transformation matrix T between the truck bucket and the excavator is obtained, T= T1 T2 T3.

[0086] When calculating the second transformation matrix T2, the excavator loading and unloading can be considered as two rigid bodies. It is assumed that the two components will only rotate around the z-axis, and the rotation angle angle is calculated based on the reading of the rotary encoder. The translation between the loading and unloading needs to be measured based on the manufacturing drawings. Generally, there is only an offset D in the z direction between the loading and unloading coordinate systems of the excavator. The calculation formula is as follows:

[0087] ,

[0088] According to the transformation matrix T, the 3D coordinates P_3d_truck of the eight corner points of the truck bucket in the truck bucket coordinate system are transformed to obtain the 3D coordinates of the eight corner points of the truck bucket in the excavator coordinate system for use in regulation and control.

[0089] Example 3

[0090] This embodiment provides a mining truck bucket corner positioning system based on key points, including:

[0091] Mining truck image acquisition module: used to acquire multi-viewing mining truck images through an image acquisition device installed on the excavator;

[0092] A mine truck bucket detection module is used to perform target detection on the multi-view mine truck image to obtain a detected mine truck bucket of a corresponding category;

[0093] The bucket key point detection module is used to detect the key points of the corresponding category of the bucket of the mining truck, and obtain the two-dimensional coordinates of the corner points of the bucket of the mining truck in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the bucket of the mining truck, including the corner points;

[0094] A bucket-excavator coordinate transformation matrix acquisition module is used to acquire a transformation matrix between a bucket coordinate system of a mining truck and a coordinate system of an excavator based on the two-dimensional coordinates;

[0095] Corner point three-dimensional coordinate acquisition module: used to obtain the three-dimensional coordinates of the corner points of the mining truck bucket in the excavator coordinate system according to the transformation matrix.

[0096] Example 4

[0097] This embodiment provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for locating the bucket corner of a mining truck based on key points as described in any step of Embodiment 2 is implemented.

[0098] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for locating the bucket corner of a mining truck based on key points, characterized in that: include: The image acquisition device installed on the excavator obtains images of the mining truck from multiple perspectives; Performing target detection on the multi-view mining truck images to obtain a detected mining truck bucket of a corresponding category; Detect and infer key points of corresponding categories on the mining truck bucket, and obtain the two-dimensional coordinates of each corner point of the mining truck bucket in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the mining truck bucket, including the corner points; Acquire a transformation matrix between the mining truck bucket coordinate system and the excavator coordinate system based on the two-dimensional coordinates; The three-dimensional coordinates of the bucket corner points of the mining truck in the excavator coordinate system are obtained according to the transformation matrix.

2. The method for locating the bucket corner of a mining truck based on key points according to claim 1 is characterized in that: The method of acquiring multi-perspective working environment images by an image acquisition device installed on an excavator includes: acquiring working environment images respectively by using a multi-field-of-view camera pre-installed on the excavator, wherein the working environment includes an unmanned mining truck for loading materials.

3. The method for positioning the bucket corner of a mining truck based on key points according to claim 1, characterized in that: The performing target detection on the multi-view mining truck images to obtain a detected mining truck bucket of a corresponding category includes: The mining truck image is input into a pre-trained target detection model to obtain a mining truck bucket detection frame of the corresponding category; The method for obtaining the pre-trained target detection model includes: Obtain historical samples of mining truck images containing the truck bed detection frame; The historical samples are classified according to the types of mining truck buckets and the corresponding mining truck category labels are marked, wherein the mining truck with a rectangular bucket longitudinal section is labeled Truck1, the mining truck with a rectangular bucket longitudinal section is labeled Truck2, and the mining truck with a trapezoidal bucket longitudinal section is labeled Truck3; The classified and labeled historical samples are input into the pre-built target detection model to obtain a trained target detection model.

4. The method for locating the bucket corner of a mining truck based on key points according to claim 3 is characterized in that: The detecting and inferring of key points of the corresponding category on the mining truck bucket to obtain the two-dimensional coordinates of each corner point of the mining truck bucket in the coordinate system of the image acquisition device includes: Establishing a rectangular coordinate system in the mining truck bucket detection frame; The mining truck bucket detection frame after the coordinate system is established is input into the pre-trained key point detection model to obtain the key points covered by the mining truck image and their two-dimensional coordinates in the camera coordinate system; Based on the key points covered by the mining card image, the corner points not covered by the mining card image are inferred, and the two-dimensional coordinates of the uncovered corner points in the camera coordinate system are obtained.

5. The method for locating the bucket corner of a mining truck based on key points according to claim 4 is characterized in that: The inferring the key points not covered by the mining card image based on the key points covered by the mining card image, and obtaining the two-dimensional coordinates of the uncovered corner points in the camera coordinate system, includes: Obtain two edges that should intersect at the corner point not covered by the mining card image, fit corresponding straight line equations according to the coordinates of the key points covered by the mining card image on the two edges, and obtain the coordinates of the uncovered corner point through the straight line equations of the two edges; Or, obtain an edge that should extend to the corner point not covered by the mining card image, fit a straight line equation according to the coordinates of the key points covered by the mining card image on the edge, and obtain the distance between two key points covered by the mining card image on the edge according to the straight line equation, obtain the scale of the edge according to the distance, and obtain the coordinates of the corner point not covered by the mining card image according to the scale.

6. The method for locating the bucket corner of a mining truck based on key points according to claim 4 is characterized in that: The method of obtaining a transformation matrix between a mining truck bucket coordinate system and an excavator coordinate system based on the two-dimensional coordinates includes: According to the pre-calibration of the camera installed on the excavator, a first transformation matrix T1 of the camera coordinate system relative to the coordinate system of the excavator is obtained; According to the rotation angle between the excavator onboard and the excavator offboard during operation, a second transformation matrix T2 of the coordinate system of the excavator onboard relative to the coordinate system of the excavator offboard is obtained; Acquire a third transformation matrix T3 of the mining truck bucket coordinate system relative to the camera coordinate system according to the two-dimensional coordinates; According to the first, second and third transformation matrices, the transformation matrix T of the mining truck bucket coordinate system relative to the excavator undercarriage coordinate system is obtained, T = T1 T2 T3.

7. The method for locating the bucket corner of a mining truck based on key points according to claim 6 is characterized in that: The method of obtaining a third transformation matrix T3 of the mining truck bucket coordinate system relative to the camera coordinate system according to the two-dimensional coordinates includes: Construct a three-dimensional coordinate system based on the actual contour of the mining truck bucket; Use parameters W, H1, H2, L1, L2 to describe the x-coordinate, y-coordinate and z-coordinate of each corner point of the truck bucket, respectively, and obtain the three-dimensional coordinates of each corner point in the truck bucket coordinate system, where L1 represents the rectangular length of the longitudinal section of Truck1, Truck2 and Truck3 buckets, H1 represents the rectangular height of the longitudinal section of Truck1 and Truck2 buckets and the length of the longer bottom side of the trapezoid of the longitudinal section of Truck3 bucket, H2 represents the rectangular height of the longitudinal section of Truck1 and Truck2 buckets and the length of the shorter bottom side of the trapezoid of the longitudinal section of Truck3 bucket, L2 represents the perpendicular distance from the vertex of the triangle of the longitudinal section of Truck2 and Truck3 buckets as the corner point of the bucket to the side where H2 is located, W represents the width of the truck bucket, W is the x-coordinate of the corner point of the bucket in the three-dimensional coordinate system, H1 and H2 are the y-coordinates of the corner point of the bucket in the three-dimensional coordinate system, and L1 and L2 are the z-coordinates of the corner point of the bucket in the three-dimensional coordinate system; The two-dimensional coordinates of each corner point in the camera coordinate system are matched with the three-dimensional coordinates of each corner point in the truck bucket coordinate system in sequence to obtain a two-dimensional-three-dimensional point pair, and the two-dimensional-three-dimensional point pair is input into the PNP solution module to obtain the third transformation matrix T3 of the truck bucket coordinate system relative to the camera coordinate system.

8. The method for locating the bucket corner of a mining truck based on key points according to claim 6, characterized in that: The three-dimensional coordinates of the corner points of the mining truck bucket in the excavator coordinate system are obtained according to the transformation matrix, including: substituting the three-dimensional coordinates of each corner point in the mining truck bucket coordinate system into the transformation matrix T to obtain the coordinates of each corner point of the mining truck relative to the excavator in space.

9. A mining truck bucket corner positioning system based on key points, characterized in that: include: Mining truck image acquisition module: used to acquire multi-viewing mining truck images through an image acquisition device installed on the excavator; A mine truck bucket detection module is used to perform target detection on the multi-view mine truck image to obtain a detected mine truck bucket of a corresponding category; The bucket key point detection module is used to detect the key points of the corresponding category of the bucket of the mining truck, and obtain the two-dimensional coordinates of the corner points of the bucket of the mining truck in the coordinate system of the image acquisition device, wherein the key points are the marked points on each edge of the bucket of the mining truck, including the corner points; A bucket-excavator coordinate transformation matrix acquisition module is used to acquire a transformation matrix between a bucket coordinate system of a mining truck and a coordinate system of an excavator based on the two-dimensional coordinates; Corner point three-dimensional coordinate acquisition module: used to obtain the three-dimensional coordinates of the corner points of the mining truck bucket in the excavator coordinate system according to the transformation matrix.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for locating the bucket corner of a mining truck based on key points as described in any one of claims 1 to 8 is implemented.

Images