Method and system for correcting unloading deviation of loader

Abstract

The invention provides a method and a system for correcting unloading deviation of a loader. The method comprises the following steps: carrying out image acquisition on an identifiable label by using a camera to obtain a label image; camera pose information is determined according to the label image, and the camera pose information is used for representing the pose of the camera in a local coordinate system; coordinate transformation is conducted on the camera pose information, target pose information of the loader bucket is obtained, and the target pose information is used for representing the pose of the bucket in a global coordinate system; and correcting the initial pose information of the bucket through the target pose information. When the loader works in a non-unloading area, initial pose information is obtained through a global positioning map provided by SLAM, when the loader works in an unloading area, an Apriltag positioning function is started, target pose information of a bucket is obtained, then the initial pose information is corrected through the target pose information, and the positioning precision of the bucket at an unloading opening is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of loader operations, and in particular to a method and system for correcting unloading deviations of a loader. Background Art

[0002] The unmanned vehicle is composed of an environmental perception module, a path planning module, and a vehicle control module. At present, in the mixing station scenario, due to the existence of occlusion, the RTK (Real-time Kinematic) positioning method cannot be used, so the laser radar SLAM (Simultaneous Localization and Mapping) positioning method or visual SLAM positioning method is mostly used. The laser radar is installed above the cab of the rear body of the loader, and the real-time positioning function is realized by obtaining the current point cloud frame data in real time and comparing it with the map data.

[0003] However, the SLAM positioning method has the problem of error accumulation, and coupled with the control error of the control module, the positioning accuracy decreases after long-term operation. When the loader is unloading, there is a risk of collision between the bucket and the unloading port, resulting in a large unloading deviation of the loader and reduced operating efficiency. Summary of the invention

[0004] In view of this, the present application provides a method and system for correcting unloading deviation of a loader, which solves the technical problem that the unloading deviation of the loader is large and the operating efficiency is reduced.

[0005] As the first aspect of the present application, the present application provides a method for correcting unloading deviations of a loader, comprising: using a camera to capture images of an identifiable tag to obtain a label image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port; determining the camera posture information according to the label image, wherein the camera posture information is used to characterize the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin; performing coordinate transformation on the camera posture information to obtain target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin; correcting the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to characterize the initial posture of the bucket obtained based on the global positioning map of the loader.

[0006] In a possible implementation of the present application, the camera posture information is coordinate transformed to obtain the target posture information of the loader bucket, including: according to the coordinate association relationship between the camera and the front body of the loader, the camera posture information is coordinate transformed to obtain the posture information of the front body of the loader; according to the posture information of the front body of the loader, the first position information of the loader bucket is determined, wherein the first position information is used to characterize the posture information of the bucket in the local coordinate system; based on a preset transformation matrix, the first position information is matrix transformed to obtain the target posture information, wherein the preset transformation matrix is ​​used to characterize the preset matrix for converting the local coordinate system into the global coordinate system.

[0007] In a possible implementation of the present application, the initial posture information of the bucket is corrected through the target posture information, including: determining the adjustment parameters of the bucket according to the target posture information and the initial posture information; adjusting the current operating parameters of the bucket based on the adjustment parameters so that the unloading accuracy of the loader bucket at the unloading port meets the preset unloading accuracy.

[0008] In a possible implementation of the present application, before correcting the initial posture information of the bucket through the target posture information, the method includes: obtaining the lidar point cloud data and sensor data of the loader; using the lidar point cloud data and sensor data to build a scene map to obtain a global positioning map; and obtaining the initial posture information of the bucket based on the global positioning map.

[0009] In a possible implementation of the present application, the label image contains multiple corner points of the label, and the camera pose information is determined based on the label image, including: obtaining the two-dimensional coordinates of the multiple corner points in the pixel coordinate system of the camera; obtaining the three-dimensional coordinates of the multiple corner points in the local coordinate system; determining the homography matrix by matching the two-dimensional coordinates and the three-dimensional coordinates, wherein the homography matrix is ​​used to represent the transformation matrix between the identifiable label and the pixel coordinate system; determining the transformation matrix of the identifiable label in the camera coordinate system based on the two-dimensional coordinates, the three-dimensional coordinates and the homography matrix, wherein the transformation matrix is ​​used to represent the matrix of the transformation between the camera coordinate system and the local coordinate system, and the transformation matrix includes a rotation matrix and a translation matrix; inverting the transformation matrix to obtain the camera pose information in the local coordinate system.

[0010] In a possible implementation of the present application, after the image of the identifiable tag is captured by a camera and the tag image is obtained, the method further includes: performing dedistortion processing on the tag image.

[0011] In a possible implementation of the present application, before dedistorting the label image, the method further includes: calibrating the camera's internal parameters to obtain an internal parameter matrix; and correcting the label image based on the internal parameter matrix.

[0012] In a possible implementation of the present application, after the label image is dedistorted, the method further includes: performing threshold segmentation on the standard label image obtained by the dedistortion process to obtain multiple segmented images; performing connected domain search on the multiple segmented images to obtain identifiable labels in the label image, wherein the identifiable labels include multiple corner points of the label.

[0013] As a second aspect of the present application, the present application also provides a unloading deviation correction system for a loader, including: an image acquisition module, used to use a camera to acquire an image of an identifiable tag to obtain a label image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port; a posture calculation module, used to determine the camera posture information according to the label image, wherein the camera posture information is used to characterize the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin; a coordinate transformation module, used to perform coordinate transformation on the camera posture information to obtain the target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in the global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin; a posture correction module, used to correct the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to characterize the initial posture of the bucket obtained based on the global positioning map of the loader.

[0014] As a third aspect of the present application, the present application further provides a loader, including: a unloading deviation correction system of the loader as described above.

[0015] As a fourth aspect of the present application, the present application further provides a computer-readable storage medium, the storage medium storing a computer program, and the computer program is used to execute any of the above-mentioned methods for correcting unloading deviations of a loader.

[0016] The present application provides a method for correcting unloading deviations of a loader, comprising: using a camera to capture an image of an identifiable tag to obtain a tag image, wherein the camera is installed below a hydraulic cylinder of a front body of the loader, the identifiable tag is installed at a horizontal middle position of a retaining wall of a discharge port, and the identifiable tag is used to uniquely identify a corresponding discharge port; determining camera posture information according to the tag image, wherein the camera posture information is used to characterize the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin; performing coordinate transformation on the camera posture information to obtain target posture information of a bucket of the loader, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at an initial moment as the origin; and correcting the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to characterize the initial posture of the bucket obtained based on a global positioning map of the loader. It is easy to notice that by installing a camera under the hydraulic cylinder of the front body of the loader and installing an identifiable label in the horizontal middle position of the unloading port retaining wall, when the loader is in the unloading operation scene, the camera posture information is determined by the label image collected by the camera, and then the target posture information of the loader bucket is determined according to the camera posture information, and then the initial posture information of the bucket is corrected by the target posture information to correct the positioning error of the initial posture of the bucket under the global positioning map, so as to ensure the positioning accuracy and unloading accuracy of the loader bucket at the unloading port, thereby achieving the technical effect of reducing the unloading deviation of the loader and thus improving the operating efficiency of the loader, thereby solving the technical problem of large unloading deviation of the loader and reduced operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 Shown is a flow chart of a method for correcting a discharging deviation of a loader provided in one embodiment of the present application.

[0019] Figure 2 The figure is a schematic diagram of the framework of the unloading deviation correction process of the loader operation scenario provided by an embodiment of the present application.

[0020] Figure 3 Shown is a schematic diagram of a unloading deviation correction system for a loader provided in one embodiment of the present application.

[0021] Figure 4Shown is a device block diagram of a loader provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] In the description of the application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically limited. In the embodiment of the present application, all directional indications (such as up, down, left, right, front, back, top, bottom ...) are only used to explain the relative position relationship, motion conditions, etc. between the components under a certain specific posture (as shown in the drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their deformations are intended to cover non-exclusive inclusions. For example, the process, method, system, product or equipment comprising a series of steps or units are not limited to the steps or units listed, but optionally also include the steps or units not listed, or optionally also include other steps or units inherent to these processes, methods, products or equipment.

[0023] In addition, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] Exemplary Methods

[0025] As a first aspect of the present application, the present application provides a method for correcting a discharging deviation of a loader. Figure 1 FIG. 1 is a flow chart of a method for correcting a discharging deviation of a loader provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0026] S101, using a camera to collect an image of the identifiable tag to obtain a tag image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at a horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port;

[0027] Specifically, the above-mentioned identifiable label can be used to represent the label installed in the horizontal middle position of the discharge port retaining wall. Common ones include but are not limited to Tag labels. Tag labels are used as an example in this application.

[0028] Tag generally refers to an automatic identification technology that can help vehicles be identified and tracked in various automated systems.

[0029] Generally, when a loader is performing unloading operations, there are generally multiple unloading ports at the unloading site. By uniquely identifying the corresponding identifiable tag for each unloading port, during the unloading operation, the identifiable tag installed in the horizontal middle position of the unloading port retaining wall can be collected through the camera installed under the hydraulic cylinder of the front body of the loader. By obtaining the identity identification information of the collected identifiable tag, it can be determined whether the current unloading port matches accurately.

[0030] It should be noted that in order to prevent the bucket from lifting and blocking the camera's field of view when the loader is unloading, the camera is deployed in the center position directly below the hydraulic cylinder of the front body of the loader, with the field of view facing forward, and the Tag is deployed in the horizontal middle position of the unloading port retaining wall, and the center point of the Tag is kept consistent with the optical center of the camera in the height direction.

[0031] In addition, the camera is used to collect the identifiable tags installed in the horizontal middle position of the unloading port retaining wall. Since the identifiable tags are installed in the horizontal middle position of the unloading port retaining wall, the coordinates of multiple corner points of the collected identifiable tags are obtained. The positioning accuracy and unloading accuracy of the loader bucket at the unloading port can be determined by the corner point coordinates, thereby avoiding the risk of collision between the bucket and the baffles on both sides of the unloading port.

[0032] The above-mentioned label image is used to represent an image containing a recognizable label.

[0033] The above-mentioned multiple corner points can be used to represent the corner points on the identifiable label. The number of corner points is not specifically limited and can be determined according to the shape of the identifiable label. For example, if the identifiable label is a rectangle, the corresponding number of corner points is 4, etc. The number of multiple corner points can be determined according to actual conditions.

[0034] In addition, before using the camera to capture images of identifiable tags and obtaining label images, generally, when the loader works in a non-unloading area, since there is no need to align the loader bucket and the side baffles of the unloading port, the global positioning map provided by SLAM can be used to meet the positioning and recognition in general scenarios. Therefore, when the loader works in a non-unloading area, the global positioning map provided by SLAM can be used to navigate the loader. If the loader needs to perform unloading operations, the loader can be navigated to the unloading area through the global positioning map provided by SLAM. After the camera recognizes the Tag deployed at the unloading port, the Apriltag positioning function can be enabled to accurately position the posture of the loader bucket in the global coordinate system to ensure that the average distance error between the loader bucket and the baffles on both sides of the unloading port is less than the established error, thereby improving the precise positioning of the loader bucket.

[0035] S102, determining camera pose information according to the tag image, wherein the camera pose information is used to represent the pose of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin;

[0036] Specifically, the above-mentioned camera posture information can be used to represent the posture of the camera in the local coordinate system of the Tag.

[0037] In an optional embodiment, after obtaining the above-mentioned tag image, it is necessary to extract the corner point information of the Tag label from the tag image, and then construct a Tag local coordinate system with the center point of the Tag label as the origin. Based on the constructed Tag local coordinate system, the camera posture information can be determined according to the above-mentioned tag image.

[0038] Specifically, based on the constructed local coordinate system of the tag, the two-dimensional coordinates of the four corner points in the tag in the pixel coordinate system and the three-dimensional coordinates in the world coordinate system can be obtained respectively, and based on the mapping relationship from the pixel coordinate system to the world coordinate system, the homography matrix can be obtained. Among them, the homography matrix is ​​used to describe the projection transformation relationship between two planes and express the position mapping relationship between the world coordinate system and the pixel coordinate system.

[0039] Furthermore, the two-dimensional coordinates of the four corner points in the above-mentioned Tag label in the pixel coordinate system and the three-dimensional coordinates in the world coordinate system are put into the homography matrix, and the rotation matrix and translation vector between the camera coordinate system and the Tag local coordinate system are obtained by the singular value calculation method, and the transformation matrix of the Tag in the camera coordinate system is obtained. By inverting the transformation matrix, the camera posture in the Tag coordinate system is obtained, that is, the above-mentioned camera posture information.

[0040] Among them, the above-mentioned camera coordinate system and pixel coordinate system are coordinate systems for the camera, and are three different coordinate systems involved in the imaging process. The world coordinate system can be considered as a coordinate system established with a fixed point in the real scene as the origin of the coordinate system.

[0041] S103, performing coordinate transformation on the camera posture information to obtain target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin;

[0042] Specifically, after obtaining the camera pose information, the camera coordinate system and the vehicle coordinate system can be associated to obtain the pose of the loader body in the Tag local coordinate system. Using kinematics, the precise pose of the bucket relative to the Tag local coordinate system is derived. The R and t transformation matrices of the Tag local coordinate system and the SLAM global coordinate system are obtained through the correspondence between the coordinates of the Tag corner points in the Tag local coordinate system and the SLAM global coordinate system. The bucket pose information in the Tag local coordinate system is converted to the precise pose in the SLAM global coordinate system using the transformation matrix to obtain the target pose information of the loader bucket mentioned above. Specifically, it can be obtained through the following matrix conversion:

[0043]

[0044] Among them, X w is the coordinate of the tag corner point in the SLAM (Simultaneous Localization and Mapping) global coordinate system, X t It is the coordinate of the corresponding corner point of the Tag in the local coordinate system of the Tag.

[0045] It should be noted that the above global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin, that is, when the loader enters the unloading area and starts working, the laser radar position point on the loader is obtained, and the position point is used as the origin to construct the SLAM global coordinate system. Since the loader bucket needs to be accurately matched with the unloading port baffle during unloading, the loader bucket needs to be continuously adjusted before unloading to ensure accurate alignment of the unloading port. Therefore, the above camera posture information needs to be converted into the target posture information of the loader bucket to achieve dynamic adjustment of the loader bucket.

[0046] S104, correcting the initial posture information of the bucket using the target posture information, wherein the initial posture information is used to represent the initial posture of the bucket obtained based on the global positioning map of the loader.

[0047] Specifically, the above-mentioned initial posture information can be used to represent the initial posture of the bucket obtained based on the global positioning map of the loader in the SLAM global coordinate system.

[0048] Generally, in the mixing station scenario, due to the existence of occlusion, the RTK positioning method cannot be used, so the laser radar SLAM positioning method or the visual SLAM positioning method is mostly used. The laser radar is installed above the cab of the rear body of the loader, and the real-time positioning function is realized by obtaining the current point cloud frame data in real time and comparing it with the map data. However, when the loader is unloading, the baffles on both sides of the unloading port are smaller than the surplus of the loader bucket. Due to the positioning error of SLAM and the accumulation of control errors in the control module, the risk of bucket baffle collision is very likely to occur. Therefore, in this application, it is necessary to correct the initial posture information of the loader bucket based on the SLAM global coordinate system.

[0049] Therefore, in this application, when the camera detects the Tag tag, the target posture information of the loader bucket mentioned above can be obtained, and vision is used as the main source of global positioning information. At the same time, the positioning information of the radar point cloud and IMU (Inertial Measurement Unit) data is used for auxiliary positioning, that is, the initial posture information of the loader bucket in the SLAM global coordinate system is used as auxiliary positioning. The auxiliary positioning is corrected by the main source to compensate for the SLAM positioning error. At the same time, during the unloading process, the loader bucket posture is continuously adjusted according to the position information of the bucket posture and the baffles on both sides and the real-time posture feedback until the bucket can avoid the risk of collision with the baffles on both sides.

[0050] In summary, when the loader works in the non-unloading area, the loader is navigated through the global positioning map provided by SLAM. When the loader works in the unloading area, the Apriltag positioning function is enabled to obtain more precise target posture information of the loader bucket, and then the initial posture information of the bucket under the global coarse positioning is corrected through the target posture information to correct the positioning error of the initial posture of the bucket under the global positioning map, ensure the positioning accuracy and unloading accuracy of the loader bucket at the unloading port, reduce the unloading deviation of the loader under SLAM global positioning, and thus improve the operating efficiency of the loader.

[0051] The present application provides a method for correcting unloading deviations of a loader, comprising: using a camera to capture an image of an identifiable tag to obtain a tag image, wherein the tag image includes multiple corner points of the tag, the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port; determining the camera posture information according to the tag image, wherein the camera posture information is used to characterize the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin; performing coordinate transformation on the camera posture information to obtain target posture information of a bucket of the loader, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at an initial moment as the origin; and correcting the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to characterize the initial posture of the bucket obtained based on the global positioning map of the loader. It is easy to notice that by installing a camera under the hydraulic cylinder of the front body of the loader and installing an identifiable label in the horizontal middle position of the unloading port retaining wall, when the loader is in the unloading operation scene, the camera posture information is determined by the label image collected by the camera, and then the target posture information of the loader bucket is determined according to the camera posture information, and then the initial posture information of the bucket is corrected by the target posture information to correct the positioning error of the initial posture of the bucket under the global positioning map, so as to ensure the positioning accuracy and unloading accuracy of the loader bucket at the unloading port, thereby achieving the technical effect of reducing the unloading deviation of the loader and thus improving the operating efficiency of the loader, thereby solving the technical problem of large unloading deviation of the loader and reduced operating efficiency.

[0052] In a possible implementation of the present application, the camera posture information is coordinate transformed to obtain the target posture information of the loader bucket, including: according to the coordinate association relationship between the camera and the front body of the loader, the camera posture information is coordinate transformed to obtain the posture information of the front body of the loader; according to the posture information of the front body of the loader, the first position information of the loader bucket is determined, wherein the first position information is used to characterize the posture information of the bucket in the local coordinate system; based on a preset transformation matrix, the first position information is matrix transformed to obtain the target posture information, wherein the preset transformation matrix is ​​used to characterize the preset matrix for converting the local coordinate system into the global coordinate system.

[0053] Specifically, the above-mentioned coordinate correlation relationship can be obtained by measuring the structural dimensions of the connection between the camera and the front body of the loader.

[0054] The above-mentioned loader front body posture information can be used to represent the posture of the loader front body in the Tag local coordinate system.

[0055] The above preset conversion matrix can be used to represent the R and t transformation matrices of the pre-obtained Tag local coordinate system and the SLAM global coordinate system, where R is the rotation matrix and t is the translation matrix.

[0056] In an optional embodiment, after obtaining the camera pose information, the pose information of the front body of the loader can be obtained based on the coordinate association between the camera and the front body of the loader. At this time, the pose information of the front body of the loader is the pose in the local coordinate system of the Tag. Then, the kinematics can be used to deduce the precise pose of the bucket of the loader relative to the local coordinate system of the Tag, and the first pose information of the bucket of the loader can be obtained. In order to convert the local coordinate system of the Tag to the global coordinate system of SLAM, the R and t transformation matrices of the local coordinate system of the Tag and the global coordinate system of SLAM can be obtained by the correspondence between the coordinates of the four corner points of the Tag label in the local coordinate system of the Tag and the coordinates in the global coordinate system of SLAM. The R and t transformation matrix is ​​the preset transformation matrix mentioned above. Therefore, the first pose information is transformed by using the preset transformation matrix, that is, the bucket pose information in the local coordinate system of the Tag is converted to the precise pose of the bucket in the global coordinate system of SLAM, and the target pose information can be obtained. By using the target pose information as the main positioning information and correcting the auxiliary positioning, the compensation correction of the SLAM positioning error can be realized.

[0057] In a possible implementation of the present application, the initial posture information of the bucket is corrected through the target posture information, including: determining the adjustment parameters of the bucket according to the target posture information and the initial posture information; adjusting the current operating parameters of the bucket based on the adjustment parameters so that the unloading accuracy of the loader bucket at the unloading port meets the preset unloading accuracy.

[0058] Specifically, the above adjustment parameters include but are not limited to adjusting parameters such as the lifting height, tilt angle, and flip angle of the bucket.

[0059] The above-mentioned preset unloading accuracy can be used to indicate the preset unloading accuracy of the loader bucket at the unloading port, that is, the allowable alignment error between the loader bucket and the baffles on both sides of the unloading port. Generally, it can be 2cm or 3cm. There is no specific restriction on the preset unloading accuracy here, and it can be adjusted according to actual conditions.

[0060] In an optional embodiment, after obtaining the target posture information and the initial posture information, the adjustment parameters of the bucket can be determined based on the positioning error of the initial posture information relative to the target posture information, and the current operating parameters of the bucket can be adjusted by adjusting the parameters so that the unloading accuracy of the loader bucket at the unloading port meets the preset unloading accuracy.

[0061] Specifically, during the process of correcting the bucket posture, the positioning information of the loader bucket is fed back to the decision-making control module in real time, and the decision-making control module outputs the above-mentioned adjustment parameters to realize closed-loop control, thereby realizing the precise unloading process of the loader.

[0062] In a possible implementation of the present application, before correcting the initial posture information of the bucket through the target posture information, the method includes: obtaining the lidar point cloud data and sensor data of the loader; using the lidar point cloud data and sensor data to build a scene map to obtain a global positioning map; and obtaining the initial posture information of the bucket based on the global positioning map.

[0063] Specifically, the above-mentioned laser radar point cloud data is obtained through detection by a laser radar installed above the cab of the rear body of the loader.

[0064] The above-mentioned sensing data is obtained by detecting the sensors installed on the loader body.

[0065] In an optional embodiment, after obtaining the laser radar point cloud data and sensor data of the loader, the SLAM algorithm uses the sensor data and the laser radar point cloud data to complete the scene mapping, realize the global positioning of the loader, and obtain the above-mentioned global positioning map, and then obtain the initial posture information of the bucket in the SLAM global coordinate system based on the global positioning map. Due to the accumulation of positioning errors in SLAM and control errors in the control module, the loader bucket is not directly controlled by the initial posture information of the bucket in this application. Instead, the initial posture information of the bucket in the SLAM global coordinate system is used as auxiliary positioning, and the target posture information is used as the main positioning information to correct the auxiliary positioning.

[0066] In a possible implementation of the present application, the label image contains multiple corner points of the label, and the camera pose information is determined based on the label image, including: obtaining the two-dimensional coordinates of the multiple corner points in the pixel coordinate system of the camera; obtaining the three-dimensional coordinates of the multiple corner points in the local coordinate system; determining the homography matrix by matching the two-dimensional coordinates and the three-dimensional coordinates, wherein the homography matrix is ​​used to represent the transformation matrix between the identifiable label and the pixel coordinate system; determining the transformation matrix of the identifiable label in the camera coordinate system based on the two-dimensional coordinates, the three-dimensional coordinates and the homography matrix, wherein the transformation matrix is ​​used to represent the matrix of the transformation between the camera coordinate system and the local coordinate system, and the transformation matrix includes a rotation matrix and a translation matrix; inverting the transformation matrix to obtain the camera pose information in the local coordinate system.

[0067] Specifically, the homography matrix is ​​used to describe the projection transformation relationship between two planes, and to express the position mapping relationship between the world coordinate system and the pixel coordinate system. Applied in the embodiment of the present application, after matching the coordinate points of the two-dimensional coordinates and the three-dimensional coordinates, the homography transformation matrix between the tag label and the pixel coordinate system can be obtained by the PNP (Perspective-n-Point) algorithm. The homography matrix contains both the intrinsic and extrinsic parameters of the camera.

[0068] Then, using the above homography matrix, the coordinates of the corresponding reference points in the world coordinate system and the pixel coordinate system are substituted, and the rotation matrix and translation matrix between the camera coordinate system and the tag local coordinate system are obtained by the singular value score calculation method, and the transformation matrix of the tag in the camera coordinate system is obtained. By inverting the transformation matrix, the position and posture of the camera in the tag coordinate system is obtained.

[0069] In a possible implementation of the present application, after the image of the identifiable tag is captured by a camera and the tag image is obtained, the method further includes: performing dedistortion processing on the tag image.

[0070] Specifically, since the acquired label image is a distorted data frame, in order to ensure positioning accuracy, the distorted data needs to be corrected.

[0071] Generally, image dedistortion is a technique in image processing that is used to correct image distortion caused by the characteristics of camera lenses. This distortion usually manifests itself as barrel distortion or pincushion distortion. The following are some common image dedistortion processing methods: camera calibration, radial distortion and tangential distortion models, perspective transformation, and machine learning methods.

[0072] In a possible implementation of the present application, before dedistorting the label image, the method further includes: calibrating the camera's internal parameters to obtain an internal parameter matrix; and correcting the label image based on the internal parameter matrix.

[0073] Specifically, the intrinsic parameter matrix mentioned above can be used to represent the intrinsic characteristic matrix of a pre-calibrated camera. It includes parameters such as focal length, principal point (image center), and radial distortion coefficient. The intrinsic parameter matrix is ​​usually used to convert three-dimensional world coordinates into two-dimensional image coordinates in the camera coordinate system. The intrinsic parameter matrix is ​​calculated during the camera calibration process and can be determined by photographing a calibration plate with known geometric features (such as a chessboard) and using a computer vision algorithm.

[0074] In an optional embodiment, the present application uses the intrinsic parameter matrix of the camera obtained by camera calibration to correct the camera data frame.

[0075] In a possible implementation of the present application, after the label image is dedistorted, the method further includes: performing threshold segmentation on the standard label image obtained by the dedistortion process to obtain multiple segmented images; performing connected domain search on the multiple segmented images to obtain identifiable labels in the label image, wherein the identifiable labels include multiple corner points of the label.

[0076] Specifically, after the label image is dedistorted, the data frame in the standard label image obtained by the dedistortion processing can be used to identify the Tag label in the image through threshold segmentation, connected domain search, encoding, and decoding, and the three-dimensional coordinates of the four corner points with the center point of the Tag label as the origin of the coordinate system and the corresponding reference point coordinates in the pixel coordinate system are matched one by one.

[0077] Among them, threshold segmentation is a simple image segmentation technique that compares the pixel value of an image with a preset threshold to generate a binary image (that is, the image contains only two colors, usually black and white). For example, if the pixel value is greater than the threshold, it is set to white (or 1), otherwise it is set to black (or 0).

[0078] A connected domain is a set of pixels with the same pixel value and adjacent to each other in a binary image. The purpose of connected domain search is to identify and mark these connected domains, which are usually used for object counting, shape analysis, and feature extraction. This process usually involves traversing every pixel in the image and grouping adjacent pixels with the same pixel value into the same connected domain.

[0079] Figure 2 The figure is a schematic diagram of the framework of the unloading deviation correction process of the loader operation scenario provided by an embodiment of the present application.

[0080] like Figure 2 As shown: The overall process framework includes: data input layer, data processing layer, output layer and downstream nodes. Among them, the data input layer includes: IMU data, lidar point cloud data and tag image data. The data processing layer includes: SLAM algorithm and April Tag local positioning information, among which April Tag local positioning information includes camera calibration, tag corner point extraction and tag pose calculation. The output layer includes: loader pose information in the global coordinate system, front vehicle pose information in the local coordinate system and loader pose information in the global coordinate system. The downstream nodes include: decision control module, which is used to feed back the positioning information of the loader bucket to the decision control module in real time to adjust the bucket pose.

[0081] AprilTag is a visual positioning system that uses specific binary tags to achieve rapid object recognition and accurate 3D spatial positioning. AprilTag's positioning method is based on feature points extracted from the tag image, and computer vision algorithms are used to determine the position and orientation of the tag relative to the camera.

[0082] The positioning process of AprilTag mainly includes the following steps: Tag detection: First, the presence of AprilTag is detected in the image, which is usually done by identifying the specific pattern and structure of the tag. Corner point extraction: After the tag is detected, the algorithm will extract the position information of the four corner points of the tag. Homography transformation: Using the corner point information, calculate the homography transformation matrix from the tag coordinate system to the image coordinate system. This matrix reflects the projection relationship from the point on the tag plane to the image plane. Pose estimation: Combined with the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.), the homography transformation matrix is ​​used to estimate the position and direction of the camera relative to the tag, that is, the camera's external parameters. Three-dimensional positioning: Through the above steps, the position of the tag in space and the direction of the camera can be determined to achieve three-dimensional positioning.

[0083] The specific unloading deviation correction process includes: when the loader drives to the unloading area, the positioning information provided by SLAM is first used to roughly position the loader in the SLAM global coordinate system, that is, the SLAM algorithm uses IMU data and lidar point cloud data to complete scene mapping and realize the global positioning of the loader, that is, the loader posture information in the global coordinate system is obtained; at the same time, the camera captures the image data of the Tag label, uses camera calibration to remove image distortion, and obtains the rotation matrix R and translation matrix T through Tag corner point extraction and Tag posture calculation methods, thereby obtaining the posture information of the camera coordinate system in the Tag local coordinate system. Then, through the coordinate transformation relationship between the camera coordinate system and the vehicle body coordinate system, the positioning information of the front body of the loader in the Tag local coordinate system is obtained, and then the high-precision positioning information obtained by the AprilTag positioning method is converted to the SLAM global coordinate system according to the transformation matrix between the Tag local coordinate system and the SLAM global coordinate system, the positioning error of SLAM is corrected, and the positioning accuracy is improved.

[0084] Exemplary Systems

[0085] As a second aspect of the present application, the present application also provides a unloading deviation correction system for a loader. Figure 3 FIG. 1 is a schematic diagram of a loader unloading deviation correction system provided by an embodiment of the present application. Figure 3 As shown, the system includes:

[0086] An image acquisition module 301 is used to acquire an image of the identifiable tag using a camera to obtain a tag image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port;

[0087] A posture calculation module 302 is used to determine the posture information of the camera according to the tag image, wherein the posture information of the camera is used to represent the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable tag as the origin;

[0088] A coordinate transformation module 303 is used to transform the camera posture information to obtain the target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in the global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin;

[0089] The posture correction module 304 is used to correct the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to represent the initial posture of the bucket obtained based on the global positioning map of the loader.

[0090] In a possible implementation of the present application, a coordinate transformation module includes: a loader front body posture information acquisition module, which is used to perform coordinate transformation on the camera posture information according to the coordinate association relationship between the camera and the loader front body, so as to obtain the loader front body posture information; a first posture information determination module, which is used to determine the first posture information of the loader bucket according to the loader front body posture information, wherein the first posture information is used to characterize the posture information of the bucket in the local coordinate system; a target posture information acquisition module, which is used to perform matrix transformation on the first posture information based on a preset transformation matrix, so as to obtain target posture information, wherein the preset transformation matrix is ​​used to characterize a preset matrix for converting the local coordinate system into the global coordinate system.

[0091] In a possible implementation of the present application, the posture correction module includes: an adjustment parameter determination module, which is used to determine the adjustment parameters of the bucket according to the target posture information and the initial posture information; an operating parameter adjustment module, which is used to adjust the current operating parameters of the bucket based on the adjustment parameters so that the unloading accuracy of the loader bucket at the unloading port meets the preset unloading accuracy.

[0092] In a possible implementation of the present application, the system also includes: a data acquisition module, used to acquire the lidar point cloud data and sensor data of the loader; a scene mapping module, used to use the lidar point cloud data and sensor data to perform scene mapping to obtain a global positioning map; an initial posture information acquisition module, used to acquire the initial posture information of the bucket according to the global positioning map.

[0093] In a possible implementation of the present application, a posture calculation module includes: a two-dimensional coordinate acquisition module, which is used to obtain the two-dimensional coordinates of multiple corner points in the pixel coordinate system of the camera; a three-dimensional coordinate acquisition module, which is used to obtain the three-dimensional coordinates of multiple corner points in the local coordinate system; a homography matrix determination module, which is used to determine the homography matrix by matching the two-dimensional coordinates and the three-dimensional coordinates, wherein the homography matrix is ​​used to represent the transformation matrix between the identifiable label and the pixel coordinate system; a transformation matrix determination module, which is used to determine the transformation matrix of the identifiable label in the camera coordinate system according to the two-dimensional coordinates, the three-dimensional coordinates and the homography matrix, wherein the transformation matrix is ​​used to represent the matrix of the transformation between the camera coordinate system and the local coordinate system, and the transformation matrix includes a rotation matrix and a translation matrix; a matrix inversion module, which is used to invert the transformation matrix to obtain the camera posture information in the local coordinate system.

[0094] In a possible implementation of the present application, the system further includes: a processing module for performing dedistortion processing on the label image

[0095] In a possible implementation of the present application, the system further includes: an intrinsic parameter matrix acquisition module, which is used to perform intrinsic parameter calibration on the camera to obtain an intrinsic parameter matrix; and an image correction module, which is used to correct the label image based on the intrinsic parameter matrix.

[0096] In a possible implementation of the present application, the system also includes: a segmentation module, which is used to perform threshold segmentation on the standard label image obtained by dedistortion processing to obtain multiple segmented images; a search module, which is used to perform connected domain search on the multiple segmented images to obtain identifiable labels in the label image, wherein the identifiable labels include multiple corner points of the label.

[0097] Example Loader

[0098] As a third aspect of the present application, the present application further provides a loader, including: a unloading deviation correction system of the loader as described above.

[0099] Figure 4 Shown is a device block diagram of a loader provided in one embodiment of the present application.

[0100] like Figure 4 As shown, it includes one or more processors 41 and a memory 42.

[0101] The processor 41 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the loader to perform desired functions.

[0102] The memory 42 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and a processor may run the program instructions to implement the methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0103] In one example, the loader may further include: an input device 43 and an output device 44, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0104] When the loader is a stand-alone device, the input device 43 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0105] In addition, the input device 43 may also include, for example, a keyboard, a mouse, etc.

[0106] The output device 44 can output various information to the outside, including the determined distance information, direction information, etc. The output device 44 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0107] Of course, to simplify, Figure 4 Only some of the components in the loader related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the loader may further include any other appropriate components according to specific application conditions.

[0108] Exemplary computer-readable storage media

[0109] As a fourth aspect of the present application, the present application provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being used to perform the following steps:

[0110] S1, using a camera to collect an image of the identifiable tag to obtain a tag image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port;

[0111] S2, determining the camera pose information according to the label image, wherein the camera pose information is used to represent the pose of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable label as the origin;

[0112] S3, coordinate transformation is performed on the camera posture information to obtain target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in the global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin;

[0113] S4, correcting the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to represent the initial posture of the bucket obtained based on the global positioning map of the loader.

[0114] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program information, which, when executed by a processor, enables the processor to execute the steps of the method for correcting the unloading deviation of a loader according to various embodiments of the present application described in this specification.

[0115] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0116] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program information is stored. When the computer program information is executed by a processor, the processor executes the steps in the method for correcting the unloading deviation of a loader according to various embodiments of the present application described in this specification.

[0117] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0118] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0119] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0120] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

Claims

1. A method for correcting unloading deviation of a loader, characterized in that: include: The identifiable tag is imaged by a camera to obtain a tag image, wherein the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port; Determine the camera pose information according to the label image, wherein the camera pose information is used to represent the pose of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable label as the origin; Performing coordinate transformation on the camera posture information to obtain target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin; The initial posture information of the bucket is corrected by using the target posture information, wherein the initial posture information is used to represent the initial posture of the bucket obtained based on the global positioning map of the loader.

2. The method for correcting the unloading deviation of a loader according to claim 1, characterized in that: The camera posture information is subjected to coordinate transformation to obtain target posture information of the loader bucket, including: According to the coordinate association relationship between the camera and the front body of the loader, coordinate transformation is performed on the camera position information to obtain the front body position information of the loader; Determine the first posture information of the bucket of the loader according to the posture information of the front body of the loader, wherein the first posture information is used to represent the posture information of the bucket in the local coordinate system; Based on a preset conversion matrix, a matrix conversion is performed on the first posture information to obtain the target posture information, wherein the preset conversion matrix is ​​used to represent a preset matrix for converting a local coordinate system into a global coordinate system.

3. The method for correcting the unloading deviation of a loader according to claim 1, characterized in that: The initial posture information of the bucket is corrected by using the target posture information, including: Determining adjustment parameters for the bucket according to the target posture information and the initial posture information; The current operating parameters of the bucket are adjusted based on the adjustment parameters so that the unloading accuracy of the loader bucket at the unloading port meets the preset unloading accuracy.

4. The method for correcting the unloading deviation of a loader according to claim 1, characterized in that: Before correcting the initial posture information of the bucket by using the target posture information, the method includes: Obtain lidar point cloud data and sensor data of the loader; Using the laser radar point cloud data and the sensor data to construct a scene map to obtain the global positioning map; According to the global positioning map, the initial position information of the bucket is obtained.

5. The method for correcting the unloading deviation of a loader according to claim 1, characterized in that: The label image contains multiple corner points of the label, and the camera position information is determined according to the label image, including: Obtaining the two-dimensional coordinates of the multiple corner points in the pixel coordinate system of the camera; Obtaining the three-dimensional coordinates of the multiple corner points in the local coordinate system; Determine a homography matrix by matching the two-dimensional coordinates with the three-dimensional coordinates, wherein the homography matrix is ​​used to represent a transformation matrix between the identifiable tag and the pixel coordinate system; Determine a transformation matrix of the identifiable tag in a camera coordinate system according to the two-dimensional coordinates, the three-dimensional coordinates, and the homography matrix, wherein the transformation matrix is ​​used to represent a matrix for transforming between the camera coordinate system and the local coordinate system, and the transformation matrix includes a rotation matrix and a translation matrix; The transformation matrix is ​​inverted to obtain the camera pose information in the local coordinate system.

6. The method for correcting the unloading deviation of a loader according to claim 1, characterized in that: After acquiring the image of the identifiable label by using a camera to obtain the label image, the method further includes: performing a dedistortion process on the label image.

7. The method for correcting the unloading deviation of a loader according to claim 6, characterized in that: Before performing dedistortion processing on the label image, the method further includes: Performing internal parameter calibration on the camera to obtain an internal parameter matrix; The label image is corrected based on the intrinsic parameter matrix.

8. The method for correcting the unloading deviation of a loader according to claim 6, characterized in that: After performing dedistortion processing on the label image, the method further includes: Performing threshold segmentation on the standard label image obtained by the dedistortion process to obtain multiple segmented images; Connected domain search is performed on the multiple segmented images to obtain identifiable labels in the label image, wherein the identifiable labels include multiple corner points of the labels.

9. A loader unloading deviation correction system, characterized in that: include: An image acquisition module, used to acquire an image of the identifiable tag using a camera to obtain a tag image, wherein the tag image includes multiple corner points of the tag, the camera is installed below the hydraulic cylinder of the front body of the loader, the identifiable tag is installed at the horizontal middle position of the unloading port retaining wall, and the identifiable tag is used to uniquely identify the corresponding unloading port; A posture calculation module, used to determine the camera posture information according to the label image, wherein the camera posture information is used to represent the posture of the camera in a local coordinate system, and the local coordinate system is constructed with the center point of the identifiable label as the origin; A coordinate transformation module, used for performing coordinate transformation on the camera posture information to obtain target posture information of the loader bucket, wherein the target posture information is used to characterize the posture of the bucket in a global coordinate system, and the global coordinate system is constructed with the laser radar position point on the loader at the initial moment as the origin; The posture correction module is used to correct the initial posture information of the bucket through the target posture information, wherein the initial posture information is used to represent the initial posture of the bucket obtained based on the global positioning map of the loader.

10. A loader, comprising the loader unloading deviation correction system according to claim 9.

Images