Vision system and method for determining the position of a tool attached to a machine

Abstract

The present application relates to vision systems and methods for determining the position of a tool attached to a machine. In particular, a vision-based sensor system for heavy equipment or other machinery is provided that determines the pose or position of a blade relative to a heavy equipment vehicle, wherein the sensor system includes a vision-based blade position system having an image acquisition device and a vision system processor.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Utility Patent Application No. 16 / 532,331, filed August 5, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates to determining the position of a working tool having a working edge, and more specifically, to a method and apparatus for determining the posture or position of a blade relative to a machine using a sensor system. Background Technology

[0004] Heavy construction machinery (e.g., bulldozers, graders, tractors, pavers, front-end loaders, backhoe excavators, etc.) typically has one or more working edges (e.g., blades) for performing a specific function. As used herein, the term working edge generally refers to any part of heavy equipment that performs a specific function requiring precise positioning; for example, a working edge may include the blade of a bulldozer or grader or the shovel of a backhoe excavator, to name a few.

[0005] Users of heavy equipment utilize work edges to perform a range of typical site functions (e.g., using bulldozer blades to level a section of the site to a desired slope and width). This work is continuously checked during the operation of the heavy equipment to ensure compliance with applicable site requirements or specifications. This often involves an iterative process until the desired slope and width are achieved, as verified according to specifications.

[0006] Therefore, equipment manufacturers typically equip their heavy equipment with some type of automated work edge positioning system to assist users and improve productivity by precisely guiding the heavy equipment across a work site. A typical work edge control system uses one or more sensors fixed to the work edge to monitor the position and / or orientation relative to a fixed point on the heavy equipment vehicle. For example, work edge position information is transmitted via one or more cables to a control system located within the vehicle's cab. The position of the heavy equipment vehicle can be provided by a variety of methods, including but not limited to LPS (Local Positioning System), SLAM (Simultaneous Localization and Mapping) and LIDAR (Light Detection and Ranging) systems based on vision or light detection, or known mission-centric positioning methods. For example, the heavy equipment vehicle may have a Global Navigation Satellite System (GNSS) receiver that provides the machine's precise position in a known manner. The work edge control system utilizes this position information, combined with the transmitted blade position, to precisely set the slope and / or level the work site.

[0007] One drawback of this sensor-based working edge control system is that the sensors are highly sensitive components mounted on the working edge and exposed to the risk of damage associated with harsh working environments. For example, the large mechanical stresses applied to heavy equipment during operation, as well as impacts from dirt or other materials, can cause these components to malfunction, become misaligned, and / or require recalibration.

[0008] Therefore, there is a need for improved sensor technology for accurately determining the attitude and / or position of blades relative to the machine during routine heavy equipment operation. Summary of the Invention

[0009] According to various embodiments, a vision-based sensor system for heavy equipment is provided, which determines the attitude or position of blades relative to the heavy equipment vehicle.

[0010] According to one implementation, an image acquisition device (e.g., a single camera) is configured as a single point of view (POI) to determine the blade position using an azimuth projection that maps a specific point of interest on or around the blade (e.g., the upper corner of the blade) in the image plane to a specific rotation in the machine reference frame. In this way, a vision-based sensor system is employed for image analysis of one or more acquired and tracked points of interest (POIs), such as at the blade corner. The working edge position information is obtained from images captured by the image acquisition device, and the position information is calculated in a specific coordinate system of the vision system. The working edge position information can be transformed into a GNSS coordinate system (or other methods for confirming the position and orientation of associated heavy equipment, such as LiDAR), allowing the information to be used to control the blades of the associated heavy equipment (e.g., a bulldozer).

[0011] According to an embodiment, a vision-based sensor system is configured to accurately determine the position of a blade relative to heavy equipment. Illustratively, an image acquisition device (e.g., a single camera) is mounted on the heavy equipment and has a fixed field of view with respect to a spatial region of the blade (e.g., the upper edge of a blade with a corresponding upper corner). Illustratively, the vision system identifies the upper edge of the blade (i.e., the first and second corners defining two corresponding points of interest), wherein such corners fall on a spherical surface including the blade (i.e., a spherical surface of interest, mathematically defined by the vision system), and wherein movement of each corner is defined to a so-called corresponding blade portion on the spherical surface of interest. After identifying these first and second corners of the blade in an image acquired by the image acquisition device, azimuth projection is then used to map the sphere onto an image plane in front of the image acquisition device. That is, according to an embodiment, an appropriate radius (R) of the spherical surface of interest is used.E The azimuth projection bijectively maps the position of the corner on the sphere to its position on the plane. In this way, the corresponding corner is identified in the acquired image and the position of the corresponding corner is calculated in the vision system coordinate system, so that perspective projection is used for a single POV (and thus eliminates the need for a second camera). According to the embodiment, the position of the corresponding blade corner (in the corresponding first and second blade segments) and therefore the position of the working edge are determined based on the X, Y, Z position and / or rotation based on the coordinate system centered on the vision system.

[0012] According to one implementation, calibration is performed during installation to determine the position and length of specific features of the associated heavy equipment (e.g., a bulldozer), the position and length of which are required to calculate the blade position through a so-called transformation projection. Calibration typically involves a combination of measurements between specific accessible features on the associated heavy equipment and their corresponding positions in camera images. These measurements allow the calculation of the required position and length of the specific features on the associated heavy equipment.

[0013] According to the implementation method, calibration is performed to obtain a unique transformation from the vision system to the machine reference frame. In the presence of GNSS, a second transformation can be applied to generate the position of the working edge of the blade according to the coordinate system of the GNSS system, and then the coordinate system of the GNSS system can be used to control the blade and / or heavy equipment to perform specific functions on the site (e.g., precisely setting the slope).

[0014] According to one embodiment, a vision system for determining the position of a tool attached to a machine includes an image acquisition device for acquiring an image of the tool. The image acquisition device is positioned such that: (1) at least two points of interest located on the tool at a distal end are observed such that the points of interest are visible in the image over the entire range of motion of the tool; and (2) any combination of motions of the tool's common origin will cause the points of interest to be mapped onto the image plane of the image acquisition device via azimuth projection. In one example, the points of interest may be associated with a corresponding target fixed to the tool. The vision system also includes a processor and memory for calculating the corresponding position of the points of interest in the reference frame of the vision system by using the corresponding position of the points of interest in the image and the known geometry of the glued joint of the tool with respect to the machine hinge, transforming the projection of the points of interest in the image.

[0015] In one illustrative embodiment, the vision system is calibrated by collecting multiple images and distances associated with at least two points of interest for multiple positions of the tool, each position representing the tool's motion about a corresponding center of rotation for each of the multiple positions. The corresponding position of each of the respective centers of rotation in the vision system's reference frame can then be calculated. In one example, the machine is a bulldozer and the tool is a blade, with the point of interest located at a corresponding upper distal corner of the blade. The spherical joint of the blade can have a limited range of rotational motion about the center of the spherical joint, such that the spherical joint is supported by a C-frame and is located in an arcuate portion of a fixed radius about a fixed axis of rotation on the bulldozer's C-frame.

[0016] These and other advantages of the present invention will become apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0017] Figure 1 An exemplary heavy equipment vehicle constructed according to an embodiment is shown;

[0018] Figure 2 An exemplary sensor system for determining the attitude or position of a blade relative to a machine, according to an embodiment, is shown.

[0019] Figure 3 Various illustrative azimuth projections using a single camera are shown according to embodiments;

[0020] Figure 4A and Figure 4B The method for determining according to the embodiment is shown. Figure 1 An example of blade motion in heavy equipment vehicles;

[0021] Figure 5A and Figure 5B The mapping of the sphere to the embodiment is shown. Figure 4A / Figure 4B The illustrative orientation projection of the picture frame determined by the movement of the blades;

[0022] Figure 6 The embodiment of the present invention is shown. Figure 5A / Figure 5B An illustrative mapping of the spherical sheet portion in the image to the image;

[0023] Figure 7 It is shown that, according to the implementation method and according to Figure 5A / Figure 5B and Figure 6 Illustrative leaf images of the mapping;

[0024] Figure 8 A flowchart illustrating the operation of a sensor system for the attitude and position of a blade relative to a machine, according to an embodiment, is shown; and

[0025] Figure 9 A high-level block diagram of an exemplary computer according to an embodiment is shown. Detailed Implementation

[0026] According to various embodiments, a vision-based sensor system for heavy equipment is provided, which determines the attitude or position of blades relative to the heavy equipment vehicle.

[0027] Figure 1 An exemplary heavy equipment vehicle constructed according to an embodiment is shown. More specifically, a bulldozer 100 (also referred to herein as a scraper 100) exemplarily includes a working edge element (i.e., blade 110) and an image acquisition device 120 (e.g., a single camera) mounted on the top of the bulldozer 100's cab to obtain one or more images of one or more points of interest (POIs) 140 associated with the blade (e.g., the blade tip edge 130). It should be noted that, in Figure 1 In this design, the image acquisition device 120 is mounted on top of the bulldozer 100, but various other suitable mounting locations exist for the image acquisition device 120. For example, alternative mounting locations could be on the front grille of the bulldozer 100 or any other location sufficiently close to the blade 110 via a visual feature line (discussed further below). Furthermore, the pivot position 190 (e.g., spherical joint) of the C-frame 180 for the bulldozer 100 is shown in the vision system reference frame and will be discussed further below with respect to calibration operations. The one or more POIs 140 can typically be located at the upper corner of the blade edge 130 of the blade 110 or adjacent to the upper corner of the blade edge 130 of the blade 110, rather than necessarily (in all cases) at the corner itself. For example, the POI 140 can be located to some extent inside and below the top of the blade 110. Antenna 150 is used to enable bulldozer 100 to send and receive communication signals in various known ways, and positioning system 160 is used to provide position and forward information in a conventional manner. As shown, according to an embodiment, image acquisition device 120 has a fixed field of view 170 with respect to the spatial region of blade 110 (and the top edge of the blade, such as blade top edge 130). It should be noted that although the description herein relates to bulldozer 100 with blade 110, the principles of the embodiment are equally applicable to any type of heavy equipment or other machinery employing working edges. It should also be noted that... Figure 1 The proportions shown are not proportional and are representational in nature.

[0028] Figure 2 An exemplary sensor system for determining the posture or position of a blade relative to a machine, according to an embodiment, is shown. Specifically, the sensor system includes a vision-based blade position system 200 having a camera 120 and a vision system processor 245. Illustratively, a positioning system 160 provides position and forward information 255 associated with the bulldozer 100 to a machine control system 220 in a conventional manner. The machine control system 220 also receives blade posture information 260 relative to the position of blade 110, such as that processed by the vision system processor 245 in the vision-based blade position system 200, thereby facilitating the transmission of hydraulic system commands 265 to heavy equipment (e.g., the bulldozer 100), the details of which will now be discussed further.

[0029] As noted above, the embodiments described herein employ image analysis and use a single camera (e.g., camera 120) to acquire and track one or more points of interest (POIs) on or around a blade (e.g., blade 110), without requiring fixation to a specific target on the blade. Illustratively, as noted above, one or more POIs according to the embodiments represent fixed locations located on or near the edge of the blade (e.g., the upper corner of the blade edge). Visual characteristics of the blade features, such as linear features (e.g., edges, line elements, etc.), parallel features, fixed angles (e.g., right angles), contours, textures, and colors, can be employed (alone or in any combination) to enhance the transmission of vision-based blade localization in the disclosed embodiments. These features can be grouped into three basic criteria defining a particular POI marker: appearance (e.g., brightness, color, texture, and variations thereof), geometry (e.g., size, shape, scale, and orientation), and location (e.g., left vs. right, known last location, and trajectory). For example, POIs can be obtained by evaluating visual features using various target recognition operations.

[0030] The vision-based blade positioning system according to the disclosed embodiments allows for the capture of blade motion in multiple degrees of freedom, such as rotation, tilt, lift, yaw, and displacement (i.e., degrees of freedom are any chance of translation or rotation within a reference frame), and the vision-based blade positioning system utilizes a specific azimuth projection associated with the blade motion (as will be further disclosed below). It will be understood that the rotation, tilt, and yaw of the blade generally share a common center of rotation, such that characteristic motions on the blade equidistant from the center of rotation (e.g., the corresponding upper corner of the blade, i.e., the corner of the blade furthest from the tilted ground) lie on a spherical (mathematically speaking) surface. Therefore, according to the embodiments herein, an azimuth projection bijectively maps the position of interest on the sphere to a position on a plane that can be used to position the blade, provided the projection point corresponds to the aperture of the camera in the vision system. The mapping between the position of the blade feature (e.g., the corner) and its position in the image plane is unique if the projection point corresponds to the aperture of the camera in the vision system. The transformation of this projection (e.g., calculating the position of the target features of the blade in the reference frame of the vision system) requires knowing the distances between these features and the corresponding distances of these features from the center of rotation (calculated during calibration).

[0031] For example, Figure 3 Various forms of azimuth projection 300 employing a single camera are illustrated; the camera can be useful in the embodiments described herein and will be discussed further below. Additional calibration data is required to include lifting capabilities (e.g., the hinge between the C-frame and the spherical joint of a bulldozer), because the rotational center controlling the lifting capability (e.g., the rotational axis of the C-frame of the bulldozer) is necessarily far from the rotational center of the previously discussed degrees of freedom (e.g., the spherical joint of the bulldozer). In this case, knowing the distance between the two rotational centers discussed above (e.g., the radius of motion of the spherical joint about the C-frame axis) is sufficient. Typically, an increase in the degrees of freedom for positioning the blades will require corresponding geometric constraints (e.g., the radius between the geometrical position of the center of the hinge chain and any fixed center).

[0032] Figure 4A and Figure 4B The method for determining according to the embodiment is shown. Figure 1 An example of blade motion in heavy equipment vehicles. (As will be discussed together.) Figure 4A and Figure 4BAs shown, a bulldozer 100 with blades 110 moves 470 around a work site 480. According to an embodiment, an image acquisition device 120 (e.g., a single camera) mounted on top of the bulldozer 100's cab will be used to acquire one or more images of one or more points of interest (POIs) associated with a blade (e.g., along a first corner 410 and a second corner 420 along the upper edge of the blade). Illustratively, as viewed from the image acquisition device 120, the first corner 410 is the upper left corner of blade 110, and the second corner 420 is the upper right corner of blade 110. As detailed above, these upper corners of blade 110 will always fall (in a mathematical sense) on the surface of a sphere 430 (i.e., a spherical surface of interest), centered at the intersection 460 of the movement axes of blade 110. According to an embodiment, the movement of each corner is restricted to a so-called patch. Illustratively, the first corner portion 410 is confined to the first piece portion 440 and the second corner portion 420 is confined to the second piece portion 450 (e.g., Figure 4B (as shown in the image).

[0033] According to this embodiment, an image acquisition device 120 is used to obtain position information of the first corner 410 and the second corner 420, and thus the position of the working edge (i.e., blade 110) where the first corner 410 and the second corner 420 are located. As the blade 110 moves through the fixed field of view 170 of the image acquisition device 120, one or more images of each of the first corner 410 and the first corner 420 are obtained. According to the principles of the embodiments herein, all tracking and vision-based calculations are performed using the positions of visually recognizable features on the blade. These features may or may not be upper corners. If the feature used for positioning is not an upper corner, the position of the upper corner is not calculated. Furthermore, the positions of the two (2) features must be distal and are generally positioned close to the upper corner. If the features are symmetrically positioned on the blade, the features will fall on the same sphere, but symmetrical positioning of features on the blade is not a necessary condition. If the radial distances between the features and the spherical joints are different, the features will each be placed on concentric spheres with corresponding radii.

[0034] As previously noted, azimuth projection is used to map sphere 430 onto the picture frame. Figure 5A and 5B The embodiment of the present invention is shown. Figure 4A and Figure 4BAn illustrative orientation projection of the sphere determined by the blade motion is shown on the picture frame. As noted above, orientation projection is used in the vision-based sensor technology described herein to map the sphere (i.e., the surface of the sphere of interest, on which the motion of the aforementioned blade corners is respectively located) onto the picture frame, and will be discussed to facilitate understanding of the disclosed embodiments. Figure 5A A specific illustrative azimuth projection with tangent 560 is shown. The type of azimuth projection corresponding to the use of a single camera according to the embodiment is called the "far-side conventional perspective" type, and this type of azimuth projection has the property of visualizing a portion of the interior of a spherical surface from a point close to but outside the corresponding sphere. For example... Figure 5A As shown, the far-side projection (i.e., R) E <-2) Mapping the hemisphere through an actual lens. As shown, the POV (as defined by the image acquisition device 120) must be close enough to make both sections (e.g., the first section 440 and the second section 450) visible from inside the bulldozer 100. In this way, as Figure 5B As shown, sphere 430 is cut from the tangential cone surface 510 of POV (in Figure 5B As can also be seen, for example, the tangent portion 520 along the tangent line 560 makes R in this illustrative example E It is approximately -2.4.

[0035] return Figure 2 Images from the image acquisition device 120 (illustratively shown as a camera) are processed by the vision system processor 245 to determine the positions of the first corner 410 and the second corner 420 in the vision system coordinate system of the vision-based blade position system 200. In this way, the vision-based blade position system 200 identifies the position of the working edge (i.e., the blade 110) in the vision system coordinate system by illustratively using a predetermined spatial relationship established with respect to the first corner 410 and the second corner 420.

[0036] Illustratively, the fixed field of view 170 includes the spatial region through which the first corner 410 and the second corner 420 move during the operation of the blade 110. The image acquisition device 120 is calibrated (e.g., during installation) to establish a scale for the acquired images, which (in terms of image pixels) defines the relationship between the first corner 410 and the second corner 420. This helps to establish a baseline regarding the scale and position of the respective corners within the field of view. Illustratively, the original position of the blade 110 is obtained (and consequently, the original positions of the first corner 410 and the second corner 420 at the blade edge). It is advantageous to establish a baseline identifying the position of the first corner 410 and / or the second corner 420 as the first corner 410 and / or the first corner 420 move through the field of view 170. During operation, the image acquisition device 120 acquires one or more images of the fixed field of view 170, and the vision processor 245 locates the first corner 410 and the second corner 420 in each image. Calibration may also include correcting distortions in the camera lens in a known manner. Other aspects of the aforementioned calibration are discussed below.

[0037] Furthermore, if the working edge is raised, the positions of the first corner 410 and the second corner 420 will be raised. Similarly, if the working edge moves to the left or right and / or rotates, the positions of the first corner 410 and the second corner 420 within the field of view will shift and / or rotate accordingly. In this way, the vision-based blade position system 200 outputs a set of working edge information (e.g., blade attitude information 260) to the machine control system 220, which can combine this information with navigation information provided in a known manner by a conventional GNSS system.

[0038] Figure 6 The use of azimuth projection according to the embodiment is shown. Figure 5A / Figure 5B An illustrative mapping of the spherical portions 440 and 450 projected onto image 640. For example... Figure 6 As shown, image 640 is constructed through image boundary 650 of the projectable surface in the image, such that the blade corner 410 / blade 440 is associated with the left blade 620 in image 640 and the blade corner 420 / blade 450 is associated with the right blade 630 in image 640. Additional constraints can be utilized to analyze one or more additional degrees of freedom through image 640. Illustratively, considering the data from calibration and the pixel position of the positional feature on the blade for a given image, the position of the positional feature in the visual system reference frame can be represented by a single unknown quantity corresponding to the orientation (i.e., angle) of the C-frame in the visual system reference frame.

[0039] Figure 7 The following is illustrated according to the implementation method. Figure 5A / Figure 5B and Figure 6 An illustrative image of the blade 700 is provided for the mapping. As noted above, taking into account the data from the above calibrations and the pixel positions of the positional features on the blade for a given image, the position of the positional features in the vision system reference frame can be represented by a single unknown quantity corresponding to the orientation (i.e., angle) of the C-frame in the vision system reference frame. Illustratively, all calibrations include (1) factory calibration and (2) installation calibration.

[0040] For factory calibration (i.e., camera / lens system distortion correction to establish a mapping between pixel positions and angular space in the vision system reference frame), the distortion correction calibration is performed once at the factory and provides an accurate mapping between pixel positions in the image and angular space in the vision system reference frame. Calibration is required when mounting to the machine and is achieved in multiple stages depending on the degrees of freedom and the arrangement of the blade hinges. For a bulldozer with blades mounted to a spherical joint, calibration must find the sphere radius (centered on the sphere) corresponding to each of the two (2) feature positions and the position of the axis or pivot of the C-frame in the vision system reference frame. Finding the sphere radius corresponding to each of the two (2) feature positions is achieved by positioning the blade to the limit of the blade hinge (as appropriate) and recording the distance 710 between the position feature (corresponding to the "upper corner", e.g., 410) for each obtained pose and the pixel position 715 of the position feature in the image 640. During this step, the C-frame (e.g., C-frame 180) must remain stationary. Measurements combining the distances between the four (4) limit poses and features on the leaf blade are sufficient to calculate the radius of the sphere (centered on the sphere) corresponding to each of the two (2) feature positions. The position of the axis or pivot of the C-frame in the visual system reference frame is achieved by rotating the C-frame about its axis and recording the distance between the position features and the camera along with the corresponding pixel position in the image. The four (4) positions of the C-frame are sufficient to calculate the position and orientation of the axis and the rotation plane of the C-frame. In a similar manner, additional degrees of freedom that do not share the origin of rotation with the degrees of freedom captured above can be characterized for pose calculation.

[0041] Figure 8A flowchart illustrating the illustrative operation 800 of a sensor system for the attitude or position of a blade relative to a machine, according to an embodiment, is shown. In step 805, an evaluation is made as to whether calibration should be performed for the current installation. This decision may be made based on installation history (stored in memory) or as input from a regular user interface. In step 815, the vision system is calibrated to obtain features of the blade in an image (e.g., image 640) corresponding to reference positions (e.g., 410 and 420) on the blade (e.g., blade 110). Features are identified by special descriptors, which may include, but are not limited to, information having the following: the feature's position on the blade, the nature of certain neighboring points, the feature's orientation, or the trajectory of the feature's position (e.g., feature position 715) captured by the vision system in a series of images (e.g., image 640). At step 820, the pivot position (e.g., pivot position 190) of the C-frame (e.g., C-frame 180) of the bulldozer (e.g., bulldozer 100) is calibrated to capture the pivot position of the bulldozer (e.g., bulldozer 100) in the vision system reference frame. Figure 8 The subsequent calibration step assumes that a distance measuring device (e.g., a yo-yo) is temporarily installed between the camera (e.g., camera 120) and reference positions (e.g., reference positions 410 and 420) on the blade. To perform this calibration step, the blade 110 is held fixed relative to the spherical joint (e.g., spherical joint 460). The distance (e.g., distance 710) and image position (e.g., image position 715) of each of the reference positions 410 and 420 on the blade 110 are then captured for at least three (3) positions of the C-frame 180. In a more understandable manner, the position of the point corresponding to the reference positions 420 and 430 on the blade 110 in the visual system reference frame can be calculated. The pivot axis of the pivot position 190 of the C-frame 180 is the same as the line between the centers of two circles, each fitted to a set of points associated with each side.

[0042] Similarly, at step 825, the position of blade 110 relative to spherical joint 460 can be characterized. In this case, the C-frame 180 is fixed in place at a height, thereby allowing the blade 110 to maximize its actual range of motion around spherical joint 460. The distance (e.g., distance 710) and image position (e.g., image position 715) of each of reference positions 410 and 420 on blade 110 can be collected, and the associated points corresponding to the corners of the spherical segments (e.g., segments 440 and 450) can be calculated. In a more easily understood manner, the center position and radius of the sphere 430 associated with each segment 440 and segment 450 can be calculated using the points corresponding to the corners of segment 440 and segment 450. The center of sphere 430 associated with segments 440 and segment 450 coincides with the center of spherical joint 460.

[0043] In step 830, the final step of calibration, with the bulldozer 100 positioned on a flat surface and the blade 110 resting on that plane, the distance 710 and image position 715 of each of the reference positions 410 and 420 on the blade 110 are collected and the position of the associated point in the vision system reference frame is calculated. This data will be used to provide the machine control with the zero position used in setting the slope in a conventional manner. At step 835, the calibration step is completed by saving the calculated values ​​used to calculate the blade posture during operation and editing the installation history to reflect successful calibration. In operation (i.e., the cyclical reporting of blade posture), the decision to perform the calibration step in step 805 is evaluated as "No," and the next decision to track the blade posture in step 810 is evaluated as "Yes." In this case, the first task in step 840 during the operation cycle is to locate the blade features in image 715 using the same techniques detailed above as during calibration.

[0044] In one implementation, the next step, at step 845, is to evaluate the orientation (i.e., pose) of the C-frame relative to the vision system reference frame. Methods for generating this evaluation may include, but are not limited to, any fixed value that can be customized or non-customized with respect to the mounting, an approximation based on feature positions in image 715, or a motion trajectory of the C-frame 180 based on the orientation determined in the immediate preceding step. In a more readily understood manner, at step 850, using the evaluated orientation of the C-frame 180 and the feature positions in image 715, the positions of reference positions 410 and 420 on the blade 110 in the vision system reference frame can then be calculated. At step 855, the distances between these calculated positions can be calculated, and at step 860, the calculated distances between these positions can be verified relative to the distances between reference positions found during calibration. The error assessed during the test is used to modify the predicted pose of the C-frame from step 845, and the cycle continues at step 860 until the error assessed during the test is considered sufficiently small (e.g., within a certain error threshold) so that the result of the blade pose is reported at step 865.

[0045] As detailed above, the various embodiments described herein can be implemented in the form of methods and apparatus for practicing these methods. The disclosed methods can be performed by a combination of hardware, software, firmware, middleware, and computer-readable media (collectively, the “Computer”) installed in and / or communicatively connected to the user equipment. Figure 9 This is a high-level block diagram of an exemplary computer 900, which can be used to implement a method for performing vision-based blade positioning according to various embodiments herein. The computer 900 includes a processor 910 operatively coupled to a data storage device 920 and a memory 930. The processor 910 controls the overall operation of the computer 900 by executing computer program instructions that define the overall operation of the computer 900. A communication bus 960 facilitates connection and communication between the various components of the computer 900. The computer program instructions may be stored in the data storage device 920 or a non-transitory computer-readable medium and loaded into the memory 930 when execution is required. Therefore, the steps of the disclosed method (e.g., see...) Figure 8(As discussed above) may be defined by computer program instructions stored in memory 930 and / or data storage device 920 and controlled by processor 910 that executes the computer program instructions. For example, the computer program instructions may be implemented as computer executable code programmed by a person skilled in the art to perform illustrative operations defined by the disclosed methods. Thus, by executing the computer program instructions, processor 910 executes an algorithm defined by the methods of this disclosure. Computer 900 also includes one or more communication interfaces 950 for communicating with other devices via a network (e.g., a wireless communication network) or one or more known communication protocols. For example, such communication interfaces may be receivers, transceivers, or modems for exchanging wired or wireless communications in a variety of known ways. Computer 900 also includes one or more input / output devices 940 (e.g., camera, display, keyboard, mouse, speaker, microphone, buttons, etc.) enabling a user to interact with computer 900.

[0046] Processor 910 may include a general-purpose or special-purpose microprocessor and may be the sole processor of computer 900 or one of multiple processors. For example, processor 910 may include one or more central processing units (CPUs). Processor 910, data storage device 920, and / or memory 930 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) in a complementary manner or in a combination thereof.

[0047] Data storage device 920 and memory 930 each include a tangible, non-transitory computer-readable storage medium. Data storage device 920 and memory 930 may each include: high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices; and may include non-volatile memory, such as one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor storage devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), optical disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0048] Input / output device 940 may include peripheral devices such as cameras, printers, scanners, display screens, etc. For example, input / output device 940 may include: a display device for displaying information to a user, such as a cathode ray tube (CRT) monitor, plasma or liquid crystal display (LCD) monitor; a keyboard; and a pointing device, such as a mouse or trackball, through which the user can provide input to computer 900.

[0049] It should be noted that, for clarity, the illustrative embodiments described herein may be shown as comprising single functional blocks or combinations of functional blocks. The functionality represented by these blocks may be provided using dedicated or shared hardware, including but not limited to hardware capable of executing software. Illustrative embodiments may include digital signal processor (“DSP”) hardware and / or software performing the operations described herein. Thus, for example, it will be understood by those skilled in the art that the block diagrams herein represent conceptual views of the illustrative functionality, operation, and / or circuitry of the principles described in the various embodiments herein. Similarly, it will be understood that any flowchart, flowchart, state transition diagram, virtual code, program code, etc., represents various processes that can be substantially represented in a computer-readable medium and therefore executed by a computer, machine, or processor, whether or not such computer, machine, or processor is explicitly shown. Those skilled in the art will recognize that actual implementations of computers or computer systems may also have other structures and may include other components, and that high-level representations of some components of such computers are for illustrative purposes.

[0050] The foregoing detailed description is to be understood as illustrative and exemplary in all respects, and not restrictive, and the scope of the invention disclosed herein is not determined from the specific embodiments, but rather by the claims interpreted in the full breadth permitted by patent law. It is understood that the embodiments shown and described herein are merely illustrative of the principles of the invention and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A vision system for determining the position of a tool attached to a machine, wherein, The vision system includes: An image acquisition device for acquiring an image of the tool, the image acquisition device being positioned such that: (i) at least two points of interest located on the tool at a distal end are observed such that the at least two points of interest are visible in the image over the entire range of motion of the tool; and (ii) any combination of motions of the tool’s common origin will cause the at least two points of interest to be mapped onto the image plane of the image acquisition device by azimuth projection. Processor; and The memory stores computer program instructions that, when executed on the processor, cause the processor to perform operations including: The projections of the at least two points of interest in the image are transformed using the corresponding positions of the at least two points of interest located within the spherical plate and the known geometry of the tool with respect to the hinge of the machine hinge, and the corresponding positions of the at least two points of interest located within the spherical plate are calculated in the visual system reference frame of the visual system.

2. The vision system according to claim 1, wherein, The operation also includes: The vision system is calibrated using the following methods: For multiple locations of the tool, multiple images and multiple distances associated with the at least two points of interest are collected, the multiple locations of the tool representing the motion of the tool about a corresponding rotation center for each of the multiple locations; and The corresponding position of each of the rotation centers in the visual system reference frame is calculated.

3. The vision system according to claim 2, wherein, The machine is a bulldozer and the tool is a blade, the at least two points of interest are located at corresponding upper distal corners of the blade, and the vision system further includes: The blade has a spherical joint, the spherical joint of the blade having a limited range of rotational movement about the center of the spherical joint; and The bulldozer has a C-frame, wherein the spherical joint is supported by the C-frame and is located in an arc-shaped portion with a fixed radius around a fixed axis of rotation on the C-frame.

4. The vision system according to claim 1, wherein, The at least two points of interest are associated with corresponding targets fixed to the tool.

5. The vision system according to claim 1, wherein, The operation also includes: Select one or more visual features of the tool, wherein the one or more visual features of the tool are selected from at least one of the following: (i) one or more edge or line elements and the corresponding image orientation of the one or more edge or line elements; (ii) one or more geometric shapes and the associated geometric proportions; (iii) variations in color or intensity; (iv) feature combinations including relationships between the features; (v) the region in which one or more visual features of the image are found; (v) the trajectory of the one or more visual features in the image plane; and The at least two points of interest are obtained by evaluating one or more selected visual features using multiple target recognition operations.

6. The vision system according to claim 3, wherein, The calibration of the vision system also includes: Determine the fixed axis of rotation of the C-shaped frame; Determine the arcuate portion of the spherical joint with a fixed radius around the fixed axis of rotation of the C-shaped frame; Determine a corresponding first distance between each of the at least two points of interest on the blade and the center of the spherical joint; and Determine the corresponding second distance between the at least two points of interest on the blade.

7. The vision system according to claim 6, wherein, The calibration of the vision system also includes: Determine a reference position for the blade, wherein the reference position is a known position of the blade relative to the machine's reference frame.

8. The vision system according to claim 3, wherein, The operation also includes: The position of the blade is determined by the following: Acquire an image in which each of the at least two points of interest is displayed; The orientation of the C-frame is determined by minimizing an error threshold of the calculated distance between the at least two points of interest, the error threshold being defined using at least one of the plurality of distances; and When the error threshold is minimized, the corresponding positions of the at least two points of interest are output.

9. The vision system according to claim 2, wherein, The operation of calibrating the vision system further includes: determining the corners of the two spherical portions and recording the determined corners as pixel positions in the image.

10. The vision system according to claim 1, wherein, The azimuth projection is a remote conventional perspective type.

11. A method for determining the position of a tool attached to a machine, wherein, The method includes: An image of the tool is acquired by an image acquisition device, wherein the image acquisition device is positioned such that: (i) at least two points of interest located on the tool at a distal end are observed such that the at least two points of interest are visible in the image over the entire range of motion of the tool; and (ii) any combination of motions of the tool's common origin will cause the at least two points of interest to be mapped onto the image plane of the image acquisition device by azimuth projection; and The projections of the at least two points of interest in the image are transformed using the corresponding positions of the at least two points of interest located within the spherical plate and the known geometry of the tool with respect to the hinge of the machine hinge, and the corresponding positions of the at least two points of interest located within the spherical plate are calculated in the visual system reference frame of the visual system.

12. The method according to claim 11, wherein, The method further includes: The vision system is calibrated using the following methods: For multiple locations of the tool, multiple images and multiple distances associated with the at least two points of interest are collected, the multiple locations of the tool representing the motion of the tool about a corresponding rotation center for each of the multiple locations; and The corresponding position of each of the rotation centers in the visual system reference frame is calculated.

13. The method according to claim 12, wherein, The machine is a bulldozer with a C-frame, the tool is a blade with a spherical joint supported by the C-frame, and the spherical joint is located in an arcuate portion of a fixed radius around a fixed axis of rotation on the C-frame. The at least two points of interest are located at corresponding upper distal corners of the blade, and the spherical joint of the blade has a limited range of rotational movement around the center of the spherical joint.

14. The method according to claim 11, wherein, The method further includes: Associating the at least two points of interest with corresponding targets fixed to the tool.

15. The method according to claim 11, wherein, The method further includes: Select one or more visual features of the tool, wherein the one or more visual features of the tool are selected from at least one of the following: (i) one or more edge or line elements and their corresponding image orientations; (ii) one or more geometric shapes and their associated proportions; (iii) variations in color or intensity; (iv) combinations of features including relationships between features; (v) the region in which the one or more visual features of the image are found; (v) the trajectory of the one or more visual features in the image plane; and The at least two points of interest are obtained by evaluating one or more selected visual features using multiple target recognition operations.

16. The method according to claim 13, wherein, The calibration of the vision system also includes: Determine the fixed axis of rotation of the C-shaped frame; Determine the arcuate portion of the spherical joint with a fixed radius around the fixed axis of rotation of the C-shaped frame; Determine a corresponding first distance between each of the at least two points of interest on the blade and the center of the spherical joint; and Determine the corresponding second distance between the at least two points of interest on the blade.

17. The method according to claim 16, wherein, The calibration of the vision system also includes: Determine a reference position for the blade, wherein the reference position is a known position of the blade relative to the machine's reference frame.

18. The method according to claim 13, wherein, The method further includes: The position of the blade is determined in the following manner: Acquire an image in which each of the at least two points of interest is displayed; The orientation of the C-frame is determined by minimizing an error threshold of the calculated distance between the at least two points of interest, the error threshold being defined using at least one of the plurality of distances; and When the error threshold is minimized, the corresponding positions of the at least two points of interest are output.

19. The method according to claim 12, wherein, The method further includes: determining the corners of the two spherical pieces and recording the determined corners as pixel positions in the image.

20. The method according to claim 11, wherein, The azimuth projection is a remote conventional perspective type.

Images