System and method for displaying topographic data for an earthmoving operation

Abstract

A method for displaying a video image of a construction site (220, 400) for earthmoving operations, the method comprising the steps of: Receiving target design data for the construction site (220, 400); Receiving a spatial location and orientation for an earthmoving machine (205) operating with respect to the construction site (220, 400); Receiving a video image of at least one area of ​​the construction site (220, 400) from an imaging device (210) arranged on the earthmoving machine (205); and Displaying the video image of the construction site area (220, 400), wherein a subset of the target design data is superimposed on the video image, wherein the subset of the target design data relates to a target design surface (615) for the construction site area (220, 400), characterized in that displaying the video image further comprises updating the subset of target design data based on a change in a topographical feature of the construction site (220, 400).

Description

FIELD OF THE INVENTION

[0001] The present invention relates generally to a method and apparatus for displaying topographical data, and more particularly, to displaying a target design surface and a video image of a construction site to an operator of an earthmoving machine. BACKGROUND

[0002] Earthmoving operations typically utilize various types of earthmoving equipment to transform the topographical features of a construction site. For example, an excavator may be used to dig a trench. Similarly, a dozer may be used to remove or shift material to a desired grade. Typically, a marker, such as a stake or flag, may be placed to mark a specific location. However, conventional markers may be limited in their ability to indicate an area of ​​interest to an earthmoving equipment operator. These conventional markers provide little more than marking a specific location. For some earthmoving operations, additional information may be required to determine whether earth should be moved to or removed from the location represented by the marker.Furthermore, performing an earthmoving operation at the location of a conventional marking may remove and / or obscure the marking. Thus, conventional markings may not be suitable for certain earthmoving operations.

[0003] Another disadvantage of conventional earthmoving operations can be complications associated with viewing a reference marker and / or a construction site from within an earthmoving machine. For example, with certain earthmoving machines, an operator may not have a complete view of a construction site from within the cab of the earthmoving machine. Similarly, an operator may not be able to identify a reference marker from within the cab of certain earthmoving machines. Thus, operating the earthmoving machine can be difficult. Furthermore, operating the earthmoving machine can be further complicated by the size of a particular earthmoving machine and / or obstructions in the surrounding area.

[0004] Although conventional methods for earthmoving operations may use flags and stakes as markers, such conventional markers may not meet the operational requirements of certain earthmoving machines.

[0005] The prior art relevant to the present invention is given by the documents DE 11 2008 000 307 T5, US 6 094 625 A, EP 0 674 975 A1, US 4 855 822 A and US 2006 / 0 224 280 A1.

[0006] DE 11 2008 000 307 T5 describes a machine simulation and control system with a user interface configured to display a simulated environment, and a controller communicatively connected to the user interface and a remotely positioned machine (10). The controller is configured to receive real-time information relating to the operation of the machine at a work location. Furthermore, the work location, the operation of the machine, and the movement of a tool of the machine are simulated based on the received information.

[0007] US Pat. No. 6,094,625 A describes methods and devices that enable the use of augmented vision in surveying and related work. Virtual objects are presented to the operator on a transparent display. Some of the objects correspond to real objects in the operator's current field of view, while others offer interactive functions for the operator. Functions can be provided to support navigation on a construction site, the measurement of new survey points, the virtual input of control commands, site inspection, and survey calculations such as intersections and offsets.

[0008] EP 0 674 975 A1 discloses an augmented reality maintenance system for use in hazardous environments. It utilizes an environment modeler to create a computer model of the environment. A remote-controlled robot arm is attached to a stationary structure at one end, for example, a video camera, and has actuators that move the robot arm to desired positions in the environment. The position and orientation of the robot arm are determined by a position and attitude sensor unit. This information is passed to a robot arm renderer, which creates an image from a stored model of the robot arm, viewed from different angles.

[0009] US 4,855,822 A describes methods and apparatus for controlling a vehicle from a remote control station to enable remote control of the vehicle. A vehicle-mounted television sensor provides video image information. Vehicle position and attitude data, which can be derived from the vehicle's on-board inertial reference unit, are also transmitted to the control station. Transformation coefficients are calculated from the instantaneous position and attitude data, which a pipeline processor uses to extrapolate the image data to generate a real-time video display.

[0010] US 2006 / 0 224 280 A1 discloses a remote control system for remotely controlling a vehicle, comprising a pair of hand controls operated by a user's hands. The hand controls can be movable to cause movement of the entire vehicle and accessories of the vehicle. The remote control system enables precise remote control with visual feedback of vehicles in a variety of situations. BRIEF SUMMARY OF THE INVENTION

[0011] Disclosed and claimed herein are a system and method for displaying a video image of a jobsite for an earthmoving operation. Target design data for the jobsite is received. A spatial location and orientation for the earthmoving machine operating with respect to the jobsite is received. Additionally, a video image of at least a portion of the jobsite is received from an imaging device attached to the earthmoving machine. A video image of the jobsite area is displayed with a subset of the target design data overlaid on the video image, the subset of the target design data relating to the jobsite area. Displaying the video image includes updating the subset of the target design data based on a change in a topographical feature of the jobsite.

[0012] Further aspects, features and techniques of the invention will become apparent to one skilled in the relevant art in view of the following detailed description of the invention. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 illustrates a simplified block diagram of a controller according to one or more embodiments of the invention; Fig. 2 illustrates embodiments of a system according to one or more embodiments of the invention; Fig. 3 illustrates a process according to one or more embodiments of the invention; Fig. 4 illustrates a graphical representation of a construction site according to an embodiment of the invention; Fig. 5 illustrates a process according to one or more embodiments of the invention; and Fig. 6A-6B illustrate a graphical representation of a display screen according to one or more embodiments of the invention. DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0013] One aspect of the present invention is directed to providing data to an operator of an earthmoving machine for an earthmoving operation. In one embodiment, a controller may be configured to receive data corresponding to the spatial location and orientation of an earthmoving machine. According to another embodiment, the controller may be configured to receive target design data and mapping data corresponding to the worksite. In one embodiment, target design data may refer to topographical data associated with the worksite. The controller may further be configured to display a video image of the worksite area and a target design surface profile overlaid on the video image. In one embodiment, the target design surface profile may be transparently overlaid, translucently overlaid, etc.In this way, the controller can provide an earthmoving machine operator with desired grade, elevation, or fill information as one or more of a plan, a profile, or a representation of a three-dimensional model.

[0014] According to another aspect of the invention, a process for presenting a target design surface profile to an operator of an earthmoving machine is provided. The process includes receiving design surface data for a jobsite and a position and orientation of an earthmoving machine. The process further includes receiving a video image from an area of ​​the jobsite. The process includes overlaying a target design surface profile onto the video image of the jobsite.

[0015] Another aspect of the invention relates to a system that can be provided for presenting a real-time visual display of a construction site to an operator of an earthmoving machine. In one embodiment, the system can comprise a controller arranged in an earthmoving machine. The system can further comprise a plurality of reference markers. According to a further embodiment, the system can comprise a control center in communication with the earthmoving machine, the control center being configured to collect the orientation and position of the earthmoving machine and reference markers to generate a topographical model of a construction site. The controller can communicate with elements of the system to provide a real-time display of the construction site, including a target design surface.

[0016] When implemented in software, the elements of the invention are essentially the code segments that perform the necessary outputs. The program or code segments may be stored in a processor-readable medium. The "processor-readable medium" may include any medium capable of storing or transmitting information. Examples of a processor-readable medium may include an electronic circuit, a semiconductor memory device, a read-only memory (ROM), a flash memory or other non-volatile memory, a floppy disk, a CD-ROM, an optical disk, a hard disk, a fiber-optic medium, etc. The code segments may be downloaded over computer networks such as the Internet, intranets, etc.

[0017] Now with reference to the drawings, the Fig. 1 is a simplified block diagram of a controller according to one or more embodiments of the invention. According to one embodiment of the invention, a controller 100 may be configured to provide design data to an operator of an earthmoving machine. As used herein, design data may refer to mapping features within a jobsite, including a position and / or elevation of features within the jobsite. It may also be understood that the design data may include a target design surface for at least a portion of the jobsite, where the target design surface refers to a desired grade, elevation, or presentation of the jobsite. According to another embodiment, design data may refer to a desired cutout and / or fill for material.For example, design data can specify the height or depth of stones to be moved to a particular location. As described in the . Fig. 1, the controller 100 includes a processor 105 coupled to a GNSS input 110, a video input 115, a memory 120, a display 125, and an input / output (I / O) interface 130. The GNSS input 110 may be configured to receive data corresponding to a spatial location and / or orientation of an earthmoving machine. In one embodiment, the controller 100 may receive at least any of global navigation satellite system (GNSS) data, global positioning system (GPS) data, and geospatial positioning data generally via the GNSS input 110. According to another embodiment, the controller 100 may receive earth-based radio positioning data, 3D positioning data generated by a total station, and / or any type of 3D positioning data in general.The video input 115 of the controller 110 may be coupled to an imaging source, such as a digital still camera, a video camera, or a video device in general. As such, the video input 115 may be usable to receive imaging data from an imaging device. According to another embodiment, the input / output (I / O) interface 130 may be usable to obtain design and / or updated topographical data for a construction site via a wired or wireless connection. The processor 105 is configured to provide one or more output signals via the I / O interface 130 such that video data received from the video input 115 and topographical data received from the I / O interface 130 are presented on the display 125. It can also be understood that data received by the processor 105 may be stored in the memory 120.Memory 120 may be either ROM or RAM. According to another embodiment, processor 105 may be any type of processor, such as a microprocessor, a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC).

[0018] With reference now to the Fig. 2 shows a simplified system diagram 200 which illustrates the controller of the Fig. 1. As stated in the Fig. 2, the system 200 includes a controller 225 (e.g., the controller 100), which may be disposed in the earthmoving machine 205. In one embodiment, an imaging device 210 and a GPS receiver 215 may be coupled to the earthmoving machine 205. Thus, the imaging device 210 may be configured to capture visual data from an area of ​​the worksite 220 (e.g., video stream data or still image data). The area of ​​the worksite 220 may refer to a field of view of the imaging device 210. As shown in the Fig. 2, the worksite area 220 is shown as the forward area of ​​the earthmoving machine 205. However, it should also be understood that the imaging device 210 may be configured to capture a video image related to any direction of the earthmoving machine 205. According to one embodiment, the imaging device 210 may be configured to capture a panoramic image. The GPS receiver 215 may be configured to determine a spatial location and / or orientation of the earthmoving machine 205. According to another embodiment, the controller 225 may be coupled to the imaging device 210 and the GPS receiver 215 through either a wireless or a wired communication link. In this manner, the controller 225 may be configured to display a video image of the worksite area 220.Further, the controller 225 may be configured to display layout and topographical data on a display 230 (e.g., the display 125). For example, a target layout surface for the construction site may be displayed, as described in further detail below with reference to FIG. Fig. 6A-6B. In one embodiment, the display 230 may be a liquid crystal display (LCD). It should also be understood that other types of displays may be used by the controller 225. In one embodiment, the controller 225 may include at least one port 235 that can be used by a user to adjust the display 230.

[0019] With reference now to the Fig. 3, a process 300 for displaying topographic data is shown in accordance with one or more embodiments of the invention. The process 300 is initiated by receiving design and / or topographic data for a jobsite (e.g., jobsite 220) at block 305. Design data received at block 305 may include a target elevation and / or slope for a jobsite. Similarly, topographic data received at block 305 may relate to cut and / or fill data for the jobsite. The spatial position and / or orientation of an earthmoving machine (e.g., earthmoving machine 205) is received at block 310. The process 300 includes receiving a video image of a jobsite at block 315. In one embodiment, the video image may be one of a video stream or one or more digital still images.Video data received at block 315 is correlated with an area of ​​a construction site. Layout or topographical data received at block 315 is overlaid on the video image at block 320. In one embodiment, overlaying a target layout area at block 320 may include using at least one auxiliary marker or a reference marker. According to another embodiment, layout data presented by process 300 may be updated, as described in further detail below with reference to FIG. Fig. 5 is described.

[0020] Fig. Figure 4 illustrates a simplified graphic representation of a construction site 400 in which the system of Fig. 2. In one embodiment, the construction site 400 may be a predefined area having at least one boundary. As described in the Fig. 4, the worksite 400 generally conforms to a rectangular shape. However, it should be understood that the worksite 400 may conform to other shapes. An earthmoving machine 405 (e.g., the earthmoving machine 205) may be configured to receive spatial positioning and orientation data using a GPS receiver 410. In one embodiment, the earthmoving machine 405 may receive spatial positioning and orientation data from a GPS satellite 425 via the GPS receiver 410. Based at least in part on one or more signals received from the GPS receiver 410, a controller (e.g., the controller 225) coupled to the earthmoving machine 405 may present layout and / or topographical data related to the worksite 400 to an operator of the machine.According to another embodiment, the earthmoving machine 405 may receive topographic data via an I / O interface (e.g., the I / O interface 130) of a control station 415. Furthermore, the control center 415 may receive topographic reference data from one or more of a plurality of reference markers 4301-430. n collect, as well as from a slope of the construction site, which is marked 435.

[0021] Now with reference to the Fig. 5 shows a process 500 for displaying topographical data for a construction site according to an embodiment of the invention. The process 500 may be initiated by displaying a real-time image of a construction site at block 505. The real-time image may be captured from an imaging device (e.g., imaging device 210) attached to an earthmoving machine (e.g., earthmoving machine 205). Layout and topographical data for the construction site may be received at block 510. At block 515, a position and / or orientation of the earthmoving machine may be determined. For example, a GPS receiver (e.g., GPS receiver 215) may be configured to determine the spatial location and orientation of the machine. It can also be understood that a controller (e.g., controller 225) may be configured to determine an orientation of the earthmoving machine.

[0022] The process 500 may further include determining whether cut and / or fill data exists or can be calculated for the current position and / or orientation of the machine. According to one embodiment of the invention, cut / fill data may be received from a controller (e.g., controller 215) disposed within the earthmoving machine. The cut / fill data may refer to data indicative of at least one of a desired elevation and grade of the jobsite as displayed to an operator of the earthmoving machine. In another embodiment, cut / fill data may be determined by the controller from the design surface and / or the most recent topographic surface data available to the controller. If no design data exists for the current spatial position of the earthmoving machine (e.g.,If the path is "No" (e.g., the path out of block 520), the position and orientation of the earthmoving machine may be monitored, as described in block 515. However, if topographical data for the current spatial position of the earthmoving machine is available (e.g., the path out of block 520), a display (e.g., display 230) representing a portion of the jobsite may be updated, as shown in block 525. In one embodiment, a target design surface may be overlaid on the real-time image of the jobsite. Thus, an operator of the earthmoving machine may be provided with the desired design data while operating the earthmoving machine.

[0023] Further with reference to the Fig. 5, the process 500 may include determining, at block 530, whether a position and / or orientation of the earthmoving machine has changed. If the position or orientation of the earthmoving machine has changed (e.g., path "Yes" out of block 530), the position and orientation of the earthmoving machine may be determined by the controller (e.g., controller 225) as shown in block 515. However, if the position or orientation of the earthmoving machine has not changed (e.g., path "No" out of block 530), a controller may determine whether design and / or topographic data for the jobsite 535 has been updated. If the design and / or topographic data for the jobsite has been updated (e.g., path "Yes" out of block 535), the position and orientation of the earthmoving machine may be determined as described in block 515.However, if the design and / or topographic data for the jobsite has not been updated (e.g., the "No" path out of block 535), the controller may be configured to monitor the position of the earthmoving machine as described in block 515. According to another embodiment of the invention, it can be understood that the operator of the earthmoving machine may use input ports (e.g., ports 235) of a controller to update at least one of the video image and the jobsite topographic data.

[0024] With reference now to the Fig. 6A-6B show exemplary views of a display window 600a and 600b (e.g., display window 230) according to one or more embodiments of the invention. A target design surface 615 is overlaid on the video image of the construction site 610. As shown in the Fig. 6a, the target design surface 615 may be represented as lines indicating at least one of a desired elevation and gradient for the worksite area. It may also be understood that the target design surface 615 may be represented as shaded, colored, or patterned areas. In certain embodiments, the display window 600a may refer to an area of ​​the worksite 610 (e.g., an area of ​​a worksite 220) in the vicinity of an earthmoving machine. According to another embodiment, GPS reference stations 620 may be used as reference markers for the display window. For example, a controller (e.g., the controller 225) may be configured to use a GPS reference station 620 to align a target design surface with the video image of the worksite 610.In this way, the video image of the jobsite 610 displayed in the display window 600a can be aligned with the design surface 615. In one embodiment, the alignment of the video image of the jobsite 610 and the target design surface 615 can be determined using position and / or alignment data collected from angle sensors, inertial sensors, GNSS receivers, and positioning devices in general. In yet another embodiment, the display window 600a can refer to a readout device worn by an operator of the earthmoving machine.

[0025] With reference now to the Fig.6B, a display window 600b is shown depicting a video image of an area of ​​the jobsite 610 corresponding to a design surface 615. It can also be understood that the target design surface 615 may be displayed as a second color when the target elevation or grade of the jobsite 610 is reached. In this way, the earthmoving machine operators may be provided with an indication that a target grade or grade has been reached.

[0026] The invention is defined by the subject matter of the appended independent claims. Particular embodiments are provided by the additional features of the appended dependent claims.

Claims

[1] A method for displaying a video image of a construction site (220, 400) for earthmoving operations, the method comprising the steps of: Receiving target design data for the construction site (220, 400); Receiving a spatial location and orientation for an earthmoving machine (205) operating with respect to the construction site (220, 400); Receiving a video image of at least one area of ​​the construction site (220, 400) from an imaging device (210) arranged on the earthmoving machine (205); and Displaying the video image of the construction site area (220, 400), wherein a subset of the target design data is superimposed on the video image, wherein the subset of the target design data relates to a target design surface (615) for the construction site area (220, 400), characterized bythat displaying the video image further comprises updating the subset of the target design data based on a change in a topographical feature of the construction site (220, 400). [2] The method of claim 1, wherein the target design surface (615) corresponds to at least one of position data, elevation data, grade data, cut / fill data, and a spatial reference marker for the construction site (220, 400). [3] The method of claim 1, wherein the spatial location data comprises at least one of global navigation satellite system (GNSS) data, global positioning system (GPS) data, and geospatial positioning data in general. [4] The method of claim 1, wherein displaying the video image further comprises updating the subset of target design data based on at least one of a change in position and a change in orientation of the earthmoving machine (205). [5] The method of claim 1, wherein displaying the video image further comprises displaying the subset of the design data as at least one of color attenuation and colorization of the video image. [6] The method of claim 1, further comprising adjusting the display (125, 230) of the video image of the construction site (220, 400) based at least in part on a received user input. [7] The method of claim 6, wherein adjusting the video image on the display (125, 230) comprises at least one of enlarging, reducing, horizontally positioning, and vertically positioning a viewing angle of the area of ​​the construction site (220, 400). [8] The method of claim 1, further comprising adjusting the display (125, 230) of the subset of design data based at least in part on lateral, longitudinal and vertical rotation. [9] An operator interface for displaying a video image of a construction site (220, 400) for an earthmoving machine (205), the interface comprising: an advertisement (125, 230); at least one input (110) designed to receive target design data for the construction site (220, 400), a spatial location and orientation for an earthmoving machine (205) operating with respect to the construction site (220, 400), and receives a video image of at least one area of ​​the construction site (220, 400) from an imaging device (210) arranged on the earthmoving machine (205); and a processor (105) coupled to the at least one input (110) and the display (125, 230), the processor (105) being configured to: outputs data received at the at least one input (110) to the display (125, 230) so that a video image of the construction site area (220, 400) is displayed; and a subset of the target design data is displayed overlying the video image thereof, wherein the subset of the target design data relates to a target design surface (615) for the area of ​​the construction site (220, 400), characterized by that displaying the video image further comprises updating the subset of the target design data based on a change in a topographical feature of the construction site (220, 400). [10] The operator interface of claim 9, wherein the target design surface (615) corresponds to at least one of position data, elevation data, slope data, cut / fill data, and a spatial reference marker for the construction site (220, 400). [11] The operator interface of claim 9, wherein the spatial location data comprises at least global navigation satellite system (GNSS) data, global positioning system (GPS) data, earth-based radio positioning data, and geospatial positioning data in general. [12] The operator interface of claim 9, wherein the processor (105) is further configured to update the subset of target design data based on at least one of a change in position and a change in orientation of the earthmoving machine (205). [13] The operator interface of claim 9, wherein displaying the video image further comprises displaying the subset of the target design data as at least one of color attenuation and colorization of the video image. [14] The operator interface of claim 9, wherein the processor (105) is further configured to adjust the display (125, 230) of the video image of the construction site (220, 400) based at least in part on a received user input. [15] The operator interface of claim 14, wherein adjusting the video image of the display (125, 230) comprises at least one of enlarging, reducing, horizontally positioning, and vertically positioning a viewing angle of the area of ​​the construction site (220, 400). [16] The operator interface of claim 9, wherein the processor (105) is further configured to adjust the display (125, 230) of the subset of design data based at least in part on lateral, longitudinal, and vertical rotation of the earthmoving machine (205). [17] Operator interface according to claim 9, wherein the video image of at least a portion of the construction site (220, 400) is received by imaging device (210) mounted on the earthmoving machine (205).

Images