3D Exclusion Zone for Machines

Abstract

A work machine (10), a first method (500) of defining a virtual 3D exclusion zone (99), and a second method (600) of preventing a collision involving the work machine (10) are disclosed. The work machine (10) comprises a body (11), an implement arm (20), an imaging assembly (40), and an electro-hydraulic assembly (30) configured to prevent the implement arm (20) from intersecting the 3D exclusion zone (99). The first method (500) includes scanning a local environment (80), generating a virtual 3D representation (93), identifying a primary structure (84), and generating a virtual 3D exclusion zone (99) that encompasses the primary structure (84). The second method (600) includes defining the virtual 3D exclusion zone (99), monitoring operation of the work machine (10), and adjusting operation to avoid intersection between the work machine (10) and the 3D exclusion zone (99). The 3D exclusion zones (99) may be implemented for multiple machines (10), environments (80), and primary structures (84) without unduly removing control from the human operator.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to systems and methods for protecting work machines, and more particularly to systems and methods for creating virtual 3D exclusion zones designed to prevent collisions involving work machines. [Background technology]

[0002] Earth moving equipment such as excavators, backhoes, loaders, etc. are designed to handle heavy loads and operate in difficult working environments. Thus, proper and careful operation of such equipment is a primary concern for manufacturers, employers, and personnel alike. As a human endeavor, operators may require formal training and certification to drive any given machine and must remain fully mindful of the machine's operation and surrounding environment while operating. To assist operators, protection protocols are also built into the machine itself. Common examples of machine protection features may include audible reverse gear alarms, speed or weight limiters, scissor point guards, overhead guards, traction control, and other possible systems and devices depending on the machine in question and its use case.

[0003] One category of machine guarding features relates to the spaces that a machine may enter and / or occupy. That is, the machine and its components must not enter spaces where there are obstacles such as another vehicle, personnel, or physical features, and should preferably avoid said spaces with some tolerance. Similarly, heavy loads should not be maneuvered above other vehicles or personnel. While these guidelines are traditionally within the purview of a human operator, they may be enforced or protected by automated means.

[0004] In the related art, the use of computer vision is generally known to identify patterns and features in images and / or videos. For example, methods and techniques related to image processing, feature extraction, and object recognition may be used to, among other things, identify objects and terrain, track surfaces or planes in three-dimensional (3D) environments, and generally augment or augment optical perception by the human eye alone.

[0005] In the related art, the use of 3D scanning is generally known to enable digital reconstruction of real-world objects. For example, methods and techniques including contact scanning, time-of-flight (TOF) 3D laser scanners, or stereo photography, among others, can be used to generate virtual 3D representations of objects or environments.

[0006] With regard to protective features including allowable maneuvering space, the prior art discloses a number of systems designed to prevent collisions between a work machine and surrounding obstacles and / or vehicles. These systems may utilize a combination of the machine's proprioceptive sensors and known coordinates of adjacent obstacles to calculate allowable maneuvers on a portion of the machine. However, the prior art stops short of providing a method to protect the work machine through the application of virtual 3D exclusion zones, which may be adjustable to multiple work machines, operating environments, and / or obstacles for avoidance.

[0007] An example of the prior art can be found in U.S. Publication No. 2020 / 0199853, designed by Konda et al. and assigned to Komatsu Corporation. Konda discloses a method and control system for operating a loading machine relative to a transport vehicle. During an earth haul operation, the method first obtains the position and azimuth orientation of both the loading machine and the transport vehicle. The method then specifies an earth haul position relative to the machine and a bucket position relative to the machine's bucket, i.e., a position where a target load can best be transferred from the bucket onto the transport vehicle. Konda further discloses an interference avoidance plan that outputs limit positions for the loading machine and its bucket, including the space physically occupied by the transport vehicle.

[0008] The system disclosed by Konda suffers from being applicable only in a limited number of situations, and may lose effectiveness in less than optimal operating conditions. For example, Konda is silent about the polar angle of any machine, i.e., the inclination of the vehicle when fixed on uneven terrain. Furthermore, the disclosed interference avoidance scheme is designed to prevent only collisions involving swing-body type loading machines and uniform height transport vehicles, and may not be applicable to work machines involving other ranges of motion or collision risks. Finally, the disclosed controller removes finer manual control from the hands of the human operator by automating the signals output to the machine actuators. Each of these problems indicates a general lack of versatility in the solutions proposed by the prior art.

[0009] Thus, there remains a need in the art for systems and methods that implement virtual 3D exclusion zones for work machines, which 3D exclusion zones are operable in a plurality of diverse environments and can be programmed to avoid any number of diverse obstacles or spaces without unduly removing control from a human operator. Summary of the Invention

[0010] According to a first aspect of the present disclosure, a method for defining a virtual 3D exclusion zone is disclosed, the method including scanning a local environment of a first work machine, identifying a second work machine within the local environment, generating a virtual 3D representation of the local environment and the second work machine, identifying one or more primary structures of the second work machine, orienting the first work machine relative to the local environment, and generating a virtual 3D exclusion zone encompassing the primary structures, where the 3D exclusion zone defines a space that should not be occupied by the first work machine.

[0011] According to a second aspect of the present disclosure, a method of preventing collisions involving a work machine through application of a virtual 3D exclusion zone is disclosed, the method including orienting the work machine relative to a local environment, defining a virtual 3D exclusion zone, monitoring motion of the work machine, predicting whether the motion will result in an intersection between the work machine and the 3D exclusion zone, and, if an intersection is predicted, adjusting the motion to avoid the intersection.

[0012] According to a third aspect of the present disclosure, a work machine is disclosed. The work machine includes a body, an implement arm supported by the body, an imaging assembly configured to define a 3D exclusion zone, a power unit, and an electro-hydraulic assembly operably powered by the power unit. The imaging assembly further includes one or more 3D imaging devices supported by the implement arm, and an image processing unit. The electro-hydraulic assembly further includes one or more implement actuators configured to control movement of the implement arm, one or more implement sensors configured to monitor movement of the implement arm, and an electro-hydraulic control unit (ECU) configured to command the implement actuators to prevent the implement arm from intersecting the 3D exclusion zone. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an illustration of a work machine according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is an illustration of a field of view (FOV) of one or more stereo cameras associated with a work machine according to another embodiment of the present disclosure. [Diagram 3] FIG. 3 is an illustration of a stereo camera associated with a work machine according to another embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram of a system architecture for a work machine according to another embodiment of the present disclosure. [Diagram 5] FIG. 5 is a flow chart illustrating a method for defining a 3D exclusion zone for a work machine according to another embodiment of the present disclosure. [Figure 6]FIG. 6 is a flow chart illustrating a first method for preventing collisions involving a work machine through application of 3D exclusion zones in accordance with another embodiment of the present disclosure. [Figure 7] FIG. 7 is a flow chart illustrating a second method for preventing collisions involving a work machine through application of 3D exclusion zones in accordance with another embodiment of the present disclosure. [Figure 8] FIG. 8 is an illustration of a first work machine, a second work machine, a local environment, a virtual 3D exclusion zone, and a virtual 3D deceleration zone in accordance with another embodiment of the present disclosure.

[0014] These and other aspects and features of the present disclosure will be more readily understood after reading the following description in conjunction with the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Referring now to the drawings, and in particular to FIG. 1, a diagram of a work machine is generally referred to by the reference numeral 10. While the work machine 10 is shown as a shovel, the work machine 10 may alternatively be a front loader, backhoe, rope shovel, material handler, or equivalent machine or vehicle with components operable relative to a body 11 of the machine 10. More specifically, the work machine 10 may comprise the body 11, an implement arm 20 attached to and supported by the body 11, a power unit 12, a traction device 13 operably powered by the power unit 12, and an electro-hydraulic assembly 30 operably powered by the power unit 12 and operably controlling the implement arm 20.

[0016] Power unit 12 may include an internal combustion engine (ICE), rechargeable batteries, hydrogen fuel cells, alternative power systems, and / or any combination of the above for providing power to machine 10 and its components. Traction device 13 may feature a pair of crawler belts as shown, or may engage the ground in a separate manner, such as by employing wheels and tires, tracks, or the like, to propel work machine 10. Body 11 may be supported by traction device 13 and, in some embodiments, may be pivotable relative to traction device 13 about a pivot axis that is perpendicular to the ground.

[0017] With continued reference to FIG. 1 , the work machine 10 may include an implement arm 20. In an exemplary configuration, the work machine 10 may be a shovel and the implement arm 20 may include a boom 21 pivotally connected to the body 11 at a proximal end and pivotally attached to a stick 22 at a distal end. Similarly, the stick 22 may be pivotally attached to the boom 21 at a proximal end and pivotally attached to a tool 23 at a distal end. However, in other embodiments, the implement arm 20 may instead include only the boom 21, only the stick 22, only the tool 23, alternative or additional attachments such as a lift arm or dipper handle (not shown), and / or some combination of the above. In still other embodiments, the work machine 10 may include two or more implement arms 20, and the aforementioned implement arms 20 may or may not be identical. In any embodiment, the implement arm 20 and its attachments may be actuable, i.e., movable, relative to the body 11.

[0018] 1 , the work machine 10 includes an electro-hydraulic assembly 30 including one or more implement actuators 31 configured to control movement of the implement arm 20 and one or more implement sensors 32 configured to monitor movement of the implement arm 20. In an exemplary configuration, the implement actuators 31 may be hydraulic cylinders and may include a boom actuator 311 configured to raise or lower the boom 21, a stick actuator 312 configured to raise or lower the stick 22, and a tool actuator 313 configured to manipulate the tool 23. However, it should be understood that additional or alternative actuators 31 may be included depending on the type of work machine 10, the particular attachments of the implement arm 20, their respective ranges of motion, and other application requirements.

[0019] The instrument sensor 32 is configured to monitor the motion of the instrument arm 20. In an embodiment, the instrument sensor 32 may be configured to operatively monitor at least the position, velocity and / or angular velocity, and acceleration and / or angular acceleration of the instrument arm 20 relative to the machine body 11 and / or another point in a given reference frame. In the same or other embodiments, the instrument sensor 32 may operatively monitor the jerk and / or angular jerk, as well as higher order displacements, of the instrument arm 20 relative to the machine body 11 or a point. In an embodiment, the position of the instrument arm 20, i.e., the space occupied by the instrument arm 20, may be derived from known schematics. It is understood that no limitations are intended with respect to the type or amount of kinematic characteristics monitored or the reference frame through which the characteristics are interpreted.

[0020] In an exemplary configuration, the implement sensors 32 may include a boom sensor 321 configured to monitor the angle of elevation of the boom 21 relative to the body 11, a stick sensor 322 configured to monitor the angle of elevation of the stick 22 relative to the boom 21, and a tool sensor 323 configured to monitor the angle of elevation of the tool 23. It should be further understood that additional or alternative sensors 32 may be included in the electro-hydraulic assembly 30 and additional or alternative displacements, angles, and derivatives thereof may be monitored depending on the type of work machine 10, the particular appendages of the implement arm 20, their respective ranges of motion, and other application requirements.

[0021] The above measurements may be collected by an electrohydraulic control unit (ECU) 33 and interpreted by various means common in the art to calculate movement of the implement arm 20. The ECU 33 may be included within the electrohydraulic assembly 30 and may be configured to receive input from a machine operator (not shown) and operatively receive or calculate movement data from the implement sensors 32 and command the implement actuators 31.

[0022] Additionally, the ECU 33 may receive 3D coordinate data defining an "exclusion zone" 99 and may command the instrument actuator 31 to prevent the instrument arm 20 from intersecting said exclusion zone 99, a function discussed in more detail below. In some embodiments, the ECU 33 may override input received from the machine operator if a resulting intersection with the exclusion zone 99 is predicted.

[0023] Work machine 10 further includes an imaging assembly 40 including one or more 3D imaging devices 41 supported by implement arm 20 and an image processing unit 42. Referring now to Figure 2, imaging assembly 40 may be configured to scan the work machine's local environment 80, identify particular objects 82 and key structures 84, and define 3D exclusion zones 99. The one or more 3D imaging devices 41 may continuously scan the environment 80 and provide data to image processing unit 42.

[0024] In some embodiments, the 3D imaging device 41 may include two or more stereo cameras 41, an exemplary stereo camera 41 is shown in FIG. 3. The stereo camera 41 may include at least two separate image sensors 411 and 412 and respective lens assemblies that enable the cameras to capture 3D images. In various embodiments, the image sensors 411 and 412 may both be monochrome sensors, such as those using CCD or CMOS technology, or one or both may be color sensors, such as those using Bayer filter or 3-CCD technology. In the same or other embodiments, each stereo camera 41 may further include a third image sensor 413 that is a dedicated color sensor.

[0025] In other embodiments (not shown), the 3D imaging device 41 may instead comprise an alternative imaging device such as a TOF 3D laser scanner, a structured light 3D scanner, an ultrasonic sensor, a photometric camera, or other technologies that may enable 3D reconstruction.

[0026] Each 3D imaging device 41 may further include one or more communication buses 47 configured to transfer real-time data to the image processing unit 42, and one or more mounting brackets 49 configured to mount the 3D imaging device 41 to the instrument arm 20. The mounting brackets 49 may be designed to associate with the instrument arm 20 by a variety of mechanisms common in the art, including but not limited to machine screws, bolts, magnetic mounts, interference fits, welding, etc., which may or may not be reversible.

[0027] Returning to FIG. 2, several possible mounting locations and associated fields of view (FOV) of the 3D imaging device 41 are shown. In an exemplary configuration, a first pair of stereo cameras 41A, 41B may be symmetrically positioned on the left and right sides of the boom 21 and may capture FOV 41X. Additionally, a second pair of stereo cameras 41C, 41D may be symmetrically positioned on the left and right sides of the stick 22 and may capture FOV 41Y. In other embodiments, only the first pair 41A, 41B may be mounted, only the second pair 41C, 41D may be mounted, only one stereo camera 41 of each pair may be mounted, and / or the stereo cameras 41 may not be mounted symmetrically with respect to the implement arm 20. It should be understood that alternative mountings may be provided and / or additional stereo cameras 41 may be included depending on the attachments of the implement arm 20, their respective ranges of motion, and the application requirements of the work machine 10. The illustrated FOVs 41X, 41Y are exemplary only and may be wider or narrower than shown, and may or may not be conical in shape, depending on application requirements. If alternative 3D imaging devices 41 are employed, their number, placement, and FOV may further depend on the characteristics of the imaging devices 41 and their principles of operation.

[0028] It can be appreciated that in any embodiment, the installation of the 3D imaging device 41 is designed to capture the local environment 80 of the work machine 10, including the terrain and any obstacles, vehicles, or personnel in the vicinity of the work machine 10 and / or in the vicinity of the implement arm 20.

[0029] Referring now to Figure 4, the system architecture of the work machine 10 is generally referred to by the reference numeral 50. As shown in Figure 4, the system 50 may include an imaging assembly 40 having one or more 3D imaging devices 41 and an image processing unit 42, an electro-hydraulic assembly 30 having one or more implement actuators 31, one or more implement sensors 32, and an ECU 33.

[0030] The system 50 may be configured to capture, via the 3D imaging device 41, images 91 (or equivalent sensory data) describing the local environment 80. For purposes of this disclosure, a captured "image" may refer to any data recorded by the 3D imaging device 41, regardless of its type or file format. The captured images 91 may then be transferred in real-time to the image processing unit 42. In an embodiment, the image processing unit 42 may be configured to generate a 3D representation 93 of the local environment 80 from the captured images 91 to identify specific objects 82 within the local environment 80, to identify key structures 84 within or relative to the object 82, and to define a 3D exclusion zone 99 encompassing the key structures 84.

[0031] Further in real time, the 3D coordinate data defining the 3D exclusion zone 99 may be communicated to the ECU 33. Simultaneously, the ECU 33 may monitor the movement of the instrument arm 20 via the instrument sensor 32 and may receive input from a human operator of the machine 10 and command the instrument actuator 31 to prevent the instrument arm 20 from entering or occupying the 3D exclusion zone 99.

[0032] 8, an exemplary first work machine 10, object 82, main structure 84, and exclusion zone 99 are shown. In an embodiment, the object 82 identified by image processing unit 42 may comprise a second work machine 82A. In the same or other embodiments, the second work machine 82A may be a transport vehicle such as a dump truck, dump trailer, mining truck, off-highway truck, or similar machine designed to haul and transport heavy loads, and the identified structure 84 may include a dump bed 841, a cabin 842, and a space 843 above the cabin 842 of the second work machine 82A. In the above situation, the ECU 33 may be operatively programmed to avoid an intersection or collision between the implement arm 20 and the aforementioned structures 841, 842, 843.

[0033] However, in other embodiments, other objects 82 and structures 84 may be identified by image processing unit 42 and applied to define 3D exclusion zone 99. In various examples, objects 82 may include terrain features, people, animals, other types and / or models of work machines, and / or combinations of the above, and no limitation is intended by this disclosure. Similarly, primary structure 84 may include additional or alternative components of second work machine 82A, additional or alternative spaces relative to second work machine 82A, components and / or spaces relative to other work machines, components and / or spaces relative to other objects 82, and / or combinations of the above, and no limitation is intended by this disclosure.

[0034] Returning now to FIG. 4 , image processing unit 42 may include a central processing unit (CPU) 421, a memory 422 in the form of a non-transitory computer-readable medium, and a graphics processing unit (GPU) 423, and may optionally include a field programmable gate array (FPGA) 424. In various embodiments, image processing unit 42 may be capable of performing stereo image processing and disparity mapping, 3D image mapping and object reconstruction, volume rendering, image filtering and segmentation, color analysis, edge detection, gaging and searching, optical character recognition, pattern recognition, and / or object recognition, among other possible functions. In an embodiment, image processing unit 42 may be configured to perform neural nets and / or machine learning processes to enable object recognition functions discussed in more detail below.

[0035] The ECU 33 may include a central processing unit 331 and memory 332 in the form of a non-transitory computer readable medium. The ECU 33 may comprise, but is not limited to, a gateway computer, an FPGA, an application specific integrated circuit (ASIC), the control system of the work machine 10, or equivalent computing device capable of receiving a data stream in real time and outputting commands to the implement actuators 31. In some embodiments, the ECU 33 and the image processing unit 42 may occupy the same computing device and / or share the same computing resources.

[0036] 4, the work machine 10 may further include a positioning unit 17 and a communications unit 18. The positioning unit 17 may be configured to position the work machine 10 relative to a global environment 88, such as a project work site, a construction zone, a mining site, etc. In various embodiments, the positioning unit 17 may use a global positioning system, such as the Global Positioning System (GPS) for short, and / or may use local positioning techniques common in the art.

[0037] Communications unit 18 may enable communication between work machine 10 and an external database, such as an external database stored in a central management station (not shown), another work machine (not shown), or a fleet of work machines (not shown). In various embodiments, communications unit 18 may employ short or medium range wireless communication technologies, such as, but not limited to, Wi-Fi, Bluetooth, cellular, and / or other protocols common in the art.

[0038] In one embodiment, the location of work machine 10 may be used as a key to search for existing data related to local environment 80, such as an existing 3D representation 93 of local environment 80. In another embodiment, the identification of object 82 may be used as a key to search for existing data related to said object 82, such as an existing 3D representation 93 of object 82 and / or its primary structure 84.

[0039] By employing the designs of the present disclosure, the work machine 10 may avoid collisions associated with various local environments 80, objects 82, primary structures 84, obstacles, equipment, and / or personnel through application of virtual 3D exclusion zones 99. Thus, the work machine 10 and its environment 80 may be better protected without relinquishing control from the human operator. [Industrial Applicability]

[0040] The present disclosure may find industrial applicability to a number of work machines or vehicles and may be implemented to protect the work machine, its operator, its operating environment, and any adjacent equipment or personnel.

[0041] The present disclosure may be particularly applicable to work machines with actuatable implement arms, i.e., implement arms that are movable relative to the body of the machine. Various exemplary work machines may include, but are not limited to, shovels, backhoes, material handlers, wheel loaders, skid steers, forestry machines, and the like, whose implement arms may include, but are not limited to, booms, sticks, lift arms, dippers, tools, and / or other common appendages and attachments.

[0042] The work machines may be employed in a number of different operating environments, such as, but not limited to, open pit mines, road construction zones, loading hubs, etc., and may be employed in a number of different industries, such as, but not limited to, earth moving, mining, construction, transportation, agriculture, etc. The present disclosure may be particularly applicable to environments where uneven terrain and / or other operating precautions may be desirable.

[0043] By using the imaging systems and methods disclosed herein, a virtual 3D representation can be generated to simulate the local environment and any objects and structures therein. Such objects can include surrounding machinery or vehicles, personnel or animals, terrain features, and other physical entities. Additionally, 3D exclusion zones can be defined to encompass key structures for avoidance. Such structures can include specific aspects of objects such as vehicle components, spaces relative to the vehicle, "bubbles" surrounding pedestrians, etc., depending on application requirements. Additionally, 3D exclusion zones can be defined with tolerances and can be further enveloped and protected by deceleration zones.

[0044] The 3D exclusion zone may be used in a number of protection protocols designed to protect the work machine and its surroundings. In various examples, occupancy or proximity to the 3D exclusion zone may trigger warnings, alerts, automatic driver overrides, and other possible responses. In particular, an automatic override may slow or adjust the operation of the work machine without significantly interrupting machine operations.

[0045] By employing the work machines and methods of the present disclosure, collisions between the machine and its surroundings may be prevented, thereby protecting the machine and its operator. Similarly, the risk of collisions posed to the local environment, including equipment and personnel, may be reduced. The 3D exclusion zones may be further programmed to include overhead space, thereby reducing the risk posed by falling debris or falling loads. Advantageously, the present disclosure may accomplish the above protocols without unduly removing control from the operator. The disclosed 3D exclusion zones may be defined and applied to the background and may generally go unnoticed unless inappropriate activity is detected.

[0046] 5, a method for defining a virtual 3D exclusion zone is generally referred to by the reference numeral 500. Method 500 includes scanning a local environment for a first work machine (block 510), identifying a second work machine within the local environment (block 520), generating a virtual 3D representation of the local environment and the second work machine (block 530), identifying one or more key structures of the second work machine (block 540), orienting the first work machine relative to the local environment (block 550), and generating a virtual 3D exclusion zone encompassing the key structures (block 560), each of which steps are described in more detail below. It will be appreciated that the 3D exclusion zone defines a space that should not be occupied by the first work machine, and that the first work machine, once defined, may be prevented from intersecting the 3D exclusion zone.

[0047] The method 500 may initially include scanning a local environment of a first work machine (block 510).

[0048] In some embodiments, the scanning step may include capturing a plurality of images using one or more stereo cameras located on the first work machine. The images may be captured continuously in real-time, at regular intervals, and / or whenever the FOV of the stereo cameras is changed.

[0049] In the same or other embodiments, the scanning step may be performed by a 3D scanning laser or similar 3D imaging device, such as a photometric camera. The 3D imaging device may capture data comparable to an optical image, such as, but not limited to, a matrix of TOF data and / or depth data within the FOV of the device. For purposes of this disclosure, a captured "image" may refer to any data recorded by the 3D imaging device, regardless of its type or format.

[0050] The captured images may be transmitted from the 3D imaging device to the image processing unit via one or more communication buses, and may be transmitted continuously and in real time, The frequency of image transmission may depend on the interval of capture or frame rate of the 3D imaging device, the bandwidth of the communication bus, the processing power of the image processing unit, and / or other specific application requirements.

[0051] At block 520, a second work machine, if present, may be identified within the local environment.

[0052] In a first embodiment, the second work machine may include fiducial markings indicative of the make, model, serial number, and / or other defining characteristics of the second work machine. The image processing unit may recognize the fiducial markings in the captured image using, for example, pattern recognition programs, character recognition programs, and / or similar machine vision programs known in the art, no limitation intended herein. It will be appreciated that the captured image may be initially processed by a number of image transformations intended to improve the accuracy and / or effectiveness of subsequent data manipulations, such as, but not limited to, high pass filtering, low pass filtering, convolution filtering, denoising, smoothing, sharpening, etc.

[0053] In a second embodiment, the second work machine may be identified by processing the captured image with a first neural network configured to identify the make, model, serial number, and / or other defining characteristics of the second work machine. For example, the captured image may be processed by a convolutional neural network (CNN) executed by the image processing unit to analyze the visual image and identify the second work machine in a database of work machines. Depending on the design of the neural network and the depth of the vehicle database, the second work machine may be identified by its type of machine, i.e., transport vehicle, haul truck, tractor trailer, etc., and / or by its make or model. The CNN may utilize any of a number of known neural network techniques and structures, and is not intended to be limited by this disclosure. In some embodiments, the captured image may be transformed before being processed by the CNN. In the same or other embodiments, the first neural network and / or the vehicle database may also be updated with newly captured images to improve future effectiveness.

[0054] With continued reference to FIG. 5, the method 500 may include generating a virtual 3D representation of the local environment and the second work machine (Block 530).

[0055] In a first embodiment in which the 3D imaging device includes a stereo camera, multiple disparity maps may be constructed from multiple captured stereo images. In the same or other embodiments, multiple depth maps may be constructed from the disparity maps and / or directly from the captured stereo images. The disparity maps and / or depth maps may then be stitched together to construct a 3D representation that may show some or all of the subject matter within the FOV of the 3D imaging device. In various embodiments, the 3D representation may be a 3D point cloud, a CAD model, a CAM model, a polygonal mesh, and / or other equivalent 3D data format.

[0056] In a second embodiment, where the 3D imaging device is a 3D scanning laser, each captured image may instead include a TOF matrix, a depth map, or an equivalent data format. Thus, the matrix may be converted to a depth map, and / or the matrix or depth map may be directly synthesized to construct the 3D representation by appropriate means. No limitations are intended on the manner in which the 3D representation is synthesized from the captured images, which may depend on the 3D imaging device, the type of its data, and other application requirements.

[0057] In some embodiments, the generated 3D representation of the local environment may be fused with an existing environment model (block 532). For example, the position of the first work machine may be known from a positioning unit and may be used as a key to search for an existing 3D model corresponding to said local environment. The existing 3D model may have been captured by an overhead drone scan, by a previous ground survey, by another work machine in the past, and / or by other possible sources, including but not limited to. The existing 3D model may be stored locally on the work machine and / or stored in an external database and downloaded via the communication unit. Block 532 may include accessing the existing 3D model to enhance, supplement, and / or improve the accuracy or resolution of the instant 3D representation. In some embodiments, the generation of the 3D representation by the first work machine may be entirely by default, and the 3D representation may be entirely provided from existing data. In other embodiments, the generated 3D representation (block 530) may be uploaded to a local and / or external database for future use.

[0058] In some embodiments, the generated 3D representation of the second work machine may be fused with an existing work machine model (block 534). For example, after the second work machine is identified, either by fiducial marking or determination by the first neural network, the make, model, and / or serial number of the second work machine may be used as a key to search for an existing 3D model of said work machine. The existing 3D model may be provided by the manufacturer, cataloged in a 3D scanning procedure, generated by another work machine in the past, and / or provided by other sources, without limitation. The existing 3D model may be stored locally on the work machine and / or an external database. Block 534 may include accessing the existing 3D model to enhance, supplement, and / or improve the accuracy and / or resolution of the instant 3D representation. In some embodiments, the generation of the 3D representation by the first work machine may be entirely by default, and the 3D representation may be provided entirely from existing data. In other embodiments, the generated 3D representation may be uploaded to a local and / or external database for future use.

[0059] 5, one or more primary structures of the second work machine, if present, may be identified (block 540). The primary structure may include some or all of the components of the second work machine and may further include one or more spaces or plenums relative to the second work machine, which will be discussed in more detail below.

[0060] In a first embodiment, the primary structures may already be identified in the existing work machine model that was fused according to block 534. In such a case, the primary structures may be obtained directly from the existing work machine model. In various embodiments, the primary structures may be identified as a collection of points, planes, surfaces, volumes, parts, components, assemblies, etc. in a 3D point cloud, a CAD file, a CAM file, a polygonal mesh, or equivalent data format.

[0061] In a second embodiment, the key structures may be identified by processing the plurality of captured images with a second neural network configured to identify key structures. For example, the captured images may be processed by a CNN executed by an image processing unit to analyze the visual images and identify the key structures among a universal key structure or a database of key structures associated only with the second work machine. The key structures may be identified by their form, function, texture, color, size, location, relationships, and / or other qualities, depending on the design of the neural network and the depth of the structure database. The CNN may utilize any of a number of known neural network techniques and structures, and is not intended to be limited by this disclosure. In some embodiments, the captured images may be transformed before being processed by the CNN. In the same or other embodiments, the second neural network and / or the structure database may also be updated with newly captured images to improve future effectiveness.

[0062] At block 550, the first work machine is oriented relative to a 3D representation of the local environment. In an embodiment, the position, elevation, azimuth orientation, and polar orientation may be obtained or calculated relative to the reference frame of the local environment. Additionally, the location or space occupied by the work machine and its components may be derived from a known schematic. Block 550 may utilize information from any combination of positioning units, implement sensors, 3D imagers, and even other sources, and is not intended to be limited by this disclosure.

[0063] At block 560, a virtual 3D exclusion zone is generated encompassing the primary structure of the second work machine. In an embodiment where the 3D representation is a 3D point cloud, the primary structure may first be defined as a number of "exclusion points" (block 562). The volume of the 3D exclusion zone may then be defined to include the number of exclusion points and additional tolerances (block 564), with the definition of specific boundaries and differences being dictated by application requirements. In other embodiments, the 3D exclusion zone may be mapped to the primary structure in other ways, such as by tracing or extrapolating individual points, segments, surfaces, vertices, edges, faces, polygons, parts, assemblies, etc. of the 3D representation, and no limitation is intended by this disclosure.

[0064] In some embodiments, a "deceleration zone" may be further defined as a set of points or as a volume surrounding the 3D exclusion zone (block 566). To prevent an intersection between the first work machine and the 3D exclusion zone, the first work machine may be slowed down in the deceleration zone accordingly.

[0065] In certain embodiments, the deceleration zone may be a volume defined by a distance to a surface of the 3D exclusion zone and / or a function of distance and other factors. In the same or other embodiments, the deceleration zone may be further divided into separate deceleration subzones based on a distance to a surface of the 3D exclusion zone and / or a function thereof. Also, in still other embodiments, other methods of defining the boundaries of the deceleration zone and / or deceleration subzones may be dictated by the requirements of a particular application and are not intended to be limited by the present disclosure.

[0066] It will be understood that the steps of method 500 may occur in a different order than presented, one or more steps may be omitted, one or more steps may be performed simultaneously, In some embodiments, method 500 may return to block 510 after generating the 3D exclusion zone, while in other embodiments, each step may be repeated continuously while the work machine is operating.

[0067] Referring now to FIG. 8, an exemplary first work machine, second work machine, main structure, and exclusion zone are shown. The first work machine is shown as a shovel, but may alternatively comprise any loading machine having an implement arm that is operable, i.e., movable, relative to the body of the machine. The second work machine is shown as a transport vehicle whose main structure includes a dump bed, a cabin, and a space above the cabin, but other machines, equipment, and / or main structures are possible and contemplated. The 3D exclusion zone is shown by a solid rectangle encompassing the main structure, and the deceleration zones and sub-zones are shown by dashed rectangles radiating from the exclusion zone. While 2D zones are shown, this is for illustrative purposes only, and it will be understood that equivalent 3D zones apply to the operation. By slowing the first work machine in the deceleration zone and preventing the first work machine from occupying the exclusion zone, the risk of collisions and falling debris can be substantially reduced.

[0068] Referring now to Figure 6, a first method of preventing collisions involving a work machine through application of a virtual 3D exclusion zone is generally referred to by the reference numeral 600. As shown in Figure 6, the method may include orienting the work machine relative to a local environment (block 610), defining a virtual 3D exclusion zone (block 620), monitoring motion of the work machine (block 630), predicting whether the motion will result in an intersection between the work machine and the 3D exclusion zone (block 640), and, if an intersection is predicted, adjusting the motion to avoid the intersection (block 650).

[0069] In block 610, the work machine is first oriented relative to the local environment. Similar to block 550 of method 500, position, elevation, azimuth orientation, and polar orientation may be obtained or calculated relative to the reference frame of the local environment. Additionally, the location or space occupied by the work machine and its components may be derived from known schematics. Block 610 may utilize information from any combination of positioning units, implement sensors, 3D imaging devices, and even other sources, and is not intended to be limited by this disclosure.

[0070] At block 620, a virtual 3D exclusion zone may be defined. The 3D exclusion zone may be defined according to an embodiment of method 500, although other methods are possible and contemplated.

[0071] At block 630, the motion of the work machine is monitored. The motion may be operatively monitored by the ECU using implement sensors. In some embodiments, the motion may specifically include a position in 3D space occupied by the implement arm of the work machine. In the same or other embodiments, the motion may include displacement and / or angular displacement vectors, velocity and / or angular velocity vectors, and acceleration and / or angular velocity vectors of the implement arm relative to the frame of reference of the machine and / or the local environment. Each vector may originate from a particular point within the body of the work machine or the local environment. No limitation to the frame of reference through which the vectors are interpreted is intended herein, provided that it remains consistent throughout the method 600. In some embodiments, the motion may further include jerk and / or angular jerk, and / or higher order displacement and / or angular displacement of the implement arm.

[0072] At block 640, the motion is predicted to result in an intersection or not between the work machine and the 3D exclusion zone. The prediction may be performed by the ECU and may utilize the motion of the implement arm and may further consider input received from the operator. If an intersection is not predicted, the method 600 may restart the process at block 610.

[0073] If an intersection is predicted between the work machine and the 3D exclusion zone, operation may be adjusted by the ECU to avoid the intersection (block 650), where various strategies may be employed.

[0074] In a first embodiment, the magnitude of the velocity vector may be slowed down. In the same or other embodiments, only the magnitude of the velocity vector in a direction perpendicular to the surface of the 3D exclusion zone may be slowed down. The strength of the slowdown may be a function of the magnitude of the velocity vector and / or a function of the distance between the work machine and the surface of the 3D exclusion zone.

[0075] In a second embodiment, the magnitude of the angular velocity vector may be slowed down. In the same or other embodiments, only the magnitude of the angular velocity vector in a direction parallel to the surface of the 3D exclusion zone may be slowed down. The magnitude of the slowdown may be a function of the magnitude of the angular velocity vector and / or a function of the distance between the work machine and the surface of the 3D exclusion zone.

[0076] In a third embodiment, the direction of the velocity vector may be reoriented, and in a fourth embodiment, the direction of the angular velocity vector may be reoriented. Additional embodiments may combine one or more of the above strategies, and additional strategies, to adjust the operation of the work machine. Each adjustment may be operatively commanded by the ECU and implemented by an implement actuator on the work machine.

[0077] It will be understood that the steps of method 600 may occur in a different order than presented, one or more steps may be omitted, and one or more steps may be performed simultaneously. In some embodiments, method 600 may return to block 610 after the operations have been adjusted, while in other embodiments, the steps may be repeated continuously while the work machine is operated.

[0078] 7, a second method of preventing collisions involving a work machine through application of a virtual 3D exclusion zone is generally referred to by the reference numeral 700. Similar to method 600, method 700 also includes orienting the work machine (block 710), defining a virtual 3D exclusion zone (block 730), and monitoring operation of the work machine (block 740). For the sake of brevity, the description of the above steps may be considered identical to the description of method 600 and will not be repeated herein. However, method 700 may further include defining a virtual deceleration zone (block 730) and detecting an intersection between the work machine and the deceleration zone (block 750), which will be discussed in more detail below.

[0079] At block 730, a virtual deceleration zone may be defined, and in some embodiments, individual deceleration subzones within the deceleration zone may be defined. The deceleration zone and / or deceleration subzones may be defined through the embodiments described in method 500, although other methods are possible and contemplated.

[0080] At block 750, an intersection between the work machine and the deceleration zone is detected. The detection may be performed by the ECU and may utilize the current position of the implement arm. If an intersection is not detected, the method 700 may restart the process from block 710.

[0081] If an intersection is detected between the work machine and the deceleration zone, the movement of the implement arm may be adjusted by the ECU to avoid a potential intersection between the implement arm and the 3D exclusion zone (block 760). Several strategies for adjusting the movement of the implement arm are described below.

[0082] In a first embodiment, the magnitude of the velocity vector may be slowed down. In the same or other embodiments, only the magnitude of the velocity vector in a direction perpendicular to the surface of the 3D exclusion zone may be slowed down. The intensity of the slowdown may be a function of the magnitude of the velocity vector and / or a function of the distance between the work machine and the surface of the 3D exclusion zone. In embodiments including separate slowdown subzones, the intensity of the slowdown may be determined by and / or a function of the particular subzone that is intersected.

[0083] In a second embodiment, the magnitude of the angular velocity vector may be slowed down. In the same or other embodiments, only the magnitude of the angular velocity vector in a direction parallel to the surface of the 3D exclusion zone may be slowed down. The magnitude of the slowdown may be a function of the magnitude of the angular velocity vector and / or a function of the distance between the work machine and the surface of the 3D exclusion zone. Also, in embodiments including separate slowdown subzones, the magnitude of the slowdown may be determined by and / or a function of the particular subzone that is intersected.

[0084] In other embodiments, one or more of the above strategies, and even additional strategies, may be combined to slow down the work machine at block 760. Each slowdown may be operatively commanded by the ECU and executed by an implement actuator on the work machine.

[0085] It will be understood that the steps of method 700 may occur in a different order than presented, one or more steps may be omitted, and one or more steps may be performed simultaneously. In some embodiments, method 700 may return to block 710 after the machine has been slowed down, while in other embodiments, each step may be repeated continuously while the work machine is operating.

[0086] Although the foregoing text has described detailed descriptions of many different embodiments, it should be understood that the legal scope of protection is defined by the language of the claims at the end of this patent. The detailed description should be construed as merely exemplary and does not describe all possible embodiments, as describing all possible embodiments would be impractical, if not impossible. Numerous alternative embodiments may be implemented using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims that define the scope of protection.

Claims

1. A method for defining a virtual 3D exclusion zone (99), comprising: Scanning a local environment (80) of a first work machine (10); identifying a second work machine (82A) within the local environment (80); generating a virtual 3D representation (93) of said local environment (80) and said second work machine (82A); identifying one or more primary structures (84) of said second work machine (82A); orienting the first work machine (10) with respect to the local environment (80); generating a virtual 3D exclusion zone (99) encompassing said primary structure (84), said 3D exclusion zone (99) defining a space that should not be occupied by said first work machine (10).

2. The step of scanning the local environment (80) comprises:

2. The method of claim 1, further comprising capturing a plurality of images (91) using one or more stereo cameras (41) located on the first work machine (10).

3. said step of identifying said second work machine (82A) comprising: recognizing fiducial markings on the second work machine (82A); and / or 3. The method of claim 2, further comprising processing the plurality of images (91) with a first neural network configured to identify a model of the second work machine (82A).

4. said step of generating said virtual 3D representation (93) comprising: constructing a plurality of disparity maps from said plurality of images (91); and / or The method of claim 2, further comprising constructing a 3D point cloud representing the local environment (80) and the second work machine (82A).

5. said step of generating said virtual 3D exclusion zone (99) comprising: defining said primary structure (84) as a plurality of excluded points in said 3D point cloud; defining a volume of the 3D exclusion zone (99) based on the plurality of exclusion points and a tolerance; and 5. The method of claim 4, further comprising: defining a deceleration zone surrounding the 3D exclusion zone (99), wherein the first work machine (10) is decelerated within the deceleration zone.

6. The step of identifying the primary structure (84) comprises: said primary structure (84) being obtained from an existing work machine model; and / or 3. The method of claim 2, further comprising processing the plurality of images (91) with a second neural network configured to identify the primary structure (84) of the second work machine (82A).

7. 1. A method of preventing collisions involving a work machine (10) by application of a virtual 3D exclusion zone (99), comprising: orienting the work machine (10) with respect to a local environment (80); Defining a virtual 3D exclusion zone (99); monitoring operation of the work machine (10); predicting whether said movement will result in an intersection between said work machine (10) and said 3D exclusion zone (99); and if a crossing is predicted, adjusting said operation to avoid said crossing.

8. 8. The method of claim 7, wherein the motion includes a position, a displacement vector and / or an angular displacement vector, a velocity vector and / or an angular velocity vector, and an acceleration vector and / or an angular velocity vector of an implement arm (20) of the work machine.

9. defining a virtual deceleration zone surrounding said 3D exclusion zone (99); detecting an intersection between the work machine (10) and the deceleration zone; and / or reducing the magnitude of the velocity vector perpendicular to the surface of the exclusion zone when the work machine (10) crosses the deceleration zone; and / or 9. The method of claim 8, further comprising the step of: decelerating a magnitude of the angular velocity vector parallel to the surface of the exclusion zone when the work machine (10) intersects the deceleration zone.

10. said 3D exclusion zone (99) comprising a first volume encompassing a dump bed (841) of a transport vehicle (82A) and a second volume encompassing a cabin (842) of said transport vehicle (82A) and a space above said cabin (843); 8. The method of claim 7, wherein each of the method steps is repeated continuously while the work machine (10) is operated.

Images