Object Detection Vision System

Abstract

An object detection vision system and method is disclosed. A method (300) for detecting objects in a vision system of an industrial machine includes receiving image data (110) from one or more vision cameras (40) and receiving detection data (112) from one or more detection devices (50). The detection data (112) includes one or more detected objects. The method includes combining the detection data (112) with the image data (110) and transforming the detection data (112) in the image data (110) based on the one or more objects in the image data (110). The method (300) also includes displaying a representation (120) of the one or more objects detected in the image data (110) based on the transformed detection data (112).

Description

[Technical field]

[0001] The present disclosure relates generally to vision systems for industrial machines, and more particularly to object detection vision systems for such industrial machines. [Background technology]

[0002] Various industrial machines, such as those used to perform excavation, loosening, conveying, compaction, or other earthmoving operations, may be equipped with a vision system having one or more cameras. The vision system may provide the operator of such industrial machines with various views of the environment around the industrial machine and / or a complete peripheral view of the environment (e.g., a 3D view). Such vision systems may utilize a typical video camera to provide a video image for each view. However, a typical video camera only has the ability to visually provide a representation of the environment, and does not have the ability to detect objects around the environment near the industrial machine or provide a representation of such objects. Object detection and object classification may be useful when an object is difficult to see on the display from the video camera. Furthermore, automatic object detection and object classification capabilities may enable autonomous operation of the industrial machine.

[0003] Replacing or upgrading a typical video camera with an object detection device (e.g., a detection camera capable of object detection) may require complex software and hardware updates on the respective industrial machine, requiring unnecessary downtime of the industrial machine from operation. Thus, operation and / or maintenance costs may increase due to the need to replace a typical video camera with such a detection device. Furthermore, it may be difficult to correctly map object detection information from the object detection device onto the displayed image provided by the typical video camera.

[0004] U.S. Patent Application Publication No. 20210181351 (the "'351 Publication"), published on June 17, 2021, describes methods and apparatus related to autonomous driving. The method of the '351 Publication includes detecting an object in a received camera image and detecting the same object in a received LIDAR image. The method further includes combining the detected object in the camera image with the detected object in the LIDAR image to display two bounding boxes in the camera image (e.g., one from the camera object detection and one from the LIDAR object detection). However, the method and apparatus of the '351 Publication do not address providing object detection in existing vision systems having cameras that do not have object detection capabilities.

[0005] The object detection vision system of the present disclosure may solve one or more of the problems set forth above and / or other problems in the art, however, the scope of the disclosure is defined by the appended claims, and not by the ability to solve any particular problem. Summary of the Invention

[0006] In one aspect, a method for detecting objects in a vision system of an industrial machine is disclosed, the method including receiving image data from one or more vision cameras, receiving detection data from one or more detection devices, the detection data including one or more detected objects, combining the detection data and the image data, transforming the detection data in the image data based on the one or more objects in the image data, and displaying a representation of the one or more objects detected in the image data based on the transformed detection data.

[0007] In another aspect, an object detection vision system is disclosed that includes one or more visual cameras, one or more detection devices, and a controller configured to receive image data from the one or more visual cameras, receive detection data from the one or more detection devices, the detection data including one or more detected objects, combine the detection data and the image data, transform the detection data in the image data based on the one or more objects in the image data, and display a representation of the one or more objects detected in the image data based on the transformed detection data.

[0008] In yet another aspect, a method for detecting objects in a vision system of an industrial machine is disclosed that includes receiving image data from one or more vision cameras, receiving detection data from one or more object detection cameras having one or more processors for detecting objects, the detection data including first coordinates of one or more detected objects, combining the first coordinates with the image data, transforming the first coordinates in the image data based on the one or more objects in the image data to generate second coordinates of the one or more detected objects in the image data, and displaying a representation of the one or more objects detected in the image data based on the second coordinates of the one or more detected objects. [Brief description of the drawings]

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0010] [Figure 1] FIG. 1 is a schematic diagram of an industrial machine having an object detection vision system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of an example object detection vision system for the industrial machine of FIG. [Diagram 3]FIG. 3 provides a flow chart illustrating an exemplary method for detecting an object in the object detection vision system of FIGS. [Figure 4] FIG. 4 provides a flow chart illustrating an exemplary method for calibrating the object detection vision system of FIGS. [Figure 5A] 5A-5C illustrate various fields of view of the object detection vision system of FIGS. 1 and 2. [Figure 5B] Same as above. [Figure 5C] Same as above. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Both the general description above and the detailed description below are merely exemplary and explanatory and do not limit the claimed features. As used herein, "comprises," "comprising," "has," "having," "includes," "including," or other variations thereof are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that includes a list of elements does not merely include those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. In this disclosure, unless otherwise stated, relative terms such as "about," "substantially," and "approximately" are used to indicate a possible variation of ±10% of the stated value.

[0012] FIG. 1 illustrates a schematic diagram of an industrial machine 10 having an object detection vision system 100 according to an embodiment of the present disclosure. Although the disclosure herein may be applicable to any type of industrial machine or vehicle, the following refers specifically to an excavator. For example, the industrial machine 10 may include, but is not limited to, an off-highway truck, a bulldozer, an articulated dolly, a scraper, a compactor, a paver, a wheel loader, a motor grader, or any other type of industrial machine. As shown in FIG. 1, the industrial machine 10 may include a frame 12 and a machine 14. The frame 12 generally supports various assemblies and mechanical systems of the industrial machine 10 and may be supported on the ground by one or more trucks 16 or a transport mechanism such as wheels. The frame 12 may support the machine 14, which may include an engine, a motor, a generator, a battery, a pump, an air compressor, a hydraulic fluid storage tank, and / or any other equipment necessary to power and operate the industrial machine 10. The industrial machine 10 may also include implements such as an excavation tool, a forklift, a blade, a dump bed, or any other device for performing a task.

[0013] Frame 12 may further support an operator cab 18, in which a user or operator may operate and control the industrial machine 10 via one or more operator interfaces 20, such as user interfaces, controls, and / or displays 22. Operator interfaces 20 may include a steering device (e.g., a steering wheel and / or joystick) and one or more displays 22, such as a touch screen display device, a keypad with buttons, or the like. It is understood that aspects of the operator interface 20 and displays 22 may be located remotely from the industrial machine 10.

[0014] The industrial machine 10 further includes an object detection vision system 100 for enhancing awareness around the industrial machine 10 and, if desired, for controlling aspects of the industrial machine 10. The object detection vision system 100 may include a controller 104, such as an electronic control module (ECM), one or more vision cameras 40, one or more detection devices 50, and one or more of the operator interface 20 and display 22. The one or more vision cameras 40, the one or more detection devices 50, the operator interface 20 and the display 22 may communicate with the controller 104 (as shown in FIG. 1 by the dashed lines) via wired communication lines and / or wireless means.

[0015] The one or more visual cameras 40 may include any type of video camera or the like for capturing image data of the environment around the industrial machine 10. The one or more visual cameras 40 may be attached to or mounted on any portion of the industrial machine 10. As used herein, "camera" generally refers to a device configured to capture and transmit image data, such as still images, video streams, time lapse sequences, or the like. The one or more visual cameras 40 may include a camera without object detection capabilities. The one or more visual cameras 40 may include a monocular or stereo digital camera, a high resolution digital camera, or any suitable digital camera. For example, the one or more visual cameras 40 may include a color monocular digital camera or a stereo digital camera (e.g., a camera set including two monochrome camera imagers and a color monocular imager). The one or more cameras 40 may transmit or forward the captured image data to the controller 104, as described in further detail below. The camera 40 may capture two-dimensional (2D) images and may not provide range or object detection information in the image data. For example, the camera 40 may not have object detection capabilities. In some cases, the camera 40 may capture a complete peripheral view of the industrial machine 10's environment. Thus, the camera 40 may have a horizontal field of view (FOV) of 360 degrees. Although only two visual cameras 40 are shown in FIG. 1, it is understood that the industrial machine 10 may include any number of cameras 40 arranged on the industrial machine 10 in any manner or position. The camera 40 may include intrinsic and extrinsic properties. For example, the intrinsic properties may include internal parameters of the camera 40, such as focal length, optical center, pixel azimuth angle, pixel elevation angle, or any other internal parameter associated with the camera 40. The extrinsic parameters may include external parameters of the camera 40, such as roll, pitch, yaw, angle of depression relative to ground level, horizontal position, vertical position, or any other external parameter associated with the camera 40.

[0016] The one or more detection devices 50 may include any type of object detection device, such as a detection camera (including monocular and / or stereo cameras), optical sensor, radar sensor, LIDAR sensor, sonar sensor, etc., for scanning or otherwise detecting one or more objects in front of and / or around the industrial machine 10. The objects may include, for example, a person, an animal, a piece of equipment, another vehicle, and / or any other type of object in the environment around the industrial machine 10. The detection devices 50 may be attached or mounted to any portion of the industrial machine 10 at a location generally corresponding to the location of the vision camera 40. For example, each detection device 50 may be positioned substantially adjacent to a vision camera 40 such that the vision camera 40 and the corresponding detection device 50 include substantially similar but different fields of view (FOV). The detection devices 50 may include substantially similar but different extrinsic parameters as the vision camera 40 (e.g., the detection devices 50 may include slightly different roll, pitch, yaw, dip angle, horizontal position, and / or vertical position than the vision camera 40). The intrinsic parameters of the detector 50 may be different from the intrinsic parameters of the visual camera 40. For example, the focal length, optical center, pixel azimuth angle, and pixel elevation angle of the detector 50 may be different from the focal length, optical center, pixel azimuth angle, and pixel elevation angle of the visual camera 40.

[0017] The detection device 50 may include any type of object detection device (e.g., monocular and / or stereo detection cameras, optical sensors, radar sensors, LIDAR sensors, sonar sensors, etc.), but refers specifically to a detection camera. As used herein, a "detection camera" may include a device having a processor (not shown), such as a field programmable gate array (FPGA), image signal processor, or the like, configured to capture detection data (e.g., image data) and detect one or more objects present in the detection data (e.g., within the FOV of the detection camera). The detection camera may generate three-dimensional (3D) coordinates of one or more objects in the detection data. For example, the processor of the detection device 50 may use machine learning and / or artificial intelligence (e.g., neural networks or the like) to generate a 2D bounding box using known detection algorithms, identify objects within the bounding box, and calculate the 3D position and / or velocity of the objects. Thus, the detection device 50 may calculate the 3D coordinates of the center of the object in coordinate space (e.g., the center of a bounding box) based on the extrinsic and intrinsic parameters of the detection camera 50. The detection device 50 may transmit the 3D coordinates of the object to the controller 104, as described in further detail below. Of course, the detection device 50 may detect the object and generate the 3D coordinates by any means known in the art.

[0018] Display 22 may include one or more devices used to present the output of controller 104 to an operator of industrial machine 10. Display 22 may include a single screen display, such as an LCD display device, or a multi-screen display. Display 22 may include multiple displays for displaying different content (e.g., image data from different cameras 40) on the same display or on different displays 22. Display 22 may communicate with controller 104 via wired communication and / or wireless means (as shown by the dashed lines in FIG. 1) and may be located on-board industrial machine 10 and / or remote from industrial machine 10. As described in more detail below, display 22 may be used to display a representation of the environment around industrial machine 10 based on image data captured by camera 40, and may display a representation of detected objects 504, as described in more detail below (see FIGS. 5A-5C). Display 22 may include a touch-sensitive screen and thus have the ability to input data (e.g., from user input) and record information.

[0019] 2 is a schematic diagram of an example object detection vision system 100 for operating and / or controlling at least a portion of an industrial machine 10. The object detection vision system 100 may include an input 102, a controller 104, and an output 106. The input 102 may include image data 110 from one or more vision cameras 40, detection data 112 from one or more detection devices 50, and / or user input 114 from an operator interface 20 or display 22. The image data 110 may include image data captured by the vision cameras 40. The detection data 112 may include 3D coordinates of detected objects detected and generated by the detection devices 50. The user input 114 may include selections received from a user via the operator interface 20 or display 22 to convert the 3D coordinates from the detection devices 50 into 3D coordinates of objects in the image data 110 from the vision cameras 40, as described in further detail below.

[0020] The output 106 may include, for example, an indication 120 of the detected object. The controller 104 also includes an object detection module 108. The object detection module 108 may receive the input 102, implement a method 300 for detecting an object in a vision system and / or a method 400 for calibrating the object detection vision system 100, and control the output 106, as described with reference to Figures 3 and 4 below.

[0021] The controller 104 may embody a single microprocessor or multiple microprocessors, which may include means for detecting objects in the vision system of the industrial machine 10. For example, the controller 104 may include a memory, secondary storage, and a processor, such as a central processing unit or any other means for accomplishing tasks consistent with the present disclosure. The memory or secondary storage associated with the controller 104 may store data and / or software routines that may assist the controller 104 in performing its functions, such as the functions of the method 300 of FIG. 3 and the method 400 of FIG. 4. Additionally, the memory or secondary storage associated with the controller 104 may also store data received from the various inputs 102 associated with the object detection vision system 100. Many commercially available microprocessors may be configured to perform the functions of the controller 104. Of course, the controller 104 may easily embody a general machine controller capable of controlling many other machine functions. Alternatively, a dedicated machine controller may be provided. Additionally, the controller 104, or portions thereof, may be located remotely from the industrial machine 10. Various other known circuits may be associated with the controller 104, including signal conditioning circuits, communication circuits, hydraulic or other actuation circuits, and other suitable circuits.

[0022] The controller 104 may also include stored values ​​for use by the module 108. For example, the stored values ​​may include coordinates of detected objects from the detection device 50, coordinate transformations, and configuration files. The coordinates of detected objects may be updated as the industrial machine 10 and / or objects move. The coordinate transformations may include a transformation of coordinates from the detection device 50 to coordinates in the image data 110 from the vision camera 40, as described in further detail below. The configuration files may include stored calibrations of coordinate transformations such that the module 108 may transform coordinates of newly detected objects into the coordinate system in the image data 110 to provide a representation 120 of detected objects in the image data 110 via the display 22, as described in further detail below.

[0023] The representation 120 of the detected object may include a representation (e.g., a highlight, a bounding box, etc.) of the detected object that is displayed in the image data 110 via the display 22, as described in further detail below. The controller 104 may derive the representation 120 based on the coordinate transformation and may transmit the representation 120 to the display 22 for displaying the representation 120 on the display 22, as described in further detail below. [Industrial Applicability]

[0024] The disclosed embodiments of the object detection vision system 100 of the present disclosure may be used with any type of industrial machine 10 having a vision system having one or more vision cameras 40 and one or more detection devices 50.

[0025] With reference to FIG. 1, during operation of the industrial machine 10, an operator may control the industrial machine 10 to maneuver around a surface to perform tasks. The vision camera 40 may provide various views of the environment around the industrial machine 10 on the display 22 as the industrial machine 10 operates. However, the vision camera 40 itself cannot detect objects in the environment around the industrial machine 10. Furthermore, replacing the vision camera 40 with a detection device 50 may require a complete software and / or hardware overhaul and therefore unnecessarily long downtime of the particular industrial machine 10, which may lead to increased costs. Thus, as described in more detail below with reference to FIGS. 3 and 4, the object detection vision system 100 may detect objects using the detection device 50 and provide an indication of the detected objects in image data 110 from the vision camera 40.

[0026] 3 shows a flow chart illustrating an example method 300 for detecting objects in the vision system 100 of the industrial machine 10. In step 305, the module 108 may receive image data 110 from the vision cameras 40. For example, the module 108 may receive video streams from one or more vision cameras 40 and display the image data 110 on the display 22.

[0027] In step 310, the module 108 may receive detection data 112 from the detection device 50. In step 315, the module 108 may detect an object based on the detection data 112. Object detection may be performed by any means known in the art, as detailed above. With reference to FIG. 5A, an exemplary FOV 500 of the detection device 50 is shown. The FOV 500 is not displayed on the display 22 because the detection device 50 is not in direct communication with the display 22. Thus, the FOV 500 of FIG. 5A is for illustrative purposes only. FIG. 5A shows multiple detected objects 504 around the work site in the FOV 500 of the detection device 50, and a display 506 of the detected objects 504. For clarity, only a single object 504 and a single display 506 are referenced in FIG. 5A. However, it is understood that the detection device 50 may detect any number of objects within the FOV 500 of each detection device 50. The FOV 500 in Figure 5A also illustrates an exemplary grid or coordinate system for the detection device 50. Although a square coordinate system is illustrated in Figure 5A, it will be understood that any type of coordinate system or grid may be used, such as, for example, a triangular grid or the like.

[0028] In step 320, the module 108 may generate first coordinates of the detected object. For example, the module 108 may generate 3D coordinates of the detected object in the coordinate system of the detection device 50. In some examples, the received detection data 112 may include the generated first coordinates of the detected object. For example, the detection device 50 may detect an object and generate first coordinates of the detected object before transmitting the detection data 112 to the module 108. Thus, steps 315 and 320 may be optional steps in that the module 108 does not separately detect the object and does not generate first coordinates of the detected object.

[0029] In step 325, the module 108 may combine the detection data (e.g., the generated first coordinates) with the image data 110. For example, the module 108 may overlay or map the generated first coordinates of the detected object 504 from the detection data 112 onto the image data 110 from the vision camera 40. In some examples, the module 108 may map the image data onto the first coordinates. With reference to FIG. 5B, a display of an exemplary FOV 502 of the vision camera 40 is shown. The FOV 502 may be provided by the image data 110 from the vision camera 40 and may be displayed on the display 22 as detailed above. The example of FIG. 5B shows multiple objects 504 around the work site that are not detected or otherwise shown within the FOV 502 of FIG. 5B. The first coordinates of the detected object 504 from the detection device 50 on the FOV 502 of the vision camera 40 may include the coordinate system of the detection device 50. As shown in Figure 5B, the objects 504 may be offset from the objects 504 in the FOV 500 of Figure 5A due to differences in extrinsic and intrinsic parameters between the vision camera 40 and the detection device 50. Thus, in Figure 5B, a first coordinate of a representation 506 of the detected object 504 from the detection device 50 may not align with the object 504 in the image data 110.

[0030] In step 330, the module 108 may determine whether a configuration file has been generated and saved. As detailed above, the configuration file may include a saved coordinate transformation. The coordinate transformation may include a mesh transformation of a first coordinate from the detection device 50 to a second coordinate in the coordinate system of the FOV 502 of the vision camera 40 based on extrinsic and inherent differences between the detection device 50 and the vision camera 40. For example, the saved coordinate transformation may include a transformation of the coordinate system of the detection device 50 to the coordinate system of the vision camera 40 to align a first coordinate of a representation 506 of the detected object 504 with the object 504 in the image data 110 from the vision camera 40 to generate a second coordinate of a representation 508 of the detected object 504.

[0031] Thus, upon determining that there is a saved configuration file (e.g., a saved coordinate transformation) (step 330: yes), the module 108 may transform the first coordinates to the object 504 in the image data 110 based on the configuration file (step 335). The module 108 may generate a representation 508 of the detected object 504 in the image data 110 based on the transformed second coordinates (as shown in FIG. 5C). In step 340, the module 108 may display the image data 110 having the representation 508 of the detected object 504 on one or more displays 22. As detailed above, the representation 508 may include a highlight, a bounding box, a point, or any other representation 508 for showing the detected object 504 on the display 22. As shown in FIG. 5C, the coordinate system or grid has been transformed from the coordinate system of the detection device 50 by the saved coordinate transformation.

[0032] During the initial calibration, the module 108 may generate a configuration file. For example, if a configuration file has not been saved (step 330: NO), the module 108 may perform a method 400 for calibrating the object detection vision system 100.

[0033] 4 provides a flowchart illustrating an example method 400 for calibrating an object detection vision system 100. In step 405, the module 108 may receive user input. For example, a user may interact with a touch screen display 22 (or other input device) to drag or otherwise move a representation 506 to align the representation 506 with an object 504 in image data 110 displayed on one or more displays 22.

[0034] In step 410, module 108 may transform first coordinates to objects 504 in image data 110 based on user input. For example, as the user moves through each representation 506, module 108 may transform the first coordinates of representation 506 to generate transformed representations 508 that transform second coordinates in the transformed coordinate system (as shown in FIG. 5C ).

[0035] In step 415, module 108 may then generate a configuration file based on the transformed second coordinates and save the configuration file for use as detailed above. Module 108 may then display the image data 110 with a representation 508 of the detected object 504 (step 340).

[0036] Module 108 may then continue to receive image data 110 from vision camera 40 (step 305) and detect data from detection device 50 (step 310) to detect new objects. Thus, module 108 may continually perform method 300 to display representations 508 of newly detected objects 504 within FOV 500 and FOV 502. Newly detected objects 504 may include new objects within FOV 500 and FOV 502 and / or may include the same detected objects 504 that have moved relative to FOV 500 and FOV 502 (e.g., because object 504 itself moves and / or because industrial machine 10 moves).

[0037] The object detection vision system 100 of the present disclosure may provide an improved object detection system for industrial machines 10 having an existing vision system with one or more visual cameras 40. For example, the object detection vision system 100 may enable the installation of a detection device 50 without requiring a complex and complete update of the on-board software of the industrial machine 10. Furthermore, the object detection vision system 100 may more accurately display a representation of a detected object on the display 22 of the existing vision system compared to prior art systems with a dedicated detection device 50. Thus, the object detection vision system 100 may provide an aftermarket system that may be relatively easily installed on industrial machines 10 having an existing vision system with a visual camera 40 to enable object detection on such industrial machines 10. Thus, the object detection vision system 100 may reduce unnecessary downtime of the industrial machine 10, thus saving maintenance and installation costs associated with installing a detection device 50 on the industrial machine 10. Furthermore, the object detection vision system 100 may enable object detection and object classification in the existing vision system of the industrial machine 10 and may provide a specific location of the detected object in the existing display 22 from the visual camera 40. Thus, the object detection vision system 100 may enable operators to complete their tasks more efficiently and quickly and / or may enable autonomous operation of the industrial machine 10.

[0038] It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the scope of the disclosure. Other embodiments of the system will be apparent to those skilled in the art from consideration of the specification and practice of the system disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope of the disclosure being indicated by the following claims and their equivalents.

Claims

1. A method (300) for detecting an object in a vision system (100) of an industrial machine (10), comprising: receiving image data (110) from one or more visual cameras (40); receiving detection data (112) from one or more detection devices (50), the detection data (112) including one or more detected objects (504); combining the detection data (112) with the image data (110); transforming the detected data (112) in the image data (110) based on one or more objects in the image data (110); and displaying a representation (120, 508) of the one or more detected objects within the image data (110) based on the transformed detection data (112).

2. The method (300) of claim 1, wherein the one or more detection devices (50) include one or more processors for detecting objects.

3. The method (300) of claim 1, wherein the one or more visual cameras (40) include one or more cameras without object detection capabilities.

4. The method (300) of any of claims 1 to 3, wherein the detection data (112) comprises first coordinates of the one or more detected objects (504).

5. 5. The method (300) of claim 4, further comprising combining the first coordinates with the image data (110).

6. 6. The method (300) of claim 5, further comprising: transforming the first coordinates to an object in the image data (110) to generate second coordinates of the detected object (504) in the image data (110).

7. 7. The method of claim 6, wherein transforming the first coordinates comprises transforming the first coordinates based on a stored coordinate transformation.

8. Transforming the first coordinates comprises: receiving a user input (114) that moves the first coordinate within the image data (110); and converting the first coordinates to second coordinates based on the user input.

9. one or more visual cameras (40); one or more detection devices (50); A controller (104), receiving image data (110) from one or more visual cameras (40); receiving detection data (112) from one or more detection devices (50), the detection data (112) including one or more detected objects (504); combining the detection data (112) with the image data (110); transforming the detected data (112) in the image data (110) based on one or more objects in the image data (110); and a controller (104) configured to display a representation (120) of the one or more objects detected in the image data (110) based on the converted detection data (112).

10. 10. The object detection vision system (100) of claim 9, wherein the one or more vision cameras (40) include one or more cameras without object detection capabilities.

Images