Sensor fusion-based object detection system and method, and construction equipment including same system

Abstract

This sensor fusion-based object detection system may comprise: a sensor device including a first sensor for outputting an image, a second sensor for outputting the location and speed of at least one object, and a third sensor for outputting equipment position information of construction equipment; and a processing device for generating a driving control signal or a turning control signal of the construction equipment.

Description

Sensor fusion-based object detection system and method and construction equipment including the system

[0001] Various embodiments of the present disclosure relate to a sensor fusion-based object detection system and method and construction equipment including the system.

[0002]

[0003] Recently, various safety solutions for construction equipment, such as wheel loaders and excavators, have been released to prevent collisions between construction equipment.

[0004] Various construction equipment manufacturers are also developing or mass-producing collision avoidance features, primarily utilizing cameras or radar sensors for object detection.

[0005] Effectively implementing a construction equipment collision avoidance safety solution requires a function that minimizes equipment control and prevents collisions based on accurate information about hazardous objects in a complex and constantly changing construction equipment environment. Otherwise, frequent misdetection or over-control of equipment can lead to inconvenience in equipment operation and a significant decrease in work productivity.

[0006] Accurate risk object information must include the classification of risk objects that must avoid collision among various objects around the equipment, as well as the exact location information of the detected risk objects.

[0007] Radar sensors can detect the exact distance to the detected object, but cannot classify the object. Camera sensors can classify objects, but can only detect the approximate distance to the detected object.

[0008]

[0009] The present disclosure provides a sensor fusion-based object detection system and method, and construction equipment including the system, that estimates relatively robust and accurate location information in various equipment postures as well as detects / classifies objects around the equipment through AI technology and sensor fusion of cameras and radar sensors, in order to implement a more effective collision avoidance safety solution for construction equipment.

[0010]

[0011] According to one embodiment of the present invention, a sensor fusion-based object detection system may include a sensor device including a first sensor outputting an image, a second sensor outputting a position and a velocity of at least one object, and a third sensor outputting attitude information of construction equipment; and a processing device generating a driving control signal or a turning control signal of the construction equipment, wherein the processing device may include an object determination unit that determines a dangerous object from the image and outputs pixel information of the dangerous object; a coordinate conversion unit that converts the pixel information of the dangerous object into a position on radar space coordinates; a matching unit that receives a position on radar space coordinates of the dangerous object and receives a position on radar space coordinates of the at least one object from the second sensor, and matches the dangerous object with the at least one object; a filtering unit that estimates precise distance information of the dangerous object based on a position and a velocity of an object matched with the dangerous object among the at least one object; and a control signal generation unit that generates the driving control signal or the turning control signal of the construction equipment based on the precise distance information of the dangerous object.

[0012] The first sensor may include a camera sensor, the second sensor may include a radar sensor, and the third sensor may include an IMU (Inertial Measurement Unit) sensor.

[0013] The above-mentioned posture information may include roll values, pitch values, and yaw values ​​indicating the degree of inclination for each axis.

[0014] The object identification unit includes a deep learning-based model trained to identify a specific object, including a pedestrian with which the construction equipment should not collide, as the dangerous object, and pixel information of the dangerous object may include the coordinates of the width, height, and center point of the dangerous object on the image.

[0015] The above coordinate transformation unit can primarily correct pixel information of the dangerous object based on the detailed information of the construction equipment, and secondarily correct camera distortion included in the pixel information of the dangerous object based on the lens information of the camera.

[0016] The above matching unit can match the dangerous object and the at least one object using the object location output from the camera sensor, the object location output from the radar sensor, and the probability distribution of the object location.

[0017] The above matching part is a mathematical formula (Here, is the position vector of the hazardous object i, is the average vector of the position probability distribution of object j output from the radar sensor, A sensor fusion-based object detection system that calculates the distance between the dangerous object and each of the at least one object using a covariance matrix of object positions output from the radar sensor, and matches the dangerous object with the object with the shortest distance among the at least one object.

[0018] The above filtering unit can be implemented using a Kalman filter-based filtering algorithm.

[0019] According to one embodiment of the present invention, construction equipment includes a sensor device including a camera that outputs an image, a radar sensor that outputs a position and a velocity of at least one object, and an IMU sensor that outputs attitude information of the construction equipment; and a processor that generates a driving control signal or a turning control signal of the construction equipment, wherein the processor is configured to determine a dangerous object from the image, obtain pixel information of the determined dangerous object, convert the pixel information of the dangerous object into a position on radar space coordinates, receive a position on radar space coordinates of the at least one object from the radar sensor, match the dangerous object with the at least one object, estimate precise distance information of the dangerous object based on a position and a velocity of an object matched with the dangerous object among the at least one object, and generate the driving control signal or the turning control signal of the construction equipment based on the precise distance information of the dangerous object.

[0020] According to one embodiment of the present invention, a sensor fusion-based object detection method by construction equipment may include a step of obtaining an image including an object using a camera sensor, a step of obtaining a position and a speed of at least one object using a radar sensor, a step of obtaining attitude information of the construction equipment using an IMU sensor, a step of determining a dangerous object from the image and obtaining pixel information of the determined dangerous object, a step of converting the pixel information of the dangerous object into a position on radar space coordinates, a step of matching the dangerous object with the at least one object, a step of estimating precise distance information of the dangerous object based on a position and a speed of an object matched to the dangerous object among the at least one object, and a step of generating the driving control signal or the turning control signal of the construction equipment based on the precise distance information of the dangerous object.

[0021] The step of determining the above-mentioned dangerous object includes a step of determining the dangerous object using a deep learning-based model learned to identify a specific object including a pedestrian with which the construction equipment should not collide as the dangerous object, and the pixel information of the dangerous object may include the coordinates of the width, height, and center point of the dangerous object on the image.

[0022] The sensor fusion-based object detection method may further include a step of primarily correcting pixel information of the dangerous object based on the posture information of the construction equipment, and a step of secondarily correcting camera distortion included in the pixel information of the dangerous object based on lens information of the camera.

[0023] The step of matching the risk object with the at least one object may include the step of matching the risk object with the at least one object using the object location output from the radar sensor and the probability distribution of the object location.

[0024] The step of matching the dangerous object and the at least one object using the object location output from the radar sensor and the probability distribution of the object location is as follows: (Here, is the position vector of the hazardous object i, is the average vector of the position probability distribution of object j output from the radar sensor, The method may include a step of calculating a distance between the dangerous object and each of the at least one object using a covariance matrix of object positions output from the radar sensor, and a step of matching the dangerous object with an object having the shortest distance among the at least one object.

[0025] The step of estimating precise distance information of the above-mentioned risk object may include a step of estimating precise distance information of the above-mentioned risk object using a Kalman filter-based filtering algorithm.

[0026]

[0027] According to one embodiment of the present invention, by applying AI technology and sensor fusion technology, even in complex surrounding environments and when the equipment shakes greatly due to the characteristics of construction equipment operation, information on objects surrounding the equipment can be more accurately estimated for precise control to prevent collisions.

[0028]

[0029] FIG. 1 is a diagram illustrating an autonomous operation system according to various embodiments of the present disclosure.

[0030] FIG. 2 is a drawing for explaining an excavator according to various embodiments of the present disclosure.

[0031] FIG. 3 is a drawing showing a functional block diagram of construction equipment according to various embodiments of the present disclosure.

[0032] FIG. 4 is a functional block diagram of an object detection system based on construction equipment sensor fusion according to various embodiments of the present disclosure.

[0033] Fig. 5 is a drawing for explaining the operation of the object determination unit of Fig. 4.

[0034] Fig. 6 is a drawing for explaining the operation of the matching unit of Fig. 4.

[0035] FIG. 7 is a flowchart illustrating an object detection method based on construction equipment sensor fusion according to various embodiments of the present disclosure.

[0036]

[0037] The advantages and features of the present disclosure, as well as the devices and methods for achieving them, will become clearer with reference to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure is complete and to fully inform those skilled in the art of the scope of the disclosure, and the present disclosure is defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

[0038] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0039] The terminology used herein is for the purpose of describing embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular also includes the plural unless the context clearly dictates otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0040] Although the terms first, second, etc. are used to describe various components, these components are not limited by these terms. These terms are used only to distinguish one component from another.

[0041] Accordingly, it should be understood that the first component mentioned below may also be a second component within the technical spirit of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0042] The term 'part' or 'module' used in this embodiment means a software or hardware component such as an FPGA or ASIC, and the 'part' or 'module' performs certain roles. However, the 'part' or 'module' is not limited to software or hardware. The 'part' or 'module' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the 'part' or 'module' may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'sub-components' or 'modules' may be combined into a smaller number of components and 'sub-components' or 'modules' or further separated into additional components and 'sub-components' or 'modules'.

[0043] The steps of a method or algorithm described in connection with some embodiments of the present disclosure may be implemented directly in hardware, a software module, or a combination of the two executed by a processor. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, such that the processor can read information from the storage medium, and write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a user terminal.

[0044] Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the attached drawings.

[0045] FIG. 1 is a diagram illustrating an autonomous operation system (100) according to various embodiments of the present disclosure.

[0046] Referring to FIG. 1, an autonomous operation system (100) according to various embodiments may include a control center (110) and at least one piece of construction equipment (or autonomous operation construction equipment) (120 to 150).

[0047] According to various embodiments, construction equipment (120 to 150) refers to machines that perform work at civil engineering or construction sites, and may include a mixer truck (120), a dump truck (130), a dozer (140), and an excavator (150), as illustrated in FIG. 1. However, this is merely exemplary, and construction equipment may include various machines such as a crane, a wheel loader, a scraper, and the like.

[0048] In one embodiment, the construction equipment (120 to 150) can perform work by an operator according to work instructions received from the control center (110). In another embodiment, the construction equipment (120 to 150) can perform work autonomously without an operator. The work instructions may include information related to the work area in which the construction equipment is to perform work, the work to be performed in the work area, etc. For example, the construction equipment (120 to 150) can move to the work area and perform work without or based on the user's operation according to the work instructions.

[0049] Construction equipment (120 to 150) may be equipped with various sensors, and based on information obtained through the sensors, the status of the construction equipment and / or the surrounding environment of the construction equipment may be detected, and the detection results may be considered in performing work.

[0050] In addition, the construction equipment (120 to 150) may be equipped with a dashboard that displays information about the construction equipment (120 to 150) or can set control settings for the construction equipment (120 to 150). According to one embodiment, the dashboard is equipped with a touch sensor that can receive a user's touch input, so that information can be obtained about which image on the dashboard the user has touched to execute an action. The construction equipment (120 to 150) can collect the user's dashboard touch information and transmit it to the control center (110).

[0051] According to various embodiments, the control center (110) may be a system that manages at least one piece of construction equipment (120 to 150) deployed at a work site. In one embodiment, the control center (110) may direct work to at least one piece of construction equipment (120 to 150). For example, the control center (110) may generate a work order defining a work area and work to be performed in the work area, and transmit the work order to at least one piece of construction equipment (120 to 150).

[0052] FIG. 2 is a drawing for explaining an excavator (200) according to various embodiments of the present disclosure. In the following description, the excavator among the construction equipment illustrated in FIG. 1 is used as an example, but the construction equipment is not limited to an excavator.

[0053] Referring to FIG. 2, the excavator (200) may be composed of a lower body (210) that functions as a mover, an upper body (220) that is mounted on the lower body (210) and rotates 360 degrees, and a front working device (230) coupled to the front of the upper body (220). However, this is merely exemplary, and the embodiments of the present disclosure are not limited thereto. For example, in addition to the components of the excavator (200) described above, one or more other components (e.g., a plate coupled to the rear of the lower body (210) may be added.

[0054] According to various embodiments, the upper body (220) may be provided with an interior space (not shown) in which a driver's cabin (222) is built and in which a power generation device (e.g., an engine) can be mounted. The driver's cabin (222) may be provided in a location close to the work area. The work area is a space in which the excavator (200) works and is located in front of the excavator (200). For example, considering that the driver on board performs work under a secured field of vision and the location where the front work device (230) is mounted, the driver's cabin (222) may be located in a location that is biased to one side from the upper body (220) while being close to the work area, as shown in FIG. 2.

[0055] According to various embodiments, the front work device (230) may be a device mounted on the upper surface of the upper body (220) and used for performing tasks such as excavating land or transporting heavy objects. According to one embodiment, the front work device (230) may include a boom (231) rotatably coupled to the upper body (220), a boom cylinder (232) for rotating the boom (231), an arm (233) rotatably coupled to the tip of the boom (231), an arm cylinder (234) for rotating the arm (233), a bucket (235) rotatably coupled to the tip of the arm (233), and a bucket cylinder (236) for rotating the bucket (235). When the excavator (200) is operating, one end of the boom (231), one end of the arm (233), and one end of the bucket (235) may each individually rotate to maximize the area that the bucket (235) can reach. The front working device (230) described above is known in many documents, so a detailed description thereof is omitted.

[0056] According to various embodiments, the lower body (210) may be coupled to the lower surface of the upper body (220). The lower body (210) may include a driving body formed as a wheel type using wheels or a crawler type using an infinite track. The driving body may implement forward, backward, left, and right movements of the excavator (200) using power generated by a power generation device as a driving force. According to one embodiment, the lower body (210) and the upper body (220) may be rotatably coupled by a center joint.

[0057] According to various embodiments, the excavator (200) may include a number of sensors for collecting information related to the operation of the excavator and / or information related to the surrounding environment.

[0058] According to one embodiment, the plurality of sensors may include a first sensor for detecting the motion of the excavator (200). For example, the motion of the excavator (200) may include a rotational motion of the upper body (220) (or the lower body (210)). The first sensor may be disposed at the center joint to detect the rotational motion of the upper body (220). Additionally, the motion of the excavator (200) may include a rotational motion of the front working device (230). The first sensor may be disposed at each of the boom (231), the arm (233), and the bucket (235), or at joints (e.g., hinge joints) of the boom (231), the arm (233), and the bucket (235) to detect the rotational motion of at least each of the boom (231), the arm (233), and the bucket (235). The location of the first sensor described above is not limited to one embodiment of the present disclosure, and the first sensor may be placed in various locations capable of detecting the status of the excavator (200).

[0059] According to one embodiment, the plurality of sensors may include a second sensor for detecting a work area in which the excavator (200) performs work. As described above, the work area is a space in which the excavator (200) performs work and may be located at the front of the excavator (200). The second sensor may be positioned on a portion of the upper body (220) close to the work area, for example, on a side of the upper surface of the cab (222) close to the front work device (230), to detect the work area. However, this is merely an example, and the location of the second sensor is not limited thereto. For example, the second sensor may be additionally or selectively positioned on the front work device (230), for example, the arm (233) or the bucket (235), to detect the work area.

[0060] According to one embodiment, the plurality of sensors may include a third sensor for detecting obstacles around the excavator (200). The third sensor may be positioned at the front, side, and rear of the upper body (220) to detect obstacles around the excavator (200). The location of the third sensor described above is not limited to one embodiment of the present disclosure, and the third sensor may be positioned at various locations capable of detecting obstacles around the excavator (200).

[0061] According to various embodiments, the various sensors described above may include an angle sensor, an inertial sensor, a rotation sensor, an electromagnetic wave sensor, a camera sensor, a radar, a lidar, or an ultrasonic sensor. For example, the first sensor may be composed of at least one of an angle sensor, an inertial sensor, or a rotation sensor, and the second sensor and the third sensor may be composed of at least one of an electromagnetic wave sensor, a camera sensor, a radar, a lidar, or an ultrasonic sensor. For example, a camera sensor disposed on the upper surface of the cab (222) and on the arm (233) of the excavator (200) may be used as the second sensor. In addition, a lidar disposed on the front of the excavator (200), an ultrasonic sensor disposed on the side and rear of the excavator (200), or a camera sensor disposed on the front, side, and rear of the excavator (200) may be used as the third sensor. Additionally or optionally, when the image sensor is used as the second sensor and the third sensor, it may be configured as a stereo vision system capable of acquiring images that can provide distance information of an object.

[0062] Additionally, each of the first, second, and third sensors may perform the same or similar operations as the other sensors. For example, the third sensor, which detects obstacles around the excavator (200), may be used to perform the operations of the second sensor, which detects the work area in which the excavator (200) is performing work.

[0063] According to various embodiments, the excavator (200) is capable of performing unmanned automation, that is, autonomous operations, and includes at least one positioning device or can obtain positioning information from an external device.

[0064] According to one embodiment, the positioning device may use a Global Navigation Satellite System (GNNS) module capable of receiving satellite signals, and an RTK (Real Time Kinematic) GNSS module may also be used for precise measurements. For example, at least one positioning device may be placed on the upper body (220) of the excavator (200).

[0065] FIG. 3 is a functional block diagram of construction equipment (300) according to various embodiments of the present disclosure. The construction equipment (300) described in the following drawings may be any one of the construction equipment (120 to 150) illustrated in FIG. 1.

[0066] Referring to FIG. 3, the construction equipment (300) may include a processor (310), a communication device (320), a storage device (330), an operating device (340), an output device (350), and a sensor device (360). However, this is merely exemplary, and the embodiments of the present disclosure are not limited thereto. For example, at least one of the components of the construction equipment (300) described above may be omitted, or one or more other components may be added to the configuration of the construction equipment (300).

[0067] According to various embodiments, the communication device (320) can transmit and receive data with a remote external device using wireless communication technology. The external device may include a control center (110), a remote control device, other display devices (e.g., smartphones, laptops, tablets, etc.), and / or other construction equipment. In this case, the communication technology used by the communication device (320) includes GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc. In addition, the communication device (320) may include at least one positioning device.

[0068] According to various embodiments, the storage device (330) may store various data used by at least one component of the construction equipment (300) (e.g., the processor (310), the communication device (320), the operating device (340), the output device (350), or the sensor device (360)). According to one embodiment, the storage device (330) may store specifications (e.g., model name, unique number, basic specifications), map data, etc. of the construction equipment (300). According to one embodiment, the storage device (330) may store design drawings that the construction equipment (300) is to work on. The design drawings may be directly stored in the storage device (330) by the user, or the construction equipment (300) may be connected to the control center (110) via the communication device (320) to obtain the design drawings and store them in the storage device (330). The storage device (330) may include at least one of a non-volatile memory device and a volatile memory device.

[0069] According to various embodiments, the operating device (340) can receive commands or data to be used for controlling the operation of the construction equipment (300). The operating device (340) can include an operating lever for operating at least a portion of the front working device (230) (e.g., a boom (231), an arm (233), and a bucket (235)), a handle for operating the steering of the lower body (210), a gear lever for operating the moving speed or forward and backward travel of the construction equipment (300), etc. In addition, the operating device (340) can include an emergency stop switch. The emergency stop switch can be a hardware switch attached to a specific location of the construction equipment (300) or a wireless emergency stop receiving device that operates by receiving a wireless signal. In the case of a wireless emergency stop receiving device, the construction equipment (300) can be brought to an emergency stop by receiving an emergency stop signal operated remotely.

[0070] According to various embodiments, the output device (350) may generate output related to the operation of the construction equipment (300). According to one embodiment, the output device (350) may include a display that outputs visual information, an audio data output device that outputs auditory information, a haptic module that outputs tactile information, etc. For example, the display may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical system (MEMS) display, or electronic paper. In addition, the audio data output device may include at least one of a speaker, earphone, earset, or headset included in the construction equipment (300) or connected to the construction equipment (300) via wired / wireless.

[0071] According to various embodiments, the sensor device (360) may include a first sensor for detecting the status of the construction equipment (300), a second sensor for detecting a work area in which the construction equipment (300) performs work, and / or a third sensor for detecting obstacles around the construction equipment (300), as described above in FIG. 2. According to one embodiment, the construction equipment (300) may include a posture recognition sensor for detecting the posture of the vehicle body, and in addition, sensors necessary for the operation of the construction equipment (300) may be added.

[0072] According to various embodiments, the processor (310) may be configured to control the overall operation of the construction equipment (300). According to one embodiment, the processor (310) may execute software (e.g., a program) stored in the storage device (330) to control at least one component among components connected to the processor (310) (e.g., a communication device (320), a storage device (330), an operating device (340), an output device (350), or a sensor device (360)), and perform various data processing or calculations. For example, as at least a part of the data processing or calculation, the processor (310) may store commands or data received from other components in the storage device (330), process the commands or data stored in the storage device (330), and store result data in the storage device (330). The processor (310) may be configured as a main processor and a secondary processor that can operate independently or together with the main processor. According to one embodiment, the processor (310) may perform CAN (Controller Area Network) communication with the aforementioned components (e.g., communication device (320), storage device (330), operating device (340), output device (350), or sensor device (360)), but the present disclosure is not limited thereto.

[0073] FIG. 4 is a functional block diagram of an object detection system based on construction equipment sensor fusion according to various embodiments of the present disclosure.

[0074] Referring to FIG. 4, the construction equipment sensor fusion-based object detection system (400) may be implemented within the construction equipment (300). Alternatively, the construction equipment sensor fusion-based object detection system (400) may be implemented as a separate device.

[0075] A construction equipment sensor fusion-based object detection system (400) may include a sensor device (410) that measures equipment surrounding information and equipment information, and a processing device (420) that processes data acquired from the sensor device (410) to generate a control signal.

[0076] Here, the processing policy (420) may be implemented as hardware, such as a separate ASIC, or as a software program. If implemented as a software program, it may be executed by the processor of the construction equipment sensor fusion-based object detection system (400) or by the processor (310) of the construction equipment of FIG. 3.

[0077] The sensor device (410) can obtain and provide information on objects surrounding the construction equipment and position information of the construction equipment. To this end, the sensor device (410) may include a camera sensor (412) that captures and outputs images of objects surrounding the construction equipment, a radar sensor (416) that detects and outputs position and speed information of objects surrounding the construction equipment, and an IMU (Inertial Measurement Unit) sensor (414) that detects and outputs position information of the construction equipment.

[0078] A plurality of camera sensors (412) can be mounted to enable monitoring of the entire surroundings of the construction equipment, and can capture images of objects surrounding the construction equipment and transmit or output them to a processing device (420).

[0079] The IMU sensor (414) can detect roll / pitch / yaw values ​​indicating the degree of inclination of each axis of the construction equipment and transmit or output them to a processing device (420).

[0080] A plurality of radar sensors (416) can be installed to monitor the entire surroundings of the construction equipment, detect objects, and transmit or output position and speed information of objects surrounding the construction equipment to a processing device (420). The positions of objects surrounding the construction equipment can be expressed as positions on a radar space coordinate system.

[0081] The processing device (420) receives images of surrounding objects from the camera sensor (412), receives position and speed information of surrounding objects from the radar sensor (416), and receives attitude information of the construction equipment from the IMU sensor (414).

[0082] The processing device (420) may include an object determination unit (421), a coordinate transformation unit (422), a matching unit (423), a filtering unit (424), and a control signal generation unit (425).

[0083] The object identification unit (421) can receive an image provided from the camera sensor (412). The object identification unit (421) can identify a dangerous object in the image and detect and output pixel information of the dangerous object on the image. The pixel information of the dangerous object can include the width, height, and coordinates of the center point of the dangerous object on the image.

[0084] FIG. 5 is a diagram illustrating the operation of the object identification unit of FIG. 4. As shown in box (501) of FIG. 5, the object identification unit (421) can recognize pedestrians, etc., around construction equipment. The object identification unit (421) can utilize known techniques to recognize pedestrians, etc., in an image. For example, the object identification unit (421) includes a deep learning-based model trained to identify specific objects, including pedestrians with which construction equipment should not collide, as dangerous objects, and can use the model to identify the objects.

[0085] The coordinate transformation unit (422) can receive pixel information of a dangerous object on an image from the object determination unit (421). The pixel information of the dangerous object can represent a feature point in the image.

[0086] In addition, the coordinate conversion unit (422) can receive the attitude information of the construction equipment from the IMU sensor (414). When the coordinate conversion unit (422) receives the attitude information of the construction equipment, it can primarily calibrate the pixel information of the hazardous object based on the degree of inclination of the construction equipment according to the attitude information of the construction equipment. In addition, the coordinate conversion unit (422) can secondarily correct the distortion of the camera included in the pixel information of the hazardous object by utilizing the distortion coefficient of the camera lens information, etc., thereby converting the pixel information output as a feature point into coordinates of the radar space coordinate system.

[0087] Referring back to FIG. 4, the matching unit (423) can receive position and speed information of objects surrounding the construction equipment from the radar sensor (416), and receive radar space coordinates of dangerous objects from the coordinate conversion unit (422). The matching unit (423) can match the radar space coordinates of the dangerous objects with the positions of the objects surrounding the construction equipment. Since the positions of the objects surrounding the construction equipment are acquired from the radar sensor (416), they can be expressed in radar space coordinates without coordinate conversion. Therefore, the dangerous objects acquired through the camera (412) image and the surrounding objects acquired through the radar sensor (416) can be expressed in the same radar space coordinates, and can be matched with each other when the coordinate values ​​are similar.

[0088] FIG. 6 is a diagram for explaining the operation of the matching unit of FIG. 4. Referring to FIG. 6, it may be necessary to filter out only objects classified as dangerous objects by the object determination unit (421) among objects detected by the radar sensor (416). To this end, the matching unit (423) compares pixel information of dangerous objects converted into radar space coordinates with object information detected by the radar sensor (416), and can match objects detected by the radar sensor (416) with dangerous objects classified by the object determination unit (421) among objects in the camera (412) image through distance calculation based on probability distribution.

[0089] In Fig. 6, rectangular areas (601, 602) represent location information of camera-based hazardous objects, and blue circles (611 to 617) represent location information of radar-based objects. In addition, blue ovals (621, 626, 627) represent location probability distributions of radar-based objects.

[0090] Referring to FIG. 6, assuming that the object determination unit (421) detects two people (601, 602) as dangerous objects in the image from the camera sensor (412), and the radar sensor (416) detects a total of seven objects (611 to 617) including the two people, it is necessary to match the seven objects (611 to 617) detected by the radar sensor (416) with the two dangerous objects (601, 602).

[0091] According to one embodiment, the matching unit (423) can match objects based on simple distance. The matching unit (423) can match the radar detection object located closest to the dangerous object to the dangerous object. In the example of FIG. 6, according to the simple distance-based matching, the matching unit (423) can match the first person (601) with the leftmost radar detection object (611), and the second person (602) with the second radar detection object (616) from the right.

[0092] According to another embodiment, the matching unit (423) may utilize the probability distribution of the radar for matching. Specifically, the matching unit (423) may calculate the position probability distribution of the radar for seven objects (611 to 617) detected by the radar sensor (416) and obtain the average vector of the position probability distribution of each object calculated. Then, the matching unit (423) may substitute the average vector of the position probability distribution of each object into the distance calculation formula based on the position probability distribution below to calculate the distance to the dangerous object as in the following mathematical expression 1.

[0093]

[0094] Here, is the position vector of the camera-based hazard object i, is the average vector of the position probability distribution of object j output from the radar sensor (416), represents the covariance matrix of the object position output from the radar sensor (416).

[0095] In other words, the matching unit (423) can use the mathematical expression 1 to calculate the distance between the dangerous object and each of the radar detection objects, and match the dangerous object with the radar detection object having the shortest distance among the radar detection objects.

[0096] In the example of Fig. 6, if the probability distribution of the radar is utilized for matching, the matching unit (423) can determine that the rightmost radar detection object (617) is more likely to be the second person (602).

[0097] Referring again to FIG. 4, the filtering unit (424) can estimate precise distance information of a risk object based on the position and speed of the matched risk object. To this end, the filtering unit (424) can use a Kalman filter-based filtering algorithm. The filtering unit (424) receives the matched camera-based risk object position information and the radar-based risk object position and speed information through the matching unit (423), and can ultimately estimate precise distance information of the risk object.

[0098] In this case, the filtering unit (424) can estimate more precise distance information by using the detection cycle for each sensor and the longitudinal / lateral measurement accuracy for each sensor.

[0099] The control signal generation unit (425) receives precise distance information of a dangerous object from the filtering unit (424) and can generate a driving control signal or a turning control signal of the construction equipment based on the precise distance information of the dangerous object. If the control signal generation unit (425) uses the precise distance information of the estimated dangerous object to generate equipment driving and turning control signals, it can be utilized for a more effective collision prevention safety solution.

[0100] FIG. 7 is a flowchart illustrating an object detection method based on construction equipment sensor fusion according to various embodiments of the present disclosure.

[0101] The flowchart of FIG. 7 can be performed by dedicated hardware of the construction equipment sensor fusion-based object detection system (400) of FIG. 4, or a dedicated processor, or a processor (310) of the construction equipment of FIG. 3.

[0102] In operation S710, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can obtain an image including an object, object information including the position and speed of the object, and posture information of the construction equipment.

[0103] As described above, the camera sensor (412) can capture images of objects around the construction equipment and transmit or output them to the processor. The IMU sensor (414) can detect roll / pitch / yaw values ​​indicating the degree of inclination of each axis of the construction equipment and transmit or output them to the processor (420). The radar sensor (416) can detect objects and transmit or output position and speed information of objects around the construction equipment to the processor.

[0104] According to one embodiment, the location of objects around the construction equipment can be expressed in radar space coordinates.

[0105] In operation S720, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can identify a hazardous object in an image and obtain pixel information about the hazardous object in the image. The pixel information about the hazardous object may include the width, height, and coordinates of the center point of the hazardous object in the image. Additionally, the pixel information about the hazardous object may indicate a feature point in the image.

[0106] In operation S730, when the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) receives the posture information of the construction equipment, the construction equipment can primarily calibrate the pixel information of the dangerous object based on the degree of inclination of the construction equipment according to the posture information of the construction equipment. In addition, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can secondarily correct the distortion caused by the camera included in the pixel information of the dangerous object by utilizing distortion coefficients such as camera lens information.

[0107] In operation S740, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can convert pixel information output as feature points into coordinates in a radar space coordinate system.

[0108] In operation S750, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can match the radar spatial coordinates of a hazardous object with the locations of objects around the construction equipment. The locations of the objects around the construction equipment can be expressed in radar spatial coordinates, as described above.

[0109] At this time, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can utilize the radar probability distribution for matching. The processing device (420) calculates the radar position probability distribution for the positions of the objects and obtains the average vector of the position probability distribution of each object calculated. Then, the processing device (420) can calculate the distance of the dangerous object by substituting the average vector of the position probability distribution of each object into the distance calculation formula of the position probability distribution of [Mathematical Formula 1].

[0110] In operation S760, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can estimate precise distance information of the dangerous object based on the position and speed of the matched dangerous object. To this end, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can use a Kalman filter-based filtering algorithm. In this case, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can estimate more precise distance information by using the detection cycle for each sensor and the longitudinal / lateral measurement accuracy for each sensor.

[0111] In operation S770, the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) can generate a driving control signal or a turning control signal of the construction equipment based on the precise distance information of the dangerous object. If the construction equipment (300) or the construction equipment sensor fusion-based object detection system (400) uses the precise distance information of the estimated dangerous object to generate the equipment driving and turning control signals, it can be utilized for a more effective collision avoidance safety solution.

[0112] According to the embodiments of the present invention as described above, by applying AI technology and sensor fusion technology, even in complex surrounding environments and when the equipment shakes greatly due to the characteristics of construction equipment operation, information on objects around the equipment can be more accurately estimated for precise control to prevent collisions.

[0113] Although all components constituting the embodiments of the present invention have been described as being combined or operated in combination as one, the present invention is not necessarily limited to such embodiments. That is, within the scope of the present invention, all components may be selectively combined and operated at least as one.

[0114] Additionally, while each of these components may be implemented as a single, independent piece of hardware, some or all of these components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions of one or more hardware components. The codes and code segments that constitute the computer program will be readily inferred by those skilled in the art.

[0115] These computer programs can be stored on computer-readable media and read and executed by a processor of a computer or construction equipment, thereby implementing embodiments of the present invention. Storage media for the computer program may include magnetic recording media, optical recording media, etc.

[0116] In addition, terms such as “include,” “comprise,” or “have” described above, unless specifically stated to the contrary, should be interpreted to mean that the corresponding component may be included, and thus should not be interpreted to exclude other components, but rather to include other components.

[0117] All terms, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains, unless otherwise defined. Commonly used terms, such as terms defined in dictionaries, should be interpreted in a way consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0118] The above description is merely an example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention.

[0119] Accordingly, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be construed as being included within the scope of the present invention.

Claims

1. In a sensor fusion-based object detection system, A sensor device including a first sensor outputting an image, a second sensor outputting a position and velocity of at least one object, and a third sensor outputting posture information of construction equipment; and It includes a processing device that generates a driving control signal or a turning control signal of the above construction equipment, The above processing device An object determination unit that determines a dangerous object in the image and outputs pixel information of the dangerous object; A coordinate transformation unit that transforms pixel information of the above-mentioned hazardous object into a location on radar space coordinates; A matching unit that receives a position on the radar space coordinates of the above-mentioned dangerous object and receives a position on the radar space coordinates of the at least one object from the second sensor, and matches the at least one object with the above-mentioned dangerous object; A filtering unit that estimates precise distance information of the dangerous object based on the position and speed of an object matched with the dangerous object among at least one object; and A sensor fusion-based object detection system, comprising a control signal generation unit that generates the driving control signal or the turning control signal of the construction equipment based on the precise distance information of the dangerous object.

2. In paragraph 1, A sensor fusion-based object detection system, wherein the first sensor includes a camera sensor, the second sensor includes a radar sensor, and the third sensor includes an IMU (Inertial Measurement Unit) sensor.

3. In paragraph 1, A sensor fusion-based object detection system, wherein the above-mentioned posture information includes roll values, pitch values, and yaw values ​​indicating the degree of inclination for each axis.

4. In paragraph 1, The object identification unit includes a deep learning-based model trained to identify a specific object, including a pedestrian with which the construction equipment should not collide, as the dangerous object. A sensor fusion-based object detection system, wherein pixel information of the above-mentioned dangerous object includes the coordinates of the width, height, and center point of the above-mentioned dangerous object on the image.

5. In paragraph 2, A sensor fusion-based object detection system, wherein the coordinate transformation unit primarily corrects pixel information of the dangerous object based on the posture information of the construction equipment, and secondarily corrects camera distortion included in the pixel information of the dangerous object based on the lens information of the camera.

6. In paragraph 2, A sensor fusion-based object detection system in which the matching unit matches the dangerous object and the at least one object using the object location output from the radar sensor and the probability distribution of the object location.

7. In paragraph 6, The above matching part is, Mathematical formula (Here, is the position vector of the hazardous object i, is the average vector of the position probability distribution of object j output from the radar sensor, A sensor fusion-based object detection system that calculates the distance between the dangerous object and each of the at least one object using a covariance matrix of object positions output from the radar sensor, and matches the dangerous object with the object with the shortest distance among the at least one object.

8. In paragraph 1, A sensor fusion-based object detection system, wherein the above filtering unit is implemented using a Kalman filter-based filtering algorithm.

9. In construction equipment, A sensor device including a camera that outputs an image, a radar sensor that outputs the position and velocity of at least one object, and an IMU sensor that outputs attitude information of the construction equipment; and A processor for generating a driving control signal or a turning control signal of the above construction equipment is included, The above processor, Identifying a dangerous object in the image above and obtaining pixel information of the identified dangerous object, Convert the pixel information of the above-mentioned hazardous object into a location on radar space coordinates, Receive the position of the at least one object in radar space coordinates from the radar sensor, Matching the above risk object with at least one object, Estimating precise distance information of the dangerous object based on the position and speed of an object matched with the dangerous object among at least one object, Construction equipment that generates the driving control signal or turning control signal of the construction equipment based on the precise distance information of the above-mentioned hazardous object.

10. In paragraph 9, The above detailed information includes roll values, pitch values, and yaw values ​​indicating the degree of inclination for each axis, construction equipment.

11. In paragraph 9, The above processor, A deep learning-based model trained to identify specific objects, including pedestrians, with which the construction equipment should not collide as dangerous objects, Construction equipment, wherein the pixel information of the above-mentioned risk object includes the width, height, and coordinates of the center point of the above-mentioned risk object on the image.

12. In paragraph 9, The above processor, Based on the detailed information of the above construction equipment, the pixel information of the above hazardous object is first corrected, Construction equipment that secondarily corrects camera distortion included in pixel information of the above-mentioned hazardous object based on lens information of the above-mentioned camera.

13. In paragraph 9, The above processor, Construction equipment that matches the dangerous object and the at least one object using the object location output from the radar sensor and the probability distribution of the object location.

14. In paragraph 13, The above processor, Mathematical formula (Here, is the position vector of the hazardous object i, is the average vector of the position probability distribution of object j output from the radar sensor, A construction equipment that calculates the distance between the dangerous object and each of the at least one object using the covariance matrix of the object positions output from the radar sensor, and matches the dangerous object with the object with the shortest distance among the at least one object.

15. In paragraph 9, The above processor, Construction equipment that estimates precise distance information of the above-mentioned hazardous object using a Kalman filter-based filtering algorithm.

16. In a method for object detection based on sensor fusion using construction equipment, A step of acquiring an image including an object using a camera sensor; A step of acquiring the position and velocity of at least one object using a radar sensor; A step of obtaining attitude information of construction equipment using an IMU sensor; A step of identifying a dangerous object in the image and obtaining pixel information of the identified dangerous object; A step of converting pixel information of the above-mentioned hazardous object into a location on radar space coordinates; A step of matching the above risk object with the at least one object; A step of estimating precise distance information of the dangerous object based on the position and speed of an object matched with the dangerous object among the at least one object; and A sensor fusion-based object detection method, comprising a step of generating the driving control signal or the turning control signal of the construction equipment based on the precise distance information of the risk object.

17. In paragraph 16, A sensor fusion-based object detection method, wherein the above-mentioned posture information includes roll values, pitch values, and yaw values ​​indicating the degree of inclination for each axis.

18. In paragraph 16, The step of determining the above-mentioned dangerous object includes a step of determining the dangerous object using a deep learning-based model trained to identify a specific object including a pedestrian with which the construction equipment should not collide as the dangerous object, A sensor fusion-based object detection method, wherein pixel information of the above-mentioned dangerous object includes the width, height, and coordinates of the center point of the above-mentioned dangerous object on the image.

19. In paragraph 16, A step of primarily correcting pixel information of the hazardous object based on the detailed information of the construction equipment; and A sensor fusion-based object detection method further comprising a step of secondarily correcting camera distortion included in pixel information of the dangerous object based on lens information of the camera.

20. In paragraph 16, The step of matching the above risk object with the at least one object is: A sensor fusion-based object detection method, comprising a step of matching the dangerous object and the at least one object using the object location output from the radar sensor and the probability distribution of the object location.

21. In paragraph 20, The step of matching the dangerous object and the at least one object using the object location output from the radar sensor and the probability distribution of the object location is as follows: Mathematical formula (Here, is the position vector of the hazardous object i, is the average vector of the position probability distribution of object j output from the radar sensor, A step of calculating the distance between the dangerous object and each of the at least one object using the covariance matrix of the object positions output from the radar sensor; and A sensor fusion-based object detection method, comprising a step of matching the object having the shortest distance among the at least one object with the dangerous object.

22. In paragraph 16, The step of estimating the precise distance information of the above risk object is: A sensor fusion-based object detection method, comprising a step of estimating precise distance information of the dangerous object using a Kalman filter-based filtering algorithm.

Images