Apparatus and method for autonomous driving

Abstract

An unmanned autonomous driving device according to an embodiment of the present invention is characterized by comprising: a driving control module that generates first driving control information according to set path information; a sensor module equipped with a camera that acquires images according to driving; a processing module that determines obstacles, road conditions, and / or distances in the driving direction based on the acquired images, and determines second driving control information based on the determination result; an integrated driving control module that generates integrated driving control information for the speed, direction, and braking of a cart based on the first driving control information and the second driving control information; and a driving module that controls unmanned autonomous driving based on the integrated driving control information.

Description

Technology Field

[0001] The present invention relates to an autonomous driving device and method, and more particularly to a device and method capable of controlling unmanned driving by collecting at least one sensor information during autonomous driving. Background Technology

[0003] An unmanned driving device refers to a device capable of driving autonomously along a set path. A golf cart is a vehicle used on a golf course, designed to allow golf players to ride in it and transport golf clubs. Golf carts enable the convenient and rapid movement of golf players and golf equipment across a large golf course. Golf carts operate autonomously when a player is in the middle of a game, and switch between autonomous and manual driving while transporting a player.

[0004] Figure 1 is a drawing illustrating an example of driving a golf cart, which is an autonomous driving device.

[0005] Referring to FIG. 1, in order to drive a golf cart (10) autonomously, it is necessary to bury a guide line (20) underground in the cart road (30). Thus, the guide line (20) is a wire or coil buried in the cart road (30) constructed for the golf cart (10) to drive on at the golf course. The golf cart (10) detects the guide line (20) and is controlled to drive along the guide line (20). Since the golf cart (10) drives along the guide line in autonomous driving mode, it is possible to prevent the golf cart (10) from driving off the cart road (30). The guide line (20) can be buried in a line underground in the center of the cart road (30).

[0006] However, if an obstacle that is moving or stationary is located on the road (30) or damage to the road occurs while the golf cart (10) is driving autonomously along the guide line (20), a person located remotely must manually stop the golf cart (10). In addition, if an obstacle and / or damage on the cart road (30) is not detected, there is a risk that the cart may collide with the obstacle or overturn. The problem to be solved

[0008] Accordingly, the objective of the present invention is to provide a device and method capable of controlling the speed, direction, and / or braking of a driving device by collecting at least one sensor data while driving autonomously without a driver.

[0009] Another objective of the present invention is to provide an apparatus and method capable of controlling autonomous driving by collecting image data, inertial data and / or distance data during unmanned autonomous driving, and estimating object detection, road conditions and / or distance to an object, etc. means of solving the problem

[0011] An unmanned autonomous driving device according to an embodiment of the present invention for achieving the above objective is characterized by comprising: a driving control module that generates first driving control information according to set path information; a sensor module equipped with a camera that acquires images according to driving; a processing module that determines obstacles, road conditions, and / or distances in the driving direction based on the acquired images, and determines second driving control information based on the determination result; an integrated driving control module that generates integrated driving control information for the speed, direction, and braking of a cart based on the first driving control information and the second driving control information; and a driving module that controls unmanned autonomous driving based on the integrated driving control information.

[0012] The set path of the autonomous driving device is set by a guide line embedded in the road, and the driving control module is equipped with left and right sensors that detect the guide line, and can generate first driving control information that determines the driving direction of the autonomous driving device based on information measured by the left and right sensors.

[0013] The sensor module further comprises an inertial sensor and / or a time of flight (TOF) sensor, wherein the inertial sensor generates road inclination, the driving speed and direction information of the cart based on inertial information resulting from the driving of the vehicle, and the TOF sensor can generate distance information to an obstacle in the driving direction.

[0014] The processing module may include: a collection unit that collects image data, inertial data, and distance data from a camera, an inertial sensor, and a TOF sensor; a preprocessing unit that preprocesses and filters the collected sensor data; a processing unit that detects an object from the image data, analyzes the road based on the image and inertial data, and measures the distance to the object based on the image data and distance data; and a judgment unit that analyzes the road condition based on the results of object detection, road analysis, and distance measurement, and generates driving control information such as driving speed, direction, and braking according to the analyzed road condition.

[0015] The processing unit may include an object detection unit that analyzes image data to detect and track moving objects and detects, identifies, and classifies static and dynamic obstacles; an analysis unit that analyzes image data to recognize road boundaries and lanes and analyzes inertial data to detect road gradient, driving speed, and direction; and a distance measurement unit that calculates the distance to an object through stereo matching of image data and fuses distance data and image data to measure the distance to an object.

[0016] An unmanned autonomous driving device according to an embodiment of the present invention for achieving the above objective comprises: a process of generating first driving control information of the autonomous driving device based on a set driving path of a road; an information acquisition process of acquiring image data while in a state of unmanned autonomous driving; a processing process of determining obstacles, road conditions, and / or distances in the driving direction based on the acquired image information, and determining second driving control information based on the determination result; an integrated control process of generating integrated driving control information for driving speed, direction, and / or braking by fusing the first driving control information and the second driving control information; and a driving process of controlling unmanned autonomous driving based on the integrated driving control information.

[0017] The information acquisition process may further include a process of acquiring inertial information for calculating the road gradient, the driving speed and direction of the cart; and a process of acquiring distance information for calculating the distance to an obstacle in the driving direction.

[0018] The processing process may include: a collection process for collecting image data, inertial data, and distance data from a camera, an inertial sensor, and a TOF sensor; a preprocessing process for preprocessing and filtering the collected sensor data; a processing process for detecting an object from the image data, analyzing the road based on the image and inertial data, and measuring the distance to the object based on the image data and distance data; and a judgment process for analyzing the road condition based on the results of object detection, road analysis, and distance measurement, and generating driving control information such as driving speed, direction, and braking according to the analyzed road condition.

[0019] The processing process may include an object detection process that analyzes image data to detect and track moving objects and detects, identifies, and classifies static and dynamic obstacles; an analysis process that analyzes image data to recognize road boundaries and lanes and analyzes road gradients by analyzing inertial data; and a distance calculation process that calculates the distance to an object through stereo matching of image data and calculates the distance to an object by fusing distance data and image data. Effects of the invention

[0021] The unmanned autonomous device of the present invention can prevent safety accidents that may occur during autonomous driving. The unmanned autonomous device fuses real-time video data acquired through a camera, inertial data acquired through an inertial sensor, and distance data acquired through a TOF sensor to detect obstacles, pedestrians, and road conditions within the driving environment; based on this, it determines the optimal driving speed according to the situation and automatically controls the vehicle, and has the effect of performing emergency braking when necessary. Brief explanation of the drawing

[0023] Figure 1 is a drawing illustrating an example of driving a golf cart, which is an autonomous driving device. FIG. 2 is a diagram illustrating the configuration of an unmanned autonomous driving device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the configuration of a processing module of an autonomous driving device according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a driving control method of an autonomous driving device according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating a method for generating driving control information in an autonomous driving device according to an embodiment of the present invention. Specific details for implementing the invention

[0024] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the attached embodiments are merely examples intended to aid in understanding the present invention and are not intended to limit the invention. Furthermore, detailed descriptions regarding technologies that are obvious to those skilled in the art or easily derivable will be omitted.

[0025] FIG. 2 is a diagram illustrating the configuration of an unmanned autonomous driving device according to an embodiment of the present invention.

[0026] Referring to FIG. 2, the unmanned autonomous driving device may include a driving control module (110), a sensor module (120), a processing module (130), an integrated driving control module (140), and a driving module (150). The sensor module (120) may include at least one sensor module among a camera module (123), an inertial sensor module (125), and a time of flight (TOF) module (127). In some embodiments, the unmanned autonomous driving device may omit at least one of the above components or additionally provide other components.

[0027] Unmanned autonomous vehicles can be referred to as carts, trolleys, AGVs (automated guided vehicles), or AGCs (automated guided carts). A cart is a device capable of transporting people or goods, and it may be called a trolley in countries such as the UK or Australia. An AGV is an unmanned transport vehicle that can transport people, goods, or materials within a controlled environment without human control or driving. An AGC can be the simplest form of AGV that moves along a predetermined path or a magnetic track. Unmanned autonomous vehicles can navigate their driving environment using various navigation methods. Autonomous vehicles can utilize a guide line system. Guide lines can be installed by embedding magnetic tape or wires into the road surface along a designated path, and the autonomous vehicle can drive along these lines. In other words, autonomous vehicles can drive along a predetermined path by detecting magnetic fields or electrical signals. Autonomous vehicles can also use laser navigation. Autonomous vehicles equipped with laser navigation can utilize reflectors placed around the facility and calculate their position based on the time it takes for a laser beam emitted from the vehicle to strike the reflector, reflect back, and return. Autonomous vehicles can use optical navigation. Autonomous vehicles using optical navigation can drive along markers or lines visually marked on the floor. Autonomous vehicles can use inertial navigation. Autonomous vehicles using inertial navigation methods use gyroscopes and accelerometers to track movement from a given location and can navigate a path based on position changes calculated over time. Autonomous vehicles can use GPS navigation. Autonomous vehicles using GPS navigation can navigate a path via GPS in outdoor environments.

[0028] In an embodiment of the present invention, the unmanned autonomous driving device may be a golf cart, and it will be described assuming that it drives autonomously using a guide line method. In this case, the guide line may be embedded in the road of the golf course in the form shown in FIG. 1.

[0029] The driving control module (110) can generate first driving control information for the cart based on the guide line of the road. The driving control module (110) may include a guide sensor module capable of detecting the guide line. The guide sensor module may include a left sensor and a right sensor. The left sensor and the right sensor may be installed on the left and right sides of the unmanned autonomous device, respectively, and can generate sensor values ​​by detecting the guide line. The driving control module (110) can drive the movement speed of the autonomous driving device at a designated speed and can perform movement initiation and provision operations according to the user's remote control. In addition, in the autonomous driving mode, the driving control module (110) determines whether the sensor values ​​detected by the left and right sensors are below a reference value, and if the sensor values ​​of the left sensor and the right sensor are above the reference value, it can generate driving information that controls the steering of the golf cart so that the deviation between the sensor values ​​of the left sensor and the right sensor is eliminated. The driving control information generated by the driving control module (110) may be the first driving control information.

[0030] The sensor module (120) may be equipped with at least one sensor that acquires images of the autonomous driving device as it moves. The sensor module (120) according to an embodiment of the present invention may include at least one of a camera module (123), an inertial sensor module (125), and a TOF module (127).

[0031] The camera module (123) may be a stereo camera. The camera module (123) may be mounted on the front of the autonomous driving device and may acquire stereo image data in real time while driving autonomously. The acquired image data may be a still image or a video. The acquired stereo image data may be used to detect moving objects and stationary objects, and to estimate the road environment and the distance to the object. The inertial sensor module (125) may measure the inertia of the moving autonomous driving device. The measured inertial data may be used to estimate the road environment (e.g., the angle of inclination of the road). The inertial sensor module (125) may include a gyroscope, an accelerometer, and a geomagnetic sensor. The Time of Flight (ToF) sensor module (127) may acquire distance data to an object (e.g., an obstacle) using a ToF sensor. There are two basic types of ToF sensors: optical sensors and electromagnetic sensors. Optical ToF sensors may use light pulses in the infrared range for distance measurement. Pulses are emitted from the sensor, reflected off an object, and return to the sensor; the travel time is measured and the distance is used to calculate the distance. Electromagnetic ToF sensors measure distance using electromagnetic waves, such as radar or lidar.

[0032] The processing module (130) can determine obstacles, road conditions, and / or distances in the driving direction based on image data, inertial data, and / or distance data acquired from the sensor module (120), and determine second driving control information based on the determination result. The second driving control information may be information related to the detection of objects (moving objects and stationary objects) located on the driving road, road environment (lane condition, road damage, slope, etc.), distance to objects, etc.

[0033] The integrated driving control module (140) can generate control information such as driving speed, direction, and braking of the driving device based on the first driving control information and the second driving control information. The first driving control information may be driving control information such as driving speed, direction, and braking based on movement path information (e.g., information on guide lines embedded in the road). The second driving control information may be driving control information related to object detection located on the movement path, road environment, and / or distance from the object. The integrated driving control module (140) may be information for controlling driving on a set path based on the first driving control information, and the second driving control information may be information for controlling driving according to object detection located on the set path, distance from the detected object, road environment, etc. The integrated driving control module (140) may generate driving control information by fusing the first driving control information and the second driving control information.

[0034] The driving module (150) can control the driving of the autonomous driving device by controlling the driving speed, direction, and / or braking of the autonomous driving device based on driving control information. The driving module (150) may be equipped with a driving module that controls the driving speed of the autonomous driving device, a steering module that controls the driving direction of the autonomous driving device, a braking module that controls the braking of the autonomous driving device, etc.

[0035] An autonomous driving device according to an embodiment of the present invention can prevent safety accidents that may occur during driving by detecting objects and road environments along a set path based on first driving control information while driving along the path. By analyzing real-time information acquired through a stereo camera, an inertial sensor, and / or a TOF sensor, it can detect obstacles and pedestrians within the driving environment, detect road lanes, damage, and inclination, and measure accurate distances through data fusion. Based on this, the autonomous driving device determines the optimal driving speed according to the situation, automatically controls acceleration and deceleration, and can perform emergency braking when necessary.

[0036] The processing module (130) of the autonomous driving device may be an AI (artificial intelligence) processor mounted on the main CPU. The processing module (130) can integrally process stereo image data captured by a stereo camera, inertial data acquired from an inertial sensor, and distance measurement data from a TOF sensor. The integration controller (140) can generate driving control information for the autonomous driving device by fusing the second driving control information generated by the processing module (130) and the first driving control information generated by the driving control module (110). That is, the integration controller (140) controls in parallel with a set path control (e.g., guide line control) system and can implement additional control functions without affecting existing control. Through this, advanced AI-based functions can be implemented while minimizing hardware changes to the autonomous driving device. This function operates only in automatic mode. For example, the driving control module (Legacy Cart Controller) may be a device that drives automatically using road guidance lines and performs acceleration and deceleration in specific sections. In the autonomous driving device according to the embodiment of the present invention, these functions may be maintained as they are. Through this hierarchical structure, a consistent pipeline can be constructed from the acquisition of sensor data to processing and control, and maintainability and scalability can be secured by ensuring the independence of each module.

[0037] FIG. 3 is a diagram illustrating the configuration of a processing module of an autonomous driving device according to an embodiment of the present invention.

[0038] Referring to FIG. 3, the processing module (130) may include a collection unit (310), a preprocessing unit (320), a processing unit (330), and a judgment unit (340). In some embodiments, the processing module of the unmanned autonomous driving device may omit at least one of the above components or additionally include other components.

[0039] The processing module (130) of the autonomous driving device may be composed of an image processing algorithm and a deep learning model structure. The image processing algorithm can perform functions such as preprocessing real-time image data acquired from a stereo camera, removing noise, and extracting feature points. The deep learning model can be optimized to enable real-time object detection based on YOLOv5 and can recognize road conditions and surrounding environments.

[0040] The collection unit (310) can collect sensor data obtained from the sensor module (120). The collection unit (310) may include a camera driver (313), an inertial sensor driver (315), and a TOF sensor driver (317). The camera driver (313) can acquire synchronized image data from the stereo camera of the camera module (123). The camera driver (313) can perform real-time image streaming processing and camera parameter adjustment (exposure, focus, etc.). The inertial sensor driver (315) can collect data obtained from the inertial sensor module (125) (data from the accelerometer, gyroscope, and geomagnetic sensors). The inertial sensor driver (315) can manage sensor calibration and adjust the sampling rate. The TOF sensor driver (317) can collect precise distance measurement data, adjust sensor settings and measurement ranges, and verify data reliability.

[0041] The preprocessing unit (320) may include an image preprocessing unit (323) and a sensor filtering unit (325). The image preprocessing unit (323) may remove image noise using an image Gaussian filter, etc. The image preprocessing unit (323) may perform an image normalization process to correct the brightness and contrast of the image and standardize it. The image preprocessing unit (323) may detect feature points of the image using algorithms such as SIFT and SURF. The sensor filtering unit (325) may perform inertial data filtering (e.g., noise removal using a Kalman filter) and TOF data correction (e.g., outlier removal and interpolation).

[0042] The processing unit (330) may be an AI processing layer. The processing unit (330) may include an object detection unit (333), a road analysis unit (335), and a distance measurement unit (337). The object detection unit (333) may use YOLOv5. The object detection unit (333) can process image data to detect and track moving objects (e.g., pedestrians). The object detection unit (333) can detect, identify, and classify obstacles (static / dynamic obstacles). The road analysis unit (335) can process image data to recognize road boundaries (road boundaries and lanes). The road analysis unit can analyze inertial data to calculate road inclination (road slope). The distance measurement unit (337) can calculate the distance to the detected object by matching stereo image data (using binocular parallax). The distance measuring unit (337) can measure the distance to the detected object by fusing stereo camera and TOF sensor data.

[0043] The judgment unit (340) may include a situation analysis unit (343) and a control decision unit (345). The situation analysis unit (343) can evaluate the risk level by analyzing risk factors in the surrounding environment and evaluate the safety of the current path by analyzing the path. The control decision unit (345) can determine the speed by calculating an appropriate speed according to the situation and determine the braking by determining the necessity and intensity of braking. The judgment unit (340) can analyze the risk level and path based on the results of the situation analysis, make control decisions such as speed and braking based thereon, and generate second driving control information based thereon.

[0044] The integrated driving control module (150) can perform speed control, emergency control, and integrated control operations based on the first driving control information and the second driving information. The integrated control module (150) can perform speed control. The speed control of the integrated control module (150) implements speed feedback control by PID control and can dynamically adjust the control gain according to the situation by adjusting the adaptive gain. The integrated driving control module (140) can perform emergency control. The emergency control of the integrated driving control module (150) can perform an immediate stop when a dangerous situation is detected by executing an emergency stop logic. The integrated driving control module (150) can perform integrated control interface operations. The integrated control interface can be linked with a set path driving control system and can manage the priority of control commands.

[0045] FIG. 4 is a flowchart illustrating a driving control method of an autonomous driving device according to an embodiment of the present invention.

[0046] Referring to FIG. 4, the autonomous driving device can generate first driving control information based on the driving path of the road set in step 411. The autonomous driving device can acquire sensor data while driving. In step 421, the autonomous driving device can acquire at least one sensor data according to the driving. The sensor data may include image data. In step 423, the autonomous driving device can determine obstacles, road conditions, and / or distances in the driving direction based on the acquired image information, and determine second driving control information based on the determination result. In step 425, the autonomous driving device can generate speed, direction, and braking control information of the autonomous driving device based on the first driving control information and the second driving control information. In step 427, the autonomous driving device can control unmanned driving of the autonomous driving device based on the speed, direction, and braking information.

[0048] FIG. 5 is a flowchart illustrating a method for generating driving control information in an autonomous driving device according to an embodiment of the present invention.

[0049] Referring to FIG. 5, the autonomous driving device collects image data, inertial data, and / or distance data from a camera, an inertial sensor, and a TOF sensor, and can preprocess and filter the collected sensor data. Subsequently, the autonomous driving device can detect and track objects by analyzing image data among the sensor data collected in step 511. Here, the object may be a moving body (e.g., a pedestrian, an animal, an object moved by wind, etc.), or a stationary object or a fixed obstacle. The autonomous driving device can detect, identify, and classify objects (static and dynamic obstacles). In step 513, the autonomous driving device can analyze the road environment by analyzing image data and / or inertial data from the sensor data. For example, the autonomous driving device can recognize road boundaries and lanes by analyzing image data, and can detect road gradient, driving speed, and direction by analyzing inertial data. In step 515, the autonomous driving device can analyze the distance to the detected object by analyzing image data and / or TOF data from the sensor data collected. For example, the autonomous driving device can calculate the distance to an object through stereo matching of image data and can measure the distance to an object by fusing TOF data and image data. Subsequently, the autonomous driving device can analyze the driving and road environment based on the information measured and calculated in step 517. For example, the autonomous driving device can analyze road conditions based on the results of object detection, road analysis, and distance measurement, and generate control information such as driving speed, direction, and braking according to the analyzed road conditions. Subsequently, the autonomous driving device can generate control information for the driving speed, steering, and braking of the autonomous driving device by fusing the first driving control information and the second driving control information in step 519.

[0050] Although the foregoing description has been made in relation to the preferred embodiments mentioned above, those skilled in the art will readily recognize that various other modifications and variations are possible without departing from the essence and scope of the invention, and it is obvious that all such changes and modifications fall within the scope of the appended claims.

Claims

Claim 1 A device comprising: a driving control module that generates first driving control information according to set path information; a sensor module equipped with a camera that acquires images according to driving; a processing module that determines obstacles, road conditions, and / or distances in the driving direction based on the acquired images, and determines second driving control information based on the determination result; an integrated driving control module that generates integrated driving control information for the speed, direction, and braking of a cart based on the first driving control information and the second driving control information; and a driving module that controls unmanned autonomous driving based on the integrated driving control information. Claim 2 A device according to claim 1, wherein the set path is set by a guide line embedded in the road, and the driving control module is equipped with left and right sensors that detect the guide line, and generates first driving control information that determines the driving direction of the autonomous driving device based on information measured by the left and right sensors. Claim 3 A device according to paragraph 2, wherein the sensor module further comprises an inertial sensor and / or a time of flight (TOF) sensor, wherein the inertial sensor generates road inclination, cart driving speed and direction information based on inertial information according to the driving of the vehicle, and the TOF sensor generates distance information to an obstacle in the driving direction. Claim 4 In paragraph 3, the processing module comprises: a collection unit that collects image data, inertial data, and distance data from the camera, inertial sensor, and TOF sensor; a preprocessing unit that preprocesses and filters the collected sensor data; a processing unit that detects an object from the image data, analyzes the road based on the image and inertial data, and measures the distance to the object based on the image data and distance data; and a judgment unit that analyzes the road condition based on the results of the object detection, road analysis, and distance measurement, and generates driving control information such as driving speed, direction, and braking according to the analyzed road condition. Claim 5 The device according to claim 4, wherein the processing unit includes: an object detection unit that analyzes image data to detect and track moving objects and detects, identifies, and classifies static and dynamic obstacles; an analysis unit that analyzes image data to recognize road boundaries and lanes and analyzes inertial data to detect road gradients, driving speeds, and directions; and a distance measuring unit that calculates the distance to an object through stereo matching of image data and fuses the distance data and image data to measure the distance to an object. Claim 6 A driving control method for an autonomous driving device that generates first driving control information based on a set driving path of a road, comprising: an information acquisition process for acquiring image data while in a state of unmanned autonomous driving; a processing process for determining obstacles, road conditions, and / or distances in the driving direction based on the acquired image information, and determining second driving control information based on the determination result; an integrated control process for generating integrated driving control information for driving speed, direction, and / or braking by fusing the first driving control information and the second driving control information; and a driving process for controlling unmanned autonomous driving based on the integrated driving control information. Claim 7 In claim 6, the above information acquisition process further comprises: a process of acquiring inertial information for calculating the road slope, the driving speed and direction of the cart; and a process of acquiring distance information for calculating the distance to an obstacle in the driving direction. Claim 8 In claim 7, the processing process comprises: a collection process for collecting image data, inertial data, and distance data from the camera, inertial sensor, and TOF sensor; a preprocessing process for preprocessing and filtering the collected sensor data; a processing process for detecting an object from the image data, analyzing the road based on the image and inertial data, and measuring the distance to the object based on the image data and distance data; and a judgment process for analyzing the road condition based on the results of the object detection, road analysis, and distance measurement, and generating driving control information such as driving speed, direction, and braking according to the analyzed road condition. Claim 9 A method according to claim 8, wherein the processing process comprises: an object detection process that analyzes image data to detect and track moving objects and detects, identifies, and classifies static and dynamic obstacles; an analysis process that analyzes image data to recognize road boundaries and lanes and analyzes road gradients by analyzing inertial data; and a distance calculation process that calculates the distance to an object through stereo matching of image data and calculates the distance to an object by fusing the distance data and image data.

Images