A system and method for autonomously performing work orders based on ground and surrounding recognition of an autonomous ai-based forklift

Abstract

The present invention relates to a ground and surroundings recognition and work order-based autonomous execution system and method for an autonomous AI-based forklift. In the ground and surroundings recognition and work order-based autonomous execution system for an autonomous AI-based forklift, the system comprises: a sensor unit that collects distance, image, inertia, load, and proximity data from a LiDAR sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor provided on the forklift and generates sensing data; a preprocessing unit that preprocesses the sensing data; a ground recognition unit that receives the preprocessed sensing data, analyzes the type and physical characteristics of the ground, and calculates a ground stability index; an object detection unit that detects objects including people and obstacles using the sensing data; a work data generation unit that receives one of natural language commands and field-based commands from a preset user terminal, identifies a work target and a work location, and generates work data; a work planning unit that generates a driving path and a work sequence corresponding to the ground stability index, object detection results, and work data; and an automatic forklift operation based on the collision and overturning risk calculated in response to the driving of the forklift. It is characterized by including a control unit that performs control.

Description

Technology Field

[0001] The present invention relates to an autonomous AI-based autonomous forklift system and method, and more specifically, to a ground and surrounding recognition and work order-based autonomous execution system and method of an autonomous AI-based forklift that is deployed at a designated work location, receives work instructions, recognizes ground characteristics and surrounding environment at the location using AI, interprets work orders, and performs work autonomously. Background Technology

[0003] Forklifts are key equipment used to perform loading, unloading, and movement of heavy loads in industrial sites and logistics warehouses, and as the complexity of the work environment and the frequency of operations increase, the risk of safety accidents continues to be raised.

[0004] In particular, the driving stability of forklifts is significantly affected by various factors such as changes in load capacity, ground slope and friction conditions, and the tilt of the vehicle body while driving, and there is a high possibility of collisions with workers or obstacles present in the work environment and rollover accidents.

[0005] Conventional forklift operation methods often rely on the skill level of the operator running the forklift for driving and work, or are limited to simple distance-based sensing or auxiliary control centered on fixed paths, and thus have limitations in that they cannot sufficiently reflect changes in ground conditions or dynamic working environments.

[0006] For example, although ground characteristics such as friction characteristics, slope, and unevenness have a direct impact on driving stability, technology to quantitatively evaluate them and reflect them in driving conditions has been applied only to a limited extent.

[0007] In addition, even when surrounding environment recognition technology is introduced, the method of determining collision risk based mainly on simple distance information to objects is used, so temporal factors such as the driving speed of the forklift or whether the object is moving are not sufficiently considered, and there was a problem in that it was difficult to make a comprehensive judgment on the risk of tipping over as well, as it relied on individual sensor signals or threshold comparisons.

[0008] As a result, there were limitations in predicting in advance situations where risk gradually increases due to changes in the work environment, or in flexibly adjusting driving strategies according to the risk level.

[0009] Meanwhile, regarding work instructions, the methods mainly involved the worker directly operating the forklift or inputting predefined simple commands, which required repeated intervention by the worker when the work object or work location changed, and there were limitations in mechanically understanding the work intent and autonomously performing the work.

[0010] In particular, technology for comprehensively interpreting natural language-based work instructions or work orders (work commands) in various input forms and reflecting them in driving and work planning has not been sufficiently developed.

[0011] Therefore, research is required on a ground and surroundings recognition and work order-based autonomous execution system and method for an autonomous AI-based forklift that comprehensively analyzes the type and physical characteristics of the ground and the dynamic behavior of the forklift to quantitatively evaluate driving stability, judges the risk of collision with the surrounding environment and overturning from a temporal and relative perspective, and autonomously interprets the contents of work orders to safely and efficiently control the overall driving and work execution. Prior art literature

[0013] Korean Registered Patent No. 10-2416201 The problem to be solved

[0014] The present invention aims to reduce the possibility of accidents occurring in response to forklift driving by comprehensively recognizing the surrounding environment and objects of the forklift using a plurality of sensors.

[0015] In addition, the purpose is to reduce the burden of manual input for workers and improve the convenience of the work instruction process by identifying work targets and work locations from received natural language and field-based commands.

[0016] In addition, the purpose is to ensure the driving stability of the forklift and improve work performance efficiency by selecting a driving path corresponding to the maximum driving speed and maximum allowable angle of inclination based on collected load data and calculated ground stability index. means of solving the problem

[0018] A ground and surroundings recognition and work order-based autonomous execution system for an autonomous AI-based forklift according to an embodiment of the present invention comprises: a sensor unit that collects distance, image, inertia, load, and proximity data from a LiDAR sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor provided in the forklift and generates sensing data; a preprocessing unit that preprocesses the sensing data; a ground recognition unit that receives the preprocessed sensing data, analyzes the type and physical characteristics of the ground, and calculates a ground stability index; an object detection unit that detects objects including people and obstacles using the sensing data; a work data generation unit that receives one of natural language commands and field-based commands from a preset user terminal, identifies a work target and a work location, and generates work data; a work planning unit that generates a driving path and a work sequence corresponding to the ground stability index, object detection results, and work data; and an automatic forklift operation based on the collision and overturning risk calculated in response to the driving of the forklift. It may include a control unit that performs control.

[0019] In addition, the ground recognition unit inputs point cloud data generated using distance data collected from the lidar sensor and ground image data collected from the camera sensor into a preset deep learning model to classify ground types, calculates a friction coefficient by applying a preset ground physical property parameter corresponding to the classified ground type, and calculates a ground stability index using the friction coefficient, the slope of the ground, and attitude information of the forklift collected from the inertia measuring device.

[0020] In addition, the above-mentioned work data generation unit includes a natural language analysis unit that performs morphological analysis in response to the input of a natural language command received from the above-mentioned user terminal, a work type classification unit that classifies work types in response to the morphological analysis results analyzed by the above-mentioned natural language analysis unit, an entity name extraction unit that extracts names corresponding to work objects and work locations from the above-mentioned natural language command, and a field value extraction unit that extracts field values ​​corresponding to work objects and work locations in response to the input of a field-based command received from the above-mentioned user terminal, and can generate work data by identifying work targets and work locations in response to the output results of the work type classification unit, the entity name extraction unit, and the field value extraction unit.

[0021] In addition, the above work planning unit can calculate the maximum driving speed and maximum allowable angle of inclination of the forklift using the ground stability index and load data, and determine a driving path less than the maximum driving speed and maximum allowable angle of inclination as the final driving path.

[0022] In addition, the method for autonomous execution based on ground and surrounding recognition and work orders of an autonomous AI-based forklift may include: a sensing data generation step for generating sensing data by collecting distance, image, inertia, load, and proximity data through a LiDAR sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor provided in the forklift; a preprocessing step for preprocessing the sensing data; a ground recognition step for analyzing the type and physical characteristics of the ground using the preprocessed sensing data and calculating a ground stability index; an object detection step for detecting objects including people and obstacles using the sensing data; a work data generation step for receiving one of a natural language command or a field-based command from a preset user terminal, identifying a work target and a work location in response to the command, and generating work data; a work planning step for generating a driving path and a work sequence in response to the ground stability index, object detection results, and work data; and a driving control step for performing automatic control of the forklift based on the collision and overturning risk calculated in response to the forklift driving. Effects of the invention

[0024] According to the present invention, by comprehensively recognizing the surrounding environment and objects of a forklift using a plurality of sensors, the possibility of accidents occurring in response to forklift driving can be reduced.

[0025] In addition, by identifying the work target and work location from received natural language and field-based commands, the burden of manual input for the operator can be reduced and the convenience of the work instruction process can be improved.

[0026] In addition, by selecting a driving path corresponding to the maximum driving speed and maximum allowable inclination angle based on collected load data and calculated ground stability index, the driving stability of the forklift can be secured and work performance efficiency can be improved. Brief explanation of the drawing

[0028] FIG. 1 is a block diagram illustrating a ground and surrounding recognition and work order-based autonomous execution system of an autonomous AI-based forklift according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an intermediate block diagram of a work data generation unit within a ground and surrounding recognition and work order-based autonomous execution system of an autonomous AI-based forklift according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating the ground and surroundings recognition and work order-based autonomous execution method of an autonomous AI-based forklift according to an embodiment of the present invention. Specific details for implementing the invention

[0029] Specific details regarding the problem to be solved, the means for solving the problem, and the effects of the invention as described above are included in the embodiments and drawings to be described below. The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings.

[0030] The scope of the present invention is not limited to the embodiments described below, and various modifications can be made by those skilled in the art within the scope of the technical essence of the present invention.

[0031] Hereinafter, the ground and surroundings recognition and work order-based autonomous execution system and method of the autonomous AI-based forklift of the present invention will be described in detail with reference to the attached FIGS. 1 to 3.

[0032] First, FIG. 1 is a block diagram illustrating a ground and surrounding recognition and work order-based autonomous execution system of an autonomous AI-based forklift according to an embodiment of the present invention, FIG. 2 is an intermediate block diagram illustrating a work data generation unit within a ground and surrounding recognition and work order-based autonomous execution system of an autonomous AI-based forklift according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a ground and surrounding recognition and work order-based autonomous execution method of an autonomous AI-based forklift according to an embodiment of the present invention.

[0033] Referring to FIG. 1, the ground and surroundings recognition and work order-based autonomous execution system (100) of an autonomous AI-based forklift according to one embodiment of the present invention may include a sensor unit (110), a preprocessing unit (120), a ground recognition unit (130), an object detection unit (140), a work data generation unit (150), a work planning unit (160), and a control unit (170).

[0034] The ground and surroundings recognition and work command-based autonomous execution system (100) of the high-autonomous AI-based forklift of the present invention recognizes the ground conditions and surrounding environment in real time while the forklift is driving and performing work, and receives and interprets work commands input from a user terminal, thereby enabling the forklift to autonomously perform the corresponding work.

[0035] The sensor unit (110) can collect distance, image, inertia, load, and proximity distance data from a lidar sensor, camera sensor, inertial measurement device, load sensor, proximity sensor, and ultrasonic sensor provided in the forklift and generate sensing data.

[0036] Here, the LiDAR sensor provided in the sensor unit (110) can be positioned at the front, side, or top of the forklift to measure distance information regarding the ground and surrounding objects and collect distance data. At this time, the distance data may include information on changes in ground height, slope, and unevenness, and may be converted into point cloud data in the ground recognition unit (130).

[0037] The camera sensor may be positioned at the front or side of the forklift to collect image data of the workspace. Here, the image data includes image data containing the surface condition of the ground, the shape of the work object, and visual features of people or obstacles, and may be transmitted to the ground recognition unit (130) and the object detection unit (140).

[0038] The above-described inertial measuring device can be mounted on the main body of the forklift to collect inertial data including acceleration, angular velocity, and attitude information. At this time, the inertial data is data representing changes in attitude, tilt, and dynamic behavior of the forklift while driving, and can be used to determine ground stability index and driving stability.

[0039] More specifically, the inertial measuring device can collect linear acceleration data for the forward / backward, left / right, and up / down directions of the forklift. Here, the linear acceleration data may include dynamic changes occurring during acceleration, deceleration, and braking of the forklift, as well as vibration characteristics occurring when driving on a surface with unevenness.

[0040] In addition, the inertial measuring device can collect angular velocity data representing the rotational motion of the forklift. At this time, the angular velocity data may include rotational speeds in the yaw, roll, and pitch directions of the forklift, and can convert the rotational behavior of the vehicle body that occurs when the forklift performs a rotational drive or drives on an inclined surface into data.

[0041] In addition, the inertial measuring device can calculate attitude information of the forklift based on the acceleration data and angular velocity data. Here, the attitude information includes the inclination angle, roll angle, or pitch angle of the forklift body and can be used as a basis for determining the state of the forklift, such as the state of the forklift driving on an inclined ground or the state of the body being tilted due to the influence of a load.

[0042] Meanwhile, since the above inertia data includes not only the value at a single point in time but also the amount of change over time, the dynamic behavior occurring during the operation of the forklift can be continuously tracked.

[0043] For example, when the forklift performs rapid acceleration or rapid turning, the inertia data can be used to display the possibility of reduced driving stability, including rapid changes in acceleration and angular velocity.

[0044] The load sensor described above may be provided on the fork or lift device of the forklift and may collect load data corresponding to the weight of the loaded cargo.

[0045] The above proximity sensor and ultrasonic sensor can be positioned at the front, side, or rear of the forklift to collect distance data with objects around the forklift.

[0046] Through this, the sensor unit (110) can collect and integrate distance data, image data, inertia data, load data, and proximity data collected from a plurality of sensors provided on the forklift to generate sensing data, and transmit the generated sensing data to the preprocessing unit (120).

[0047] The above preprocessing unit (120) can preprocess the sensing data.

[0048] More specifically, the preprocessing unit (120) can receive the sensing data and perform at least one of temporal and spatial alignment, noise removal, and data format standardization.

[0049] Here, the temporal alignment can be performed by performing synchronization processing based on the time information of sensing data collected from multiple sensors.

[0050] In other words, distance, image, inertia, load, and proximity data collected from different sensors at different times can be aligned to the same time.

[0051] Through this, it is possible to prevent misrecognition and judgment errors caused by temporal discrepancies between different sensor data in the subsequent components, the ground recognition unit (130), object detection unit (140), and work planning unit (160).

[0052] In addition, the spatial alignment mentioned above may mean converting sensing data collected from multiple sensors having different installation locations and coordinate systems into a single common coordinate system to perform alignment so that observation results for the same object or the same ground area correspond to each other.

[0053] More specifically, the spatial alignment can normalize (coordinate transform) point cloud data based on distance data collected from the LiDAR sensor, camera images, inertial data, and proximity data into a forklift reference coordinate system or a world coordinate system, based on a transformation matrix according to the sensor mounting position and orientation, which are external parameters defining the geometric relationship between the plurality of sensors, and the focal length, principal point, and distortion coefficient, which are internal parameters of the camera sensor.

[0054] For example, the preprocessing unit (120) can convert the coordinates of the point cloud data generated in a coordinate system based on distance data collected from the LiDAR sensor into the coordinate system of the forklift body, and project the converted point cloud data onto the camera image plane so that the pixel area of ​​the ground image and the ground point cloud data indicate the same ground area.

[0055] In addition, the spatial alignment may include calculating a correction transformation using a preset alignment algorithm to correct coordinate transformation errors that may occur due to sensor mounting errors, vehicle body vibrations, changes in load, etc.

[0056] For example, the preprocessing unit (120) can calculate the degree of discrepancy between the edge / texture features of the camera-based ground area and the ground plane, which is a reference plane generated from the LiDAR-based ground separation result, and update the rotation / translation parameters, which are correction transformations, so that the discrepancy is minimized.

[0057] As another example, the preprocessing unit (120) can improve the degree of input data alignment of the ground recognition unit (130) and object detection unit (140) by performing alignment between point cloud data of continuous frames, such as ICP-based alignment or visual-inertial fusion-based attitude correction, to reflect coordinate system variations corresponding to changes in the attitude of the forklift body.

[0058] In addition, the preprocessing unit (120) can perform noise removal included in the sensing data.

[0059] Here, the noise removal may include processing to ensure the reliability of the recognition accuracy and stability index calculation of the downstream module by reducing measurement noise of the sensor itself, LiDAR spikes / ghost points due to reflector characteristics, image noise due to changes in lighting, inertial data drift due to vehicle body vibration, and missing values ​​due to communication delay or packet loss.

[0060] More specifically, the preprocessing unit (120) can remove point cloud noise: the preprocessing unit (120) can remove point clouds unnecessary for ground recognition, such as floating points and reflection noise, by performing radius-based outlier removal, statistical outlier removal, downsampling such as voxel grid, and applying a preset height / slope threshold when separating the ground on the LiDAR-based point cloud.

[0061] In addition, for image noise removal, the preprocessing unit (120) can perform blurring / sharpening, illumination normalization, distortion correction, and region of interest (ROI) cropping in response to image data within the sensing data, thereby securing image quality that is advantageous for ground surface conditions and object contour extraction.

[0062] Additionally, in the case of removing noise from inertia, load, and proximity data, the preprocessing unit (120) can reduce vibration components and drift by applying preset filters such as a Kalman filter, a complementary filter, and a low-pass filter to the inertia data, and remove abrupt spikes by applying at least one of a moving average, a median filter, and outlier clipping to the load data and proximity data.

[0063] In addition, in the case of missing values ​​and delay correction, the preprocessing unit (120) may apply an interpolation method such as linear interpolation and maintaining recent values ​​or a reliability flag when sensor communication delay or temporary missing values ​​occur, so that the subsequent module can perform inference considering the missing value interval.

[0064] In addition, the preprocessing unit (120) can perform data processing standardization.

[0065] Here, the data processing standardization may mean unifying different sensor output formats and units into a common schema so that the ground recognition unit (130), object detection unit (140), work planning unit (160), and control unit (170) can use the sensing data through a consistent input interface.

[0066] More specifically, the above data processing standardization can perform unit and range standardization, data structure standardization, coordinate and reference system standardization, and quality indicator addition.

[0067] Here, the unit and range standardization can stabilize the model input distribution by unifying distance data, speed and acceleration, angular velocity, angle, load, etc. within the sensing data into a preset unit system and normalizing the value range for each sensor.

[0068] In addition, the above data structure standardization can be encapsulated into a data packet structure including point cloud (point coordinates, reflection intensity, timestamp), image frame (resolution, frame ID, correction parameter), inertial data (acceleration / angular velocity / attitude, covariance), load and proximity data (measurement value, reliability, sensor ID), etc.

[0069] In addition, the above coordinate and reference system standardization can unify the sensing data into either the forklift reference coordinate system or the world coordinate system of the work environment for storage and transmission, and add a coordinate system identifier and transformation history (correction matrix version, etc.) as metadata.

[0070] The above quality indicator addition allows for recording reliability scores, outliers, missing values, and timestamp deviations, which are synchronization errors, as quality indicators for each data, so that the ground recognition unit (130) or work planning unit (160) can adjust weights or perform conservative control according to the input quality.

[0071] In this way, the preprocessing unit (120) can perform temporal and spatial alignment to secure the degree of alignment of sensor fusion-based recognition, and at the same time reduce the possibility of errors and misrecognition in the next configuration through noise removal and data processing standardization.

[0072] The above ground recognition unit (130) can receive preprocessed sensing data, analyze the type and physical characteristics of the ground, and calculate a ground stability index.

[0073] That is, the ground recognition unit (130) can perform ground type classification by inputting point cloud data generated using distance data collected from the lidar sensor and ground image data collected from the camera sensor into a preset deep learning model.

[0074] The ground recognition unit (130) can generate point cloud data that represents the distance to the ground, height change, and degree of inclination as a set of three-dimensional points using distance data collected from a lidar sensor, and generate a two-dimensional image including the color, texture, reflection characteristics, contamination state, cracks, and surface pattern of the ground using ground image data collected from a camera sensor.

[0075] Afterwards, the ground recognition unit (130) can set ground candidate areas using the point cloud data and ground image data.

[0076] Here, the setting of the ground candidate region may include the step of separating a point cloud estimated to be the ground from point cloud data and setting an area determined to be the ground in ground image data as a region of interest.

[0077] In addition, the ground recognition unit (130) can generate feature information for the ground candidate area.

[0078] At this time, the generation of feature information may include the step of generating flatness, degree of unevenness, slope, and point cloud density of the ground surface as feature quantities from point cloud data, and generating color distribution, texture characteristics, reflectance, and pattern repeatability as feature information from ground image data.

[0079] Afterwards, the ground recognition unit (130) can classify the ground type into one of a plurality of candidates by inputting the point cloud data, ground image data, and feature information into a preset deep learning model.

[0080] Here, the deep learning model may include at least one of a convolutional neural network (CNN), a point cloud-only neural network (PointNet family), or a combination thereof, and may be configured to determine the ground type by comprehensively learning the surface roughness, reflection intensity distribution, height change pattern, texture, and color features of the ground.

[0081] For example, the ground recognition unit (130) can classify the ground into at least one of asphalt, concrete, steel plate, soil floor, gravel, wet ground, or frozen ground by inputting flatness, slope, irregularity frequency, and point cloud density extracted from point cloud data, and color histogram, brightness contrast, texture directionality, and pattern repeatability extracted from ground image data into a deep learning model.

[0082] In this way, the ground recognition unit (130) can perform ground classification using fusion data that fuses the point cloud data and ground image data collected from LiDAR and camera sensors, rather than a single sensor-based system.

[0083] In addition, the ground recognition unit (130) can calculate a friction coefficient by applying a stored ground physical property parameter corresponding to the classified ground type.

[0084] Here, the above ground material property parameter is the reference friction coefficient (μ) pre-stored for each ground type according to [Table 1] below. ref ), surface roughness / wear correction factor (k r ) and wetting correction factor (k w Can save ).

[0085] [Table 1]

[0086]

[0087] At this time, the above ground material property parameters are set based on experimental data, simulation results, empirical values, or a combination thereof, and can be stored so as to be updated during the system initial setup phase or maintenance process.

[0088] Through this, the ground recognition unit (130) can calculate a friction coefficient (μ) corresponding to the actual driving environment by combining the stored ground physical property parameters according to the following [Equation 1].

[0089] [Mathematical Formula 1]

[0090]

[0091] (here, μ ref is the standard friction coefficient corresponding to the classified ground type, k r is the correction coefficient based on the state, k w represents a correction factor based on the wet or contaminated state)

[0092] At this time, the presence of the above-mentioned wear traces is determined based on at least one of the frequency of cracks in the ground image, the non-uniformity of the surface texture, changes in brightness contrast, or reflection characteristics, and the above-mentioned wet state can be determined using at least one of the decrease in LiDAR reflection intensity, changes in image brightness distribution, increases in reflection patterns, or changes in color saturation.

[0093] For example, if the ground is classified as 'concrete' by the ground recognition unit (130) above, the standard friction coefficient (μ ref ) can be set to 0.75. In addition, if the image analysis results determine that wear marks exist on the ground surface, the roughness correction factor (k r ) is 0.9, and if the ground is determined to be in a wet state based on the analysis of LiDAR reflection intensity and image brightness, the wetness correction coefficient (k w ) is set to 0.8, so the friction coefficient (μ) can be calculated as 0.54 (=0.75*0.9*0.8).

[0094] In this way, the ground recognition unit (130) can have the effect of more accurately reflecting the actual driving environment by using a friction coefficient that is dynamically calculated in response to changes in ground conditions, rather than a simple fixed friction coefficient.

[0095] That is, even for the same ground type, the friction coefficient (μ) can be calculated differently in response to changes in surface condition or wet condition and can be reflected in the driving stability evaluation.

[0096] Afterwards, the ground recognition unit (130) can calculate the ground stability index (SI) according to the following [Equation 2] using the friction coefficient, the slope of the ground, and the attitude information of the forklift collected from the inertia measuring device.

[0097] [Mathematical Formula 2]

[0098]

[0099] (Here, μ represents the coefficient of friction, θ represents the ground inclination angle, and φ represents the roll / pitch angle of the forklift body calculated from the inertia measurement device.)

[0100] For example, when the friction coefficient (μ) is 0.54, the ground inclination angle (θ) is 10˚, and the pitch angle (φ) of the forklift is 5˚, the ground stability index (SI) can be calculated as approximately 2.02 ((0.54*cos10˚) / (sin10˚+tan5˚)).

[0101] Meanwhile, in the above [Equation 2], the denominator (sinθ+tanφ) represents a stable element that suppresses the slipping of the forklift in response to friction, and the numerator (μ*cosθ) represents an unstable element that causes overturning and slipping due to ground slope and vehicle body tilt.

[0102] Accordingly, the above ground stability index (SI) may indicate a state where driving stability is higher the larger the value.

[0103] Meanwhile, the above ground inclination angle (θ) can be calculated using a point group determined to be the ground in the point cloud data generated from the lidar sensor.

[0104] For example, the ground recognition unit (130) can estimate (fit) a ground plane corresponding to a ground point group and calculate a ground inclination angle (θ) using the normal vector of the estimated ground plane and the inclination relationship with respect to a reference horizontal plane in the direction of gravity.

[0105] In addition, the roll / pitch angle (φ) of the forklift body calculated from the inertial measuring device may represent attitude data that expresses the degree to which the forklift body is tilted in the left-right direction or the front-back direction as an angle.

[0106] More specifically, the roll angle is an angle indicating the degree to which the forklift body is tilted in the lateral direction, left and right, and if the forklift is tilted to the left, the roll angle can change to a positive (+) or negative (-) value. In addition, the pitch angle is an angle indicating the degree to which the forklift body is tilted in the longitudinal direction, front and back, and if the forklift travels uphill or the front descends due to the movement of the load, the pitch angle can change.

[0107] Meanwhile, the inertial measuring device includes an acceleration sensor and a gyroscope sensor, wherein the acceleration sensor measures linear acceleration including a gravitational component acting on the forklift body, and the gyroscope sensor measures angular velocity, which is the rotational speed of the forklift body.

[0108] At this time, the inertial measuring device can estimate the attitude of the forklift using the acceleration and angular velocity data and calculate the roll / pitch angle based thereon.

[0109] That is, the inertial measuring device can measure a 3-axis acceleration value including a gravitational component acting on the forklift body through an acceleration sensor, and measure a 3-axis angular velocity value of the forklift body through a gyroscope sensor.

[0110] Through this, the inertial measuring device obtains an initial roll angle estimate (φ) according to the following [Equation 3] to [Equation 4]. r ) and initial pitch angle estimate (φ p ) can be produced.

[0111] [Mathematical Formula 3]

[0112]

[0113] (here, a x is the x-axis acceleration value, a y is the y-axis acceleration value, a z represents the z-axis acceleration value)

[0114] [Mathematical Formula 4]

[0115]

[0116] At this time, the ground recognition unit (130) is the initial roll angle estimate (φ) calculated from the inertial measuring device. r ) and initial pitch angle estimate (φ p The value with the larger absolute value among ) can be reflected in the above ground stability index (SI) as the roll / pitch angle (φ) of the forklift body calculated from the inertia measuring device.

[0117] For example, the x-axis acceleration value (a) measured from the above acceleration sensor x ) is 0.52g, y-axis acceleration value (a y ) is 0g, z-axis acceleration value (a z When ) is 0.85g, the initial estimated roll angle (φ) of the forklift r ) is approximately 0˚(=tan -1 (0 / √(0.52 2 +0.85 2 ))), initial pitch angle estimate (φ p ) is approximately -31˚(=-0.52 / √(0 2 +0.85 2 It can be calculated as )). This means that the forklift body is hardly tilted in the left-right direction, but is tilted approximately 31 degrees in the forward or rear direction, and the initial pitch angle estimate (φ p This can be reflected in the calculation of the above ground stability index (SI).

[0118] Meanwhile, the ground recognition unit (130) can determine that the forklift is in a state where stable driving is possible when the calculated ground stability index (SI) exceeds a preset threshold, and when it is below the preset threshold, it can transmit a signal to the work planning unit (160) and the control unit (170) to perform driving speed reduction, detour route setting, and work stoppage control commands.

[0119] In this way, the ground recognition unit (130) can be used as an indicator that serves as a basis for predicting the risk of overturning and slipping that may occur during autonomous driving of the forklift and for performing control to prevent such accidents by calculating a ground stability index (SI) that is calculated as a quantitative value by comprehensively reflecting ground characteristics, forklift attitude, and driving environment.

[0120] The object detection unit (140) can detect objects including people and obstacles using the sensing data.

[0121] More specifically, the object detection unit (140) can determine the presence and location of an object located near the driving path of the forklift and a preset distance by using camera image data included in the sensing data, LiDAR-based distance data, and distance data collected from a proximity sensor and an ultrasonic sensor.

[0122] Here, the object may include a person including a worker, a pallet, a load, a rack, a wall, a vehicle, a fixed object and a moving object within the work environment.

[0123] In addition, the object detection unit (140) can determine whether the object is an obstacle affecting the driving or operation of the forklift when an object is detected from the sensing data, and can calculate the relative position of the detected object or distance information from the forklift.

[0124] At this time, the object detection unit (140) includes a determination of whether the object is in a stationary state or a moving state, and if it is determined to be a moving object, it may consider whether it is in a relative approach to the forklift.

[0125] Here, the object detection unit (140) can determine whether the object is moving by calculating the change in object position or relative speed between consecutive frames.

[0126] Meanwhile, the object detection result output from the object detection unit (140) can be used by the work planning unit (160) to generate or modify a driving path, or by the control unit (170) to perform deceleration, stopping, or avoidance control of the forklift.

[0127] In this way, the object detection unit (140) can use the sensing data to recognize people and obstacles present in the surrounding environment of the forklift, thereby reducing the risk of collision that may occur during the autonomous driving and work performance of the forklift.

[0128] The above-mentioned work data generation unit (150) can identify a work target and a work location by receiving one of a natural language command and a field-based command from a preset user terminal.

[0129] More specifically, the above-mentioned work data generation unit (150) can serve as an interface that autonomously understands and executes work instructions input by a user, even without a person directly executing operation commands on the forklift.

[0130] At this time, the pre-configured user terminal may refer to a terminal such as a PC, laptop, smartphone, or tablet PC capable of screen output, input, and wired and wireless communication.

[0131] Here, the above-mentioned work data generation unit (150) is explained in more detail with reference to FIG. 2.

[0132] Referring to FIG. 2, the above-mentioned work data generation unit (150) may include a natural language analysis unit (151), a work type classification unit (152), an entity name extraction unit (153), and a field value extraction unit (154).

[0133] The above natural language analysis unit (151) can perform morphological analysis in response to the input of a natural language command received from the above-configured user terminal.

[0134] More specifically, the natural language analysis unit (151) can perform morphological analysis in response to the input of the natural language command when a natural language command is received from the preset user terminal.

[0135] Here, the morphological analysis is a process of breaking down a sentence into words or semantic units, and may refer to a preprocessing step for distinguishing action expressions, target expressions, and location expressions included within a command sentence.

[0136] The above work data generation unit (152) can classify work types in correspondence with the morphological analysis results analyzed by the above natural language analysis unit (151).

[0137] Here, the task type classification unit (153) can classify the type of task instructed by the user by using the morphological analysis results analyzed by the natural language analysis unit (151).

[0138] More specifically, the above-mentioned work type classification unit (152) can determine that the work is one of loading, unloading, moving, waiting, or driving to a specific location if a word matching a preset keyword is included among the action-related expressions included in the above-mentioned natural language command.

[0139] For example, the above-mentioned work type classification unit (152) may determine the corresponding work as a loading work if the natural language command includes action-related expressions such as 'loading', 'loading', 'lifting', 'lifting', etc., and may classify the corresponding work as an unloading work if the natural language command includes expressions such as 'unloading', 'lowering', 'putting down', 'unloading', etc. Additionally, the above-mentioned work type classification unit (152) may determine the corresponding work as a moving work if the natural language command includes expressions such as 'moving', 'transporting', 'taking', 'transporting', etc., and may classify the corresponding work as a waiting work if expressions such as 'waiting', 'stopping', 'stopping', 'waiting', etc. are included. Additionally, the above-mentioned work type classification unit (152) can determine that the task is a driving task to a specific location if the above-mentioned natural language command includes expressions indicating a specific location, such as 'go forward', 'move to location A', 'go to location B', or 'drive to location C'. That is, if the user inputs a natural language command such as 'move the pallet to rack 3', the above-mentioned work type classification unit (152) can classify the task as a moving task by recognizing the action-related expression 'move' as a pre-stored movement-related keyword. As another example, if the user inputs a natural language command such as 'lower the fork and unload', the above-mentioned work type classification unit (152) can determine that the task is an unloading task by recognizing the expression 'unload' as a unloading-related keyword.

[0140] In this way, the work type classification unit (152) classifies the type of work by comparing the action-related expression included in the natural language command with a preset work type keyword, so that the work data generation unit (150) can classify the user's work instruction into a work form that can be performed by a forklift.

[0141] The above entity name extraction unit (153) can extract names corresponding to work objects and work locations from the above natural language command.

[0142] More specifically, the named entity extraction unit (153) can identify expressions corresponding to nouns or noun phrases among the words included in the natural language command using the morphological analysis results analyzed by the natural language analysis unit (151), and determine the expressions that match the stored work target and work location words as work target or work location.

[0143] Here, the previously stored work object and work location words may include words referring to objects handled by a forklift, such as 'pallet', 'pallet', 'box', 'container', 'drum', 'load', etc., and the location words may include words referring to work locations, such as 'rack 1', 'rack 3', 'area A', 'line B', 'shipping area', 'receiving area', 'waiting area', etc.

[0144] Additionally, the entity name extraction unit (153) can determine that the word included in the natural language command does not match the previously stored word, but corresponds to a similar spelling or synonym, by mapping it to the same entity.

[0145] For example, assuming that 'pallet' and 'pallet' refer to the same work object even if their notations differ, the entity name extraction unit (153) can be configured to treat them as the same entity. Additionally, expressions that may be used interchangeably within the workplace, such as 'outbound place' and 'shipping place', can also be determined to be the same or related location according to pre-registered mapping information.

[0146] In addition, the entity name extraction unit (153) can distinguish between a work object and a work location by determining which category each noun or noun phrase belongs to when a natural language command includes multiple nouns or noun phrases.

[0147] For example, when a natural language command such as ‘move the pallet to rack 3’ is entered, the entity name extraction unit (153) can determine the work object by matching ‘pallet’ with the object dictionary and determine the work location by matching ‘rack 3’ with the location dictionary.

[0148] In addition, the entity name extraction unit (153) can recognize and extract a sequence of nouns or a modifier-noun combination form as a single entity name from the morphological analysis result, even if the extracted work object or work location is a noun phrase composed of multiple words rather than a single word.

[0149] For example, 'Rack No. 3', 'Right Shipping Area', 'End of Line A', etc., can be combined into a single work location and extracted even if they consist of multiple words.

[0150] Meanwhile, the entity name extraction unit (153) can supplement the extraction result according to a preset correction rule when the work object or work location is not clearly specified in the natural language command.

[0151] For example, if a location expression is omitted in a natural language command, one of the default location value received from the pre-configured user terminal and the current location of the forklift is assumed to be the work location, and if an object expression is omitted, an object with high priority in the recent work history or work environment can be set as a candidate.

[0152] Here, the correction rule may refer to a criterion applied when the work object or work location is not clearly specified from the natural language command.

[0153] For example, if the work location is not included in the natural language command, the current location of the forklift, the location where the previous work was performed, and one of the preset default work zones may be corrected to be set as the work location; and if the work object is unclear, the work object may be corrected to be set as one of the object currently loaded on the forks of the forklift, the object used in the recent work history, and one of the preset representative object. Additionally, if multiple work objects or multiple work locations are extracted from the natural language command, the work object and work location may be corrected to be selected in correspondence with the order of mention in the natural language command, the association with the work type, and the preset priority.

[0154] At this time, the correction rule can be set and changed by the user using the pre-configured user terminal and management interface.

[0155] Through this, the named entity extraction unit (153) can identify a noun or noun phrase from the result of morphological analysis based on natural language commands and then match it with the stored object and location words to extract the work object and work location.

[0156] The field value extraction unit (154) can extract field values ​​corresponding to a work object and a work location in response to the input of a field-based command received from the preset user terminal.

[0157] More specifically, the field value extraction unit (154) can extract field values ​​corresponding to the work type, work target, and work location from a structured field-based command received from the preset user terminal.

[0158] Here, the field-based command may refer to a command generated by a user entering or selecting a value into a predefined item, rather than in the form of natural language.

[0159] For example, when a field-based command is received in which 'movement' is entered or selected in the work type field, 'pallet' in the work object field, and 'rack 3' in the work location field, the field value extraction unit (154) can identify the work object and work location by extracting the corresponding field values ​​as they are.

[0160] Through this, the field value extraction unit (154) can recognize the type of work, the work object, and the work location, etc., from a field-based command with a structured input format without a natural language interpretation process.

[0161] Additionally, the field value extraction unit (154) checks whether the value included in the field-based command matches the previously stored work environment information, and if it does not match, it can perform at least one of applying a preset correction rule and requesting re-input to the preset user terminal.

[0162] Here, the above-mentioned stored work environment information refers to basic information about the workspace where the forklift is operated, and may include the type of work object, the name and location identification information of the work location, the configuration of the work area, and identification information for racks, areas and lines, etc.

[0163] This work environment information can be registered in advance through a user terminal or management system, or updated according to changes in the work environment.

[0164] In addition, the above-mentioned preset correction rule may refer to a criterion for correcting a value when the value included in the field-based command does not exactly match the above-mentioned work environment information.

[0165] More specifically, if an input field value differs slightly in spelling or notation from a previously stored name but is semantically identical, the corresponding name can be automatically mapped and corrected.

[0166] For example, if 'Palette' is entered as a field value, it can be corrected by mapping it to the pre-stored work object name 'Palette'.

[0167] In addition, if the input field value is similar to multiple previously stored names and it is difficult to identify it as a single one, it can be configured to select one candidate based on pre-set priority criteria.

[0168] For example, if a field value of 'Area A' is entered but multiple location candidates exist, such as 'Area A-1' and 'Area A-2', the location can be corrected to the final working location based on one of the priorities: the default working area, the recently used location, or the priority.

[0169] In this way, the field value extraction unit (154) can identify the work target and work location using the structured input value included in the field-based command, thereby supporting the work data generation unit (150) to reliably interpret the user command.

[0170] Meanwhile, the above-mentioned work data generation unit (150) can identify a work target and a work location in response to the output results of the above-mentioned work type classification unit (152), entity name extraction unit (153), and field value extraction unit (154) and generate work data.

[0171] More specifically, the above-described work data generation unit (150) can generate work data by determining the work target and work location of the work to be performed by the forklift, by combining work type information output from the above-described work type classification unit (152) and work target information output from the above-described entity name extraction unit (153) when the work command received from the above-described user terminal is a natural language command, or by combining field value information output from the above-described field value extraction unit (154) when the received work command is a field-based command.

[0172] For example, when a natural language command is received from the preset user terminal, the work data generation unit (150) can identify the work target and work location using the output result of the entity name extraction unit (153) and determine the type of the corresponding work using the output result of the work type classification unit (152). On the other hand, when a field-based command is received from the preset user terminal, the work data generation unit (150) can identify the work target and work location using the output result of the field value extraction unit (154) and simultaneously refer to the output result of the work type classification unit (152) or the work type value included in the field-based command.

[0173] Additionally, the above-mentioned work data generation unit (150) may be configured to select one in response to a preset priority or correction rule when a natural language command-based output result and a field-based command-based output result exist simultaneously or are mutually inconsistent, or to combine the two results to identify the final work target and work location.

[0174] For example, the field-based command may be applied first because it is a structured value, and the natural language command may be referenced secondarily, or conversely, the natural language command may be configured to be applied first according to user settings.

[0175] In addition, the work data generation unit (150) can check whether the identified work target and work location are valid targets and locations registered in the work environment information, and if they are not valid, apply the preset correction rule or request re-input via the user terminal.

[0176] Referring again to FIG. 1, the work planning unit (160) can generate a driving path and a work sequence in response to the ground stability index, object detection results and work data.

[0177] More specifically, the work planning unit (160) can set the starting point and destination of the forklift based on the work target and work location included in the work data, and generate a candidate path that can be driven by considering the ground stability index (SI) and object detection results.

[0178] At this time, the above driving path candidate is selected as one of a plurality of paths within the work environment, and each path can be evaluated by reflecting ground conditions, obstacle distribution, and work efficiency.

[0179] In addition, the above work planning unit (160) can generate a work sequence by using the object detection result to avoid sections where people or obstacles exist, or by adjusting the order of passage through the sections, so as to reduce the risk of collision during the work.

[0180] For example, if a person moving along the work path is detected, the path can be modified to bypass that section, or the work order can be adjusted by placing the task at a lower priority so that it is performed after waiting.

[0181] Meanwhile, the above work planning unit (160) can calculate the maximum driving speed and maximum allowable angle of inclination of the forklift using the ground stability index and load data.

[0182] More specifically, the above work planning unit (160) can set driving constraint conditions by simultaneously reflecting the ground stability index (SI) and load data, as the lower the ground stability index (SI), the greater the risk of ground slip or overturning, and the greater the load, the lower the driving stability due to changes in the inertia and center of gravity of the vehicle body.

[0183] To this end, the above work planning unit (160) has a maximum driving speed (V) corresponding to the ground stability index (SI) and load data. max ) and maximum allowable slope angle (θ max) can be calculated according to [Equation 5] to [Equation 6] below.

[0184] [Mathematical Formula 5]

[0185]

[0186] (Here, V b is the standard driving speed, SI r is the standard ground stability index, k v is the speed weighting factor, W is the load, W r represents the standard load.

[0187] [Mathematical Formula 6]

[0188]

[0189] (here, θ b is the standard allowable slope angle, k θ represents the slope angle weighting factor)

[0190] At this time, the maximum driving speed (V) max ) and maximum allowable slope angle (θ max The above [Equations 5] to [Equations 6], which calculate the ), are configured such that the driving speed and the allowable angle of inclination decrease simultaneously as the ground stability index (SI) decreases, and the speed and angle of inclination are further limited in proportion to the load, thereby allowing for a maximum allowable speed (V) according to ground and loading conditions. max ) and maximum allowable slope angle (θ max ) may be restricted.

[0191] For example, the ground stability index (SI) calculated by the ground recognition unit (130) is 1.61, and the preset ground stability index (SI r ) is 2, the load (W) of the forklift collected from the load sensor in the sensor unit (110) is 1,200 kg, the reference load (Wr) is 2,000 kg, and the reference driving speed (V b ) is 3 m / s, standard allowable slope angle (θ b ) is 12˚, velocity weighting factor (k vWhen ) is 0.5 and the slope angle weighting factor (kθ) is 0.4, the maximum driving speed (V max ) is, 1.6905(=3*min(1, 1.61 / 2)*(1-0.5*min(1,1200 / 2000)))m / s, maximum allowable slope angle(θ max ) can be calculated as 7.3416(=12*min(1, 1.61 / 2)*(1-0.4*min(1,1200 / 2000)))˚.

[0192] Through this, the work planning unit (160) can determine the final driving path among a plurality of driving path candidates, a driving path with a maximum driving speed and maximum allowable angle of inclination less than the calculated maximum driving speed.

[0193] Meanwhile, the above-mentioned work planning unit (160) can generate a driving path candidate by generating graph-based map or grid-based map data that represents driving sections within a work environment as nodes and edges when generating a plurality of driving path candidates, or by using a pre-set path search algorithm by referring to the data stored in advance.

[0194] Here, the preset path search algorithm can generate multiple driving path candidates connecting a starting point to a destination by using at least one of the path search algorithms including the Dijkstra algorithm, A* (A-star) algorithm, D* (D-star) algorithm, RRT (Rapidly-exploring Random Tree) based algorithm and PRM (Probabilistic Roadmap) based algorithm.

[0195] Afterwards, the above-mentioned work planning unit (160) can generate path data including slope angle information corresponding to each path section by configuring each generated driving path candidate with a plurality of path sections and waypoints.

[0196] At this time, the inclination angle information may be calculated based on a point cloud generated from the actual path driving of the forklift including the sensor unit (110) and a ground model received from the ground recognition unit (130), or may be calculated from elevation information included in the map data of the corresponding work environment.

[0197] Through this, the above-mentioned work planning unit (160) can include slope angle information for each path segment corresponding to each generated driving path candidate as an attribute value.

[0198] Subsequently, the above-mentioned work planning unit (160) collects the maximum value among the slope angle information for each path segment included in each driving path candidate to calculate the slope angle information of the corresponding path candidate, and uses this as the maximum allowable slope angle (θ max It is possible to determine whether driving is possible by comparing with ).

[0199] Additionally, the above-mentioned work planning unit (160) calculates or collects the required driving speed required for each driving path candidate and the maximum driving speed (V max It is possible to determine whether the speed constraint is satisfied by comparing with ).

[0200] Through this, the work planning unit (160) determines only the driving path that satisfies all conditions of the calculated maximum driving speed and maximum allowable inclination angle among a plurality of driving path candidates as a valid path, and if a plurality of valid paths are calculated, the valid path with a low work distance and work time and high safety can be determined as the final driving path.

[0201] In this way, the above-mentioned work planning unit (160) can generate a candidate driving path using a known path generation algorithm, and compare and verify the maximum driving speed and maximum allowable angle of inclination calculated using ground stability index and load data with the angle of inclination information and required speed information of the driving path to generate a safe final driving path that reflects the ground condition of the candidate driving path and the loading condition of the forklift scheduled to drive.

[0202] Meanwhile, the above-mentioned work planning unit (160) may be configured to determine the order of work to be performed in correspondence with the above-mentioned final driving path, and to re-plan the above-mentioned final driving path and work order if the ground stability index (SI) or object detection result changes during the driving of the above-mentioned forklift.

[0203] The control unit (170) can perform automatic control of the forklift based on the collision and overturning risk calculated in response to the driving of the forklift.

[0204] More specifically, the control unit (170) controls the driving of the forklift according to the driving path set by the work planning unit (160), and at the same time uses sensing data collected in real time during driving to determine the collision risk (R c ) and conduction risk (R t ) can be calculated according to [Equation 7] to [Equation 8] below.

[0205] [Mathematical Formula 7]

[0206]

[0207] (Here, T r is the reference collision time, V r ε represents the relative speed between the forklift and the object, ε is a preset infinitesimal value, and d represents the distance between the forklift and the object.

[0208] For example, the reference collision time (T r ) is 3s, the distance between the forklift and the object (d) is 2.4m, and the relative speed between the forklift and the object (Vr When ) is 1.2 m / s and the preset infinitesimal value (ε) is 0.01 m / s, the collision risk (R c ) can be calculated as 1(=min(1,3*max(1.2, 0.01) / 2.4)).

[0209] Through this, the control unit (170) determines the risk of collision during the forklift's operation not merely based on the distance from the object, but in correspondence with the forklift's driving speed, whether the object is moving, and the relative approach speed, the remaining time term (d / max(V)) until collision between the forklift and the object. r By calculating the collision risk based on ε), distance and speed information can be integrated into a time concept to determine the collision risk as a quantitative value.

[0210] In other words, [Equation 7] quantifies the risk of collision that may occur during forklift operation in terms of time margin, and enables rapid and stable control decisions such as deceleration and stopping.

[0211] [Mathematical Formula 8]

[0212]

[0213] (Here, θ is the ground slope angle, θ max θ is the maximum allowable inclination angle, φ is the forklift body tilt angle, φ r is the standard body tilt angle, a l is lateral acceleration, a r represents the reference lateral acceleration)

[0214] For example, the above conduction risk (R t ) is that the ground inclination angle (θ) collected from the inertial measuring device provided in the forklift is 6˚, and the maximum allowable inclination angle (θ max ) is 8˚, body inclination angle (φ) is 7˚, reference body inclination angle (φ r ) is 10˚, lateral acceleration(a l ) is 1.6 m / s 2 , reference lateral acceleration(a r ) is 2 m / s 2In this case, the conduction risk (Rt) can be calculated as 0.8 (=max(6 / 8, 7 / 10, 1.6 / 2)).

[0215] Through the above process, the control unit (170) can reflect the ground inclination angle, body tilt angle, and lateral acceleration as factors that contribute most significantly to the overturning in response to the operation of the forklift in response to the overturning of the forklift, and calculate the risk of overturning by comparing the percentage of the collected values ​​with the preset reference value and corresponding to the highest item.

[0216] Through this, the control unit (170) can quantify the risk of conduction by reflecting the condition that has the greatest influence on the risk of conduction among the various dynamic states input from multiple sensors.

[0217] Subsequently, the control unit (170) calculates the collision risk (R) c ) and conduction risk (R t By comparing each risk level with a preset threshold, if the risk level is less than the preset first threshold, normal driving is maintained; if it exceeds the preset first threshold but is less than the preset second threshold, it switches to a deceleration or low-speed driving mode; and if the risk level exceeds the preset second threshold, automatic stop or driving stop control can be performed.

[0218] At this time, the first threshold value set above may be set to a value less than the second threshold value.

[0219] Through the above process, the ground and surroundings recognition and work order-based autonomous execution system (100) of the above-described autonomous AI-based forklift can collect and preprocess sensing data from sensors provided inside and outside the forklift, recognize ground conditions and surrounding objects in the ground recognition unit (130), calculate driving conditions according to ground stability and work order in the work planning unit (160), and calculate collision risk and overturning risk in the control unit (170) and compare them with threshold values, thereby controlling the entire driving and work process to be performed autonomously according to the work order.

[0220] Below, a method (300) for autonomous driving AI-based forklift ground and surrounding area recognition and work order-based autonomous execution, which includes processing performed in stages in each component of the autonomous driving AI-based forklift ground and surrounding area recognition and work order-based autonomous execution system (100), will be described in detail.

[0221] Here, the ground and surrounding recognition and work order-based autonomous execution method (300) of the above-mentioned autonomous AI-based forklift is explained in more detail with reference to FIG. 3.

[0222] Referring to FIG. 3, the ground and surroundings recognition and work order-based autonomous execution method (300) of the above-described autonomous AI-based forklift includes a sensing data generation step (310), a preprocessing step (320), a ground recognition step (330), an object detection step (340), a work command recognition step (350), a work planning step (360), a final driving path generation step (370), and an autonomous driving control step (380).

[0223] The above sensing data generation step (310) can generate sensing data by collecting distance, image, inertia, load, and proximity distance data through a lidar sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor provided in the forklift.

[0224] At this time, the sensing data may include distance data including distance to the ground, height change, and inclination information collected from a lidar sensor; image data including ground surface condition and object shape information collected from a camera sensor; inertial data including acceleration, angular velocity, and attitude information collected from an inertial measurement device; load data corresponding to the load weight collected from a load sensor; and distance data to objects around the forklift collected from a proximity sensor and an ultrasonic sensor.

[0225] The above preprocessing step (320) can preprocess the sensing data.

[0226] More specifically, the preprocessing step (320) can receive the sensing data generated in the sensing data generation step (310) and perform temporal alignment, spatial alignment, noise removal, and data processing standardization.

[0227] Here, the preprocessing step (320) can perform temporal alignment by synchronizing sensing data collected at different times from different sensors based on the same time standard, and spatial alignment by converting sensor data having different coordinate systems into a forklift reference coordinate system or a world coordinate system.

[0228] Additionally, the preprocessing step (320) can perform outlier removal for the LiDAR-based point cloud collected from the sensing data, illumination correction and distortion correction for the image data, and filtering for the inertial data.

[0229] The above ground recognition step (330) can analyze the type and physical characteristics of the ground using preprocessed sensing data and calculate a ground stability index.

[0230] More specifically, the ground recognition step (330) can classify the ground into one of a plurality of ground types by inputting LiDAR-based point cloud data and camera-based ground image data, and calculate the friction coefficient (μ) according to [Equation 1] by applying the ground physical property parameters stored in advance according to [Table 1] corresponding to the classified ground type.

[0231] [Table 1]

[0232]

[0233] [Mathematical Formula 1]

[0234]

[0235] (here, μ ref is the standard friction coefficient corresponding to the classified ground type, k r is the correction coefficient based on the state, k w represents a correction factor based on the wet or contaminated state)

[0236] That is, the ground recognition step (330) can calculate the friction coefficient (μ) corresponding to the actual driving environment by combining ground physical property parameters corresponding to [Table 1] according to [Equation 1].

[0237] Afterwards, the ground recognition step (330) can calculate the ground stability index (SI) according to the following [Equation 2] using the calculated friction coefficient (μ), the ground inclination angle (θ) calculated from the lidar point cloud, and the roll or pitch angle (φ) of the forklift body calculated from the inertia measuring device.

[0238] [Mathematical Formula 2]

[0239]

[0240] (Here, μ represents the coefficient of friction, θ represents the ground inclination angle, and φ represents the roll / pitch angle of the forklift body calculated from the inertia measurement device.)

[0241] At this time, the above ground stability index (SI) indicates a state where driving stability is higher the higher the value, and can be compared with a preset threshold value to be used as a standard for setting driving constraint conditions in subsequent steps.

[0242] Meanwhile, the ground inclination angle (θ) used in calculating the ground stability index (SI) can be calculated using a point group determined to be the ground in the point cloud data generated from the lidar sensor.

[0243] In addition, the roll / pitch angle (φ) of the forklift body calculated from the inertial measuring device may represent attitude information in which the degree to which the forklift body is tilted in the left-right direction or the front-back direction is expressed as an angle.

[0244] In addition, the forklift roll / pitch angle based on the acceleration and angular velocity data can be calculated by measuring the linear acceleration, including the gravitational component acting on the forklift body, and the angular velocity, which is the rotational speed of the forklift body, from the inertial measurement device including the acceleration sensor and the gyroscope sensor.

[0245] Through this, the inertial measuring device obtains an initial roll angle estimate (φ) according to the following [Equation 3] to [Equation 4]. r ) and initial pitch angle estimate (φ p ) can be produced.

[0246] [Mathematical Formula 3]

[0247]

[0248] (here, a x is the x-axis acceleration value, a y is the y-axis acceleration value, a z represents the z-axis acceleration value)

[0249] [Mathematical Formula 4]

[0250]

[0251] At this time, the ground recognition step (330) is the initial roll angle estimate (φ) calculated from the inertial measuring device. r ) and initial pitch angle estimate (φ p The value with the larger absolute value among ) can be reflected in the above ground stability index (SI) as the roll / pitch angle (φ) of the forklift body calculated from the inertia measuring device.

[0252] The object detection step (340) can detect objects including people and obstacles using the sensing data.

[0253] More specifically, the object detection step (340) can detect objects including people or obstacles located near the forklift driving path using preprocessed sensing data.

[0254] At this time, the object includes people, pallets, loads, racks, walls, vehicles, and fixed or moving objects within the work environment, and the location of the detected object, the distance from the forklift, and whether it is moving relative to it can be determined.

[0255] The above-mentioned task data generation step (350) can receive one of a natural language command or a field-based command from a preset user terminal, and identify a task target and a task location in response to the command to generate task data.

[0256] More specifically, the above-mentioned work data generation step (350) can perform morphological analysis in response to the input of the natural language command when the received work command is a natural language command.

[0257] Here, the morphological analysis is a process of breaking down a sentence into words or semantic units, and may refer to a preprocessing step for distinguishing action expressions, target expressions, and location expressions included in a command sentence.

[0258] Afterwards, the above-mentioned task data generation step (350) can classify the type of task instructed by the user using the morphological analysis results.

[0259] That is, if a word matching a preset keyword is included among the action-related expressions classified as a result of morphological analysis included in the above natural language command, the above operation can be determined as one of loading, unloading, moving, waiting, and driving to a specific location.

[0260] For example, as a result of performing morphological analysis in response to the above natural language command, if the data contains 'load / load / lift / lift', it can be classified as a loading operation; if it contains 'unload / lower / lower / departure', it can be classified as an unloading operation; if it contains 'movement / movement / take / transport', it can be classified as a movement operation; and if it contains 'waiting / stop / stop / wait', it can be classified as a waiting operation. Additionally, if specific location indication expressions such as 'move to location A / go to location B / drive to location C' are included, it can be determined as a driving operation to a specific location.

[0261] In addition, the above-mentioned work data generation step (350) can extract entity names, which are names corresponding to work objects and work locations, from the above-mentioned natural language command.

[0262] In this case, if the above natural language command includes multiple nouns or noun phrases, the work object and work location can be distinguished by determining which category each noun or noun phrase belongs to, either an object or a location.

[0263] For example, if the command 'move pallet to rack 3' is entered, 'pallet' can be extracted as the work object and 'rack 3' as the work location.

[0264] In addition, if the extracted work object or work location is a noun phrase composed of multiple words rather than a single word, a sequence of nouns or a modifier-noun combination form can be recognized and extracted as a single entity.

[0265] Meanwhile, the above-mentioned work data generation step (350) can supplement the extraction result according to a preset correction rule when the work object or work location is not clearly specified in the above-mentioned natural language command.

[0266] For example, if a location expression is omitted in a natural language command, one of the default location values ​​received from the user terminal or the current location of the forklift can be assumed as the work location, and if an object expression is omitted, an object with high priority in the recent work history or work environment can be set as a candidate.

[0267] Additionally, the above-mentioned work data generation step (350) can extract field values ​​corresponding to the work type, work object, and work location from the structured received field-based command when the received work command is a field-based command.

[0268] Here, the field-based command may refer to a command generated by a user entering or selecting a value into a predefined item, rather than in the form of natural language.

[0269] For example, if the work type field='Movement', work object field='Palette', and work location field='Rack 3' are entered, the corresponding field values ​​are extracted as is, allowing the work object and work location to be identified.

[0270] Additionally, the above-mentioned work data generation step (350) checks whether the value included in the field-based command matches the previously stored work environment information, and if it does not match, it may perform at least one of applying a correction rule or requesting re-entry to the user terminal.

[0271] In addition, the above-mentioned work data generation step (350) can generate work data by combining the work type classification result and the entity name extraction result when the command received from the above-mentioned user terminal is a natural language command, and combining the field value extraction result when the command is a field-based command, thereby determining the work target and work location of the work to be performed by the forklift.

[0272] In addition, if natural language-based output results and field-based output results exist simultaneously or are inconsistent with each other, it can be configured to select one according to preset priority or correction rules, or to combine the two results to identify the final task target and task location.

[0273] For example, it may be configured to prioritize field-based commands, which are structured values, and refer to natural language commands secondarily, or to prioritize natural language commands according to user settings.

[0274] In addition, it is possible to check whether the confirmed work target and work location are valid targets and locations registered in the work environment information, and if they are invalid, apply correction rules or request re-entry via the user terminal.

[0275] The above work planning step (360) can generate a driving path and a work sequence in response to the ground stability index, object detection results and work data.

[0276] More specifically, the above work planning step (360) can set the starting point and destination of the forklift based on the work target and work location included in the work data, and generate multiple driving path candidates using a pre-set path search algorithm by referring to graph-based map or grid-based map data in which driving sections within the work environment are represented as nodes and edges.

[0277] At this time, the above-mentioned preset path search algorithm may mean at least one of the Dijkstra algorithm, A* (A-star) algorithm, D* (D-star) algorithm, RRT (Rapidly-exploring Random Tree) based algorithm and PRM (Probabilistic Roadmap) based algorithm.

[0278] Additionally, the above work planning step (360) can adjust the work order to reduce the risk of collision by using the object detection result to avoid sections where people or obstacles exist on the driving path, or by adjusting the order of passing through such sections.

[0279] Meanwhile, the above work planning step (360) uses the ground stability index (SI) and load data to determine the maximum driving speed (V) of the forklift. max ) and maximum allowable slope angle (θ max ) can be calculated according to [Equation 5] to [Equation 6] below.

[0280] [Mathematical Formula 5]

[0281]

[0282] (Here, V b is the standard driving speed, SI r is the standard ground stability index, k v is the speed weighting factor, W is the load, W r represents the standard load.

[0283] [Mathematical Formula 6]

[0284]

[0285] (here, θ b is the standard allowable slope angle, k θ represents the slope angle weighting factor)

[0286] Through this, the above work planning step (360) is the maximum driving speed (V) among a plurality of driving path candidates. max) and maximum allowable slope angle (θ max The final driving path can be determined by determining only driving paths that satisfy conditions less than ) as valid paths.

[0287] Afterwards, the above-mentioned work planning step (360) may configure each generated driving path candidate into a plurality of path segments and waypoints, and generate path data including slope angle information corresponding to each path segment.

[0288] At this time, the inclination angle information may be calculated based on a point cloud generated from a LiDAR sensor and a ground model generated in the ground recognition step, or calculated from elevation information included in the work environment map data.

[0289] Subsequently, the above-mentioned work planning step (360) collects the maximum value among the slope angle information for each path segment included in each driving path candidate to calculate the slope angle of the corresponding path candidate, and this is the maximum allowable slope angle (θ max It is possible to determine whether driving is possible by comparing with ).

[0290] In addition, the required driving speed for each driving path candidate is calculated or collected, and this is the maximum driving speed (V max It is possible to determine whether the speed constraint is satisfied by comparing with ).

[0291] Through this, the above work planning step (360) is the maximum driving speed ( Vmax ) and maximum allowable slope angle (θ max Only driving path candidates that do not exceed ) are determined as valid paths, and if multiple valid paths are determined, the path with the highest working distance, working time, and safety can be determined as the final driving path and work execution order.

[0292] The above driving control step (370) can perform automatic control of the forklift based on the collision and overturning risk calculated in response to the driving of the forklift.

[0293] More specifically, the driving control step (370) controls the driving of the forklift according to the driving path set in the work planning step (360), and at the same time uses sensing data collected in real time during driving to determine the collision risk (R c ) and conduction risk (R t ) can be calculated according to [Equation 7] to [Equation 8] below.

[0294] [Mathematical Formula 7]

[0295]

[0296] (Here, T r is the reference collision time, V r ε represents the relative speed between the forklift and the object, ε is a preset infinitesimal value, and d represents the distance between the forklift and the object.

[0297] Accordingly, the driving control step (370) includes a time term (d / max(V) remaining until collision between the forklift and the object, corresponding to the driving speed of the forklift, whether the object is moving, and the relative approach speed. r By calculating the collision risk based on ε)) and integrating distance and speed information into a time concept to determine the collision risk as a quantitative value, the collision risk that may occur during the operation of the forklift can be quantified in terms of time margin.

[0298] [Mathematical Formula 8]

[0299]

[0300] (Here, θ is the ground slope angle, θ max θ is the maximum allowable inclination angle, φ is the forklift body tilt angle, φ r is the standard body tilt angle, a l is lateral acceleration, a r represents the reference lateral acceleration)

[0301] Accordingly, the driving control step (370) reflects the ground inclination angle, body tilt angle, and lateral acceleration as factors that contribute most significantly to the overturning in response to the operation of the forklift in response to the occurrence of overturning of the forklift, and compares the percentage of the collected values ​​with a preset reference value to determine the overturning risk (R) corresponding to the highest item. t ) can be quantified and calculated.

[0302] Subsequently, the driving control step (370) calculates the collision risk (R c ) and conduction risk (R t By comparing each risk level with a preset threshold, if the risk level is less than the preset first threshold, normal driving is maintained; if it exceeds the preset first threshold but is less than the preset second threshold, it switches to a deceleration or low-speed driving mode; and if the risk level exceeds the preset second threshold, automatic stop or driving stop control can be performed.

[0303] At this time, the first threshold value set above may be set to a value less than the second threshold value.

[0304] Through the above process, the autonomous operation system and method based on ground and surrounding recognition and work orders for an AI-based autonomous forklift can quantitatively determine driving stability by comprehensively analyzing ground characteristics and the vehicle's dynamic behavior during driving and work processes, and can evaluate the risk of collision and overturning in advance by considering the relative position and approach situation with people and obstacles in the surrounding environment. Subsequently, the evaluation results are combined with the content of work instructions and reflected in driving speed, path selection, and work sequence; and in response to environmental changes occurring during driving, the driving strategy and work execution method are automatically adjusted based on the corresponding risk level, thereby enabling autonomous operation that can simultaneously ensure safety and work efficiency in various work environments without direct control intervention by an operator.

[0305] According to one embodiment of the present invention, by comprehensively recognizing the surrounding environment and objects of a forklift using a plurality of sensors, the possibility of accidents occurring in response to forklift driving can be reduced.

[0306] In addition, by identifying the work target and work location from received natural language and field-based commands, the burden of manual input for the operator can be reduced and the convenience of the work instruction process can be improved.

[0307] In addition, by selecting a driving path corresponding to the maximum driving speed and maximum allowable inclination angle based on collected load data and calculated ground stability index, the driving stability of the forklift can be secured and work performance efficiency can be improved.

[0308] As described above, although an embodiment of the present invention has been explained by limited embodiments and drawings, the embodiment of the present invention is not limited to the embodiments described above, and various modifications and variations are possible from this description by those skilled in the art to which the present invention pertains. Accordingly, an embodiment of the present invention should be understood only by the claims described below, and all equivalent or analogous variations thereof shall be considered to fall within the scope of the inventive concept. Explanation of the symbols

[0310] 100: Ground and Surroundings Recognition and Work Order-Based Autonomous Execution System for AI-Based Autonomous Forklifts 110 : Sensor part 120 : Preprocessing section 130 : Ground recognition unit 140 : Object detection unit 150 : Work data generation section 151 : Natural Language Analysis Unit 152 : Task Type Classification Section 153 : Named Entity Extraction Section 154 ; Field value extraction section 160 : Work Planning Department 170 : Control unit 300: Ground and Surroundings Recognition and Work Order-Based Autonomous Execution Method for AI-Based Autonomous Forklifts 310: Sensing data generation step 320 : Preprocessing step 330: Ground recognition stage 340: Object detection step 350: Job data generation step 360: Work Planning Phase 370: Driving control stage

Claims

Claim 1 In a ground and surroundings recognition and work order-based autonomous execution system for an autonomous AI-based forklift, the system comprises: a sensor unit that collects distance, image, inertia, load, and proximity data from a LiDAR sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor installed in the forklift and generates sensing data; a preprocessing unit that preprocesses the sensing data; a ground recognition unit that receives the preprocessed sensing data, analyzes the type and physical characteristics of the ground, and calculates a ground stability index; an object detection unit that detects objects including people and obstacles using the sensing data; a work data generation unit that receives one of a natural language command and a field-based command from a preset user terminal, identifies a work target and a work location, and generates work data; and a work planning unit that generates a driving path and a work sequence corresponding to the ground stability index, object detection results, and work data. A control unit that performs automatic control of a forklift based on collision and overturning risk calculated in response to the driving of the forklift; wherein the work data generation unit includes: a natural language analysis unit that performs morphological analysis in response to the input of a natural language command received from the preset user terminal; a work type classification unit that classifies work types in response to the morphological analysis results analyzed by the natural language analysis unit; an entity name extraction unit that extracts names corresponding to work objects and work locations from the natural language command; and a field value extraction unit that extracts field values ​​corresponding to work objects and work locations in response to the input of a field-based command received from the preset user terminal; and is characterized by identifying work targets and work locations in response to the output results of the work type classification unit, the entity name extraction unit, and the field value extraction unit to generate work data. Claim 2 In claim 1, the ground recognition unit inputs point cloud data generated using distance data collected from the lidar sensor and ground image data collected from the camera sensor into a preset deep learning model to perform ground type classification, calculates a friction coefficient by applying a preset ground physical property parameter corresponding to the classified ground type, and calculates a ground stability index using the friction coefficient, the slope of the ground, and attitude information of the forklift collected from the inertia measuring device, characterized by an autonomous AI-based forklift ground and surrounding area recognition and work order-based autonomous execution system. Claim 3 delete Claim 4 A ground and surroundings recognition and work order-based autonomous execution system for an autonomous AI-based forklift according to claim 1, wherein the work planning unit calculates the maximum driving speed and maximum allowable angle of inclination of the forklift using the ground stability index and load data, and determines a driving path less than the maximum driving speed and maximum allowable angle of inclination as the final driving path. Claim 5 A method for ground and surrounding recognition and autonomous execution based on work orders for an autonomous AI-based forklift, comprising: a sensing data generation step of generating sensing data by collecting distance, image, inertia, load, and proximity data through a LiDAR sensor, camera sensor, inertial measurement unit, load sensor, proximity sensor, and ultrasonic sensor provided in the forklift; a preprocessing step of preprocessing the sensing data; a ground recognition step of analyzing the type and physical characteristics of the ground using the preprocessed sensing data and calculating a ground stability index; an object detection step of detecting objects including people and obstacles using the sensing data; a work data generation step of receiving one of a natural language command or a field-based command from a preset user terminal, identifying a work target and a work location in response to the command, and generating work data; and a work planning step of generating a driving path and a work sequence in response to the ground stability index, object detection results, and work data. The method comprises a driving control step for performing automatic control of a forklift based on collision and overturning risk calculated in response to the driving of the forklift, wherein the driving data generation step is characterized by, when the received driving command is a natural language command, performing morphological analysis of the natural language command to classify the type of work instructed by the user, extracting entity names corresponding to the work object and work location, synthesizing the driving type result and the entity name extraction result to generate driving data, and when the received driving command is a field-based command, extracting field values ​​corresponding to the driving type, work object, and work location from the structured field-based command received, and synthesizing the field value extraction result to determine the driving target and work location to generate driving data.

Images