Method and Apparatus for Recognizing a Moving Environment Based on Sensor Data

Abstract

A method for recognizing a moving environment based on sensor data and an apparatus for the same are disclosed. A method for recognizing a moving environment according to an embodiment of the present invention may include: a data acquisition step of acquiring point cloud data from at least one sensor provided in a moving body; a preprocessing step of correcting the point cloud data based on the movement of the moving body; a first classification processing step of performing movable ground modeling based on the corrected point cloud data and performing a first classification processing for ground data and non-ground data; and a second classification processing step of generating a moving environment recognition result by performing a second classification processing for an object using the non-ground data.

Description

Technology Field

[0001] The present invention relates to a sensor data-based moving environment recognition method for autonomous driving in rough terrain environments and an apparatus for the same. Background Technology

[0002] The content described in this section merely provides background information regarding embodiments of the present invention and does not constitute prior art.

[0003] Companies providing services such as delivery and transportation, as well as autonomous vehicles, are increasingly utilizing LiDAR to operate delivery robots navigating diverse environments, including roads and sidewalks. Consequently, recognizing movable areas and objects from 3D point cloud data is a critical requirement for robot operation in real-world scenarios. In particular, for ground-based mobile robots and vehicle platforms, determining the movable terrain capable of driving and exploring via wheels is a critical issue. Furthermore, accurate recognition of obstacles located on the terrain is essential for making safe driving decisions.

[0004] As the demand for autonomous driving technology increases, various technologies are emerging to recognize the platform's movement range and surrounding obstacles. One approach utilizes distance sensors (or laser sensors) for floor and object detection. While this method is designed to efficiently detect small obstacles on the ground, it is applicable only at relatively slow speeds and fails to account for terrain slopes.

[0005] Furthermore, recently, there is a method that utilizes 3D LiDAR for relatively accurate distance information. While this method performs ground recognition first for efficient object recognition, it is difficult to apply to environments with varying slopes because it divides the surrounding environment into cylindrical polar grids for ground recognition. Additionally, due to the box-fitting module-based object recognition method, the algorithm slows down in environments where object data is highly dense.

[0006] Additionally, for ground and object recognition, there is a method based on extracting depth information from images using deep learning rather than laser sensors. However, this training data-based method faces difficulties in collecting data from unfamiliar environments, objects, and terrains, and performance cannot be guaranteed.

[0007] Therefore, the need for ground recognition technology that can be utilized for real-time driving and is robust against environmental changes and object recognition technology that is robust against obscuration by other objects, as well as technology that enables robust driving even when encountering unknown environments such as rough terrain and uneven terrain, is emphasized. The problem to be solved

[0008] The main objective of the present invention is to provide a sensor data-based moving environment recognition method and an apparatus for the same, which performs classification of objects corresponding to ground and non-ground areas within a movable area based on point cloud data of a lidar sensor for autonomous driving in a rough terrain environment. means of solving the problem

[0009] According to one aspect of the present invention, in a method for recognizing a moving environment based on sensor data in a moving environment recognition device for achieving the above objective, the moving environment recognition method may include: a data acquisition step of acquiring point cloud data from at least one sensor provided in a moving body; a preprocessing step of correcting the point cloud data based on the movement of the moving body; a first classification processing step of performing movable ground modeling based on the corrected point cloud data and performing a first classification processing for ground data and non-ground data; and a second classification processing step of generating a moving environment recognition result by performing a second classification processing for an object using the non-ground data.

[0010] In addition, according to another aspect of the present invention, a moving environment recognition device for achieving the above objective comprises: a memory storing one or more programs for moving environment recognition; and one or more processors performing operations for moving environment recognition according to the one or more programs, wherein the operations performed by the processors may include: a data acquisition step of acquiring point cloud data from at least one sensor provided in a moving body; a preprocessing step of correcting the point cloud data based on the movement of the moving body; a first classification processing step of performing movable ground modeling based on the corrected point cloud data and performing a first classification processing for ground data and non-ground data; and a second classification processing step of performing a second classification processing for an object using the non-ground data to generate a moving environment recognition result. Effects of the invention

[0011] As explained above, the present invention is mounted on a mobile body and ensures real-time determination of movable terrain search and object classification, and has the effect of minimizing under-segmentation or over-segmentation problems in object recognition. Brief explanation of the drawing

[0012] FIGS. 1 to 5 are drawings for explaining the overall operation concept of a moving environment recognition according to an embodiment of the present invention. FIG. 6 is a block diagram schematically showing a moving body that performs moving environment recognition according to an embodiment of the present invention. FIG. 7 is a block diagram schematically showing a mobile environment recognition device according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method for recognizing a moving environment according to an embodiment of the present invention. FIG. 9 is an illustrative diagram for explaining sensor data acquisition according to an embodiment of the present invention. FIG. 10 is an exemplary diagram illustrating a moving environment recognition operation according to an embodiment of the present invention. FIGS. 11 and FIGS. 12 are drawings illustrating a preprocessing step for recognizing a moving environment according to an embodiment of the present invention. FIG. 13 is a flowchart illustrating a first classification processing step for recognizing a moving environment according to an embodiment of the present invention. FIGS. 14 to 17 are drawings for explaining a first classification processing operation according to an embodiment of the present invention. FIG. 18 is a flowchart illustrating a second classification processing step for recognizing a moving environment according to an embodiment of the present invention. FIGS. 19 to 22 are drawings for explaining a second classification processing operation according to an embodiment of the present invention. Specific details for implementing the invention

[0013] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, detailed descriptions of related known configurations or functions are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention. Furthermore, while preferred embodiments of the present invention will be described below, the technical concept of the present invention is not limited or restricted thereto and can be modified and implemented in various ways by those skilled in the art. Hereinafter, the sensor data-based mobile environment recognition method and the apparatus for the same proposed in the present invention will be described in detail with reference to the drawings.

[0014] FIGS. 1 to 5 are drawings for explaining the overall operation concept of a moving environment recognition according to an embodiment of the present invention.

[0015] Recognizing movable areas and objects from 3D point clouds is a critical issue required in robot navigation systems. In particular, for ground-based mobile robots and vehicle platforms, determining movable terrain capable of driving and exploring via wheels is one of the critical problems. Furthermore, to make safe driving decisions, recognition of objects located on that terrain follows. However, over-segmentation and under-segmentation phenomena have an adverse effect on robot navigation.

[0016] Therefore, for safe autonomous driving, the present invention proposes an algorithm that utilizes a 3D point cloud-based graph structure to simultaneously detect movable areas and perform object clustering (object classification).

[0017] For the detection of movable terrain, the present invention assigns point cloud data to nodes based on xy-coordinate values ​​in a tri-grid field, which is a graph structure using triangular grids as nodes. Then, movable regions are explored and redefined through local convexity and concavity checks on the relationships between nodes. On the other hand, ground object segmentation uses a graph structure that defines groups of horizontally adjacent 3D points in a spherical projection space as nodes and represents the horizontal and vertical relationships between nodes as edges.

[0018] Figure 1 shows an overview of the LiDAR sensor data-based movable terrain detection and object segmentation algorithm of the present invention. This method is a two-stage method that independently performs terrain detection and object segmentation, and performs detection and classification through two different graph representation and search methods.

[0019] First, to model movable terrain, point cloud data of the surrounding environment is acquired through a LiDAR sensor (S10), and the acquired point cloud data is projected onto a tri-grid field to be represented as a graph, after which node-based point distribution-based terrain modeling is performed. Accordingly, movable terrain is explored using breadth-first graph search, and the overall terrain modeling is fitted. Then, ground data is classified based on the model (S20).

[0020] Next, object classification that is robust to object shapes or occlusion by other objects is performed based on non-surface data (S30). This is achieved through graph representation based on a spherical projection space and binary graph search within that graph, thereby enabling real-time object classification that is robust to the surrounding environment.

[0021] Figure 2 illustrates the result of representing LiDAR sensor data as a tri-grid field, and each tri-grid contains one graph node.

[0022] In addition, Figures 3(a) and 3(b) show a comparison of the results of determining movable areas based on a grid field and a triangular grid field, respectively. The blue polygons in Figures 3(a) and 3(b) are terrain model results formed based on the same terrain modeling method.

[0023] Conventional grid-based technologies commonly used have many discontinuous parts within the movable terrain model, even though the terrain is actually continuous and curved, as shown in Fig. 3 (a). In contrast, the triangular grid-based technology newly introduced in the present invention overcomes these disadvantages, allowing for consideration of mobility and continuity between terrain models.

[0024] Figure 4 shows the results of applying the technology of the present invention, in order from left to right: in a simulation environment (Figure 4 (a)), in a narrow alley environment in an urban area (Figure 4 (b)), in an urban intersection environment (Figure 4 (c)), and in a forest environment (Figure 4 (d)).

[0025] As such, even if environmental changes are significant, it can be confirmed that ground data (gray dot data) is properly classified through the movable area detection algorithm based on the technology of the present invention.

[0026] In the results of the object classification algorithm (color point data), particularly as can be seen in the results from the simulation environment, it can be confirmed that even objects with shapes that are easily divided into multiple parts, such as thin and long cylinders like streetlights, are classified as a single object. Additionally, through the results from the urban intersection environment, it can be seen that an object can be classified as a single object even when a wall is obscured by another thin object.

[0027] FIG. 5(a) shows a mobile platform in which the present invention is actually applied, and the performance of the mobile platform can be verified through autonomous exploration based on the present invention. The mobile platform of FIG. 5(a) may be a robot selected as being suitable for driving in an off-road environment.

[0028] Figure 5 shows an environment for self-testing the movable terrain search and object classification algorithm based on the present invention, and autonomous exploration based on the technology of the present invention is conducted in the environment.

[0029] To demonstrate the applicability of the technology of the present invention to various environments, a lakeside open field environment (Fig. 5(b)) and a forest including a steep slope (Fig. 5(c)) may be selected as length test environments.

[0030] FIG. 6 is a block diagram schematically showing a moving body that performs moving environment recognition according to an embodiment of the present invention.

[0031] The mobile environment recognition system (100) according to the present embodiment includes a sensor device (200) and a moving body (300). The mobile environment recognition system (100) of FIG. 1 is according to one embodiment, and not all blocks shown in FIG. 1 are essential components, and some blocks included in the mobile environment recognition system (100) in other embodiments may be added, changed, or deleted.

[0032] The sensor device (200) refers to a device that senses the surrounding environment and the moving body (300) for recognizing the moving environment.

[0033] The sensor device (200) preferably includes at least one lidar sensor, but is not necessarily limited thereto.

[0034] Additionally, the sensor device (200) may include an inertial measuring device (not shown) for measuring inertial information of the moving body (300), or a device for sensing attitude information of the moving body (300).

[0035] In FIG. 1, the sensor device (200) is shown as being a separate device from the moving body (300), but it is not necessarily limited to this and may be implemented in a form such as a sensor part inside the moving body (300).

[0036] The moving body (300) refers to a device that moves in an external environment based on autonomous driving.

[0037] The mobile body (300) may be one of a cleaning robot, a mobile robot for public buildings, a logistics robot, a delivery robot, an industrial robot, etc.

[0038] The technology regarding movement control for controlling the movement of the moving body (300), driving control of the moving wheels, and control related to autonomous driving is omitted in detail as general technology is applied.

[0039] The moving body (300) includes a moving environment recognition device (400) that recognizes the moving environment.

[0040] The moving environment recognition device (400) acquires point cloud data, performs movable ground modeling based on the point cloud data corrected based on the movement of the moving body (300), and performs a first classification process for ground data and non-ground data. Subsequently, the moving environment recognition device (400) performs a second classification process for an object using the non-ground data to generate a moving environment recognition result.

[0041] The mobile environment recognition device (400) transmits the mobile environment recognition result to the mobile body (300) so that autonomous driving is operated based on the mobile environment recognition result. A detailed description of the mobile environment recognition device (400) is provided in FIG. 7.

[0042] FIG. 7 is a block diagram schematically showing a mobile environment recognition device according to an embodiment of the present invention.

[0043] The mobile environment recognition device (400) according to the present embodiment includes an input unit (710), an output unit (720), a processor (730), a neural network (740), a memory (750), and a database (760). The mobile environment recognition device (400) of FIG. 7 is according to one embodiment, and not all blocks shown in FIG. 7 are essential components, and some blocks included in the mobile environment recognition device (400) in other embodiments may be added, changed, or deleted. Meanwhile, the mobile environment recognition device (400) may be implemented as a computing device, and each component included in the mobile environment recognition device (400) may be implemented as a separate software device or as a separate hardware device combined with software.

[0044] The input unit (710) refers to a means for inputting or acquiring signals or data to perform a movement environment recognition operation. The input unit (710) may input various forms of signals or data in conjunction with the processor (730), or may acquire data directly in conjunction with an external device and transmit it to the processor (730).

[0045] The output unit (720) can output a preprocessing result, a first classification processing result, a second classification processing result, a movement environment recognition result, etc. in conjunction with the processor (730).

[0046] The output unit (720) can transmit the movement environment recognition result generated by the processor (730) to the moving body (300) so that movement driving can be performed.

[0047] The input unit (710) and the output unit (720) may be connected to an input / output interface unit (not shown). The input / output interface unit may be configured to transmit information obtained from the input unit (710) to the processor (730), or to receive a control signal from the processor (730) and convert it into a signal for substantially controlling the input unit (710) and the output unit (720).

[0048] The processor (730) performs the function of executing at least one instruction or program contained in memory (750).

[0049] The processor (730) according to the present embodiment acquires point cloud data, performs movable ground modeling based on the point cloud data corrected based on the movement of the moving body (300), performs a first classification process for ground data and non-ground data, and then performs a second classification process for an object using non-ground data to generate a moving environment recognition result.

[0050] The processor (730) may also process movement environment recognition operations in conjunction with the neural network (740).

[0051] Meanwhile, although the processor (730) and the neural network (740) are described as being different modules, they are not necessarily limited to this and may be implemented as a single module to perform their respective operations.

[0052] The neural network (740) performs neural network processing related to determining sudden events, calculating classification values, and assigning weights based on artificial intelligence (AI).

[0053] The neural network (740) has input nodes, intermediate nodes, and output nodes, and has a structure specified by decision weights that have been pre-learned through training data as connection weights connecting each node. The output value of the neural network (740) may be the coordinate value of an expansion area or the coordinate value of a unit block area, and may be implemented in the form of a feature value matrix for the expansion area or the unit block area.

[0054] Below, the movement environment recognition operation performed by the processor (730) will be described.

[0055] The processor (730) obtains point cloud data from at least one sensor device (200) equipped in the moving body (300).

[0056] The processor (730) can additionally acquire at least one of inertial measurement data and attitude data for the moving body (300).

[0057] Additionally, the processor (730) corrects the point cloud data based on the movement of the moving body (300).

[0058] The processor (730) corrects the point cloud data by integrating and processing at least one sensor data and inertial measurement data included in the point cloud data. Here, the integration processing is preferably a summation processing between the data, but is not necessarily limited to this.

[0059] The processor (730) can correct point cloud data based on pitch and roll angles calculated from the posture of the moving body (300). Here, the correction of point cloud data means correcting tilted sensor data into planar sensor data.

[0060] Additionally, the processor (730) performs movable ground modeling based on corrected point cloud data and performs a first classification process for ground data and non-ground data.

[0061] The processor (730) projects the point cloud data onto the graph structure of the triangular grid field.

[0062] The processor (730) forms a ground model for each of at least one node included in a triangular grid field. The processor (730) forms a ground model by performing ground modeling based on Principal Component Analysis (PCA).

[0063] The processor (730) determines the possibility of movement between at least one node. The processor (730) calculates the possibility of movement based on the relationship between nodes due to local convexity and concavity.

[0064] The processor (730) redefines the ground model for the at least one node based on mobility and classifies and processes ground data and non-ground data based on the redefined final ground model.

[0065] The processor (730) redefines the ground model selected based on mobility by assigning weights to each corner of the node to the ground model selected based on mobility and re-establishing it as the final ground model.

[0066] The processor (730) classifies ground data by determining the node corresponding to the final ground model as ground, and classifies the remaining nodes excluding the node corresponding to the final ground model as non-ground data by determining them as non-ground.

[0067] The processor (730) can classify a node whose planar value corresponding to the final ground model is less than a preset distance threshold as ground data by determining it as ground.

[0068] Additionally, the processor (730) uses non-ground data to perform a second classification process for the object to generate a moving environment recognition result.

[0069] The processor (730) projects three-dimensional non-surface data into a two-dimensional spatial structure and converts it into a two-dimensional graph structure.

[0070] The processor (730) processes horizontal connectivity updates to provide connectivity between nodes based on a two-dimensional graph. The processor (730) applies at least one of skipped linkage and circular linkage tests to minimize errors caused by object overlap.

[0071] The processor (730) processes vertical connectivity updates to provide connectivity between nodes based on a two-dimensional graph so that classification of objects is processed.

[0072] The processor (730) can minimize redundant calculations by prioritizing point distance calculations for duplicate nodes based on index in a two-dimensional graph structure.

[0073] Additionally, the processor (730) generates a moving environment recognition result including ground data and object classification data that classifies objects.

[0074] The memory (750) includes at least one instruction or program executable by the processor (730). The memory (750) may include instructions or programs for a preprocessing operation for a moving environment recognition operation, a first classification processing operation, a second classification processing operation, a moving environment recognition operation, etc. Additionally, the memory (750) may include instructions or programs for an operation for preprocessing a neural network learning result and an input or output value of the neural network, etc.

[0075] A database (760) refers to a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS), and refers to a data storage form in which data can be freely searched (extracted), deleted, edited, or added. It can be implemented to suit the purpose of an embodiment of the present invention using relational database management systems (RDBMS) such as Oracle, Informix, Sybase, and DB2, object-oriented database management systems (OODBMS) such as Gemston, Orion, and O2, and XML native databases such as Excelon, Tamino, and Sekaiju, and has appropriate fields or elements to achieve its functions. Meanwhile, the database (760) may also be implemented in the cloud, virtual memory, etc.

[0076] The database (760) according to the present embodiment can store and provide data or information related to preprocessing operations, first classification processing operations, second classification processing operations, movement environment recognition operations, etc.

[0077] The database (760) is described as being implemented within the mobile environment recognition device (400), but is not necessarily limited thereto and may be implemented as a separate data storage device.

[0078] FIG. 8 is a flowchart illustrating a method for recognizing a moving environment according to an embodiment of the present invention.

[0079] The mobile environment recognition device (400) acquires sensor data (S810).

[0080] The moving environment recognition device (400) can additionally acquire at least one of inertial measurement data and attitude data for the moving body (300).

[0081] The moving environment recognition device (400) performs preprocessing to correct point cloud data based on the movement of the moving body (300) (S820). The processor (730) can correct the point cloud data by integrating and processing at least one sensor data and inertial measurement data included in the point cloud data.

[0082] The moving environment recognition device (400) performs movable ground modeling based on corrected point cloud data and performs a first classification process for ground data and non-ground data (S830).

[0083] The moving environment recognition device (400) projects point cloud data onto a graph structure of a triangular grid field, forms a ground model for each of at least one node included in the triangular grid field, determines the possibility of movement between at least one node, redefines the ground model for said at least one node based on the possibility of movement, and classifies and processes ground data and non-ground data based on the redefined final ground model.

[0084] The mobile environment recognition device (400) performs a second classification process for an object using non-ground data to generate a mobile environment recognition result (S840).

[0085] The mobile environment recognition device (400) projects the three-dimensional non-ground data into a two-dimensional spatial structure to convert it into a two-dimensional graph structure, processes horizontal connectivity updates to provide connectivity between nodes based on the two-dimensional graph, and processes vertical connectivity updates to provide connectivity between nodes based on the two-dimensional graph so that classification of objects is processed.

[0086] The overall algorithm flow described in the preprocessing step, the first classification processing step, and the second classification processing step of the mobile environment recognition device (400) according to the present embodiment can be processed at a computational speed of 50 Hz or more on the platform of the actual mobile body (300). This processing speed is a result that can be achieved by operating based on efficient modeling and data classification based on two new graph structures.

[0087] Although FIG. 3 describes each step as being executed sequentially, it is not necessarily limited to this. In other words, FIG. 3 is not limited to a chronological order, as it may be applicable to execute the steps described in FIG. 3 by modifying them or to execute one or more steps in parallel.

[0088] The mobile environment recognition method according to the present embodiment described in FIG. 3 may be implemented as an application (or program) and recorded on a recording medium readable by a terminal device (or computer). The recording medium on which the application (or program) for implementing the mobile environment recognition method according to the present embodiment is recorded and which is readable by a terminal device (or computer) includes all types of recording devices or media in which data that can be read by a computing system is stored.

[0089] FIG. 9 is an illustrative diagram for explaining sensor data acquisition according to an embodiment of the present invention.

[0090] Figure 9 shows a simulation for self-data acquisition and a platform for a mobile body in possession.

[0091] Data for operating the algorithm is acquired through 3D LiDAR sensors attached to various platforms. The operation of the algorithm can be verified through proprietary data acquired based on CARLA vehicle simulations, owned autonomous vehicles equipped with Ouster OS1-64 3D LiDAR, and owned driving robots equipped with Ouster OS0-128 3D LiDAR, as well as publicly available KITTI data based on Velodyne HDL-64 3D LiDAR. Through this, the algorithm proposed in this invention enables robust ground recognition and object classification regardless of the platform and sensor model.

[0092] FIG. 10 is an exemplary diagram illustrating a moving environment recognition operation according to an embodiment of the present invention.

[0093] The overall flowchart of the proposed algorithm of the present invention is as shown in FIG. 10.

[0094] The present invention comprises a pre-processing process for determining appropriate traversability (S820), a traversability terrain modeling and ground / non-ground data separation algorithm (TGS: traversability ground segmentation, S830), and an above-ground object classification algorithm (AOS: above-ground object segmentation, S840). In particular, TGS and AOS enable efficient recognition and classification because they are performed by structuring LiDAR data (point cloud data) into a self-proposed graph.

[0095] The present invention, which performs movable terrain search and object classification based on two graph structures, is mounted on a robot to ensure real-time decision-making and can mitigate the under-segmentation phenomenon in object recognition where multiple objects are recognized as one object and the over-segmentation problem where one object is recognized as multiple objects.

[0096] The present invention has the effect of being able to form a graph structure through a field of a triangular grid structure for the first time, thereby enabling terrain modeling in rough environments with many curves and providing robustness against under-segmentation problems based thereon, real-time movable terrain search through breadth-first movable node search considering mobility between nodes, robustness against environmental changes such as flat structures like simulations and urban environments and rough environments with many curves and slopes, real-time object classification through binary graph search in a spherical projection space and robustness against over-segmentation through this, and applicable to various ground platforms such as autonomous vehicles and driving robots, and providing robustness against changes in LiDAR sensors.

[0097] FIGS. 11 and FIGS. 12 are drawings illustrating a preprocessing step for recognizing a moving environment according to an embodiment of the present invention.

[0098] FIG. 11 is an example diagram showing sensor point data acquired within a predetermined specific time and inertial measurement data acquired from an inertial measurement device (not shown) in chronological order.

[0099] FIG. 12 (a) shows distorted sensor data (black) and corrected data (green) caused by the movement of a moving body (300) in an open field environment,

[0100] Figure 12 (b) shows distorted sensor data (black) and corrected data (green) due to the degree of tilt of the moving body (300).

[0101] The moving environment recognition device (400) according to the present embodiment performs a preprocessing step to correct LiDAR data distortion that occurs according to the movement and state of the moving body (300) in the field environment before performing the proposed algorithm (first classification processing and second classification processing).

[0102] Accordingly, the moving environment recognition device (400) corrects point data acquired at different times through an inertial measuring device (not shown) as if acquired at once, as in FIG. 11, and replaces it with data for performing the proposed algorithm as in FIG. 12 (a). Additionally, if the moving body (300) is tilted, the data can be corrected by determining that a flat surface is sloped or a sloped surface is flat, as in FIG. 12 (b).

[0103] The moving environment recognition device (400) can operate an algorithm that is robust even to the tilting of the moving body (300) through inertial measurement data of an inertial measurement device (not shown).

[0104] The preprocessing step is essential to improve classification performance, especially in uneven terrain.

[0105] As illustrated in FIG. 12 (a), sensor data (point cloud) may be distorted due to the movement of the moving body (300). The moving environment recognition device (400) can integrate sensor data (LiDAR data) and inertial measurement data (IMU data) using a pre-integration approach of an inertial measurement device (not shown) to correct the distortion of the sensor data (point cloud).

[0106] Additionally, as shown in FIG. 12 (b), the moving environment recognition device (400) can correct data by applying a 6-DoF attitude to mitigate problems that occur during classification processing due to a tilted moving body (300).

[0107] The moving environment recognition device (400) corrects the original point cloud to rotate in a vertical orientation using pitch and roll angles calculated from the orientation of the moving body (300), and after correction, can accurately estimate terrain characteristics in the world coordinate system.

[0108] FIG. 13 is a flowchart illustrating a first classification processing step for recognizing a moving environment according to an embodiment of the present invention.

[0109] The mobile environment recognition device (400) projects point cloud data onto a graph structure of a triangular grid field (S1310). The mobile environment recognition device (400) forms a ground model for each of at least one node included in the triangular grid field.

[0110] The mobile environment recognition device (400) forms a ground model by performing ground modeling based on Principal Component Analysis (PCA) (S1320).

[0111] The movement environment recognition device (400) determines the possibility of movement between at least one node. The processor (730) calculates the possibility of movement based on the relationship between nodes due to local convexity and concavity (S1330).

[0112] The mobile environment recognition device (400) redefines the ground model for the at least one node based on mobility and classifies and processes ground data and non-ground data based on the redefined final ground model (S1340). The mobile environment recognition device (400) assigns weights to each corner of the node for the ground model selected based on mobility to redefine it as the final ground model. The mobile environment recognition device (400) classifies ground data by determining the node corresponding to the final ground model as ground, and classifies non-ground data by determining the remaining nodes excluding the node corresponding to the final ground model as non-ground. The mobile environment recognition device (400) may classify ground data by determining the node where the planar value of the node corresponding to the final ground model is less than a preset distance threshold as ground.

[0113] FIGS. 14 to 17 are drawings for explaining a first classification processing operation according to an embodiment of the present invention.

[0114] Below, the first classification processing step of the mobile environment recognition device (400) will be schematically explained through FIGS. 14 to 17.

[0115] The 3D LiDAR data (point cloud data) corrected through the preprocessing step can be processed into movable ground modeling through the first classification processing step shown in FIG. 14, and classified into ground data and non-ground data.

[0116] The first classification processing step may include detailed steps such as a tri-grid field projection step, a node-by-node ground modeling step, a breadth-first graph search-based movable terrain search step, a re-definition of the overall ground model based on mobility, and a ground data and non-ground data classification step based on the model.

[0117] The mobile environment recognition device (400) projects three-dimensional point cloud data onto a graph structure of a triangular grid field. Each triangular grid within the triangular grid field acts as a node, and the graph structure of the triangular grid field can accurately represent various terrains more than a general grid structure field. The result of projecting point cloud data onto the graph structure of the triangular grid field is as shown in FIG. 2 (b).

[0118] Based on the method illustrated in FIG. 15, the mobile environment recognition device (400) can define a ground model for each triangular grid node. This is performed based on Principal Component Analysis (PCA) and can extract more information faster than the commonly used RANSAC (RANdom SAmple consensus).

[0119] After an independent ground model is formed for each node, the moving environment recognition device (400) determines the possibility of movement between nodes by considering the relationship between nodes as shown in FIG. 16. Here, the relationship between nodes may be local convexity and concavity. In addition, the degree of similarity of normal vectors and discontinuity between ground models may be considered in the relationship between nodes.

[0120] After determining the possibility of movement between nodes, the movement environment recognition device (400) generates a final ground model by redefining the overall ground model centered on nodes including the movable ground model.

[0121] The movement environment recognition device (400) redefines the ground model based on the result of determining the possibility of movement between nodes (Fig. 17 (a)) as shown in Fig. 17 (b).

[0122] The ground model redefinition processing step of the mobile environment recognition device (400) contributes to correcting areas where the ground model is not determined or parts that have formed an incorrect ground model, thereby enabling ground modeling that is robust even to terrain including environmental changes and various slopes.

[0123] Additionally, the mobile environment recognition device (400) can classify at least one of ground data and non-ground data based on the final ground model.

[0124] The moving environment recognition device (400) can produce a result in which the under-segmentation phenomenon, in which two or more object data are combined into one and judged, is significantly reduced as a result of utilizing the possibility of movement in the ground model and data classification.

[0125] Below, the specific operation of the first classification processing step of the mobile environment recognition device (400) will be described.

[0126] <Triangular Grid Field Projection Phase and Node-Specific Ground Modeling Phase>

[0127] The moving environment recognition device (400) encodes three-dimensional point cloud data in a form that projects it onto a graph of a triangular grid field (see FIG. 2). Here, the triangular grid field has a constant resolution r T It refers to a pre-constructed field on the xy coordinates that has

[0128] In an encoded triangular grid field, each triangular-shaped node is the corresponding partial point cloud Includes

[0129] The encoded triangular grid field is a set of nodes and edge set It consists of. Here Is and It is an edge connecting, is a set of node indices.

[0130] Therefore, the mobile environment recognition device (400) uses a ground plane fitting approach based on principal component analysis (PCA) as a plane model , initial ground point and each partial point cloud Unique values ​​sorted in descending order for can be estimated. Here, is the normalized surface normal vector , in other words and plane coefficient It consists of.

[0131] also, The average point obtained by averaging the initial ground points within and weights that score the traversability of the corresponding triangular grid do It is included in. These can be expressed as [Equation 1].

[0132]

[0133] Here, cohesion = λ1 / λ3, planarity = λ2 / λ3, and linearity = λ1 / λ2 represent characteristic coefficients derived from the distribution of points.

[0134] Each node corresponds as in (b) of FIG. 2. It can be represented as a triangular grid having.

[0135] Afterwards, node is the slope threshold parameter and the number of points It is classified into three types according to. Here, the three types are terrain nodes , obstacle node and unknown nodes am.

[0136] The node with a smaller slope is Classified as, The node with a steeper slope is It is classified as, Nodes with fewer points It is classified as.

[0137] The mobile environment recognition device (400) performs the next step using only terrain nodes.

[0138] <Width-First Graph Search-Based Movable Terrain Search Steps>

[0139] The mobile environment recognition device (400) is a set of terrain nodes A set of nodes that can be traversed To explore it, a breadth-first search approach called B-TGS (Breadth-First Traversable Graph Search) can be applied.

[0140] The mobile environment recognition device (400) has the highest weight and is closest to the sensor is the seed node Select the seed node and select the adjacent nodes of the seed node as neighbor nodes. The operation of selecting the seed node and neighbor nodes is as shown in (b) of FIG. 16.

[0141] Then, the movement environment recognition device (400) recognizes a movable node among neighboring nodes. In order to decide and The geometric relationship between Calculates the possibility of movement.

[0142] The movability of is determined based on the relationship between nodes due to local convexity and concavity, as illustrated in Fig. 16 (a).

[0143] [Mathematical Formula 2] is the edge It indicates whether it has acceptable local convexity and concavity.

[0144]

[0145] Here, represents the movement vector between two nodes, and silver (or )go (or Represents the angle that can be located on ), represents the critical angle for similarity.

[0146] For example, between a seed node and another predetermined node If is true, the corresponding specified node is It is included in and can become another seed node in a subsequent exploration phase.

[0147] The mobile environment recognition device (400) repeats the comparison operation between the seed node and other nodes until there are no more adjacent nodes.

[0148] As shown in FIG. 17 (a), the moving environment recognition device (400) is movable at the end of the movable terrain search step. The set of Corresponding to It increases by connecting.

[0149] Ground Model Redefinition and Non-Ground Data Classification Steps

[0150] The mobile environment recognition device (400) performs a redefinition step for the ground model.

[0151] The mobile environment recognition device (400) is triangle corners of the node For the entire movable ground on the triangular grid field (TGF) by applying weighted corner fitting. of cast It can be improved as. In weighted corner fitting, all corners are It is grouped as. Here represents a set of grouped corner indices.

[0152] Subsequently, the mobile environment recognition device (400) through a weighted average step to allow a node with a high weight to have a greater influence on the model of a neighboring node The height of each corner is , in other words, It is updated to. Here, the weighted average step can be expressed as [Equation 3].

[0153]

[0154] Also, three All surrounded by is a terrain node with updated elements as in [Equation 4] It is modified to.

[0155]

[0156] As illustrated in FIG. 17(b), the mobile environment recognition device (400) each for The point cloud is classified using [Equation 5]. Here, the mobile environment recognition device (400) classifies ground data and non-ground data.

[0157]

[0158] Here, each point Regarding, represents the distance threshold between the point and the plane.

[0159] FIG. 18 is a flowchart illustrating a second classification processing step for recognizing a moving environment according to an embodiment of the present invention.

[0160] The mobile environment recognition device (400) generates a mobile environment recognition result by performing a second classification process on an object using non-ground data (S1810). The mobile environment recognition device (400) projects the 3-dimensional non-ground data into a 2-dimensional spatial structure and converts it into a 2-dimensional graph structure.

[0161] The mobile environment recognition device (400) processes horizontal connectivity updates to provide connectivity between nodes based on a two-dimensional graph (S1820). The mobile environment recognition device (400) applies at least one test of skipped linkage and circular linkage to minimize errors caused by object overlap.

[0162] The mobile environment recognition device (400) processes vertical connectivity updates to provide connectivity between nodes based on a two-dimensional graph so that classification of objects is processed (S1830). The mobile environment recognition device (400) can minimize redundant calculations by prioritizing point distance calculations for duplicate nodes based on indices in the two-dimensional graph structure.

[0163] The mobile environment recognition device (400) generates a mobile environment recognition result including ground data and object classification data that classifies objects (S1840).

[0164] FIGS. 19 to 22 are drawings for explaining a second classification processing operation according to an embodiment of the present invention.

[0165] In the second classification processing step, object classification is performed based on non-surface data extracted through the first classification processing step.

[0166] The second classification processing step may include detailed steps such as a spherical projection step, a horizontal connectivity update step, and a vertical connectivity update step. In the second classification processing step, updates in the horizontal and vertical directions proceed sequentially, enabling robust object classification.

[0167] In the spherical projection step, the moving environment recognition device (400) projects three-dimensional data into a two-dimensional space and forms a graph structure in two dimensions.

[0168] As illustrated in FIG. 20, the spherical projection step forms a graph structure where each cell becomes a node of the graph. For example, the spherical projection result can be observed by looking at the second ring with applied color in FIG. 20 (a) and (b). Here, data at a distance closer than a preset distance can be treated as a single node.

[0169] The mobile environment recognition device (400) performs horizontal connectivity updates and vertical connectivity updates, as shown in FIGS. 21 and 22, to provide connectivity between nodes based on a two-dimensional graph formed in the spherical projection step.

[0170] The mobile environment recognition device (400) can minimize problems regarding errors that may occur during object classification processing (e.g., object obscuration or object shape error) through a horizontal connectivity update step and a vertical connectivity update step.

[0171] In the horizontal connectivity update step, the mobile environment recognition device (400) can form a robust edge by applying concepts such as skipped linkage and circular linkage, rather than simply considering the relationship with neighbor nodes. This can contribute to the object being recognized as a single object even when it is obscured by another object.

[0172] In the vertical connectivity update step, the mobile environment recognition device (400) has a local index buffer size (local index buffer size, T) in the vertical direction. ext By applying the concept of ), robust object classification of thin, long cylindrical objects such as streetlights or trees can be performed.

[0173] Below, the specific operation of the second classification processing step of the mobile environment recognition device (400) will be described.

[0174] The second classification processing step performs classification of objects on the ground.

[0175] In the second classification processing step, the mobile environment recognition device (400) is a point cloud that is a point label (label(p) = Obstacle) non-surface. It processes the spherical projection of. Afterwards, the moving environment recognition device (400) repeatedly performs horizontal and vertical updates for each row of the projection space for efficient processing time.

[0176] Spherical Projection Step

[0177] Spherical projection is efficient for verifying the spatial adjacency of points based on reflection angles. Additionally, spherical projection can simplify subsequent horizontal and vertical update steps.

[0178] The moving environment recognition device (400) arranges points according to the azimuth and elevation angles of reflection for spherical projection of the point cloud, and width and height having Map into space. Figure 20 (b) shows an example of spherically projected points.

[0179] The mobile environment recognition device (400) has a horizontally adjacent distance threshold. The points below are nodes It forms. Here, , and Each represents the local index of the start point, end point, and label.

[0180] Since points are organized in horizontal order based on azimuth, registering indices for the local start and end of nodes can be useful for subsequent clustering.

[0181] Simply put, a set of directed edges connecting two nodes class A set of nodes is a graph structure for clustering Constitutes. Here, And, is ring It is a list of node indices within.

[0182] <Horizontal Connectivity Update Phase>

[0183] The horizontal connectivity update step uses the horizontal merge threshold in the direction of the local index. Creates nodes by clustering neighboring points within.

[0184] After the moving environment recognition device (400) configures the node-edge through horizontal updates, the second ring of (b) in FIG. 20 can be concisely represented as in FIG. 21.

[0185] In Fig. 21 (a), the nodes formed in the second ring of Fig. 20 (b) are colored with labels and connected by designated edges.

[0186] The circular connection test is the same ring The last node within of The point in and the first node of The horizontal distance between the points in Check if it is less than or equal to. Here, silver This is the total number of nodes within.

[0187] A circular connection test is necessary to prevent point separation in 3D LiDAR.

[0188] The skip connection test involves two nodes Non-neighboring nodes are merged only if they are located within the range. Skip connection testing plays an important role in situations where objects occur frequently.

[0189] In Fig. 21 (b), the green and purple nodes belong to the same object but are separated by the blue node due to occlusion. The two separated nodes pass the skip connection test, and the label of the latter node is updated from purple to green according to the color of the previous node.

[0190] In Fig. 21 (c), since the last and first nodes pass the circular connection test, the labels of the fourth and last nodes can be updated to red according to the color of the first node.

[0191] The mobile environment recognition device (400) can effectively utilize concise expressions and apply tests based on skipped linkage and circular linkage to prevent over-segmentation caused by obscuring objects.

[0192] <Vertical Connectivity Update Phase>

[0193] The mobile environment recognition device (400) performs a vertical connectivity update step when the ring completes the horizontal connectivity update step, and the horizontal connectivity update step and the vertical connectivity update step are repeatedly executed for each of the plurality of rings.

[0194] The vertical connectivity update step finds overlaps and the edge distance between two nodes. Binary search can be performed to efficiently calculate.

[0195] As shown in FIG. 22(a), the search space expands the window size to reduce the possibility of vertical separation by configuring more possible edges. It is expanded. Subsequently, the node in the current ring is obtained by using binary search in the previous ring. Find overlapping lower and upper bound nodes for each index. This step can create ring-to-ring connections from all nodes between the current node and the lower and upper bounds.

[0196] Figure 22 (b) shows the vertical distance between the edges of the rings. It indicates the operation in which is produced.

[0197] In the general way Is class It is defined as the distance between points. However, when a large number of points are clustered into a single node, the order in which the distance is calculated is important.

[0198] Points identified as having been collected with the same local index within two rings are more likely to be related as they were measured from a single object, so the point distance is calculated starting from the index where the points overlap. This method can save a significant amount of redundant calculations.

[0199] The foregoing description is merely an illustrative explanation of the technical concept of the embodiments of the present invention, and those skilled in the art to which the embodiments of the present invention pertain will be able to make various modifications and variations within the scope that does not deviate from the essential characteristics of the embodiments of the present invention. Accordingly, the embodiments of the present invention are intended to explain, not limit, the technical concept of the embodiments of the present invention, and the scope of the technical concept of the embodiments of the present invention is not limited by these embodiments. The scope of protection of the embodiments of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the embodiments of the present invention. Explanation of the symbols

[0200] 100: Mobile Environment Recognition System 200: Sensor device 300: Moving object

Claims

Claim 1 A method for recognizing a moving environment based on sensor data in a moving environment recognition device comprises: a data acquisition step of acquiring point cloud data from at least one sensor equipped in a moving body; a preprocessing step of correcting the point cloud data based on the movement of the moving body; a first classification processing step of forming a ground model by projecting the corrected point cloud data onto a graph structure of a triangular grid field and performing ground modeling based on Principal Component Analysis (PCA), and performing a first classification processing for ground data and non-ground data; and a second classification processing step of generating a moving environment recognition result by performing a second classification processing for an object using the non-ground data, wherein the first classification processing step comprises: a step of projecting the point cloud data onto a graph structure of a triangular grid field; a step of forming the ground model for each of at least one node included in the triangular grid field; and a step of determining the possibility of movement between the at least one node based on the relationship between nodes due to local convexity and concavity. A method for recognizing a moving environment, characterized by including the step of redefining a ground model for at least one node based on the mobility possibility, and classifying and processing ground data and non-ground data based on the redefined final ground model. Claim 2 A method for recognizing a moving environment according to claim 1, wherein the data acquisition step further acquires at least one of inertial measurement data and attitude data for the moving body. Claim 3 A method for recognizing a moving environment according to claim 2, wherein the preprocessing step is characterized by correcting the point cloud data by integrating and processing at least one sensor data included in the point cloud data and the inertial measurement data. Claim 4 A method for recognizing a moving environment according to claim 3, wherein the preprocessing step corrects the point cloud data based on pitch and roll angles calculated from the attitude of the moving body. Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A method for recognizing a movement environment according to claim 1, wherein the step of classifying and processing ground data and non-ground data is characterized by assigning weights to each corner of a node for a ground model selected based on the mobility possibility and redefining it as the final ground model. Claim 9 A method for recognizing a moving environment according to claim 8, wherein the step of classifying and processing ground data and non-ground data comprises classifying ground data by determining a node corresponding to the final ground model as ground, and classifying non-ground data by determining the remaining nodes excluding the node corresponding to the final ground model as non-ground. Claim 10 A method for recognizing a moving environment according to claim 9, wherein the step of classifying and processing ground data and non-ground data is characterized by determining a node whose planar value corresponding to the final ground model is less than a preset distance threshold as ground and classifying it as ground data. Claim 11 A method for recognizing a moving environment according to claim 1, wherein the second classification processing step comprises: a step of projecting the three-dimensional non-ground data onto a two-dimensional spatial structure to convert it into a two-dimensional graph structure; a step of processing a horizontal connectivity update to provide connectivity between nodes based on the two-dimensional graph; and a step of processing a vertical connectivity update to provide connectivity between nodes based on the two-dimensional graph so that classification of the object is processed. Claim 12 A method for recognizing a moving environment according to claim 11, wherein the second classification processing step further comprises a step of generating a moving environment recognition result including the ground data and object classification data that classifies the object. Claim 13 A method for recognizing a moving environment according to claim 11, wherein the step of processing the horizontal connectivity update is characterized by applying at least one test among skipped linkage and circular linkage to minimize errors caused by object overlap. Claim 14 A method for recognizing a moving environment according to claim 11, wherein the step of processing the vertical connectivity update is characterized by minimizing redundant calculations by prioritizing point distance calculations for duplicate nodes based on indices in the two-dimensional graph structure. Claim 15 A moving environment recognition device comprises: a memory storing one or more programs for moving environment recognition; and one or more processors performing operations for moving environment recognition according to the one or more programs, wherein the operations performed by the processors include: a data acquisition step of acquiring point cloud data from at least one sensor equipped in a moving body; a preprocessing step of correcting the point cloud data based on the movement of the moving body; a first classification processing step of forming a ground model by projecting the corrected point cloud data onto a graph structure of a triangular grid field and performing ground modeling based on Principal Component Analysis (PCA), and performing a first classification processing for ground data and non-ground data; and a second classification processing step of generating a moving environment recognition result by performing a second classification processing for an object using the non-ground data, wherein the first classification processing step comprises: a step of projecting the point cloud data onto a graph structure of a triangular grid field; and a step of forming the ground model for each of at least one node included in the triangular grid field. A moving environment recognition device characterized by comprising: a step of determining the possibility of movement between at least one node based on the relationship between nodes based on local convexity and concavity; and a step of redefining a ground model for at least one node based on the possibility of movement, and classifying and processing ground data and non-ground data based on the redefined final ground model.

Images