Vehicle utilizing spatial information acquired using sensors, sensing device utilizing spatial information acquired using sensors, and server

Abstract

To provide a vehicle utilizing space information acquired using sensors, a sensing device utilizing space information acquired using the sensors, and a server.SOLUTION: There are disclosed a vehicle and a sensing device utilizing information relating to a tracked three-dimensional space, configured to: utilize at least one sensor; continuously sense a three-dimensional space; acquire time-based space information relating to the sensed three-dimensional space; identify at least one object in the sensed three-dimensional space by use of an object classification model of a neural network base on the acquired time-based information; and track the sensed three-dimensional space including at least one identified object. A server therefor is also disclosed.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a vehicle and a sensing device that utilizes spatial information acquired using a sensor, and a server therefor. [Background technology]

[0002] The Fourth Industrial Revolution has sparked interest in technology fields such as autonomous vehicles, drones, and robots. For autonomous vehicles, drones, and robots to operate stably and accurately, it is important to collect data necessary for motion control. In this regard, research has been conducted into methods for utilizing various sensors. Summary of the Invention [Problem to be solved by the invention]

[0003] The problem to be solved by the present invention is to provide a vehicle and a sensing device that utilize spatial information acquired using sensors, as well as a server therefor.

[0004] Additional aspects will be set forth in part in the description that follows, and will be obvious from the description, or may be learned by practice of presented embodiments of the present disclosure. [Means for solving the problem]

[0005] The vehicle according to the first aspect also includes a sensor unit that continuously senses a three-dimensional space using at least one sensor; a memory that stores computer executable instructions; and a processor that executes the computer executable instructions to acquire time-varying spatial information related to the sensed three-dimensional space, identifies at least one object in the sensed three-dimensional space using a neural network-based object classification model for the acquired time-varying spatial information, tracks the sensed three-dimensional space including the identified at least one object, and controls driving of the vehicle based on information related to the tracked three-dimensional space and information related to the movement and attitude of the vehicle.

[0006] A sensing device according to a second aspect may include a sensor unit that continuously senses a three-dimensional space using at least one sensor; a communication interface device; a memory that stores computer-executable instructions; and a processor that executes the computer-executable instructions to acquire time-varying spatial information related to the sensed three-dimensional space, identifies at least one object in the sensed three-dimensional space using a neural network-based object classification model for the acquired time-varying spatial information, tracks the sensed three-dimensional space including the identified at least one object, and transmits information related to the tracked three-dimensional space to an external device via the communication interface device.

[0007] According to the third aspect, the server may include a communication interface device, a memory storing computer-executable instructions, and a processor that executes the computer-executable instructions to receive, via the communication interface device, information relating to a three-dimensional space corresponding to a moving position of at least one vehicle tracked by the at least one vehicle, and information relating to a three-dimensional space corresponding to a fixed position of the at least one sensing device tracked by at least one sensing device installed on a route along which the vehicle moves, and reconstruct information relating to a three-dimensional space corresponding to a predetermined area to which both the moving position of the at least one vehicle and the fixed position of the at least one sensing device belong, based on the information relating to the three-dimensional space corresponding to the moving position of the at least one vehicle and the information relating to the three-dimensional space corresponding to the fixed position of the at least one sensing device.

[0008] The foregoing and other aspects, features, and advantages of one embodiment of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an arbitrary driving environment in which a vehicle and a sensing device are located, according to an embodiment; [Figure 2] 1 is a block diagram showing a configuration of a vehicle according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing a configuration of a sensing device according to an embodiment. [Figure 4] FIG. 2 is a block diagram illustrating a configuration of a server according to an embodiment. [Figure 5] FIG. 2 is a block diagram illustrating a hierarchical structure of servers according to one embodiment. [Figure 6] 1 is a diagram illustrating a manner in which a vehicle travels based on information related to a tracked three-dimensional space according to an embodiment; [Figure 7]10 is a diagram illustrating a state in which a vehicle travels based on information related to a tracked three-dimensional space according to another embodiment; [Figure 8] 10 is a diagram illustrating how a sensing device tracks information related to a three-dimensional space corresponding to a fixed position of the sensing device, according to an embodiment; [Figure 9] 1 is a diagram illustrating a state in which a vehicle travels based on information related to a three-dimensional space that is tracked by the vehicle and corresponds to a moving position of the vehicle, according to an embodiment; [Figure 10] 1 is a diagram illustrating how a vehicle travels based on information related to a three-dimensional space tracked by the vehicle and corresponding to the vehicle's moving position, and information related to a three-dimensional space tracked by a sensing device and corresponding to the fixed position of the sensing device, according to one embodiment. [Figure 11] 10 is a flowchart illustrating a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. [Figure 12] 10 is a diagram illustrating each step of a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. [Figure 13] 10 is a diagram illustrating each step of a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. [Figure 14] 10 is a diagram illustrating each step of a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. [Figure 15] 10 is a diagram illustrating each step of a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. [Figure 16]10 is a diagram illustrating each step of a process of tracking a sensed three-dimensional space and predicting information related to the tracked three-dimensional space based on time-dependent spatial information related to the three-dimensional space. DETAILED DESCRIPTION OF THE INVENTION

[0010] A vehicle according to an embodiment of the present disclosure may include a sensor unit that continuously senses a three-dimensional space using at least one sensor; a memory that stores computer executable instructions; and a processor that executes the computer executable instructions to acquire time-varying spatial information related to the sensed three-dimensional space, identifies at least one object in the sensed three-dimensional space using a neural network-based object classification model for the acquired time-varying spatial information, tracks the sensed three-dimensional space including the identified at least one object, and controls driving of the vehicle based on information related to the tracked three-dimensional space and information related to movement and attitude of the vehicle.

[0011] Hereinafter, various embodiments will be described in detail with reference to the drawings. The embodiments described below may be implemented in various different forms. In order to more clearly describe the features of the present embodiments, detailed descriptions of matters that are well known to those skilled in the art to which the following embodiments pertain will be omitted.

[0012] This embodiment relates to a vehicle, a sensing device, and a server therefor that utilize spatial information acquired using sensors, and detailed explanations of matters that are well known to those skilled in the art in the technical field to which the following embodiments belong will be omitted.

[0013] FIG. 1 is a diagram illustrating an arbitrary driving environment in which a vehicle 100 and a sensing device 200 are located, according to an embodiment.

[0014] Referring to FIG. 1, a vehicle 100 stops at an intersection while traveling and waits for a traffic light. Sensing devices 200-1, 200-2, 200-3, and 200-4 (hereinafter also referred to as "sensing devices 200") are located near each corner of the intersection.

[0015] The vehicle 100 can be a vehicle such as a car or train that runs on a road or rail. However, the meaning can be expanded to include an aerial vehicle such as a drone or airplane when it runs in the air, not on a road or rail, or a watercraft such as a boat or ship when it runs on the sea. For convenience of explanation, the following description will be given assuming that the vehicle 100 is an autonomous vehicle. For autonomous driving, the vehicle 100 can sense the surrounding space and acquire spatial information using sensors.

[0016] The sensing device 200 is a device capable of sensing the surrounding space and acquiring spatial information, and may include at least one sensor. The sensing device 200 may be installed on the ground or at a predetermined height from the ground. The sensing device 200 may also be installed by being attached or fixed to an existing facility.

[0017] Each of the vehicle 100 and the sensing device 200 may include at least one of various types of sensors, such as a light detection and ranging (LiDAR) sensor, a radar sensor, a camera sensor, an infrared image sensor, an ultrasonic sensor, etc. To acquire spatial information related to three-dimensional space, each of the vehicle 100 and the sensing device 200 may use a plurality of sensors of the same type or a combination of different types of sensors, taking into consideration the sensing ranges of the various sensors and the types of data that can be acquired.

[0018] Depending on the types of sensors provided in the vehicle 100 and the sensing device 200, the sensing ranges that the vehicle 100 and the sensing device 200 can detect may be the same or different. Referring to FIG. 1, the sensing ranges that the vehicle 100 and the sensing device 200 can detect are illustrated. As illustrated in FIG. 1, a first sensing range (sensing range 1) that the vehicle 100 can detect is illustrated as being narrower than a second sensing range (sensing range 2) that the sensing device 200 can detect, but this is not limited thereto. Even if the types of sensors provided in the vehicle 100 and the sensing device 200 are the same, the sensing ranges of the vehicle 100 and the sensing device 200 may differ depending on the positions at which the sensors are installed and the surrounding environment. For example, the sensing device 200 may be located higher than the vehicle 100, and a moving vehicle 100 may encounter various objects in close proximity that interfere with three-dimensional spatial sensing compared to a fixed sensing device 200, so the second sensing range that the sensing device 200 can sense is wider than the first sensing range that the vehicle 100 can sense.

[0019] The vehicle 100 can directly acquire spatial information related to the surrounding three-dimensional space using sensors for autonomous driving. The vehicle 100 can receive spatial information that it could not acquire directly from the outside in order to acquire spatial information related to a wider space corresponding to the driving direction in advance. For example, the vehicle 100 can receive spatial information acquired by other vehicles or the sensing device 200 in the vicinity.

[0020] Below, we will explain in detail a method for using spatial information acquired by the vehicle 100 for autonomous driving, a method for transmitting spatial information acquired by the sensing device 200 to other devices in the vicinity, and a method for combining spatial information acquired by the vehicle 100 and the sensing device 200 to acquire and utilize spatial information related to a larger three-dimensional space.

[0021] FIG. 2 is a block diagram showing the configuration of a vehicle according to one embodiment.

[0022] 2, a vehicle 100 according to an embodiment may include a memory 110, a processor 120, a communication interface device 130, a sensor unit 140, and a user interface device 150. A person skilled in the art of the present invention would know that other general components may be included in addition to the components illustrated in FIG.

[0023] The memory 110 may store software and / or programs. For example, the memory 110 may store programs such as applications, application programming interfaces (APIs), and various data. The memory 110 may store instructions that can be executed by the processor 120.

[0024] The processor 120 can access and use data stored in the memory 110 and store new data in the memory 110. The processor 120 can execute instructions stored in the memory 110. The processor 120 can execute computer programs installed in the vehicle 100. The processor 120 can also install computer programs or applications received from an external source into the memory 110. The processor 120 may include at least one processing module. The processing module may be a dedicated processing module for executing a predetermined program. For example, the processor 120 may include various processing modules, each in the form of a separate dedicated chip, for executing a vehicle control program for autonomous driving, such as an advanced driver assistance system (ADAS), or a processing module for executing a three-dimensional space tracking program. The processor 120 can control other components included in the vehicle 100 to perform operations corresponding to execution results, such as instructions or computer programs.

[0025] The communication interface device 130 can perform wireless communication with other devices or networks. To this end, the communication interface device 130 may include a communication module supporting at least one of various wireless communication methods. For example, the communication interface device 130 may include a communication module for short-range communication such as Wi-Fi (wireless fidelity), various types of mobile communication such as 3G, 4G, and 5G, or ultra-wideband communication. The communication interface device 130 can be connected to a device located outside the vehicle 100 to transmit and receive signals or data. The vehicle 100 communicates with the sensing device 200 or other vehicles via the communication interface device 130, or may be connected to a regional server that manages the area in which the vehicle 100 is located.

[0026] The sensor unit 140 may include at least one sensor for sensing a three-dimensional space. The sensor unit 140 may sense an object located within a sensing range and acquire data that can be used to generate coordinates of the sensed object in a three-dimensional space. The sensor unit 140 may acquire shape data or distance data related to an object located within the sensing range. The sensor unit 140 may include at least one of various sensors, such as a LiDAR sensor, a radar sensor, a camera sensor, an infrared image sensor, or an ultrasonic sensor. For example, the sensor unit 140 may include at least one three-dimensional LiDAR sensor to acquire data related to a 360° range of space, or may further include at least one of a radar sensor and an ultrasonic sensor to acquire data related to a blind spot that the three-dimensional LiDAR sensor cannot sense or a nearby space within a predetermined distance from the vehicle 100.

[0027] The user interface device 150 can receive user inputs from a user. The user interface device 150 can display information such as execution results of computer programs in the vehicle 100, processing results corresponding to user inputs, and the status of the vehicle 100. For example, a user can select and execute a computer program to be executed from among various computer programs installed in the vehicle 100 through the user interface device 150. The user interface device 150 can include hardware units for receiving inputs and providing outputs, and can also include dedicated software modules for driving them. For example, the user interface device 150 can be, but is not limited to, a touch screen.

[0028] Although not shown in FIG. 2 , the vehicle 100 may further include components required for autonomous driving, such as a global positioning system (GPS) and inertial measurement units (IMUs). The GPS is a satellite navigation system that receives signals sent from GPS satellites and calculates the current position of the vehicle 100. The IMU is a device that measures the speed, direction, gravity, and acceleration of the vehicle 100. The processor 120 can obtain information related to the movement and attitude of the vehicle 100 using the GPS and IMU. The processor 120 can also obtain other information related to controlling the vehicle 100 from other sensors and memories provided in the vehicle 100.

[0029] The processor 120 may execute computer-executable instructions to acquire time-varying spatial information related to a continuously sensed three-dimensional space using at least one sensor. The processor 120 may identify at least one object in the sensed three-dimensional space using a neural network-based object classification model for the acquired time-varying spatial information, and track the sensed three-dimensional space including the identified at least one object. The processor 120 may control the driving of the vehicle 100 based on information related to the tracked three-dimensional space and information related to the movement and attitude of the vehicle. The information related to the tracked three-dimensional space may include spatial information related to a space in which the identified at least one object is located and dynamic information related to the movement of the identified at least one object.

[0030] The processor 120 receives a timestamp recording time information and data related to the sensed three-dimensional space from the sensor unit 140, and generates time-dependent spatial information related to the sensed three-dimensional space as a three-dimensional image. The spatial information related to the sensed three-dimensional space can obtain movement coordinate values ​​corresponding to the movement position of the vehicle 100 through the GPS, and is therefore also mapped to a corresponding part of a coordinate system based on a predetermined coordinate, for example, an absolute coordinate system based on an origin.

[0031] The sensor unit 140 may continuously sense a three-dimensional space in different sensing ranges that are concentric spheres using a plurality of three-dimensional lidar sensors. The processor 120 may acquire time-dependent spatial information related to the three-dimensional spaces sensed in the different sensing ranges, and may assign weights related to accuracy to commonly identified objects based on the acquired time-dependent spatial information, thereby tracking the three-dimensional space.

[0032] The processor 120 determines at least one attribute information of the type, three-dimensional shape, position, posture, size, trajectory, and speed of at least one identified object in the sensed three-dimensional space, tracks the three-dimensional space, predicts information related to the tracked three-dimensional space, and controls the driving of the vehicle 100 based on the predicted information.

[0033] The processor 120 can reflect information related to the tracked three-dimensional space, which changes as the vehicle 100 moves, and information related to the movement and posture of the vehicle 100 in real time at a processing speed of 10 Hz to 20 Hz, and control the driving of the vehicle 100.

[0034] The processor 120 can receive, via the communication interface device 130, information related to the three-dimensional space that is tracked by the at least one sensing device 200 and that corresponds to a fixed position of the at least one sensing device 200, from the at least one sensing device 200 provided on a path traveled by the vehicle 100. The processor 120 can control the traveling of the vehicle 100 further based on the information related to the three-dimensional space that corresponds to the fixed position of the at least one sensing device 200.

[0035] The processor 120 may classify the at least one identified object using the neural network-based object classification model into one of a first type object corresponding to a vehicle equal to or larger than a predetermined standard, a second type object corresponding to a light vehicle or a motorcycle below a predetermined standard, a third type object corresponding to a pedestrian, a fourth type object corresponding to a path of the vehicle 100, and a fifth type object corresponding to other detected objects other than the first type object or the fourth type object. The processor 120 may display the at least one classified object separately in the tracked three-dimensional space via the user interface device 150.

[0036] FIG. 3 is a block diagram showing the configuration of a sensing device 200 according to an embodiment.

[0037] 3, a sensing device 200 according to an embodiment may include a memory 210, a processor 220, a communication interface device 230, and a sensor unit 240. A person skilled in the art of the present invention would know that other general components may be included in addition to the components illustrated in FIG.

[0038] The memory 210 may store software and / or programs, and may store instructions that are executable by the processor 220.

[0039] The processor 220 can access and use data stored in the memory 210 and can store new data in the memory 210. The processor 220 can execute instructions stored in the memory 210. The processor 220 can execute computer programs installed in the sensing device 200. The processor 220 can also install computer programs or applications received from the outside into the memory 210. The processor 220 may include at least one processing module. For example, the processor 220 may include a processing module in the form of a dedicated processing module that executes a three-dimensional space tracking program. The processor 220 can control other components included in the sensing device 200 to perform operations corresponding to execution results of instructions or computer programs, etc.

[0040] The communication interface device 230 can perform wired or wireless communication with other devices or networks. To this end, the communication interface device 230 may include a communication module supporting at least one of various wired and wireless communication methods. For example, the communication interface device 230 may include a communication module that performs short-range communication such as Wi-Fi, wireless communication such as various mobile communications, or wired communication using a coaxial cable or an optical cable. The communication interface device 230 may be connected to a device located outside the sensing device 200 to transmit and receive signals or data. The sensing device 200 may communicate with the vehicle 100 or other sensing devices via the communication interface device 230, or may be connected to a regional server that manages the area in which the sensing device 200 is located.

[0041] The sensor unit 240 may include at least one sensor for sensing a three-dimensional space. The sensor unit 240 may sense an object located within a sensing range and acquire data that can be used to generate coordinates of the sensed object in a three-dimensional space. The sensor unit 240 may acquire shape data or distance data related to an object located within the sensing range. The sensor unit 240 may include at least one of various sensors, such as a LiDAR sensor, a radar sensor, a camera sensor, an infrared image sensor, or an ultrasonic sensor. For example, the sensor unit 240 may include at least one three-dimensional LiDAR sensor to acquire data related to a 360° range of space, or may further include at least one of a radar sensor and an ultrasonic sensor to acquire data related to a blind spot that the three-dimensional LiDAR sensor cannot sense or a nearby space within a predetermined distance from the sensing device 200.

[0042] The processor 220 executes computer-executable instructions to acquire time-varying spatial information related to a continuously sensed three-dimensional space using at least one sensor. The processor 220 may identify at least one object in the sensed three-dimensional space using a neural network-based object classification model for the acquired time-varying spatial information, and track the sensed three-dimensional space including the identified at least one object. The processor 220 may transmit information related to the tracked three-dimensional space to an external device via the communication interface device 230.

[0043] The processor 220 may receive a timestamp recording time information and data related to the sensed 3D space from the sensor unit 240 and generate time-dependent spatial information related to the sensed 3D space as a 3D image. The spatial information related to the sensed 3D space may have fixed coordinate values ​​corresponding to a fixed position of the sensing device 200, and may therefore be mapped to a corresponding part of a coordinate system based on a predetermined coordinate, for example, an absolute coordinate system based on an origin.

[0044] The processor 220 may determine at least one attribute information of the type, three-dimensional shape, position, posture, size, trajectory, and speed of at least one object identified in the sensed three-dimensional space, and track the object in the three-dimensional space. The processor 220 may transmit information related to the tracked three-dimensional space to at least one of a vehicle 100, another sensing device 200, and a server 300 within a predetermined distance from the sensing device 200 via the communication interface device 230.

[0045] FIG. 4 is a block diagram showing the configuration of the server 300 according to an embodiment.

[0046] 4, a server 300 according to an embodiment may include a memory 310, a processor 320, and a communication interface device 330. A person skilled in the art of the present invention will know that other general components may be included in addition to the components illustrated in FIG.

[0047] The memory 310 may store software and / or programs, and may store instructions that are executable by the processor 320.

[0048] The processor 320 can use data stored in the memory 310 and store new data in the memory 310. The processor 320 can execute instructions stored in the memory 310. The processor 320 can execute computer programs installed in the server 300. The processor 320 may include at least one processing module. The processor 320 can control other components included in the server 300 to perform operations corresponding to the execution results of the instructions or computer programs, etc.

[0049] The communication interface device 330 may perform wired or wireless communication with other devices or networks. The communication interface device 330 may be connected to devices located outside the server 300 and may transmit and receive signals or data. The server 300 may communicate with the vehicle 100 or the sensing device 200 via the communication interface device 330, or may be connected to other servers connected to a network.

[0050] The processor 320 executes computer-executable instructions to receive, via the communication interface device 330, information on a three-dimensional space corresponding to a moving position of the at least one vehicle 100 tracked by the at least one vehicle 100, and information on a three-dimensional space corresponding to a fixed position of the at least one sensing device 200 tracked by the at least one sensing device 200 installed on a route traveled by the vehicle 100. The processor 320 can reconstruct information on a three-dimensional space corresponding to a predetermined area to which both the moving position of the at least one vehicle 100 and the fixed position of the at least one sensing device 200 belong, based on the information on a three-dimensional space corresponding to the moving position of the at least one vehicle 100 and the information on a three-dimensional space corresponding to the fixed position of the at least one sensing device 200.

[0051] The processor 320 can transmit information relating to the reconstructed three-dimensional space corresponding to the predetermined area to a higher-level integrated server via the communication interface device 330 .

[0052] FIG. 5 is a block diagram illustrating the hierarchical structure of servers 300-1, 300-2 through 300-N, and 400 according to one embodiment.

[0053] 5, it can be seen that the sensing ranges that the vehicle 100 and the sensing device 200 can sense belong to a predetermined area. Such a predetermined area is managed by one of the servers 300-1, 300-2, to 300-N (hereinafter, collectively referred to as the server 300) corresponding to the area, which corresponds to the predetermined area.

[0054] The server 300 can reconstruct information about the three-dimensional space corresponding to the entire area by collecting information about the three-dimensional space tracked by the vehicles 100 and the sensing devices 200 within the area managed by the server 300 and mapping it to the corresponding space within the area. In other words, the server 300, which corresponds to a regional server, can obtain information about the three-dimensional space related to the area it manages. If a predetermined condition is satisfied or if requested, the server 300, which corresponds to a regional server, can transmit part or all of the information about the three-dimensional space related to the area it manages to the vehicles 100 and the sensing devices 200 located within the area. The server 300 can also transmit the information about the three-dimensional space related to the area it manages to the server 400, which corresponds to a global server that manages the regional servers. The server 400 is also an integrated server at a higher level than the server 300.

[0055] The server 400 can acquire information about the three-dimensional space corresponding to the entire area managed by the server 400 by collecting information about the three-dimensional space corresponding to a predetermined area reconstructed by the regional server and mapping it to the corresponding space within the entire area. In other words, the server 400 corresponding to the global server can acquire information about the three-dimensional space related to the entire area managed by the server 400. The server 400 corresponding to the global server can transmit part or all of the information about the three-dimensional space related to the entire area managed by the server 400 to the server 300 corresponding to the regional server when a predetermined condition is satisfied or when requested.

[0056] 5, a hierarchical structure can be formed among vehicle 100 including sensor unit 140, sensing device 200 including sensor unit 240, server 300 corresponding to a regional server, and server 400 corresponding to a global server. Information related to the three-dimensional space tracked by vehicle 100 and sensing device 200 is transferred to a higher layer and integrated, and becomes information related to the three-dimensional space corresponding to the global space.

[0057] FIG. 6 is a diagram illustrating how the vehicle 100 travels based on information related to a tracked three-dimensional space according to an embodiment.

[0058] 6, it can be seen that the vehicle 100 travels based on information related to a three-dimensional space corresponding to the moving position of the vehicle 100 and information related to the movement and posture of the vehicle 100. The information related to the three-dimensional space corresponding to the moving position of the vehicle 100 may include at least one object. As shown in FIG. 6, a vehicle above a predetermined standard is a first type of object (object of type 1), a light vehicle or a two-wheeled vehicle below a predetermined standard is a second type of object (object of type 2), a pedestrian is a third type of object (object of type 3), the path of the vehicle 100 is a fourth type of object (object of type 4), and other detected objects other than the first or fourth type of object are also classified as a fifth type of object (object of type 5), and the objects of each type may have different three-dimensional shapes including heights, sizes, colors, etc. The position, posture, speed, etc. of each object are determined based on information related to the three-dimensional space tracked according to the moving position of the vehicle 100, and by continuously tracking each object, it is possible to confirm or predict its displacement, amount of change, trajectory, transition, etc. If the vehicle 100 is an autonomous vehicle, it is also used as data for tracking surrounding objects that change as the vehicle 100 moves and controlling the driving of the vehicle 100.

[0059] As shown in the lower right corner of FIG. 6, the speed, direction and steering angle of the vehicle 100 can be confirmed, and changes therein can be tracked as the vehicle 100 moves.

[0060] FIG. 7 is a diagram illustrating how a vehicle 100 travels based on information related to a tracked three-dimensional space according to another embodiment.

[0061] 7, it can be seen that the vehicle 100 travels based on information related to a three-dimensional space corresponding to the moving position of the vehicle 100 and information related to the movement and attitude of the vehicle 100. Compared to the previously described FIG. 6, it can be seen that the number and spacing of the concentric circles surrounding the vehicle 100 are different. In the case of FIG. 7, it can be seen that the vehicle 100 continuously senses the three-dimensional space in different sensing ranges that are concentric spheres using a plurality of three-dimensional lidar sensors, and acquires time-dependent spatial information related to the three-dimensional space sensed in the different sensing ranges, thereby tracking the three-dimensional space corresponding to the moving position of the vehicle 100.

[0062] FIG. 8 is a diagram illustrating how the sensing device 200 tracks information related to a three-dimensional space corresponding to a fixed position of the sensing device 200, according to an embodiment.

[0063] Referring to FIG. 8, two sensing devices 200 are positioned a predetermined distance apart, and each sensing device 200 continuously senses a three-dimensional space using at least one sensor and tracks information related to the three-dimensional space corresponding to the fixed position of the sensing device 200. It can be seen that a car corresponding to ID 185 is moving away from the sensing device 200 at a speed of approximately 20 km / h, and a pedestrian corresponding to ID 156 is moving from the sensing device 200 on the left to the sensing device 200 on the right at a speed of approximately 5 km / h. Arrows corresponding to the moving direction of each object can be displayed to indicate the moving direction of each object. Since such sensing devices 200 can determine which objects are passing or entering within their sensing range, they can be used for security purposes and for monitoring purposes, such as observing traffic volume in specific areas.

[0064] 9 is a diagram illustrating how the vehicle 100 travels based on information related to a three-dimensional space that is tracked by the vehicle 100 and corresponds to the moving position of the vehicle 100, according to an embodiment. FIG. 10 is a diagram illustrating how the vehicle 100 travels based on information related to a three-dimensional space that is tracked by the vehicle 100 and corresponds to the moving position of the vehicle 100, and information related to a three-dimensional space that is tracked by the sensing device 200 and corresponds to the fixed position of the sensing device 200, according to an embodiment.

[0065] 9 and 10, the vehicle 100 shown in Fig. 9 uses only the sensors provided in the vehicle 100 to acquire information related to the three-dimensional space corresponding to the moving position of the vehicle 100, and therefore the sensing range that the sensors can sense is limited when there are objects surrounding the vehicle 100. In contrast, the vehicle 100 shown in Fig. 10 not only uses the sensors provided in the vehicle 100 to acquire information related to the three-dimensional space corresponding to the moving position of the vehicle 100, but also uses sensors provided in at least one sensing device 200 installed along the path that the vehicle 100 moves to acquire information related to the three-dimensional space corresponding to the fixed position of the sensing device 200, and therefore it can be seen that the three-dimensional space that the vehicle 100 in Fig. 10 can sense is much wider than the three-dimensional space that the vehicle 100 in Fig. 9 can sense. The vehicle 100 of Figure 10 can track pedestrians that fall under the third type of object and all vehicles that fall under the first type of object and check the information related thereto, but the vehicle 100 of Figure 9 cannot track pedestrians that fall under the third type of object and some vehicles that fall under the first type of object.

[0066] Figure 11 is a flowchart illustrating a process of tracking a sensed three-dimensional space based on time-dependent spatial information related to the three-dimensional space and predicting information related to the tracked three-dimensional space. Figures 12 to 16 are diagrams illustrating each step of a process of tracking a sensed three-dimensional space based on time-dependent spatial information related to the three-dimensional space and predicting information related to the tracked three-dimensional space. Hereinafter, with reference to Figures 11 to 16, a process of tracking a sensed three-dimensional space based on time-dependent spatial information related to the three-dimensional space and predicting information related to the tracked three-dimensional space will be described.

[0067] In operation 1110, the vehicle 100 or the sensing device 200 may distinguish a ground region from an object region based on the time-varying spatial information related to the three-dimensional space. The time-varying spatial information related to the three-dimensional space sensed by the vehicle 100 or the sensing device 200 may be in the form of point cloud data.

[0068] The vehicle 100 or the sensing device 200 may classify a ground region from temporal spatial information related to a three-dimensional space. The vehicle 100 or the sensing device 200 may classify point cloud data corresponding to a ground region from point cloud data. The vehicle 100 or the sensing device 200 may first classify a ground region from temporal spatial information related to a three-dimensional space, and then classify the remaining portion into an object region including at least one object. The vehicle 100 or the sensing device 200 may apply fitting based on a stochastic model to find a ground estimation model. The vehicle 100 or the sensing device 200 may learn ground shapes in real time and classify each point cloud data as whether it is point cloud data corresponding to a ground region.

[0069] 12 shows a process of separating a ground area and an object area from spatial information at a specific time related to a three-dimensional space sensed by the vehicle 100 or the sensing device 200. For convenience of explanation, FIGS. 12 to 16 will be described using an example of spatial information including three objects and the ground, as shown in the drawings.

[0070] Spatial information at a specific time related to a three-dimensional space sensed by the vehicle 100 or the sensing device 200 is point cloud data without distinction between objects and the ground, and is also point cloud data corresponding to all sensed objects, as shown in the upper part of Fig. 12. The vehicle 100 or the sensing device 200 can separate point cloud data estimated to be a ground region from the overall point cloud data as shown in the upper part of Fig. 12, thereby separating point cloud data corresponding to the ground region and point cloud data corresponding to an object region, as shown in the lower part of Fig. 12. In this case, the object region includes at least one object, but is not separated for each object, and is also point cloud data corresponding to the entire object.

[0071] 11 again, the vehicle 100 or the sensing device 200 may classify individual object regions by clustering the object regions in operation 1120. The vehicle 100 or the sensing device 200 may further classify the object region classified as the ground region into individual object regions so that each object can be classified separately. Since the object region classified as the ground region corresponds to point cloud data corresponding to the entire object, the vehicle 100 or the sensing device 200 may classify the point cloud data for each object by clustering the point cloud data corresponding to the entire object.

[0072] Referring to FIG. 13, a process in which the vehicle 100 or the sensing device 200 separates individual object regions corresponding to each object from the ground region and the separated object region is shown.

[0073] As shown in the upper part of Fig. 13, point cloud data corresponding to an object region including the entire object excluding the ground may be separated from point cloud data corresponding to the ground region. The vehicle 100 or the sensing device 200 may cluster the point cloud data corresponding to the object region based on at least one of distance information, shape information, and distribution information, thereby separating point cloud data corresponding to individual object regions of "Object 1," "Object 2," and "Object 3" from the point cloud data corresponding to the object region including the entire object, as shown in the lower part of Fig. 13. As a result, the vehicle 100 or the sensing device 200 may acquire information related to the position, shape, number, etc. of the objects.

[0074] Referring again to FIG. 11 , in operation 1130, the vehicle 100 or the sensing device 200 may acquire object information of at least one object identified using a neural network-based object classification model for time-varying spatial information related to a three-dimensional space. The vehicle 100 or the sensing device 200 may input time-varying spatial information related to a three-dimensional space to the neural network-based object classification model, identify at least one object, and acquire object information of the identified object. The neural network-based object classification model may be trained using a training video stored in a database for each object. The neural network-based object classification model may estimate object information of each identified object based on at least one of distance information, shape information, and distribution information for point cloud data corresponding to all detected objects. The vehicle 100 or the sensing device 200 may identify only movable objects of interest, such as automobiles, light vehicles, motorcycles, and pedestrians, through a neural network-based object classification model, and estimate object information only about the objects of interest. Step 1130 may be processed in parallel with steps 1110 and 1120.

[0075] Referring to FIG. 14, a process of acquiring object information of each object from space information at a specific time related to a three-dimensional space sensed by the vehicle 100 or the sensing device 200 is shown.

[0076] The spatial information at a specific time related to the three-dimensional space sensed by the vehicle 100 or the sensing device 200 is also point cloud data corresponding to all sensed objects, as shown in the upper part of Fig. 14. The vehicle 100 or the sensing device 200 can identify and classify "Object 1," "Object 2," and "Object 3" using a neural network-based object classification model for the entire point cloud data as shown in the upper part of Fig. 14, and acquire object information for "Object 1," "Object 2," and "Object 3." By previously setting an object of the type corresponding to "Object 1" as an object of interest, it is also possible to identify and classify only "Object 1," which is the object of interest, and acquire object information only for "Object 1." As shown in the lower part of Figure 14, the vehicle 100 or the sensing device 200 can estimate the type, position, and size of each object, and can determine various types of boundary lines or bounding boxes related to each object.

[0077] 11 again, in step 1140, the vehicle 100 or the sensing device 200 may track the sensed three-dimensional space based on object information of at least one object identified using a neural network-based object classification model for the time-dependent spatial information related to the three-dimensional space acquired in step 1130 and the individual object region acquired in step 1120. Since the neural network-based object classification model used in step 1130 has difficulty identifying objects and estimating object information for objects that have not been fully trained, the vehicle 100 or the sensing device 200 may acquire information related to the position, shape, or number of objects from the individual object region acquired in step 1120 for such unidentifiable objects. In addition, since the object information estimated by the neural network-based object classification model utilized in step 1130 may differ from the actual information of the object in the sensed three-dimensional space, it can be corrected using information related to the position, shape, or number of objects acquired from the individual object region acquired in step 1120. As a result, by integrating the object information estimated by the neural network-based object classification model with information related to the object acquired from the individual object region divided through clustering, each object can be identified and accurate information related to the object's position, shape, etc. In addition, even for objects that cannot be identified or classified through the neural network-based object classification model, it is possible to track all objects in the sensed three-dimensional space without missing any of them by using information related to the object acquired from the individual object region divided through clustering.

[0078] Referring to Figures 15 and 16, the process is shown in which the vehicle 100 or the sensing device 200 integrates information related to objects acquired from individual object areas divided through clustering and information about objects estimated by a neural network-based object classification model, and tracks detected objects in three-dimensional space.

[0079] 15, the vehicle 100 or the sensing device 200 can acquire accurate information about all objects in the sensed three-dimensional space by integrating information about each object acquired based on point cloud data corresponding to the individual object regions of "Object 1," "Object 2," and "Object 3" with object information about each object identified and classified as "Object 1," "Object 2," and "Object 3." The vehicle 100 or the sensing device 200 can acquire accurate information about the objects by correcting various types of boundary lines or bounding boxes related to each object with information about each object acquired based on point cloud data corresponding to the individual object regions.

[0080] 16, the vehicle 100 or the sensing device 200 can acquire continuous information related to all objects in the sensed three-dimensional space over time from time-dependent spatial information related to the sensed three-dimensional space, and can track all objects in the sensed three-dimensional space. For example, the vehicle 100 or the sensing device 200 can track each object over time using an object tracking method using a Kalman filter. As shown in FIG. 16, the vehicle 100 or the sensing device 200 can track the speed and movement direction of each object based on the amount of change in position from continuous frame information over time, and can also record the tracking results.

[0081] 11 again, the vehicle 100 or the sensing device 200 may predict information related to a tracked three-dimensional space in operation 1150. The vehicle 100 or the sensing device 200 may accumulate information on tracked objects, analyze the movement patterns of the objects from the accumulated tracking information, and predict the movement of the objects. The vehicle 100 or the sensing device 200 may predict the movement in the tracked three-dimensional space of only the object of interest among at least one identified object, thereby reducing the amount of processing calculations related thereto and enabling efficient driving or monitoring planning.

[0082] Each of the above-described embodiments may also be provided in the form of a computer program or application stored on a computer-readable recording medium that causes an electronic device to perform a method including predetermined steps of utilizing spatial information acquired using a sensor. In other words, each of the above-described embodiments may also be provided in the form of a computer program or application stored on a computer-readable recording medium that causes an electronic device to perform a method including predetermined steps of utilizing spatial information acquired using a sensor.

[0083] The above-described embodiments may also be implemented in the form of a computer-readable recording medium storing instructions and data executable by a computer or processor, where at least one of the instructions and data may be stored in the form of program code, which, when executed by a processor, may generate a predetermined program module and perform a predetermined operation. Such computer-readable recording media include read only memory (ROM), random access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-RLTHs, BD-REs, magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid state disks (SSDs), and any device that can store instructions or software, associated data, data files, and data structures, and can provide instructions or software, associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0084] The above description focuses on the present embodiment. Those skilled in the art to which the disclosed embodiment pertains will understand that the disclosed embodiment may be embodied in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiment should be considered from an illustrative rather than a restrictive perspective. The scope of the invention is defined by the claims, not the above description of the embodiments, and all differences within the range of equivalents thereof should be construed as being within the scope of the invention.

Claims

1. a sensor unit that senses three-dimensional space; a memory for storing one or more instructions; a processor that executes the one or more commands to acquire point cloud data related to the sensed three-dimensional space, classify individual object regions from the acquired point cloud data, acquire object information of the identified objects using an object classification model, track the sensed three-dimensional space using the object information and information related to the objects acquired based on the individual object regions, and control vehicle driving based on the information related to the tracked three-dimensional space, The processor executes the one or more instructions to: The vehicle tracks the sensed three-dimensional space by correcting the object information using information related to the object.

2. The processor executes the one or more instructions to: The vehicle of claim 1 , wherein, in the case of an object that cannot be identified through the object classification model, the sensed three-dimensional space is tracked using information related to the object obtained from the individual object region.

3. The processor executes the one or more instructions to: The vehicle of claim 1 , wherein the sensed three-dimensional space is tracked by correcting a bounding box corresponding to the object information using information related to the object.

4. The processor executes the one or more instructions to: The vehicle of claim 1 , further comprising: analyzing a movement pattern of an object from the information about the tracked three-dimensional space, and predicting the information about the tracked three-dimensional space.

5. The processor executes the one or more instructions to:

2. The vehicle of claim 1, further comprising: extracting object regions from the acquired point cloud data; classifying the individual object regions by clustering the object regions; identifying objects in the three-dimensional space from the acquired point cloud data using the object classification model; and estimating object information of the identified objects.

6. The vehicle of claim 1 , wherein the information about the object is information about the position, shape, and number of the object acquired based on point cloud data corresponding to the individual object region.

7. The vehicle of claim 1 , further comprising a communication interface device, wherein the processor executes the one or more instructions to transmit information related to the tracked three-dimensional space to an external device through the communication interface device.

8. a sensor unit that senses three-dimensional space; a memory for storing one or more instructions; a processor that executes the one or more commands to acquire point cloud data related to the sensed three-dimensional space, classify individual object regions from the acquired point cloud data, acquire object information of the identified objects using an object classification model, and track the sensed three-dimensional space using information related to the objects acquired based on the individual object regions and the object information, The processor executes the one or more instructions to: The sensing device tracks the sensed three-dimensional space by correcting the object information using information related to the object.

9. 10. The sensing device of claim 8, wherein the processor executes the one or more instructions to track the sensed three-dimensional space using information related to the object acquired from the individual object region in the case of an object that cannot be identified through the object classification model.

10. The processor executes the one or more instructions to: The sensing device of claim 8 , wherein the sensed three-dimensional space is tracked by correcting a bounding box corresponding to the object information using information related to the object.

11. 10. The sensing device of claim 8, wherein the processor executes the one or more instructions to analyze a movement pattern of an object from information related to the tracked three-dimensional space and predict information related to the tracked three-dimensional space.

12. 10. The sensing device of claim 8, wherein the processor executes the one or more instructions to extract object regions from the acquired point cloud data, classify the individual object regions by clustering the object regions, identify objects in the three-dimensional space from the acquired point cloud data using the object classification model, and estimate object information of the identified objects.

13. The sensing device of claim 8 , wherein the information about the objects is information about positions, shapes, and numbers of objects acquired based on point cloud data corresponding to the individual object regions.

14. The sensing device of claim 8 , further comprising a communication interface device, wherein the processor executes the one or more instructions to transmit information related to the tracked three-dimensional space to an external device through the communication interface device.

15. In an electronic device, acquiring point cloud data relating to a three-dimensional space; Segmenting individual object regions from the acquired point cloud data; acquiring object information of the identified objects using an object classification model for the acquired point cloud data; tracking a sensed three-dimensional space using information related to the object acquired based on the individual object region and the object information; The tracking step includes: A computer program stored on a recording medium for performing a method for tracking the sensed three-dimensional space by correcting the object information using information related to the object.

Images