System and method for segmentation of detected objects using autonomous driving sensors

By using LiDAR segmentation technology to segment and fuse sensor data from autonomous vehicles into multiple categories, the problem of insufficient accuracy of sensor systems in identifying the surrounding environment and objects is solved, thereby improving the safety and driving convenience of autonomous vehicles.

CN123119418APending Publication Date: 2026-11-17HYUNDAI MOTOR CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511727104.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2025-11-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing sensor systems in autonomous vehicles suffer from insufficient recognition accuracy and inadequate data fusion when identifying the surrounding environment and objects, affecting vehicle safety and driving convenience.

Method used

By employing LiDAR segmentation technology, the processor performs multi-category segmentation and fusion operations on multiple point data points. Combined with occupancy models, road models, and scene understanding models, feature points and odometer readings of the surrounding environment are extracted to achieve accurate object tracking and autonomous driving control of vehicles.

Benefits of technology

It improves the accuracy of object recognition and the ability to identify the vehicle's surrounding environment, thereby enhancing the safety and ease of driving of autonomous vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123119418A_ABST
    Figure CN123119418A_ABST
Patent Text Reader

Abstract

An apparatus and method for identifying objects using segmentation from autonomous driving sensors are disclosed. The apparatus can: receive multiple point data corresponding to objects in the vehicle's surrounding environment from the vehicle's sensors; perform multi-class segmentation on the multiple point data; identify points corresponding to the ground based on the multiple point data; perform a first fusion operation on the results of the multi-class segmentation and the identification of points corresponding to the ground; perform a second fusion operation on the object's cuboid bounding box and the object's outline; extract feature points and odometer readings from the surrounding environment; track the object based on the result of the second fusion operation; and control the vehicle's autonomous driving operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to autonomous vehicles, and more specifically, to object recognition systems and methods in autonomous vehicles. Background Technology

[0002] With the development and commercialization of autonomous vehicles, various sensors and artificial intelligence (AI) technologies are being employed to support their autonomous driving functions. For example, research is actively underway to identify various objects in front of a moving vehicle, measure the distance between objects and the vehicle, and determine which algorithms the vehicle should use to handle specific situations to ensure safety.

[0003] As a result, vehicle sensor technology continues to advance, and high-performance sensors are being installed on vehicles, such as LiDAR sensors that use laser beams to identify the vehicle's surroundings, radar sensors that use radio waves, ultrasonic sensors, fisheye cameras that can capture 360-degree images, multifocal lenses, and Global Positioning System (GPS).

[0004] A so-called super-sensor vehicle can be realized by aggregating measurement results from multiple sensors (e.g., various types of sensors). In autonomous driving (or driverless driving), the concept of a super-sensor refers to a technology aimed at improving driving convenience and safety by combining measurements from various different sensors (rather than relying solely on individual sensors) to more accurately identify the surrounding environment. With the advent of information and communication technologies (ICT) and cloud computing, the sensors required for autonomous driving and their associated artificial intelligence algorithms have become more complex than ever before. This involves not only targeting individual vehicles but also remotely accumulating data from numerous vehicle units (e.g., fleets) and training AI servers and databases to improve the accuracy of vehicle sensor identification.

[0005] The content described in this background section is only for enhancing understanding of the background of this disclosure and should not be construed as an admission that it conforms to prior art known to those skilled in the art. Summary of the Invention

[0006] This disclosure aims to address some of the problems that arise in at least some of the embodiments, while maintaining the advantages achieved by these embodiments.

[0007] One aspect of this disclosure provides a system and method for identifying the surrounding environment and objects of an autonomous vehicle based on autonomous driving sensors. More specifically, it provides a system and method that can improve object recognition, surrounding environment recognition, and the calculation of the movement of the vehicle (equipped with autonomous driving sensors) by utilizing LiDAR segmentation as the primary technical task.

[0008] The technical problems to be solved by this disclosure are not limited to those described above, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art from the following description.

[0009] To solve all or at least some of the above-mentioned technical problems, one or more embodiments of this disclosure may be implemented in the following aspects.

[0010] According to one or more exemplary embodiments of this disclosure, a vehicle apparatus may include a processor and a memory. The memory may store at least one instruction configured to, when executed by the processor, cause the apparatus to: receive multiple point data corresponding to objects in the vehicle's surrounding environment from the vehicle's sensors; store the multiple point data in the memory; detect objects based on the multiple point data; perform multi-class segmentation on the multiple point data; identify points corresponding to the ground based on the multiple point data; perform a first fusion operation on the result of the multi-class segmentation and the result of identifying points corresponding to the ground; perform a second fusion operation on the object's cuboid bounding box and the object's outline; extract feature points and odometer readings of the surrounding environment based on the result of the first fusion operation; track the object based on the result of the second fusion operation; track the surrounding environment based on the feature points and odometer readings; and control the vehicle's autonomous driving operation based on the object tracking and the surrounding environment tracking.

[0011] At least one instruction can be configured, when executed by the processor, to cause the device to perform a second fusion operation by combining the fusion results of the object's cuboid frame and contour.

[0012] At least one instruction can be configured to, when executed by the processor, cause the device to perform combined processing on the fusion result of the cuboid bounding box and the outline of an object based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

[0013] At least one instruction can be configured to, when executed by the processor, cause the device to extract feature points and odometers by combining measurements of the surrounding environment with the odometers.

[0014] At least one instruction can be configured to, when executed by the processor, cause the device to combine measurements of the surrounding environment and the odometer based on at least one of the following: an occupancy model, a road model, a scene understanding model, and the odometer.

[0015] At least one instruction can be configured to, when executed by the processor, also cause the device to: correct sensor biases based on continuous odometer estimates during vehicle operation.

[0016] At least one instruction can be configured, when executed by the processor, to cause the device to perform a first fusion operation in such a way as to determine the final category based on the result of identifying points corresponding to the ground if the region of interest condition is met or the network confidence condition is not met; or to determine the final category based on the result of multi-class segmentation if the region of interest condition is not met but the network confidence condition is met.

[0017] At least one instruction can be configured, when executed by the processor, to cause the device to perform a second fusion operation in one of the following ways: registering the cuboid box to the list of valid boxes if the region of interest condition is met or the network confidence condition is not met; or registering the detection result of the object to the list of valid boxes if the region of interest condition is not met but the network confidence condition is met.

[0018] At least one instruction can be configured to, when executed by the processor, also cause the device to perform a second fusion operation by associating the object detection results with the cuboid bounding box based on the intersection-union ratio of the object detection results with the cuboid bounding box before determining the region of interest conditions.

[0019] At least one instruction can be configured, when executed by the processor, to cause the device to perform the second fusion operation by: marking the object and registering it in a list of valid bounding boxes based on the determination that the object has an irregular shape. At least one instruction can be configured, when executed by the processor, to cause the device to: further track the object based on the multi-class segmentation results.

[0020] According to one or more exemplary embodiments of this disclosure, a method performed by a vehicle's apparatus may include: receiving from the vehicle's sensors a plurality of point data corresponding to an object in the vehicle's surrounding environment; detecting the object based on the plurality of point data; performing multi-class segmentation on the plurality of point data; identifying points corresponding to the ground based on the plurality of point data; performing a first fusion operation on the result of the multi-class segmentation and the result of identifying points corresponding to the ground; performing a second fusion operation on the object's cuboid bounding box and the object's outline; extracting feature points and odometer readings of the surrounding environment based on the result of the first fusion operation; tracking the object based on the result of the second fusion operation; tracking the surrounding environment based on the feature points and odometer readings; and controlling the vehicle's autonomous driving operation based on the tracking of the object and the tracking of the surrounding environment.

[0021] The second fusion operation may include combining and processing the fusion results of the cuboid frame and the object outline.

[0022] The combined processing can be based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

[0023] Extracting feature points and odometers can include combining and processing measurement results of the surrounding environment with odometer readings.

[0024] Combined processing can be based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

[0025] The method may further include: correcting sensor biases based on continuous odometer estimations during the vehicle's operation.

[0026] The first fusion operation may include: determining the final category based on the results of identifying points corresponding to the ground if the region of interest condition is met or the network confidence condition is not met; or determining the final category based on the results of multi-class segmentation if the region of interest condition is not met but the network confidence condition is met.

[0027] The second fusion operation may include: registering the cuboid bounding box to the list of valid bounding boxes if the region of interest condition is met or the network confidence condition is not met; or registering the detection result of the object to the list of valid bounding boxes if the region of interest condition is not met but the network confidence condition is met.

[0028] The second fusion operation may include: before determining the region of interest conditions, associating the object detection results with the cuboid bounding box based on the intersection-union ratio of the object detection results and the cuboid bounding box.

[0029] The second fusion operation may include: based on the determination that an object has an irregular shape, marking the object and registering it in a list of valid bounding boxes. Object tracking may also be based on multi-class segmentation. Attached Figure Description

[0030] The above and other objects, features and advantages of this disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings:

[0031] Figure 1 This is a block diagram illustrating an example system for automatically identifying objects and controlling the autonomous driving operation of a vehicle;

[0032] Figure 2 This is a flowchart illustrating an example algorithm for object recognition architecture utilizing LiDAR segmentation;

[0033] Figure 3 This is a flowchart illustrating an example algorithm for the segmentation and fusion process of an object recognition architecture utilizing LiDAR segmentation;

[0034] Figure 4 This is a flowchart illustrating an example algorithm for object fusion using an object recognition architecture that utilizes LiDAR segmentation;

[0035] Figure 5This is a flowchart illustrating an example algorithm for refining (or processing) object measurements using an object recognition architecture that utilizes LiDAR segmentation;

[0036] Figure 6 This is a flowchart illustrating an example algorithm for refining (or processing) a surrounding environment measurement-related process using an object recognition architecture segmented by LiDAR; and

[0037] Figure 7 This is a block diagram illustrating an example computing system for controlling autonomous vehicles and recognizing operating objects. Detailed Implementation

[0038] One or more exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. When adding reference numerals to components in the drawings, it should be noted that identical or equivalent components are represented by the same numerals, even if the same components are shown in other drawings. Furthermore, in describing exemplary embodiments of this disclosure, detailed descriptions of related known structures or functions will be omitted if it is believed that such detailed descriptions would hinder understanding of this disclosure.

[0039] Furthermore, in describing the components of this disclosure, terms such as first, second, A, B, (a), (b), etc., may be used. These terms are used only to distinguish the various elements, and the nature, order, sequence, and number of elements are not limited by these terms. Moreover, unless otherwise defined, all terms used herein, including technical or scientific terms, have the meaning commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in common dictionaries should be interpreted as having a meaning consistent with the relevant technical context and should not be interpreted as having an idealized or overly formal meaning unless explicitly defined in the specification of this disclosure. For example, in this disclosure, "object" and "body" have almost the same meaning, and the terms "object" and "body" can be used interchangeably in this disclosure.

[0040] According to the Society of Automotive Engineers (SAE) standards, the automation levels of autonomous vehicles can be categorized as follows: Level 0, corresponding to "no automation," involves the autonomous system intervening only temporarily in emergency situations (e.g., automatic emergency braking) and / or providing warnings (e.g., blind spot warning, lane departure warning), requiring driver intervention. Level 1, corresponding to "driver assistance," involves the system performing some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control) while the driver operates the vehicle normally. The driver must assess the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. Level 2, corresponding to "partial automation," involves the system performing steering, acceleration, and / or braking under driver supervision. The driver must assess the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard can correspond to "conditional automation," meaning the system drives the vehicle under specific conditions (e.g., performing driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver when the required conditions are not met. The driver needs to assess the system's operating status and / or timing, and take over control in emergencies, but otherwise, the driver does not need to operate the vehicle (e.g., steering, acceleration, and / or braking). At Level 4 of autonomous driving, the SAE classification standard can correspond to "high automation," meaning the system performs all driving functions, and the driver only needs to take over vehicle control in emergencies. At Level 5 of autonomous driving, the SAE classification standard can correspond to "full automation," meaning the system performs all driving functions without any driver assistance in all situations, including emergencies. The driver only needs to assess the system's operating status and does not need to perform any other driving functions. Although this disclosure may adopt the SAE autonomous driving classification standard, other classification methods and / or algorithms may also be used in one or more configurations described herein. One or more functions related to autonomous driving control may be activated based on the configured autonomous driving control settings (e.g., based on at least one of the following: autonomous driving classification, selection of vehicle autonomous driving level, etc.).

[0041] Based on one or more of the features described herein (e.g., multi-class segmentation of sensor data), vehicle operation can be controlled. Vehicle control can include various operational controls related to the vehicle (e.g., autonomous driving control, sensor control, braking control, braking time control, acceleration control, rate of change of acceleration control, warning time control, forward collision warning time control, etc.).

[0042] For example, based on one or more features described herein (e.g., multi-class segmentation of sensor data), one or more auxiliary devices (e.g., engine braking, exhaust braking, hydraulic retarder, electric retarder, regenerative braking, etc.) can also be controlled. For example, based on one or more features described herein (e.g., multi-class segmentation of sensor data), one or more communication devices (e.g., modems, network adapters, radio transceivers, antennas, etc., capable of communicating via one or more wired or wireless communication protocols (e.g., Ethernet, Wi-Fi, Near Field Communication (NFC), Bluetooth, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle-to-Everything (V2X) etc.)) can also be controlled.

[0043] Minimum Risk Maneuver (MRM) operations can also be controlled based on one or more features described herein, such as multi-class segmentation of sensor data. A minimum risk maneuver (e.g., minimum risk maneuver, lowest risk maneuver) refers to a maneuver performed by a vehicle to minimize the risk of collision with surrounding vehicles, achieving a lower (e.g., lowest) risk state. A minimum risk maneuver is an operation that can be triggered during autonomous driving when the driver is unable to respond to an intervention request. During a minimum risk maneuver, one or more processors in the vehicle can control the vehicle's driving operations for a set period of time.

[0044] Bias driving operations can also be controlled based on one or more features described herein (e.g., multi-class segmentation of sensor data). The driving control unit can perform bias driving control. To perform bias driving, the driving control unit can control the vehicle to travel within the lane by maintaining a lateral distance between the vehicle's center position and the lane's center position. For example, the driving control unit can control the vehicle to stay within the lane, but not necessarily to stay in the lane center.

[0045] The driving control unit can identify a target lateral distance for offset driving control. For example, the target lateral distance may include the lateral distance that the vehicle intentionally adjusts during maneuvers such as lane changes to maintain a certain distance from a reference point (e.g., lane center or other vehicles). This adjustment is made to improve the vehicle's stability, safety, and / or performance under different driving conditions. For example, during a lane change, the driving control system may apply a lateral distance offset to maintain a safer distance from adjacent vehicles, taking into account factors such as vehicle speed, road conditions, and / or the presence of obstacles.

[0046] Autonomous driving levels and / or activation / deactivation of autonomous driving can also be controlled, for example, based on one or more features described herein (e.g., multi-class segmentation of sensor data). The driving control unit can perform autonomous driving level control (e.g., change the autonomous driving level, change the required driver attention, etc.) or cause autonomous driving operation to be deactivated. For example, by changing the required driver attention, the driver can be instructed to keep their hands on the steering wheel more frequently (e.g., at least once within a threshold time period, such as 5 seconds, 30 seconds, 1 minute, etc.). By changing the required driver attention, the driver can be instructed to look forward more frequently (e.g., at least once within a threshold time period, such as 5 seconds, 30 seconds, 1 minute, etc.). By changing the autonomous driving level, the display of one or more video contents on the vehicle's display screen can be prevented.

[0047] For example, based on one or more features described herein (e.g., multi-class segmentation of sensor data), one or more sensors (e.g., inertial measurement unit (IMU) sensors, cameras, LiDAR, radar, blind spot monitoring sensors, lane departure warning sensors, parking sensors, light sensors, rain sensors, traction control sensors, anti-lock braking system sensors, tire pressure monitoring sensors, seat belt sensors, airbag sensors, fuel sensors, emission sensors, throttle position sensors, inverters, converters, motor controllers, power distribution units, high-voltage lines and connectors, auxiliary power modules, charging interfaces, etc.) can also be controlled.

[0048] The operational control of autonomous driving can include various driving controls of the vehicle by the vehicle control unit (e.g., acceleration, deceleration, steering control, gear shifting control, braking system control, traction control, stability control, cruise control, lane keeping assist control, collision avoidance system control, emergency braking assist control, traffic sign recognition control, adaptive headlight control, driver warning control, autonomous driving operation design domain (ODD), enabling and / or disabling autonomous driving modes, etc.). For example, based on one or more features described herein (e.g., multi-class segmentation of sensor data), one or more auxiliary devices (e.g., engine braking, exhaust braking, hydraulic retarder, electric retarder, regenerative braking, etc.) can also be controlled. For example, based on one or more functions described herein (e.g., assessing driver engagement), one or more communication devices (e.g., modems, network adapters, radio transceivers, antennas, etc., capable of communicating via one or more wired or wireless communication protocols (e.g., Ethernet, Wi-Fi, Near Field Communication (NFC), Bluetooth, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle-to-Everything (V2X) etc.)) can also be controlled.

[0049] The vehicle actively controlled by the autonomous driving system may be referred to as the vehicle itself, the master vehicle, or the autonomous driving vehicle. The vehicle may also be referred to as a self-driving vehicle, an autonomous driving (AC) vehicle, a driverless vehicle, a robot taxi, a robotic vehicle, or an intelligent driving vehicle. The vehicle may be equipped with an autonomous driving system. Alternatively, the autonomous driving system may control the vehicle via external and / or remote devices (e.g., a server). The vehicle may be partially or fully remotely controlled by a remote driver (e.g., navigation, driving, etc.). A vehicle located in front of the vehicle (e.g., in the same lane as the vehicle) may be referred to as the vehicle in front (e.g., the vehicle directly in front), the vehicle in front (e.g., the vehicle directly in front), the leading vehicle, the preceding vehicle, or the vehicle ahead. A vehicle following the vehicle (e.g., in the same lane as the vehicle) may be referred to as the vehicle behind, the following vehicle, the following vehicle, or the successor vehicle. An adjacent vehicle may refer to any vehicle located in any direction from the vehicle (e.g., in front, behind, to the left, to the right, diagonally opposite, etc.), provided that there are no other vehicles (e.g., spaced vehicles) between it and the vehicle (e.g., regardless of their distance from the vehicle). Alternatively, in some cases, only vehicles located within a certain distance threshold of the vehicle (e.g., within the vehicle's line of sight and / or the detection limits of one or more of the vehicle's sensors) can be considered adjacent vehicles. A target vehicle can refer to any vehicle close to the vehicle (e.g., within the threshold distance). A target vehicle can be any vehicle actively or passively monitored, identified, tracked, and / or analyzed by the automated driving system. Monitoring, identification, tracking, and / or analysis can be single or multiple times, intermittent or continuous. The threshold distance can be, for example, the vehicle's line of sight and / or the detection limits of one or more sensors, or a value smaller than the vehicle's line of sight and / or the detection limits of one or more sensors (e.g., an adjustable value). A target vehicle can be, for example, a vehicle in front, a vehicle behind, a vehicle in a lane different from the vehicle's lane (e.g., a vehicle on the left, a vehicle on the right, a vehicle diagonally opposite, etc.), and / or an adjacent vehicle (e.g., regardless of its distance from the vehicle, and regardless of whether there are vehicles between the target vehicle and the vehicle). Target vehicles can also be referred to as surrounding vehicles, nearby vehicles, external vehicles, other vehicles, etc.

[0050] Figure 1 This is a block diagram illustrating an example system for automatically identifying objects and controlling the autonomous driving operation of a vehicle.

[0051] Reference Figure 1The vehicle control device 100 can be located inside or outside the vehicle, and some components of the vehicle control device 100 can also be located inside or outside the vehicle. In this case, the vehicle control device 100 can be integrally formed with the vehicle's internal control unit, or it can be a separate device connected to the vehicle's control unit via a separate connection method. For example, the vehicle control device 100 may also include Figure 1 Components not shown in the diagram.

[0052] The vehicle control unit 100 may include a processor 110, a lidar radar (LiDAR) 120, and a memory 130. The processor 110, LiDAR 120, or memory 130 may be electronically and / or operatively coupled to each other via electronic components including a communication bus.

[0053] In the following, hardware operable coupling can refer to the establishment of a direct or indirect connection between hardware via wired or wireless means, enabling the first hardware to control the second hardware.

[0054] Although shown in different block diagrams, this disclosure is not limited thereto. Figure 1 Some of the hardware in the device can be contained in a single integrated circuit, such as a system-on-a-chip (SoC). The type and / or quantity of hardware contained in the vehicle control unit 100 are not limited to... Figure 1 The hardware shown. For example, the vehicle control unit 100 may only include... Figure 1 The hardware shown is a portion of the hardware.

[0055] The vehicle control unit 100 may include hardware for processing data based on one or more instructions. For example, the hardware for processing data may include a processor 110. For example, the hardware for processing data may include an arithmetic logic unit (ALU), a floating-point unit (FPU), a field-programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), and / or an application processor (AP). The processor 110 may have a single-core processor architecture or a multi-core processor architecture, including dual-core, quad-core, hexa-core, or octa-core.

[0056] Processor 110 may include a graphics processing unit (GPU), a neural processing unit (NPU), or any combination thereof. For example, a GPU may be referred to as a visual processing unit (VPU). For example, an NPU may refer to a neural network processing unit.

[0057] The vehicle control unit 100 may include a depth sensor for detecting external objects. For example, the depth sensor for detecting external objects may include at least one of a time-of-flight (ToF) sensor, a LiDAR 120, a structured light sensor, an ultrasonic sensor, an infrared sensor, a radar (RADAR), an optical distance sensor, or any combination thereof. For ease of explanation, LiDAR will be described in detail below.

[0058] The vehicle control device 100 may include a LiDAR 120 that acquires multiple points based on pulsed laser signals. These multiple points may be, for example, multiple point data (data points) or a point cloud. For instance, the LiDAR 120 may acquire a dataset for identifying objects surrounding the vehicle control device 100 (or the vehicle containing the vehicle control device 100). For example, the LiDAR 120 may identify the position, direction of motion, velocity, or any combination thereof of surrounding objects based on the pulsed laser signals it emits reflected from them.

[0059] For example, LiDAR 120 can obtain a dataset representing external objects in space defined by the x-axis, y-axis, and z-axis based on pulsed laser signals reflected from surrounding objects. For example, LiDAR 120 can obtain a dataset containing multiple points in space defined by the x-axis, y-axis, and z-axis based on pulsed laser signals received at specified time intervals. For example, the multiple points can include points representing external objects in a three-dimensional virtual coordinate system. This three-dimensional (3D) virtual coordinate system can include at least one of a vehicle coordinate system, a LiDAR coordinate system, or any combination thereof. However, examples of three-dimensional virtual coordinate systems are not limited to those described above.

[0060] The memory 130 of the vehicle control unit 100 may include hardware components for storing data and / or instructions input to and / or output to the processor 110. For example, the memory 130 may include volatile memory (including random access memory (RAM)) and / or non-volatile memory (including read-only memory (ROM)).

[0061] For example, volatile memory may include at least one of the following: dynamic RAM (DRAM), static RAM (SRAM), cache RAM, pseudo SRAM (PSRAM), or any combination thereof. For example, non-volatile memory may include at least one of the following: programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, optical disk, solid-state drive (SSD), embedded multimedia card (eMMC), or any combination thereof.

[0062] The memory 130 of the vehicle control unit 100 may store one or more instructions (or commands) that instruct the processor 110 of the vehicle control unit 100 how to perform calculations and / or operations using data. A set of instructions may be referred to as a program, firmware, operating system, process, routine, subroutine, and / or application.

[0063] In the following text, the fact that “the application has been installed in the vehicle control unit 100” may refer to one or more instructions provided in the form of an application stored in memory 130, and one or more applications stored in a format executable by the processor 110 of the vehicle control unit 100 (e.g., a file with an extension specified by the operating system of the vehicle control unit 100).

[0064] For example, memory 130 may include a first neural network model for detecting objects. For example, memory 130 may include a second neural network model for outputting the type and / or scores of multiple points obtained by LiDAR 120.

[0065] The processor 110 can obtain at least one of the following based on multiple points obtained by LiDAR 120 and a first neural network model stored in memory 130: a first virtual box representing a target object, a first category representing the type of the target object, or a combination thereof.

[0066] For example, processor 110 can obtain at least one of the following based on inputting multiple points into a first neural network model: a first virtual bounding box representing a target object, a first category representing the type of the target object, or a combination thereof. For example, the first neural network model may include an object detection model. For example, the target object may include an external object located within a specified distance from vehicle control device 100 (or the host vehicle containing vehicle control device 100). For example, the target object may include an object identified and continuously tracked by vehicle control device 100. For example, the type of the target object may include multiple types for classifying the target object. For example, the type of the target object may include at least one of the following: a first type representing the ground, a second type representing a type different from the ground, or any combination thereof. However, the type of the target object is not limited to the above. For example, the type of the target object may include at least one of the following: a third type representing a person, a fourth type representing a vehicle, or any combination thereof, but is not limited thereto.

[0067] The processor 110 can obtain at least one of the following based on multiple points and a second neural network model: a first part of the multiple points corresponding to at least a portion of the target object; a second category identified by the first part of the points and indicating the type of the target object; or any combination thereof. For example, the second neural network model may include a segmentation model.

[0068] For example, the second neural network model may include a neural network model for obtaining the type of multiple points and the score of multiple points.

[0069] For example, processor 110 can obtain a first set of points corresponding to at least a portion of a target object from the multiple points by inputting multiple points into a second neural network model. For example, processor 110 can identify the type of the multiple points by inputting multiple points into the second neural network model. For example, processor 110 can obtain a first set of points corresponding to at least a portion of a target object from the multiple points based on the type of each point among the multiple points.

[0070] Processor 110 can perform a first specified algorithm on multiple points. For example, processor 110 can perform the first specified algorithm on multiple points to classify the type of each point. For example, processor 110 can classify a second portion of points among the multiple points that correspond to a specified type. For example, the specified type may include a type representing the ground.

[0071] For example, processor 110 may classify a second part of points corresponding to a specified type based on performing a first specified algorithm on multiple points, and obtain (or identify) a first part of points by excluding the second part of points from the multiple points.

[0072] The processor 110 can obtain at least one of the following based on inputting multiple points into a second neural network model: a partial category for obtaining a second category, a score for each of the multiple points, or a combination thereof. For example, the processor 110 can obtain a partial category and a score for each of the multiple points based on inputting multiple points into the second neural network model. For example, the partial category may include classifying each of the multiple points into an arbitrary type.

[0073] For example, processor 110 may fuse the partial category, the score of each point in the multiple points, and the second partial points. For example, processor 110 may perform clustering based on the fusion of the partial category, the score of each point in the multiple points, and the second partial points. For example, clustering may include grouping first partial points that correspond to at least a portion of the target object.

[0074] For example, processor 110 can obtain a point cloud for generating a second virtual bounding box based on the first subset of points. For example, processor 110 can obtain the point cloud based on grouping the first subset of points.

[0075] For example, processor 110 can generate a second virtual bounding box for representing a target object based on a point cloud, which is different from the first virtual bounding box. For example, the second virtual bounding box may include a box containing at least some of the first portion of points.

[0076] For example, processor 110 can identify a heading that indicates the direction of movement of a target object based on at least one of a first portion of points, a point cloud, or any combination thereof.

[0077] For example, processor 110 can identify the position of the second virtual bounding box in the virtual coordinate system based on at least one of the first partial points, point cloud, or any combination thereof. For example, processor 110 can identify the size of the second virtual bounding box based on at least one of the first partial points, point cloud, or any combination thereof. For example, processor 110 can identify a second category based on at least one of the first partial points, point cloud, or any combination thereof. For example, processor 110 can identify at least one of the following based on at least one of the first partial points, point cloud, or any combination thereof: a heading indicating the direction of movement of the target object, the position of the second virtual bounding box in the virtual coordinate system, the size of the second virtual bounding box, a second category, or any combination thereof. For example, processor 110 can identify the heading of the bounding box based on at least one of the first virtual bounding box, the first category, the heading of the second virtual bounding box, the position of the second virtual bounding box, the size of the second virtual bounding box, the second category, or any combination thereof. For example, processor 110 can identify the position of the bounding box in the virtual coordinate system based on at least one of the first virtual bounding box, the first category, the heading of the second virtual bounding box, the position of the second virtual bounding box, the size of the second virtual bounding box, the second category, or any combination thereof. For example, processor 110 can obtain a third category indicating the type of target object corresponding to the bounding box based on at least one of the first virtual frame, the first category, the heading of the second virtual frame, the position of the second virtual frame, the size of the second virtual frame, the second category, or any combination thereof. For example, processor 110 can obtain at least one of the following based on at least one of the first virtual frame, the first category, the heading of the second virtual frame, the position of the second virtual frame, the size of the second virtual frame, the second category, or any combination thereof: the heading direction of the bounding box, the position of the bounding box in the virtual coordinate system, a third category indicating the type of target object corresponding to the bounding box, or any combination thereof.

[0078] For example, processor 110 can assign a first identifier for tracking the second virtual box. For example, processor 110 can assign a second identifier corresponding to the first identifier to the bounding box.

[0079] For example, processor 110 can use the second identifier to track bounding boxes. For example, processor 110 can track a target object based on identifying multiple bounding boxes (including bounding boxes assigned the second identifier) ​​across multiple frames. For example, since the second identifier is an identifier assigned to the bounding box corresponding to the target object, processor 110 can track the target object by identifying multiple bounding boxes with the second identifier across multiple frames.

[0080] The processor 110 can output a bounding box corresponding to the target object based on at least one of a first virtual box, a first category, a first partial point, a second category, or any combination thereof. For example, the bounding box may include an example of the target object represented in hexahedral form in a virtual coordinate system.

[0081] The following is a brief description of the operations performed by the CPU, GPU, and / or NPU included in processor 110.

[0082] Processor 110 may include at least one of a CPU, GPU, NPU, or any combination thereof. For example, at least one of a GPU, NPU, or any combination thereof may obtain a first virtual bounding box and a first category based on a first neural network model. For example, at least one of a GPU and NPU may obtain a first virtual bounding box and a first category. For example, at least one of a GPU, NPU, or any combination thereof may obtain a partial category for obtaining a second category, and a score for each of a plurality of points, based on a second neural network model. For example, at least one of a GPU and NPU may obtain a partial category for obtaining a second category, and a score for each of a plurality of points, based on a second neural network model. For example, a CPU may classify a second subset of points corresponding to a specified type by performing a first specified algorithm on the plurality of points.

[0083] As described above, the vehicle control device 100 may include at least one processor 110. The vehicle control device 100 can detect target objects accurately through at least one processor 110. Furthermore, by performing parallel processing, the vehicle control device 100 can reduce the load on each processor.

[0084] In description Figure 2 Previously, to facilitate understanding of the AI ​​object recognition disclosed herein, the following description related to LiDAR-based object recognition was added.

[0085] The object recognition process of LiDAR 120 and the AI ​​module includes three operations: preprocessing, segmentation, and tracking.

[0086] The object recognition system 1000 of this disclosure performs a preprocessing procedure before performing the object recognition function. For example, preprocessing may include removing points constituting the ground from laser sensing data (e.g., unprocessed raw data) input from the LiDAR 120. Since laser beams reflected from the ground may be incorrectly identified as objects on the ground, ground and non-ground distinctions are made during the preprocessing operation, and this can also be done during the segmentation operation if necessary.

[0087] In short, preprocessing in artificial intelligence object recognition can be understood as the process by which the image processing tool of the artificial intelligence module in processor 110 removes noise from the LiDAR point cloud image and reduces the total number of points in the LiDAR point cloud image through techniques such as voxel downsampling, thereby improving computational efficiency.

[0088] For reference, LiDAR point cloud images can also be displayed as a bird's-eye view (BEV). When a LiDAR map is created from the perspective of a bird flying overhead over a city, this type of map is called a bird's-eye view (BEV).

[0089] In other words, as described above, LiDAR 120 emits laser beams into the surrounding environment and records the time it takes for the laser beams to be reflected by external objects and return, thereby generating a point (i.e., a single point) for each of the numerous laser signals and calculating the distance to that point. By repeatedly emitting a large number of laser beams in this manner, processor 110 can generate a real-time LiDAR map of the surrounding environment, which is a three-dimensional map of the bird's-eye view type and can also serve as a two-dimensional map when necessary.

[0090] The black and white lines or surfaces displayed on a LiDAR point cloud map are actually composed of countless points (each generated by a laser beam from the LiDAR120). Therefore, LiDAR sensing images are also called LiDAR point cloud images. For example, by combining a red-green-blue depth (RGB-D) sensor with a LiDAR sensor, a color LiDAR point cloud image can be reconstructed.

[0091] While humans may struggle to identify an object from a single point in a LiDAR point cloud image, by comprehensively considering point cloud information—such as from a BEV perspective or a two-dimensional planar view—it's possible to estimate the surrounding environment of an autonomous vehicle. Furthermore, it's possible to identify cars, buses, pedestrians, roadside trees, traffic signs, and other objects within LiDAR point cloud images. In artificial intelligence image recognition technology, things or people can be referred to as objects, and each object can be categorized into a class with specific characteristics, such as cars, buses, and so on.

[0092] To classify objects in LiDAR point cloud images into car and bus categories, a deep artificial intelligence neural network is needed. To use this neural network to find objects and identify their categories in LiDAR point cloud images, artificial intelligence training can be performed first.

[0093] For example, the dataset called PANDASET™ (source: https: / / pandaset.org / #data-collection) contains more than 48,000 camera images (mainly taken in Silicon Valley, USA) and more than 16,000 LiDAR scan images, which are labeled in a total of 28 categories, including pedestrians, vehicles, bicycles, construction site signs, traffic signs, etc.

[0094] In addition, point cloud processing tools such as Open3D™ (source: https: / / www.open3d.org / ) can be used to visualize LiDAR point cloud images according to user needs. Because the LiDAR 120 has distance detection capabilities, it can reproduce 3D LiDAR images more realistically. For example, when using Open3D™ for visual processing, distant objects can be displayed in dark blue, and nearby objects in light blue.

[0095] Furthermore, as mentioned above, the raw LiDAR image (e.g., raw data) can be preprocessed, for example, by applying voxel (e.g., 3D pixel) downsampling techniques to the LiDAR point cloud image. In this case, a voxel refers to a cubic 3D pixel. Voxel downsampling is a technique that reduces the number of points, thus reducing the number of points, avoiding excessive AI computation, and preserving the structure of the various objects contained in the LiDAR point cloud.

[0096] For example, the LiDAR 120 fires m laser beams n times in one scan cycle. In this case, the scan values ​​returned after each laser beam collides with an external object form an (m×n) matrix, and this (m×n) matrix data is called the range image. Each point that makes up the LiDAR point cloud image contains depth (i.e., distance) information, as well as the intensity, azimuth, tilt, and other additional information of the returned laser pulses. For the range image, large datasets such as the Waymo™ Open Dataset (WOD) can be used for AI learning.

[0097] Distance View (RV) is a technique that converts 3D point clouds into 2D or 2.5D scenes, representing them as a LiDAR 3D map that humans can intuitively understand, similar to a simulated image rather than a large number of points. In a distance view image, the 3D LiDAR point cloud image has two-dimensional coordinates, but the three-dimensional laser-related information (angle, tilt, intensity, etc.) recorded when obtaining the depth image is not lost. By applying a variable called "width" to the (x, y, z) coordinate values ​​of the 3D LiDAR image, a 2D axis coordinate is obtained; then, by applying a variable called "height" to the (z) coordinate and distance image information representing distance (depth), the other 2D axis coordinate is obtained, thus generating a 2D distance view image.

[0098] Furthermore, the AI ​​algorithms disclosed herein may include convolutional neural networks (CNNs), a common AI training module used to extract features (or feature points) from image data. Commercial datasets containing tens of thousands of images are currently available, and CNN versions capable of processing images from one-dimensional to three-dimensional are also available. In other words, the results of distance-view image processing tools are used to train the CNN neural network, thereby helping AI accurately identify objects in images.

[0099] Since objects around autonomous vehicles are ultimately identified by machines, generating ground truth (GT) bounding boxes on the aforementioned LiDAR map is a crucial step in the object recognition process. In machine learning, ground truth (GT) refers to the original or actual values ​​of the data that the AI ​​is learning from. It can be understood as an image annotation applied to LiDAR point cloud images, typically a bounding box with box-shaped boundaries.

[0100] In other words, to identify objects, the AI ​​module adds labels and groups various objects. The 3D data point spacing used to output the ground truth (GT) bounding box can be set to contain approximately 50 to 1000 LiDAR point clouds within a GT bounding box.

[0101] During vehicle movement, the raw data collected by sensors such as LiDAR 120 does not contain such ground truth (GT) annotations. The processor 110 identifies various objects, including road signs, crosswalks, pedestrians, other vehicles, and the center lane. GT annotation is a method of measuring object recognition error by comparing the object recognition results identified by the processor 110's AI algorithm with the actual objects; it is sometimes also used to evaluate AI performance. The GT bounding boxes applied to the raw image in the form of annotations can be manually set by the user, but there are also commercially available GT calculation tools, such as grid-striding.

[0102] On the other hand, when driving the AI ​​object recognition module, the predicted bounding boxes can also be viewed. Specific categories of objects identified by the processor 110 from raw image data obtained from the LiDAR sensor 120, etc., are displayed as another bounding box, similar to the ground truth (GT) bounding box. Unlike the GT bounding box, the predicted bounding box is the result of calculations by the autonomous driving AI. The predicted bounding box may coincide with the GT bounding box, or it may not, or it may not coincide at all.

[0103] For reference, since it is difficult to determine whether a specific type of object exists at a certain location based solely on the predicted bounding box, the predicted bounding box is often called a probabilistic box (P-Box).

[0104] Preprocessing segmentation refers to, for example, displaying specific parts of a road (e.g., traffic lights) in red and other parts (e.g., asphalt pavement) in blue. The segmentation operation can also include clustering the point cloud into specified groups and generating P-Boxes. For reference, there is also a technique called cluster expansion, which aims to cluster all points within an ε-distance (the minimum distance to form a cluster) from the seed point.

[0105] During the segmentation process, clustering and P-box generation are performed based on the point cloud. In short, point labels are obtained from the LiDAR sensor 120 using an AI network that performs segmentation.

[0106] To address the ground identification error problem that occurs during segmentation, this disclosure proposes a rule-based road surface identification and label fusion technique. This road surface identification algorithm can apply various techniques, including slope-based road surface identification, grid-based road surface identification, and other non-planar road surface identification. However, this disclosure recommends using slope-based road surface identification technology when identifying road surfaces within tunnels.

[0107] For reference, semantic segmentation refers to the task of attaching (e.g., applying) unique category labels to each point in a point cloud generated by LiDAR 120. In LiDAR image processing technology, semantic segmentation is a technique for finding and utilizing meaningful information from LiDAR data for object recognition and scene reconstruction, which are essential for autonomous driving. Various semantic segmentation AI models already exist, such as projection-based methods, point-based methods, sparse convolution-based methods, and so on. For example, the semantic segmentation result can be the result of AI operations performed by the processor 110 with the NVIDIA DRIVE™ AGX system; with this configuration, for example, various colors can be added to the LiDAR point cloud image.

[0108] After the segmentation process described above, the LiDAR images will be processed through a process called post-processing. Post-processing refers to converting the point cloud data into a 3D map or model, which is meaningful for autonomous driving. Post-processing also includes the final removal of noise and errors from the LiDAR point cloud images, the identification of objects such as vehicles or pedestrians from the point cloud, and, if necessary, the attachment (e.g., application) of unique identifiers to the point cloud information for registration.

[0109] Figure 2 This is a flowchart illustrating an example algorithm 200 that utilizes LiDAR segmentation for object recognition architecture.

[0110] Reference Figure 2 First, in operation S100, preprocessing as described above is performed. That is, in operation S100, the main inputs from the autonomous driving sensor 120 are preprocessed, and the raw data from the point cloud or vehicle network is processed into the form required by each software or hardware module according to the architecture of this disclosure.

[0111] In operation S110, object detection is performed. Operation S110 is performed by a recognition network, which performs the detection task of recognizing the location, shape, and category of stereotyped shapes such as vehicles, pedestrians, and pillars.

[0112] In operation S120, multi-class segmentation is performed. Operation S120 is performed by a segmentation task network, which identifies the class of objects and their surrounding environment in units of points or grids, given that they exist at a unique ratio (whether normalized or not).

[0113] In operation S130, ground identification is performed, and in operation S130, a rule-based algorithm or supplementary network is executed to identify road surface-related categories on a point or grid basis.

[0114] In operation S140, segmentation fusion is performed, and a rule-based algorithm or ensemble network is executed to fuse road surface-related category predictions at the point or grid level, thereby deriving the final result.

[0115] In operation S150, instantiation or noise filtering is performed. Operation S150 is the process of filtering out noise while performing instantiation, analyzing information on an instance-by-instance basis based on the category prediction results of point or grid units.

[0116] In this context, instantiation can include window search and clustering processes. Furthermore, noise filtering can be performed before or after instantiation, or concurrently. Points or grids identified as noise may not be analyzed in subsequent modules, but in some cases, they can be used as additional features for extracting information on a frame-by-frame basis. For example, noisy points or grids can be used to predict weather information for a specific frame.

[0117] exist Figure 2 In operation S160, a cuboid object can be generated. That is, in operation S160, a rule-based algorithm or supplementary network can be executed to generate a cuboid based on the points contained in the instance.

[0118] In operation S170, the outline of the object is extracted. In operation S170, a rule-based algorithm or supplementary network can be executed to represent the external shape of the instance by generating a shape containing multiple edges or vertices.

[0119] Operation S180 can be an object fusion process. This can be a module that fuses detection, cuboid bounding boxes, and contour data, which contain information about the object. For regular objects, the output of the object detection network is mainly used, while for irregular objects, it is difficult to detect them using the object detection network. Therefore, the remaining boxes in the object's cuboid bounding box output that are unrelated to the object detection network output are mainly used. Not all objects are 3D cuboid shapes. Even for objects with 3D cuboid shapes, such as vehicles, when considering shape anomalies such as rearview mirrors and state changes such as open doors, it may be difficult to accurately estimate the shape using only 3D cuboid bounding boxes. Therefore, in operation S180, representing the object's outer surface through the object contour yields the most accurate shape inference results.

[0120] In operation S190, an occupancy model can be executed, along with a rule-based algorithm or supplementary network for predicting occupancy areas in the surrounding environment. The occupancy area can be output simply as a grid, or the output of a segmented network can be used to predict category information and dynamic flow per grid. Operation S190 may include a free space module, which represents only the drivable area in a more simplified form.

[0121] In operation S200, a road model can be executed, and rule-based algorithms or supplementary networks can be used to predict lane, road boundary, and pavement geometry.

[0122] In operation S210, a scene understanding model can be executed. Rule-based algorithms or supplementary networks are used to understand the current frame in the desired form based on category predictions at the point or grid level. For example, the weather in the current frame can be predicted, anomalies in the frame can be detected to predict contamination of sensor information, and various scene understanding results can be derived, such as predictions of the current condition and whether drivability is possible.

[0123] In operation S220, feature point extraction can be performed. This is point-by-point feature extraction that meets desired conditions, allowing the use of category information to improve performance or to extract previously unextracted feature points. As an example, we can illustrate how to extract edge points of a static object, in which case category information may be required.

[0124] In operation S230, odometry data can be obtained. That is, performance can be greatly improved if only static objects are used when calculating movement, and further improved because category information can be utilized in this case. The output of operation S230, in addition to being used to determine or control autonomous driving, can also be used as the main input to mapping algorithms used for map construction.

[0125] The online calibration module operating the S240 can be a module that corrects the deviations of the autonomous driving sensors during driving based on continuous odometer estimation results.

[0126] Operation S250 can be an object measurement refinement / processing operation. This is a module used to combine the measurement values ​​required for object tracking. In object tracking, the results of the object fusion module in operation S180 are mainly utilized. However, the object fusion results can be precisely measured and processed by using the occupancy model (OM), road model (RM), scene understanding model, and odometry as inputs. Therefore, these processes are performed in operation S250.

[0127] Operation S260 is a measurement refinement / processing operation related to the surrounding environment. This is a module used to combine and process the measurement results required for tracking. When tracking the surrounding environment, the calculation results of the occupancy model (OM), road model (RM), and scene understanding model are mainly used, but this can be processed into more complex measurement values ​​in Operation S260 by using object fusion and odometry as inputs.

[0128] Operation S270 performs tracking and post-processing of object information. By utilizing continuous frame information, the final output is processed into the desired form, while stabilization compensation is possible, providing results for various autonomous driving determination / control and mapping processes.

[0129] In operation S280, the tracking and post-processing of surrounding environment information are performed. Similar to operation S270, by utilizing continuous frame information, the final output is processed into the desired form, while stability compensation is possible, and results are provided for various autonomous driving determination / control and mapping processes.

[0130] Figure 3 This is a flowchart illustrating an example algorithm for the segmentation and fusion process of an object recognition architecture that utilizes LiDAR segmentation. Figure 3 Detailed illustration Figure 2 Operation S140 in the middle.

[0131] In operation S141, the segmentation results (operation S120) and ground recognition results are updated (operation S130). Then, in operation S142, it is checked whether the region of interest (ROI) condition is met. For example, the benchmark could be whether the currently analyzed object is within a short distance of 5 meters from the autonomous driving sensor 120.

[0132] If the ROI condition is not met as determined by operation S141, the process proceeds to operation S143: determining whether the network confidence score meets a specified threshold (e.g., whether the confidence score is above 0.9). The network confidence score can be achieved through the network output score, or through a combination of the score and an empirically determined value.

[0133] When the result of operation S143 is "yes", the process proceeds to operation S144: making a final decision on the category based on the segmentation result. When the result of operation S142 is "yes" or the result of operation S143 is "no", the process proceeds to operation S145: making a final decision on the category based on the ground recognition result. In operation S146, the above operations are repeated until the category cursor index curIdx value becomes equal to the MaxIdx value.

[0134] Figure 4 This is a flowchart illustrating an example algorithm for object fusion processes using an object recognition architecture that leverages LiDAR segmentation. In other words, Figure 4 Detailed illustration Figure 2 Operation S180 in the middle.

[0135] In operation S181, an association operation is performed between the object detection result and the object cuboid bounding box based on the intersection-over-union ratio (IoU).

[0136] In operation S182, the curIdx bounding box of the object detection and the object cuboid bounding box associated with the corresponding bounding box are updated.

[0137] In operation S183, check whether the ROI conditions are met. For example, whether there is an object within the field of view (FOV) boundary can be a criterion for judgment.

[0138] For example, when operation S183 determines that the object is not within the FOV boundary, the process proceeds to operation S184: determining whether the network confidence meets a preset threshold (e.g., exceeding 0.6).

[0139] If the network confidence score is met in operation S184, the process proceeds to operation S185: register the list of valid bounding boxes based on the object detection results. If the network confidence score is not met in operation S184, or if the ROI condition is met in operation S183, the process proceeds to operation S186: register the list of valid bounding boxes based on the object cuboid bounding box results.

[0140] Then, repeat the above operation by operating S187 until the category cursor index curIdx value becomes equal to the MaxIdx value.

[0141] Then, in operation S188, the “SEG ONLY” flag is assigned to the unrelated object cuboid bounding boxes and registered in the list of valid bounding boxes; in operation S189, the possible object contour data in the list of valid bounding boxes is updated based on the IoU and the object cuboid bounding box ID. For reference, the “SEG ONLY” flag is used to track irregular objects that are difficult for the detection network to detect based on segmentation.

[0142] Figure 5 This is a flowchart illustrating an example algorithm for refining (or processing) object measurements using an object recognition architecture that utilizes LiDAR segmentation. In other words, Figure 5 It shows Figure 2 Operation S250 in the middle.

[0143] Reference Figure 5 In operation S251, the measurement values ​​of the object fusion result box instances are updated, and in operation S252, feature points (f1, f2, f3, etc.) are extracted based on the associated input. This can be a function of the containment box, occupancy model (OM), road model (RM), scene understanding model (SUM), and odometry (ODO).

[0144] In operation S253, the measured values ​​are refined / processed based on feature points. For example, when a threshold for a specific feature point is not met, the bounding box is deleted or the shape is corrected based on the specific feature point value. In operation S254, the above operations are repeated until the category cursor curIdx value becomes equal to the MaxIdx value.

[0145] Figure 6 This is a flowchart illustrating an example algorithm for refining (or processing) the surrounding environment measurement related to an object recognition architecture utilizing LiDAR segmentation. In other words, Figure 6 Detailed illustration Figure 2 Operation S260 in the middle.

[0146] Reference Figure 6 In operation S261, the measurements of the surrounding environment data are updated based on the results of the occupancy model (OM), road model (RM), and scene understanding model (SUM). In operation S262, feature points (f1, f2, f3, ...) are extracted based on the associated input. This can be a function that includes measurements of the surrounding environment (ENV), bounding boxes (BOX), and odometry (ODO).

[0147] In operation S263, the measured values ​​are refined / processed based on feature points. For example, when a threshold for a specific feature point is not met, the ENV is deleted or the ENV confidence is corrected based on the specific feature point value. In operation S264, the above operations are repeated until the category cursor curIdx value becomes the MaxIdx value.

[0148] This disclosure presents a system and method that can utilize LiDAR segmentation to improve recognition, surrounding environment recognition, and the calculation of the movement of the vehicle (equipped with autonomous driving sensors).

[0149] In other words, according to this disclosure, it is configured to: generate effective recognition output for object / environment / vehicle movement based on a logical architecture based on point category information by identifying the main unit logic modules that can utilize point unit category information (which is the result of the segmentation task) and the relationships between the modules. Furthermore, by utilizing the point unit category information obtained from the segmentation task, it is expected to improve recognition performance, make it robust to various types of noise, and handle oversegmentation / undersegmentation phenomena.

[0150] Furthermore, according to this disclosure, since classification is applied from the clustering operation (which is the first instantiation process in LiDAR recognition), shape analysis algorithms such as adaptive bounding box search or contour algorithms can be applied to each category in the shape analysis operation (which is the logic for subsequent operations), thereby further improving shape analysis performance. In addition to other shape analysis modules, point category information from various logic modules can also be used to configure the cell architecture.

[0151] Figure 7 This is a block diagram illustrating an example computing system 1000 for controlling autonomous vehicles and identifying operating objects.

[0152] Reference Figure 7 The computing system 1000 may include at least one processor 1100, memory 1300, user interface input device 1400, user interface output device 1500, storage device 1600, and network interface 1700, which are connected via bus 1200.

[0153] Processor 1100 may be a central processing unit (CPU) or a semiconductor device for processing instructions stored in memory 1300 and / or storage device 1600. Memory 1300 and storage device 1600 may include various volatile or non-volatile storage media. For example, memory 1300 may include read-only memory (ROM) and random access memory (RAM).

[0154] Therefore, the flow of the methods or algorithms described in this disclosure can be directly implemented by hardware, software modules, or a combination of both executed by processor 1100. The software modules can reside in storage media (i.e., memory 1300 and / or storage device 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, solid-state drive (SSD), removable disk, or CD-ROM.

[0155] Example storage media can be coupled to processor 1100, which can read information from and write information to the storage media. Alternatively, the storage media can be integrated with processor 1100. The processor and storage media can reside in an application-specific integrated circuit (ASIC). The ASIC can reside in the user terminal. In another approach, the processor and storage media can reside as independent components in the user terminal.

[0156] According to one aspect of this disclosure, a system for identifying objects using autonomous driving sensor segmentation includes: a processor having an object recognition module that identifies objects present in a surrounding environment based on multiple point data input from autonomous driving sensors; and a memory storing the input multiple point data, wherein the object recognition module performs the following steps: detecting objects based on the multiple point data; performing multi-class segmentation; identifying points corresponding to the ground; performing a first fusion of the results of the multi-class segmentation and the ground recognition; performing a second fusion of the object's cuboid bounding box and the object's outline; extracting feature points and odometry of the surrounding environment based on the result of the first fusion; performing object tracking based on the result of the second fusion; and performing tracking of the surrounding environment based on the feature points and odometry.

[0157] The second fusion of the object's cuboid frame and object outline may include the following process: combining and processing the fused cuboid frame and object outline results.

[0158] Extracting feature points and odometers can include the following process: combining and processing the measurement results of the surrounding environment and the odometer.

[0159] The combined processing can be performed based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

[0160] The object recognition module can also correct the deviations of autonomous driving sensors during driving based on continuous estimation results from the odometer.

[0161] The first fusion of the identification results may include the following process: when the ROI condition is met or the network confidence condition is not met, the final category is determined based on the ground identification results; otherwise, the final category is determined based on the multi-class segmentation results.

[0162] The second fusion of the object's cuboid bounding box and the object's outline may include the following process: when the ROI condition is met or the network confidence condition is not met, the cuboid bounding box is registered in the list of valid bounding boxes; otherwise, the object detection result is registered in the list of valid bounding boxes.

[0163] The second fusion of the object's cuboid bounding box and the object's outline may include the following process: before determining the ROI conditions, associating the object detection results with the cuboid bounding box based on the intersection-over-union ratio (IoU).

[0164] The second fusion of the object's cuboid bounding box and the object's outline may include the following process: after registering irregular objects that are difficult to detect as values ​​in the list of valid bounding boxes, attaching a specific flag to the irregular object; and tracking the object based on the segmentation that is the result of the first fusion.

[0165] According to another aspect of this disclosure, a method for identifying objects and the surrounding environment using multiple point data input from autonomous driving sensors includes: detecting objects by an object recognition module based on the multiple point data, performing multi-class segmentation, and identifying points corresponding to the ground; performing a first fusion of the results of the multi-class segmentation and the ground recognition by the object recognition module; performing a second fusion of the object's cuboid bounding box and the object's outline by the object recognition module; extracting feature points and odometry of the surrounding environment by the object recognition module based on the result of the first fusion; and performing object tracking by the object recognition module based on the result of the second fusion, and performing tracking of the surrounding environment based on the feature points and odometry.

[0166] According to this disclosure, a system and method are provided that can utilize LiDAR segmentation to improve techniques for object recognition, surrounding environment recognition, and vehicle (vehicle equipped with autonomous driving sensors) movement calculation.

[0167] In other words, according to this disclosure, the configuration is as follows: by identifying the main unit logic modules that can utilize point unit category information (which is the result of the segmentation task) and the relationships between the modules, an effective output for recognizing the movement of the object / environment / vehicle is generated based on a logical architecture based on point category information. Furthermore, by utilizing the point unit category information obtained from the segmentation task, it is expected to improve recognition performance, making it robust to various types of noise and able to handle oversegmentation / undersegmentation phenomena.

[0168] Furthermore, according to this disclosure, since classification is applied from the clustering operation (which is the first instantiation process in LiDAR recognition), shape analysis algorithms such as adaptive bounding box search or contour algorithms can be applied to each category in the shape analysis operation (which is the logic for subsequent operations), thereby further improving shape analysis performance. In addition to other shape analysis modules, point category information from various logic modules can also be used to configure the cell architecture.

[0169] Furthermore, through the detailed description and accompanying drawings of this disclosure, those skilled in the art will be able to understand various other effects of this disclosure besides those described above.

[0170] While one or more exemplary embodiments of this disclosure have been described for ease of illustration, those skilled in the art will understand that various modifications, additions, and substitutions may be made without departing from the scope and spirit of this disclosure.

[0171] Therefore, the exemplary embodiments disclosed herein are for illustrative purposes only and do not limit the technical concepts of this disclosure. It should be understood that these exemplary embodiments are not intended to limit the scope of the technical concepts of this disclosure. The scope of protection of this disclosure should be understood through the following claims, and all technical concepts within the equivalent scope should be interpreted as being within the scope of the rights of this disclosure.

Claims

1. A vehicle apparatus, the apparatus comprising: processor; and A memory storing at least one instruction configured to, when executed by the processor, cause the device to: Receive data from the vehicle's sensors, which corresponds to multiple points in the vehicle's surrounding environment. The multiple point data are stored in the memory; Based on the multiple point data, the object is detected; Perform multi-class segmentation on the multiple point data; Based on the multiple point data, identify the points corresponding to the ground; Perform the first fusion operation on the results of multi-class segmentation and the results of identifying points corresponding to the ground; Perform a second blending operation on the cuboid frame and the outline of the object; Based on the results of the first fusion operation, feature points and odometry of the surrounding environment are extracted; Based on the result of the second fusion operation, the object is tracked; Based on the feature points and the odometer, the surrounding environment is tracked; as well as The vehicle's autonomous driving operation is controlled based on the tracking of the object and the surrounding environment.

2. The apparatus according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform the second fusion operation in the following manner: The fusion result of the cuboid frame and contour of the object is combined and processed.

3. The apparatus according to claim 2, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform combined processing on the fusion result of the cuboid frame and contour of the object based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

4. The apparatus according to claim 1, wherein, The at least one instruction is configured, when executed by the processor, to cause the device to extract the feature points and the odometer in the following manner: The measurement results of the surrounding environment are combined with the odometer for processing.

5. The apparatus according to claim 4, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform combined processing of measurements of the surrounding environment and the odometer based on at least one of the following: an occupancy model, a road model, a scene understanding model, and the odometer.

6. The apparatus according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, also cause the device to: During vehicle operation, sensor biases are corrected based on continuous estimations of the odometer readings.

7. The apparatus according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform the first fusion operation in the following manner: If the region of interest condition is met or the network confidence condition is not met, the final category is determined based on the results of identifying points corresponding to the ground. or In cases where the region of interest condition is not met but the network confidence condition is met, the final category is determined based on the results of multi-class segmentation.

8. The apparatus according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform the second fusion operation in one of the following ways: If the region of interest condition is met or the network confidence condition is not met, the cuboid box is registered in the list of valid boxes; or If the region of interest condition is not met but the network confidence condition is met, the detection result of the object is registered in the list of valid bounding boxes.

9. The apparatus according to claim 8, wherein, The at least one instruction is configured to, when executed by the processor, also cause the device to perform a second fusion operation in the following manner: Before determining the region of interest conditions, the detection results of the object are associated with the cuboid bounding box based on the intersection-union ratio of the detection results of the object and the cuboid bounding box.

10. The apparatus according to claim 8, wherein, The at least one instruction is configured to, when executed by the processor, cause the device to perform the second fusion operation in the following manner: Based on the determination that the object has an irregular shape, the object is marked and registered in the list of valid boxes, and The at least one instruction is configured to, when executed by the processor, cause the device to: further track the object based on the multi-class segmentation.

11. A method performed by a device of a vehicle, the method comprising the steps of: Receive data from the vehicle's sensors, which corresponds to multiple points in the vehicle's surrounding environment. Based on the multiple point data, the object is detected; Perform multi-class segmentation on the multiple point data; Based on the multiple point data, identify the points corresponding to the ground; Perform the first fusion operation on the results of multi-class segmentation and the results of identifying points corresponding to the ground; Perform a second blending operation on the cuboid frame and the outline of the object; Based on the results of the first fusion operation, feature points and odometry of the surrounding environment are extracted; Based on the result of the second fusion operation, the object is tracked; Based on the feature points and the odometer, the surrounding environment is tracked; as well as The vehicle's autonomous driving operation is controlled based on the tracking of the object and the surrounding environment.

12. The method according to claim 11, wherein, The second fusion operation includes the following steps: The fusion result of the cuboid frame and contour of the object is combined and processed.

13. The method according to claim 12, wherein, The combined processing is based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

14. The method according to claim 11, wherein, The steps for extracting the feature points and the odometer include: The measurement results of the surrounding environment are combined with the odometer for processing.

15. The method according to claim 14, wherein, The combined processing is based on at least one of the following: occupancy model, road model, scene understanding model, and odometer.

16. The method of claim 11, further comprising the step of: During vehicle operation, the sensor's bias is corrected based on continuous estimations of the odometer readings.

17. The method according to claim 11, wherein, The first fusion operation includes the following steps: If the region of interest condition is met or the network confidence condition is not met, the final category is determined based on the results of identifying points corresponding to the ground; or In cases where the region of interest condition is not met but the network confidence condition is met, the final category is determined based on the results of multi-class segmentation.

18. The method according to claim 11, wherein, The second fusion operation includes the following steps: If the region of interest condition is met or the network confidence condition is not met, the cuboid box is registered in the list of valid boxes; or If the region of interest condition is not met but the network confidence condition is met, the detection result of the object is registered in the list of valid bounding boxes.

19. The method according to claim 18, wherein, The second fusion operation includes the following steps: Before determining the region of interest conditions, the detection results of the object are associated with the cuboid bounding box based on the intersection-union ratio of the detection results of the object and the cuboid bounding box.

20. The method according to claim 18, wherein, The second fusion operation includes the following steps: Based on the determination that the object has an irregular shape, the object is marked and registered in the list of valid boxes, and The tracking of the object is also based on multi-class segmentation.