Apparatus and method for controlling autonomous driving of a vehicle

Abstract

An apparatus and method of controlling autonomous driving of a vehicle are disclosed. An apparatus of a vehicle can be configured to receive, from a sensor of the vehicle, sensor data associated with an object existing in a surrounding environment of the vehicle; receive, from the vehicle, a set of parameters for calibrating a coordinate system for object recognition and information about a resource state of a processor or a memory; determine a motion score and a confidence score based on driving environment information and motion information of the vehicle; adjust the confidence score based on a result of segmenting the sensor data and a result of recognizing the object; perform online calibration of the coordinate system for object recognition based on the adjusted confidence score; and control autonomous driving operation of the vehicle based on the calibrated coordinate system for object recognition.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit and priority of Korean Patent Application No. 10-2025-0042965, filed with the Korean Intellectual Property Office on April 2, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to autonomous vehicles, and more specifically, to systems and methods for identifying objects based on artificial intelligence. Background Technology

[0004] With the development and commercialization of autonomous vehicles, the number and types of sensors used in vehicles, as well as the application of artificial intelligence (AI) technology, are increasing to achieve autonomous driving capabilities. For example, extensive research is underway on recognition technologies used to identify objects present in front of a moving vehicle, determine the distance of these objects from the vehicle, and determine which algorithms the vehicle should use to respond to specific situations to ensure the safety of its occupants.

[0005] Therefore, vehicle sensor technology has become more advanced. There is a trend toward implementing high-performance sensors in vehicles, such as light detection and ranging (LiDAR) sensors that use laser beams to identify the surrounding environment, radar sensors that use radio waves, ultrasonic sensors, fisheye cameras that can capture 360-degree images, multifocal lenses, and Global Positioning System (GPS).

[0006] Measurements from multiple sensors and sensor types can be aggregated to create so-called super-sensor vehicles. The concept of super-sensors in self-driving or autonomous vehicles refers to technologies that combine measurements from various sensors and sensor types to more accurately identify the vehicle's surroundings, improving driver convenience and safety, rather than relying on and analyzing individual sensor measurements separately. With the advent of information and communication technologies (ICT) and cloud technologies, sensors used for autonomous driving and their associated AI algorithms have become more advanced than ever before. For example, instead of utilizing data collected from a single vehicle, data can be collected remotely from a fleet of vehicles, and the accumulated data can be used to train AI servers and databases to increase the reliability and accuracy of vehicle sensors.

[0007] The descriptions in this background section are intended only to enhance the understanding of the background of this disclosure and should not be construed as an admission that they correspond to prior art known to those skilled in the art. Summary of the Invention

[0008] This disclosure has been made to address the problems that arise in at least some of the implementations, while fully preserving the advantages achieved by those implementations.

[0009] One aspect of this disclosure provides a system and method for identifying objects and analyzing the driving environment using artificial intelligence (AI)-based autonomous driving sensors, and more specifically, provides a calibration system and method for autonomous driving, the calibration system being used to adaptively adjust calibration parameters to selectively apply a calibration mode best suited to the current road.

[0010] The technical problems to be solved by this disclosure are not limited to those described above, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art from the following description.

[0011] This disclosure can be implemented in the following ways to solve all or at least some of the above-mentioned technical problems.

[0012] According to one or more exemplary embodiments of the present disclosure, a device for a vehicle may include: a processor; and a memory storing at least one instruction that, when executed by the processor, is configured to cause the device to: receive sensor data from the vehicle's sensors associated with objects present in the vehicle's surrounding environment; store the sensor data in the memory; receive from the vehicle: a set of parameters for calibrating a coordinate system for object recognition, and information regarding the resource status of at least one of the processor and the memory; determine a motion score and a confidence score based on the vehicle's driving environment information and motion information; adjust the confidence score based on the results of segmenting the sensor data and the results of object recognition; perform online calibration of the coordinate system for object recognition based on the adjusted confidence score; and control autonomous driving operations of the vehicle based on the calibrated coordinate system for object recognition.

[0013] At least one instruction can be configured to, when executed by the processor, cause the device to: adjust the confidence score by segmenting sensor data based on driving environment information and motion information.

[0014] At least one instruction can be configured to, when executed by the processor, cause the device to: adjust the confidence score by identifying objects based on driving environment information and motion information.

[0015] At least one instruction can be configured to further enable the device, when executed by the processor, to receive driving environment information via a global positioning system (GPS) or navigation device in the vehicle.

[0016] At least one instruction can be configured to, when executed by the processor, further cause the device to determine whether to perform online calibration by entering a waiting state without performing online calibration based on the resource status being less than a reference value.

[0017] At least one instruction can be configured to, when executed by the processor, further enable the device to: determine a motion score based on a direct or inverse proportional relationship between driving environment information and motion information.

[0018] At least one instruction can be configured to, when executed by the processor, further cause the device to determine whether to perform online calibration by entering a waiting state without performing online calibration based on a motion score less than a reference value.

[0019] At least one instruction can be configured to, when executed by the processor, further cause the device to determine whether to perform online calibration by entering a waiting state without performing online calibration based on a confidence score less than a reference value.

[0020] At least one instruction can be configured to cause the device to perform online calibration in such a way that, when executed by the processor, the online calibration parameters and online calibration confidence scores obtained for online calibration are stored in a memory buffer.

[0021] At least one instruction can be configured to cause the device to perform online calibration when executed by the processor by updating the parameter values ​​associated with the buffer to clear the buffer based on the buffer being full.

[0022] According to one or more exemplary embodiments of this disclosure, a method performed by a vehicle device may include: receiving sensor data from the vehicle's sensors associated with objects present in the vehicle's surrounding environment; receiving from the vehicle a set of parameters for calibrating a coordinate system for object recognition, and information about the resource status of at least one of the vehicle's processor and memory; determining a motion score and a confidence score based on the vehicle's driving environment information and motion information; adjusting the confidence score based on the results of segmenting the sensor data and the results of object recognition; performing online calibration of the coordinate system for object recognition based on the adjusted confidence score; and controlling autonomous driving operations of the vehicle based on the calibrated coordinate system for object recognition.

[0023] Adjusting the confidence score can include segmenting sensor data based on driving environment information and motion information.

[0024] Adjusting the confidence score can include identifying objects based on driving environment information and motion information.

[0025] The method may further include receiving driving environment information via a global positioning system (GPS) or navigation device in the vehicle.

[0026] The method may further include determining whether to perform online calibration by entering a waiting state without performing online calibration based on the resource status being less than a reference value.

[0027] This method may further include: determining the motion score based on the direct or inverse proportional relationship between driving environment information and motion information.

[0028] The method may further include determining whether to perform online calibration by entering a waiting state without performing online calibration based on a motion score that is less than a reference value.

[0029] The method may further include determining whether to perform online calibration by entering a waiting state without performing online calibration based on a confidence score that is less than a reference value.

[0030] Performing online calibration may include storing the online calibration parameters and online calibration confidence scores obtained for online calibration in a buffer of the memory.

[0031] Performing online calibration may include: updating the parameter values ​​associated with the buffer to clear the buffer if the buffer is full. Attached Figure Description

[0032] The above and other objects, features, and advantages of this disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Figure 1 This is a block diagram illustrating an exemplary system for controlling a vehicle to automatically identify objects and perform autonomous driving; Figure 2 This is a flowchart illustrating an exemplary operational process of an autonomous driving system with a LiDAR recognition software architecture; Figure 3 This is a flowchart illustrating an exemplary algorithm for applying calibration techniques from LiDAR sensors to an autonomous driving LiDAR identification software architecture; Figure 4 This is a flowchart of the calibration portion of an exemplary algorithm used to apply LiDAR sensor calibration techniques to an autonomous driving LiDAR recognition software architecture; Figure 5 This is a flowchart illustrating an exemplary process for performing calibration management; Figure 6 This is a flowchart illustrating an exemplary process for updating static calibration values; Figure 7 This is a block diagram illustrating an exemplary process for real-time verification of calibration and evaluation results; and Figure 8 This is a block diagram illustrating an exemplary computing system for autonomous vehicle control and object recognition computation according to embodiments of the present disclosure. Detailed Implementation

[0033] In the following, one or more exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, when affixing reference numerals to components in the various drawings, it should be noted that the same components shown in other drawings will also be indicated by the same reference numerals. In addition, detailed descriptions of well-known features or functions will be omitted in the description of exemplary embodiments of the present disclosure so as not to unnecessarily obscure the spirit of the disclosure.

[0034] In describing components of exemplary embodiments of this disclosure, the terms first, second, A, B, (a), (b), etc., may be used herein. These terms are used only to distinguish one component from another, regardless of the order or priority of the corresponding components, and do not limit the respective components. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms as defined in commonly used dictionaries should be interpreted as having a meaning equivalent to that in the context of the relevant field and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined as having an ideal or overly formal meaning in this application.

[0035] Unless otherwise defined, the terms used herein (including technical or scientific terms) may have the meanings commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0036] Expressions such as “comprise,” “may comprise,” “include,” “may include,” “have,” and “may have” as used herein are intended to indicate the presence of a characteristic (e.g., a function, operation, component, etc.) without excluding the presence of other additional characteristics. That is, these expressions should be understood as open-ended terms, encompassing the possibility that other examples may be included.

[0037] Unless the context otherwise indicates, the singular expressions used herein may include the meaning of the plural, and also apply to the singular expressions described in the claims.

[0038] As used herein, expressions such as “first” or “second” are used to distinguish one object from another when referring to multiple similar objects, unless otherwise indicated by the context, and do not limit their order or importance. For example, multiple chips according to this disclosure can be distinguished from each other by referring to them respectively as “first chip” and “second chip”.

[0039] As used herein, the term "cell" can refer to software or hardware components, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc. However, "cell" is not limited to hardware and software. A "cell" can be configured to be stored in addressable memory or can be configured to execute one or more processors. A "cell" can include components such as software components, object-oriented software components, class components, and task components, as well as processors, functions, properties, processes, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0040] As used herein, the expression “based on” is intended to describe one or more factors that influence the behavior or operation of a determination or decision described in a phrase or sentence that includes the expression, and the expression does not exclude any additional factors that influence the behavior or operation of a determination or decision.

[0041] When a component (e.g., a first component) as used herein is described as being “connected” or “coupled” to another component (e.g., a second component), it may mean that the component is not only directly connected or coupled to the other component, but also connected or coupled through yet another component (e.g., a third component).

[0042] Depending on the context, as used herein, the expression “configured to” can have meanings such as “set to,” “capable of,” “modified to,” “made,” “able to,” etc. The expression is not limited to the meaning of “specifically designed in hardware.” For example, a processor configured to perform a specific operation can refer to a general-purpose processor capable of performing a specific operation by executing software, or to a special-purpose computer constructed by programming to perform a specific operation.

[0043] For the purposes of this application and claims, the exemplary phrases “at least one of A; B; or C” or “at least one of A, B, or C” are used, which means “at least one A, or at least one B, or at least one C, or at least one A, at least one B, and at least one C.” Furthermore, as used herein, exemplary phrases such as “A, B, or C,” “at least one of A, B, and C,” “at least one of A, B, or C,” etc., may represent each listed item or all possible combinations of listed items. For example, “at least one of A or B” may refer to (1) at least one A; (2) at least one B; or (3) at least one A and at least one B.

[0044] According to the Society of Automotive Engineers (SAE), the autonomy levels of autonomous vehicles can be classified as follows: At Autonomous Level 0, the SAE classification standard corresponds to "No Autonomous Operation," where the autonomous driving system temporarily handles emergency situations (e.g., autonomous emergency braking) and / or only provides warnings (e.g., blind spot warning, lane departure warning, etc.) and expects the driver to operate the vehicle. At Autonomous Level 1, the SAE classification standard corresponds to "Driver Assistance," where the system performs some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.) when the driver operates the vehicle in normal operating mode, and expects the driver to determine the system's operating status and / or timing, perform other driving functions, and handle (e.g., resolve) emergency situations. At Autonomous Level 2, the SAE classification standard corresponds to "Partial Autonomy," where the system performs steering, acceleration, and / or braking under driver supervision, and expects the driver to determine the system's operating status and / or timing, perform other driving functions, and handle (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard can correspond to "conditional autonomy," where the system drives the vehicle under limited conditions (e.g., performing driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver when the desired conditions are not met. The driver is expected to determine the system's operating state and / or timing and take over control in emergency situations, but not otherwise operate the vehicle (e.g., steering, acceleration, and / or braking). At Level 4 of autonomous driving, the SAE classification standard can correspond to "high autonomy," where the system performs all driving functions and the driver is expected to control the vehicle only in emergency situations. At Level 5 of autonomous driving, the SAE classification standard can correspond to "full autonomy," where the system performs all driving functions without any assistance from the driver, including in emergency situations, and the driver is not expected to perform any driving functions other than determining the system's operating state. While this disclosure applies the SAE classification standard to autonomous driving classification, other classification methods and / or algorithms can be used in one or more configurations described herein. One or more features associated with autonomous driving control can be activated based on the configured autonomous driving control settings (e.g., based on at least one of the following: autonomous driving classification, selection of the vehicle's autonomous driving level, etc.).

[0045] Based on one or more features described herein (e.g., adaptive LiDAR calibration), vehicle operation can be controlled. Vehicle control can include various operational controls associated with the vehicle (e.g., autonomous driving control, sensor control, braking control, braking time control, acceleration control, acceleration rate of change control, warning timing control, forward collision warning timing control, etc.).

[0046] One or more auxiliary devices (e.g., engine brakes, exhaust brakes, hydraulic reducers, electric reducers, regenerative brakes, etc.) can also be controlled, for example, based on one or more features described herein (e.g., adaptive LiDAR calibration). For example, one or more communication devices (e.g., modems, network adapters, radio transceivers, antennas, etc.) capable of communicating via one or more wired or wireless communication protocols such as Ethernet, Wi-Fi, Near Field Communication (NFC), Bluetooth, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle-to-Everything (V2X), etc., can also be controlled based on one or more features described herein (e.g., adaptive LiDAR calibration).

[0047] One or more Minimum Risk Maneuvering (MRM) operations can also be controlled, for example, based on one or more features described herein (e.g., adaptive LiDAR calibration). A minimum risk maneuvering operation (e.g., minimum risk manipulation) can be a vehicle maneuvering operation designed to minimize (e.g., reduce) the risk of collision with surrounding vehicles to achieve a reduced (e.g., minimum) risk state. A minimum risk maneuvering operation can be an operation activated during autonomous driving when the driver is unable to respond to an intervention request. During minimum risk manipulation, one or more processors in the vehicle can control the vehicle's driving operations for a set time period.

[0048] For example, bias driving operations can also be controlled based on one or more features described herein (e.g., adaptive LiDAR calibration). The driving control device can perform bias driving control. To perform bias driving, the driving control device can control the vehicle to travel within the lane by maintaining the lateral distance between the vehicle's center position and the lane center. For example, the driving control device can control the vehicle to remain within the lane but not in the center of the lane.

[0049] The driving control equipment can identify a target lateral distance for offset driving control. For example, the target lateral distance may include an intentionally adjusted lateral distance that the vehicle can maintain from a reference point (such as the center of the lane or another vehicle) during maneuvers such as lane changes. This adjustment can be made to improve the vehicle's stability, safety, and / or performance under changing driving conditions. For example, during lane changes, the driving control system may offset the lateral distance to maintain a safer clearance from adjacent vehicles, taking into account factors such as vehicle speed, road conditions, and / or the presence of obstacles.

[0050] The level of autonomous driving and / or activation / deactivation of autonomous driving can also be controlled, for example, based on one or more features described herein (e.g., adaptive LiDAR calibration). The driving control device can perform autonomous driving level control (e.g., change of the level of autonomous driving, change of required user attention, etc.) or deactivate autonomous driving operation. For example, by changing the required user attention, the driver can be instructed to place his / her hands on the steering wheel more frequently (e.g., at least once within a threshold time period (e.g., 5 seconds, 30 seconds, 1 minute, etc.)). By changing the required user attention, the driver can be instructed to look forward more frequently (e.g., at least once within a threshold time period (e.g., 5 seconds, 30 seconds, 1 minute, etc.)). By changing the level of autonomous driving, one or more video contents may not be displayed on the vehicle's displays.

[0051] One or more sensors (e.g., IMU sensor, camera, LIDAR, RADAR, blind spot monitoring sensor, lane departure warning sensor, parking sensor, light sensor, rain sensor, traction control sensor, anti-lock braking system sensor, tire pressure monitoring sensor, seat belt sensor, airbag sensor, fuel sensor, emission sensor, throttle position sensor, inverter, converter, motor controller, power distribution unit, high voltage wiring and connectors, auxiliary power module, charging interface, etc.) can also be controlled, for example, based on one or more features described herein (e.g., adaptive LiDAR calibration).

[0052] Operational control for autonomous driving of a vehicle may include various driving controls of the vehicle via vehicle control equipment (e.g., acceleration, deceleration, steering control, gear shifting control, braking system control, traction control, stability control, cruise control, lane keeping assist control, collision avoidance system control, emergency braking assist control, traffic sign recognition control, adaptive headlight control, driver alert control, autonomous driving operation design domain (ODD), engaging and / or disengaging autonomous driving modes, etc.). One or more auxiliary devices (e.g., engine brakes, exhaust brakes, hydraulic reducers, electric reducers, regenerative brakes, etc.) may also be controlled, for example, based on one or more features described herein (e.g., adaptive LiDAR calibration). For example, one or more communication devices (e.g., modems, network adapters, radio transceivers, antennas, etc.) capable of communicating via one or more wired or wireless communication protocols such as Ethernet, Wi-Fi, Near Field Communication (NFC), Bluetooth, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle to Everything (V2X), etc.) may also be controlled based on one or more features described herein (e.g., assessing driver engagement).

[0053] The vehicle actively controlled by the autonomous driving system may be referred to as the vehicle itself, the master vehicle, or the autonomous vehicle. The vehicle may also be referred to as an automated driving vehicle, an autonomous vehicle (AC), a driverless vehicle, a robotaxi, a robotic car, or a robo-car. The vehicle may be equipped with an autonomous driving system. Alternatively, for example, the autonomous driving system may control the vehicle from an external and / or remote device such as a server. The vehicle may be partially or wholly controlled remotely by a remote human driver (e.g., guiding, driving, etc.). A vehicle in front of the vehicle (e.g., in the same lane as the vehicle) may be referred to as the vehicle in front (e.g., the vehicle directly in front), the vehicle in front (e.g., the vehicle directly in front), the lead vehicle, the leading vehicle, or the vehicle in front. A vehicle following the vehicle (e.g., in the same lane as the vehicle) may be referred to as the vehicle behind, the following vehicle, the vehicle in charge, or the vehicle behind. Adjacent vehicles can refer to any vehicle located in any direction (e.g., front, rear, left, right, diagonally, etc.) from the vehicle, provided that there are no other vehicles (e.g., intermediate vehicles) between them (e.g., regardless of their distance from the vehicle). Optionally, in some contexts, only those vehicles located within a threshold distance (e.g., the line-of-sight and / or detection limits of one or more sensors of the vehicle) can be referred to as adjacent vehicles. Target vehicles can be any vehicle close to the vehicle (e.g., within the threshold distance). Target vehicles can be any vehicle actively or passively, once or multiple times, and sporadically or continuously monitored, identified, identified, tracked, and / or analyzed by the autonomous driving system. For example, the threshold distance can be the line-of-sight and / or detection limits of one or more sensors of the vehicle, but the threshold distance can be a value smaller than the line-of-sight and / or detection limits of one or more sensors of the vehicle (e.g., an adjustable value). The target vehicle can be, for example, the vehicle in front, the vehicle behind, a vehicle in a lane different from the vehicle's lane (e.g., the vehicle to the left, the vehicle to the right, the vehicle diagonally opposite, etc.), and / or an adjacent vehicle (e.g., regardless of the distance from the vehicle and / or regardless of whether there is an intermediate vehicle between the target vehicle and the vehicle). The target vehicle can also be referred to as a surrounding vehicle, a nearby vehicle, an external vehicle, another vehicle (other vehicle), etc.

[0054] Figure 1 This is a block diagram illustrating an exemplary system for controlling a vehicle to autonomously identify objects and perform autonomous driving.

[0055] Reference Figure 1The vehicle control device 100 can be implemented inside or outside the vehicle, and some components included in the vehicle control device 100 can be implemented inside or outside the vehicle. In this case, the vehicle control device 100 can be configured integrally with the control unit in the vehicle, or it can be implemented as a separate device connected to the control unit of the vehicle via a separate connection device. For example, the vehicle control device 100 may further include Figure 1 Components not shown in the diagram.

[0056] The vehicle control device 100 may include a processor 110, a light detection and ranging (LiDAR) device 120, and a memory 130. The processor 110, LiDAR device 120, and memory 130 may be electronically or operatively coupled to each other via electronic components including a communication bus.

[0057] In the following text, operatively coupling hardware blocks to each other may include direct or indirect connections between hardware blocks being established wired and / or wirelessly, such that the second hardware is controlled by the first hardware in the hardware block.

[0058] Different blocks are shown, but this disclosure is not limited thereto. For example, Figure 1 The hardware blocks may be included in a single integrated circuit comprising a system-on-a-chip (SoC). The type and / or number of hardware blocks included in the vehicle control device 100 are not limited to... Figure 1 Those shown in the diagram. For example, vehicle control device 100 may include... Figure 1 Only a portion of the hardware is shown.

[0059] The vehicle control device 100 may include hardware for processing data based on one or more instructions. For example, the hardware for processing data may include a processor 110. For example, the hardware for processing data may include an arithmetic and logic unit (ALU), a floating-point unit (FPU), a field-programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 110 may have a single-core processor architecture, or it may have a multi-core processor architecture including dual-core, quad-core, hexa-core, or octa-core processors.

[0060] The processor 110 may include at least one or any combination of a graphics processing unit (GPU) and a neural processing unit (NPU). For example, a GPU may be referred to as a visual processing unit (VPU). For example, an NPU may be referred to as a neural network processing unit.

[0061] The vehicle control device 100 may include a depth sensor for detecting external objects. For example, the depth sensor for detecting external objects may include at least one or any combination of a time-of-flight (ToF) sensor, a LiDAR 120, a structured light sensor, an ultrasonic sensor, an infrared sensor, a radio detection and ranging (RADAR), and an optical distance sensor. In the following description, for ease of description, a description of LiDAR will be given.

[0062] The vehicle control device 100 may include a LiDAR 120 for acquiring multiple points based on pulsed laser signals. For example, the LiDAR 120 may acquire a dataset for identifying surrounding objects around the vehicle control device 100 (or a vehicle including the vehicle control device 100). For example, based on the reflection of pulsed laser signals emitted from the LiDAR 120 from surrounding objects, the LiDAR 120 may identify at least one or any combination of the position of the surrounding objects, the direction of motion of the surrounding objects, and the speed of the surrounding objects.

[0063] For example, LiDAR 120 can obtain a dataset representing external objects in a space formed by the x-axis, y-axis, and z-axis based on pulsed laser signals reflected from surrounding objects. For example, LiDAR 120 can obtain a dataset including multiple points in a space formed by the x-axis, y-axis, and z-axis based on receiving pulsed laser signals at specified intervals. For example, the multiple points can include points representing external objects in a three-dimensional (3D) virtual coordinate system. The 3D virtual coordinate system can include at least one of a vehicle coordinate system and a LiDAR coordinate system, or any combination thereof. However, instances of 3D virtual coordinate systems are not limited to those described above.

[0064] The memory 130 of the vehicle control device 100 may include hardware components for storing data and / or instructions input and / or output from the processor 110 of the vehicle control device 100. For example, the memory 130 may include volatile memory containing random access memory (RAM) and / or non-volatile memory containing read-only memory (ROM).

[0065] For example, volatile memory may include at least one or any combination of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). For example, non-volatile memory may include at least one or any combination of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid-state drive (SSD), and embedded multimedia card (eMMC).

[0066] One or more instructions that instruct the processor 110 of the vehicle control device 100 to perform calculations and / or operations using data may be stored in the memory 130 of the vehicle control device 100. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, subroutine, and / or application program.

[0067] In the following text, "the application is installed in the vehicle control device 100" may mean that one or more instructions provided in the form of an application are stored in the memory 130. This may mean that one or more applications are stored in a format that can be executed by the processor 110 of the vehicle control device 100 (e.g., as a file with an extension specified by the operating system of the vehicle control device 100).

[0068] For example, memory 130 may include a first neural network model for detecting objects. For example, memory 130 may include a second neural network model for outputting the type and / or scores of multiple points obtained by LiDAR 120.

[0069] The processor 110 can obtain at least one or any combination of a first virtual box representing a target object and a first category indicating the type of the target object, based on multiple points obtained via LiDAR 120 and a first neural network model stored in memory 130.

[0070] The processor 110 can obtain at least one or any combination of a first virtual bounding box representing a target object and a first category indicating the type of the target object by inputting multiple points into a first neural network model. For example, the first neural network model may include an object detection model. For example, the target object may include an external object located within a specified distance from the vehicle control device 100 (or the host vehicle including the vehicle control device 100). For example, the target object may include an object identified and continuously tracked by the vehicle control device 100. For example, the type of the target object may include multiple types for classifying the target object. For example, the type of the target object may include at least one or any combination of a first type indicating ground and a second type indicating a type different from ground. However, the type of the target object is not limited to the types described above. For example, the type of the target object may include at least one or any combination of a third type indicating a person and a fourth type indicating a vehicle.

[0071] The processor 110 can obtain at least one or any combination of a first part point corresponding to at least a portion of a target object and a second category identified via the first part point and indicating the type of the target object, based on a plurality of points and a second neural network model. For example, the second neural network model may include a segmentation model.

[0072] For example, the second neural network model may include a neural network model for obtaining the type of multiple points and the score of multiple points.

[0073] For example, processor 110 can obtain a first portion of points corresponding to at least a portion of a target object by inputting multiple points into a second neural network model. For example, processor 110 can identify the type of multiple points by inputting multiple points into a second neural network model. For example, processor 110 can obtain a first portion of points corresponding to at least a portion of a target object based on the type of each point in the multiple points.

[0074] Processor 110 can execute a first specified algorithm for multiple points. For example, processor 110 can execute a first specified algorithm for classifying the type of each point among the multiple points. For example, processor 110 can classify a second subset of points among the multiple points corresponding to a specified type. For example, the specified type may include a type representing the ground.

[0075] For example, processor 110 can classify a second part of points corresponding to a specified type based on performing a first specified algorithm on multiple points, and can exclude the second part of points from the multiple points to obtain (or identify) the first part of points.

[0076] Processor 110 can obtain at least one or any combination of partial categories and scores for each of the multiple points, used to obtain a second category, based on inputting multiple points into a second neural network model. For example, processor 110 can obtain a partial category and score for each of the multiple points based on inputting multiple points into the second neural network model. For example, the partial category may include classifying each of the multiple points as any type.

[0077] For example, processor 110 may fuse partial categories, the scores of each point in a plurality of points, and a second set of points. For example, processor 110 may perform clustering based on the fused partial categories, the scores of each point in a plurality of points, and the second set of points. For example, clustering may include grouping first set of points corresponding to at least a portion of a target object.

[0078] For example, processor 110 can obtain a point cloud for generating a second virtual bounding box based on the first subset of points. For example, processor 110 can obtain the point cloud by grouping the first subset of points.

[0079] For example, processor 110 can generate a second virtual bounding box, different from the first virtual bounding box, to represent the target object based on the point cloud. For example, the second virtual bounding box may include a box containing at least some of the first portion of points.

[0080] For example, processor 110 can identify a forward direction indicating the direction of the target object based on at least one or any combination of the first part of the points and the point cloud.

[0081] For example, processor 110 can identify the position of the second virtual bounding box in the virtual coordinate system based on at least one or any combination of the first partial points and the point cloud. For example, processor 110 can identify the size of the second virtual bounding box based on at least one or any combination of the first partial points and the point cloud. For example, processor 110 can identify a second category based on at least one or any combination of the first partial points and the point cloud. For example, processor 110 can identify at least one or any combination of the following: a direction of travel indicating the direction of movement of a target object, the position of the second virtual bounding box in the virtual coordinate system, the size of the second virtual bounding box, and the second category, based on at least one or any combination of the first partial points and the point cloud. For example, processor 110 can identify the direction of travel of the bounding box based on at least one or any combination of the first virtual bounding box, the first category, the direction of movement of the second virtual bounding box, the position of the second virtual bounding box, the size of the second virtual bounding box, and the second category. For example, processor 110 can identify the position of the bounding box in the virtual coordinate system based on at least one or any combination of the following: the first virtual bounding box, the first category, the direction of travel of the second virtual bounding box, the position of the second virtual bounding box, the size of the second virtual bounding box, and the second category. For example, processor 110 can obtain a third category indicating the type of target object corresponding to the bounding box based on at least one or any combination of a first virtual frame, a first category, the direction of movement of the second virtual frame, the position of the second virtual frame, the size of the second virtual frame, and a second category. For example, processor 110 can obtain at least one or any combination of a direction of movement of the bounding box, the position of the bounding box in the virtual coordinate system, and a third category indicating the type of target object corresponding to the bounding box based on at least one or any combination of a first virtual frame, a first category, the direction of movement of the second virtual frame, the position of the second virtual frame, the size of the second virtual frame, and a second category.

[0082] For example, processor 110 may assign a first identifier to the second virtual box for tracking the second virtual box. For example, processor 110 may assign a second identifier corresponding to the first identifier to the bounding box.

[0083] For example, processor 110 can use a second identifier to track bounding boxes. For example, processor 110 can track a target object at multiple frames based on identifying multiple bounding boxes, including those assigned the second identifier. For example, because the second identifier is an identifier assigned to the bounding box corresponding to the target object, processor 110 can identify multiple bounding boxes assigned the second identifier at multiple frames to track the target object.

[0084] The processor 110 can output a bounding box corresponding to a target object based on at least one of a first virtual box, a first category, a first partial point, and a second category, or any combination thereof. For example, the bounding box may include an instance of the target object represented in a virtual coordinate system in the form of a hexahedron.

[0085] The following is a brief description of the operations performed by the CPU, GPU and / or NPU included in processor 110.

[0086] Processor 110 may include at least one or any combination of a CPU, GPU, and NPU. For example, at least one or any combination of a GPU and NPU may obtain a first virtual bounding box and a first category based on a first neural network model. For example, at least one GPU and NPU may obtain a first virtual bounding box and a first category. For example, at least one GPU and NPU may obtain a partial category and a score for each of a plurality of points based on a second neural network model. For example, at least one GPU and NPU may obtain a partial category and a score for each of a plurality of points based on a second neural network model. For example, for a plurality of points, the CPU may classify a second subset of points corresponding to a specified type based on executing a first specified algorithm for classifying the type of each of the plurality of points.

[0087] As described above, the vehicle control device 100 may include at least one processor 110. The vehicle control device 100 can use at least one processor 110 to accurately detect target objects. Furthermore, by performing parallel processing, the vehicle control device 100 can reduce the load on each processor.

[0088] At the same time, in reference Figure 2 Previously, a description related to LiDAR-based object recognition was added to provide an overall understanding of AI object recognition according to this disclosure, as follows.

[0089] Object recognition processing by the LiDAR 120 or AI module involves three steps: preprocessing, segmentation, and tracking.

[0090] The object recognition system according to this disclosure can undergo preprocessing before performing the object recognition function. Preprocessing may include, for example, removing points forming the ground based on laser sensing data (e.g., raw data) input from the LiDAR 120. Because objects on the ground may be misidentified by laser beams reflected from it, a process to separate the ground from objects that are not on the ground can be performed in the preprocessing step, and this process can also be performed in the segmentation step if necessary.

[0091] For example, preprocessing in AI object recognition is understood as the following process: wherein the image processing tool of the AI ​​module in the processor 110 removes noise from the LiDAR point cloud image via voxel downsampling technology, for example, reducing the total number of points present in the LiDAR point cloud image to improve computational efficiency.

[0092] For reference, LiDAR point cloud images can be displayed using a bird's-eye view (BEV) scheme. If a LiDAR map is generated as if it were an aerial view of a city with birds flying in the air, this is called a BEV image.

[0093] In other words, as described above, LiDAR 120 transmits laser beams to the surrounding environment and records the time it takes for the laser beams to reflect back from objects existing in the outside, thus generating a point every few laser signals and calculating the distance to that point. By repeatedly emitting a large number of laser beams, processor 110 can generate a real-time LiDAR map of the surrounding environment as a 3D map of the BEV type, and can generate a real-time LiDAR map as a two-dimensional (2D) map if necessary.

[0094] The lines or surfaces shown in black and white on a LiDAR point cloud map are actually composed of countless points (each generated by a laser beam from the LiDAR120). Therefore, a LiDAR-sensed image is called a LiDAR point cloud image. Of course, for example, a LiDAR point cloud image can be recreated in color by combining red, green, and blue depth (RGB-D) sensors with a LiDAR sensor.

[0095] While it may be difficult for a human to identify an object using only one point from a large number of points in a LiDAR point cloud image, (for example) if a person views the point cloud from the perspective of a BEV or synthetically in the same way as a 2D planar diagram, then a person can roughly guess whether the surrounding environment of the vehicle currently performing autonomous driving has any shape. Furthermore, for example, vehicles, buses, pedestrians, street trees, traffic signs, etc., can be identified in the LiDAR point cloud image. In AI image recognition technology, such things or people are called objects. Objects can be classified into categories belonging to groups of specific properties, such as vehicle categories or bus categories.

[0096] The assistance of a deep AI neural network is needed to classify any object in a LiDAR point cloud image as either a vehicle or a bus. The AI ​​should first be trained to locate objects in the LiDAR point cloud image and identify their categories via the AI ​​neural network.

[0097] For example, it is called PANDASET TMThe dataset (source: https: / / pandaset.org / #data-collection) includes over 48,000 camera images (primarily captured in the Silicon Valley area of ​​the United States) and over 16,000 LiDAR scam images. These images are annotated across a total of 28 categories, such as pedestrians, cars, bicycles, construction site signs, and traffic signs.

[0098] In addition, point cloud working tools (such as Open3D) can be used. TM (Source: https: / / www.open3d.org / ) Visualizes LiDAR point cloud images into options suitable for the user's expectations. Because the LiDAR 120 is capable of distance detection, it is suitable for use with tools such as Open3D. TM When visually processing LiDAR point cloud images, the LiDAR 120 can more realistically reproduce 3D LiDAR images by displaying distant objects in dark blue and short objects in light blue, for example.

[0099] Furthermore, as mentioned above, techniques such as voxel (3D pixel) downsampling can be applied to LiDAR point cloud images to preprocess raw LiDAR images (e.g., raw data). Here, a voxel refers to a 3D pixel with a regular hexahedral shape, and voxel downsampling is a technique used to reduce the number of points without requiring excessive AI computation, even while preserving the structure of various objects included in the LiDAR point cloud.

[0100] Simultaneously, for example, in one scan cycle, LiDAR 120 fires m laser beams n times. In this case, the scan values ​​of the laser beams returning from collisions with external objects constitute an (m x n) matrix. This (m x n) matrix data is called a range image. Each point constituting the LiDAR point cloud image can include depth (e.g., distance) information, and may further include the intensity, azimuth, tilt, or other additional information of the returning laser pulses. Range images are large datasets, such as Waymo... TM Open Datasets (WOD). Enables AI learning from illustrated images.

[0101] Range view (RV) is a technique that transforms a 3D point cloud into a 2D scene (e.g., a 2.5D scene) to represent the 3D point cloud as a 3D LiDAR map (such as a simulated image) that humans can intuitively understand, rather than a large number of points. The 3D LiDAR point cloud image has 2D coordinates in the range view image, but 3D laser-related information (e.g., angles, tilt, intensity, etc.) recorded when the range image was previously acquired is not discarded. A 2D range view image is generated by applying a variable called width to the (x, y, z) coordinate values ​​of the 3D LiDAR image to obtain coordinates on one axis in two dimensions, and applying range image information indicating the range (depth) and variable information called height to the (z) coordinate to obtain coordinates on the other axis in two dimensions.

[0102] Furthermore, the AI ​​algorithms according to this disclosure may include convolutional neural networks (CNNs). CNNs are AI training modules frequently used to extract features (or feature points) from image data. For this purpose, there exists a dataset consisting of tens of thousands of commercially available images. CNNs currently have versions capable of processing each of one-dimensional to three-dimensional images. In other words, CNNs learn from the results of image processing tools across a range of perspectives to perform functions that help AI accurately identify objects in images.

[0103] Since objects surrounding autonomous vehicles are ultimately recognized by machines, generating ground truth (GT) bounding boxes on the aforementioned LiDAR map is also a crucial process in object recognition. Ground truth (GT) in machine learning is a term used to indicate the raw and actual values ​​that the AI ​​wants to learn from the data. It can generally be viewed as an image annotation overlaid on a LiDAR point cloud image with bounding boxes having box-shaped boundaries.

[0104] In other words, the AI ​​module extracts labels to group different objects for object identification. Of course, the interval between the 3D data points used to output the ground truth bounding box can be set, allowing for approximately 50 to 1000 LiDAR point cloud points to be included in a single ground truth bounding box.

[0105] Of course, when a vehicle is in motion, there are no GT annotations in the raw data (or raw data) captured by sensors such as LiDAR 120. Processor 110 should identify targets belonging to various categories such as road signs, crosswalks, pedestrians, other vehicles, and center lines as objects. GT annotations are a means of comparing the results of identifying objects by the AI ​​algorithm of processor 110 with reality to measure errors in object recognition and sometimes to evaluate AI performance. The GT bounding boxes overlaid on the raw image in the form of annotations can be manually set by the user, but commercially available GT calculation tools, such as grid stripes, are representative.

[0106] Simultaneously, if the AI ​​object recognition module is activated, the predicted bounding boxes can be examined. The results of the processor 110 identifying objects of a specific category based on raw image data obtained from LiDAR 120, etc., are represented as another bounding box similar to the ground truth (GT) bounding box. Unlike the GT bounding box, the predicted bounding box is the result of computation by the autonomously driven AI. The predicted bounding box may be identical to the GT bounding box, but it may not be identical, or it may not have any overlapping area between the predicted and GT bounding boxes.

[0107] For reference, since it is not possible to definitively conclude from the prediction bounding box alone that an object of a particular category actually exists in a particular location, the prediction bounding box is often referred to as a probability box (P-box) or prediction bounding box.

[0108] The segmentation process performed after preprocessing refers to displaying specific portions of the road (e.g., traffic lights) in red and the remaining portions (asphalt roads) in blue. During the segmentation step, point clouds can be clustered into groups, generating P-boxes. For reference, there is a technique called cluster expansion. Here, the expansion target can include clusters up to the seed point and all points within an ε-distance (the minimum distance that constitutes a cluster).

[0109] In other words, clustering based on point clouds and P-boxes is performed during segmentation. For example, the AI ​​network that performs segmentation is used to obtain point labels from LiDAR 120.

[0110] This disclosure proposes a rule-based road surface recognition and label fusion technique to address the problem of ground surface recognition errors that occur during segmentation. Fortunately, various techniques, such as slope-based road surface recognition, grid-based road surface recognition, and other non-planar road surface recognition methods, are representatively applied to road surface recognition algorithms. However, this disclosure proposes using a slope-based road surface recognition technique for identifying road surfaces within tunnels.

[0111] For reference, semantic segmentation is the task of attaching unique category labels to corresponding points in a point cloud generated by LiDAR 120. Semantic segmentation in LiDAR imaging technology is a technique for finding and using meaningful information from LiDAR data for object recognition or scene representation required for autonomous driving. Various semantic segmentation AI models already exist, such as projection-based methods, point-based methods, and sparse convolution-based methods. For example, the semantic segmentation result can be the result of AI computation performed by processor 110 along with the NVIDIA DRIVE™ AGX system. With this configuration, various colors can be added to, for example, LiDAR point cloud images.

[0112] LiDAR images undergo a process called post-processing after segmentation. Post-processing refers to converting point cloud data into 3D maps or models, which provides information meaningful for autonomous driving. Post-processing also includes removing noise or errors from the LiDAR point cloud images, identifying objects (such as vehicles or pedestrians) from the point cloud, and attaching and registering unique identifiers to the point cloud information if necessary.

[0113] Figure 2 This is a flowchart illustrating an example operation process 200 of an autonomous driving system with a LiDAR recognition software architecture.

[0114] S101 represents the process of converting optical data into packet data using LiDAR 120. In other words, the input to S101 is optical data, and the output of S101 is packet data. S101 performs the operation of converting data detected by the LiDAR sensor's optical system into digital information.

[0115] In S102, the grouped data of the grouped point cloud can be converted into point cloud data in the sensor coordinate system. The output of S102 can be point cloud data configured as a point cloud based on the sensor coordinate system. S102 can preferably be computed by a GPU processor. In other words, the input to S102 can be LiDAR grouped data and sensor metadata.

[0116] S103 may be a process of calibrating a point cloud sensor coordinate system version relative to the vehicle coordinate system to the point cloud. This process is used to remove dependencies on vehicle type, etc., in object recognition. S103 may preferably be computed by a GPU processor.

[0117] S104 can be a deep learning-based process for detecting objects from a point cloud. For example, the output of S104 can be a LiDAR data bounding box. In S104, the bounding box, including title information and the result of category recognition, can be used by the output of an AI network for category recognition. S104 can preferably be computed by a GPU processor.

[0118] In S111, the output in S104 can be converted into P-box information and category information derived from deep learning. The input to S104 can include point cloud data, sensor information, and timestamp information. As a result, the output in S111 can be box information calculated based on the results of deep learning and category recognition (e.g., whether a moving object or a static object exists).

[0119] Simultaneously, in S105, LiDAR data can be used as input and output for category classification information and related score information. In other words, the segmentation network described above can output the category and score results in points. S105 can preferably be computed by a GPU processor. The input to S105 can include point cloud data, sensor information, and timestamp information.

[0120] In S106, ground point classification can be performed after inputting LiDAR data. S106 can perform ground point classification based on signal processing rather than deep learning. Such a task can be performed on a point-by-point basis and can be configured to perform binary classification. The input to S106 can include point cloud data, sensor information, and timestamp information.

[0121] In S107, the category and ground recognition result for each point can be input, and the category and score can be output. In other words, S107 can be understood as a process used to fuse the outputs from S105 and S106 to enhance ground points and ensure ground recognition performance for range data. This can be a process of fusing ground data through a point cloud library (PCL).

[0122] In S108, LiDAR data and category information can be input, and the output can be a cluster ID and cluster-related information (e.g., cluster size, average number of points included in the cluster, etc.). In other words, the operation of dividing into clusters using point characteristics and distance information can be performed in S108. For example, the output in S108 can be a cluster ID assigned to each point.

[0123] In S109, LiDAR data and cluster ID information can be input, and P-boxes and related category information can be output. The output can be based on signal processing, including the 3D location, size, and title of the P-box, as well as its category. The output in S109 can be box information calculated based on signal processing and the result of category identification (e.g., whether a moving or stationary object exists).

[0124] In S110, LiDAR data and cluster ID information can be input, and contour information about the identified target object can be output. The output in S110 can be the result of contour recognition calculated based on signal processing.

[0125] In S112, meta-fusion of the results from S109, S110, and S111 can be performed. In other words, in S112, meta-fusion can be performed based on bounding box and category information based on deep learning output, as well as bounding box, category, and contour information obtained based on signal processing.

[0126] In S113, multi-object tracking IDs can be generated based on bounding box and category information.

[0127] For reference, in S103, during the process of converting the coordinates of each sensor into common sensor coordinates and then converting the common sensor coordinates into vehicle coordinates, the main parameters can be derived from... constitute. These refer to the amplitude in the horizontal, vertical, and height directions around the 3D reference axis, respectively. These refer to the distance, elevation angle, and azimuth angle to a point P represented in 3D space, respectively. These values ​​are measured by autonomous driving sensors (e.g., LiDAR 120). , , , , , It can be modified according to the default installation (DM) (i.e., the common design value for each vehicle type in S103) by the following Equation 1.

[0128] Equation 1

[0129] Equation 2

[0130] The data transformed according to Equation 1 above can be used to obtain a calibration transformation that best suits the current road conditions, based on Equation 2, through S117 described below. , , , , , A set of parameters (e.g., default data or a default parameter set). Existing. Figure 3 Algorithm 300, in which, Figure 2 This calibration process is reflected in Algorithm 200.

[0131] Figure 3 This is a flowchart illustrating algorithm 300 for applying LiDAR sensor calibration techniques to an autonomous driving LiDAR recognition software architecture. (The remaining text will be omitted.) Figure 2 The steps described are repeated.

[0132] In S114, Vehicle Dynamic Input Signal Processing (VDISP) information can be transmitted to the calibration manager and used for calibration in S117. VDISP information may include information such as the speed or direction of travel of the vehicle currently equipped with the autonomous driving sensor 120 (i.e., the master vehicle), and may have a common reference to the meaning of various input signals associated with the dynamic operation of the master vehicle in this disclosure. Represented as... Figure 3 The 301 information can correspond to VDISP information.

[0133] In S115, Global Positioning System (GPS) information can be transmitted to the calibration manager and used for calibration in S117. For example, information about whether the current driving environment is a highway or a city center can be included in the GPS information. (This is represented as...) Figure 3 The information in 302 can correspond to GPS information.

[0134] In S116, information input from an external source can be transmitted to the calibration manager and used for calibration in S117. In other words, any calibration values ​​received from the AI ​​module or the user can be included in the information input from an external source. This is represented as... Figure 3 The information in 303 can correspond to information input from an external source.

[0135] In S120, resource information can be transferred to the calibration manager via the resource manager and used for calibration in S117. This is because if S117 is executed, the AI ​​module should know the available computing power or resource status, such as memory resources. Specifically, the resource status can be the information necessary to determine whether the requirements for online calibration, as described below, are met. This is represented as... Figure 3 The 304 information can correspond to resource information.

[0136] The output in S107 can be sent to the calibration manager as a segmentation result and can be used for calibration in S117. (This is represented as...) Figure 3 The information in 305 can correspond to the output.

[0137] The meta-fusion results in S112 can also be transferred to the calibration manager and used for calibration in S117. The meta-fusion results can include information about objects that are moving or stationary in the current driving environment. (Represented as...) Figure 3 The information in 306 can correspond to the meta-fusion result.

[0138] In S118, online calibration can be performed and the results can be transferred to the calibration manager for use in calibration in S117. (This is represented as...) Figure 3 The information in section 307 can correspond to the results. Parameters associated with online calibration, confidence scores, etc., can be included in the results.

[0139] like Figure 3 As shown, one of the outputs in S117 can provide feedback to S118, which can be information about whether the on / off command of S118 is executed. This is represented as... Figure 3 The information in 308 can correspond to this output.

[0140] In S119, static calibration can be performed and the results can be transferred to the calibration manager for use in calibration in S117. (This is represented as...) Figure 3 The information in 309 can correspond to this result and can include parameters associated with static calibration.

[0141] like Figure 3 As shown, one of the outputs in S117 can be used as feedback for S119. Updated values ​​of the static calibration parameters required to perform S119 can be fed back. This is represented as... Figure 3 The information in 310 corresponds to this output.

[0142] In S117, the calibration process for the LiDAR 120 required for autonomous vehicles can be performed. All the information mentioned above, 301 to 310, can be used for the calibration process. In S117, calibration parameters suitable for the current operating conditions can be output, comparing the results of static calibration and online calibration. The prerequisites for online calibration can be checked using the aforementioned resource status, VDISP information, etc. Furthermore, in S117, the parameters used for static calibration can be updated using the confidence level regarding the calibration, and any calibration value desired by the user can be output if external input exists. As a result, the results of executing S117 can be provided to S103 as the final calibration parameters. (This is represented as...) Figure 3 The information in 311 corresponds to the result.

[0143] Figure 4 This is a flowchart illustrating the calibration process of Algorithm 300, which applies LiDAR sensor calibration techniques to the autonomous driving LiDAR recognition software architecture. (The remaining text will be omitted.) Figure 2 and 3 The details described are repeated.

[0144] S114' shows the immediate following Figure 3Information 301 was previously input and then transmitted to the state before S117. Here, the main vehicle speed, direction of travel, and yaw angle information can be included as VDISP information.

[0145] S115' shows the immediate following Figure 3 Information 302 was previously input and then transmitted to the state before S117. Information regarding driving environment identification can be included in information 302. For S117, information about whether the current driving environment is a highway or a city center can be transmitted.

[0146] S107' shows the immediate following Figure 3 Information 305 refers to the state before it was previously input and then transmitted to S117. Confidence scores, point information, etc., can be included in information 305.

[0147] S112' shows what follows immediately Figure 3 Information 306 represents the state prior to being input and then transmitted to S117. For example, the percentage of static objects occupied can be included in information 306.

[0148] S120' indicates that immediately following Figure 3 Information 304 represents the state that was previously entered and then transmitted to S117. Resource status of the processor 110 (specifically, the CPU) and GPU can be provided to S117.

[0149] It is possible Figure 4 The above checks are in Figure 3 Information 309 described in the document is transmitted from S119 to S117.

[0150] The information 308 transmitted from S117 to S118 can be used to calculate (e.g., determine) online calibration parameters (e.g., S118a) and calculate (e.g., determine) online calibration confidence scores (e.g., S118b). The online calibration parameters and online calibration confidence scores can be stored in memory (e.g., a predetermined area of ​​memory, such as a buffer).

[0151] The information 311 provided from S117 to S103 can be the final calibration parameters as described above, which can be used to switch the LiDAR reference coordinates to the vehicle coordinate system. To this end, in S117, a calibration switch (S117a) can occur to select the optimal coordinates, and static calibration-related parameters can be updated (e.g., S117b).

[0152] S113' indicates that immediately following Figure 3 Information 303 refers to the state that was previously entered and then transmitted to S117. Calibration values ​​randomly entered by the user can be included in information 303.

[0153] Figure 5 This is a flowchart illustrating an exemplary process 400 for performing calibration management. In other words, there exists a more detailed algorithm executed in S117. Figure 5 .

[0154] Reference Figure 5 In S401, resource monitoring can be performed. This can be corresponding to... Figure 3 The processing in S120. In S401, the current resource information of the system can be integrated and monitored. In S402, the resource information can be checked to ensure that it is in a state sufficient to perform online calibration.

[0155] If the result is "No" in S402, the process can proceed to S403 to disable the online calibration function. Due to resource shortage, the process can re-enter the waiting state and then wait until sufficient resources are available.

[0156] If resources are sufficient, the driving environment can be identified in S404. The vehicle's current road environment can be determined using GPS, navigation information, etc. For example, it can be determined whether the vehicle is traveling on a highway, in a tunnel, an underground road, in a typical city center, or in a narrow alley. The obtained driving environment information can be used to calculate motion scores and confidence scores.

[0157] In S405, the motion information of the main vehicle can be identified. The motion information of the main vehicle (e.g., speed, direction of travel, etc.) can be calculated using VDISP, etc. The motion information of the main vehicle can be combined with the driving environment information obtained in S404 to calculate the motion score. For example, the motion score can be configured to be lower than that on a highway, even at the same high speed (e.g., 70 km / h) in an urban environment.

[0158] In S406, it can be determined whether the calculated motion score is greater than a specific standard. If the motion score is less than the standard, the motion score can be moved to S407 to enter a waiting state.

[0159] In S408 and S409, where the movement is determined to be "yes" in S406, the motion and environmental information obtained above can be used. First, in the analysis of the segmentation results in S408, the label information and score of each point can be received from the segmentation output layer. This can be reflected in the confidence score, which uses factors that affect the performance of online calibration, such as the number (or ratio) of ground points, the number (or ratio) of noise points, and the category confidence score. For example, if the number of ground points is insufficient to detect the ground, the confidence score can be set to a small value, and even if the ratio of noise points is large, the confidence score can still be set to a small value.

[0160] If the detection results are analyzed in S409, the results of identifying moving and static objects can be received from the object recognition information output layer. This can be reflected in the confidence score using the ratio between moving objects (or points) and static objects (or points). For example, since online calibration is performed by reverse-tracking the movement of static objects, a smaller number of static objects allows for a smaller confidence score. As another example, with a larger number of dynamic objects, the score can be set lower due to the risk of error during calibration.

[0161] In S410, the results from S404, S405, S408, and S409 can be fused to calculate the fused result as the final confidence score. If the confidence score is greater than the reference threshold, the process can move to S412 to output the "online calibration probability" signal. If the confidence score is less than or equal to the threshold, the process can move to S411 to enter a waiting state.

[0162] At the same time, depending on the execution environment or as needed, at least some of S401 to S412 can be executed in parallel or sequentially.

[0163] Figure 6 This is a flowchart illustrating an exemplary process 500 for updating static calibration values. Figure 6 It is used for detailed description Figure 3 and Figure 4 The diagram of S119.

[0164] Reference Figure 6 In the online calibration step S118, parameter result values ​​(e.g., S118a) and confidence information (e.g., S118b) can be sent for processing according to S117. In S117, the data sent in S118 can be stored in a separately configured dedicated buffer (not shown). If the buffer (not shown) is full or online calibration is turned off (e.g., an on / off switch according to S117a), the optimal parameter values ​​can be updated (e.g., S117b) to clear the buffer. Alternatively, the optimal parameter values ​​can be provided to S119 as updated parameters and confidence scores.

[0165] Figure 7 This is a block diagram illustrating an exemplary process 600 for real-time verification of calibration and evaluation results. According to... Figure 7 Calibration scores can be calculated based on thresholds, confidence scores, etc. These calibration scores can be used to evaluate... Figure 3 The system's performance.

[0166] In some implementations, if software for object recognition is executed, the calibration value can be fixed based on the parameters specified initially (or static calibration), and calibration of the value may fail while driving. This can be considered a software architecture without considering the scalability of additional calibration logic for applications, such as supporting online calibration.

[0167] In view of the above problems, in at least some implementations, this disclosure is able to adaptively apply calibration parameters to select the calibration mode most suitable for the current road. For example, because online calibration has high accuracy on highways, autonomous driving can be controlled based on online calibration.

[0168] Furthermore, this disclosure allows for flexible changes to the calibration mode during driving. For example, in at least some implementations, autonomous driving operation may be aborted if a problem occurs during online calibration. However, despite problems in online calibration, this disclosure is able to change the calibration mode to a static calibration mode for continuous autonomous driving.

[0169] Furthermore, this disclosure ensures stability and accuracy by continuously updating static calibration parameters.

[0170] As mentioned above, some implementations use end-of-line (EOL) calibration values ​​that are fixed values. However, this disclosure can use values ​​updated by static calibration. Furthermore, the present invention can ensure resource connectivity with other logic based on an architecture design that considers CPU / GPU resources, and can introduce design schemes that consider external driving conditions, the motion information of the main vehicle, or the confidence level of the calibration to further ensure accuracy. This disclosure can use logic outputs (e.g., the number of ground points, the ratio of static objects, etc.) to ensure robust calibration performance for several scenarios, and can use logic outputs to extend and apply to different calibration algorithms. For example, this disclosure can perform ground-based calibration, target-based calibration, motion-based calibration, etc.

[0171] Figure 8 This is a block diagram illustrating an example computing system 1000 for autonomous vehicle control and object recognition computation.

[0172] Reference Figure 8 The computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 connected to each other via a bus 1200.

[0173] Processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory 1300 and / or storage device 1600. Memory 1300 and storage device 1600 may include various types of volatile or non-volatile storage media. For example, memory 1300 may include read-only memory (ROM) 1310 and random access memory (RAM) 1320.

[0174] Therefore, the operation of the methods or algorithms described herein can be directly implemented using hardware modules, software modules, or a combination of hardware and software modules, which are executed by processor 1100. Software modules can reside on storage media such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, and CD-ROMs (e.g., memory 1300 and / or storage device 1600).

[0175] An exemplary storage medium may be coupled to processor 1100. Processor 1100 may read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with processor 1100. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside within the user terminal. In another scenario, the processor and storage medium may reside as separate components in the user terminal.

[0176] According to one aspect of this disclosure, a system for identifying objects via a vehicle equipped with an artificial intelligence (AI) module and autonomous driving sensors may include a processor equipped with an AI module, the AI ​​module including a calibration module for identifying objects present in the surrounding environment and driving environment from data input from the autonomous driving sensors, and a memory for storing the input data. The AI ​​module may perform actions in parallel or sequentially, including receiving default data required for calibration from the autonomous driving sensors or the vehicle, receiving information about the resource status of the processor or memory, calculating motion scores and confidence scores based on driving environment information and vehicle motion information, adjusting the confidence scores using the results of segmenting the data input from the autonomous driving sensors and the results of identifying moving or static objects present in the driving environment, and determining whether to perform online calibration based on the adjusted confidence scores.

[0177] Adjusting the confidence score can be included in the results of segmented data to further reflect the processing of driving environment and motion information.

[0178] Adjusting the confidence score can include additional processing that reflects driving environment and motion information in the results of identifying moving or stationary objects.

[0179] Information about the driving environment can be obtained using the vehicle's Global Positioning System (GPS) or navigation device.

[0180] Determining whether to perform online calibration may include a process where, if the resource status is less than a first reference value, the user enters a waiting state without performing online calibration.

[0181] The motion score can be calculated variably based on the direct or inverse proportional relationship between driving environment information and motion information.

[0182] Determining whether to perform online calibration may include the following process: if the motion score is less than the second reference value, enter a waiting state without performing online calibration.

[0183] Determining whether to perform online calibration may include the following process: if the confidence score is less than the third reference value, enter a waiting state without performing online calibration.

[0184] Determining whether to perform online calibration may include the following process: if it is determined that online calibration should be performed, the online calibration parameters and online calibration confidence scores obtained through online calibration are stored in a certain storage space.

[0185] Determining whether to perform online calibration may include the following processes: updating optimal parameter values ​​at a point in time when a specific storage space is full and emptying the full specific storage space.

[0186] A method for identifying the surrounding environment and targets from data of a vehicle equipped with an artificial intelligence (AI) module and autonomous driving sensors and adjusting coordinates required for object recognition may include: the AI ​​module receiving default data required for calibration from the autonomous driving sensors or the vehicle, and receiving information about the resource status of the processor or memory; calculating a motion score and a confidence score based on driving environment information and vehicle motion information; adjusting the confidence score using the results of segmenting data input from the autonomous driving sensors and the results of identifying moving or static objects present in the driving environment; and determining whether to perform online calibration based on the adjusted confidence score.

[0187] Furthermore, based on the detailed description and accompanying drawings of this disclosure, those skilled in the art can understand various effects other than those described above.

[0188] While this disclosure has been described above with reference to one or more exemplary embodiments and accompanying drawings, it is not limited thereto, but various modifications and changes can be made by those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure as claimed in the appended claims.

[0189] Therefore, the exemplary embodiments of this disclosure are not intended to limit the technical spirit of this disclosure, but are provided for illustrative purposes only. The scope of this disclosure should be interpreted based on the appended claims, and all technical concepts within the scope of the claims should be included within the scope of this disclosure.

Claims

1. A device for controlling autonomous driving of a vehicle, the device comprising: processor; as well as The memory stores at least one instruction configured to cause the device to: when executed by the processor. Receive sensor data from the vehicle's sensors that is associated with objects present in the vehicle's surrounding environment; The sensor data is stored in the memory; Received from the vehicle: A set of parameters used to calibrate the coordinate system for object recognition, and Information about the resource status of at least one of the processor and memory; The motion score and confidence score are determined based on the vehicle's driving environment information and motion information; The confidence score is adjusted based on the results of segmenting the sensor data and identifying the object. Based on the adjusted confidence scores, online calibration of the coordinate system used for object recognition is performed; as well as The autonomous driving operation of the vehicle is controlled based on a calibrated coordinate system for object recognition.

2. The device according to claim 1, wherein, The at least one instruction is configured to cause the device, when executed by the processor, to: The confidence score is adjusted by segmenting the sensor data based on the driving environment information and the motion information.

3. The device according to claim 1, wherein, The at least one instruction is configured to cause the device, when executed by the processor, to: The confidence score is adjusted by identifying the object based on the driving environment information and the motion information.

4. The device according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, further cause the device to: The driving environment information is received via the vehicle's global positioning system or navigation device.

5. The device according to claim 1, wherein, The at least one instruction is configured to, when executed by the processor, further cause the device to determine whether to perform the online calibration by entering a waiting state without performing the online calibration based on the resource status being less than a reference value.

6. The device according to claim 1, wherein, The at least one instruction is configured to further cause the device, when executed by the processor: The motion score is determined based on the direct or inverse proportional relationship between the driving environment information and the motion information.

7. The device according to claim 1, wherein, The at least one instruction is configured, when executed by the processor, to further cause the device to determine whether to perform the online calibration in the following manner: If the motion score is less than the reference value, the system enters a waiting state and does not perform the online calibration.

8. The device according to claim 1, wherein, The at least one instruction is configured, when executed by the processor, to further cause the device to determine whether to perform the online calibration in the following manner: If the confidence score is less than the reference value, the system enters a waiting state and does not perform the online calibration.

9. The device according to claim 1, wherein, The at least one instruction is configured to cause the device to perform the online calibration in the following manner when executed by the processor: The online calibration parameters and online calibration confidence scores obtained for the online calibration are stored in the buffer of the memory.

10. The device according to claim 9, wherein, The at least one instruction is configured to cause the device to perform the online calibration in the following manner when executed by the processor: Since the buffer is full, update the parameter value associated with the buffer to clear the buffer.

11. A method for controlling autonomous driving of a vehicle, the method comprising: Receive sensor data from the vehicle's sensors that is associated with objects present in the vehicle's surrounding environment; Received from the vehicle: A set of parameters used to calibrate the coordinate system for object recognition, and Information regarding the resource status of at least one of the vehicle's processor and the vehicle's memory; The motion score and confidence score are determined based on the vehicle's driving environment information and motion information; The confidence score is adjusted based on the results of segmenting the sensor data and identifying the object. Based on the adjusted confidence scores, online calibration of the coordinate system used for object recognition is performed; as well as The autonomous driving operation of the vehicle is controlled based on a calibrated coordinate system for object recognition.

12. The method according to claim 11, wherein, Adjusting the confidence score includes: The sensor data is segmented based on the driving environment information and the motion information.

13. The method according to claim 11, wherein, Adjusting the confidence score includes: The object is identified based on the driving environment information and the motion information.

14. The method of claim 11, further comprising: The driving environment information is received via the vehicle's global positioning system or navigation device.

15. The method of claim 11, further comprising determining whether to perform the online calibration by: If the resource status is less than the reference value, the system enters a waiting state and does not perform the online calibration.

16. The method of claim 11, further comprising: The motion score is determined based on the direct or inverse proportional relationship between the driving environment information and the motion information.

17. The method of claim 11, further comprising determining whether to perform the online calibration by: If the motion score is less than the reference value, the system enters a waiting state and does not perform the online calibration.

18. The method of claim 11, further comprising determining whether to perform the online calibration by: If the confidence score is less than the reference value, the system enters a waiting state and does not perform the online calibration.

19. The method according to claim 11, wherein, Performing the online calibration includes: The online calibration parameters and online calibration confidence scores obtained for the online calibration are stored in the buffer of the memory.

20. The method according to claim 19, wherein, Performing the online calibration includes: Since the buffer is full, update the parameter value associated with the buffer to clear the buffer.

Images