Method, device and computer program product for generating input feature map of artificial intelligence model

Abstract

This invention provides a method, apparatus, and computer program for generating input feature maps for an artificial intelligence model based on lidar object detection. The method for generating input feature maps for an artificial intelligence model based on lidar object detection, according to various embodiments of the invention, is executed by a computing device. The method includes: collecting lidar sensor data in the form of a three-dimensional point cloud for a predetermined region; processing the collected lidar sensor data based on one or more predefined metrics to generate one or more first feature maps; and combining the generated one or more first feature maps with second feature maps pre-generated based on distance and angle as separate channels to generate an input feature map for the artificial intelligence model.

Description

Technical Field

[0001] Various embodiments of the present invention relate to a method, apparatus, and computer program product for generating input feature maps for an artificial intelligence model of object detection based on lidar. Background Technology

[0002] To make driving easier for users, there is a growing trend to install various sensors and electronic devices (e.g., Advanced Driver Assistance Systems, ADAS), and in particular, an autonomous driving system is being developed that can identify the surrounding environment and drive the vehicle to a given destination without driver intervention.

[0003] Among them, autonomous vehicles refer to vehicles equipped with autonomous driving systems that can identify the surrounding environment and drive to a given destination without the intervention of a driver.

[0004] In order to control the driving behavior of autonomous vehicles without driver intervention, it is necessary to first identify the movement of objects (e.g., obstacles) in the surrounding environment of the autonomous vehicle. In order to prevent accidents and ensure safety, more accurate object identification is needed. Therefore, technologies that can detect objects around autonomous vehicles through various sensor data are being actively developed.

[0005] Previously, by using sensors (e.g., LiDAR sensors) installed in autonomous vehicles, sensor data perceiving the surroundings of the autonomous vehicle was used as learning data to enable artificial intelligence models to learn. The LiDAR sensor data was collected in real time using the learned artificial intelligence model, and the data was analyzed to control the vehicle to detect objects (e.g., obstacles) around it and avoid them.

[0006] However, conventional methods have the following problems: using LiDAR sensor data in the form of 3D point clouds (e.g., a set of (x, y, z) points) as learning data for artificial intelligence models, or in order to effectively utilize the input data for learned artificial intelligence models, requires converting it into a suitable tensor form as input data for the artificial intelligence model. However, the computational cost of generating input feature maps for 3D or 2D convolution is very high, thus the process of detecting objects takes a long time.

[0007] Furthermore, when increasing the number of channels in the input feature map to derive more accurate result values, there is a problem of increased inefficiency in the learning and reasoning actions of the artificial intelligence model. Summary of the Invention

[0008] Technical issues

[0009] The technical problem to be solved by this invention is to provide a method, apparatus, and computer program product for generating input feature maps for an artificial intelligence model based on LiDAR object detection. The method pre-computed the distance and angle of a preset object detection area around an autonomous vehicle. Using feature maps related to the generated and stored distance and angle, and LiDAR sensors in the form of real-time collected 3D point clouds, the generated feature maps are channel-combined to generate input feature maps for an artificial intelligence model based on LiDAR object detection. That is, by using pre-computed feature maps to generate input feature maps more quickly, faster learning and inference computation of the artificial intelligence model can be achieved, and input feature maps capable of deriving accurate result values ​​can be generated.

[0010] The technical problems to be solved by the present invention are not limited to those described above. Those skilled in the art will clearly understand other unmentioned problems through the following description.

[0011] Technical solution

[0012] According to an embodiment of the present invention, a method for generating input feature maps for an artificial intelligence model based on lidar object detection, which addresses the aforementioned technical problems, is executed by a computing device. The method may include: collecting lidar sensor data in the form of a three-dimensional point cloud for a predetermined region; processing the collected lidar sensor data based on one or more predefined indicators to generate one or more first feature maps; and combining the generated one or more first feature maps with second feature maps pre-generated based on distance and angle as separate channels to generate an input feature map for the artificial intelligence model.

[0013] In various embodiments, the step of generating one or more first feature maps may include: a step of meshing the collected lidar sensor data based on the XY plane to generate multiple mesh pillars; a step of calculating one or more predefined indicators for each of the generated multiple mesh pillars; and a step of inputting the calculated one or more indicators into a matrix corresponding to the collected lidar sensor data to generate one or more first feature maps.

[0014] In various embodiments, the more than one predefined index refers to the number of LiDAR sensor points, the maximum height, the minimum height, and the average height. The step of generating more than one first feature map may include the step of generating a first feature map corresponding to each of the number of LiDAR sensor points, the maximum height, the minimum height, and the average height as an independent channel.

[0015] In various embodiments, the method may further include: a step of meshing a preset object detection region based on the XY plane to generate multiple meshes with a predetermined size; a step of generating a distance-related second feature map based on the distance between the generated multiple meshes and a reference point; and a step of generating an angle-related second feature map based on the angle between the generated multiple meshes and the reference point.

[0016] In various embodiments, the step of generating a distance-related second feature map may include: calculating the distance between the center point coordinates of each of the generated plurality of grids and the reference point coordinates; standardizing the calculated distance values ​​using a preset maximum distance value; and inputting the standardized distance values ​​into a matrix corresponding to the preset object detection region to generate the distance-related second feature map.

[0017] In various embodiments, the step of generating a distance-related second feature map may include: calculating the distance value between the center point coordinates of each of the generated plurality of grids and the reference point coordinates; transforming the calculated distance value using a predefined logarithmic function; and inputting the transformed distance value into a matrix corresponding to the preset object detection region to generate the distance-related second feature map.

[0018] In various embodiments, the step of generating the angle-related second feature map may include: calculating the angle value between the line connecting the center point of each of the generated plurality of grids and the reference point and the X-axis or Y-axis; converting the calculated angle value using Euler angle transformation; and inputting the converted angle value into a matrix corresponding to the preset object detection region to generate the angle-related second feature map.

[0019] In various embodiments, the step of generating the angle-related second feature map may include: calculating the angle between the line connecting the center point of each of the generated plurality of grids and the reference point and the X-axis or Y-axis; calculating the trigonometric ratio for the calculated angle value; and inputting the calculated trigonometric ratio into a matrix corresponding to the preset object detection region to generate two angle-related second feature maps.

[0020] An input feature map generation apparatus for an artificial intelligence model based on lidar object detection according to another embodiment of the present invention, for solving the above-mentioned technical problems, may include: a processor; a network interface; a memory; and a computer program loaded into the memory and executed by the processor. The computer program may include: instructions for collecting lidar sensor data in the form of three-dimensional point clouds for a predetermined region; instructions for processing the collected lidar sensor data based on one or more predefined indicators to generate one or more first feature maps; and instructions for generating an input feature map for an artificial intelligence model by combining the generated one or more first feature maps and second feature maps pre-generated based on distance and angle as separate channels.

[0021] According to another embodiment of the present invention, a computer program recorded on a computer-readable recording medium is combined with a computing device to solve the above-mentioned technical problems. The computer program is stored in the computer-readable recording medium to perform the following steps: collecting lidar sensor data in the form of a three-dimensional point cloud for a predetermined region; processing the collected lidar sensor data based on one or more predefined indicators to generate one or more first feature maps; and combining the generated one or more first feature maps and second feature maps pre-generated based on distance and angle as separate channels to generate an input feature map for an artificial intelligence model.

[0022] Further details of the invention are included in the detailed description and accompanying drawings.

[0023] Invention Effects

[0024] According to various embodiments of the present invention, the following advantages are available: the distance and angle of a preset object detection area around the autonomous vehicle are pre-computed; feature maps related to the generated and stored distance and angle are combined with a LiDAR sensor in the form of a real-time collected 3D point cloud; and the generated feature maps are channel-wise combined to generate an input feature map for an artificial intelligence model based on LiDAR object detection. In other words, the input feature map is generated faster by using the pre-computed feature map, thereby enabling faster learning and inference calculations of the artificial intelligence model, and generating an input feature map that can derive accurate result values.

[0025] The effects of the present invention are not limited to those mentioned above, and those skilled in the art will clearly understand other effects not mentioned through the following description. Attached Figure Description

[0026] Figure 1 This diagram illustrates an input feature map generation system for an artificial intelligence model of object detection based on lidar, according to an embodiment of the present invention.

[0027] Figure 2 This is a hard disk structure diagram of an artificial intelligence model for detecting objects based on lidar, according to another embodiment of the present invention.

[0028] Figure 3 This is a flowchart of an input feature map generation method for an artificial intelligence model for object detection based on lidar, according to another embodiment of the present invention.

[0029] Figure 4 This is a flowchart describing a method for generating more than one first feature map in various embodiments.

[0030] Figure 5 This diagram illustrates the process of generating more than one first feature map in various embodiments.

[0031] Figure 6 This is a flowchart describing a method for generating a distance-related second feature map in various embodiments.

[0032] Figure 7 This is a flowchart illustrating a method for generating an angle-related second feature map in various embodiments.

[0033] Figure 8 This diagram illustrates the process of generating a second feature map related to distance and angle in various embodiments. Detailed Implementation

[0034] The advantages and features of the present invention, as well as the methods of implementing them, will become apparent from the following detailed description of embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and these embodiments are provided merely to complete the disclosure of the invention and to fully inform those skilled in the art of the scope of the invention, which is defined only by the scope of the claims.

[0035] The terminology used herein is for describing embodiments and not for limiting the invention. Throughout this document, the singular form includes the plural form unless otherwise specified in the text. As used herein, “comprises” and / or “comprising” do not exclude the presence or addition of more than one other structural element besides the stated structural element. Throughout the specification, the same reference numerals refer to the same structural elements, and “and / or” includes each of the mentioned structural elements and all combinations thereof. Although “first,” “second,” etc., are used to describe individual structural elements, it is understood that these structural elements are not limited by these terms. These terms are used only to distinguish one structural element from another. Therefore, it is understood that the first structural element mentioned below, within the technical spirit of the invention, can also be a second structural element.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) may have the meaning commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, unless specifically and explicitly defined, terms as defined in commonly used dictionaries should not be ideally or excessively interpreted.

[0037] The terms “department” or “module” as used herein refer to software, hard disk structural elements such as FPGAs or ASICs, and “departments” or “modules” that perform certain functions. However, a “department” or “module” is not limited to software or hard disks. A “department” or “module” can be configured to reside on an addressable storage medium or be reproduced by more than one processor. Thus, as an example, a “department” or “module” includes software structural elements, software structural elements pointing to objects, structural elements such as class structural elements and task structural elements, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided in structural elements and “departments” or “modules” can be combined into a smaller number of structural elements and “departments” or “modules”, or further separated into additional structural elements and “departments” or “modules”.

[0038] As shown in the figure, spatial relative terms such as "below," "below," "lower," "above," and "upper" can be used to readily describe the relationship between a structural element and other structural elements. Spatial relative terms should be understood to include terms describing different orientations of the structural element during use or operation, in addition to those shown in the figure. For example, when the structural element shown in the figure is flipped, a structural element described as "below" or "below" of other structural elements can be placed "above" of other structural elements. Therefore, the exemplary term "below" can include both "below" and "above." Structural elements can also be oriented in other directions, thus the spatial relative terms can be interpreted according to orientation.

[0039] In this document, "computer" refers to all types of hard disk devices including at least one processor, and can be understood to also include the meaning of software configuration running in the respective hard disk device according to the embodiments. For example, "computer" can be understood to include, but is not limited to, smartphones, tablets, desktop computers, laptops, and user clients and applications running on each device.

[0040] Furthermore, in the input feature map generation method for an artificial intelligence model for detecting objects based on lidar according to various embodiments of the present invention, the purpose of controlling the driving of an autonomous vehicle is to describe the generation of input feature maps for detecting objects in the surrounding area of ​​the autonomous vehicle, but it is not limited thereto. It can be applied to general vehicles (e.g., vehicles in which the driver directly intervenes to control the driving) or vehicles that operate according to semi-autonomous driving functions (e.g., functions in which the driver directly intervenes to control the driving, but only performs partial autonomous control on a portion of the functions).

[0041] In this context, when applied to general vehicles or vehicles operating under semi-autonomous driving functions, the input feature map is generated to assist driving (e.g., to provide collision prevention notifications through object detection), rather than to directly control driving by detecting objects.

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0043] The steps described herein are described as being performed by a computer, but the subject of each step is not limited thereto, and at least a portion of each step may be performed in different devices according to embodiments.

[0044] Figure 1 This diagram illustrates an input feature map generation system for an artificial intelligence model of object detection based on lidar, according to an embodiment of the present invention.

[0045] Reference Figure 1 An input feature map generation system for an artificial intelligence model for detecting objects based on lidar, according to an embodiment of the present invention, may include an input feature map generation device 100, a user terminal 200, an external server 300, and a network 400.

[0046] in, Figure 1 The input feature map generation system shown for an artificial intelligence model of object detection based on LiDAR is according to one embodiment, and its structural elements are not limited to... Figure 1 The embodiments shown may be added, changed, or deleted as needed.

[0047] In one embodiment, the input feature map generation device 100 can learn an artificial intelligence model for detecting objects around the autonomous vehicle 10 by analyzing LiDAR sensor data in the form of a three-dimensional point cloud, and can generate an input feature map of the artificial intelligence model to detect objects around the autonomous vehicle 10 using the pre-learned artificial intelligence model. For example, the input feature map generation device 100 can be connected to the autonomous vehicle 10 via a network 400 and can receive LiDAR sensor data collected from sensors (e.g., LiDAR sensors) installed in the autonomous vehicle 10 to generate the input feature map.

[0048] The artificial intelligence model can be a deep learning model (e.g., CNN) used to analyze lidar sensor data in the form of three-dimensional clouds, but is not limited to this.

[0049] Artificial intelligence models consist of one or more network functions, which can be composed of interconnected computational units commonly referred to as "nodes." These "nodes" can also be called "neurons." One or more network functions are configured by including at least one node. The nodes (or neurons) that make up one or more network functions can be interconnected through one or more "links."

[0050] In artificial intelligence models, multiple nodes connected by links can form relative relationships between input and output nodes. The concepts of input and output nodes are relative; for a given node, any node with an output node relationship can also be an input node relationship with other nodes, and vice versa. As mentioned above, relationships from input nodes to output nodes can be created around links. More than one output node can be connected to an input node through links, and vice versa.

[0051] In a relationship between input and output nodes connected by a link, the value of the output node can be determined based on the data input to the input nodes. The nodes interconnecting the input and output nodes can have weights. These weights can be variable, and can be made variable by the user or an algorithm, to perform the functions required by the artificial intelligence model. For example, when more than one input node is interconnected with an output node through its own links, the output node can determine its value based on the values ​​input to the input nodes connected to the output node and the weights set in the links corresponding to the respective input nodes.

[0052] As mentioned above, in an artificial intelligence (AI) model, one or more nodes are interconnected through one or more links to form relationships between input and output nodes. The AI ​​model can be defined based on the number of nodes and links, the relationships between nodes and links, and the weight values ​​assigned to each link. For example, if two AI models exist with the same number of nodes and links but different weight values ​​between the links, the two AI models can be identified as distinct.

[0053] A subset of nodes constituting an artificial intelligence model can be configured as a layer based on their distance from the initial input node. For example, a set of multiple nodes at a distance of n from the initial input node can constitute an n-layer. The distance from the initial input node can be defined by the minimum number of links required to reach the corresponding node from the initial input node. However, this definition of a layer is arbitrary and for interpretation purposes; the order of layers in an artificial intelligence model can be defined in different ways than described above. For example, a node's layer can also be defined by its distance from the final output node.

[0054] An initial input node can refer to one or more nodes in an artificial intelligence model whose relationships with other nodes are not linked but rather through direct data input. Alternatively, it can refer to a node in the artificial intelligence model network whose relationships are based on links and do not have other input nodes connected by links. Similarly, a final output node can refer to one or more nodes in an artificial intelligence model whose relationships with other nodes do not have an output node. Furthermore, a hidden node can refer to any node constituting the artificial intelligence model other than the initial input node and the final output node. An artificial intelligence model according to an embodiment of the present invention can be a model in which the number of nodes in the input layer may be greater than the number of nodes in the hidden layer near the output layer, and the number of nodes decreases as the model progresses from the input layer to the hidden layer.

[0055] Artificial intelligence models can include more than one hidden layer. Hidden nodes in a hidden layer can use the output of the previous layer and the outputs of surrounding hidden nodes as input. The number of hidden nodes in each hidden layer can be the same or different. The number of nodes in the input layer can be determined based on the number of data fields in the input data and can be the same as or different from the number of hidden nodes. Input data fed into the input layer can be computed through the hidden nodes of the hidden layer and can be output by a fully connected layer (FCL) as the output layer.

[0056] In various embodiments, the artificial intelligence model can undergo supervised learning by using LiDAR sensor data labeled with object information as learning data. However, the invention is not limited to this, and various learning methods can be applied.

[0057] Supervised learning is a method of generating learning data by labeling specific data and information related to that specific data, and learning from that data. It is also a method of generating learning data by labeling two data points that have a causal relationship, and learning from the generated learning data.

[0058] More specifically, the input feature map generation device 100 can perform the learning of one or more network functions constituting an artificial intelligence model using labeled learning data. For example, the input feature map generation device 100 inputs each learning input data to one or more network functions, and each output data calculated by one or more network functions can be compared with each learning output data corresponding to the label of each learning input data to calculate the error. That is, in the learning of the artificial intelligence model, the learning input data can be input to the input layer of one or more network functions, and the learning output data can be compared with the output of one or more network functions.

[0059] The input feature map generation device 100 can enable the artificial intelligence model to learn based on the error between the calculation results of the learning input data and the learning output data (label) using one or more network functions.

[0060] Furthermore, the input feature map generation device 100 can adjust the weights of one or more network functions based on the error using a backpropagation method. That is, the input feature map generation device 100 can adjust the weights based on the error between the calculation results of one or more network functions for the learning input data and the learning output data, so that the output of one or more network functions is close to the learning output data.

[0061] When the learning of more than one network function has been performed for a predetermined epoch, the input feature map generation device 100 can use validation data to determine whether to stop learning. The predetermined epoch can be a part of the entire learning target period.

[0062] The validation data may consist of at least a portion of the labeled learning data. That is, the input feature map generation device 100 performs learning of the artificial intelligence model using the learning data. After the learning of the artificial intelligence model has been repeated for a predetermined period, the validation data can be used to determine whether the learning effect of the artificial intelligence model is above a predetermined level. For example, when the input feature map generation device 100 uses 100 pieces of learning data to perform learning with a target number of iterations of 10, after performing 10 iterations of learning for a predetermined period, it uses 10 pieces of validation data to perform 3 iterations of learning. If the change in the output of the artificial intelligence model during the 3 iterations of learning is below a predetermined level, it is determined that further learning is meaningless and learning is terminated.

[0063] That is, validation data can be used to determine whether the learning effect at each stage of iterative learning based on the artificial intelligence model is above or below a specified level, thus determining the completion of learning. The quantity and number of repetitions of the learning data and validation data mentioned above are merely examples, and the present invention is not limited thereto.

[0064] The input feature map generation device 100 can use test data to test the performance of more than one network function to determine whether to activate more than one network function, thereby generating an artificial intelligence model. The test data can be used to verify the performance of the artificial intelligence model and can constitute at least a portion of the learning data. For example, in the learning data, 70% can be effectively used for learning the artificial intelligence model (i.e., adjusting weights to output result values ​​similar to the labels), and 30% can be effectively used as test data to verify the performance of the artificial intelligence model. The input feature map generation device 100 can input the test data into the completed artificial intelligence model and measure the error to determine whether to activate the artificial intelligence model based on whether the performance exceeds a predetermined level.

[0065] The input feature map generation device 100 can use test data to verify the performance of the AI ​​model after it has finished learning. When the performance of the AI ​​model after it has finished learning is above a predetermined standard, the corresponding AI model can be activated for use in other applications.

[0066] Furthermore, when the performance of the AI ​​model that has finished learning falls below a predetermined standard, the input feature map generation device 100 can deactivate and discard the corresponding AI model. For example, the input feature map generation device 100 can judge the performance of the generated AI model based on factors such as accuracy, precision, and recall. The above performance evaluation criteria are merely examples, and the present invention is not limited thereto. According to one embodiment of the present invention, the input feature map generation device 100 can enable each AI model to learn independently to generate multiple AI models, and through performance evaluation, only AI models with performance above a preset level are used. However, this is not a limitation.

[0067] In various embodiments, the input feature map generation device 100 can be connected to the user terminal 200 via the network 400. Using the input feature map generated according to the input feature map generation method for an artificial intelligence model for detecting objects based on lidar, it can provide detection results of objects around the autonomous vehicle 10, or provide guidance (e.g., driving guidance, etc.) information based on the object detection results.

[0068] The user terminal 200 is a wireless communication device that ensures portability and mobility. It may include various handheld wireless communication devices such as navigation systems, personal communication systems (PCS), global system for mobile communications (GSM), personal digital cell (PDC), personal handyphone systems (PHS), personal digital assistants (PDAs), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), wireless broadband internet (Wibro) terminals, smartphones, smartpads, and tablet PCs. However, it is not limited to these devices. The user terminal 200 may also be an in-vehicle infotainment system installed in the vehicle 10.

[0069] Furthermore, Network 400 refers to a connection structure that enables information exchange between multiple terminals, servers, and other nodes. Examples of such networks include Local Area Networks (LANs), Wide Area Networks (WANs), the World Wide Web (WWW), wired / wireless data communication networks, telephone networks, and wired / wireless television communication networks. The wireless data communication network may include, but is not limited to, 3G, 4G, 5G, the 3rd Generation Partnership Project (3GPP), the 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, the Internet, Local Area Network (LAN), Wireless Local Area Network (Wireless LAN), Wide Area Network (WAN), Personal Area Network (PAN), Radio Frequency (RF), Bluetooth networks, Near-Field Communication (NFC) networks, satellite broadcasting networks, analog broadcasting networks, and Digital Multimedia Broadcasting (DMB) networks.

[0070] In one embodiment, an external server 300 can be connected to the input feature map generation device 100 via a network 400. The input feature map generation device 100 can store and manage the information / data required for the input feature map generation process of the artificial intelligence model for LiDAR-based object detection, as well as the information / data generated during the execution of the input feature map generation process of the artificial intelligence model for LiDAR-based object detection. Hereinafter, reference will be made to... Figure 2 The hard disk configuration of the input feature map generation apparatus 100, which performs an input feature map generation method for an artificial intelligence model for detecting objects based on lidar, is described.

[0071] Figure 2This is a hard disk structure diagram of an input feature map generation apparatus for an artificial intelligence model of object detection based on lidar, according to another embodiment of the present invention.

[0072] Reference Figure 2 According to another embodiment of the present invention, an input feature map generation apparatus 100 (hereinafter, "computing device 100") for an artificial intelligence model of object detection based on lidar may include: one or more processors 110; a memory 120 loaded with a computer program 151 executed by the processors 110; a bus 130; a communication interface 140; and a storage device 150 storing the computer program 151. Figure 2 Only structural elements relevant to embodiments of the present invention are illustrated. Therefore, those skilled in the art will recognize that, in addition to... Figure 2 In addition to the structural elements shown, other general structural elements may also be included.

[0073] Processor 110 controls the overall operation of each structure of computing device 100. Processor 110 includes a central processing unit (CPU), a microprocessor unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), or any type of processor known in the art of this invention.

[0074] Furthermore, the processor 110 is used to execute at least one application of the method according to an embodiment of the present invention or to perform calculations on which the application can be performed, and the computing device 100 may include more than one processor.

[0075] In various embodiments, the processor 110 may further include random access memory (RAM, not shown) and read-only memory (ROM, not shown) for temporarily and / or permanently storing signals (or data) processed internally by the processor 110. Furthermore, the processor 110 may be implemented as a system-on-chip (SoC) of at least one of the graphics processing unit, RAM, and ROM.

[0076] Memory 120 stores various data, instructions, and / or information. Memory 120 can load computer program 151 from storage device 150 to implement methods / actions according to various embodiments of the present invention. When computer program 151 is loaded into memory 120, processor 110 executes one or more instructions constituting computer program 151, thereby enabling the execution of the methods / actions. Memory 120 may be implemented as volatile memory, such as RAM, but the scope of the present invention is not limited thereto.

[0077] Bus 130 provides communication functionality between the structural elements of computing device 100. Bus 130 can be implemented as various types of buses, such as address bus, data bus, and control bus.

[0078] The communication interface 140 supports wired / wireless internet communication of the computing device 100. Furthermore, the communication interface 140 can also support various communication methods other than internet communication. Therefore, the communication interface 140 can be configured to include communication modules known in the art of this invention. In some embodiments, the communication interface 140 may be omitted.

[0079] Storage device 150 can non-temporarily store computer program 151. When the computing device 100 executes the input feature map generation process for an artificial intelligence model for LiDAR-based object detection, storage device 150 can store various information required to provide the input feature map generation process for the artificial intelligence model for LiDAR-based object detection.

[0080] Storage device 150 may be configured to include non-volatile memory, such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk, removable disk, or any form of computer-readable recording medium known in the art to which this invention pertains.

[0081] Computer program 151 may include one or more instructions that, when loaded into memory 120, cause processor 110 to execute methods / actions according to various embodiments of the present invention. That is, processor 110 can execute the methods / actions according to various embodiments of the present invention by executing the one or more instructions.

[0082] In one embodiment, computer program 151 may include one or more instructions to perform an input feature map generation method for an artificial intelligence model of object detection based on lidar, comprising the following steps: collecting lidar sensor data in the form of a three-dimensional point cloud for a predetermined region; processing the collected lidar sensor data based on one or more predefined metrics to generate one or more first feature maps; and combining the generated one or more first feature maps and second feature maps pre-generated based on distance and angle as separate channels to generate an input feature map for the artificial intelligence model.

[0083] The steps of the methods or algorithms described in conjunction with the embodiments of the present invention can be directly implemented as a hard disk, or as a software module executed by a hard disk, or a combination thereof. The software module can reside in random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or any form of computer-readable recording medium known in the art to which this invention pertains.

[0084] To be implemented in conjunction with a computer as a hard disk, the structural elements of this invention can be implemented as programs (or application programs) and stored in a medium. The structural elements of this invention can be implemented as software programming or software elements; similarly, embodiments can be implemented using programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as combinations of data structures, procedures, routines, or other programming configurations. Functionally, it can be implemented as an algorithm that executes on more than one processor. Hereinafter, reference will be made to... Figures 3 to 8 This describes a method for generating input feature maps for an artificial intelligence model based on lidar object detection, executed by the computing device 100.

[0085] Figure 3 This is a flowchart of an input feature map generation method for an artificial intelligence model for object detection based on lidar, according to another embodiment of the present invention.

[0086] Reference Figure 3In step S110, the computing device 100 can collect lidar sensor data for a predetermined area. For example, the computing device 100 can be connected to the autonomous vehicle 10 via network 400 and can receive collected lidar sensor data from lidar sensors installed in the autonomous vehicle 10. However, it is not limited to this.

[0087] The lidar sensor data can be in the form of a three-dimensional point cloud including multiple lidar sensor points with three-dimensional coordinate values, but is not limited to this.

[0088] Furthermore, the computing device 100 is described as collecting LiDAR sensor data to generate an input feature map, but is not limited thereto; any sensor data for detecting objects around the autonomous vehicle 10 may be used.

[0089] In step S120, through step S110, the computing device 100 can generate one or more first feature maps by processing the collected lidar sensor data.

[0090] In various embodiments, the computing device 100 can pre-process lidar sensor data in the form of a three-dimensional point cloud based on one or more predefined indicators (e.g., density (e.g., the number of lidar sensor points), maximum height, minimum height, average height, etc.), thereby generating a first feature map corresponding to each indicator. Hereinafter, refer to... Figure 4 and Figure 5 The method for generating the first feature map will be described in more detail below.

[0091] Figure 4 This is a flowchart describing a method for generating more than one first feature map in various embodiments. Figure 5 This diagram illustrates the process of generating more than one first feature map in various embodiments.

[0092] Reference Figure 4 and Figure 5In step S210, the computing device 100 can generate multiple grid pillars 21. For example, the computing device 100 can grid the lidar sensor data 20 in the form of a three-dimensional point cloud, including the XY plane, XZ plane, and XY plane, based on the XY plane, thereby generating multiple grid pillars 21. For example, the computing device 100 can use the gridded XY plane to divide the space of the lidar sensor data 20 in the form of a three-dimensional point cloud to include a grid of a predetermined size (e.g., 0.1m * 0.1m), thereby generating multiple grid pillars 21 with a bottom surface of the same size and shape as the grid of the predetermined size and a height of a predetermined size in the Z-axis direction. However, it is not limited to this, and the computing device 100 can be implemented in various forms, such as dividing the lidar sensor data 20 in the form of a three-dimensional point cloud based on the XZ plane or based on the YZ plane.

[0093] In step S220, the computing device 100 can calculate the value of one or more predefined indicators for each of the plurality of grid pillars 21 generated in step S210. For example, when one or more predefined indicators include the number (density) of LiDAR sensor points, the maximum height, the minimum height, and the average height, the computing device 100 can calculate the number of LiDAR sensor points, the maximum height, the minimum height, and the average height for each of the plurality of grid pillars 21.

[0094] One or more metrics are predefined by the user for detecting objects around the autonomous vehicle 10 and are described as the number (density) of LiDAR sensor points, maximum height, minimum height and average height, but are not limited to these; any metric used for object detection can be applied.

[0095] In step S230, the computing device 100 can use the index calculated in step S220 to generate a first feature map.

[0096] In various embodiments, the computing device 100 can generate a first feature map by inputting the calculated index value of each of the plurality of grid pillars 21 into a matrix corresponding to the lidar sensor data 20.

[0097] The matrix corresponding to the lidar sensor data 20 is a preset template. The first feature map is generated by inputting and arranging the index values ​​of each of the multiple grid pillars 21, and is implemented as an M-row N-column matrix. The number of rows and columns in the matrix can be determined according to the number and arrangement of the multiple grid pillars 21 (for example, the same shape as the XY plane (a plane that is meshed into multiple grids of a predetermined size) used when generating the multiple grid pillars 21).

[0098] In various embodiments, the computing device 100 inputs the indexes for the grid pillars 21 located in the first row and first column of the plurality of grid pillars 21 into the first row and first column of the matrix, and inputs the indexes for the grid pillars 21 located in the first row and second column into the first row and second column of the matrix, thereby generating a first feature map.

[0099] In various embodiments, the computing device 100 can generate a first feature map corresponding to each of the number of lidar sensor points, maximum height, minimum height, and average height as an independent channel, and combine the independently generated first feature maps to generate a three-dimensional first feature map. For example, as Figure 5 As shown, the computing device 100 can generate a total of four first feature maps based on the first feature map corresponding to the lidar sensor point, the first feature map corresponding to the maximum height, the first feature map corresponding to the minimum height, and the first feature map corresponding to the average height, that is, based on the number of calculated indicators, and perform channel combination on the four first feature maps, thereby generating a three-dimensional first feature map 30 with horizontal and vertical lengths of x and y and a height of 4.

[0100] Revisit Figure 3 In step S130, the computing device 100 combines one or more first feature maps generated in step S120 with each of the second feature maps pre-generated based on distance and angle into a separate channel, thereby generating the input feature map of the artificial intelligence model.

[0101] The second feature map, which is pre-generated based on distance and angle, can refer to the pre-generated and stored object detection regions so that the artificial intelligence model can derive feature maps based on distance and angle by taking into account the distance and angle of the LiDAR sensor points.

[0102] Typically, when lidar sensor data 20 is represented in X and Y space, it has characteristics based on distance and angle. For example, lidar sensor data 20 has the characteristic that the number (density) of lidar sensor points decreases rapidly with increasing distance. Furthermore, when occlusion occurs in X and Y space, lidar sensor data 20 is represented as a line and has the characteristic that the position of the obstacle changes according to the position of the origin reference coordinates.

[0103] On the other hand, existing artificial intelligence models perform position-independent actions, so the input feature map used for learning and inference of artificial intelligence models does not contain position information for each of the multiple grids (pixels), which limits the ability to take into account features such as distance and angle as described above.

[0104] With this in mind, the computing device 100 pre-generates and stores a second feature map related to distance and angle, enabling the artificial intelligence model to consider the distance and angle of multiple LiDAR sensor points and combine it with one or more first feature maps generated from the real-time collected LiDAR sensor data 20. That is, an input feature map including distance and angle information can be generated, making it possible to take into account the distance and angle features as described above. Hereinafter, referring to... Figures 6 to 8 This describes a method for generating a second feature map related to distance and angle, performed by a computing device 100.

[0105] Figure 6 This is a flowchart describing a method for generating a distance-related second feature map in various embodiments.

[0106] Reference Figure 6 In step S310, the computing device 100 can mesh a preset object detection area based on the XY plane to generate multiple meshes with a predetermined size.

[0107] The object detection area can be the area around the autonomous vehicle 10 sensed by the LiDAR sensor, or it can be an area with the same attributes (e.g., size, form) as the collection area of ​​the LiDAR sensor data 20 collected in step S110, but is not limited thereto.

[0108] Furthermore, the predetermined size may have the same size as the grid contained in the XY plane (which is gridded in a way that includes a grid of predetermined size) used to divide the space of the lidar sensor data 20 (e.g., 0.1m * 0.1m), but is not limited thereto.

[0109] In step S320, the computing device 100 can calculate the distance values ​​between the multiple grids generated in step S310 and the reference point.

[0110] In various embodiments, the computing device 100 can calculate the distance between the center point coordinates of each of the plurality of grids and the reference point coordinates (e.g., the origin coordinates), and standardize the calculated distance values ​​using a preset maximum distance value. For example, the computing device 100 can use the following mathematical formula 1 to calculate the standardized distance value (e.g., a first distance value) between the center point coordinates of each of the plurality of grids and the reference point coordinates.

[0111] Mathematical Formula 1

[0112]

[0113] Among them, D 1(ij)This can be a standardized distance value (e.g., the first distance value) between the grid located in row i and column j and the reference point, x ij and y ij The x and y coordinates of the grid center point located in row i and column j can be given. ref and y ref The x and y coordinates can be the reference point, and Dmax can be the preset maximum distance value. The reference point can be the origin, and its coordinates can be (0, 0), but are not limited to these.

[0114] That is, the computing device 100 calculates the distance between the reference point coordinates and the center point coordinates of each of the multiple grids, and divides the calculated distance value into a preset maximum distance value (e.g., 100m) for standardization, so that the distance between the reference point coordinates and the center point coordinates of each of the multiple grids can be a value in the range of 0 to 1.

[0115] In various embodiments, the computing device 100 can calculate the distance value between the center point coordinates and the reference point coordinates (e.g., the origin coordinates) of each of the plurality of grids, and can use a predefined logarithmic function to transform the calculated distance value. For example, the computing device 100 can use the following mathematical formula 2 to calculate the transformed distance value (e.g., a second distance value) between the center point coordinates and the reference point coordinates of each of the plurality of grids.

[0116] Mathematical formula 2

[0117]

[0118] Among them, D 2(ij) This can be the distance value (e.g., a second distance value) between the grid located in row i and column j and the reference point, x ij and y ij The x and y coordinates of the grid center point located in row i and column j, x ref and y ref These can be the x and y coordinates of a reference point. The reference point can be the origin, and its coordinates can be (0, 0), but are not limited to these.

[0119] That is, the computing device 100 calculates the distance between the coordinates of the reference point and the coordinates of the center point of each of the multiple grids, and transforms the calculated distance value by using a logarithmic function, thereby limiting the range of the input value.

[0120] In step S330, the computing device 100 can use the distance value calculated in step S320 to generate a second feature map related to distance.

[0121] In various embodiments, the computing device 100 may input the distance value calculated according to the method described above (e.g., a normalized distance value or a distance value transformed by a logarithmic function) into a matrix corresponding to a preset object detection region to generate a second feature map.

[0122] The matrix corresponding to the object detection region is a preset template for generating the second feature map as the distance values ​​between the reference point coordinates and the center point coordinates of each of the multiple grids are input and arranged. It can be implemented as an M-row N-column matrix. In this case, the matrix corresponding to the object detection region can be implemented in the same form as the matrix corresponding to the LiDAR sensor data 20, so that the second feature map related to distance and angle can be combined with the first feature map through channels, but it is not limited to this.

[0123] In various embodiments, such as Figure 8 As shown, in the computing device 100, the distance value D between the grid located in the second row and fifth column of multiple grids and the reference point is calculated. 25 (For example, the first distance value or the second distance value) is input into the second row and fifth column of the matrix, and the distance value D between the grid located in the fifth row and third column and the reference point is input. 53 The input is fed into the fifth row and third column of the matrix, which generates a second feature map related to distance.

[0124] In various embodiments, the computing device 100 can generate a second feature map of each of the first distance value and the second distance value calculated according to the above method as an independent channel, and combine multiple independently generated second feature maps to generate a distance-related three-dimensional second feature map 40.

[0125] Figure 7 This is a flowchart illustrating a method for generating an angle-related second feature map in various embodiments.

[0126] Reference Figure 7 In step S410, the computing device 100 can mesh a pre-defined object detection region based on the XY plane to generate multiple meshes of predetermined sizes. The multiple mesh generation actions performed by the computing device 100 can be implemented in conjunction with... Figure 6 The multiple mesh generation actions performed in step S310 are of the same form, but are not limited to this.

[0127] In step S420, the computing device 100 can calculate the angle values ​​between the multiple grids generated in step S410 and the reference point.

[0128] In various embodiments, the computing device 100 uses Euler angle transformation to convert the angles between multiple grids and a reference point into Euler angles, thereby calculating a first angle value. Known methods for converting angle values ​​to Euler angles using Euler angle transformation include techniques such as using a predefined transformation matrix (e.g., a transformation matrix for rotation with each of the X, Y, and Z axes as the rotation axis) to convert angle values ​​to Euler angles. These known techniques can be selectively applied, and this document does not specifically limit the method of using Euler angle transformation to convert angle values ​​to Euler angles.

[0129] In various embodiments, the computing device 100 can calculate the trigonometric ratio of the angle values ​​between multiple grids and a reference point as a second angle value. For example, the computing device 100 can calculate the sine and cosine values ​​of the angles between multiple grids and the reference point. That is, the computing device 100 calculates the sine and cosine values ​​of the angles between multiple grids and the reference point to generate a second feature map related to the angles, thereby limiting the range of calculated angle values ​​to values ​​within the range of -1 to 1, and solving the error problem of periodic distances in Eulerian space.

[0130] In step S430, the computing device 100 can use the angle value calculated in step S420 to generate a second feature map related to the angle.

[0131] In various embodiments, the computing device 100 may input angle values ​​(e.g., a first angle value and a second angle value) calculated according to the method described above into a matrix corresponding to a preset object detection region to generate a second feature map. For example, as Figure 8 As shown, the computing device 100 calculates the angle value (θ) between the grid located in the second row and fifth column of the multiple grids and the reference point. 25 Input the value into the second row and fifth column of the matrix, and input the angle θ between the grid located in the fifth row and third column and the reference point. 53 The input is placed into the fifth row and third column of the matrix, thereby generating a second feature map related to the angle.

[0132] At this time, in the computing device 100, when the angle value calculated according to the above method has two or more second angle values ​​between multiple grids and reference points, that is, when the angle value calculated according to the above method uses the trigonometric ratio calculated between multiple grids and reference points as a sine value and a cosine value, the second feature map corresponding to the sine value and the second feature map corresponding to the cosine value are generated as separate channels respectively.

[0133] In various embodiments, the computing device 100 can generate a second feature map of each of the first angle value and the second angle value calculated according to the above method as an independent channel, and combine the multiple independently generated second feature maps to generate a three-dimensional second feature map 40 related to the angle.

[0134] That is, the computing device 100 has the advantage that by combining the second feature map related to distance, the second feature map related to angle, and the first feature map generated by real-time collection of lidar sensor data in a channel to generate the input feature map of the artificial intelligence model, it is possible to generate an input feature map that takes into account the distance and angle related features of lidar sensor data.

[0135] Furthermore, the computing device 100 has the advantage that, according to the method described above (for example, Figure 6 and Figure 7 Using the object detection area as the object, a second feature map related to distance and a second feature map related to angle are pre-generated. Subsequently, multiple first feature maps and the same second feature map are combined to effectively generate the input feature map. The multiple first feature maps are generated by processing LiDAR sensor data collected at a preset period or in real time.

[0136] The above-described method for generating input feature maps for an artificial intelligence model based on LiDAR object detection is described with reference to the flowcharts shown in the accompanying drawings. For brevity, the method for generating input feature maps for an artificial intelligence model based on LiDAR object detection is described using a series of illustrated block diagrams. However, the invention is not limited to the order of the blocks, and some blocks can be executed in a different order or simultaneously than those illustrated and described herein. Furthermore, new blocks not described herein and in the accompanying drawings can be added, or some blocks can be executed with altered states.

[0137] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that the invention can be implemented in other specific forms without altering its technical spirit or essential characteristics. Therefore, it should be understood that the above embodiments are exemplary in all respects and not restrictive.

Claims

1. A method for generating input feature maps for an artificial intelligence model based on lidar object detection, the method being executed by a computing device, wherein, The method includes: Steps for collecting lidar sensor data in the form of 3D point clouds for a predetermined area; The steps of processing the collected lidar sensor data based on one or more predefined metrics to generate one or more first feature maps; and The step of generating an input feature map for an artificial intelligence model by combining one or more first feature maps generated and second feature maps pre-generated based on distance and angle as separate channels.

2. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 1, wherein, The step of generating one or more first feature maps includes: The step of meshing the collected lidar sensor data based on the XY plane to generate multiple mesh pillars; The steps of calculating one or more predefined indices for each of the generated plurality of grid columns; and The step of inputting one or more calculated indicators into a matrix corresponding to the collected lidar sensor data to generate one or more first feature maps.

3. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 2, wherein, The one or more predefined indicators include the number of LiDAR sensor points, maximum height, minimum height, and average height. The step of generating one or more first feature maps includes: The step of generating a first feature map as an independent channel corresponding to each of the number of lidar sensor points, the maximum height, the minimum height, and the average height.

4. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 1, wherein, The method further includes: The steps of meshing a preset object detection area based on the XY plane to generate multiple meshes with a predetermined size; The steps of generating a distance-related second feature map based on the distances between the generated multiple grids and the reference point; and The step of generating a second feature map related to the angles between the generated multiple grids and the reference point.

5. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 4, wherein, The step of generating the distance-related second feature map includes: The step of calculating the distance between the center point coordinates of each of the generated multiple grids and the reference point coordinates; The step of standardizing the calculated distance value using a preset maximum distance value; and The step of inputting the standardized distance value into a matrix corresponding to the preset object detection region to generate the distance-related second feature map.

6. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 4, wherein, The step of generating the distance-related second feature map includes: The step of calculating the distance between the center point coordinates of each of the generated multiple grids and the reference point coordinates; The steps of transforming the calculated distance values ​​using a predefined logarithmic function; and The step of inputting the transformed distance value into a matrix corresponding to the preset object detection region to generate the distance-related second feature map.

7. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 4, wherein, The step of generating the angle-related second feature map includes: The step of calculating the angle between the line used to connect the center point of each of the generated multiple meshes to the reference point and the X-axis or Y-axis; The steps of using Euler angle transformation to convert the calculated angle values; and The step of inputting the converted angle value into a matrix corresponding to the preset object detection region to generate the angle-related second feature map.

8. The method for generating input feature maps for an artificial intelligence model based on lidar object detection according to claim 4, wherein, The step of generating the angle-related second feature map includes: The step of calculating the angle between the line used to connect the center point of each of the generated multiple meshes to the reference point and the X-axis or Y-axis; The steps of calculating the trigonometric ratio for the calculated angle value; and The step of inputting the calculated trigonometric ratio into a matrix corresponding to the preset object detection region to generate two angle-related second feature maps.

9. An input feature map generation device for an artificial intelligence model based on lidar object detection, wherein, The device comprises: processor; Network interface; Memory; and The computer program is loaded into the memory and executed by the processor. The computer program includes: Instructions to collect lidar sensor data in the form of 3D point clouds for a predetermined area; Instructions for processing the collected lidar sensor data based on one or more predefined metrics to generate one or more first feature maps; and Instructions for generating input feature maps for an artificial intelligence model are given by combining one or more first feature maps generated and second feature maps pre-generated based on distance and angle as separate channels.

10. A computer program product recorded on a computer-readable recording medium, comprising a computer program that performs the following steps: Steps for collecting lidar sensor data in the form of 3D point clouds for a predetermined area; The steps of processing the collected lidar sensor data based on one or more predefined metrics to generate one or more first feature maps; and The step of generating an input feature map for an artificial intelligence model by combining one or more first feature maps generated and second feature maps pre-generated based on distance and angle as separate channels.

Images