Method for domain generalization for three-dimensional object segmentation

Abstract

According to an embodiment of the present disclosure, a computer program stored in a computer-readable storage medium is disclosed. The computer program performs the following methods for recognizing an object in a three-dimensional space. The method may comprise the steps of: performing voxelization of point cloud data obtained from a LiDAR sensor and converting the voxelized point cloud data into a sparse tensor; encoding the sparse tensor by using a backbone encoder to generate a latent feature; generating a semantic segmentation map by using a semantic decoder on the basis of the latent feature; generating a density map by using a density decoder on the basis of the latent feature; and training at least one of the backbone encoder, the semantic decoder, and the density decoder.

Description

Domain generalization methods for 3D object segmentation

[0001] The present disclosure relates to a technology for object recognition in three-dimensional space for point cloud data acquired from a LiDAR sensor, and more specifically, to provide consistent object recognition performance in LiDAR sensors of various performances.

[0002] LiDAR sensors are widely used for 3D object recognition in various applications, such as autonomous vehicles, robots, and drones. LiDAR sensors provide 3D information about the surrounding environment as point cloud data, and processing this data to recognize objects is a key task.

[0003] Point cloud data can contain various feature information, such as laser regression values, along with the location information of each point. While this data enables high-resolution, precise object recognition, its density can vary significantly depending on the environment and sensor settings, making it difficult to maintain consistent performance. In particular, object recognition accuracy can degrade significantly in low-density environments.

[0004] Existing 3D object recognition methods have primarily focused on performance optimization in fixed environments, but have limited ability to address performance degradation in environments with varying density. To address this issue, this disclosure proposes a semantic segmentation method that reflects density information from LiDAR data. This method aims to provide consistent object recognition performance across diverse environments and enhance its applicability, particularly in fields such as autonomous vehicles and robotics.

[0005] The present disclosure is conceived based on the aforementioned background technology, and enables consistent 3D object recognition to be performed even in environments where the density of point cloud data acquired from a lidar sensor varies.

[0006] According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. The computer program performs the following methods for recognizing an object in a three-dimensional space, the methods including: converting point cloud data acquired from a lidar sensor into a sparse tensor by voxelizing the data; encoding the sparse tensor through a backbone encoder to generate latent features; generating a semantic segmentation map through a semantic decoder based on the latent features; generating a density map through a density decoder based on the latent features; and learning at least one of the backbone encoder, the semantic decoder, and the density decoder.

[0007] Alternatively, the step of converting the point cloud data into a sparse tensor by voxelizing the point cloud data may include the step of dividing the point cloud data into grids of a predetermined size and converting points within each grid into one voxel.

[0008] Alternatively, the size of the grid may be variably set depending on at least one of the resolution of the lidar sensor or the application environment.

[0009] Alternatively, the step of converting the point cloud data into a sparse tensor by voxelizing the point cloud data may include the step of generating voxels at spatially regular intervals, and representing a plurality of points within the same voxel as a single voxel by averaging them.

[0010] Alternatively, the averaging of the point cloud data may also average feature information including position information, reflection intensity, or laser regression value of each point within each voxel.

[0011] Alternatively, the step of converting the point cloud data into a sparse tensor by voxelizing it may further include a filtering step of removing or merging voxels below a certain density while considering continuity with surrounding voxels during the voxelization process.

[0012] Alternatively, the sparse tensor can store only non-empty voxels to optimize memory usage and computation.

[0013] Alternatively, the step of encoding the sparse tensor through a backbone encoder to generate latent features may include the step of generating a sparse voxel feature including the spatial location and feature vector of each voxel of the sparse tensor.

[0014] Alternatively, the step of encoding the sparse tensor through a backbone encoder to generate a latent feature may include the step of generating kernel occupancy information (Kernel Occupancy Binary Vector) based on the spatial location of each voxel, converting the kernel occupancy information into a feature vector through 1×1 convolution, and combining the feature vector with the sparse voxel feature to generate the latent feature.

[0015] Alternatively, the step of encoding the sparse tensor through the backbone encoder to generate latent features may include the step of performing operations on the sparse voxel features and the feature vector through a density-aware sparse convolution residual block, and residually concatenating the results of the operations to generate final latent features.

[0016] Alternatively, the density-aware sparse convolution residual block can process the input feature vector through 3*3*3 sparse convolution, compress the feature vector through 1*1 sparse convolution, and then residually concatenate the vectors to generate the final latent feature.

[0017] Alternatively, the backbone encoder may be characterized by including a plurality of density-aware sparse convolution residual blocks, each block stepwise enhancing features of an input sparse tensor to generate latent features.

[0018] Alternatively, the semantic decoder can receive the latent features as input and predict class labels at the voxel level to generate a semantic segmentation map.

[0019] Alternatively, the semantic decoder may include multiple decoding layers, each of which may progressively restore the spatial resolution of latent features to ultimately generate a semantic segmentation map.

[0020] Alternatively, the density decoder can receive the latent feature as input and predict the density at the voxel level to generate a density map.

[0021] Alternatively, the density decoder may include multiple decoding layers, each of which may progressively refine density information of latent features to ultimately generate a density map.

[0022] Alternatively, the step of training at least one of the backbone encoder, the semantic decoder, and the density decoder may include the step of receiving the latent feature of the backbone encoder as an input and generating outputs of each of the semantic decoder and the density decoder, wherein the output of the semantic decoder is trained using a cross entropy loss function and a Lovasz-Softmanx loss function, and the output of the density decoder is trained using a Smooth-L1 loss function.

[0023] Alternatively, the step of learning at least one of the backbone encoder, the semantic decoder and the density decoder may include a step of optimizing the backbone encoder through a joint learning method by taking into account a correlation between the output of the semantic decoder and the output of the density decoder.

[0024] Alternatively, the joint learning may include adjusting the weights of the loss functions during the learning process to achieve a balance between the cross-entropy loss function for the output of the semantic decoder and the Smooth-L1 loss function for the output of the density decoder.

[0025] Alternatively, the step of learning at least one of the backbone encoder, the semantic decoder and the density decoder may include a step of learning by applying independent learning rates to each of the backbone encoder, the semantic decoder and the density decoder.

[0026] Alternatively, the method may further include an inference step of recognizing an object in a three-dimensional space using the learned backbone encoder and the learned semantic decoder.

[0027] Alternatively, a method for recognizing an object in a three-dimensional space is disclosed. The method may include the steps of: voxelizing point cloud data acquired from a lidar sensor and converting it into a sparse tensor; encoding the sparse tensor through a backbone encoder to generate latent features; generating a semantic segmentation map through a semantic decoder based on the latent features; generating a density map through a density decoder based on the latent features; and training at least one of the backbone encoder, the semantic decoder, and the density decoder.

[0028] Alternatively, a computer device is disclosed. The computer device includes one or more processors; and a memory storing instructions executable by the one or more processors; wherein the one or more processors are configured to voxelize point cloud data acquired from a lidar sensor and convert it into a sparse tensor, encode the sparse tensor through a backbone encoder to generate latent features, generate a semantic segmentation map through a semantic decoder based on the latent features, generate a density map through a density decoder based on the latent features, and train at least one of the backbone encoder, the semantic decoder, and the density decoder.

[0029] The present disclosure can enable consistent 3D object recognition even in environments where the density of point cloud data acquired from a lidar sensor varies.

[0030] This disclosure proposes a semantic segmentation method that can adapt to density fluctuations in lidar data, thereby providing object recognition performance in a variety of environments. This can enhance 3D object recognition performance in applications such as autonomous vehicles, robots, and drones.

[0031] FIG. 1 is a block diagram of a computer device that performs object recognition in three-dimensional space according to one embodiment of the present disclosure.

[0032] FIG. 2 is a schematic diagram showing a process of processing lidar data for object recognition in three-dimensional space according to one embodiment of the present disclosure.

[0033] FIG. 3 is a diagram visually illustrating a density calculation process for point cloud data of one embodiment of the present disclosure.

[0034] FIG. 4 is a general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0035] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

[0036] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0037] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0038] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0039] And, the term “at least one of A or B” should be interpreted to mean “if it includes only A”, “if it includes only B”, or “if it is combined in the composition of A and B”.

[0040] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0041] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments set forth herein. The present disclosure is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0042] In the present disclosure, network function, artificial neural network, artificial neural network network, and neural network may be used interchangeably.

[0043] FIG. 1 is a block diagram of a computer device that performs depth estimation according to one embodiment of the present disclosure.

[0044] The configuration of the computer device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computer device (100) may include other configurations for performing a computer environment, and only some of the disclosed configurations may constitute the computer device (100).

[0045] A computer device (100) may include a processor (110), a memory (130), and a network unit (150). The processor (110) may be composed of one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of the computer device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) can process network function learning. For example, the CPU and GPGPU can jointly process network function learning and data classification using the network function. Furthermore, in one embodiment of the present disclosure, processors of multiple computer devices can be used together to process network function learning and data classification using the network function. Furthermore, a computer program executed in a computer device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0046] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0047] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computer device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the Internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

[0048] The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0049] In addition, the network unit (150) presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0050] In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth.

[0051] Through the network portion (150) of the present disclosure, the computer device (100) can communicate with other computer devices, such as a data storage device storing data, a cloud data storage device, a cloud computer system for utilizing computing power, etc. The techniques described herein can be used not only in the networks mentioned above, but also in other networks.

[0052] FIG. 2 is a schematic diagram showing a process of processing lidar data for object recognition in three-dimensional space according to one embodiment of the present disclosure.

[0053] A computer device (100) can acquire point cloud data (10) from a lidar sensor. The point cloud data (10) acquired from the lidar sensor consists of individual points in three-dimensional space. This point cloud data (10) is voxelized and converted into a sparse tensor composed of voxels (11). The voxelized data is then input to a backbone encoder (200) as a sparse tensor.

[0054] A computer device (100) can process point cloud data (10) acquired via a LiDAR sensor and convert it into a voxelized sparse tensor. This step effectively structures the point cloud data and prepares basic data required for subsequent object recognition and semantic segmentation.

[0055] The computer device (100) divides the point cloud data (10) into grids of a certain size and converts the points within each grid into voxels (11). In this process, the point cloud data is converted into a discontinuous three-dimensional space, and the voxelized data facilitates subsequent processing. Referring to FIG. 2, the process of reconstructing the point cloud data (10) within a grid of a certain size through the voxelization step (11) can be visually confirmed. Here, the size of the grid can be variably set according to at least one of the resolution of the lidar sensor or the application environment. For example, the performance of the computer device that will process the lidar sensor data, the task to which the lidar sensor is applied, etc. can be examples of the application environment.

[0056] The voxelization process is a method that reduces the size and complexity of data while retaining important spatial information. It allows for variable grid sizes to adapt to various resolutions and environments. During this process, multiple points within a voxel are averaged or considered representative based on specific criteria, transforming them into features representing each voxel. However, in actual point cloud data, many grid cells may be empty, meaning they lack data.

[0057] To handle this, the computer device (100) can convert voxelized point cloud data into a sparse tensor. After voxelizing (11) the point cloud data (10), the computer device (100) can generate a sparse tensor by storing only non-empty voxels. By storing only voxels that actually have values ​​among the voxelized data, memory efficiency can be increased and computational efficiency can be achieved.

[0058] In the process of converting point cloud data acquired from a LiDAR sensor into a sparse tensor by voxelizing it, the computer device (100) can represent a plurality of points included in each voxel by averaging them into one voxel.

[0059] This process is necessary to efficiently process voxelized data while maintaining its accuracy. Point cloud data can be spatially irregularly distributed, and points within each voxel may have different locations and feature information. Therefore, the representative value of a voxel is calculated by averaging the locations of points within the voxel and their corresponding feature information. This representative value is then converted into a sparse tensor and utilized for semantic segmentation and other object recognition tasks.

[0060] For example, when multiple points exist within the same voxel, the location information of each point can be averaged to determine the voxel's center location. Furthermore, other feature information, such as reflection intensity or laser regression values, can also be averaged and used as a representative value for the voxel. This process reduces data noise and yields more accurate object recognition results.

[0061] When performing averaging of points within a voxel in voxelized point cloud data, the computer device (100) averages not only position information but also feature information such as reflection intensity or laser regression value.

[0062] The positional information for points within a voxel is based on the 3D coordinates of each point, which are averaged to calculate the center position of the voxel. Simultaneously, feature information, such as the reflectance intensity or laser retracement value of each point, is averaged within the voxel and used as a representative value for the voxel. This averaging process improves the quality of the voxelized data and provides more accurate results in object recognition.

[0063] This process is particularly useful when points within a voxel are densely packed, and it increases the reliability of object recognition by combining various feature information to generate a representative value for each voxel. Through this averaging process, the voxelized data is converted into a sparse tensor and passed on to subsequent processing steps.

[0064] During the voxelization process, the computer device (100) may perform filtering to remove or merge voxels below a certain density, taking into account continuity with surrounding voxels.

[0065] This filtering step removes unnecessary voxels from voxelized data, retaining only data with continuity. Voxels below a certain density may lack continuity with surrounding voxels or be judged to have low data reliability. These voxels are removed or merged with adjacent voxels to maintain data consistency.

[0066] The filtering process improves the quality of voxelized data, optimizes memory usage, and enables efficient utilization of computational resources. By removing empty or unnecessary voxels, the data converted to sparse tensors can perform better in subsequent object recognition tasks.

[0067] The computer device (100) stores only non-empty voxels when constructing a sparse tensor to optimize memory usage and computation.

[0068] Point cloud data consists of a very large number of points, but meaningful data for object recognition is concentrated in a few voxels. Therefore, storing empty voxels can waste memory. To prevent this, sparse tensors store only non-empty voxels, minimizing memory usage and efficiently utilizing computational resources.

[0069] This sparse tensor structure accelerates the processing of voxelized data and plays a crucial role in improving performance in object recognition and segmentation tasks. This approach is an essential optimization technique, especially for processing large-scale point cloud data.

[0070] Below, we will explain sparse tensors in detail.

[0071] sparse tensor can be defined by the following mathematical formula 1.

[0072]

[0073] Here is the location of the ith voxel, is the feature vector of the ith voxel. Is , which means a set of three-dimensional coordinates of all voxels, Is , which may be a set of feature vectors of voxels. Here, M may be a set of feature vectors of non-empty voxels.

[0074] Equation 2 defines the components of a sparse tensor.

[0075]

[0076] Here, P is a set of coordinates representing the voxel location in 3D space, and F consists of a C-dimensional feature vector corresponding to each voxel. This allows the sparse tensor to effectively store location information in 3D space and feature information corresponding to each location.

[0077] The computer device (100) can generate latent features by encoding the sparse tensor through a backbone encoder (200). The backbone encoder (200) encodes the input sparse tensor and generates latent features (620) for each voxel.

[0078] The backbone encoder (200) first generates sparse voxel features (510) for each voxel of the sparse tensor. These features may include location information of the voxel and a corresponding feature vector.

[0079] The backbone encoder (200) retrieves kernel occupancy information (530) based on sparse voxel features (510). Kernel occupancy information represents the relationship between each voxel and its neighboring voxels, and this information generates a more sophisticated feature vector for each voxel. This process incorporates the density information of the sparse tensor, resulting in more precise encoding results.

[0080] The computer device (100) processes voxelized point cloud data converted into sparse tensors through a backbone encoder, generating kernel occupancy information reflecting the spatial relationship of each voxel and a feature vector derived therefrom. This process plays a key role in improving the performance of object recognition and semantic segmentation.

[0081] Kernel occupancy information reflects the density and spatial continuity of voxelized point cloud data, indicating the spatial relationship each voxel has with its neighboring voxels. This allows computer programs to efficiently encode information about the location and density of each voxel.

[0082] The computer device (100) sets a kernel area for each voxel in voxel data expressed as a sparse tensor, and searches for neighboring voxels existing within the area to generate kernel occupancy information.

[0083] This process is defined by mathematical equation 3.

[0084]

[0085] Here, represents the area of ​​the kernel, and K represents the size of the kernel. The computer device (100) determines the occupancy status of each neighboring voxel based on the distance between the center voxel and neighboring voxels within this kernel area.

[0086] Kernel Occupancy Information (530) is defined by mathematical formula 4.

[0087]

[0088] Here represents the location of the center voxel, represents the location of neighboring voxels, and P is a set of non-empty voxels.

[0089] Finally, kernel occupancy information can be flattened and converted into a one-dimensional vector. is flattened It can be expressed as . Through this, the original multidimensional occupancy information can be organized into a single vector and efficiently processed in a computer device (100).

[0090] This formula indicates whether a neighboring voxel exists within the kernel region, and is represented as 1 if the voxel exists, and 0 otherwise. This explicitly encodes the occupancy state within the kernel region.

[0091] The kernel occupancy information generated in this way reflects the spatial relationship between voxels and includes information about the density of each voxel and the surrounding environment.

[0092] The computer device (100) can generate kernel occupancy information (Kernel Occupancy Binary Vector) based on the spatial location of each voxel, convert the kernel occupancy information into a feature vector through 1×1 convolution, and combine the feature vector with the sparse voxel feature to generate the latent feature.

[0093] The generated kernel occupancy information is converted into a multidimensional feature vector through a 1x1 convolution operation. This process transforms the kernel occupancy information from simple binary information to a complex feature that reflects the density and spatial relationships of each voxel.

[0094] Generated kernel occupancy information is a multidimensional feature vector (kernel occupancy binary vector (540)) through a 1x1 convolution operation. ) is converted into a complex feature that reflects the density and spatial relationship of voxels, rather than a simple binary information.

[0095] Feature vectors combined with the feature vectors of the sparse tensor is defined by mathematical expression 5 using the kernel occupancy binary vector.

[0096]

[0097] represents a 1x1 convolution operation, through which the kernel occupancy information of each voxel is converted into a multidimensional feature vector. This feature vector plays an important role in subsequent object recognition, and reflects the location and density information of the voxel.

[0098] The combined feature vector is a sparse tensor It is stored in the form of , and sparse convolution is applied to generate an output sparse tensor. Afterwards, it is calculated into more sophisticated latent features through a density-aware sparse convolution residual block (600).

[0099] During sparse convolution operations, certain voxels may receive more weight depending on kernel occupancy information, which can lead to computational imbalance. To mitigate this imbalance, the paper proposes an output feature vector A process of normalizing may be required. This can be defined by Equation 6.

[0100]

[0101] is the output feature vector of a specific voxel i, represents the voxels within the kernel region to which the voxel belongs. represents the occupancy status of neighboring voxels and means the number of voxels that actually exist within the kernel.

[0102] This equation adjusts the output feature vector for each voxel based on how occupied its neighboring voxels are. In other words, if a particular voxel has more neighboring voxels, its output vector is adjusted according to the number of neighboring voxels, maintaining consistency throughout the entire computation.

[0103] A computer device (100) processes point cloud data to recognize a three-dimensional object, and generates features reflecting the density and spatial correlation of voxel data by utilizing a density-aware sparse convolution residual block (600). This process includes operations on sparse voxel features (510) and feature vectors (540) of each voxel, and ultimately applies a residual connection technique to generate latent features (620).

[0104] The density-aware sparse convolution residual block (600) receives the combined feature vector as input and performs a convolution operation that considers the spatial correlation between voxels. This process includes normalizing the feature vector to mitigate imbalances between convolution layers. This results in more consistent computational results for each voxel and enhances sensitivity to density differences.

[0105] A density-aware sparse convolution residual block (600) performs an operation on each voxel of voxel data expressed as a sparse tensor. This block extracts unique features of the voxel data through a sparse convolution operation based on spatial relationships between voxels and density information.

[0106] The density-aware sparse convolution residual block includes the following main computational steps:

[0107] 1×1 convolution (602): A 1×1 convolution is applied to convert kernel occupancy information into a multidimensional feature vector.

[0108] 3×3×3 Sparse Convolution (603, 606): A 3×3×3 sparse convolution operation is applied to reflect spatial relationships between voxels. This process integrates spatial information that each voxel can obtain from its relationships with its neighbors, thereby richly reflecting the density and location information of the voxels.

[0109] Vector multiplication (604, 607): An operation is performed to calculate the correlation between voxels by combining the spatial location of the voxel and the feature vector.

[0110] Voxel-wise Summation (601): The computational results obtained from each voxel are combined and the computational results between voxels are combined.

[0111] The feature vectors obtained from the density-aware sparse convolution residual block are combined with existing input features to form residual connections. Residual connections are a technique that directly connects information between input and output, minimizing information loss that may occur during computation and improving learning speed.

[0112] The backbone encoder (200) of the present disclosure includes a plurality of density-aware sparse convolution residual blocks (600) to gradually enhance the features of an input sparse tensor, thereby ultimately generating unique latent features.

[0113] The backbone encoder (200) receives location information and feature information of each voxel of the input sparse tensor T=(P,F), and sequentially processes them to generate high-dimensional latent features. The backbone encoder is composed of multiple density-aware sparse convolution residual blocks (600), and each block processes the input feature vector to derive progressively enhanced features.

[0114] The backbone encoder gradually enhances the features of the input sparse tensor by passing it through a series of density-aware sparse convolution residual blocks.

[0115] Each density-aware sparse convolution residual block (600) performs sparse convolution on the input feature vector to extract features reflecting the density and spatial characteristics of the voxel.

[0116] These feature vectors are progressively strengthened as they pass through each block, realizing cumulative strengthening of information by using the output of the previous block as the input of the next block.

[0117] Multiple density-aware sparse convolution residual blocks included in the backbone encoder further refine voxel features through computations for each block.

[0118] The first block processes the sparse tensor provided as initial input to extract basic spatial features, taking into account density and correlation between voxels.

[0119] Blocks after the second block accept features generated from the previous block as input and use these to analyze deeper relationships between voxels. With each block, the feature vectors become increasingly sophisticated, evolving into high-dimensional features.

[0120] The semantic decoder (400) of the present disclosure receives as input the latent features (620) generated through the backbone encoder (200), predicts a class label in voxel units, and generates a semantic segmentation map.

[0121] The semantic decoder determines the class of each voxel based on the spatial and contextual information contained in the latent features. This class label prediction process enables the distinction of various objects and plays a crucial role in performing voxel-level semantic segmentation based on 3D point cloud data, such as LiDAR data.

[0122] The semantic decoder performs object recognition or segmentation tasks on each voxel, and the results are output as a semantic segmentation map (410) containing the location of each voxel within a three-dimensional space and corresponding class information. This map visualizes the segmentation of objects on a voxel-by-voxel basis, thereby providing an accurate understanding of the surrounding environment.

[0123] The semantic decoder (400) of the present disclosure includes multiple decoding layers, each layer gradually restoring the spatial resolution of latent features to ultimately generate a semantic segmentation map.

[0124] Each decoding layer processes the input latent features, gradually enhancing detailed information while restoring their resolution. Early layers analyze the overall structure and characteristics of the latent features, while subsequent layers restore finer spatial resolution and detailed class information.

[0125] These decoding layers are hierarchically structured within the semantic decoder, and with each layer, the spatial and content information of the voxels is refined more clearly. The final generated semantic segmentation map (410) includes class labels for each voxel, enabling object recognition and classification within three-dimensional space.

[0126] The density decoder (300) of the present disclosure receives as input the latent feature (620) generated through the backbone encoder (200), predicts the density of voxels, and generates a density map.

[0127] The density decoder analyzes the density of point cloud data within a voxel based on input latent features to predict density information for each voxel. This analysis is particularly important for point cloud data such as LiDAR data, and allows for more accurate 3D object recognition by accounting for density differences between voxels.

[0128] The density decoder uses the predicted voxel density information to generate a density map (310), which contains density values ​​for each voxel location. This density map is used in conjunction with the semantic segmentation map to further improve object recognition accuracy and reduce distortion caused by density differences between voxels.

[0129] The density decoder (300) of the present disclosure includes a plurality of decoding layers, each of which gradually refines density information of latent features to ultimately generate a density map.

[0130] Each decoding layer analyzes density information between voxels based on input latent features, and based on this, progressively refines the density of point cloud data within voxels. Early layers identify overall density trends, while subsequent layers extract more detailed density information, ultimately generating a sophisticated density map.

[0131] In this process, each layer receives the output of the previous layer as input, progressively refines density information, and enables more distinct density differences between voxels. The final generated density map (310) contains predicted density values ​​for each voxel, which serves as important reference information in the 3D object recognition process.

[0132] The step of learning at least one of the backbone encoder (200), the semantic decoder (400) and the density decoder (300) of the present disclosure receives the latent feature (620) generated from the backbone encoder as input, generates outputs of the semantic decoder and the density decoder respectively, and performs learning by applying various loss functions based on the outputs.

[0133] First, the backbone encoder (200) generates latent features from the input sparse tensor. These latent features are high-dimensional representations that comprehensively reflect the spatial and content characteristics of voxel data, and play an important role in the subsequent learning process.

[0134] The latent features generated from the backbone encoder are input to the semantic decoder (400) and the density decoder (300), respectively.

[0135] The semantic decoder predicts voxel-level class labels based on latent features to generate a semantic segmentation map. In this process, the cross-entropy loss function and the Lovasz-Softmax loss function are used to improve the accuracy of the output semantic segmentation map.

[0136] Cross-Entropy Loss: Used to improve prediction accuracy by measuring the difference between the predicted class and the actual class of each voxel.

[0137] Lovasz-Softmax loss function: This loss function is used to improve accuracy in semantic segmentation and is particularly effective in solving class imbalance problems. It helps accurately predict voxel boundaries and minority classes in the segmentation map.

[0138] The density decoder generates a density map by predicting voxel-level density based on latent features. The Smooth-L1 loss function is used in this process.

[0139] Smooth-L1 loss function: This function is used to reduce the difference between the predicted and actual density values ​​of a voxel, and it enables stable learning by balancing large and small errors. This is effective when processing point cloud data with large density differences.

[0140] The step of learning at least one of the backbone encoder (200), the semantic decoder (400) and the density decoder (300) of the present invention includes a process of optimizing the backbone encoder through a joint learning method that takes into account the correlation between the outputs of the semantic decoder and the density decoder.

[0141] In the task of recognizing and classifying 3D objects by processing LiDAR data, the semantic decoder and density decoder are responsible for semantic segmentation and density estimation, respectively. These two tasks are highly interrelated, and semantic segmentation results can vary in areas with high or low density. Therefore, rather than training these two tasks separately, joint learning, which considers the correlation between the two tasks, is advantageous for producing more consistent results.

[0142] In the present invention, since both the semantic decoder and the density decoder receive latent features from the same backbone encoder, it is crucial to optimize the relationship between these two outputs. In a joint learning approach, the outputs of the semantic decoder and the density decoder are simultaneously considered to optimize the parameters of the backbone encoder.

[0143] The correlation between the outputs of the semantic decoder and the outputs of the density decoder provides important feedback during the training process of the backbone encoder, which enables the backbone encoder to produce more consistent feature vectors.

[0144] In the joint learning process of the present invention, the weights of the loss functions are adjusted during the learning process to achieve a balance between the cross-entropy loss function for the output of the semantic decoder and the Smooth-L1 loss function for the output of the density decoder.

[0145] Semantic decoders and density decoders perform different tasks, and loss functions tailored to their characteristics are used. However, if the loss functions for these two tasks are not appropriately balanced, the learning process can overweight certain tasks, hindering optimal learning results.

[0146] The cross-entropy loss function is used to evaluate the output of the semantic decoder and is a loss function to improve the accuracy in predicting the class label of a voxel.

[0147] The Smooth-L1 loss function is used to evaluate the output of the density decoder, and is a loss function that balances large and small errors in the density prediction of voxels.

[0148] The process of adjusting the weights of these loss functions helps to optimize the outputs of the semantic decoder and the density decoder in a balanced manner during the learning process.

[0149] Loss function weight adjustments can be applied from the early stages of learning and dynamically adjusted as learning progresses. These weight adjustments appropriately reflect the correlation between semantic segmentation and density estimation, particularly in complex 3D data, thereby improving overall learning performance.

[0150] In the present invention, learning is performed by applying independent learning rates to each of the backbone encoder (200), the semantic decoder (400), and the density decoder (300).

[0151] The backbone encoder and two decoders perform different functions, which may result in different optimization speeds and ranges required during the training process. By applying independent learning rates, each module can be optimized to its required speed, contributing to more detailed learning and more stable results.

[0152] The backbone encoder is responsible for basic feature extraction and latent feature generation, and may require a relatively stable learning rate.

[0153] The semantic decoder is responsible for predicting class labels at the voxel level and may require fine-tuning of training to improve object classification accuracy.

[0154] The density decoder is responsible for predicting the density of voxels and may require learning rate adjustments that are sensitive to density changes.

[0155] By applying the optimal learning rate for each module, we can prevent overfitting or underfitting of specific modules during the training process. This helps each module to be independently optimized, especially for complex three-dimensional data.

[0156] The framework of the present invention integrates density awareness into LiDAR data processing, aiming to learn generalized representations that maintain high performance in diverse environments. To achieve this, it employs a self-supervised learning approach by introducing an auxiliary task that predicts voxel-level density.

[0157] In the present invention, input voxel features Silver shared encoder through latent features Here, M represents the number of voxels before encoding, and m represents the number of voxels after encoding. The latent features are processed by two decoders, each performing semantic segmentation and density prediction. This is expressed in mathematical formula 7 as follows:

[0158]

[0159] Here represents the semantic segmentation prediction result, shows the density prediction results.

[0160] Voxel-specific density labels may be required for learning. Below, a self-supervised learning method for generating voxel-specific density labels is described with reference to Figure 3.

[0161] In the process of generating density labels through self-supervised learning, the k-NN algorithm is used to find neighboring voxels of a given voxel. This allows the density of surrounding voxels for each voxel to be calculated.

[0162] Below, the k-NN algorithm is applied and explained as in mathematical formula 8.

[0163]

[0164] k represents the number of neighboring voxels of each voxel, and in this process, for example, k=10 can be set. M represents the total number of voxels. Referring to Fig. 3, the distance d from the center voxel 701 to the predetermined kth voxel is determined. The density can be expressed as the number of points in the volume of a three-dimensional sphere based on the distance d.

[0165] The calculation of density labels can be defined using mathematical expression 9.

[0166]

[0167] represents the density label for voxel i, and a log transformation can be applied to estimate the density value for each voxel. Euclidean distance of is used in density calculations and reflects the density relationship between neighboring voxels. This process uses the k-NN algorithm to perform density calculations that consider the distances between neighboring voxels, and normalizes the scale change through log transformation to generate stable density labels.

[0168] The invention uses two main loss functions to predict voxel-to-voxel density and perform semantic segmentation while processing 3D point cloud data. This optimizes model learning to achieve more generalized performance.

[0169] In the density prediction task, the Smooth-L1 loss function is used to accurately predict the density difference between voxels. This loss function is used to estimate the predicted density and actual density labels It works in a way that minimizes the difference between the two.

[0170] In semantic segmentation tasks, a combination of the cross-entropy loss function and the Lovasz-Softmax loss function is used. This composite loss function is designed to improve the accuracy of class label prediction for each voxel.

[0171] The loss function related to density prediction and segmentation can be expressed by mathematical expression 10.

[0172]

[0173] Here, CE is the cross entropy loss function.

[0174] To achieve a balance between density prediction and semantic segmentation during the learning process, the overall loss function is defined as a weighted sum of the two loss functions. This weighted sum loss function is defined as Equation 11.

[0175]

[0176] Here is the weighting coefficient. For example, This can help us better learn the influence of density.

[0177] The present invention utilizes a trained backbone encoder (200) and a trained semantic decoder (400) to perform an inference step for recognizing objects in three-dimensional space. This process utilizes a trained model to input new LiDAR data, and aims to recognize objects by generating a semantic segmentation map from the data.

[0178] In the inference process, point cloud data (10) acquired from a lidar sensor is first voxelized and converted into a sparse tensor. This sparse tensor is input to a backbone encoder (200), which generates latent features including the spatial location and feature vector of each voxel.

[0179] The latent features generated by the backbone encoder are input to a semantic decoder (400), which predicts class labels at the voxel level. The semantic decoder analyzes the semantic information of voxels based on parameters optimized during the learning process and generates a semantic segmentation map based on this.

[0180] The generated semantic segmentation map contains information that assigns each voxel in the 3D space to a specific object class. This allows for accurate recognition of the location and shape of each object, even when new data is input from the lidar sensor.

[0181] During the training process, the semantic decoder also learns the influence of density, thereby acquiring knowledge about density perception. Therefore, when performing inference using a trained model, the density decoder can be disabled and only the semantic decoder activated, achieving the desired semantic object segmentation.

[0182] FIG. 4 is a general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0183] Although the present disclosure has been described above as being generally implemented by a computer device, those skilled in the art will appreciate that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0184] Generally, program modules include routines, programs, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0185] The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0186] Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0187] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0188] An exemplary environment (1100) implementing various aspects of the present disclosure is illustrated, including a computer (1102) comprising a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0189] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0190] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0191] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate that other types of media readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0192] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0193] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0194] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0195] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

[0196] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0197] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0198] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0199] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0200] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0201] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0202] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0203] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0204] The following references are incorporated herein by reference in their entirety for the purpose of supporting the invention of the present disclosure.

[0205] [1] P. Sun et al., “Scalability in perception for autonomous driving: Waymoopen dataset,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2446-2454.

[0206] [2] H. Caesar et al., “nuscenes: A multimodal dataset for autonomous driving,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11 621-11 631.

[0207] [3] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “Semantickitti: A dataset for semantic scene understanding of lidar sequences,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2019, pp. 9297-9307

[0208] [4] L. Yi, B. Gong, and T. Funkhouser, “Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15 363-15 373.

[0209] [5] M. Rochan, S. Aich, E. R. Corral-Soto, A. Nabatchian, and B. Liu, “Unsupervised domain adaptation in lidar semantic segmentation with self-supervision and gated adapters,”in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2022,

[0210] pp. 2649-2655.

[0211] [6] L. Kong, N. Quader, and V. E. Liong, “Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation,”in Proceedings of IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 9338-9345.

[0212] [7] A. Shaban, J. Lee, S. Jung, X. Meng, and B. Boots, “Lidar-uda: Self ensembling through time for unsupervised lidar domain adaptation,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023, pp. 19 784-19 794.

[0213] [8] H. Kim, Y. Kang, C. Oh, and K.-J. Yoon, “Single domain generalization for lidar semantic segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17 587-17 598.

[0214] [9] K. Ryu, S. Hwang, and J. Park, “Instant domain augmentation for lidar semantic segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9350-9360.

[0215]

[0010] J. Sanchez, J.-E. Deschaud, and F. Goulette, “Domain generalization of 3d semantic segmentation in autonomous driving,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023, pp. 18 077-18 087.

[0216]

[0011] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, “Kpconv: Flexible and deformable convolution for point clouds,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2019, pp. 6411-6420.

[0217]

[0012] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, “Randla-net: Efficient semantic segmentation of large scale point clouds,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11 108-11 117.

[0218]

[0013] S. Fan, Q. Dong, F. Zhu, Y. Lv, P. Ye, and F.-Y. Wang, “Scf net: Learning spatial contextual features for large-scale point cloud segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14 504-14 513.

[0219]

[0014] T. Cortinhal, G. Tzelepis, and E. Erdal Aksoy, “Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds,”in Advances in Visual Computing: 15th International Symposium, ISVC 2020, San Diego, CA, USA, October 5-7, 2020, Proceedings, Part II 15. Springer, 2020, pp. 207-222.

[0220]

[0015] Y. Zhang, Z. Zhou, P. David, X. Yue, Z. Xi, B. Gong, and H. Foroosh, “Polarnet: An improved grid representation for online lidar point clouds semantic segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9601-9610.

[0221]

[0016] K. Peng, J. Fei, K. Yang, A. Roitberg, J. Zhang, F. Bieder, P. Heidenre ich, C. Stiller, and R. Stiefelhagen, “Mass: Multi-attentional semantic segmentation of lidar data for dense top-view understanding,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 15 824-15 840, 2022.

[0222]

[0017] T.-H. Chen and T. S. Chang, “Rangeseg: Range-aware real time segmentation of 3d lidar point clouds,”IEEE Transactions on Intelligent Vehicles, vol. 7, no. 1, pp. 93-101, 2022.

[0223]

[0018] A. Ando, S. Gidaris, A. Bursuc, G. Puy, A. Boulch, and R. Marlet, “Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5240-5250.

[0224]

[0019] H.-X. Cheng, X.-F. Han, and G.-Q. Xiao, “Transrvnet: Lidar semantic segmentation with transformer,”IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 6, pp. 5895-5907, 2023.

[0225]

[0020] B. Graham, M. Engelcke, and L. Van Der Maaten, “3d semantic segmentation with submanifold sparse convolutional networks,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 9224-9232.

[0226]

[0021] C. Choy, J. Gwak, and S. Savarese, “4d spatio-temporal convnets: Minkowski convolutional neural networks,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3075-3084

[0227]

[0022] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 9939-9948.

[0228]

[0023] Y. Hou, X. Zhu, Y. Ma, C. C. Loy, and Y. Li, “Point-to-voxel knowledge distillation for lidar semantic segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8479-8488.

[0229]

[0024] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky, “Sparse convolutional neural networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 806-814.

[0230]

[0025] B. Graham and L. Van der Maaten, “Submanifold sparse convolutional networks,” arXiv preprint arXiv:1706.01307, 2017.

[0231]

[0026] B. Wu, X. Zhou, S. Zhao, X. Yue, and K. Keutzer, “Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud,”in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2019, pp. 4376-4382.

[0232]

[0027] C. Saltori, S. Lathuiliere, N. Sebe, E. Ricci, and F. Galasso, “Sf-uda ´ 3d: Source-free unsupervised domain adaptation for lidar-based 3d object detection,”in International Conference on 3D Vision (3DV), 2020, pp. 771-780.

[0233]

[0028] C. Saltori, F. Galasso, G. Fiameni, N. Sebe, E. Ricci, and F. Poiesi, “Cosmix: Compositional semantic mix for domain adaptation in 3d lidar segmentation,”in Proceedings of European Conference on Computer Vision (ECCV). Springer, 2022, pp. 586-602.

[0234]

[0029] Z. Yuan, C. Wen, M. Cheng, Y. Su, W. Liu, S. Yu, and C. Wang, “Category-level adversaries for outdoor lidar point clouds cross domain semantic segmentation,”IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 2, pp. 1982-1993, 2022.

[0235]

[0030] M. Li, Y. Zhang, X. Ma, Y. Qu, and Y. Fu, “Bev-dg: Cross-modal learning under bird's-eye view for domain generalization of 3d semantic segmentation,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023, pp. 11 632-11 642.

[0236]

[0031] C. Saltori, A. Osep, E. Ricci, and L. Leal-Taixe, “Walking your lidog: ´A journey through multiple domains for lidar semantic segmentation,”in Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023, pp. 196-206.

[0237]

[0032] X. Pan, P. Luo, J. Shi, and X. Tang, “Two at once: Enhancing learning and generalization capacities via ibn-net,”in Proceedings of European Conference on Computer Vision (ECCV), 2018, pp. 464-479.

[0238]

[0033] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,”in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.

[0239]

[0034] M. Berman, A. R. Triki, and M. B. Blaschko, “The lovasz-softmax loss: ´ A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4413-4421.

[0240]

[0035] C.-Y. Lee, T. Batra, M. H. Baig, and D. Ulbricht, “Sliced Wasserstein discrepancy for unsupervised domain adaptation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10 285-10 295.

[0241]

[0036] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,”in Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14. Springer, 2016, pp. 443-450.

[0242]

[0037] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Perez, “Advent: ´ Adversarial entropy minimization for domain adaptation in semantic segmentation,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2517-2526.

[0243]

[0038] A. Nekrasov, J. Schult, O. Litany, B. Leibe, and F. Engelmann, “Mix3d: Out-of-context data augmentation for 3d scenes,” in International Conference on 3D Vision (3DV), 2021, pp. 116-125.

[0244]

[0039] A. Xiao, J. Huang, D. Guan, K. Cui, S. Lu, and L. Shao, “Polarmix: A general data augmentation technique for lidar point clouds,” in Neural Information Processing Systems (NeurIPS), 2022, pp. 11 035-11 048

[0245] As described above, the relevant contents have been described in the best form for carrying out the invention.

[0246] The present invention can be used in a technical field in which object recognition technology utilizing a lidar sensor is utilized.

Claims

1. A computer program stored in a computer-readable storage medium, wherein the computer program performs the following methods for recognizing an object in a three-dimensional space, wherein the method is: A step of converting point cloud data acquired from a lidar sensor into a sparse tensor by voxelizing it; A step of generating latent features by encoding the above sparse tensor through a backbone encoder; A step of generating a semantic segmentation map through a semantic decoder based on the above latent features; A step of generating a density map through a density decoder based on the above latent features; and A step of learning at least one of the backbone encoder, the semantic decoder and the density decoder; including, Computer program.

2. In paragraph 1, The step of converting the above point cloud data into a sparse tensor by voxelizing it is as follows: A step of dividing the above point cloud data into grids of a predetermined size and converting points within each grid into one voxel; including, Computer program.

3. In paragraph 2, The size of the above grid can be variably set depending on at least one of the resolution of the lidar sensor or the application environment. Computer program.

4. In paragraph 1, The step of converting the above point cloud data into a sparse tensor by voxelizing it is as follows: A step of generating voxels at spatially regular intervals, and representing multiple points within the same voxel as one voxel by averaging them; including, Computer program.

5. In paragraph 4, The averaging of the above point cloud data also averages feature information including the location information, reflection intensity, or laser regression value of each point within each voxel. Computer program.

6. In paragraph 1, The step of converting the above point cloud data into a sparse tensor by voxelizing it is as follows: A filtering step that removes or merges voxels below a certain density, taking into account continuity with surrounding voxels during the voxelization process; including more, Computer program.

7. In paragraph 1, The above sparse tensor stores only non-empty voxels to optimize memory usage and computation. Computer program.

8. In paragraph 1, The step of generating latent features by encoding the above sparse tensor through a backbone encoder is as follows: A step of generating a sparse voxel feature including the spatial location and feature vector of each voxel of the above sparse tensor; including, Computer program.

9. In paragraph 8, The step of generating latent features by encoding the above sparse tensor through a backbone encoder is as follows: A step of generating kernel occupancy information (Kernel Occupancy Binary Vector) based on the spatial location of each voxel, converting the kernel occupancy information into a feature vector through 1×1 convolution, and combining the feature vector with the sparse voxel feature to generate the latent feature; including, Computer program.

10. In paragraph 9, The step of generating latent features by encoding the above sparse tensor through a backbone encoder is as follows: A step of performing an operation on the sparse voxel features and the feature vector through a density-aware sparse convolution residual block, and generating a final latent feature by residually connecting the operation results; including, Computer program.

11. In paragraph 10, The above density-aware sparse convolution residual block is, The input feature vector is processed through 3*3*3 sparse convolution, the feature vector is compressed through 1*1 sparse convolution, and the vector is residually connected to generate the final latent feature. Computer program.

12. In paragraph 8, The above backbone encoder is characterized in that it includes multiple density-aware sparse convolution residual blocks, and each block generates a latent feature by stepwise enhancing the features of the input sparse tensor. Computer program.

13. In paragraph 1, The above semantic decoder is, Taking the above latent features as input, predicting the class label of each voxel to generate a semantic segmentation map. Computer program.

14. In paragraph 13, The above semantic decoder is, It includes multiple decoding layers, each of which progressively restores the spatial resolution of latent features, ultimately generating a semantic segmentation map. Computer program.

15. In paragraph 1, The above density decoder is, Taking the above latent features as input, predicting the density of voxels and generating a density map. Computer program.

16. In paragraph 17, The above density decoder is, It includes multiple decoding layers, each of which gradually refines the density information of latent features to ultimately generate a density map. Computer program.

17. In paragraph 1, The step of learning at least one of the above backbone encoder, the above semantic decoder and the above density decoder is: The latent features of the backbone encoder are received as input and the outputs of the semantic decoder and the density decoder are generated, respectively. The output of the above semantic decoder is trained using a cross entropy loss function and a Lovasz-Softmanx loss function, and the output of the above density decoder is trained using a Smooth-L1 loss function; including, Computer program.

18. In paragraph 17, The step of learning at least one of the above backbone encoder, the above semantic decoder and the above density decoder is: A step of optimizing the backbone encoder through a joint learning method, taking into account the correlation between the output of the semantic decoder and the output of the density decoder; including, Computer program.

19. In paragraph 18, The above joint learning is, Including adjusting the weights of the loss functions during the learning process to achieve a balance between the cross entropy loss function for the output of the semantic decoder and the Smooth-L1 loss function for the output of the density decoder. Computer program.

20. In paragraph 1, The step of learning at least one of the above backbone encoder, the above semantic decoder and the above density decoder is: A step of learning by applying independent learning rates to each of the backbone encoder, the semantic decoder, and the density decoder; including, Computer program.

21. In paragraph 1, An inference step for recognizing an object in a three-dimensional space using the learned backbone encoder and the learned semantic decoder; including more, Computer program.

22. A method for recognizing objects in three-dimensional space, A step of converting point cloud data acquired from a lidar sensor into a sparse tensor by voxelizing it; A step of generating latent features by encoding the above sparse tensor through a backbone encoder; A step of generating a semantic segmentation map through a semantic decoder based on the above latent features; A step of generating a density map through a density decoder based on the above latent features; and A step of learning at least one of the backbone encoder, the semantic decoder and the density decoder; including, method.

23. As a computer device, one or more processors; and A memory storing instructions executable by one or more processors; Including, One or more of the above processors, Point cloud data acquired from a lidar sensor is voxelized and converted into a sparse tensor. Encode the above sparse tensor through a backbone encoder to generate latent features, Based on the above latent features, a semantic segmentation map is generated through a semantic decoder, Generate a density map through a density decoder based on the above latent features, and Learning at least one of the backbone encoder, the semantic decoder and the density decoder, Computer devices.

Images