Single dataset domain generalization method in 3D point cloud data

Abstract

The present invention discloses a method for domain generalization of a single dataset in 3D point cloud data. The method comprises: pre-training a deep learning model based on a source domain dataset and a set classification loss function to obtain a baseline model, wherein the source domain dataset reflects the correspondence between 3D point cloud data and class labels; training the baseline model based on a set overall objective function with adaptation to multiple target domains as the optimization goal to obtain a domain generalization model, wherein the overall objective function includes the classification loss; and using the domain generalization model to perform category prediction for point cloud data in the target domain. The present invention enables good transfer capabilities to multiple different target domains without the restriction of touching the target domain.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and more particularly to a single data set domain generalization method in 3D point cloud data. Background Art

[0002] As a commonly used data format for describing the real world, point cloud representations retain more geometric information in three-dimensional scenes and have become a crucial data type for real-world applications such as autonomous driving, AR / VR, and robotics. In recent years, point cloud-based vision tasks have made significant progress on public benchmarks, largely due to the fact that collected point cloud datasets are often finely annotated, large in volume, and have low noise. However, in the real world, acquiring such data from a new target domain and manually annotating this 3D data relies heavily on domain experts, making data acquisition and annotation more difficult, labor-intensive, and time-consuming.

[0003] An effective solution for migrating models from a fully labeled source domain to a new domain without requiring additional human effort is unsupervised domain adaptation (UDA), which aims to learn more generalizable representations between a labeled source domain and an unlabeled target domain, allowing the model to adapt to the data distribution of the target domain. However, these techniques rely heavily on the accessibility of target domain data, an assumption that cannot be fully guaranteed in areas such as autonomous driving and smart healthcare. Therefore, it is important to study the cross-domain generalization capabilities of models under the constraints of a zero-shot target domain, which has led to the domain generalization (DG) task for 3D scenes.

[0004] In the existing technology, there are mainly domain adaptation solutions based on 2D images and domain adaptation solutions based on 3D point cloud classification. Domain adaptation (DA) works based on 2D images can be roughly divided into two categories: 1) methods based on adversarial learning, whose research focus is to use domain label discriminators to reduce inter-domain differences; 2) methods based on moment matching, that is, aligning the first-order or second-order moments of feature distributions. However, when the target domain data is unavailable, the above data generalization methods cannot be directly applied to the DG problem. To this end, some researchers began to explore how to adapt the pre-trained model from the source domain to the out-of-distribution domain using only the source data. For example, some works attempt to use mixed domains to improve the generalization ability of the model and generate new data distributions from a mixture of multiple domains. In addition, self-supervised learning (SSL) has also been applied to the DG problem to enhance transferable features by leveraging designed pre-tasks.

[0005] For domain adaptation solutions based on 3D point cloud classification, existing work often attempts to design a camouflage task to address common geometric deformations caused by differences in scanned point clouds. For example, DefRec achieves good domain adaptation results in various domain shift scenarios by deforming the regional shape of a point and reconstructing the original region of that shape. PointDAN proposes node-wise adaptive (SA) learning with node-level attention to present point geometric shape information.

[0006] Analysis reveals that current research on zero-shot domain adaptation, or domain generalization, involves adapting a trained source domain model to an unseen target domain without using any target samples. However, existing research on domain generalization remains underdeveloped on 3D point cloud data and faces challenges such as the irregular structure of point cloud data and uneven distribution of modalities across classes. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a method for generalizing a single dataset domain in 3D point cloud data. The method includes the following steps:

[0008] Pre-training a deep learning model based on a source domain dataset and a set classification loss function to obtain a baseline model, wherein the source domain dataset reflects the correspondence between three-dimensional point cloud data and category labels;

[0009] Training the baseline model based on a set overall objective function with adaptation to multiple target domains as an optimization goal to obtain a domain generalization model, wherein the overall objective function includes the classification loss;

[0010] The domain generalization model is used to perform category prediction on point cloud data in the target domain.

[0011] Compared with the existing technology, the advantage of the present invention is that, compared with the existing image-oriented 2D DG, the present invention proposes a single-dataset unified domain generalization (SUG) framework for the 3D DG problem, that is, only using a single source domain data to alleviate the unpredictable domain differences faced by the well-pre-trained source model, thereby improving the generalization ability to different target domains.

[0012] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0014] Figure 1is a flowchart of a single dataset domain generalization method in 3D point cloud data according to one embodiment of the present invention;

[0015] Figure 2 2. Schematic diagram of a unified domain generalization framework for a single dataset according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of unique features in a 3D point cloud dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0018] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0019] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0020] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0021] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0022] In the single dataset domain generalization method in 3D point cloud data provided by the present invention, a single dataset unified domain generalization (SUG) framework (also called model or network) is designed. The SUG framework includes a multi-granularity subdomain alignment (MSA) module, which constrains the model to learn domain-independent and discriminative feature representations by performing multi-granularity feature alignment between subdomains separated in a single source domain dataset. In addition, the SUG framework also includes a sample-level domain-aware attention (SDA) strategy, which selectively enhances easily adaptable samples from different subdomains based on the sample-level inter-domain distance to avoid negative transfer. The present invention is a solution paradigm for the one-to-many (single training source domain to multiple target domains) 3D DG problem.

[0023] Specifically, see Figure 1 As shown, the single dataset domain generalization method in the provided 3D point cloud data includes the following steps:

[0024] In step S110 , a one-to-many 3D domain generalization problem is defined with the goal of migrating from a single training source domain to multiple target domains.

[0025] Assume that the domain is given by the joint distribution P XY Definition, where X and Y represent the input space and label space respectively. In the research scope of DG, K source domains It can be used in the training process, where each different source domain has a different joint distribution. The goal of DG is to obtain a model f that is trained on the source domain and obtains the minimum prediction error on the unseen target domain. 3D point cloud data is a set of unordered three-dimensional points x = {pi|i = 1, ..., n}, where each point pi is usually represented by its three-dimensional coordinates (x, y, z), and n is the number of sample points of a three-dimensional object. A training sample pair is represented by (x, y), and .y is its label.

[0026] In the single dataset DG problem, model training can only access a labeled dataset S, and the trained model is required to be evaluated on M unseen target datasets T. In the problem setting of this invention, it is necessary to solve the 3D point cloud classification problem, and the objective function can be defined as:

[0027]

[0028] Express expectations.

[0029] The prediction result is obtained by the following formula:

[0030]

[0031] Where x is the input point cloud instance, is the predicted label. is Parameterized embedded networks, is a parameterized classifier, represents the parameters of the neural network feature extractor part, and θ represents the parameters of the neural network classification head part.

[0032] Step S120: construct a deep learning model, which is a unified domain generalization framework based on a single dataset, including an embedding network, a multi-granularity subdomain alignment module, and a sample-level domain-aware attention module.

[0033] In one embodiment, the SUG framework is provided by Figure 2 As shown. In addition to the embedded network (marked ), the framework also contains two new plug-and-play modules, namely the multi-granularity subdomain alignment (MSA) module and the sample-level domain-aware attention (SDA) module, which can be plugged into existing 3D backbone networks to learn more domain-independent representations.

[0034] First, based on a predefined heuristic algorithm, the single source dataset is input into the dataset segmentation module to obtain multiple subdomains of the original source dataset. Figure 2 Then, the embedding network takes all the segmented subdomains as network input and converts the point cloud instance x into a multi-level feature vector as well as Representing low-level and high-level feature vectors, respectively. To address feature differences across subdomains, the MSA module is used to align multi-granular features at low and high levels, thereby constraining the network to focus on domain-agnostic representations. Simultaneously, the SDA module selectively enforces alignment constraints on easily transferable samples to ensure uniform adaptation across subdomains.

[0035] Step S130 : setting loss items for the multi-granularity subdomain alignment module, including a category distribution loss item, a geometric difference loss item, and a semantic difference loss item.

[0036] 1) Category distribution alignment.

[0037] 3D point clouds have been deployed in a wide range of application scenarios, where the distribution of object categories has changed significantly, leading to different distribution patterns. To address this cross-dataset class imbalance, we introduce class weights into the original classification loss function and form a weighted classification loss function, as follows:

[0038]

[0039] in, Represents a batch, that is, a batch, which is the amount of data loaded into the neural network at one time. L(θ; x) represents the loss loss (under supervised signals) obtained by inputting a sample x when the neural network parameters are θ. x represents point cloud data, y represents category label, w(y) represents the weight vector related to the category, and θ represents the learnable model parameters.

[0040] In practical applications, the weight vector can be set according to different heuristics, such as FocalLoss, DLSA, etc. For example, the definition in DLSA can be continued and the weight of each training sample can be defined as:

[0041]

[0042] Among them, n iRepresents the number of samples in the i-th category, and q is a positive number used to control the distribution of weights.

[0043] 2) Geometric drift alignment.

[0044] Due to the geometric differences of objects in different scenes and inconsistent data collection processes, objects of the same category from different datasets present different geometric appearances, e.g. Figure 3 (a) The appearance of a chair. At the same time, the geometric appearance of objects varies greatly within a specific category or a single dataset, which makes it possible to effectively model the geometric differences between different datasets by leveraging the geometric differences in a single dataset.

[0045] More specifically, from the feature embedding network The low-level feature vector f is extracted from the shallow network l , and align the geometric features from different subdomains by minimizing the maximum mean difference (MMD) loss as follows:

[0046]

[0047] Where κ is the kernel function, subscripts t and s represent two different subdomains sampled from a single dataset, and n s Indicates the number of samples in subdomain s, n t represents the number of samples in subdomain t, i represents the sample index of subdomain s, and j represents the sample index of subdomain t.

[0048] 3) Semantic difference alignment.

[0049] In obtaining the high-level feature vector f h Finally, the semantic variance alignment method is used to minimize the semantic level differences between features of different subdomains before inputting them into the classifier. The intuition of the semantic alignment method comes from the observation that samples from different categories may have similar geometric appearances. Figure 3 As shown in (b), the samples in the table and cabinet categories are very similar to some samples in the chair category because they all have four legs. By performing semantic difference alignment, the model will be forced to learn representations with fewer single-domain geometric deviations. Semantic alignment constraints It can be calculated by using the high-level feature vector into the above formula. It should be noted that, The expression is Similar, no further description is given here.

[0050] In step S140 , a loss term is set for the sample-level domain-aware attention module to enhance the alignment constraint of easy-to-transfer samples.

[0051] The above-mentioned MSA module guides the model to learn more domain-independent representations. However, the mini-batch features from different subdomains do not contribute equally to the subdomain alignment process because they may contain different feature distributions. Ignoring this diversity and imposing equal importance on different samples will lead to the phenomenon of negative generalization of samples that are difficult to transfer. At the same time, the dataset segmentation module designed in the SUG framework inevitably introduces randomness to different sub-regions, thereby affecting the generalization performance of the model. In order to enable safer and more robust transfer learning, the present invention proposes an SDA module to enhance the alignment constraints of samples that are easy to transfer. More specifically, a sample-level weight is added to the alignment constraint, which is inversely proportional to the domain distance and can be generally expressed as:

[0052]

[0053] in, represents the alignment constraint after adding sample-level weight, ω represents the sample-level weight, L MMD represents the maximum mean difference regularization loss, and d represents the domain distance.

[0054] For geometric drift alignment, a 3D reconstruction metric is used as a distance function. In one embodiment, the chamfer distance (CD) is used, which can be expressed as:

[0055]

[0056] Where X and Y are two point cloud instances. Geometric weights focus more on explicit geometric consistency, such as Figure 3 As shown in the first column of (a), samples with geometric similarity have relatively small CD distances, even though they may come from different categories. For samples with different geometric appearances, the CD distance is higher and the corresponding MMD constraints will be relaxed.

[0057] For semantic difference alignment, Jensen-Shannon (JS) divergence can be used as a metric. Considering the symmetry of the metric, the JS distance is written as:

[0058]

[0059] Among them, D KL It is the discrete form of KL divergence, specifically written as:

[0060]

[0061] Where x(c) and Y(c) describe the probability of predicting a sample belonging to class C. Compared with geometric weights, semantic weights focus more on semantic consistency and tend to compare samples belonging to the same class.

[0062] Step S150: Set the overall objective function and train the deep learning model to obtain a domain generalization model.

[0063] For example, according to the aforementioned alignment constraints and alignment weights, the complete MMD loss function is expressed as:

[0064]

[0065] in, is a semantic alignment constraint, is the geometric difference constraint, ω Geo and ω Sem is the corresponding weight.

[0066] Combined with formula (6), for ω Geo , the related domain distance is the chamfer distance; for ω Sem , whose related domain distance is Jensen-Shannon (JS) distance.

[0067] The complete training loss (i.e., the overall objective function) includes the classification loss function described above and the MMD loss mentioned above, which can be written as:

[0068] L=L cls +L MMD (11)

[0069] Among them, L MMD is the maximum average discrepancy regularization loss, which is used to align the distributions between different domains.

[0070] Specifically, to implement the domain generalization strategy, the deep learning model is trained in an end-to-end manner, including:

[0071] Step 1: First, use the classification loss L cls Training the deep learning model ensures that the trained baseline model learns representative features and prepares for subsequent migration.

[0072] Step 2: In order to learn a robust representation that can be generalized to different target datasets, the baseline model (i.e., the model trained in step 1) is trained using the complete loss function L. The trained baseline model serves as a domain generalization model and can be used for subsequent migration to multiple different target domains.

[0073] To further validate the effectiveness of this invention, we conducted extensive experiments on various point cloud classification datasets, including ShapeNet, ModelNet, and ScanNet. Experimental results show that compared to existing UDA methods, our SUG framework can effectively improve the model's generalization ability to unseen target domains, and even outperforms existing unsupervised domain adaptation methods that require access to target domain data.

[0074] In summary, the present invention takes into account the existence of multiple subdomain distributions in a single training dataset, and the representation learning based on this diversity to achieve domain generalization, and provides the application of a strategy based on domain adaptation algorithm and domain generalization in point cloud classification. In the source domain training stage, an MSA module is designed to learn domain-independent and discriminative features, and in the source domain training stage, an SDA strategy is designed to calculate the sample-level inter-domain distance, and thereby balance the adaptability of different subdomains. The present invention can take into account domain generalization feature learning at both the set level and the semantic level to ensure a safer and more robust migration process. In short, the present invention proposes for the first time a domain generalization scheme for a single dataset in the field of 3D point clouds, which can achieve relatively good migration capabilities without being restricted by the target domain.

[0075] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0076] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0077] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0078] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, Python, and conventional procedural programming languages such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0079] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0082] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0083] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A single dataset domain generalization method for 3D point cloud data, comprising the following steps: Pre-training a deep learning model based on a source domain dataset and a set classification loss function to obtain a baseline model, wherein the source domain dataset reflects the correspondence between three-dimensional point cloud data and category labels; Training the baseline model based on a set overall objective function with adaptation to multiple target domains as an optimization goal to obtain a domain generalization model, wherein the overall objective function includes a classification loss; Using the domain generalization model to perform category prediction on point cloud data of the target domain; Among them, the classification loss function is set to: in, is the class weight of the training sample, Indicates the amount of data loaded into the deep learning model at one time, Indicates that the deep learning model has parameters In this case, input a point cloud data x, and the loss is obtained, x represents the point cloud data, Represents the category label; The category weight of the training sample is defined as: in, represents the number of samples in the i-th category, q is a set positive number; Wherein, the overall objective function is set as: in, is the maximum mean difference regularization loss function, represents the classification loss function; Among them, the maximum mean difference regularization loss function is expressed as: in, is the semantic alignment constraint, is the geometric difference constraint, and is the corresponding weight; The geometric difference constraint term is expressed as: in, is the kernel function, subscript t and s Represent two different subdomains sampled from a single dataset, Indicates a subdomain s The number of samples in Indicates a subdomain t The number of samples in Indicates a subdomain s The sample index of Indicates a subdomain t The sample index in , and Represents the multi-level features corresponding to the samples of subdomain s, and Represents the multi-level features corresponding to the samples in subdomain t.

2. The method according to claim 1, characterized in that is the inverse of the sample's chamfer distance, It is the reciprocal of the JS distance.

3. The method according to claim 1, characterized in that The deep learning model includes an embedding network, a multi-granularity subdomain alignment module and a sample-level domain-aware attention module. The embedding network takes multiple subdomains of a source domain dataset as input to extract multi-granularity features of point cloud instances; the multi-granularity subdomain alignment module is used to align the multi-granularity features at low and high levels, and uses the sample-level domain-aware attention module to enhance the alignment constraints of samples that are easy to transfer.

4. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

5. A computer device comprising a memory and a processor, wherein a computer program capable of being executed on the processor is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

Images