Training method of point cloud multi-modal model and electronic device

Abstract

This application discloses a training method and electronic device for a point cloud multimodal model, which relates to the field of artificial intelligence technology. By co-masking reconstruction of point cloud, text and image modalities, aligning the contrastive loss functions of the three modalities, and aligning the representations of the three modalities pairwise to a unified semantic space, the semantic details of the three modalities are fully explored, enabling the model to capture more complex cross-modal relationships, thereby improving the accuracy of model training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to training methods and electronic devices for point cloud multimodal models. Background Technology

[0002] In the fields of 3D perception and multimodal artificial intelligence, point cloud data, as the core representation of 3D space, requires accurate semantic recognition for accurate 3D perception. Currently, the encoding and decoding processes of point clouds are handled independently during the training of point cloud models, leading to fragmented semantic representations and low model training accuracy. Summary of the Invention

[0003] This application provides a training method and electronic device for point cloud multimodal models, in order to at least solve the problem of low model training accuracy in related technologies.

[0004] This application provides a method for training a point cloud multimodal model, including:

[0005] Acquire multimodal data synchronously collected for the sample object. The multimodal data includes point cloud data, text semantic data, and image data.

[0006] Multimodal data is processed by segmentation and independent random masking to obtain the visible features and masking features of each modality.

[0007] The visible features of each modality are encoded by the encoder corresponding to each modality to obtain multimodal global features, and the multimodal global features are mapped to the shared semantic space.

[0008] Based on the multimodal global features in the shared semantic space, determine the contrast loss between each pair of modalities; based on the multimodal global features in the shared semantic space and the mask features of each modality, determine the mask reconstruction loss of each modality.

[0009] Based on the contrast loss between each two modes and the mask reconstruction loss of each mode, a joint loss function is constructed, and the multimodal encoder-decoder network is iteratively trained according to the joint loss function to obtain the point cloud multimodal model.

[0010] This application also provides a training device for a point cloud multimodal model, comprising:

[0011] The acquisition module is used to acquire multimodal data synchronously collected from the sample object. The multimodal data includes point cloud data, text semantic data, and image data.

[0012] The processing module is used to perform segmentation processing and independent random masking processing on the multimodal data to obtain the visible features and masking features of each modality.

[0013] The encoding module is used to encode the visible features of each modality through the encoder corresponding to each modality to obtain multimodal global features, and then map the multimodal global features to the shared semantic space;

[0014] The loss module is used to determine the contrast loss between each pair of modalities based on the multimodal global features in the shared semantic space, and to determine the mask reconstruction loss of each modality based on the multimodal global features in the shared semantic space and the mask features of each modality.

[0015] The optimization module is used to construct a joint loss function based on the contrast loss between each two modes and the mask reconstruction loss of each mode, and to iteratively train the multimodal encoder-decoder network based on the joint loss function to obtain the point cloud multimodal model.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described point cloud multimodal model training methods.

[0017] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described point cloud multimodal model training methods.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described point cloud multimodal models training methods.

[0019] This application enables collaborative mask reconstruction of point cloud, text, and image modalities, aligns the contrastive loss functions of the three modalities, aligns the representations of the three modalities pairwise to a unified semantic space, fully explores the semantic details of the three modalities, and allows the model to capture more complex cross-modal relationships, thereby improving the accuracy of model training. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram illustrating an application scenario of a training method for a point cloud multimodal model provided in this application embodiment;

[0022] Figure 2 A flowchart illustrating a training method for a point cloud multimodal model provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating another method for training a point cloud multimodal model provided in this application embodiment;

[0024] Figure 4 A schematic diagram of a point cloud multimodal model provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram illustrating the iteration of a point cloud multimodal model provided in an embodiment of this application;

[0026] Figure 6 A schematic diagram illustrating the training and fine-tuning of a point cloud multimodal model provided in an embodiment of this application;

[0027] Figure 7 A schematic diagram of the structure of a training device for a point cloud multimodal model provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of the structure of a training device for another point cloud multimodal model provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] For example, in the fields of 3D perception and multimodal artificial intelligence, especially in scenarios such as autonomous driving, robot navigation, augmented reality (AR) / virtual reality (VR), digital twins, and metaverse, point cloud data, as the core representation of 3D space, needs to be deeply integrated with text descriptions and 2D images to achieve cross-modal semantic understanding. For instance, in autonomous driving, vehicles need to collaboratively analyze point cloud data acquired by LiDAR with camera images and road sign text information to accurately identify obstacles, traffic signs, and road structures; in AR / VR scenarios, users combine point clouds generated by 3D scanning devices with images and text information in the virtual environment to achieve an immersive interactive experience.

[0033] In related technologies, when processing the encoding and decoding of point clouds, text, and images, contrastive learning methods achieve global semantic consistency only by aligning pairwise modal representations, resulting in fragmented semantic representations, insufficient fine-grained semantic alignment between point clouds and text / images, and low training accuracy of point cloud models.

[0034] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The specific application environment architecture upon which the method's execution depends is described here. (References) Figure 1 The feature extraction process is performed on the collected point cloud data to obtain features. These features are then input into the point cloud model to obtain the 3D object classification corresponding to the point cloud data output by the point cloud model.

[0036] Figure 2 This is a flowchart illustrating the training method for a point cloud multimodal model provided in an embodiment of this application, as shown below. Figure 2 As shown, embodiments of this application provide a training method for a point cloud multimodal model. The method is described in detail below:

[0037] S201. Acquire multimodal data synchronously collected for the sample object. The multimodal data includes point cloud data, text semantic data, and image data.

[0038] For example, during the system initialization phase, it is necessary to acquire multimodal data for training. This data does not exist independently, but is collected synchronously on different sensors for the same sample object, such as a "red car model".

[0039] Point cloud data, captured by LiDAR or depth cameras, records the three-dimensional spatial coordinates of points on the vehicle's surface, accurately describing its shape and geometric details.

[0040] Textual semantic data can be a description, such as "a red four-door sedan with round headlights," which provides semantic information that is understandable to people at a higher level.

[0041] The image data was captured by a regular camera and contains visual information such as color, texture, and lighting.

[0042] By collecting data from the same source for the same object, semantic consistency is ensured for data from different modalities, thereby enabling cross-modal learning and alignment.

[0043] For example, the data acquisition process needs to ensure strict time synchronization and spatial calibration. In autonomous driving scenarios, for instance, onboard LiDAR and cameras are activated simultaneously and acquire data at the same frame rate. The system assigns a uniform timestamp to each frame of point cloud, image, and other sensor data to ensure they correspond to the environmental state at the same moment.

[0044] Optionally, the extrinsic parameter matrix between the camera and the LiDAR is also pre-calibrated, which allows the pixels in the image to be accurately mapped to the 3D point cloud coordinate system, thereby correctly learning cross-modal associations.

[0045] For example, the acquired multimodal data contains information from three different sources: point cloud data provides the most direct three-dimensional spatial structure; textual semantic data provides abstract and conceptual high-level knowledge; and image data acts as a bridge, containing both spatial structural information similar to point clouds (through two-dimensional projection) and semantic information similar to text (such as object color and category). In this way, the system can understand objects from multiple angles and levels.

[0046] S202. Perform segmentation and independent random masking processing on the multimodal data to obtain the visible features and masking features of each modality.

[0047] For example, for each modality, the raw data is segmented into multiple local units, a process called slicing. For instance, point cloud data is segmented into several small regions (local slicing), image data is segmented into several small blocks (image sub-slicing), and text data is segmented into several words or phrases (text slicing).

[0048] For each modality, a random masking operation is performed independently. This involves randomly selecting a subset of slices and replacing their original feature information with a special, learnable placeholder (mask token). These replaced slices are called mask features, while the original slices that are not replaced are called visible features.

[0049] For example, the masking strategy is designed based on the Masked Language Model (MLM) from the field of natural language processing. Its core idea is to enable the model to "predict" the masked portion based on the contextual information of the visible portion during training. In multimodal point cloud scenarios, this strategy requires the model to learn to use information from different modalities for collaborative reasoning. For instance, when a partial point cloud patch is masked, the model needs to combine the texture of the corresponding region in the image with the semantic description in the text to accurately reconstruct the 3D geometry of that patch. This "fill-in-the-blank" learning approach allows the model to deeply explore fine-grained relationships between modalities.

[0050] For example, independent random masking is performed for each modality. The masking scale and specific location of point clouds, text, and images can be different and unrelated to each other. This independence prevents the model from learning through simple location correspondence (e.g., assuming that a masked area in an image should also be masked), thus enabling the model to truly understand cross-modal semantic content. For instance, the system can mask the core noun "seat" in the text while leaving the corresponding area in the image and point cloud visible, allowing the model to infer this masked semantics from the image and point cloud information. This greatly enhances the model's reasoning and cross-modal understanding capabilities.

[0051] S203. Encode the visible features of each modality using the encoder corresponding to each modality to obtain multimodal global features, and map the multimodal global features to the shared semantic space.

[0052] For example, after obtaining the visible features of each modality, these features are input into their respective dedicated encoders. For instance, visible point cloud slices are input into a Transformer-based point cloud encoder, visible image slices are input into a Vision Transformer (ViT) encoder, and visible text word sequences are input into a text Transformer encoder.

[0053] These encoders are specifically designed to process the structure and characteristics of their specific modal data, with each encoder corresponding to a modality. The core function of the encoder is to perform contextual fusion of the locally visible features of the input through a multi-layer self-attention mechanism, capturing the long-distance dependencies between local regions within the modality, and finally outputting a global feature vector that can represent the information of the entire modality, such as the CLS (classification) label vector in the output of each modality.

[0054] After encoder processing, although the features of each modality capture rich global semantics, they still exist in their own independent feature spaces, making it impossible to directly perform cross-modal similarity calculations or comparisons. A linear mapping layer is used to transform the feature dimension of one modality to the dimension of other modalities and calibrate the feature distribution to achieve mapping to a shared semantic space.

[0055] With the help of scenario examples, mapping to a shared semantic space is used to eliminate the distribution differences between features of different modalities. For example, the numerical range and variance of text features and image features may be completely different. Linear mapping can unify them, so that positive sample pairs from the same semantics (such as the image and text of "red car") are closer in the shared space, while negative sample pairs (such as the image of "red car" and the text of "green trees") are farther apart.

[0056] For example, the shared semantic space is the core hub of the entire multimodal learning process. This space is dynamically learned during training through tasks such as contrastive learning. The encoder and mapping layer together constitute a function that embeds the raw multimodal data into this space. In this space, the geometric positions of different modal features directly reflect their semantic similarity. For instance, a textual description of "round wheel hub" will be located in the shared space very close to the global features of a point cloud dataset containing a round wheel hub, and also close to the global features of an image that clearly shows a round wheel hub.

[0057] S204. Based on the multimodal global features in the shared semantic space, determine the contrast loss between each pair of modalities. Based on the multimodal global features in the shared semantic space and the mask features of each modality, determine the mask reconstruction loss of each modality.

[0058] For example, after mapping the features of all modalities to a shared semantic space, two core loss functions are introduced to drive the model’s learning: contrastive loss and mask reconstruction loss.

[0059] The role of contrastive loss is to bring closer the distance between different modal features (positive sample pairs) from the same sample and push away the distance between different modal features (negative sample pairs) from different samples, based on global features within a shared space. For example, for a "blue chair" sample in a batch, its point cloud global features, image global features, and text global features are considered members of a positive sample group, while the features of other samples in the same batch constitute negative samples. By calculating the contrastive loss between each pair of modalities, such as point cloud-image contrastive loss, text-image contrastive loss, and point cloud-text contrastive loss, the model is forced to learn the global semantic associations between different modalities.

[0060] The masking reconstruction loss focuses on fine-grained local structure reconstruction. It utilizes multimodal global features in a shared semantic space (as context and conditions) and features of the masked portion (as the prediction target) to guide the decoder in reconstructing the masked local features. For example, for the text modality, the decoder needs to predict the masked word based on features of the visible portion of the text and global features from the image and point cloud. For the image modality, the decoder needs to use global features from the text and point cloud to predict the pixel values ​​of the masked piece. This loss requires the model not only to understand the macroscopic alignment between modalities but also to utilize information from other modalities at the microscopic level to compensate for missing information in its own modality.

[0061] For example, contrastive loss and mask reconstruction loss play different roles during training, but they work together to improve model performance. Contrastive loss is mainly responsible for global semantic alignment, ensuring conceptual consistency across different modalities; while mask reconstruction loss delves into the local level, enabling the model to learn how to utilize cross-modal information for refined prediction and reconstruction.

[0062] To illustrate with a scenario example, when the model sees an image of a car with a "metallic door handle" and the corresponding point cloud, the contrastive loss helps the model understand that all three describe a "car"; while the mask reconstruction loss helps the model try to understand the description of "metallic texture" and guides it to reconstruct the door handle area with high reflectivity from the image and point cloud.

[0063] S205. Based on the contrast loss between each two modes and the mask reconstruction loss of each mode, construct a joint loss function, and iteratively train the multimodal encoder-decoder network according to the joint loss function to obtain the point cloud multimodal model.

[0064] For example, after calculating all contrastive losses and all mask reconstruction losses, they need to be combined into a unified loss function, namely the joint loss function. The joint loss function is implemented by assigning a weight coefficient to each loss term. These weight coefficients can be set manually or learned automatically by the model during training.

[0065] For example, in the early stages of training, to quickly achieve global alignment, the weights of the image-text contrast loss might be set higher; while in the later stages, to refine local semantics, the weights of the point cloud-text mask reconstruction loss might gradually increase. The joint loss function is the basis for updating the model parameters.

[0066] For example, training refers to the process in which, in one iteration, the model calculates the total loss based on the joint loss function and, through backpropagation, simultaneously updates the parameters of all modules in the multimodal encoder-decoder network, including the point cloud encoder, image encoder, text encoder, corresponding decoders for each modality, and parameters of all mapping layers. This process is end-to-end, meaning that all parts of the model work together towards the same ultimate goal: minimizing the joint loss. During iteration, the model continuously learns from the data, gradually adjusting its internal weights to make its output features and alignment results increasingly accurate.

[0067] For example, iterative training is a repeated process. In each training epoch, the model traverses the entire training dataset and then updates its parameters based on the average loss across all batches. This process continues until the model converges, meaning the joint loss function no longer decreases or the performance metrics on the validation set no longer improve. When training ends, the result is a well-trained point cloud multimodal model. This model not only learns how to extract features from each modality, but more importantly, it tightly connects point clouds, images, and text—three heterogeneous data types—at both macro and micro levels, possessing powerful cross-modal understanding and reasoning capabilities. In downstream tasks, this pre-trained model can be used directly or fine-tuned to significantly improve task performance.

[0068] The point cloud multimodal model training method provided in this application involves acquiring multimodal data synchronously collected for sample objects, including point cloud data, text semantic data, and image data; performing segmentation and independent random masking processing on the multimodal data to obtain the visible features and mask features of each modality; encoding the visible features of each modality using the encoder corresponding to each modality to obtain multimodal global features, and mapping the multimodal global features to a shared semantic space; determining the contrast loss between each pair of modalities based on the multimodal global features in the shared semantic space, and determining the mask reconstruction loss for each modality based on the multimodal global features in the shared semantic space and the mask reconstruction loss for each modality; constructing a joint loss function based on the contrast loss between each pair of modalities and the mask reconstruction loss for each modality, and iteratively training the multimodal encoder-decoder network based on the joint loss function to obtain the point cloud multimodal model. Collaborative mask reconstruction of point cloud, text, and image modalities is performed, and the contrastive loss functions of the three modalities are aligned. The representations of the three modalities are aligned pairwise to a unified semantic space, fully mining the semantic details of the three modalities. This enables the model to capture more complex cross-modal relationships, thereby improving the accuracy of model training.

[0069] Based on any of the above embodiments, the following, in conjunction with Figure 3The training process of point cloud multimodal models is explained in detail.

[0070] Figure 3 This is a flowchart illustrating another method for training a point cloud multimodal model provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0071] S301. Acquire multimodal data synchronously collected for the sample object. The multimodal data includes point cloud data, text semantic data, and image data.

[0072] Below, in conjunction with Figure 4 The multimodal model of point cloud is explained.

[0073] Figure 4 This is a schematic diagram of the structure of a point cloud multimodal model provided in an embodiment of this application. Figure 4 As shown, fine-tuning the point cloud multimodal model to adapt to the point cloud semantic segmentation task only requires a point cloud encoder and decoder, and does not require a text encoder and decoder or an image encoder and decoder, because the point cloud encoder and decoder already contain rich text and image modal information, which can effectively improve the accuracy of semantic segmentation.

[0074] S302. Perform segmentation and independent random masking processing on the multimodal data respectively to obtain the visible features and masking features of each modality.

[0075] One feasible implementation method involves performing piecewise processing and independent random masking on point cloud data to obtain visible features and masked features, including:

[0076] The point cloud data is sampled globally using the farthest point sampling algorithm to obtain multiple spatially uniformly distributed key points. A k-nearest neighbor query is performed on each key point to obtain multiple local patches of the point cloud. Some local patches of the point cloud are randomly selected from the multiple local patches and their feature information is masked to obtain the masked features of the point cloud data. The features of the remaining local patches of the point cloud are extracted to obtain the visible features of the point cloud data.

[0077] For example, for a 3D point cloud with an irregular structure, the Farthest Point Sampling (FPS) algorithm is first used to select k representative spatial keypoints from the original point cloud. The FPS algorithm ensures that the selected keypoints are evenly distributed on the object surface, and these points can be regarded as the "skeleton" of the point cloud. Then, with each keypoint as the center, the k-Nearest Neighbor (KNN) query algorithm is executed to query the multiple nearest neighbor points to that point, forming local patches of the point cloud. Each patch describes the geometric structure of a local region of the object.

[0078] For example, the number of local patches in the point cloud is equal to the number of keypoints selected by the FPS algorithm. For masking, the system randomly selects a subset of patches from this local point cloud using a randomly determined ratio (e.g., 15%–40%). For the selected patches, the coordinates of all points within them are removed from the encoder input and replaced with learnable mask token vectors. These masked patches are called mask features, and the model's goal during training is to predict the coordinates of these masked points based on other visible patches and other modal information.

[0079] For example, the remaining unselected point cloud patches are then used for feature extraction via a shared Multi-Layer Perceptron (MLP), mapping the 3D coordinates of points in each patch to a high-dimensional feature vector. These unmasked patches and their features constitute the visible features of the point cloud data. By dividing the point cloud data into these processable basic units and hiding a portion of them, a self-supervised learning task is constructed, enabling the model to learn the local geometry of the point cloud.

[0080] In this feasible implementation, by sampling the farthest point and querying the k nearest neighbors, irregular point cloud data can be transformed into structured and processable local patches. Randomly occluding some patches and enabling the model to make predictions based on the context is an effective self-supervised learning method. It enables the model to learn highly generalizable feature representations from local geometric structures, thereby improving the accuracy of model training.

[0081] One feasible implementation involves performing image data segmentation and independent random masking to obtain visible features and mask features, including:

[0082] The image data is normalized to a preset resolution to obtain normalized image data; the normalized image data is then divided into non-overlapping, equal grid segments to obtain multiple image sub-segments; from these multiple image sub-segments, image sub-segments with a preset ratio are randomly selected and their original pixel features are replaced with a unified mask token to obtain the mask features of the image data; the features of the remaining image sub-segments are extracted to obtain the visible features of the image data.

[0083] For example, for two-dimensional image data, preprocessing is first performed to normalize its size to a uniform resolution, such as 224×224 or 512×512 pixels. This normalization ensures that all input images have consistent sizes, facilitating subsequent batch processing and non-overlapping segmentation. After normalization, the image is uniformly divided into multiple identical, non-overlapping rectangular blocks, i.e., image sub-slices. For example, a 224×224 image can be divided into 16×16=256 image sub-slices of size 14×14. Each sub-slice represents a local region of the image.

[0084] For example, after obtaining multiple image sub-slices, the system defines a mask ratio, such as 50%. Based on this ratio, 128 sub-slices are randomly selected from 256. The original pixel values ​​of these selected sub-slices are completely discarded and uniformly replaced with a learnable mask token, which is a special vector with the same dimension as the sub-slice feature vector. After the replacement, these sub-slices become the mask features of the image data.

[0085] For example, the remaining unselected sub-slices retain their original pixel information. Next, the system uses a linear embedding layer to project each image sub-slice (whether visible or masked) into a one-dimensional feature vector, forming a sequence of feature vectors. The feature vectors corresponding to these unmasked sub-slices represent the visible features of the image data. For masked slices, their feature vectors are the vectors of the mask tokens.

[0086] In this feasible implementation, images are standardized and non-overlappingly segmented to adapt them to encoders based on the Transformer architecture. Randomized mask sub-slicing and replacement with uniform tokens is an effective means of creating self-supervised learning tasks, enabling the model to learn to understand the content of occluded regions using the overall contextual information of the image (i.e., visible slices), thereby extracting more semantically expressive image features and improving the accuracy of model training.

[0087] One feasible implementation involves performing segmentation and independent random masking on the text semantic data to obtain visible features and masked features, including:

[0088] The text semantic data is preprocessed by word segmentation to obtain a continuous text character sequence; the text character sequence is segmented into multiple text segments based on a single character or word as the basic unit; a preset number of text segments are randomly selected from the multiple text segments and their semantic features are masked to obtain the masked features of the text semantic data; the features of the remaining text segments are extracted to obtain the visible features of the text semantic data.

[0089] For example, for text data, standard tokenization preprocessing is first performed. The system uses a pre-trained tokenizer, such as BERT (Bidirectional Encoder Representation from Transformers), to segment the input natural language sentence (such as "This is a red car") into a sequence of individual words or subwords, such as ["this", "is", "a", "red", "of", "car"]. This sequence is a continuous sequence of text characters.

[0090] For example, based on a text character sequence, text segmentation involves treating each word or subword in the sequence as a text segment. Then, the system randomly selects a certain number (e.g., 15%) of the text segments and replaces the words at these positions with a special MASK token, thereby masking their semantic information and obtaining the masked features of the text semantic data.

[0091] For example, for text segments that are not selected, they retain their original semantic representations. During model processing, each segment (whether visible or occluded) is transformed into a feature vector through an embedding layer (Token Embedding Layer) and a positional embedding layer. The feature vectors corresponding to the remaining, unoccluded segments constitute the visible features of the text semantic data.

[0092] In this feasible implementation, by segmenting, slicing, and randomly masking the text sequence, the text model can predict the masked words by learning the visible words in the context, thereby gaining a deeper understanding of the syntax and semantic structure of the language and improving the accuracy of model training.

[0093] S303. Perform embedding dimension transformation and position encoding enhancement on the multimodal global features respectively, and fuse them to obtain an initial feature sequence carrying spatial location and semantic information.

[0094] For example, after the visible features of each modality are input into their respective encoders, an embedding dimension transformation is performed. This operation uses a learnable linear layer to transform the original feature dimensions of different modalities, such as the 3D coordinates of point cloud slices (after MLP mapping), the 3D pixel values ​​of image slices (after MLP mapping), and the word embedding vectors of text slices, into a unified preset dimension, such as 768 dimensions. In this way, the features of all modalities have a unified dimension in subsequent processing.

[0095] For example, positional encoding enhancement is a key step because the Transformer model itself does not have the ability to process sequence order and requires additional positional information to be injected.

[0096] Different types of positional encoding are used to better fit the inherent spatial properties of different modalities. For point cloud modalities, spatial relative positional encoding is used, which calculates a relative positional offset based on the three-dimensional spatial distance between the center points of the segments. This can better represent the local geometric relationships between point cloud segments.

[0097] For image modalities, two-dimensional absolute position coding is used, which assigns a unique position vector based on the row and column positions of image sub-pieces in their original grid to represent the spatial distribution of pixels.

[0098] For text modalities, one-dimensional temporal positional encoding is used, which assigns a position vector based on the order of words in a sentence to represent the word order and temporal relationship of the text.

[0099] Finally, the visible feature sequences of each of the three modalities are element-wise superimposed (fused) with their corresponding positional encoding features. For example, for point clouds, the feature vector of the i-th point cloud segment is added to its corresponding 3D spatial relative position encoding vector to obtain an initial feature vector carrying spatial position information. The initial feature vectors of all segments are arranged in order to obtain the initial feature sequence of each modality.

[0100] S304. Deep feature extraction and semantic fusion are performed on the initial feature sequence through a multi-layer encoder to obtain global features of a unified dimension.

[0101] For example, the resulting initial feature sequences for each modality are fed into the corresponding multi-layer Transformer encoder. These encoders are typically composed of multiple (e.g., 12 layers) identical Transformer blocks stacked together. The core of each Transformer block is a multi-head self-attention (MHSA) mechanism. The self-attention mechanism enables each segment in the sequence to pay attention to all other segments in the sequence and calculate their importance (attention weights) to the current segment.

[0102] In this way, the model can capture long-distance dependencies and global contextual information between elements within the feature sequence.

[0103] For example, through multi-layer encoder processing, the features of each slice are continuously enriched and fused, incorporating the semantic information of the entire modality. The encoder output is a feature sequence with the same length as the initial input feature sequence. This sequence contains the deep-processed features of each slice. Specifically, the encoder's input sequence typically includes a special classification token (CLS). According to the Transformer's design, the CLS token interacts with attention from all other slices, thus its final output feature vector is considered to encode the global semantic information of the entire modality, i.e., a global feature of uniform dimension. For example, assuming an embedding dimension of 768, this global feature would be a vector of length 768.

[0104] Similarly, the global features output by the encoders of all modalities are consistent in dimension (768-dimensional), which ensures that they can be compared by subsequent loss functions (such as contrastive loss). Although the dimension is unified, these features are not yet projected into the same semantic space; they are merely high-dimensional representations of their respective modalities.

[0105] S305. Perform dimension alignment and distribution calibration on global features of the same dimension through a linear mapping layer to map multimodal global features to a shared semantic space.

[0106] For example, to map the global features of the unified dimension to the shared semantic space, a separate linear mapping layer needs to be configured for each modality. This linear mapping layer is essentially a single-layer, learnable, fully connected neural network. The input to this layer is the global features (e.g., 768 dimensions), and the output is the features in the shared semantic space (the dimension can be the same as the input or different, e.g., 512 dimensions). Through this linear transformation, the feature distributions of each modality are mapped to a common, comparable space.

[0107] For example, dimension alignment is the most intuitive function of a linear mapping layer. It ensures that features from different modalities have exactly the same vector length (dimension) after mapping. Distribution calibration is a more core function of this mapping layer.

[0108] To illustrate with a scenario example, consider a "blue car" sample. Its point cloud global features are mapped to vector P, image global features to vector I, and text global features to vector T. In the shared space, the contrastive loss pushes these three vectors, P, I, and T, closer together. The linear mapping layer is the specific operation learned to achieve this "closer proximity." It adjusts the positions of P, I, and T so that they all face the region in the shared space representing the semantics of "blue car," thus completing the mapping from the original modality space to a unified semantic space.

[0109] Based on the above implementation, by performing embedding dimension transformation and positional encoding enhancement, and utilizing a multi-layer encoder for deep feature extraction, and finally performing dimension alignment and distribution calibration through a linear mapping layer, global features from different modalities are mapped to a unified shared semantic space. This process not only unifies the feature scale, but more importantly, through end-to-end contrastive learning, it eliminates distribution differences between different modalities, enabling features from different modalities with the same semantic meaning to approach each other within the shared space, thereby improving the accuracy of model training.

[0110] One feasible implementation involves performing embedding dimension transformation and positional encoding enhancement on the multimodal global features, respectively, using the following methods:

[0111] Embedding dimension transformation is performed on the visible features of each modality to obtain the embedding features of each modality. For the point cloud modality, spatial relative position encoding is used to fit the local geometric spatial positional relationship of the point cloud to obtain the position encoding features of the point cloud. For the image modality, two-dimensional absolute position encoding is used to represent the row and column spatial distribution information of image pixels to obtain the position encoding features of the image. For the text modality, one-dimensional temporal position encoding is used to represent the semantic order information of text characters to obtain the position encoding features of the text. The position encoding of each modality is fused with the corresponding embedding features element by element to perform position encoding enhancement.

[0112] For example, the visible features of each modality need to pass through an independent linear embedding layer to transform their original dimensions (e.g., 3D coordinates of point cloud slices, pixel values ​​of image slices, word embeddings of text) to a common, predefined dimension, such as 768 dimensions. The output is the embedding feature of each modality. This transformation brings all modalities into a unified feature channel, facilitating subsequent processing.

[0113] For example, for point cloud modalities, the spatial relative position encoding is implemented as follows: For any two point cloud local patches i and j, first calculate the 3D spatial coordinate difference (Δx, Δy, Δz) between their center points. Then, this 3D coordinate difference is encoded into a high-dimensional vector through a sinusoidal encoding function or a small MLP network. This vector is used as an attention bias and added to the Transformer's self-attention calculation. In this way, when the self-attention module calculates the attention weights between i and j, it depends not only on their feature content but also on their spatial distance. This allows for a more refined capture of the local geometry of the point cloud; for example, neighboring regions have stronger correlations than distant regions.

[0114] For example, the implementation of two-dimensional absolute position encoding for image modalities is very intuitive. The system generates a two-dimensional coordinate (r, c) for each image sub-slice, representing its position in the r-th row and c-th column of the image grid. This two-dimensional coordinate (r, c) is then mapped to a fixed sinusoidal coded vector or a learnable vector. This vector represents the absolute position of the slice. This encoding method directly reflects the grid spatial distribution of pixels in the image.

[0115] For example, similarly, one-dimensional temporal positional encoding works for text modalities. The system generates a one-dimensional vector based on the index position t of each word in the sentence, also using sinusoidal encoding or learnable parameters. This provides the model with the sequential relationships between words, which is fundamental to understanding natural language grammar and word order.

[0116] The generated positional encoding feature vector is added element-wise to the previously obtained embedded feature vector at the corresponding positions to enhance the encoding position.

[0117] In this feasible implementation, by using three different types of encoding methods—spatial relative, two-dimensional absolute, and one-dimensional temporal—to encode the characteristics of different modalities, the inherent spatial or temporal structural information of each modality's data can be simulated and injected more accurately. This allows the Transformer encoder to better understand the relative relationships between features, thereby extracting higher-quality, more spatially aware multimodal global features, thus improving the accuracy of model training.

[0118] S306. Based on the multimodal global features in the shared semantic space, determine the contrast loss between each pair of modalities. Based on the multimodal global features in the shared semantic space and the mask features of each modality, determine the mask reconstruction loss of each modality.

[0119] One feasible implementation method is to determine the contrast loss between each pair of modes using the following methods:

[0120] The multimodal global features in the shared semantic space are normalized to obtain normalized global features. Cross-modal features from homologous samples and semantically matched are identified as positive sample pairs, and cross-modal features from non-homologous samples and semantically unmatched are identified as negative sample pairs. The contrast loss between each pair of modalities is determined by calculating the feature similarity difference between positive and negative sample pairs.

[0121] For example, before performing contrastive learning in a shared semantic space, all multimodal global features extracted from that space are normalized using the L2 norm.

[0122] Optionally, point cloud global features and semantic global features (semantic global features can include global image features and global text features) can be used as input. Each input consists of B triples. The representation vectors of the B triples are respectively... The corresponding text representation vector is The corresponding image representation vector is Normalize these triples using the 2-norm, i.e.:

[0123]

[0124]

[0125]

[0126] in, yes The One element, yes The One element, yes The One element, , , All dimensions are .

[0127] Optionally, in the same batch of inputs, , , These are the corresponding positive sample triples. The contrast loss between the point cloud and the text is:

[0128]

[0129] Optionally, the contrast loss between the point cloud and the image is:

[0130]

[0131] Optionally, the contrast loss between text and image is:

[0132]

[0133] in, This represents the contrast loss between the point cloud and the text. This represents the contrast loss between the point cloud and the image. This represents the contrast loss between text and images. This represents the inner product similarity, and τ is a preset constant used to adjust the smoothness of the softmax curve.

[0134] For example, consider a batch containing N triples (point cloud, image, text). In this batch, each sample has one positive sample pair. For instance, for the point cloud features of sample i, the corresponding image features are its positive samples, while the image features of the other N-1 samples in the same batch are its negative samples. Similarly, the text features are treated similarly. Thus, for the point cloud-image modality, there are N positive sample pairs and N×(N-1) negative sample pairs.

[0135] For example, the contrast loss is calculated using the InfoNCE loss method. Taking the point cloud-image contrast loss as an example, the calculation process is as follows: For each point cloud feature, calculate its similarity score with all image features. These scores are then transformed into a probability distribution using a Softmax function with a temperature parameter τ. The i-th term in this probability distribution, i.e., the probability of a positive sample pair, should be maximized. The average of this loss over all N point cloud features yields the contrast loss for the "point cloud-image" modality pair in that batch. Similarly, the contrast losses for "point cloud-text" and "image-text" can be calculated.

[0136] In this feasible implementation, by normalizing features in the shared semantic space and calculating contrastive loss, it is possible to efficiently bring different modal features (positive sample pairs) from the same object closer together in space, while pushing modal features (negative sample pairs) from different objects further apart. This method effectively achieves global-level semantic alignment, enabling the model to learn modality-insensitive, highly abstract semantic representations, thereby improving the accuracy of model training.

[0137] One feasible implementation method is to calculate the feature similarity difference between positive and negative sample pairs to determine the contrastive loss, including:

[0138] Based on positive and negative sample pairs, the feature similarity of each positive sample pair and the feature similarity of each negative sample pair are calculated to obtain multiple sets of similarity measurement results; the difference between the similarity of homologous matching samples and the similarity of non-homologous mismatched samples is compared to obtain the feature difference value between modes; based on the feature difference value between modes, the contrast loss between each pair of modes is determined.

[0139] For example, when determining the specific calculation of the contrast loss, the core is to quantify the difference in similarity between positive and negative sample pairs. Using the normalized feature vectors, the cosine similarity between all sample pairs (including positive and negative examples) is calculated. For a batch size of N, an N×N similarity matrix S is calculated, where S[i][j] represents the similarity between the i-th point cloud feature and the j-th image feature. This matrix contains multiple sets of similarity quantification results.

[0140] For example, ideally, the similarity of the diagonal elements of matrix S (S[0][0], S[1][1], etc.), i.e., the positive sample pairs, should be much greater than the similarity of the off-diagonal elements (negative sample pairs). The core idea of ​​calculating the loss is to compare these differences.

[0141] Specifically, it calculates the proportion of the diagonal elements (positive sample similarity) in each row (or column) of the entire similarity matrix to obtain the probability value of the difference between the similarity of homologous matching samples and the similarity of non-homologous mismatched samples.

[0142] The closer this probability value is to 1, the more overwhelmingly positive samples have similarity compared to negative samples within a row, indicating a large difference in features between modalities and a smaller loss value. Conversely, if this probability value is close to 1 / N, it indicates that the model has failed to distinguish between positive and negative samples, with very small differences in features between modalities and a large loss value.

[0143] Minimizing this loss using gradient descent directly drives the model to learn how to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, thereby clearly clustering the features of each modality in the shared semantic space.

[0144] In this feasible implementation, the alignment between modalities is measured by calculating the difference in similarity between positive and negative sample pairs, and a contrastive loss function is constructed based on this. This directly and effectively drives the model to learn a highly discriminative shared semantic space. This process forces the model to find aggregation points for multiple modalities of each sample in the shared space, thereby achieving global semantic alignment and improving the accuracy of model training.

[0145] One feasible implementation method is to determine the mask reconstruction loss for each modality using the following approach:

[0146] Based on the multimodal global features in the shared semantic space, predictive reconstruction features of each modality mask region are generated by a multimodal decoder; the predictive reconstruction features are compared with the mask features of each modality in a dimensional feature manner to obtain the reconstruction error of each modality; and the mask reconstruction loss of each modality is determined based on the reconstruction error of each modality.

[0147] For example, when performing mask reconstruction, a multimodal decoder is first required. The design of this decoder can differ for different modalities.

[0148] For example, for text modalities, the decoder might be a Transformer decoder; for image and point cloud modalities, it might be a Transformer encoder or decoder. What they have in common is that the input not only contains the visible features from the encoder output of their own modality, but also global features from the other two modalities in a shared semantic space.

[0149] For example, in point cloud mask reconstruction, the decoder's input includes: features of visible point cloud segments output by the point cloud encoder, global features of the text modality, and global features of the image modality. The decoder's role is to synthesize this information to predict the features of the masked segment regions.

[0150] For example, the output of the decoder is a series of feature vectors that correspond to the masked regions that were occluded, i.e., the predicted reconstructed features.

[0151] For example, for the k-th image sub-slice that is masked, the decoder outputs a prediction vector pred_k, whose dimension is the same as the original embedding features of that sub-slice. Next, the difference between the predicted reconstructed features and the ground truth, i.e., the reconstruction error, needs to be calculated. For text modality, the ground truth is the word embedding vector corresponding to the masked word; for image modality, the ground truth is the original pixel value vector of the masked sub-slice; for point cloud modality, the ground truth is the three-dimensional coordinate vector of all points in the masked sub-slice.

[0152] For example, dimensional feature alignment involves calculating the difference between the predicted value `pred_k` and the ground truth value. This difference can be quantified in several ways. For continuous values ​​(image pixels, point cloud coordinates), mean squared error (MSE) loss can be used. For discrete values ​​(text words), cross-entropy loss can be used. The calculated difference value is the reconstruction error of that modality. By summing the errors of all masked regions, the overall mask reconstruction loss of that modality is obtained.

[0153] Optionally, the internal structure of the text decoder, image decoder, and point cloud decoder in the point cloud multimodal model can be modeled using a Transformer encoder, a Transformer decoder, or an alternating approach of both. Specifically, when using an alternating approach, for example, using a Transformer encoder in layer l and a Transformer decoder in layer l+1, it is possible to better perform fine-grained information fusion between the three modalities and to utilize information from other modalities to guide the mask reconstruction of the target modality. For example, this flexible decoder design allows the model to select the most suitable structure for cross-modal interaction based on different data characteristics and task requirements.

[0154] The implementation steps of a Transformer encoder are described below:

[0155] The bidirectional text encoding layer is composed of several (W in this embodiment) identical Transformer encoders stacked in series. The Transformer encoder consists of two substructures: a multi-head self-attention module (MHSA) and a feedforward neural network module (FFN). The output of the former is the input of the latter. In addition, each substructure also includes a residual connection module and a layer normalization module.

[0156] Multi-head self-attention module. The input to the k-th layer Transformer encoder is The processing method is the same for each layer of the Transformer encoder, so for ease of description, we will temporarily ignore k in the explanation of the Transformer encoder. Now, let... , 1≤i≤M. In a single-head self-attention module, the vector Transformation matrices of query values Key value transformation matrix and embedding value transformation matrix Generate query value Key value and feature embedding values In the same Transformer encoder, all Share a group , and ,and , and The element values ​​are learned during model training, that is:

[0157]

[0158] Where 0≤i≤M, and They represent dimensions respectively. and The vector space; the dimensions of the query value and the key value are the same (both are vector spaces). Their dimensions can be the same as the dimension of the feature embedding values. ), or they can be different ( In the embodiments of this application, .

[0159] Secondly, for the query value With key value Perform a matching calculation, that is, for any query value and key value Calculate the dot product value to obtain M dot product values:

[0160]

[0161] in, represent The transpose vector ( It is a column vector. (For row vectors). Perform the dot product on the above values. Scaling, to obtain ( ), at this time .

[0162] Meanwhile, this invention defines the mask matrix used in the action representation coding layer as follows: to indicate and Whether attention calculations are performed, i.e.:

[0163]

[0164] In the action coding layer, and Attention calculations are performed on all of them, that is ( ).

[0165] Then, the softmax operation is used to convert the dot product and mask values ​​into probability values. ,Right now:

[0166]

[0167] Where 1≤i, j≤M.

[0168] Depend on As can be seen from the softmax of multi-head self-attention, when the Transformer encoder encodes an action sequence, any two actions in the action sequence will perform attention calculations on each other.

[0169] Finally, calculate Corresponding feature embedding ,as follows:

[0170]

[0171] so, This is the output of a single-head self-attention module, where each element... Corresponding to the input of this module .

[0172] A multi-head attention module with H heads performs the single-head attention module process in parallel H times, concatenating the results o_i of each x_i and then performing a linear mapping before outputting the result. Specifically, if the output of the h-th head is... Then the i-th element of the multi-head attention output for:

[0173]

[0174] in, The dimension is d, and The dimension d remains consistent, that is The size is .

[0175] so, This is the output of the multi-head self-attention module, where each element... Corresponding to the input of this module Then, residual joins and layer normalization are performed sequentially, as follows:

[0176] The residual connection is:

[0177]

[0178] Layer normalization layer is for any The following formula is used for calculation:

[0179]

[0180] Where μ is The mean of all elements in the set, i.e. ,here express The k-th element in the matrix. σ is the mean square error, i.e. γ and β are learnable parameters.

[0181] so,( It is the output of the multi-head self-attention module, which is then input into the feedforward neural network module, where each element Corresponding to the input of this module .

[0182] The input to the feedforward neural network module is the output of the multi-head self-attention module. After passing through two layers of multilayer perceptrons, residual connections and layer normalization are performed before the output is obtained. Specifically:

[0183]

[0184]

[0185] Here, `acti` is the activation function of the perceptron, which can be chosen from sigmoid, tanh, ReLU, GELU, and other similar functions. and The dimensions are respectively and LayerNorm is a layer normalization operation, and... The mechanism of the formula is similar, so I will not go into details.

[0186] so,( It is the output of the feedback neural network module, and also the action representation output by the action coding layer, where each element Corresponding to the input of this module ,Right now that is These are also the inputs to the (k+1)th Transformer coding layer.

[0187] The multi-head self-attention module in the Transformer encoder performs self-attention fusion between text and image representations and point cloud segmentation representations, enabling text and image representations to guide point cloud mask segmentation reconstruction.

[0188] The specific structure of the Transformer decoder is as follows:

[0189] The Transformer decoder consists of three parts: a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward neural network module (which includes a cross-attention module, unlike the Transformer encoder). These three modules are stacked in series, with the output of each module serving as the input to the next module. The output of the final feedforward neural network module serves as the input to the multi-head self-attention module of the next Transformer decoder layer. Each module is as follows:

[0190] Multi-Head Self-Attention module.

[0191] This module has the same structure as the multi-head attention module in the action coding layer that encodes action sequences, so it will not be repeated here.

[0192] Multi-Head Cross-Attention module.

[0193] The structure of this module is similar to the multi-head self-attention module described above, and the structure of each single head is also similar, but the processing of the input is different. Specifically, assume that the input of the k-th layer single-head cross-attention is... and At this time, let , 1≤i≤N, let , 1≤j≤M.

[0194] In the single-head cross-attention module, vector After query value transformation matrix Generate query value ,vector After key value transformation matrix and embedding value transformation matrix Generate key-value pairs and feature embedding values In the single-head cross-attention module of the same Transformer decoder, all Share a group ,all Share a group and ,and , and The element values ​​are learned during model training, that is:

[0195]

[0196] Where 1≤i≤N, 1≤j≤M, and They represent dimensions respectively. and The vector space.

[0197] The rest is similar to the single-head self-attention module and will not be repeated here.

[0198] Feed-Forward Network module.

[0199] This module has the same structure as the feedback neural network module in the Transformer encoder, so it will not be repeated here.

[0200] In this feasible implementation, a multimodal decoder guides the reconstruction of each modal mask region using global features from a shared semantic space, and the reconstruction results are compared with the original ground truth to calculate the loss. This is an effective fine-grained alignment method. It forces the model to do more than passively match global features; instead, it requires the model to utilize cross-modal information to supplement missing local details, thereby driving the model to deeply understand the correspondences between different modal information at a micro-level and improving the accuracy of model training.

[0201] One feasible implementation method is to determine the mask reconstruction loss for each modality using the following approach:

[0202] Based on the data characteristics of each modality, the corresponding loss calculation rules are determined; based on the loss calculation rules, the reconstruction error of each modality is calculated to obtain the mask reconstruction loss of each modality.

[0203] For example, data from different modalities have drastically different characteristics, so their corresponding loss calculation rules also need to be tailored. For the text modality, the data consists of discrete words, and its reconstruction task is essentially a classification problem. Therefore, the loss calculation rule is usually cross-entropy loss. The output of the text decoder (a high-dimensional vector) is mapped to the dimension of the vocabulary size through a linear layer to obtain a logits vector. Then, this logits vector is cross-entropy calculated with the one-hot encoded vectors of the true mask words. The smaller the loss, the more accurate the predicted words.

[0204] For example, in image modalities, the data consists of continuous pixel values, and the reconstruction task is a regression problem. Therefore, the loss calculation rule is typically mean squared error loss. The system maps the output of the image decoder (a high-dimensional vector) through a linear layer to data of the same size as the pixel matrix of the original masked image sub-segments (e.g., a tensor of size 32×32×3). Then, this predicted tensor is compared with the true pixel tensor of the masked region for pixel-by-pixel MSE calculation; the smaller the loss, the closer the reconstructed pixels are to the true values.

[0205] For example, for point cloud modalities, the data consists of continuous 3D coordinates, and the reconstruction task is also a regression problem. The output of the point cloud decoder is passed through an MLP header to predict the 3D coordinates (x, y, z) of all points in each mask slice, forming a point set. Then, the Chamfer Distance is used as the loss function to calculate the average nearest distance between the predicted point set and the ground truth point set. The smaller this distance, the more accurate the reconstructed point cloud geometry.

[0206] In this feasible implementation, selecting appropriate loss calculation rules based on the data characteristics of text, images, and point clouds can more accurately measure the reconstruction quality of each modality, thereby guiding the decoder network to learn the generation patterns of specific modalities more effectively and improving the accuracy of model training.

[0207] S307. Determine the weighting coefficients corresponding to the contrast loss between each mode and the mask reconstruction loss of each mode.

[0208] For example, the core of constructing a joint loss function is determining the weight coefficient for each loss term. For instance, we have six independent loss values: three contrastive losses (point cloud-image, point cloud-text, image-text) and three mask reconstruction losses (point cloud reconstruction, image reconstruction, text reconstruction). These loss values ​​may differ significantly in numerical magnitude and training importance. Therefore, a weight coefficient needs to be assigned to each loss term, and the final total loss is the weighted sum of these losses.

[0209] For example, the weight coefficients can be set statically or dynamically. One static method is to set them manually based on experience. For instance, in the early stages of training, to quickly establish global alignment, the weights of the contrastive loss can be set larger, while the weights of the reconstruction loss can be set smaller. Another dynamic method is to treat the weight coefficients as learnable parameters that are automatically optimized during model training.

[0210] For example, an uncertainty weighting method can be used, where the uncertainty of each task is used as its weight. The model can automatically learn to assign higher weights to simpler and more stable tasks, and lower weights to more difficult tasks.

[0211] S308. Based on the weighting coefficients, the contrast loss between each mode and the mask reconstruction loss of each mode are weighted and summed to obtain the joint loss function.

[0212] Optionally, the joint loss function can be expressed by the following formula:

[0213]

[0214] in, Denotes the joint loss function. This represents the weighting coefficients.

[0215] Based on the above implementation methods, a joint loss function is constructed by weighting and summing different loss terms, enabling the model to simultaneously optimize multiple tasks with different properties under a unified objective. This multi-task training approach overcomes the overfitting or local optima problems that may be caused by a single loss function, allowing the model to achieve good performance at both the global alignment and local reconstruction levels, thereby improving the training accuracy of the model.

[0216] One feasible implementation method is to determine the weight coefficients of each loss term using the following methods:

[0217] Determine the current training phase; based on the current training phase, determine the weight coefficients.

[0218] For example, the determination of weight coefficients can be dynamically correlated with the current training phase. For instance, at the start of training, the model's ability to extract features from any modality is weak, so global alignment can be prioritized. Therefore, the weights of the contrastive loss can be set to higher initial values, such as 1.0, while the weights of the all-mask reconstruction loss can be set to lower initial values, such as 0.1.

[0219] As training progresses, the model's global alignment capability is gradually established. When training enters the mid-stage, the weight of the mask reconstruction loss can be gradually increased, for example, the weight of the point cloud reconstruction loss can be linearly increased from 0.1 to 0.5, to encourage the model to start paying attention to cross-modal reconstruction of local details.

[0220] As training progresses and the model gains a good understanding of both global and local capabilities, the weight of the contrastive loss can be slightly reduced to balance the two aspects.

[0221] In this feasible implementation, by dynamically adjusting the loss weights according to the different training stages of the model, the learning process of the model can be made more robust, thereby improving the accuracy of model training.

[0222] One feasible implementation method is to perform iterative training using the following approaches:

[0223] Based on the joint loss function, the total loss value for multiple tasks is determined; based on the total loss value for multiple tasks, the parameters of the multimodal encoder-decoder network are iteratively updated; the parameter iteration process is corrected through a gradient pruning mechanism; the accuracy indicators of the model are monitored in real time, including the multimodal feature alignment accuracy and mask reconstruction accuracy; until the accuracy indicators reach the preset convergence value, the point cloud multimodal model is obtained.

[0224] Below, in conjunction with Figure 5 The training of point cloud multimodal models is explained.

[0225] Figure 5 This is a schematic diagram illustrating the training of a point cloud multimodal model provided in an embodiment of this application. Figure 5 As shown, the model's input is a point cloud. and the corresponding text T (i.e., text sequence) The training of the deep learning neural network employs batch training, where the model is input with B triples of point cloud, text, and image (B is a predefined constant). The point cloud, text, and image triples (P, T, I) in each batch are sequentially paired. Each point cloud, along with its corresponding text and image, forms a positive sample triple, and together with the other B-1 text and image pairs in the batch, forms B-1 negative sample triples. The training process is described for a single sample triple and does not affect the overall model architecture.

[0226] The pre-trained model for point cloud reconstruction by mask segmentation consists of three parts: (1) a text, image and point cloud encoder module, which performs semantic representation learning on text, image and point cloud respectively; (2) a contrast learning module, which aligns text, image and point cloud representations to a unified semantic space to achieve object-level semantic alignment; and (3) a text, image and point cloud decoder module, which reconstructs text mask, image mask segmentation and point cloud mask segmentation in a distributed manner to achieve fine-grained semantic alignment at the segment level.

[0227] The encoder is a feature extraction network used to transform and filter raw data, removing redundant information and converting structurally dissimilar data such as point clouds, images, and text into fixed-dimensional, comparable feature vectors. It projects heterogeneous modal information onto a shared semantic space, producing global features for cross-modal similarity matching.

[0228] The decoder is a feature reconstruction network used to predict patch features that are randomly masked during training, utilizing global semantic information. It generates predicted features for the masked regions, compares these features with the actual masked features to obtain the reconstruction error, and forms the mask reconstruction loss, thus supervising the model's learning of modal local details.

[0229] The model is explained in detail below:

[0230] The input to the text encoder is a sequence of text in triples. ,Right now ,in It is replaced (covered) by a text mask, and will be predicted and reconstructed in the text mask reconstruction loss. First, each word in the text sequence is converted into a preset dimension. The dense vector representation of , i.e., the th text Transformed into a dense vector representation The entire text sequence is converted to That is, we have the following formula:

[0231]

[0232] in, , It is a representation of all words in the text embedding layer. The dimension is . yes True embedded representation, yes The masked embedding representation is 1, and the rest are visible embedding representations. Assume there are... If a mask is embedded in the representation, then... A visible embedded representation.

[0233] Then, this layer is for each dense vector of the text sequence. Add position vector This helps to identify each word. Position within a sequence better expresses the sequential relationship between words. Specifically:

[0234]

[0235] in, , and All dimensions are Position vector It can be set to a fixed value or a learnable model variable. Additionally, a CLS vector is added to the text sequence. And add the position vector That is, we get:

[0236]

[0237] in, , and The dimension is .

[0238] At this point, the text sequence is: and The result of concatenating the visible embeddings in the sequence is denoted as . ,Right now The text encoding layer consists of multiple Transformer encoders. The first subscript The second subscript refers to the first layer input of the Transformer encoder. Refers to the first of this layer The sequence number of each word.

[0239] The input to the text Transformer encoder is This layer has Layers, each modeled using a Transformer encoder, the first layer... The input of the layer is Layer output , No. Layer output is As shown below:

[0240]

[0241]

[0242] in, , The first element is the text CLS vector. It models the semantic information of the entire text sequence. Therefore, the output of the text encoding layer is... For ease of narration, I will record... for .

[0243] The input to an image encoder is an image in triplets. First, scale the image to a uniform size, then divide it into equal-sized slices, such as those with different resolutions. The image is divided into 256 Size of the slice (in, (3 refers to RGB, which are the three color channels), thus forming an image sequence. Simultaneously, we divide it into several slices. Perform a masking operation, that is, replace it with a uniform mask token. Assume there is If a mask is embedded in the representation, then... Each visible embedding representation is then mapped using an MLP. dimensional vector, that is:

[0244]

[0245] At the same time, a CLS vector is added to the image sequence. And add the position vector This helps determine the position of each image slice in the sequence. The position vector can be set to a fixed value or a learnable model variable, thus:

[0246]

[0247] in, , and The dimension is .

[0248] At this point, the image sequence is: and The result of splicing together the visible representations in the sequence (for ease of subsequent description, ... Recorded as ), denoted as ,Right now .

[0249] The image coding layer consists of multiple Transformer encoders. The first subscript The second subscript refers to the first layer input of the Transformer encoder. Refers to the first of this layer The sequence number of each image slice.

[0250] The input to the image encoder is This layer has Layers, each modeled using a Transformer encoder, the first layer... The input of the layer is Layer output , No. Layer output is As shown below:

[0251]

[0252]

[0253] in, , The first element is the image CLS vector. It models the semantic information of the entire image sequence. Therefore, the output of the image coding layer is... For ease of narration, I will record... for .

[0254] The input to the point cloud encoder is the point cloud in the triplet. First, the point cloud needs to be sliced ​​and then input into a multi-layer Transformer encoder for processing. Point cloud slicing refers to the process of dividing large-scale point cloud data into small, local regions, which are called slices or blocks. This is a fundamental operation in point cloud processing. For example, a classic point cloud slice generation process first uses the farthest point sampling (FPS) method to extract slices from the original point cloud. Selected from The initial set consists of representative points. Then, for each sampling point, through Nearest neighbor query retrieves its nearest neighbor. 10 neighboring points, eventually forming A point cloud segment with good spatial distribution.

[0255] Farthest point sampling is an iterative selection of points. Add to collection ( In the initial empty set, that is:

[0256]

[0257] For each sampling point Its KNN partitioning definition is:

[0258]

[0259] in, This indicates sorting by distance.

[0260] We start from N fragments Random selection Each fragment is used as a mask fragment. The remaining Each fragment is used as a visible fragment. Each shard has There are 10 points, each with 3-dimensional coordinates, which can be stretched into... A dimensional vector, which is the representation vector for each piece. Then each shard is mapped to an MLP. dimensional vector, that is:

[0261]

[0262] At the same time, a CLS vector is added to the point cloud sequence. And add the position vector That is, we get:

[0263]

[0264] in, , and The dimension is .

[0265] At this point cloud sequence is and The visible mask in the sequence represents the concatenated result (for ease of subsequent description, it will be...). Recorded as ), denoted as ,Right now .

[0266] The point cloud encoding layer consists of multiple Transformer encoders. The first subscript The second subscript refers to the first layer input of the Transformer encoder. Refers to the first of this layer The sequence number of each point cloud segment.

[0267] The input of the point cloud encoder is This layer has Layers, each modeled using a Transformer encoder, the first layer... The input of the layer is Layer output , No. Layer output is As shown below:

[0268]

[0269]

[0270] in, , The first element is the image CLS vector. It models the semantic information of the entire image sequence. Therefore, the output of the point cloud coding layer is... For ease of narration, I will record... for .

[0271] The input to the text decoder is the output of the text encoder. Image representation Point cloud representation and text mask token vector The core goal of a text decoder is to leverage image representation. Point cloud representation By combining joint semantic guidance with information from the visible parts of the text, a text mask can be reconstructed. In this way, by utilizing the complementary information of the image and point cloud, the image point cloud modality and the text modality can be aligned at a fine-grained level. Right now ,in That is .

[0272] Will , , and The concatenated text is input into the text decoder, where the number of text mask fragments is... One, so Copying is required. A portion, recorded as Therefore, the input to the text decoder is For better narration below, it will be relabeled as .

[0273] Text decoders have Layers can be modeled using a Transformer encoder, a Transformer decoder, or a combination of both, which can be used alternately. This will be explained in detail below.

[0274] (1) Model using Transformer encoder.

[0275] Transformer encoder The input of the layer is the first Layer output The output is .

[0276]

[0277]

[0278] so This is the output of the text decoder.

[0279] (2) Model using Transformer decoder.

[0280] Transformer decoder The input of the layer is the first Layer output The output is .

[0281]

[0282] .

[0283] so This is the output of the text decoder.

[0284] (3) Use Transformer encoder and Transformer decoder to model alternately.

[0285] Alternating modeling is achieved using a Transformer encoder and a Transformer decoder; that is, a Transformer encoder layer is followed by a Transformer decoder layer. The input to the layer Transformer decoder is the first layer. Output of the layer Transformer encoder The output is . No. The input to the layer Transformer encoder is the first layer. The output of the layer Transformer decoder The output is ,in .

[0286]

[0287]

[0288]

[0289]

[0290] .

[0291] so This is the output of the text decoder. By alternating modeling with the Transformer encoder and Transformer decoder, fine-grained fusion between the three modalities and guided reconstruction of the text mask using image point cloud information can be achieved more effectively.

[0292] The dimensions represented above are all .

[0293] The input to the text mask reconstruction layer is the output of the text decoding layer. The text mask reconstruction layer can obtain the mask representation using a simple MLP head, then perform an inner product with the embedding vectors of all words, and finally calculate the text mask reconstruction loss using cross-entropy loss, i.e.:

[0294]

[0295]

[0296]

[0297] in, It is a vector dot product operation. , and The size is .

[0298] By using text mask reconstruction loss, fine-grained (piece-level) semantic alignment between text modalities and image point cloud modalities can be achieved.

[0299] The input to the image decoder is the output of the image encoder. (Right now ,in That is Text representation Point cloud representation and image mask token vector The core goal of an image decoder is to utilize text representations. Point cloud representation By combining the joint semantic guidance with information from visible image segments, image mask segments can be reconstructed. In this way, by utilizing the complementary information of text and point clouds, text point cloud modalities and image modalities can be aligned at a fine-grained level.

[0300] Will , , and After concatenation, the data is input into the image decoder, where the number of image mask fragments is... One, so Copying is required. A portion, recorded as Therefore, the input to the image decoder is For better narration below, it will be relabeled as .

[0301] Image decoders have Layers can be modeled using a Transformer encoder, a Transformer decoder, or a combination of both, which can be used alternately. This will be explained in detail below.

[0302] (1) Model using Transformer encoder.

[0303] Transformer encoder The input of the layer is the first Layer output The output is .

[0304]

[0305]

[0306] .

[0307] so This is the output of the image decoder.

[0308] (2) Model using Transformer decoder.

[0309] Transformer decoder The input of the layer is the first Layer output The output is .

[0310]

[0311]

[0312] .

[0313] so This is the output of the image decoder.

[0314] (3) Use Transformer encoder and Transformer decoder to model alternately.

[0315] Alternating modeling is achieved using a Transformer encoder and a Transformer decoder; that is, a Transformer encoder layer is followed by a Transformer decoder layer. The input to the layer Transformer decoder is the first layer. Output of the layer Transformer encoder The output is . No. The input to the layer Transformer encoder is the first layer. The output of the layer Transformer decoder The output is ,in .

[0316]

[0317]

[0318]

[0319]

[0320]

[0321] so This is the output of the image decoder. By alternating modeling with the Transformer encoder and Transformer decoder, fine-grained fusion between the three modalities and guided reconstruction of image mask segments using text point cloud information can be achieved more effectively.

[0322] The dimensions represented above are all .

[0323] The input to the image mask reconstruction layer is the output of the image decoding layer. The image masking layer can use a simple MLP header to obtain the predicted pixel values ​​of the mask pieces, and then use MSE to calculate the image masking reconstruction loss by comparing them with the actual pixel values.

[0324]

[0325]

[0326] in, It is a vector dot product operation. , and The size is .

[0327] By using image masking piecewise reconstruction loss, fine-grained (piecewise) semantic alignment between image modalities and text point cloud modalities can be achieved.

[0328] The input to the point cloud decoder is the output of the point cloud encoder. (Right now ,in That is Text representation Image representation Point cloud mask token vector The core objective of a point cloud decoder is to leverage text representation. With image representation By combining the joint semantic guidance with the information from visible point cloud segments, point cloud mask segments are reconstructed. In this way, by utilizing the complementary information of text and images, text-image modalities and point cloud modalities can be aligned at a fine-grained level.

[0329] Will , , and After concatenation, the data is input into the point cloud decoder. The number of point cloud mask fragments is... One, so Copying is required. A portion, recorded as Therefore, the input to the point cloud decoder is For better narration below, it will be relabeled as .

[0330] Point cloud decoder has Layers can be modeled using a Transformer encoder, a Transformer decoder, or a combination of both, which can be used alternately. This will be explained in detail below.

[0331] The input to the point cloud masking and fragmentation reconstruction layer is the output of the point cloud decoding layer. The point cloud mask piece reconstruction layer can use a simple MLP header to directly predict the coordinates of points in each mask piece.

[0332]

[0333] in , The size is Convert its data The data in row 3 and column 3 gives the coordinates of the mask fragments. The size is ,express The three-dimensional coordinates of each point.

[0334] We can use Chamfer Distance to calculate mask fragmentation. and mask prediction This distance is calculated as the average nearest distance between two point sets, which is the "point-to-set" distance.

[0335]

[0336] By using point cloud mask piecewise reconstruction loss, fine-grained (piecewise) semantic alignment between point cloud modalities and text image modalities can be achieved.

[0337] Below, in conjunction with Figure 6 The training and fine-tuning of point cloud multimodal models are explained.

[0338] Figure 6 This is a schematic diagram illustrating the training and fine-tuning of a point cloud multimodal model provided in an embodiment of this application. For example... Figure 6 As shown, in the pre-training phase, point cloud, text, and image triples are input into the multimodal pre-trained model. Adaptive weights are used for multi-task, multi-objective training using contrastive learning loss, point cloud mask reconstruction loss, text mask reconstruction loss, and image mask reconstruction loss objective functions, thereby aligning and fusing the point cloud, text, and image modal spaces. In the fine-tuning phase, the pre-trained model can be transferred to downstream tasks, such as cross-modal point cloud retrieval, point cloud category recognition, and point cloud semantic segmentation, for fine-tuning training.

[0339] The following examples illustrate three downstream tasks: point cloud cross-modal retrieval, point cloud recognition, and point cloud segmentation.

[0340] 1. Point cloud cross-modal retrieval task.

[0341] The point cloud multimodal retrieval task enables mutual retrieval of point clouds, text, and images. Retrieval tasks are categorized based on whether model fine-tuning is required: zero-shot retrieval (no fine-tuning required) and fine-tuned model retrieval.

[0342] Zero-sample retrieval.

[0343] After pre-training, we sequentially input the point cloud, text, and image triples (P, T, I) from the dataset to be retrieved into the point cloud encoder, text encoder, and image encoder of the pre-trained model to obtain their representation vectors. and , where 1≤k≤N, and N is the number of triples in the dataset. After normalization, the similarity scores between each pair of elements are calculated:

[0344]

[0345] in, Represents the similarity score. Indicates the source modality of the retrieval. Indicates the target retrieval modality. , where k represents the sample index number.

[0346] Thus, for retrieval modalities A sample s and the modality to be retrieved Given all samples, such as searching for the image most similar to a given point cloud sample *s* within all samples of an image modality, similarity can be evaluated. Sort the results (where 1≤k≤N) from largest to smallest and return the results with the highest similarity.

[0347] Fine-tuning the model retrieval.

[0348] After pre-training, the point cloud, text, and image triplet data from the training set of the fine-tuning dataset are sequentially input into the model for fine-tuning training (the process is similar to the pre-training process). When the fine-tuned model meets the preset convergence conditions, such as the number of training rounds reaching a limit or the loss function value converging, training is stopped, and the point cloud multimodal fine-tuned model is obtained.

[0349] The point cloud, text, and image triplet data (P, T, I) from the test set (i.e., the dataset to be retrieved) of the fine-tuning dataset are sequentially input into the point cloud encoder, text encoder, and image encoder of the fine-tuning model to obtain their representation vectors. and , where 1≤k≤N, and N is the number of triples in the dataset. After normalization, the similarity scores between each pair of elements are calculated:

[0350]

[0351] Thus, for retrieval modalities A sample s and the modality to be retrieved For example, in all samples of an image modality, the image most similar to a point cloud sample can be retrieved, and the similarity can be evaluated. Sort the results (where 1≤k≤N) from largest to smallest and return the results with the highest similarity.

[0352] Through the two explanations above, cross-modal retrieval with zero samples and fine-tuning models can be achieved, namely, cross-modal retrieval between point cloud and image, image and point cloud, point cloud and text, text and point cloud, image and text, and text and image.

[0353] 2. Point cloud recognition is the task of recognizing and classifying point clouds. Recognition tasks are divided into zero-shot recognition retrieval (no fine-tuning required) and fine-tuned model recognition, depending on whether model fine-tuning is needed.

[0354] Zero-shot identification.

[0355] After pre-training, we will identify the point cloud. Input the point cloud encoder of the pre-trained model to obtain its representation vector. At the same time, construct a category dataset. (n is the number of categories), such as {apple, car, ..., water cup}, categorizing the text. The text is sequentially input into the pre-trained model's text encoder to obtain the text representation. (where 1≤k≤n). [The following is a list of options:] ... and After normalization, the similarity scores between each pair of elements are calculated:

[0356]

[0357] In this way, similarity can be measured. Sort the texts in descending order (where 1≤k≤n) and return the text categories with the highest similarity.

[0358] Fine-tuning the model for recognition.

[0359] After pre-training, the point cloud, text, and image triplet data (P, T, I) of the training set of the fine-tuning dataset are sequentially input into the model for fine-tuning training (the process is similar to the pre-training process). When the fine-tuned model meets the preset convergence conditions, such as the number of training rounds reaching a limit or the loss function value converging, training is stopped, and the point cloud multimodal fine-tuned model is obtained.

[0360] The point clouds in the test set (i.e., the point cloud dataset to be recognized) of the fine-tuned dataset are used to refine the dataset. Input the point cloud encoder of the fine-tuning model to obtain its representation vector. At the same time, the text data in the test set... (n is the number of text categories), such as {apple, car, ..., water cup}, are sequentially input into the text encoder of the fine-tuning model to obtain the text representation. (where 1≤k≤n). [The following is a list of options:] ... and After normalization, the similarity scores between each pair of elements are calculated:

[0361]

[0362] In this way, similarity can be measured. Sort the texts in descending order (where 1≤k≤n) and return the text categories with the highest similarity.

[0363] The two methods described above can be used to achieve point cloud recognition with zero samples and fine-tuned models.

[0364] 3. Point cloud semantic segmentation task.

[0365] Point cloud semantic segmentation is the task of classifying each point in a point cloud. Semantic segmentation requires model fine-tuning.

[0366] Fine-tuning a point cloud multimodal model to adapt to point cloud semantic segmentation tasks requires only a point cloud encoder and decoder, eliminating the need for text encoders and decoders, and image encoders and decoders (e.g., ...). Figure 4 As shown in the figure, because the point cloud encoder and decoder already contain rich text image modal information, the accuracy of semantic segmentation can be effectively improved.

[0367] Optionally, the training dataset used for fine-tuning includes category information for each point (assuming there are n categories), i.e. The structure of the point cloud pre-trained model—point cloud segmentation (while organizing the category of each point into a one-hot encoded form), point cloud encoder, and decoder—remains unchanged. The distinction between masked segments and visible segments is no longer made. The decoder output needs to be changed from a masked segment representation to a fully segmented representation, i.e., from... , expanded to Then add new category headers for categorization:

[0368]

[0369]

[0370] Size is ; The size is , OK Each row of the column represents a point in The corresponding logit value for each category.

[0371] Therefore, the class loss function for all points is:

[0372]

[0373] The model is fine-tuned and trained according to the category loss function. When the preset convergence conditions are met, such as the number of training rounds reaching a limit or the loss function value converging, the training stops, and the fine-tuned point cloud semantic segmentation model is obtained.

[0374] The reasoning process of the point cloud semantic segmentation fine-tuning model is similar to the training process. After obtaining... After that, the first on that segment The category of each point is the category corresponding to the maximum logit value of each category, and the formula is:

[0375]

[0376] For example, the starting point of iterative training is to calculate the total multi-task loss of the entire model on the training set. In one iteration, a batch of data is input into the model, all loss terms are calculated through forward propagation, and then the total loss value for that batch is obtained according to the joint loss function. This value is a weighted sum of all losses and serves as a direct signal guiding model updates.

[0377] Next, the system performs backpropagation, calculating the gradient of the total loss value with respect to all learnable parameters in the model (including the weights and biases of the encoder, decoder, mapping layers, etc.). Then, an optimizer updates the parameters based on these gradients, shifting the loss function in the direction of decrease. This process is repeated, with each update of a set of parameters constituting one iteration.

[0378] During parameter updates, the problem of gradient explosion may occur, where the magnitude of the gradient becomes abnormally large, leading to excessively large parameter update steps and unstable model training. To address this issue, a gradient pruning mechanism is introduced to control the magnitude of the update step size and ensure training stability.

[0379] For example, to determine whether a model is properly trained, it is necessary to monitor the model's accuracy metrics in real time. These metrics should not only be evaluated on the training set, but also on the validation set.

[0380] For example, given a point cloud image, the percentage of correct images appearing in the first preset number of search results can be observed. Mask reconstruction accuracy can be compared to the similarity between the predicted mask content and the real content, such as the reconstruction accuracy of text or the peak signal-to-noise ratio (PSNR) of image pixels. When these accuracy metrics no longer improve continuously, i.e., when a preset convergence value is reached, the model training is complete, and the final point cloud multimodal model is obtained.

[0381] In this feasible implementation, iterative optimization is achieved by combining joint loss, gradient pruning, and real-time performance monitoring, ensuring the stability and effectiveness of the training process. Gradient pruning prevents training crashes. By monitoring real-world downstream task metrics, rather than simply relying on the decrease in training loss, it is possible to more objectively determine whether the model has truly learned useful representations, thereby improving the accuracy of model training.

[0382] One feasible implementation method for gradient clipping includes:

[0383] During parameter iteration, the global gradient magnitude is determined, which represents the overall magnitude of the gradient in this round; when the global gradient magnitude is greater than or equal to the preset gradient threshold, the gradient parameters are scaled proportionally.

[0384] For example, during the backpropagation process before each parameter update, the optimizer computes the gradients of all model parameters (weights and biases). This gradient is a multi-dimensional vector containing the update directions of numerous parameters. To calculate the overall update magnitude, the L2 norm of the entire gradient vector, i.e., the global gradient magnitude, needs to be calculated.

[0385] For example, a preset gradient threshold of 1.0 might be used. If the calculated global gradient magnitude is 2.0, exceeding the threshold, it indicates that the gradient has "exploded." In this case, the gradients of all parameters are scaled proportionally, forcibly limiting the updated total gradient magnitude to 1.0.

[0386] For example, if the global gradient magnitude is less than or equal to a threshold, such as 0.8, the gradient is not scaled, and the original gradient is used directly for parameter updates, enabling the model to converge stably to a better solution.

[0387] In this feasible implementation, by monitoring the global gradient magnitude and comparing it with a preset threshold, the gradient is scaled proportionally, effectively preventing gradient explosion and greatly improving the stability of multimodal large model training. This ensures that even if the model generates a very large gradient on a certain batch of data, it will not cause drastic fluctuations in model parameters, thereby improving the accuracy of model training.

[0388] Figure 7 This is a schematic diagram of the structure of a training device for a point cloud multimodal model provided in an embodiment of this application. Figure 7 As shown, the training device 70 for the point cloud multimodal model may include: an acquisition module 71, a processing module 72, an encoding module 73, a loss module 74, and an optimization module 75.

[0389] The acquisition module 71 is used to acquire multimodal data synchronously collected for the sample object. The multimodal data includes point cloud data, text semantic data, and image data.

[0390] The processing module 72 is used to perform segmentation processing and independent random masking processing on the multimodal data to obtain the visible features and masking features of each modality.

[0391] The encoding module 73 is used to encode the visible features of each modality through the encoder corresponding to each modality to obtain multimodal global features, and to map the multimodal global features to the shared semantic space.

[0392] The loss module 74 is used to determine the contrast loss between each pair of modalities based on the multimodal global features in the shared semantic space, and to determine the mask reconstruction loss of each modality based on the multimodal global features in the shared semantic space and the mask features of each modality.

[0393] The optimization module 75 is used to construct a joint loss function based on the contrast loss between each two modes and the mask reconstruction loss of each mode, and to iteratively train the multimodal encoder-decoder network based on the joint loss function to obtain a point cloud multimodal model.

[0394] Optionally, module 71 can be executed. Figure 2 S201 in the embodiment.

[0395] Optionally, processing module 72 can execute Figure 2 S202 in the embodiment.

[0396] Optionally, encoding module 73 can execute Figure 2 S203 in the embodiment.

[0397] Optionally, loss module 74 can be executed. Figure 2 S204 in the embodiment.

[0398] Optionally, optimization module 75 can be executed. Figure 2 S205 in the embodiment.

[0399] In one possible implementation, the processing module 72 is specifically used for:

[0400] By using the farthest point sampling algorithm to sample key points globally in the point cloud data, multiple spatially uniformly distributed key points are obtained.

[0401] Perform k-nearest neighbor queries on each key point to obtain multiple local fragments of the point cloud.

[0402] By randomly selecting a portion of the point cloud from multiple point cloud patches and masking its feature information, the masked features of the point cloud data are obtained.

[0403] Extract features from local patches of the remaining point cloud to obtain the visible features of the point cloud data.

[0404] In one possible implementation, the processing module 72 is specifically used for:

[0405] The image data is normalized to a preset resolution to obtain normalized image data.

[0406] Normalized image data is divided into non-overlapping, equal-grid segments to obtain multiple image sub-slices.

[0407] From multiple image sub-slices, a preset proportion of image sub-slices are randomly selected and the original pixel features are replaced by a unified mask token to obtain the mask features of the image data.

[0408] Extract the features of the remaining image sub-slices to obtain the visible features of the image data.

[0409] In one possible implementation, the processing module 72 is specifically used for:

[0410] The text semantic data is preprocessed by word segmentation to obtain a continuous sequence of text characters.

[0411] Using a single character or word as the basic unit, the text character sequence is segmented to obtain multiple text segments.

[0412] A preset number of text segments are randomly selected from multiple text segments, and their semantic features are masked to obtain the masked features of the text semantic data.

[0413] Extract the features of the remaining text fragments to obtain the visible features of the text semantic data.

[0414] In one possible implementation, the encoding module 73 is specifically used for:

[0415] Embedding dimension transformation and positional encoding enhancement are performed on the multimodal global features respectively, and then fused to obtain an initial feature sequence carrying spatial location and semantic information.

[0416] By performing deep feature extraction and semantic fusion on the initial feature sequence through a multi-layer encoder, a global feature with a unified dimension is obtained.

[0417] A linear mapping layer is used to perform dimensional alignment and distribution calibration on global features of a unified dimension, so as to map multimodal global features to a shared semantic space.

[0418] In one possible implementation, the encoding module 73 is specifically used for:

[0419] Embedding dimension transformation is performed on the visible features of each modality to obtain the embedding features of each modality.

[0420] Spatial relative position encoding is used for point cloud modes to fit the local geometric spatial positional relationship of the point cloud and obtain the position encoding features of the point cloud.

[0421] Two-dimensional absolute position coding is used for image modalities to characterize the row and column spatial distribution information of image pixels, thereby obtaining the position coding features of the image.

[0422] One-dimensional temporal positional encoding is used for text modalities to represent the semantic order information of text characters, thus obtaining the positional encoding features of the text.

[0423] The positional encoding of each modality is fused with the corresponding embedded features element by element to enhance the positional encoding.

[0424] In one possible implementation, the loss module 74 is specifically used for:

[0425] The multimodal global features within the shared semantic space are normalized to obtain normalized global features.

[0426] Cross-modal features from homologous samples and semantically matched in the normalized global features are identified as positive sample pairs, while cross-modal features from non-homologous samples and semantically unmatched are identified as negative sample pairs.

[0427] The contrast loss between each pair of modalities is determined by calculating the difference in feature similarity between positive and negative sample pairs.

[0428] In one possible implementation, the loss module 74 is specifically used for:

[0429] Based on positive and negative sample pairs, the feature similarity of each positive sample pair and the feature similarity of each negative sample pair are calculated to obtain multiple sets of similarity measurement results.

[0430] By comparing the similarity between homologous matching samples and the similarity between non-homologous mismatched samples, the feature difference value between modalities is obtained.

[0431] Based on the difference values ​​of features between modes, determine the contrast loss between each pair of modes.

[0432] In one possible implementation, the loss module 74 is specifically used for:

[0433] Based on the multimodal global features in the shared semantic space, predictive reconstruction features for each modal mask region are generated by a multimodal decoder.

[0434] The predicted reconstruction features are compared with the mask features of each modality dimensionally to obtain the reconstruction error of each modality.

[0435] The mask reconstruction loss for each mode is determined based on the reconstruction error of each mode.

[0436] In one possible implementation, the loss module 74 is specifically used for:

[0437] Based on the data characteristics of each modality, the corresponding loss calculation rules are determined.

[0438] Based on the loss calculation rules, the reconstruction error of each mode is calculated to obtain the mask reconstruction loss of each mode.

[0439] In one possible implementation, the optimization module 75 is specifically used for:

[0440] Determine the weighting coefficients corresponding to the contrast loss between each mode and the mask reconstruction loss of each mode.

[0441] Based on the weighting coefficients, the contrast loss between each mode and the mask reconstruction loss of each mode are weighted and summed to obtain the joint loss function.

[0442] In one possible implementation, the optimization module 75 is specifically used for:

[0443] Determine the current training phase.

[0444] Determine the weighting coefficients based on the current training phase.

[0445] It should be noted that the training device for the point cloud multimodal model shown in the embodiments of this application can execute the technical solution shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0446] Figure 8 This is a schematic diagram of the structure of a training device for another point cloud multimodal model provided in an embodiment of this application. Figure 7 Based on the illustrated embodiments, as Figure 8 As shown, the training device 70 for the point cloud multimodal model also includes: an iteration module 76 and an execution module 77.

[0447] Iteration module 76 is used for:

[0448] Determine the total loss value for multiple tasks based on the joint loss function;

[0449] Based on the total loss value of multiple tasks, the parameters of the multimodal encoder-decoder network are iteratively updated;

[0450] The parameter iteration process is corrected through gradient pruning mechanism;

[0451] Real-time monitoring of model accuracy metrics, including multimodal feature alignment accuracy and mask reconstruction accuracy;

[0452] The point cloud multimodal model is obtained when the accuracy index reaches the preset convergence value.

[0453] Execution module 77 is used for:

[0454] During the parameter iteration process, the global gradient magnitude is determined, which represents the overall magnitude of the gradient in this round;

[0455] When the global gradient magnitude is greater than or equal to a preset gradient threshold, the gradient parameters are scaled proportionally.

[0456] Based on the above implementation methods, collaborative mask reconstruction of point cloud, text and image modalities is performed, the contrastive loss functions of the three modalities are aligned, the representations of the three modalities are aligned pairwise to a unified semantic space, the semantic details of the three modalities are fully explored, the model can capture more complex cross-modal relationships, thereby improving the accuracy of model training.

[0457] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0458] For a description of the features in the embodiment corresponding to the training device for the point cloud multimodal model, please refer to the relevant description of the embodiment corresponding to the training method for the point cloud multimodal model, which will not be repeated here.

[0459] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the electronic device 90 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus.

[0460] In the specific implementation process, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to execute the above-described training method embodiment for point cloud multimodal models.

[0461] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0462] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0463] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0464] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0465] Based on the above implementation methods, collaborative mask reconstruction of point cloud, text and image modalities is performed, the contrastive loss functions of the three modalities are aligned, the representations of the three modalities are aligned pairwise to a unified semantic space, the semantic details of the three modalities are fully explored, the model can capture more complex cross-modal relationships, thereby improving the accuracy of model training.

[0466] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the point cloud multimodal model training method when running.

[0467] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0468] Based on the above implementation methods, collaborative mask reconstruction of point cloud, text and image modalities is performed, the contrastive loss functions of the three modalities are aligned, the representations of the three modalities are aligned pairwise to a unified semantic space, the semantic details of the three modalities are fully explored, the model can capture more complex cross-modal relationships, thereby improving the accuracy of model training.

[0469] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described examples of training methods for point cloud multimodal models.

[0470] Based on the above implementation methods, collaborative mask reconstruction of point cloud, text and image modalities is performed, the contrastive loss functions of the three modalities are aligned, the representations of the three modalities are aligned pairwise to a unified semantic space, the semantic details of the three modalities are fully explored, the model can capture more complex cross-modal relationships, thereby improving the accuracy of model training.

[0471] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described training method embodiments for a point cloud multimodal model.

[0472] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuit systems), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-chip (SoC), a complex programmable logic device (CPLD), a microcontroller unit (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The device may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or a combination thereof. The aforementioned computer program (also referred to as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for the computing environment.

[0473] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0474] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0475] The above provides a detailed description of the training method and electronic device for a point cloud multimodal model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for training a point cloud multi-modal model, the method comprising: include: Acquire multimodal data synchronously collected for the sample object, including point cloud data, text semantic data, and image data; The multimodal data is processed by segmentation and independent random masking to obtain the visible features and masking features of each modality. The visible features of each modality are encoded by the encoder corresponding to each modality to obtain multimodal global features, and the multimodal global features are mapped to a shared semantic space. Based on the multimodal global features within the shared semantic space, determine the contrast loss between each pair of modalities; Based on the multimodal global features within the shared semantic space, a multimodal decoder generates predicted reconstruction features for each modality's masked region. The predicted reconstruction features are then compared dimensionally with the masked features of each modality to obtain the reconstruction error for each modality. The masked reconstruction loss for each modality is determined based on the reconstruction error. When determining the masked reconstruction loss for any target modality, the input to the multimodal decoder includes the visible features of the target modality and global features of other modalities besides the target modality. Based on the contrast loss between each pair of modes and the mask reconstruction loss of each mode, a joint loss function is constructed, and the multimodal encoder-decoder network is iteratively trained according to the joint loss function to obtain a point cloud multimodal model.

2. The method of claim 1, wherein, The point cloud data is subjected to segmentation and independent random masking to obtain visible features and mask features, including: The point cloud data is sampled globally using the farthest point sampling algorithm to obtain multiple spatially uniformly distributed key points. Perform k-nearest neighbor queries on each key point to obtain multiple local fragments of the point cloud; Randomly select a portion of the point cloud from the plurality of point cloud local segments and mask the feature information to obtain the mask feature of the point cloud data; Features of the remaining point cloud local patches are extracted to obtain the visible features of the point cloud data.

3. The training method for a point cloud multimodal model according to claim 1, characterized in that, The image data is processed by segmentation and independent random masking to obtain visible features and mask features, including: The image data is normalized to a preset resolution to obtain normalized image data; The normalized image data is divided into non-overlapping, equal-grid segments to obtain multiple image sub-slices. From the plurality of image sub-slices, a preset proportion of image sub-slices are randomly selected and the original pixel features are replaced by a unified mask token to obtain the mask features of the image data; Extract the features of the remaining image sub-slices to obtain the visible features of the image data.

4. The training method for a point cloud multimodal model according to claim 1, characterized in that, The text semantic data is processed by segmentation and independent random masking to obtain visible features and masked features, including: The text semantic data is preprocessed by word segmentation to obtain a continuous sequence of text characters; Using a single character or word as the basic unit, the text character sequence is segmented into multiple text segments; A preset number of text segments are randomly selected from the plurality of text segments, and their semantic features are masked to obtain the masked features of the text semantic data; Extract the features of the remaining text segments to obtain the visible features of the text semantic data.

5. The training method for a point cloud multimodal model according to claim 1, characterized in that, Mapping the multimodal global features to a shared semantic space includes: Embedding dimension transformation and positional encoding enhancement are performed on the multimodal global features respectively, and then fused to obtain an initial feature sequence carrying spatial location and semantic information; The initial feature sequence is subjected to deep feature extraction and semantic fusion through a multi-layer encoder to obtain global features of a unified dimension. The global features of the unified dimension are aligned and calibrated by a linear mapping layer to map the multimodal global features to a shared semantic space.

6. The training method for a point cloud multimodal model according to claim 5, characterized in that, Perform embedding dimension transformation and positional encoding enhancement on the multimodal global features, including: Embedding dimension transformation is performed on the visible features of each modality to obtain the embedding features of each modality; Spatial relative position encoding is used for point cloud modes to fit the local geometric spatial position relationship of the point cloud and obtain the position encoding features of the point cloud; Two-dimensional absolute position coding is used for image modalities to characterize the row and column spatial distribution information of image pixels, thereby obtaining the position coding features of the image; One-dimensional temporal positional encoding is used for text modalities to represent the semantic order information of text characters, thus obtaining the positional encoding features of the text; The positional encoding of each modality is fused with the corresponding embedded features element by element to enhance the positional encoding.

7. The method according to claim 1, characterized in that, Based on the multimodal global features within the shared semantic space, determine the contrast loss between each pair of modalities, including: The multimodal global features in the shared semantic space are normalized to obtain normalized global features; Cross-modal features from homologous samples and semantically matched in the normalized global features are identified as positive sample pairs, and cross-modal features from non-homologous samples and semantically unmatched are identified as negative sample pairs. The contrast loss between each pair of modalities is determined by calculating the feature similarity difference between the positive sample pairs and the negative sample pairs.

8. The method according to claim 7, characterized in that, The contrast loss between each pair of modalities is determined by calculating the feature similarity difference between the positive sample pairs and the negative sample pairs, including: Based on the positive sample pairs and the negative sample pairs, the feature similarity of each positive sample pair and the feature similarity of each negative sample pair are calculated respectively, and multiple sets of similarity measurement results are obtained. By comparing the similarity between homologous matching samples and the similarity between non-homologous mismatched samples, the feature difference value between modalities is obtained; Based on the feature difference values ​​between the modalities, the contrast loss between each pair of modalities is determined.

9. The method according to claim 1, characterized in that, The mask reconstruction loss for each mode is determined based on the reconstruction error of each mode, including: Based on the data characteristics of each modality, determine the corresponding loss calculation rules; According to the loss calculation rules, the reconstruction error of each mode is calculated to obtain the mask reconstruction loss of each mode.

10. The method according to claim 1, characterized in that, Based on the contrast loss between each pair of modes and the mask reconstruction loss of each mode, a joint loss function is constructed, including: Determine the weighting coefficients corresponding to the contrast loss between each mode and the mask reconstruction loss of each mode, respectively; Based on the weighting coefficients, the contrast loss between the modalities and the mask reconstruction loss of the modalities are weighted and summed to obtain the joint loss function.

11. The method according to claim 1, characterized in that, Determine the weighting coefficients corresponding to the contrast loss between each mode and the mask reconstruction loss of each mode, including: Determine the current training phase; The weight coefficients are determined based on the current training phase.

12. The method according to any one of claims 1-11, characterized in that, The multimodal encoder-decoder network is iteratively trained according to the joint loss function to obtain a point cloud multimodal model, including: Based on the joint loss function, determine the total loss value for multiple tasks; Based on the total multi-task loss value, the parameters of the multimodal coding and decoding network are iteratively updated; The parameter iteration process is corrected through gradient pruning mechanism; The accuracy metrics of the model are monitored in real time, including multimodal feature alignment accuracy and mask reconstruction accuracy. The point cloud multimodal model is obtained when the accuracy index reaches the preset convergence value.

13. The method according to claim 12, characterized in that, The parameter iteration process is corrected through gradient pruning, including: During the parameter iteration process, the global gradient magnitude is determined, which represents the overall magnitude of the gradient in this round; When the global gradient magnitude is greater than or equal to a preset gradient threshold, the gradient parameters are scaled proportionally.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the training method for a point cloud multimodal model as described in any one of claims 1 to 13 when executing the computer program.

Images