Point cloud semantic segmentation model generation method, point cloud semantic segmentation method and system

Abstract

The embodiment of the application provides a point cloud semantic segmentation model generation method, a point cloud semantic segmentation method and a system, wherein the point cloud semantic segmentation model generation method comprises the following steps: receiving a basic point cloud semantic segmentation model generated by a center node; constructing a scene point cloud semantic segmentation model of a current user node based on the basic point cloud semantic segmentation model and a scene adapter corresponding to the current user node and used for processing private point cloud data of the current user node; training the scene point cloud semantic segmentation model by taking the private point cloud data of the current user node as a training sample, and obtaining a trained scene point cloud semantic segmentation model; and sending model data of the scene point cloud semantic segmentation model to the center node, so that the center node integrates and generates a global point cloud semantic segmentation model according to the model data of the scene point cloud semantic segmentation model of the current user node and the model data of the scene point cloud semantic segmentation models sent by other user nodes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a point cloud semantic segmentation model generation method, a point cloud semantic segmentation method, and a corresponding point cloud semantic segmentation system. Background Technology

[0002] A point cloud is a set of coordinates of three-dimensional points, which can reflect the geometric information of a scene or object. With the development of devices that collect point clouds, such as lasers, radars, and depth cameras, the application scenarios of point clouds are becoming more and more numerous. Capturing three-dimensional scenes based on point clouds and processing three-dimensional information have also attracted much attention, and point cloud semantic segmentation is an important part of this.

[0003] Point cloud semantic segmentation is the process of assigning a label to each point in a point cloud to indicate its corresponding object. For example, assigning a label to a point to indicate whether it is a vehicle, pedestrian, or traffic light, and so on. This makes point cloud semantic segmentation particularly important in certain fields, such as autonomous driving. In autonomous driving, devices with autonomous driving capabilities, such as vehicles, aircraft, or robots, rely on powerful point cloud semantic segmentation models to achieve robust perception capabilities. These models need to be trained on a large number of street scenes, including internal roads, to obtain high-performance models and achieve accurate point cloud semantic segmentation, thus providing perception services for subsequent applications (such as navigation, obstacle recognition, or other types of target recognition).

[0004] However, most internal road data has privacy requirements and can only be used in the form of single nodes, rather than being uniformly distributed to a single node to form a large-scale training sample for street scenes. Therefore, how to train a high-performance point cloud semantic segmentation model under the premise of data privacy requirements, and then use this model to provide better perception services for devices, has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide solutions related to point cloud semantic segmentation to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of this application, a method for generating a point cloud semantic segmentation model is provided, comprising: receiving a trained basic point cloud semantic segmentation model sent by a central node; constructing a scene point cloud semantic segmentation model for the current user node based on the basic point cloud semantic segmentation model and a scene adapter corresponding to the current user node for processing the privacy point cloud data of the current user node; training the scene point cloud semantic segmentation model using the privacy point cloud data of the current user node as training samples to obtain a trained scene point cloud semantic segmentation model; and sending the model data of the trained scene point cloud semantic segmentation model to the central node, so that the central node integrates and generates a global point cloud semantic segmentation model based on the model data of the scene point cloud semantic segmentation model of the current user node and the model data of trained scene point cloud semantic segmentation models sent by other user nodes.

[0007] According to a second aspect of the embodiments of this application, another method for generating a point cloud semantic segmentation model is provided, comprising: receiving model data of multiple scene point cloud semantic segmentation models sent by multiple user nodes, wherein the scene point cloud semantic segmentation models are generated by training using the method described in the first aspect; determining corresponding weights for the model data of the multiple scene point cloud semantic segmentation models respectively; and integrating the model data of the multiple scene point cloud semantic segmentation models and their respective corresponding weights to generate a global point cloud semantic segmentation model.

[0008] According to a third aspect of the embodiments of this application, a point cloud semantic segmentation method is provided, comprising: acquiring point cloud data of a three-dimensional scene to be detected; acquiring an encoding vector of the three-dimensional scene point cloud data through a global point cloud semantic segmentation model, and acquiring intermediate features and Gaussian distribution corresponding to the three-dimensional scene point cloud data based on the encoding vector; determining the weights corresponding to multiple scene adapters in the global point cloud semantic segmentation model based on the intermediate features and the Gaussian distribution; and performing semantic segmentation on the three-dimensional scene point cloud data based on the weights of each scene adapter and the output scene features.

[0009] According to a fourth aspect of the embodiments of this application, a point cloud semantic segmentation system is provided, comprising: a central node and multiple user nodes; wherein: the central node is configured to train a basic point cloud semantic segmentation model using its own central point cloud data, and distribute the trained basic point cloud semantic segmentation model to each user node; each of the multiple user nodes is configured to receive the trained basic point cloud semantic segmentation model distributed by the central node; construct a scene point cloud semantic segmentation model for the current user node based on the basic point cloud semantic segmentation model and a scene adapter corresponding to the current user node for processing the privacy point cloud data of the current user node; train the scene point cloud semantic segmentation model using the privacy point cloud data of the current user node as training samples to obtain a trained scene point cloud semantic segmentation model; and send the trained scene point cloud semantic segmentation model to the central node; the central node is further configured to integrate and generate a global point cloud semantic segmentation model based on the multiple scene point cloud semantic segmentation models sent by the multiple user nodes.

[0010] According to the solution provided in this application, a scene point cloud semantic segmentation model is constructed locally on the user node based on the basic point cloud semantic segmentation model issued by the central node. The model is trained using the user node's local privacy point cloud data, allowing the scene feature data of the visiting user node to be learned by the model. Furthermore, the model data of the user node's scene point cloud semantic segmentation model is sent back to the central node. Thus, the central node can integrate the models based on the model data of multiple user nodes' scene point cloud semantic segmentation models to generate a global point cloud semantic segmentation model. In this way, on the one hand, data representing internal roads locally on the user node is not sent to other nodes, satisfying data privacy requirements; on the other hand, although the central node cannot obtain the user node's privacy point cloud data, it can obtain the model data of the model trained based on this privacy point cloud data. The global point cloud semantic segmentation model constructed based on the model data of multiple user nodes can directly achieve the training results of model training based on data from different nodes, obtaining better model performance. This enables comprehensive and accurate point cloud semantic segmentation in subsequent applications, providing better perception services for corresponding devices. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1A schematic diagram of an exemplary system to which the embodiments of this application are applicable;

[0013] Figure 2A This is a flowchart illustrating the steps of a point cloud semantic segmentation model generation method according to Embodiment 1 of this application;

[0014] Figure 2B for Figure 2A A schematic diagram of the structure of a basic point cloud semantic segmentation model in the embodiment shown;

[0015] Figure 2C for Figure 2A A schematic diagram of the structure of a scene point cloud semantic segmentation model in the embodiment shown;

[0016] Figure 2D for Figure 2A A schematic diagram of the structure of a scene adapter in the embodiment shown;

[0017] Figure 2E for Figure 2A A schematic diagram of an intermediate feature acquisition process in the illustrated embodiment;

[0018] Figure 3A This is a flowchart illustrating the steps of a point cloud semantic segmentation model generation method according to Embodiment 2 of this application;

[0019] Figure 3B for Figure 3A A schematic diagram of the structure of a global point cloud semantic segmentation model in the embodiment shown;

[0020] Figure 4A This is a flowchart illustrating the steps of a point cloud semantic segmentation method according to Embodiment 3 of this application;

[0021] Figure 4B This is a schematic diagram illustrating an example of a point cloud semantic segmentation model generation and application method according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0023] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0024] Figure 1An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a central node 102, a communication network 104, and multiple user nodes 106.

[0025] In this embodiment, the central node 102 and the user node 106 are used to perform federated learning and training on the point cloud semantic segmentation model to generate the final global point cloud semantic segmentation model.

[0026] in:

[0027] The central node 102 can be any suitable device for storing information, data, and programs for training and generating point cloud semantic segmentation models, including but not limited to distributed system devices, servers or server clusters, computing cloud servers or server clusters, etc. In some embodiments, the central node 102 can perform any suitable function. For example, in some embodiments, the central node 102 can train a basic point cloud semantic segmentation model using its own central point cloud data and distribute the trained basic point cloud semantic segmentation model to each user node. As an optional example, in some embodiments, the central node 102 can be used to integrate multiple scene point cloud semantic segmentation models sent by multiple user nodes to generate a global point cloud semantic segmentation model, wherein the scene point cloud semantic segmentation model is generated and trained based on the basic point cloud semantic segmentation model. As another example, in some embodiments, the central node 102 can be used to perform semantic segmentation processing on the 3D scene point cloud data to be detected based on the global point cloud semantic segmentation model. In some embodiments, the central node 102 can also be used to distribute the integrated global semantic segmentation model to each user node 106.

[0028] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User node 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to central node 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user node 106 and central node 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0029] User node 106 can also be any suitable device used to store information, data, and programs for training point cloud semantic segmentation models. In some embodiments, user node 106 can receive a trained basic point cloud semantic segmentation model from central node 102; construct a scene point cloud semantic segmentation model for the current user node based on the basic point cloud semantic segmentation model and a scene adapter corresponding to the current user node for processing the privacy point cloud data of the current user node; train the scene point cloud semantic segmentation model using the privacy point cloud data of the current user node as training samples to obtain a trained scene point cloud semantic segmentation model; and send the trained scene point cloud semantic segmentation model to central node 102.

[0030] Based on the above system, this application provides a point cloud semantic segmentation and model generation method, which will be described below through several embodiments.

[0031] Example 1

[0032] Reference Figure 2A The flowchart illustrates the steps of a point cloud semantic segmentation model generation method according to Embodiment 1 of this application.

[0033] This embodiment describes the point cloud semantic segmentation model generation method provided in this application from the perspective of the user node. The method includes the following steps:

[0034] Step S202: Receive the completed training of the basic point cloud semantic segmentation model from the central node.

[0035] In current point cloud semantic segmentation methods based on federated learning (a distributed machine learning framework with privacy protection and secure encryption, designed to allow decentralized participants to collaboratively train machine learning models without disclosing private data to other participants), a large amount of internal data, such as internal road data in autonomous driving scenarios, is distributed across different user nodes. The central node can only collect internal data from its own local scenario and cannot collect internal data from other user nodes. Therefore, it is necessary to effectively integrate these two types of data.

[0036] Based on this, in this embodiment, the central node first trains a basic point cloud semantic segmentation model based on its own central dataset, and after training, distributes the model to each user node. Each user node then receives the trained basic point cloud semantic segmentation model from the central node.

[0037] The central node can distribute the model structure and model data (such as model parameters) of the basic point cloud semantic segmentation model to user nodes. In this way, user nodes do not need to prepare their own models. However, it is not limited to this. User nodes can also prepare a point cloud semantic segmentation model with the same model structure as the central node in advance. In this case, the central node only needs to distribute the model data, such as model parameters, of the trained basic point cloud semantic segmentation model.

[0038] In one alternative implementation, the basic point cloud semantic segmentation model can be implemented as an encoder-decoder structure, for example, such as... Figure 2B As shown, both the encoder and decoder include multiple feature extraction layers. The encoder's feature extraction layer can be represented as g(·θ), and the decoder's feature extraction layer can be represented as d(·φ). Here, "." represents the input vector, and θ and φ represent randomly initialized parameters. Based on this, the basic point cloud semantic segmentation model can be simply represented as f G (·;θ,φ). It should be noted that, in the embodiments of this application, the specific implementation of the encoder-decoder structure can be implemented by those skilled in the art in any appropriate manner according to actual needs, including but not limited to encoder-decoder structures using Transformer structures, as well as other forms of convolutional encoder-decoder structures, etc.

[0039] The central node uses the central dataset as training samples and inputs it into the basic point cloud semantic segmentation model in batches. This model is a neural network model with an encoder-decoder structure as described above. After training, the central node sends the completed basic point cloud semantic segmentation model to the user node, which then receives the model.

[0040] Step S204: Based on the basic point cloud semantic segmentation model and the scene adapter corresponding to the current user node, which is used to process the privacy point cloud data of the current user node, construct the scene point cloud semantic segmentation model of the current user node.

[0041] Each user node has a scene adapter used to process and learn from the privacy point cloud data of its own physical scene. It should be noted that the privacy point cloud data of the user node has the same properties and attributes as the data in the central dataset of the central node. It is called privacy point cloud data only because the user node cannot share its data with other nodes for data security reasons.

[0042] Based on the basic point cloud semantic segmentation model issued by the central node and the scene adapters of each user node, each user node can construct its own scene point cloud semantic segmentation model. For example, when the basic point cloud semantic segmentation model adopts an encoder-decoder structure, the scene adapter corresponding to the current user node, which is used to process the privacy point cloud data of the current user node, can be inserted between the encoder-decoder structure. The output of the encoder is used as the input of the scene adapter, and the output of the scene adapter is used as the input of the decoder to construct the scene point cloud semantic segmentation model of the current user node.

[0043] An exemplary scene point cloud semantic segmentation model is as follows: Figure 2C As shown, the scene adapter is positioned between the encoder and the decoder, taking the encoder's output as its input and its output as the decoder's input.

[0044] In one feasible approach, the scene adapter can be implemented with a convolutional layer component, a feature aggregation component, and a scene topology component. For example, one possible structure of a scene adapter is as follows: Figure 2D As shown, the convolutional layer consists of four layers: a 1×1 convolutional layer followed by three 3×3 convolutional layers connected sequentially. Optionally, these three 3×3 convolutional layers can employ dilated convolutions to expand the receptive field. The features extracted by these three 3×3 convolutional layers are then fed into the feature aggregation part, where they are concatenated along the channel dimension to obtain multi-level aggregated features. Further, the scene topology part learns the scene topology information of the current user node (scene feature information of the physical scene where the user node is located). This part uses a learnable matrix, namely the scene topology matrix shown in the figure. For ease of computation, the multi-level aggregated features are flattened along the spatial dimension, while the learnable matrix can be scaled to the size of the flattened multi-level aggregated features. Therefore, the scene topology part can convolve both to obtain the final output, i.e., the scene features.

[0045] As can be seen, after the encoded vector output by the encoder is handed over to the scene adapter, it is processed sequentially by its convolutional layer, feature aggregation layer and scene topology layer to output scene features. These scene features will then be handed over to the decoder for subsequent semantic segmentation processing.

[0046] Through the above process, the scene point cloud semantic segmentation model for each user node can be constructed.

[0047] Step S206: Using the privacy point cloud data of the current user node as training samples, train the scene point cloud semantic segmentation model to obtain the trained scene point cloud semantic segmentation model.

[0048] Since the scene point cloud semantic segmentation model is obtained by inserting a scene adapter between the encoder and decoder of the basic point cloud semantic segmentation model, and the encoder and decoder are models that have already been trained by the central node, the training of the scene point cloud semantic segmentation model can be mainly carried out on the scene adapter, while the model parameters of the encoder and decoder can remain unchanged.

[0049] Based on the encoder-scene adapter-decoder structure of the scene point cloud semantic segmentation model, in one feasible approach, training the scene point cloud semantic segmentation model using the privacy point cloud data of the current user node as training samples can be achieved as follows: The encoder in the scene point cloud semantic segmentation model encodes the input privacy point cloud data of the current user node to obtain a corresponding encoding vector; the scene adapter learns the scene topology information corresponding to the privacy point cloud data based on the encoding vector and outputs the corresponding scene features; the decoder trains point cloud semantic segmentation based on the scene features until the training termination condition is met. The training termination condition can be appropriately set by those skilled in the art according to actual needs, such as reaching a preset number of training iterations or a preset threshold for the loss value.

[0050] In adopting such Figure 2D When the scene adapter structure is shown, the scene adapter learns the scene topology information corresponding to the privacy point cloud data based on the encoded vector, and outputs the corresponding scene features, including: through the first convolutional layer in the convolutional layer part of the scene adapter (such as...). Figure 2D The "convolution 1×1" shown in the diagram convolves the encoded vector to obtain the first feature vector (e.g., ...). Figure 2D As shown ); through multiple consecutive dilated convolutional layers after the first convolutional layer (such as Figure 2D The three "3×3" convolutions shown in the diagram perform dilated convolutions on the first feature vector to obtain multiple corresponding second feature vectors (such as...). Figure 2D As shown ); By aggregating multiple second feature vectors through the feature aggregation part, multi-level aggregated features are obtained (such as...). Figure 2D u shown k ); through the learnable matrix in the scene topology (e.g. Figure 2D The scene topology matrix shown in the figure is used to learn scene topology information based on multi-level aggregated features, and the scene features are output according to the learnable matrix and multi-level aggregated features obtained after learning.

[0051] In a specific example, combined Figure 2D The scene adapter structure shown above, the process of the scene adapter outputting scene features can be specifically implemented as follows: for the k-th scene adapter A in the k-th user node kIts input is the feature g(x) output by the encoder, and its output is the scene feature U of the current user node. k .

[0052] Specifically:

[0053] First, the input g(x) is convolved by a 1×1 convolution layer to obtain... Then, the obtained The input is fed into three consecutive 3×3 dilated convolutional layers for dilated convolution, yielding the outputs of the three dilated convolutional layers respectively. The spatial dimension is the same as the channel dimension. Multi-level aggregated features are obtained by concatenating along the channel dimension. k .

[0054] To obtain scene topology information, each scene adapter has a learnable matrix M. k The length of the square matrix is ​​u k The size of the spatial dimension when flattened. (U) k After flattening the spatial dimensions, the scene features U k It can be calculated that:

[0055] U k =conv1×1(M k ⊙RESHAPE(u k ))

[0056] Where conv1×1 represents a 1×1 convolution operation, ⊙ represents matrix multiplication, and RESHAPE(·) represents the spatial dimension transformation function used for the above spatial dimension flattening.

[0057] Through the above process, the scene features of user nodes can be effectively learned. Based on these scene features, subsequent decoding is performed, and a loss value is obtained according to the final decoding result, the semantic labels corresponding to the privacy point cloud data of the user nodes, and the preset loss function. The model parameters of the scene adapter are adjusted based on the loss value until the training termination condition corresponding to the model is reached, thus obtaining the trained scene point cloud semantic segmentation model.

[0058] However, to ensure greater accuracy and efficiency when integrating the scene point cloud semantic segmentation model at the central node, one feasible approach involves extracting intermediate features from the privacy point cloud data during the training process of the scene point cloud semantic segmentation model, and establishing a multivariate Gaussian distribution based on these intermediate features. The intermediate features can be obtained as follows: acquiring the encoding vector output by the encoder; performing spatial pooling on the encoding vector to obtain high-level features; then, after restoring the encoding vector to the corresponding point cloud, performing gridding and channel pooling to obtain low-level features; and finally, concatenating the high-level features and the low-level features to obtain the intermediate features.

[0059] An exemplary intermediate feature acquisition process is as follows: Figure 2E As shown in the figure, the intermediate feature r k Features of high-level and underlying features It is pieced together along the channel dimension. One aspect is the high-level features. The output of the last layer of the encoder, for the sake of distinction, is represented here as It is obtained through pooling in the spatial dimension; on the other hand, the underlying features The output of the last layer of the encoder, for the sake of distinction, is represented here as It was obtained through a series of changes, including: First, it is projected back to the point cloud dimension, then meshed and pooled along the channel dimension to obtain... High-level characteristics and underlying features By splicing the components together, the intermediate features can be obtained.

[0060] Furthermore, a multivariate Gaussian distribution of the current user node can be established based on this intermediate feature, such as:

[0061]

[0062] in, It expresses expectation.

[0063] This intermediate feature and multivariate Gaussian distribution will help the central node to integrate the scene point cloud semantic segmentation model of multiple user nodes in the subsequent process.

[0064] Step S208: Send the model data of the trained scene point cloud semantic segmentation model to the central node, so that the central node can integrate the model data of the current user node's scene point cloud semantic segmentation model with the model data of the trained scene point cloud semantic segmentation models sent by other user nodes to generate a global point cloud semantic segmentation model.

[0065] If the operation of obtaining the intermediate features and multivariate Gaussian distribution of the scene point cloud semantic segmentation model is not performed, the model data sent to the central node is mainly model parameters, especially the model parameters of the scene adapter. The central node integrates the scene point cloud semantic segmentation models sent by multiple user nodes into a global point cloud semantic segmentation model based on the model parameters of each scene adapter, using methods such as average weight or multi-branch balancing settings corresponding to multiple scene adapters.

[0066] However, if the operation of obtaining the intermediate features and multivariate Gaussian distribution of the scene point cloud semantic segmentation model is performed, the model parameters, corresponding intermediate features, and multivariate Gaussian distribution of the trained scene point cloud semantic segmentation model can be sent to the central node. This allows the central node to integrate the model parameters, intermediate features, and multivariate Gaussian distribution of the current user node's scene point cloud semantic segmentation model with the model parameters, intermediate features, and multivariate Gaussian distribution of the trained scene point cloud semantic segmentation models sent by other user nodes to generate a global point cloud semantic segmentation model. A feasible integration method can be specifically implemented using the point cloud semantic segmentation model generation method on the central node side described in Example 2 below, which will not be detailed here.

[0067] As can be seen, through this embodiment, a scene point cloud semantic segmentation model is constructed locally on the user node based on the basic point cloud semantic segmentation model issued by the central node. The model is trained using the user node's local privacy point cloud data, allowing the scene feature data of the visiting user node to be learned by the model. Furthermore, the model data of the user node's scene point cloud semantic segmentation model is sent back to the central node. Thus, the central node can integrate the model data from multiple user nodes' scene point cloud semantic segmentation models to generate a global point cloud semantic segmentation model. In this way, on the one hand, data representing internal roads locally on the user node is not sent to other nodes, satisfying data privacy requirements; on the other hand, although the central node cannot obtain the user node's privacy point cloud data, it can obtain the model data of the model trained based on this privacy point cloud data. The global point cloud semantic segmentation model constructed based on the model data from multiple user nodes can directly achieve the training results of model training based on data from different nodes, obtaining better model performance. This enables comprehensive and accurate point cloud semantic segmentation in subsequent applications, providing better perception services for corresponding devices.

[0068] Example 2

[0069] Reference Figure 3A The flowchart illustrates the steps of a point cloud semantic segmentation model generation method according to Embodiment 2 of this application.

[0070] This embodiment describes the point cloud semantic segmentation model generation method of this application from the perspective of the central node. The point cloud semantic segmentation model generation method includes the following steps:

[0071] Step S302: Receive model data of multiple scene point cloud semantic segmentation models sent by multiple user nodes.

[0072] In this context, the scene point cloud semantic segmentation model sent by each user node is trained and generated using the method described in Example 1.

[0073] As mentioned earlier, if the operation of obtaining the intermediate features and multivariate Gaussian distribution of the scene point cloud semantic segmentation model is not performed on the user node's scene point cloud semantic segmentation model, the model data received by the central node mainly consists of the model parameters of the scene point cloud semantic segmentation model, especially the model parameters of the scene adapter. However, if the operation of obtaining the intermediate features and multivariate Gaussian distribution of the scene point cloud semantic segmentation model is performed on the user node's scene point cloud semantic segmentation model, the central node receives the model parameters, corresponding intermediate features, and multivariate Gaussian distribution of the trained scene point cloud semantic segmentation model.

[0074] In addition, since the central node has already completed the training of the basic point cloud semantic segmentation model locally, it also stores the model data of the decoder and encoder in the scene point cloud semantic segmentation model.

[0075] The above-mentioned data from various models provide a complete, comprehensive, and accurate foundation for the central node to integrate multiple semantic segmentation models.

[0076] Step S304: Determine the corresponding weights for the model data of the point cloud semantic segmentation models for multiple scenes.

[0077] As mentioned earlier, the central node can use average weighting to determine the corresponding weights for the model data of multiple scene point cloud semantic segmentation models. However, due to the different characteristics of user privacy point cloud data from different user nodes, the trained scene adapters also have different characteristics. In order to better leverage the characteristics of different scene adapters from different user nodes in the subsequent model application stage, in one feasible approach, this step can be implemented as follows: For each scene point cloud semantic segmentation model, a preset number of scene features are collected from the multivariate Gaussian distribution of the model data of the scene point cloud semantic segmentation model; the scene features are processed into one-hot codes to obtain the corresponding scene one-hot codes; based on the scene one-hot codes and the intermediate features in the model data of the scene point cloud semantic segmentation model, a Gaussian process is established to estimate the posterior probability that the intermediate features belong to the preset scene, and the posterior probability is used as the weight corresponding to the scene point cloud semantic segmentation model.

[0078] In one example, based on the intermediate features *r* and multivariate Gaussian distributions of each scene point cloud semantic segmentation model, firstly, Z sampled scene features are uniformly sampled from each collected multivariate Gaussian distribution to form *R*, where |R| = Z × N, and N represents the number of scene point cloud semantic segmentation models, i.e., the number of collected multivariate Gaussian distributions. Then, multiple one-hot vectors *I* are used to represent the scene corresponding to each sampled scene feature.

[0079] Furthermore, a Gaussian process is established to estimate the posterior probability that the intermediate feature r belongs to each scene.

[0080] include:

[0081]

[0082] i m =K(r,R)·K(R,R) -1 ·I,

[0083] i v =K(r,r)-K(r,R)·K(R,R) -1 ·K(R, r),

[0084] Where i represents the soft label corresponding to the sampling scene features, i m Let i represent the posterior mean of the Gaussian process. v Let K(·,·) represent the posterior covariance of the Gaussian process, and K(·,·) be used to calculate the covariance matrix. m It is an approximation of i. (·,·) represents a Gaussian process.

[0085] Finally, the posterior probability w of the sampled scene features belonging to each scene is:

[0086] w = softmax(i m ,τ)

[0087] Where w = (w1, ... w) N ), where τ represents the temperature parameter of the softmax function.

[0088] After obtaining w, it can be used as the weight of the corresponding scene adapter, or it can be considered as the weight of the scene point cloud semantic segmentation model corresponding to the scene adapter.

[0089] Step S306: Based on the model data and corresponding weights of multiple scene point cloud semantic segmentation models, integrate them to generate a global point cloud semantic segmentation model.

[0090] After determining the model data and corresponding weights of the point cloud semantic segmentation models for each scene, a global point cloud semantic segmentation model can be generated by integrating them. For example, a global point cloud semantic segmentation model can be generated by merging multiple scene adapters, where multiple scene adapters share an encoder and a decoder. Each scene adapter, along with the preceding encoder and the following decoder, forms a branch of the global point cloud semantic segmentation model. An exemplary global point cloud semantic segmentation model is shown below. Figure 3B As shown. In addition, after generating the global point cloud semantic segmentation model, the central node may distribute the global point cloud semantic segmentation model to multiple user nodes to update the scene point cloud semantic segmentation models of each user node. This enables each user node to comprehensively and reasonably process point cloud data in different scenarios and perform accurate semantic segmentation.

[0091] As can be seen, through this embodiment, without acquiring the privacy point cloud data of each user node, the central node integrates the scene point cloud semantic segmentation models trained by each user node, especially the scene adapters, to effectively extract scene feature information from different user nodes and generalize it. This not only effectively avoids the training difficulties caused by the data privacy of user nodes, but also generates a universal and generalized global point cloud semantic segmentation model to provide accurate perception services for the corresponding devices.

[0092] Example 3

[0093] Reference Figure 4A The diagram shows a flowchart of a point cloud semantic segmentation method according to Embodiment 3 of this application.

[0094] This embodiment describes the point cloud semantic segmentation method of this application from the perspective of the application of the generated global point cloud semantic segmentation model. The point cloud semantic segmentation method includes the following steps:

[0095] Step S402: Obtain the point cloud data of the 3D scene to be detected.

[0096] The point cloud data of the three-dimensional scene to be detected can be point cloud data in any scene, including but not limited to point cloud data of the physical environment in which the device with autonomous driving function (such as vehicle, aircraft, robot, etc.) is located, collected by LiDAR.

[0097] Step S404: Obtain the encoding vector of the 3D scene point cloud data through the global point cloud semantic segmentation model, and obtain the intermediate features and Gaussian distribution corresponding to the 3D scene point cloud data based on the encoding vector.

[0098] The global point cloud semantic segmentation model is the model generated by the central node and published to each user node in the aforementioned embodiments. Since both the central node and each user node possess this global point cloud semantic segmentation model, the point cloud semantic segmentation method in this embodiment can be implemented either at the central node or on any user node. The process of obtaining the encoding vector of the 3D point cloud data, and obtaining the intermediate features and Gaussian distribution corresponding to the 3D scene point cloud data based on this encoding vector, can be implemented as described in the aforementioned embodiments. This involves obtaining the encoding vector through an encoder, processing the output of the last layer of the encoder to obtain high-level and low-level features, and then concatenating these features to obtain intermediate features and a Gaussian distribution. Specifically, in this embodiment, after the 3D scene point cloud data is input into the global point cloud semantic segmentation model, it first passes through an encoder to generate an encoding vector. Simultaneously, the output of the last layer of the encoder is processed to obtain the corresponding intermediate features and Gaussian distribution.

[0099] Step S406: Determine the weights of the multiple scene adapters in the global point cloud semantic segmentation model based on the intermediate features and Gaussian distribution.

[0100] As described in Example 2, the corresponding posterior probability can be determined based on the intermediate features and Gaussian distribution, and the posterior probability is used as the weight corresponding to the scene adapter.

[0101] Step S408: Perform semantic segmentation on the 3D scene point cloud data based on the weights of each scene adapter and the output scene features.

[0102] Because different scene adapters form different model branches when integrating and generating a global point cloud semantic segmentation model, and different scene adapters have different weights, the final output scene features of this part can be obtained based on the scene features output by different scene adapters and the weights corresponding to those scene adapters.

[0103] Then, the scene features will be input into the decoder for processing to ultimately achieve semantic segmentation of the 3D scene point cloud data.

[0104] After obtaining the semantic segmentation results, subsequent applications can be guided according to actual needs. For example, in autonomous driving scenarios, the semantic segmentation results can be used to formulate driving rules for devices with autonomous driving functions, or to instruct them to avoid obstacles, or to instruct them to drive in intersections with other devices with autonomous driving functions, and so on.

[0105] The following uses a specific example to illustrate the solution provided in the embodiments of this application, from the entire process of point cloud semantic segmentation model generation and application. Figure 4B As shown.

[0106] S1. The central node obtains the central dataset for training, and each user node obtains its own user privacy dataset for training. The datasets are all LiDAR point cloud data.

[0107] Specifically, this step may include:

[0108] (S11) Obtain the contents of N 0 p point cloud data samples 0 and the corresponding point-by-point label y 0 Central dataset S 0 :

[0109]

[0110] in, This represents the i-th point cloud data sample in the central dataset. express The tag.

[0111] (S12) In a scenario with N user nodes, obtain the data containing N... k p point cloud data samples k and the corresponding point-by-point label y k The k-th user privacy dataset S k :

[0112]

[0113] in, This represents the i-th point cloud data sample in the k-th user privacy dataset. express The tag.

[0114] (S13) Finally, the overall dataset is

[0115] S2. Establish a basic point cloud semantic segmentation model, which adopts an encoder-decoder structure.

[0116] Specifically, including:

[0117] (S21) The encoder of the basic point cloud semantic segmentation model consists of a feature extraction module g(·; θ), and the decoder consists of a module d(·; φ), where θ and φ are randomly initialized parameters; the overall network can be represented as follows:

[0118] f G (·; θ, φ):=d(g(·; θ); φ);

[0119] S3. Input the central dataset from S1 into the basic point cloud semantic segmentation model built in S2 in batches, and combine the cross-entropy function and the Lovas Softmax function to constrain the model in order to train the model.

[0120] Specifically, including:

[0121] (S31) Transfer the training set of the central node Divide into batches and input into f in S21 G (·;θ,φ), and train according to the constraints:

[0122]

[0123] Among them, L train It consists of the cross-entropy loss function and the Lovas Softmax loss function.

[0124] S4. Distribute the basic point cloud semantic segmentation model trained in S3 to each (e.g., N) user node.

[0125] Specifically, including:

[0126] (S41) The central node will train the basic semantic segmentation parameters θ * , φ * Distribute to N user nodes;

[0127] (S42) Each user node constructs an encoder and decoder with the same structure as the central node, and assigns it the trained parameters θ. * ,φ * :

[0128] g(·;θ * ),d(·;φ * ).

[0129] S5. Each user node establishes its own scene adapter and inserts the adapter into the basic point cloud semantic segmentation model received in S4 to construct the scene point cloud semantic segmentation model.

[0130] Specifically, including:

[0131] (S51) For the k-th user node, where k∈{1,…,N}, construct scene adapter A. k (·;ψ k ), where ψ k These are parameters that are randomly initialized.

[0132] Specifically, the construction of the scene adapter can include:

[0133] The k-th scene adapter A kThe input is the feature g(x) output by the encoder, and the output is the scene feature U. k U k The calculations include:

[0134] First, perform a 1×1 convolution operation on the input g(x) to obtain Secondly, the obtained The input is fed into three consecutive dilated 3×3 convolutions, resulting in three layers of output. The spatial dimension is the same as the channel dimension. Multi-level aggregated features are obtained by concatenating along the channel dimension. k .

[0135] To obtain scene topology information, each scene adapter has a learnable matrix M. k The length of the square matrix is ​​u k The size of the spatial dimension when flattened. (U) k After the spatial dimensions are flattened, U k It can be calculated that:

[0136] U k =conv1×1(M k ⊙RESHAPE(u k )),

[0137] Where conv1×1 is a 1×1 convolution, ⊙ is matrix multiplication, and RESHAPE(·) is the dimension transformation function mentioned above.

[0138] (S52) For the k-th user node, where k∈{1,…,N}, construct the corresponding scene point cloud semantic segmentation model, represented as:

[0139] f k (·;θ * ,φ * ,ψ k ):=d(A k (g(·;θ);ψ k );φ * ).

[0140] S6. Input the privacy dataset from S1 into the scene point cloud semantic segmentation model constructed in S5 in batches. Combine the cross-entropy function and the Lovas Softmax function to constrain the network. Collect intermediate features during the training process and establish a Gaussian distribution.

[0141] Specifically, including:

[0142] (S61) Use the privacy training set of the kth user node Divide into batches and input them into f in S52 k(·;θ * ,φ * ,ψ k ), and train according to the constraints:

[0143]

[0144] (S62) Collect p during training k The corresponding intermediate feature r k And establish a multivariate Gaussian distribution:

[0145]

[0146] in,

[0147] The extraction of the aforementioned intermediate features includes the following steps:

[0148] intermediate feature r k Features of high-level and underlying features It is assembled along the channel dimension. Among them, high-level features The output of the last layer of the encoder Obtained through pooling in spatial dimensions; underlying features The output of the last layer of the encoder It was obtained through a series of changes. First, it is projected back to the point cloud dimension, then meshed and pooled along the channel dimension to obtain...

[0149] S7. Each user node sends the trained scene adapter and Gaussian distribution back to the central node.

[0150] Specifically, including:

[0151] (S71) Each user node returns the scene adapter parameters, and the central node receives a total of N sets of parameters:

[0152]

[0153] (S72) Each user node sends back a Gaussian distribution, and the central node receives a total of N distributions:

[0154]

[0155] S8, the central node integrates the scene adapter in S7 and uses Gaussian distribution to model the scene, generating a global point cloud semantic segmentation model.

[0156] Specifically, including:

[0157] (S81) The central node establishes a scene estimation module to evaluate the weights (w1,…,w) for each received scene adapter. N The fully integrated model is as follows:

[0158]

[0159] When performing specific scene estimation, the following steps can be taken:

[0160] For input p, intermediate features r are obtained through a process similar to S62, which extracts intermediate features.

[0161] Z scene features are uniformly sampled from each collected Gaussian distribution, forming R, |R| = Z × N. Then, multiple one-hot vectors I represent the scene described by each scene feature.

[0162] A Gaussian process is established to estimate the posterior probability that the intermediate feature r belongs to each scene.

[0163] include:

[0164]

[0165] i m =K(r,R)·K(R,R) -1 ·I,

[0166] i v =K(r,r)-K(r,R)·K(R,R) -1 ·K(R,r),

[0167] Where i is the soft label of the input sample to be obtained, i m Let i be the posterior mean of the Gaussian process. v Let K(·,·) be the posterior covariance of the Gaussian process, and K(·,·) be used to calculate the covariance matrix. m It is an approximation of i.

[0168] Finally, the posterior probability w of a sample belonging to each scenario can be obtained as follows:

[0169] w = softmax(i m ,τ)

[0170] Where w = (w1, ... w N ), where τ is the temperature parameter of the softmax function. This posterior probability w can then be used as the weight.

[0171] S9. Distribute the fully integrated model from S8 to each user node for application on each node (including the central node and each user node) for semantic segmentation.

[0172] include:

[0173] (S91) Input the point cloud data p to be processed into the encoder g, and calculate the intermediate features r and the Gaussian distribution set.

[0174] (S92) Based on the intermediate feature r and the Gaussian distribution set Calculate the weight w;

[0175] (S93) Calculate the segmentation result according to the formula in S81:

[0176]

[0177] As can be seen, this embodiment solves the problem that in traditional federated learning scenarios, only user nodes can access their own scenario data during training, while the goal of the task is for the central node to generate a model with good generalization performance on the data of each user node, which is impossible to achieve.

[0178] Secondly, the federated learning-based point cloud semantic segmentation scheme in this embodiment designs a scene adapter that learns scene-specific information for user nodes based on the relationship between scene and object layout. The scene adapter models the current scene, combining multi-scale features with a learnable scene topology matrix. Furthermore, during the training process of user nodes, only the parameters of the scene adapter are continuously updated; therefore, knowledge of scene-specific spatial topology can be acquired.

[0179] Furthermore, the point cloud semantic segmentation scheme based on federated learning in this embodiment incorporates a scene evaluation module to estimate the relationship between samples and known scenes, generalize the scene spatial information learned by each scene adapter, improve the overall model's generalization ability, and ultimately achieve higher segmentation performance in the federated learning scenario. Each user node establishes a multivariate Gaussian distribution based on the intermediate features during training and sends the mean and covariance matrix of the distribution back to the central node. During the evaluation process, given the intermediate features of the input samples, the Gaussian distribution of the user nodes is sampled, and the weights of the samples and each scene are obtained through a Gaussian process to comprehensively utilize the knowledge of each scene.

[0180] When the solution in this embodiment is applied to autonomous driving scenarios, it effectively solves the training difficulties caused by data privacy in autonomous driving scenarios. It extracts unique information from different scenarios and generalizes the knowledge, which can effectively improve the semantic segmentation performance in various scenarios. Furthermore, the knowledge from each scenario can be integrated into the adapter, and through the integration of the adapter, the generalization of the overall network in federated learning scenarios can be improved, resulting in good semantic segmentation performance in various scenarios.

[0181] Example 4

[0182] Refer again Figure 1The system shown is considered as a point cloud semantic segmentation system capable of performing point cloud semantic segmentation.

[0183] The central node can be used to train a basic point cloud semantic segmentation model using its own central point cloud data, and then distribute the trained basic point cloud semantic segmentation model to each user node.

[0184] Each of the multiple user nodes is used to receive the trained basic point cloud semantic segmentation model sent by the central node; based on the basic point cloud semantic segmentation model and the scene adapter corresponding to the current user node for processing the privacy point cloud data of the current user node, a scene point cloud semantic segmentation model for the current user node is constructed; using the privacy point cloud data of the current user node as training samples, the scene point cloud semantic segmentation model is trained to obtain the trained scene point cloud semantic segmentation model; and the trained scene point cloud semantic segmentation model is sent to the central node.

[0185] The central node is also used to integrate and generate a global point cloud semantic segmentation model based on multiple scene point cloud semantic segmentation models sent by multiple user nodes.

[0186] In addition, the central node is also used to distribute the global point cloud semantic segmentation model to multiple user nodes to update the scene point cloud semantic segmentation model of each user node.

[0187] as well as,

[0188] The central node and each of the multiple user nodes are used to obtain the encoding vector of the 3D scene point cloud data through the global point cloud semantic segmentation model for the 3D scene point cloud data to be detected, and obtain the intermediate features and Gaussian distribution corresponding to the 3D scene point cloud data based on the encoding vector; determine the weights of the multiple scene adapters in the global point cloud semantic segmentation model based on the intermediate features and Gaussian distribution; and perform semantic segmentation on the 3D scene point cloud data based on the weights of each scene adapter and the output scene features.

[0189] The process of point cloud semantic segmentation by the above point cloud semantic segmentation system is described in a relatively brief manner. Its specific implementation can be referred to the description of the relevant parts in the aforementioned multiple method embodiments, and it has corresponding beneficial effects, which will not be detailed here.

[0190] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0191] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0192] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0193] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for generating a point cloud semantic segmentation model, comprising: The training completed basic point cloud semantic segmentation model is sent by the central node. The basic point cloud semantic segmentation model is a neural network model with an encoder-decoder structure. Based on the aforementioned basic point cloud semantic segmentation model and the scene adapter corresponding to the current user node, which is used to process the privacy point cloud data of the current user node, a scene point cloud semantic segmentation model for the current user node is constructed. Using the privacy point cloud data of the current user node as training samples, the scene point cloud semantic segmentation model is trained to obtain the trained scene point cloud semantic segmentation model. The model data of the trained scene point cloud semantic segmentation model is sent to the central node, so that the central node can integrate the model data of the scene point cloud semantic segmentation model of the current user node with the model data of the trained scene point cloud semantic segmentation model sent by other user nodes to generate a global point cloud semantic segmentation model. The step of constructing a scene point cloud semantic segmentation model for the current user node based on the basic point cloud semantic segmentation model and the scene adapter corresponding to the current user node for processing the privacy point cloud data of the current user node includes: A scene adapter corresponding to the current user node, used to process the privacy point cloud data of the current user node, is inserted between the encoder-decoder structure. The output of the encoder is used as the input of the scene adapter, and the output of the scene adapter is used as the input of the decoder to construct the scene point cloud semantic segmentation model of the current user node.

2. The method according to claim 1, wherein, The step of training the scene point cloud semantic segmentation model using the privacy point cloud data of the current user node as training samples includes: The encoder in the scene point cloud semantic segmentation model encodes the privacy point cloud data of the current user node to obtain the corresponding encoding vector. Using the scene adapter, based on the encoding vector, the scene topology information corresponding to the privacy point cloud data is learned, and the corresponding scene features are output. The decoder is used to train point cloud semantic segmentation based on the scene features until the training termination condition is met.

3. The method according to claim 2, wherein, The scene adapter includes a convolutional layer part, a feature aggregation part, and a scene topology part; The step of learning scene topology information corresponding to the privacy point cloud data based on the encoded vector through the scene adapter and outputting the corresponding scene features includes: The encoding vector is convolved through the first convolutional layer in the convolutional layer part of the scene adapter to obtain the first feature vector; The first feature vector is subjected to dilated convolution through multiple consecutive dilated convolutional layers following the first convolutional layer to obtain multiple corresponding second feature vectors. The multiple second feature vectors are aggregated through the feature aggregation part to obtain multi-level aggregated features; The scene topology information is learned based on the multi-level aggregated features by using the learnable matrix in the scene topology part, and the scene features are output according to the learnable matrix and the multi-level aggregated features obtained after learning.

4. The method according to any one of claims 1-3, wherein, The method further includes: During the training process of the scene point cloud semantic segmentation model, intermediate features corresponding to the privacy point cloud data are extracted, and a multivariate Gaussian distribution is established based on the intermediate features.

5. The method according to claim 4, wherein, The extraction of intermediate features corresponding to the privacy point cloud data includes: Obtain the encoded vector output by the encoder; Spatial pooling is performed on the encoded vector to obtain high-level features; and after the encoded vector is restored to the corresponding point cloud, meshing and channel-level pooling are performed to obtain low-level features. The intermediate feature is obtained by concatenating the high-level feature and the low-level feature.

6. The method according to claim 4, wherein, The trained scene point cloud semantic segmentation model data is sent to the central node, so that the central node can integrate the trained scene point cloud semantic segmentation model data sent by other user nodes with the model data of the current user node's scene point cloud semantic segmentation model to generate a global point cloud semantic segmentation model, including: The model parameters, intermediate features, and multivariate Gaussian distribution of the trained scene point cloud semantic segmentation model are sent to the central node, so that the central node can integrate the model parameters, intermediate features, and multivariate Gaussian distribution of the scene point cloud semantic segmentation model of the current user node with the model parameters, intermediate features, and multivariate Gaussian distribution of the trained scene point cloud semantic segmentation model sent by other user nodes to generate a global point cloud semantic segmentation model.

7. A method for generating a point cloud semantic segmentation model, comprising: Receive model data of multiple scene point cloud semantic segmentation models sent by multiple user nodes, wherein the scene point cloud semantic segmentation models are trained and generated by the method described in any one of claims 1-6; Determine the corresponding weights for the model data of the multiple scene point cloud semantic segmentation models respectively; Based on the model data and corresponding weights of the multiple scene point cloud semantic segmentation models, a global point cloud semantic segmentation model is generated.

8. The method according to claim 7, wherein, The model data includes: model parameters, intermediate features, and multivariate Gaussian distribution of the scene point cloud semantic segmentation model.

9. The method according to claim 8, wherein, The step of determining corresponding weights for the model data of the multiple scene point cloud semantic segmentation models includes: For each scene point cloud semantic segmentation model, a preset number of scene features are collected from the multivariate Gaussian distribution of the model data of that scene point cloud semantic segmentation model. The scene features are processed using one-hot encoding to obtain the corresponding scene one-hot code; Based on the scene one-hot code and the intermediate features in the model data of the scene point cloud semantic segmentation model, a Gaussian process is established to estimate the posterior probability that the intermediate features belong to the preset scene, and the posterior probability is used as the weight corresponding to the scene point cloud semantic segmentation model.

10. The method according to any one of claims 7-9, wherein, The method further includes: The global point cloud semantic segmentation model is distributed to multiple user nodes to update the scene point cloud semantic segmentation model of each user node.

11. A point cloud semantic segmentation method, comprising: Acquire point cloud data of the 3D scene to be inspected; The encoding vector of the 3D scene point cloud data is obtained through a global point cloud semantic segmentation model, and the intermediate features and Gaussian distribution corresponding to the 3D scene point cloud data are obtained based on the encoding vector. Based on the intermediate features and the Gaussian distribution, the weights corresponding to the multiple scene adapters in the global point cloud semantic segmentation model are determined respectively. Based on the weights of each scene adapter and the output scene features, the 3D scene point cloud data is semantically segmented. The global point cloud semantic segmentation model is generated based on scene point cloud semantic segmentation models sent by multiple user nodes. Each user node's scene point cloud semantic segmentation model is constructed by inserting a scene adapter corresponding to the current user node, which is used to process the privacy point cloud data of the current user node, between the encoder and decoder structures of the basic point cloud semantic segmentation model, with the output of the encoder as the input of the scene adapter and the output of the scene adapter as the input of the decoder.

12. A point cloud semantic segmentation system, comprising: Central node and multiple user nodes; in: The central node is used to train a basic point cloud semantic segmentation model using its own central point cloud data, and to distribute the trained basic point cloud semantic segmentation model to each user node. The basic point cloud semantic segmentation model is a neural network model with an encoder-decoder structure. Each of the multiple user nodes is used to receive the trained basic point cloud semantic segmentation model sent by the central node; based on the basic point cloud semantic segmentation model, a scene adapter corresponding to the current user node, used to process the privacy point cloud data of the current user node, is inserted between the encoder-decoder structure of the basic point cloud semantic segmentation model, with the output of the encoder as the input of the scene adapter and the output of the scene adapter as the input of the decoder, to construct the scene point cloud semantic segmentation model of the current user node; the scene point cloud semantic segmentation model is trained using the privacy point cloud data of the current user node as training samples to obtain the trained scene point cloud semantic segmentation model; and the trained scene point cloud semantic segmentation model is sent to the central node. The central node is also used to integrate and generate a global point cloud semantic segmentation model based on multiple scene point cloud semantic segmentation models sent by multiple user nodes.

13. The system according to claim 12, wherein, The central node is also used to distribute the global point cloud semantic segmentation model to multiple user nodes to update the scene point cloud semantic segmentation model of each user node. as well as, The central node and the multiple user nodes are all used to obtain the encoding vector of the 3D scene point cloud data through a global point cloud semantic segmentation model for the 3D scene point cloud data to be detected, and to obtain the intermediate features and Gaussian distribution corresponding to the 3D scene point cloud data based on the encoding vector; to determine the weights corresponding to the multiple scene adapters in the global point cloud semantic segmentation model based on the intermediate features and the Gaussian distribution; and to perform semantic segmentation on the 3D scene point cloud data based on the weights of each scene adapter and the output scene features.

Images