Point cloud data processing model training methods, devices, equipment and media

Abstract

This disclosure provides a point cloud data processing model training method that can be applied to the field of artificial intelligence technology. The method includes: inputting sample point cloud data into a student network and outputting a first feature map related to a first intermediate stage of the student network; inputting sample point cloud data into a teacher network and outputting a second feature map related to a second intermediate stage of the teacher network; reconstructing knowledge from the first feature map using the target feature map to obtain a first reconstructed feature map; determining a first loss value based on the first reconstructed feature map and the second feature map; and adjusting the model parameters of the student network using the first loss value to obtain a target point cloud data processing model. Furthermore, this disclosure also provides a point cloud data processing method, a point cloud data processing model training device, a point cloud data processing apparatus, a device, and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a point cloud data processing model training method, a point cloud data processing method, a point cloud data processing model training device, a point cloud data processing device, an electronic device, and a storage medium. Background Technology

[0002] With the development of 3D sensing technology, point cloud data is widely used in many fields such as autonomous driving and robot grasping. Deep neural networks, as the mainstream solution for 3D point cloud analysis, have demonstrated excellent performance in point cloud data processing.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technologies: the computational overhead of neural networks used to process point cloud data in the related technologies is large, making it difficult to directly apply to terminal devices with limited computing resources. Summary of the Invention

[0004] In view of the above, this disclosure provides a point cloud data processing model training method, a point cloud data processing method, a point cloud data processing model training device, a point cloud data processing device, an electronic device, a storage medium, and a computer program product.

[0005] One aspect of this disclosure provides a method for training a point cloud data processing model, comprising: inputting sample point cloud data into a student network and outputting a first feature map related to a first intermediate stage of the student network; inputting the sample point cloud data into a teacher network and outputting a second feature map related to a second intermediate stage of the teacher network, wherein the first intermediate stage and the second intermediate stage correspond one-to-one; reconstructing knowledge from the first feature map using a target feature map to obtain a first reconstructed feature map, wherein the target feature map is related to a target intermediate stage of the student network and the target intermediate stage is adjacent to the first intermediate stage; determining a first loss value based on the first reconstructed feature map and the second feature map; and adjusting the model parameters of the student network using the first loss value to obtain a target point cloud data processing model.

[0006] According to embodiments of this disclosure, the target feature map includes a first target feature map and a second target feature map; the above-mentioned knowledge reconstruction of the first feature map using the target feature map to obtain a first reconstructed feature map includes: for the i-th first intermediate stage, using the first target feature map to reconstruct the knowledge of the first feature map related to the i-th first intermediate stage to obtain a second reconstructed feature map related to the i-th first intermediate stage, wherein the first target feature map is represented as a second reconstructed feature map related to the (i+1)-th first intermediate stage; and using the second target feature map to reconstruct the knowledge of the second reconstructed feature map related to the i-th first intermediate stage to obtain a first reconstructed feature map related to the i-th first intermediate stage, wherein the second target feature map is represented as a first reconstructed feature map related to the (i-1)-th first intermediate stage; wherein i is a positive integer less than or equal to N, and N represents the number of first intermediate stages in the student network.

[0007] According to an embodiment of this disclosure, for the i-th first intermediate stage, the method of reconstructing knowledge from the first feature map related to the i-th first intermediate stage using the first target feature map to obtain a second reconstructed feature map related to the i-th first intermediate stage includes: for the i-th first intermediate stage, when i is less than N, upsampling the first target feature map to obtain a first sampled feature map; processing the first feature map related to the i-th first intermediate stage using a first convolutional network to obtain a third feature map; concatenating the first sampled feature map and the third feature map to obtain a fourth feature map; processing the fourth feature map using a second convolutional network to obtain a first weight matrix; and determining the second reconstructed feature map related to the i-th first intermediate stage based on the first weight matrix, the first target feature map, and the first feature map related to the i-th first intermediate stage.

[0008] According to an embodiment of this disclosure, the method further includes: when i equals N, processing the first feature map associated with the i-th first intermediate stage using a third convolutional network to obtain a second reconstructed feature map associated with the i-th first intermediate stage.

[0009] According to an embodiment of this disclosure, the above-mentioned knowledge reconstruction of the second reconstructed feature map related to the i-th first intermediate stage using the second target feature map to obtain the first reconstructed feature map related to the i-th first intermediate stage includes: when i is greater than 1, performing downsampling processing on the second target feature map to obtain a second sampled feature map; processing the second reconstructed feature map related to the i-th first intermediate stage using a fourth convolutional network to obtain a fifth feature map; concatenating the second sampled feature map and the fifth feature map to obtain a sixth feature map; processing the sixth feature map using a fifth convolutional network to obtain a second weight matrix; and determining the first reconstructed feature map related to the i-th first intermediate stage based on the second weight matrix, the second target feature map, and the second reconstructed feature map related to the i-th first intermediate stage.

[0010] According to an embodiment of this disclosure, the method further includes: when i equals 1, using a sixth convolutional network to process a second reconstructed feature map related to the i-th intermediate stage to obtain a first reconstructed feature map related to the i-th intermediate stage.

[0011] According to embodiments of this disclosure, the method further includes: determining a first set of location information related to the first intermediate stage after the sample point cloud data is input into the student network, wherein the first set of location information includes first location information of multiple points input into the first intermediate stage; determining a second set of location information related to the second intermediate stage after the sample point cloud data is input into the teacher network, wherein the second set of location information includes second location information of multiple points input into the second intermediate stage; and performing position calibration on the first reconstructed feature map using the first set of location information and the second set of location information to obtain a calibration feature map; wherein determining a first loss value based on the first reconstructed feature map and the second feature map includes: determining the first loss value based on the calibration feature map and the second feature map.

[0012] According to embodiments of this disclosure, the above-mentioned position calibration of the first reconstructed feature map using the first location information set and the second location information set to obtain a calibration feature map includes: for each second location information in the second location information set, determining multiple target location information from multiple first location information in the first location information set; determining a third weight matrix based on the second location information and the multiple target location information; and obtaining the calibration feature map based on the third weight matrix and the first reconstructed feature map.

[0013] According to embodiments of this disclosure, the method further includes: inputting the sample point cloud data into the student network and outputting the output features of the student network; and calculating the cross-entropy loss between the output features and the labels of the sample point cloud data to obtain a second loss value; wherein, adjusting the model parameters of the student network using the first loss value to obtain the target point cloud data processing model includes: adjusting the model parameters of the student network using the first loss value and the second loss value to obtain the target point cloud data processing model.

[0014] One aspect of this disclosure provides a point cloud data processing method, comprising: inputting point cloud data to be processed into a target point cloud data processing model, and outputting point cloud data processing results; wherein the target point cloud data processing model includes a method for training point cloud data processing models.

[0015] Another aspect of this disclosure provides a point cloud data processing model training apparatus, comprising: a first input module for inputting sample point cloud data into a student network and outputting a first feature map related to a first intermediate stage of the student network; a second input module for inputting the sample point cloud data into a teacher network and outputting a second feature map related to a second intermediate stage of the teacher network, wherein the first intermediate stage and the second intermediate stage correspond one-to-one; a first processing module for performing knowledge reconstruction on the first feature map using a target feature map to obtain a first reconstructed feature map, wherein the target feature map is related to a target intermediate stage of the student network and the target intermediate stage is adjacent to the first intermediate stage; a first determining module for determining a first loss value based on the first reconstructed feature map and the second feature map; and a training module for adjusting the model parameters of the student network using the first loss value to obtain a target point cloud data processing model.

[0016] Another aspect of this disclosure provides a point cloud data processing apparatus, comprising: a second processing module for inputting point cloud data to be processed into a target point cloud data processing model and outputting point cloud data processing results; wherein the target point cloud data processing model is obtained by training through a point cloud data processing model training method.

[0017] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0018] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0019] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0020] According to embodiments of this disclosure, during the process of using a teacher network to guide the training of a student network, the semantic information of the first feature maps related to different first intermediate stages of the student network can be integrated through a knowledge reconstruction method. Then, the second feature map output by the teacher network is used to perform feature distillation on the first reconstructed feature map obtained through knowledge reconstruction, thus realizing cross-layer guidance of the student network by the teacher network. Through the above technical means, a narrow and shallow student network can be used to replace a wide and deep teacher network in point cloud data processing scenarios. Therefore, it at least partially overcomes the technical problem in related technologies where the computational overhead of neural networks used to process point cloud data is large, making it difficult to directly apply to terminal devices with limited computing resources. Applying the target point cloud data processing model obtained by training the student network can effectively reduce the computational overhead of point cloud analysis tasks. Simultaneously, the target point cloud data processing model trained through the above technical means can learn multi-level and multi-scale contextual knowledge from the teacher network, effectively improving the model's processing accuracy. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 An exemplary system architecture for applying point cloud data processing model training methods and apparatus according to embodiments of the present disclosure is illustrated.

[0023] Figure 2 A flowchart illustrating a point cloud data processing model training method according to an embodiment of the present disclosure is shown.

[0024] Figure 3A A flowchart illustrating a knowledge reconstruction method according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 3B A schematic diagram of a top-down knowledge reconstruction network according to an embodiment of the present disclosure is shown.

[0026] Figure 3C A schematic diagram of a bottom-up knowledge reconstruction network according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A flowchart illustrating a point cloud data processing model training method according to another embodiment of the present disclosure is shown.

[0028] Figure 5A schematic diagram of a sampling point calibration method according to an embodiment of the present disclosure is shown.

[0029] Figure 6 The diagram illustrates a point cloud data processing model training method according to an embodiment of the present disclosure.

[0030] Figure 7 A flowchart illustrating a point cloud data processing method according to an embodiment of the present disclosure is shown.

[0031] Figure 8 A block diagram of a point cloud data processing model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 9 A block diagram of a point cloud data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0033] Figure 10 A block diagram of an electronic device suitable for implementing a point cloud data processing model training method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0034] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0037] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0038] The development of 3D sensing devices has enabled point cloud data to be widely used in many fields, such as autonomous driving and robotic grasping. With the rapid advancement of deep learning, deep neural networks have become the mainstream solution for 3D point cloud analysis, demonstrating excellent performance in point cloud data processing. However, due to the generally high computational overhead of point cloud analysis networks based on deep neural networks, they are difficult to directly apply to terminal devices with limited computing resources in practical applications. Furthermore, directly compressing the network would lead to a decrease in processing performance.

[0039] In view of this, the embodiments of this disclosure are based on the idea of ​​knowledge distillation, and use a trained complex teacher network to guide a lightweight student network to improve the performance of the lightweight student network, thereby at least partially overcoming the technical problems existing in the related art.

[0040] Specifically, embodiments of this disclosure provide a point cloud data processing model training method, a point cloud data processing method, a point cloud data processing model training device, an electronic device, a storage medium, and a computer program product. The point cloud data processing model training method includes: inputting sample point cloud data into a student network and outputting a first feature map related to a first intermediate stage of the student network; inputting sample point cloud data into a teacher network and outputting a second feature map related to a second intermediate stage of the teacher network, wherein the first intermediate stage and the second intermediate stage correspond one-to-one; reconstructing knowledge from the first feature map using a target feature map to obtain a first reconstructed feature map, wherein the target feature map is related to a target intermediate stage of the student network, and the target intermediate stage is adjacent to the first intermediate stage; determining a first loss value based on the first reconstructed feature map and the second feature map; and adjusting the model parameters of the student network using the first loss value to obtain a target point cloud data processing model.

[0041] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0042] Figure 1 This illustration schematically depicts an exemplary system architecture to which point cloud data processing model training methods and apparatus can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0043] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105.

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with communication capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers. Terminal devices 101, 102, and 103 can have a memory that stores sample point cloud data for model training; or, terminal devices 101, 102, and 103 can be connected to an external storage device that stores sample point cloud data for model training.

[0045] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and / or wireless communication links.

[0046] Server 105 can be a server that provides computing resources to support various computing tasks. For example, when a user trains a student model using terminal devices 101, 102, and 103, server 105 can provide computing resources to support the training process of the student model.

[0047] It should be noted that the point cloud data processing model training method provided in this embodiment can generally be executed by server 105. Correspondingly, the point cloud data processing model training device provided in this embodiment can generally be located in server 105. The point cloud data processing model training method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the point cloud data processing model training device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the point cloud data processing model training method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Accordingly, the point cloud data processing model training device provided in this embodiment of the present disclosure may also be set in terminal device 101, 102, or 103, or in other terminal devices different from terminal device 101, 102, or 103.

[0048] For example, the sample point cloud data can be originally stored in any one of terminal devices 101, 102, or 103 (e.g., terminal device 101, but not limited thereto), or stored on an external storage device and can be imported into terminal device 101. Then, terminal device 101 can locally execute the point cloud data processing model training method provided in this disclosure embodiment to train a student network using the sample point cloud data to obtain a target point cloud data processing model. Alternatively, terminal device 101 can send the sample point cloud data to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters receiving the sample point cloud data can execute the point cloud data processing model training method provided in this disclosure embodiment to train a student network using the sample point cloud data to obtain a target point cloud data processing model.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] Figure 2 A flowchart illustrating a point cloud data processing model training method according to an embodiment of the present disclosure is shown.

[0051] like Figure 2 As shown, the method includes operations S201 to S205.

[0052] In operation S201, the sample point cloud data is input into the student network, and the first feature map related to the first intermediate stage of the student network is output.

[0053] In operation S202, the sample point cloud data is input into the teacher network, and the second feature map related to the second intermediate stage of the teacher network is output. The first intermediate stage and the second intermediate stage correspond one-to-one.

[0054] In operation S203, the first feature map is reconstructed using the target feature map to obtain the first reconstructed feature map. The target feature map is related to the target intermediate stage of the student network, and the target intermediate stage is adjacent to the first intermediate stage.

[0055] In operation S204, a first loss value is determined based on the first reconstructed feature map and the second feature map.

[0056] In operation S205, the first loss value is used to adjust the model parameters of the student network to obtain the target point cloud data processing model.

[0057] According to embodiments of this disclosure, the sample point cloud data may include the spatial coordinates of each point in the point cloud. The sample point cloud data can be acquired using sensing devices such as depth cameras and LiDAR.

[0058] According to embodiments of this disclosure, the teacher network can be any neural network capable of processing 3D point cloud data, including but not limited to PointNet, PointNet++, PointConv, etc. Before inputting sample point cloud data into the teacher network, the teacher network may have already been trained using other sample point cloud data.

[0059] According to embodiments of this disclosure, the student network can be a lightweight neural network obtained by simplifying the backbone of the teacher network. That is, the student network can have the same input and output layers as the teacher network, but its backbone is shallower and / or narrower than that of the teacher network. For example, the teacher network can be a PointNet++ network, and the student network can be obtained by reducing the width of the teacher network to 1 / 8 of its original width. By reducing the width or depth, the student network has fewer parameters and lower computational complexity, thus achieving the goal of lightweighting.

[0060] According to embodiments of this disclosure, the network structure of the student network is divided into stages, namely, an input stage, a first intermediate stage, and an output stage. Specifically, the first intermediate stage may include the network structure of the student network excluding the input and output layers. There may be one or more first intermediate stages, and the network structure corresponding to each first intermediate stage may include one or more intermediate layers. The feature maps output by each intermediate layer are identical in spatial dimension; that is, the feature maps output by one or more intermediate layers within the same first intermediate stage may have different numbers of channels but the same width and height.

[0061] According to embodiments of this disclosure, the first feature map may be a feature map output by the network structure corresponding to the first intermediate stage in the student network.

[0062] According to embodiments of this disclosure, the network structure of the teacher network is divided into stages, namely, an input stage, a second intermediate stage, and an output stage. Specifically, the second intermediate stage may include network structures in the teacher network other than the input and output layers. There may be one or more second intermediate stages, and the network structure corresponding to each second intermediate stage may include one or more intermediate layers. The feature maps output by each intermediate layer are identical in spatial dimension; that is, the feature maps output by one or more intermediate layers within the same second intermediate stage may have different numbers of channels but the same width and height.

[0063] According to embodiments of this disclosure, the second feature map may be a feature map output by the network structure corresponding to the second intermediate stage in the teacher network.

[0064] According to embodiments of this disclosure, the number of first intermediate stages in the student network and the number of second intermediate stages in the teacher network can be equal. For example, if the student network can have four first intermediate stages, then the teacher network used to guide the training of the student network can also have four second intermediate stages. When the number of first intermediate stages in the student network is greater than one, operations S203 to S204 can be performed on the first feature map of each first intermediate stage and the second feature map of the corresponding second intermediate stage, and in operation S205, the obtained multiple first loss values ​​are used to train the student network.

[0065] According to embodiments of this disclosure, the target intermediate stage may refer to other first intermediate stages adjacent to the first intermediate stage. For example, the first intermediate stage may be the i-th first intermediate stage in the student network, then the target intermediate stage may refer to the (i+1)-th and / or the (i-1)-th first intermediate stage.

[0066] According to embodiments of this disclosure, adjusting the model parameters of the student network using the first loss value can be achieved using gradient descent or an improved algorithm of gradient descent, without limitation herein.

[0067] According to embodiments of this disclosure, during the process of using a teacher network to guide the training of a student network, the semantic information of the first feature maps related to different first intermediate stages of the student network can be integrated through a knowledge reconstruction method. Then, the second feature map output by the teacher network is used to perform feature distillation on the first reconstructed feature map obtained through knowledge reconstruction, thus realizing cross-layer guidance of the student network by the teacher network. Through the above technical means, a narrow and shallow student network can be used to replace a wide and deep teacher network in point cloud data processing scenarios. Therefore, it at least partially overcomes the technical problem in related technologies where the computational overhead of neural networks used to process point cloud data is large, making it difficult to directly apply to terminal devices with limited computing resources. Applying the target point cloud data processing model obtained by training the student network can effectively reduce the computational overhead of point cloud analysis tasks. Simultaneously, the target point cloud data processing model trained through the above technical means can learn multi-level and multi-scale contextual knowledge from the teacher network, effectively improving the model's processing accuracy.

[0068] The following is for reference. Figures 3A-3C and Figures 4-6 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0069] Figure 3A A flowchart illustrating a knowledge reconstruction method according to an embodiment of the present disclosure is shown schematically.

[0070] like Figure 3A As shown, the method includes operations S301 to S302.

[0071] In operation S301, for the i-th first intermediate stage, the first feature map related to the i-th first intermediate stage is reconstructed using the first target feature map to obtain the second reconstructed feature map related to the i-th first intermediate stage. The first target feature map is represented as the second reconstructed feature map related to the (i+1)-th first intermediate stage.

[0072] In operation S302, the second target feature map is used to reconstruct the knowledge of the second reconstructed feature map related to the i-th first intermediate stage, resulting in the first reconstructed feature map related to the i-th first intermediate stage. The second target feature map is represented as the first reconstructed feature map related to the i.1-th first intermediate stage.

[0073] According to embodiments of this disclosure, i can be a positive integer less than or equal to N, where N can represent the number of first intermediate stages in the student network.

[0074] According to embodiments of this disclosure, for the i-th first intermediate stage of the student network, the i-th first intermediate stage may have one or two adjacent first intermediate stages, i.e., the (i+1)-th first intermediate stage and / or the (i-1)-th first intermediate stage. Correspondingly, the target feature map may include a first target feature map and / or a second target feature map. Specifically, when i equals N, the target feature map may include a second target feature map, and the second reconstructed feature map associated with the i-th first intermediate stage may be determined solely based on the first feature map associated with the i-th first intermediate stage; when i equals 1, the target feature map may include a first target feature map, and the first reconstructed feature map associated with the i-th first intermediate stage may be determined solely based on the second reconstructed feature map associated with the i-th first intermediate stage; when i is greater than 1 and less than N, the target feature map may include both a first target feature map and a second target feature map.

[0075] According to embodiments of this disclosure, the second reconstructed feature map associated with the i-th first intermediate stage can be calculated using formula (1):

[0076]

[0077] In the formula, TD i This represents the second reconstructed feature map associated with the i-th first intermediate stage; TD i+1 Represents the first target feature map; F s，i This represents the first feature map associated with the i-th first intermediate stage; TDRKR(TD i+1 F s，i ) represents using a top-down knowledge reconfiguration network to process the first target feature map and the first feature map related to the i-th first intermediate stage; Conv1×1(F s，i This can be represented as using a 1×1 convolution to process the first feature map associated with the i-th first intermediate stage.

[0078] According to embodiments of this disclosure, the first reconstructed feature map associated with the i-th first intermediate stage can be calculated using formula (2):

[0079]

[0080] In the formula, BU i This represents the first reconstructed feature map associated with the i-th first intermediate stage; BU i-1 Represents the second target feature map; BUKR(BU i-1 TD i) represents using a bottom-up knowledge reconfiguration network to process the second target feature map and the second reconstructed feature map related to the i-th first intermediate stage; Conv1×1(TD i ) represents using a 1×1 convolution to process the second reconstructed feature map associated with the i-th first intermediate stage.

[0081] Figure 3B A schematic diagram of a top-down knowledge reconstruction network according to an embodiment of the present disclosure is shown.

[0082] like Figure 3B As shown, taking the i-th first intermediate stage of the student network as an example, when i is less than N, using top-down knowledge reconstruction of the network to process the first target feature map 311 and the first feature map 312 related to the i-th first intermediate stage can include the following operations:

[0083] The first target feature map 311 is upsampled to obtain the first sampled feature map 313; the first feature map 312 related to the i-th first intermediate stage is processed by the first convolutional network to obtain the third feature map 314; the first sampled feature map 313 and the third feature map 314 are connected to obtain the fourth feature map 315; the fourth feature map 315 is processed by the second convolutional network to obtain the first weight matrix 316; and based on the first weight matrix 316, the first target feature map 311 and the first feature map 312 related to the i-th first intermediate stage, the second reconstructed feature map 317 related to the i-th first intermediate stage is determined.

[0084] According to embodiments of this disclosure, upsampling the first target feature map 311 to obtain the first sampled feature map 313 can be understood as: by upsampling, the number of features in the first target feature map 311 is matched to the feature map output by the teacher network. The first target feature map 311 can be represented as... The first sampled feature map 313 obtained after upsampling can be represented as:

[0085] According to embodiments of this disclosure, the upsampling process may be, for example, a distance-based interpolation method, and is not limited thereto.

[0086] According to embodiments of this disclosure, the first convolutional network can be a 1×1 convolutional layer. The first convolutional network can be used to change the dimension of the input feature map.

[0087] According to embodiments of this disclosure, processing the first feature map 312 related to the i-th first intermediate stage using a first convolutional network to obtain the third feature map 314 can be understood as: passing the first feature map 312 through a 1×1 convolutional layer to match the dimension of the first sampled feature map 313. The first feature map 312 can be represented as... After processing by the first convolutional network, the resulting third feature map 314 can be represented as:

[0088] According to embodiments of this disclosure, the fourth feature map 315 obtained by concatenating the first sampling feature map 313 and the third feature map 314 can be represented as follows:

[0089] According to embodiments of this disclosure, the second convolutional network can be a 1×1 convolutional layer with a sigmoid activation function. The second convolutional network can be used to change the dimension of the input feature map and normalize the elements in the feature map, that is, to transform the values ​​of each element to the (0, 1) interval.

[0090] According to embodiments of this disclosure, processing the fourth feature map 315 using a second convolutional network to obtain the first weight matrix 316 can be understood as: using a 1×1 convolution with a sigmoid activation function to process the fourth feature map 315 to generate the first weight matrix 316. The first weight matrix 316 can be represented as...

[0091] According to embodiments of this disclosure, using the first weight matrix 316 to process the first target feature map 311 and the first feature map 312 associated with the i-th first intermediate stage can be understood as using a gate mechanism to control the information flow from the first target feature map 311 and the first feature map 312 associated with the i-th first intermediate stage, respectively.

[0092] According to embodiments of this disclosure, the first weight matrix 316 can be divided into a first gate vector 318 and a second gate vector 319. The first gate vector 318 can be represented as... The second gate vector 319 can be represented as:

[0093] According to embodiments of this disclosure, the calculation process of the second reconstructed feature map 317 associated with the i-th first intermediate stage can be as shown in formula (3):

[0094]

[0095] In the formula, This can be represented as repeating the first gate vector 318 d times along the second dimension; This is represented as repeating the second gate vector 319 d times along the second dimension; · represents the dot product of the matrices.

[0096] According to embodiments of this disclosure, when i equals N, using a top-down knowledge reconstruction network to process the first target feature map 311 and the first feature map 312 associated with the i-th first intermediate stage may include the following operations:

[0097] The first feature map 312 associated with the i-th first intermediate stage is processed by the third convolutional network to obtain the second reconstructed feature map 317 associated with the i-th first intermediate stage.

[0098] According to embodiments of this disclosure, the third convolutional network can be a 1×1 convolutional layer. The third convolutional network can be used to change the dimension of the input feature map.

[0099] According to embodiments of this disclosure, processing the first feature map 312 associated with the i-th first intermediate stage using a third convolutional network can be understood as: using a 1×1 convolutional layer to make the first feature map 312 match the number of features of the feature map output by the teacher network.

[0100] Figure 3C A schematic diagram of a bottom-up knowledge reconstruction network according to an embodiment of the present disclosure is shown.

[0101] like Figure 3C As shown, taking the i-th first intermediate stage of the student network as an example, when i is greater than 1, using the bottom-up knowledge reconstruction network to process the second target feature map 321 and the second reconstructed feature map 317 related to the i-th first intermediate stage can include the following operations:

[0102] The second target feature map 321 is downsampled to obtain the second sampled feature map 322; the second reconstructed feature map 317 related to the i-th first intermediate stage is processed by the fourth convolutional network to obtain the fifth feature map 323; the second sampled feature map 322 and the fifth feature map 323 are connected to obtain the sixth feature map 324; the sixth feature map 324 is processed by the fifth convolutional network to obtain the second weight matrix 325; and, based on the second weight matrix 325, the second target feature map 321 and the second reconstructed feature map 317 related to the i-th first intermediate stage, the first reconstructed feature map 326 related to the i-th first intermediate stage is determined.

[0103] According to an embodiment of this disclosure, the second target feature map 321 is downsampled to obtain a second sampled feature map 322: the downsampling method makes the number of features in the second target feature map 321 match the number of features in the feature map output by the teacher network. The second target feature map 321 can be represented as BU. i-1The second sampled feature map 322 obtained after downsampling can be represented as:

[0104] According to embodiments of this disclosure, the fourth convolutional network can be a 1×1 convolutional layer. The fourth convolutional network can be used to change the dimension of the input feature map.

[0105] According to embodiments of this disclosure, processing the second reconstructed feature map 317 related to the i-th first intermediate stage using a fourth convolutional network to obtain the fifth feature map 323 can be understood as: passing the second reconstructed feature map 317 through a 1×1 convolutional layer to match the dimension of the second sampled feature map 322. The fifth feature map 323 obtained after processing by the fourth convolutional network can be represented as...

[0106] According to embodiments of this disclosure, connecting the second sampling feature map 322 and the fifth feature map 323 to obtain the sixth feature map 324 can be represented as follows:

[0107] According to embodiments of this disclosure, the fifth convolutional network can be a 1×1 convolutional layer with a sigmoid activation function. The fifth convolutional network can be used to change the dimension of the input feature map and normalize the elements in the feature map, that is, to transform the values ​​of each element to the (0, 1) interval.

[0108] According to embodiments of this disclosure, processing the sixth feature map 324 using a fifth convolutional network to obtain the second weight matrix 325 can be understood as follows: processing the sixth feature map 324 using a 1×1 convolution with a sigmoid activation function to generate the second weight matrix 325. The second weight matrix 325 can be represented as...

[0109] According to embodiments of this disclosure, using the second weight matrix 325 to process the second target feature map 321 and the second reconstructed feature map 317 associated with the i-th first intermediate stage can be understood as using a gate mechanism to control the information flow from the second target feature map 321 and the second reconstructed feature map 317 associated with the i-th first intermediate stage, respectively.

[0110] According to embodiments of this disclosure, the second weight matrix 325 can be divided into a third gate vector 327 and a fourth gate vector 328. The third gate vector 327 can be represented as... The fourth gate vector 328 can be represented as

[0111] According to embodiments of this disclosure, the calculation process of the first reconstructed feature map 326 associated with the i-th first intermediate stage can be as shown in formula (4):

[0112]

[0113] In the formula, This can be represented as repeating the third gate vector 327 d times along the second dimension; This is represented as repeating the fourth gate vector 328 d times along the second dimension; · represents the dot product of the matrices.

[0114] According to embodiments of this disclosure, when i equals 1, using a bottom-up knowledge reconstruction network to process the second target feature map 321 and the second reconstructed feature map 317 associated with the i-th first intermediate stage may include the following operations:

[0115] The second reconstructed feature map 317 associated with the i-th first intermediate stage is processed using the sixth convolutional network to obtain the first reconstructed feature map 326 associated with the i-th intermediate stage.

[0116] According to embodiments of this disclosure, the sixth convolutional network can be a 1×1 convolutional layer. The sixth convolutional network can be used to change the dimension of the input feature map.

[0117] According to embodiments of this disclosure, processing the second reconstructed feature map 317 associated with the i-th first intermediate stage using a sixth convolutional network can be understood as: using a 1×1 convolutional layer to make the second reconstructed feature map 317 match the number of features of the feature map output by the teacher network.

[0118] According to embodiments of this disclosure, a method for processing input features by utilizing a knowledge reconstruction network can dynamically generate a weight matrix based on the input features, and then use the weight matrix to process the input features, thereby adaptively fusing information flows from different levels that carry rich knowledge.

[0119] Figure 4 A flowchart illustrating a point cloud data processing model training method according to another embodiment of the present disclosure is shown.

[0120] like Figure 4 As shown, the method includes operations S401 to S408.

[0121] In operation S401, the sample point cloud data is input into the student network, and the first feature map related to the first intermediate stage of the student network is output.

[0122] In operation S402, the sample point cloud data is input into the teacher network, and the second feature map related to the second intermediate stage of the teacher network is output. The first intermediate stage and the second intermediate stage correspond one-to-one.

[0123] In operation S403, the first feature map is reconstructed using the target feature map to obtain the first reconstructed feature map. The target feature map is related to the target intermediate stage of the student network, and the target intermediate stage is adjacent to the first intermediate stage.

[0124] In operation S404, after inputting the sample point cloud data into the student network, a first location information set related to the first intermediate stage is determined. The first location information set includes the first location information of multiple points input into the first intermediate stage.

[0125] In operation S405, after inputting the sample point cloud data into the teacher network, a second location information set related to the second intermediate stage is determined. This second location information set includes the second location information of multiple points input into the second intermediate stage.

[0126] In operation S406, the first reconstructed feature map is calibrated using the first location information set and the second location information set to obtain a calibrated feature map.

[0127] In operation S407, a first loss value is determined based on the calibration feature map and the second feature map.

[0128] In operating S408, the first loss value is used to adjust the model parameters of the student network to obtain the target point cloud data processing model.

[0129] According to the embodiments of this disclosure, the method of operating S401 to S403 can be implemented according to the method of operating S201 to S203, and will not be described again here.

[0130] According to embodiments of this disclosure, after point cloud data is input into the neural network, before it is input into the network structure of each intermediate stage, a farthest point sampling operation can be performed on the point cloud data, that is, a portion of the point cloud data can be randomly selected from the point cloud data and input into the network structure. For the same sample of point cloud data, the point cloud data input into the same intermediate stage network structure each time can be different.

[0131] According to embodiments of this disclosure, the first location information may refer to the three-dimensional coordinates of the point input to the first intermediate stage. The first location information set may be the set of three-dimensional coordinates of all points input to the first intermediate stage. For the i-th first intermediate stage, the first location information set may be represented as P. s ={p s，1 p s，2 , ..., p s，M}, where M represents the number of points in the first intermediate stage, and correspondingly, the first reconstructed feature map associated with the i-th first intermediate stage can be represented as

[0132] According to embodiments of this disclosure, the second location information may refer to the three-dimensional coordinates of the point input to the second intermediate stage. The second location information set may be the set of three-dimensional coordinates of all points input to the second intermediate stage. For the i-th second intermediate stage, the second location information set may be represented as P. t ={p t，1 p t，2 , ..., p t，L}, where L represents the number of points in the first intermediate stage.

[0133] According to embodiments of this disclosure, operation S406 may include the following operations:

[0134] For each second location information in the second location information set, multiple target location information is determined from multiple first location information in the first location information set; a third weight matrix is ​​determined based on the second location information and the multiple target location information; and a calibration feature map is obtained based on the third weight matrix and the first reconstructed feature map.

[0135] According to the embodiments of this disclosure, determining multiple target location information from multiple first location information in a first location information set can be understood as: determining k neighboring points in the first location information set that are close to the point represented by the second location information based on relative distance, and the first location information of the k neighboring points is the multiple target location information.

[0136] According to embodiments of this disclosure, the third weight matrix can be calculated using a normalized Gaussian radial basis function, as shown in formula (5):

[0137]

[0138] In the formula, Represents the first location information set P S Information relative to the second position The index of the k neighboring points of the represented point; τ represents the temperature coefficient, which can be a hyperparameter; d j The distance between the point represented by the first location information and the point represented by the second location information is shown in formula (6):

[0139]

[0140] According to embodiments of this disclosure, the method for calculating the calibration feature map based on the third weight matrix and the first reconstructed feature map can be as shown in formula (7):

[0141]

[0142] In the formula, This can represent the calibration feature corresponding to the nth point in the second location information set. The calibration feature map associated with the i-th second intermediate stage can be represented as follows:

[0143] Figure 5 A schematic diagram of a sampling point calibration method according to an embodiment of the present disclosure is shown.

[0144] like Figure 5 As shown, by calibrating the sampling points of the teacher network and the student network, the neighborhood features in the student point set can be fitted to the features in the teacher point set based on distance weighting, so that the local representation regions of the student features and the teacher features are consistent, thereby effectively improving the performance of the student network.

[0145] Figure 6 The diagram illustrates a point cloud data processing model training method according to an embodiment of the present disclosure.

[0146] like Figure 6 As shown, the sample point cloud data 601 is input into the student network, and each first intermediate stage of the student network can output a first feature map 602. Taking the first feature map related to the i-th first intermediate stage as an example, the first feature map 602 related to the i-th first intermediate stage can be represented as F s，i .

[0147] According to embodiments of this disclosure, the first feature map 602 can be input into a top-down knowledge reconstruction network 603, so that the first feature map 602 can be reconstructed in the top-down knowledge reconstruction network 603 using a second reconstruction feature map 604 associated with the (i+1)th first intermediate stage, to obtain a second reconstruction feature map 604 associated with the ith first intermediate stage. The top-down knowledge reconstruction network 603 can be represented as TDKR, and the second reconstruction feature map 604 associated with the ith first intermediate stage can be represented as TDKR. i .

[0148] According to embodiments of this disclosure, the second reconstruction feature map 604 associated with the i-th first intermediate stage can be input into a bottom-up knowledge reconstruction network 605. In this bottom-up knowledge reconstruction network 605, the second reconstruction feature map 604 is reconstructed using the first reconstruction feature map 606 associated with the (i-1)-th first intermediate stage to obtain the first reconstruction feature map 606 associated with the i-th first intermediate stage. The bottom-up knowledge reconstruction network 605 can be represented as BUKR, and the first reconstruction feature map 606 associated with the i-th first intermediate stage can be represented as BUKR. i .

[0149] According to an embodiment of this disclosure, the first reconstructed feature map 606 associated with the i-th first intermediate stage can be input into a sampling point network 607 to perform sampling point calibration on the first reconstructed feature map 606 in the sampling point network 607 to obtain a calibration feature map 608 associated with the i-th first intermediate stage.

[0150] According to embodiments of this disclosure, sample point cloud data 601 is input into a teacher network, and each second intermediate stage of the teacher network can output a second feature map 609. The second feature map 609 associated with the i-th second intermediate stage can be represented as F t，i .

[0151] According to embodiments of this disclosure, a first loss value 610 can be determined based on the calibration feature map 608 and the second feature map 609 corresponding to the intermediate stage.

[0152] According to embodiments of this disclosure, the sample point cloud data 601 is input into the student network, and the output features of the student network can also be output. Then, the cross-entropy loss between the output features and the labels of the sample point cloud data 601 is calculated to determine the second loss value 611.

[0153] According to embodiments of this disclosure, after determining the first loss value 401 and the second loss value 402, a total loss value for adjusting the model parameters of the student network can be determined, and the model parameters of the student network can be adjusted using the total loss value. The method for calculating the total loss value can be as shown in formula (8):

[0154] L=λL1+L2 (8)

[0155] In the formula, L represents the total loss value, L1 represents the first loss value, L2 represents the second loss value, and λ represents the weight parameter, which can be a hyperparameter.

[0156] According to embodiments of this disclosure, point cloud data processing tasks may include point cloud classification tasks and point cloud segmentation tasks.

[0157] According to embodiments of this disclosure, in a point cloud classification task, the comparison results between the target point cloud data processing model trained based on embodiments of this disclosure and neural networks trained based on methods in related technologies, such as SP, FitNet, NST, AT, KD, F-L2, etc., are shown in Table 1. In Table 1, OA (Overall accuracy) represents the ratio of correctly classified samples to the total number of samples, and mAcc (Mean accuracy) represents the average accuracy of each class. The larger OA and mAcc are, the better the model performance. The image data used for comparison comes from the ModelNet40 dataset. The models used include Pointnet++ (PN++), DGCNN (DGC), PointConv (Pconv), PointTransformer (PT), etc. The prefix of the model can indicate the width compression ratio of the model.

[0158] Table 1

[0159]

[0160] According to embodiments of this disclosure, in point cloud segmentation tasks, the comparison results between the target point cloud data processing model trained based on embodiments of this disclosure and neural networks trained based on methods in related technologies, such as SP, FitNet, NST, AT, KD, F-L2, etc., are shown in Table 2. In Table 2, ins.mIoU (Instance mIoU, average intersection-union ratio) can represent the ratio of the intersection and union of the two sets of true values ​​and predicted values, and cat.mIoU (Category mIoU, average intersection-union ratio of each class) can represent the average intersection-union ratio of each class. The larger ins.mIoU and cat.mIoU are, the better the model performance. The image data used for comparison comes from the ShapeNetPart dataset. The models used include Pointnet++ (PN++), DGCNN (DGC), PointConv (Pconv), PointTransformer (PT), etc. The prefix of the model can indicate the width compression ratio of the model.

[0161] Table 2

[0162]

[0163] Based on Tables 1 and 2, it can be determined that using bidirectional knowledge reconstruction and query-based sampling point calibration can enable the student network to learn the rich shape and structure information in the teacher network, effectively improving the performance of the student network.

[0164] Figure 7 A flowchart illustrating a point cloud data processing method according to an embodiment of the present disclosure is shown.

[0165] like Figure 7 As shown, the method includes operation S701.

[0166] When operating S701, the point cloud data to be processed is input into the target point cloud data processing model, and the point cloud data processing result is output.

[0167] According to embodiments of this disclosure, the target point cloud data processing model can be trained using the methods described in the point cloud data processing model training method section, which will be further described in detail here.

[0168] Figure 8 A block diagram of a point cloud data processing model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0169] like Figure 8 As shown, the point cloud data processing model training device 800 includes a first input module 810, a second input module 820, a first processing module 830, a first determination module 840, and a training module 850.

[0170] The first input module 810 is used to input sample point cloud data into the student network and output a first feature map related to the first intermediate stage of the student network.

[0171] The second input module 820 is used to input sample point cloud data into the teacher network and output a second feature map related to the second intermediate stage of the teacher network, wherein the first intermediate stage and the second intermediate stage correspond one-to-one.

[0172] The first processing module 830 is used to reconstruct knowledge from the first feature map using the target feature map to obtain the first reconstructed feature map, wherein the target feature map is related to the target intermediate stage of the student network, and the target intermediate stage is adjacent to the first intermediate stage.

[0173] The first determining module 840 is used to determine a first loss value based on the first reconstructed feature map and the second feature map.

[0174] Training module 850 is used to adjust the model parameters of the student network using the first loss value to obtain the target point cloud data processing model.

[0175] According to embodiments of this disclosure, during the process of using a teacher network to guide the training of a student network, the semantic information of the first feature maps related to different first intermediate stages of the student network can be integrated through a knowledge reconstruction method. Then, the second feature map output by the teacher network is used to perform feature distillation on the first reconstructed feature map obtained through knowledge reconstruction, thus realizing cross-layer guidance of the student network by the teacher network. Through the above technical means, a narrow and shallow student network can be used to replace a wide and deep teacher network in point cloud data processing scenarios. Therefore, it at least partially overcomes the technical problem in related technologies where the computational overhead of neural networks used to process point cloud data is large, making it difficult to directly apply to terminal devices with limited computing resources. Applying the target point cloud data processing model obtained by training the student network can effectively reduce the computational overhead of point cloud analysis tasks. Simultaneously, the target point cloud data processing model trained through the above technical means can learn multi-level and multi-scale contextual knowledge from the teacher network, effectively improving the model's processing accuracy.

[0176] According to embodiments of this disclosure, the target feature map includes a first target feature map and a second target feature map.

[0177] According to embodiments of the present disclosure, the first processing module 830 includes a first processing unit and a second processing unit.

[0178] The first processing unit is configured to, for the i-th first intermediate stage, use the first target feature map to perform knowledge reconstruction on the first feature map related to the i-th first intermediate stage, and obtain the second reconstructed feature map related to the i-th first intermediate stage, wherein the first target feature map is represented as the second reconstructed feature map related to the (i+1)-th first intermediate stage.

[0179] The second processing unit is used to perform knowledge reconstruction on the second reconstruction feature map related to the i-th first intermediate stage using the second target feature map, so as to obtain the first reconstruction feature map related to the i-th first intermediate stage, wherein the second target feature map is represented as the first reconstruction feature map related to the (i-1)-th first intermediate stage.

[0180] According to embodiments of this disclosure, i is a positive integer less than or equal to N, where N represents the number of first intermediate stages in the student network.

[0181] According to embodiments of the present disclosure, the first processing unit includes a first processing subunit, a second processing subunit, a third processing subunit, a fourth processing subunit, and a fifth processing subunit.

[0182] The first processing subunit is used to perform upsampling processing on the first target feature map for the i-th first intermediate stage, when i is less than N, to obtain the first sampled feature map.

[0183] The second processing subunit is used to process the first feature map related to the i-th first intermediate stage using the first convolutional network to obtain the third feature map.

[0184] The third processing subunit is used to connect the first sampled feature map and the third feature map to obtain the fourth feature map.

[0185] The fourth processing subunit is used to process the fourth feature map using the second convolutional network to obtain the first weight matrix.

[0186] The fifth processing subunit is used to determine the second reconstructed feature map related to the i-th first intermediate stage based on the first weight matrix, the first target feature map, and the first feature map related to the i-th first intermediate stage.

[0187] According to embodiments of this disclosure, the first processing unit further includes a sixth processing subunit.

[0188] The sixth processing subunit is used to process the first feature map related to the i-th first intermediate stage using the third convolutional network when i equals N, to obtain the second reconstructed feature map related to the i-th first intermediate stage.

[0189] According to embodiments of this disclosure, the second processing unit includes a seventh processing subunit, an eighth processing subunit, a ninth processing subunit, a tenth processing subunit, and an eleventh processing subunit.

[0190] The seventh processing subunit is used to perform downsampling processing on the second target feature map when i is greater than 1, so as to obtain the second sampled feature map.

[0191] The eighth processing subunit is used to process the second reconstructed feature map related to the i-th first intermediate stage using the fourth convolutional network to obtain the fifth feature map.

[0192] The ninth processing subunit is used to connect the second sampled feature map and the fifth feature map to obtain the sixth feature map.

[0193] The tenth processing subunit is used to process the sixth feature map using the fifth convolutional network to obtain the second weight matrix.

[0194] The eleventh processing subunit is used to determine the first reconstructed feature map related to the i-th first intermediate stage based on the second weight matrix, the second target feature map, and the second reconstructed feature map related to the i-th first intermediate stage.

[0195] According to embodiments of this disclosure, the second processing unit further includes a twelfth processing subunit.

[0196] The twelfth processing subunit is used to process the second reconstructed feature map related to the i-th first intermediate stage using the sixth convolutional network when i equals 1, so as to obtain the first reconstructed feature map related to the i-th intermediate stage.

[0197] According to embodiments of this disclosure, the point cloud data processing model training device 800 further includes a second determination module, a third determination module, and a calibration module.

[0198] The second determining module is used to determine the first location information set related to the first intermediate stage after the sample point cloud data is input into the student network, wherein the first location information set includes the first location information of multiple points input into the first intermediate stage.

[0199] The third determining module is used to determine the second location information set related to the second intermediate stage after the sample point cloud data is input into the teacher network. The second location information set includes the second location information of multiple points input into the second intermediate stage.

[0200] The calibration module is used to perform position calibration on the first reconstructed feature map using the first location information set and the second location information set to obtain a calibration feature map.

[0201] According to an embodiment of this disclosure, the first determining module 840 includes a determining unit.

[0202] A determination unit is used to determine a first loss value based on a calibration feature map and a second feature map.

[0203] According to embodiments of this disclosure, the calibration module includes a first calibration unit, a second calibration unit, and a third calibration unit.

[0204] The first calibration unit is used to determine multiple target location information from multiple first location information in the first location information set for each second location information in the second location information set.

[0205] The second calibration unit is used to determine the third weight matrix based on the second location information and multiple target location information.

[0206] The third calibration unit is used to obtain a calibration feature map based on the third weight matrix and the first reconstructed feature map.

[0207] According to embodiments of this disclosure, the point cloud data processing model training device 800 further includes a third input module and a calculation module.

[0208] The third input module is used to input sample point cloud data into the student network and output the output features of the student network.

[0209] The calculation module is used to calculate the cross-entropy loss between the output features and the labels of the sample point cloud data to obtain the second loss value.

[0210] According to embodiments of this disclosure, training module 850 includes training units.

[0211] The training unit is used to adjust the model parameters of the student network using the first and second loss values ​​to obtain the target point cloud data processing model.

[0212] Figure 9 A block diagram of a point cloud data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0213] like Figure 9 As shown, the point cloud data processing device 900 includes a second processing module 910.

[0214] The second processing module 910 is used to input the point cloud data to be processed into the target point cloud data processing model and output the point cloud data processing result.

[0215] According to embodiments of this disclosure, the target point cloud data processing model can be trained using the methods described in the point cloud data processing model training method section.

[0216] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0217] For example, any of the first input module 810, the second input module 820, the first processing module 830, the first determining module 840, and the training module 850, or the second processing module 910, can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first input module 810, the second input module 820, the first processing module 830, the first determining module 840, and the training module 850, or the second processing module 910, can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first input module 810, the second input module 820, the first processing module 830, the first determining module 840, and the training module 850, or the second processing module 910, can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0218] It should be noted that the point cloud data processing model training device part in the embodiments of this disclosure corresponds to the point cloud data processing model training method part in the embodiments of this disclosure. The description of the point cloud data processing model training device part, as well as the training method of the target point cloud data processing model in the point cloud data processing device, are specifically referred to in the point cloud data processing model training method part, and will not be repeated here.

[0219] Figure 10 A block diagram of an electronic device suitable for implementing a point cloud data processing model training method according to an embodiment of the present disclosure is shown schematically. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0220] like Figure 10As shown, a computer electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0221] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0222] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0223] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0224] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0225] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0226] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0227] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the point cloud data processing model training method provided in the embodiments of this disclosure.

[0228] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0229] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0230] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0232] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for training a point cloud data processing model, comprising: Input sample point cloud data into the student network and output a first feature map related to the first intermediate stage of the student network; The sample point cloud data is input into the teacher network, and a second feature map related to the second intermediate stage of the teacher network is output, wherein the first intermediate stage and the second intermediate stage correspond one-to-one. The method of knowledge reconstruction of the first feature map using the target feature map to obtain the first reconstructed feature map includes: using the target feature map, processing the first feature map through a top-down knowledge reconstruction network to obtain the second reconstructed feature map, and then processing the second reconstructed feature map through a bottom-up knowledge reconstruction network to obtain the first reconstructed feature map. The target feature map is related to the target intermediate stage of the student network, the target intermediate stage is adjacent to the first intermediate stage, and the first reconstructed feature map is a feature map obtained by adaptively fusing the semantic information of the first feature map and the feature map of the first intermediate stage adjacent to the student network. Its dimension and feature number match the second feature map output by the teacher network corresponding to the second intermediate stage. Based on the first reconstructed feature map and the second feature map, a first loss value is determined; and The model parameters of the student network are adjusted using the first loss value to obtain the target point cloud data processing model.

2. The method according to claim 1, wherein, The target feature map includes a first target feature map and a second target feature map; The step of reconstructing the first feature map using the target feature map to obtain the first reconstructed feature map includes: For the i-th first intermediate stage, the first feature map related to the i-th first intermediate stage is reconstructed using the first target feature map to obtain a second reconstructed feature map related to the i-th first intermediate stage, wherein the first target feature map is represented as the second reconstructed feature map related to the (i+1)-th first intermediate stage; and The second target feature map is used to reconstruct the knowledge of the second reconstruction feature map related to the i-th first intermediate stage to obtain the first reconstruction feature map related to the i-th first intermediate stage, wherein the second target feature map is represented as the first reconstruction feature map related to the (i-1)-th first intermediate stage; Where i is a positive integer less than or equal to N, and N represents the number of the first intermediate stages of the student network.

3. The method according to claim 2, wherein, For the i-th first intermediate stage, the first feature map related to the i-th first intermediate stage is reconstructed using the first target feature map to obtain a second reconstructed feature map related to the i-th first intermediate stage, including: For the i-th first intermediate stage, if i is less than N, the first target feature map is upsampled to obtain a first sampled feature map; The first feature map associated with the i-th first intermediate stage is processed using the first convolutional network to obtain the third feature map; The first sampled feature map and the third feature map are concatenated to obtain the fourth feature map; The fourth feature map is processed using a second convolutional network to obtain a first weight matrix; and Based on the first weight matrix, the first target feature map, and the first feature map associated with the i-th first intermediate stage, a second reconstructed feature map associated with the i-th first intermediate stage is determined.

4. The method according to claim 3, further comprising: When i equals N, the first feature map associated with the i-th first intermediate stage is processed by the third convolutional network to obtain the second reconstructed feature map associated with the i-th first intermediate stage.

5. The method according to claim 2, wherein, The step of using the second target feature map to perform knowledge reconstruction on the second reconstructed feature map related to the i-th first intermediate stage, to obtain the first reconstructed feature map related to the i-th first intermediate stage, includes: When i is greater than 1, the second target feature map is downsampled to obtain the second sampled feature map; The fourth convolutional network is used to process the second reconstructed feature map related to the i-th first intermediate stage to obtain the fifth feature map; The second sampled feature map and the fifth feature map are connected to obtain the sixth feature map; The sixth feature map is processed using the fifth convolutional network to obtain the second weight matrix; and Based on the second weight matrix, the second target feature map, and the second reconstructed feature map associated with the i-th first intermediate stage, the first reconstructed feature map associated with the i-th first intermediate stage is determined.

6. The method according to claim 5, further comprising: When i equals 1, the second reconstructed feature map associated with the i-th first intermediate stage is processed using the sixth convolutional network to obtain the first reconstructed feature map associated with the i-th intermediate stage.

7. The method according to claim 1, further comprising: After the sample point cloud data is input into the student network, a first location information set related to the first intermediate stage is determined, wherein the first location information set includes the first location information of multiple points input into the first intermediate stage; After the sample point cloud data is input into the teacher network, a second set of location information related to the second intermediate stage is determined, wherein the second set of location information includes the second location information of multiple points input into the second intermediate stage; and The first reconstructed feature map is calibrated using the first location information set and the second location information set to obtain a calibrated feature map. The step of determining the first loss value based on the first reconstructed feature map and the second feature map includes: The first loss value is determined based on the calibration feature map and the second feature map.

8. The method according to claim 7, wherein, The step of using the first location information set and the second location information set to perform position calibration on the first reconstructed feature map to obtain a calibrated feature map includes: For each second location information in the second location information set, multiple target location information are determined from multiple first location information in the first location information set; Based on the second location information and the multiple target location information, a third weight matrix is ​​determined; and The calibration feature map is obtained based on the third weight matrix and the first reconstructed feature map.

9. The method according to claim 1, further comprising: The sample point cloud data is input into the student network, and the output features of the student network are output. as well as Calculate the cross-entropy loss between the output features and the labels of the sample point cloud data to obtain a second loss value; The step of adjusting the model parameters of the student network using the first loss value to obtain the target point cloud data processing model includes: The model parameters of the student network are adjusted using the first loss value and the second loss value to obtain the target point cloud data processing model.

10. A point cloud data processing method, comprising: Input the point cloud data to be processed into the target point cloud data processing model, and output the point cloud data processing result; The target point cloud data processing model is obtained by training the point cloud data processing model training method according to any one of claims 1 to 9.

11. A point cloud data processing model training device, comprising: The first input module is used to input sample point cloud data into the student network and output a first feature map related to the first intermediate stage of the student network. The second input module is used to input the sample point cloud data into the teacher network and output a second feature map related to the second intermediate stage of the teacher network, wherein the first intermediate stage and the second intermediate stage correspond one-to-one. The first processing module is used to reconstruct knowledge from the first feature map using the target feature map to obtain a first reconstructed feature map. This includes: processing the first feature map using a top-down knowledge reconstruction network to obtain a second reconstructed feature map, and then processing the second reconstructed feature map using a bottom-up knowledge reconstruction network to obtain the first reconstructed feature map. The target feature map is related to the target intermediate stage of the student network, and the target intermediate stage is adjacent to the first intermediate stage. The first reconstructed feature map is obtained by adaptively fusing the semantic information of the first feature map with the feature map of the first intermediate stage adjacent to the student network. Its dimension and feature number match the second feature map output by the teacher network corresponding to the second intermediate stage. The first determining module is configured to determine a first loss value based on the first reconstructed feature map and the second feature map; and The training module is used to adjust the model parameters of the student network using the first loss value to obtain the target point cloud data processing model.

12. A point cloud data processing device, comprising: The second processing module is used to input the point cloud data to be processed into the target point cloud data processing model and output the point cloud data processing results. The target point cloud data processing model is obtained by training the point cloud data processing model training method according to any one of claims 1 to 9.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 10.

14. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 10.

15. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 10.

Images