A 3D data processing method and system based on dual network super-resolution algorithm
By constructing a 3D dual-branch network model for joint training of low-resolution 3D data, the problem of low semantic segmentation accuracy of 3D data is solved, and the segmentation accuracy and processing efficiency are improved.
Patent Information
- Application Number
- CN202310837609.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-10
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2043-07-10
AI Technical Summary
The existing 3D data semantic segmentation models have low segmentation accuracy on low-resolution 3D data and are inefficient after conversion to high-resolution post-processing.
Using the 3D data processing method based on the dual network super-resolution algorithm, a dual-branch network model including 3D reconstruction branches and 3D segmentation branches is constructed, and the two are divided by low-resolution 3D data set, high-resolution 3D data set and 3D semantic segmentation result set. Each branch is trained in joint to obtain joint loss. When the loss is less than the preset threshold, semantic segmentation is performed.
The accuracy and processing efficiency of 3D data semantic segmentation are improved, and high-precision semantic segmentation of low-resolution 3D data is realized.
Smart Images
Figure CN116883658B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a 3D data processing method and system based on a dual-network super-resolution algorithm. Background Art
[0002] With the widespread application of 3D data in various fields, 3D data processing technology is becoming increasingly important. Efficiently and accurately processing massive 3D data and mining the rich information contained in 3D data are the basis and prerequisite for realizing 3D data applications. However, the existing 3D data semantic segmentation model directly segments low-resolution 3D data. Due to the low resolution of 3D data, the segmentation results are not accurate enough. Converting low-resolution 3D data to high-resolution 3D data before semantic segmentation leads to low data processing efficiency. Summary of the invention
[0003] The present application provides a 3D data processing method and system based on a dual-network super-resolution algorithm, aiming to solve the technical problem of low semantic segmentation accuracy of 3D data in the prior art.
[0004] In view of the above problems, the present application provides a 3D data processing method and system based on a dual-network super-resolution algorithm.
[0005] In a first aspect disclosed in the present application, a 3D data processing method based on a dual-network super-resolution algorithm is provided, the method comprising: obtaining a low-resolution 3D data set, a high-resolution 3D data set and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data and the 3D semantic segmentation result correspond to each other one by one; constructing a 3D dual-branch network model, the 3D dual-branch network model comprising a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data; jointly training the 3D reconstruction branch and the 3D segmentation branch through the low-resolution 3D data set, the high-resolution 3D data set and the 3D semantic segmentation result set to obtain a joint loss; when the joint loss is less than a preset loss threshold, obtaining target low-resolution 3D data; inputting the target low-resolution 3D data into the 3D dual-branch network model, the 3D segmentation branch performing a forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result.
[0006] Another aspect disclosed in the present application provides a 3D data processing system based on a dual-network super-resolution algorithm, the system comprising: a training data acquisition module, used to acquire a low-resolution 3D data set, a high-resolution 3D data set and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data and the 3D semantic segmentation result correspond to each other; a network model construction module, used to construct a 3D dual-branch network model, the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data; a branch joint training module, used to jointly train the 3D reconstruction branch and the 3D segmentation branch through the low-resolution 3D data set, the high-resolution 3D data set and the 3D semantic segmentation result set to obtain a joint loss; a target data acquisition module, used to acquire target low-resolution 3D data when the joint loss is less than a preset loss threshold; a processing result acquisition module, used to input the target low-resolution 3D data into the 3D dual-branch network model, and the 3D segmentation branch performs a forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result.
[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0008] Due to the adoption of a 3D dual-branch network model including a 3D reconstruction branch and a 3D segmentation branch, the two branches share the model input data; wherein, the 3D reconstruction branch is used for 3D data super-resolution reconstruction, and the 3D segmentation branch is used for 3D data semantic segmentation; then, by acquiring a low-resolution 3D dataset, a high-resolution 3D dataset, and a 3D semantic segmentation dataset, the reconstruction loss of the 3D reconstruction branch, the segmentation loss of the 3D segmentation branch, and the consistency loss of the two are calculated, thereby realizing joint training of the 3D reconstruction branch and the 3D segmentation branch; under the condition of meeting preset conditions, the target low-resolution 3D data is acquired, and the 3D segmentation branch can perform semantic segmentation on it. This technical solution solves the technical problem of low accuracy of 3D data semantic segmentation in the prior art, and achieves the technical effect of improving the accuracy of 3D data semantic segmentation.
[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A possible flow chart of a 3D data processing method based on a dual network super-resolution algorithm is provided for an embodiment of the present application;
[0011] Figure 2A schematic diagram of a possible flow chart for calculating reconstruction loss in a 3D data processing method based on a dual-network super-resolution algorithm is provided for an embodiment of the present application;
[0012] Figure 3 A schematic diagram of a possible flow chart for calculating segmentation loss in a 3D data processing method based on a dual-network super-resolution algorithm is provided for an embodiment of the present application;
[0013] Figure 4 A possible structural schematic diagram of a 3D data processing system based on a dual-network super-resolution algorithm is provided for an embodiment of the present application.
[0014] Explanation of the reference numerals: training data acquisition module 11, network model construction module 12, branch joint training module 13, target data acquisition module 14, processing result acquisition module 15. DETAILED DESCRIPTION
[0015] The overall idea of the technical solution provided by this application is as follows:
[0016] The embodiment of the present application provides a 3D data processing method and system based on a dual-network super-resolution algorithm. First, a 3D dual-branch network model is constructed, which includes a 3D reconstruction branch and a 3D segmentation branch. The two branches share the model input data, wherein the 3D reconstruction branch is used for 3D data super-resolution reconstruction, and the 3D segmentation branch is used for 3D data semantic segmentation. Then, by obtaining a low-resolution 3D data set, a high-resolution 3D data set, and a 3D semantic segmentation data set, the reconstruction loss of the 3D reconstruction branch, the segmentation loss of the 3D segmentation branch, and the consistency loss of the two are calculated to achieve joint training of the 3D reconstruction branch and the 3D segmentation branch. In this process, the 3D reconstruction branch reconstructs high-resolution 3D data and provides training constraint information for the 3D segmentation branch to achieve high-precision semantic segmentation of low-resolution 3D data by the 3D segmentation branch. When the joint training meets the preset conditions, the target low-resolution 3D data is obtained and input into the 3D dual-branch network model, and the 3D segmentation branch performs high-precision semantic segmentation on the target low-resolution 3D data.
[0017] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.
[0018] Embodiment 1
[0019] like Figure 1 As shown, the embodiment of the present application provides a 3D data processing method based on a dual network super-resolution algorithm, the method comprising:
[0020] Step S100: obtaining a low-resolution 3D data set, a high-resolution 3D data set, and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data, and the 3D semantic segmentation result correspond one to one;
[0021] Specifically, low-resolution 3D data refers to a data set consisting of fewer data points or lower-precision data points in three-dimensional space; high-resolution 3D data refers to a data set consisting of more data points or higher-precision data points in three-dimensional space; 3D semantic segmentation results refer to the results obtained after semantic segmentation of 3D data, that is, each data point is assigned to the corresponding semantic category. First, access the historical 3D scanning segmentation database, traverse all low-resolution 3D data, obtain low-resolution 3D data with corresponding high-resolution 3D data and 3D semantic segmentation results, and form data strips respectively, each data strip contains low-resolution 3D data, high-resolution 3D data and 3D semantic segmentation results. Then, uniquely number each data strip, and associate the three types of data in each data strip. Next, the low-resolution 3D data in all data strips are combined into a low-resolution 3D data set, the high-resolution 3D data of all data strips are combined into a high-resolution 3D data set, and the 3D semantic segmentation results of all data strips are combined into a 3D semantic segmentation result set; at this time, extract a piece of data in any data set, and the corresponding data can be retrieved in other data sets according to the data strip code and the associated code.
[0022] Step S200: constructing a 3D dual-branch network model, wherein the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data;
[0023] Specifically, in order to achieve high-precision semantic segmentation of low-resolution 3D data, a 3D dual-branch network model is constructed, which consists of two branches, 3D reconstruction branch and 3D segmentation branch. In the model training stage, the two branches use the same input data, namely low-resolution 3D data, and in the data processing process, the reconstruction branch and the high-resolution reconstructed image it produces are not used. The reconstruction branch is only used for model training to improve the learning ability of the segmentation branch, avoid interfering with the learning process of the segmentation branch, and improve its segmentation ability for low-resolution 3D data.
[0024] The 3D reconstruction branch is implemented using a convolutional LSTM model. First, the input low-resolution 3D data is preprocessed, such as normalization, downsampling, etc., to adapt to the input requirements of the branch; next, a series of convolutional layers are used to extract the features of the low-resolution 3D data; after the convolutional layer, the LSTM layer is introduced to process the time series data. The LSTM layer can capture the long-term dependencies in the data and has memory units and gating mechanisms to effectively process time series data; after the LSTM layer, an upsampling layer is used to restore the resolution of the feature map to the original size; finally, the activation function and loss function are used to define the output layer to map the output of the network to a high-resolution 3D reconstruction result.
[0025] The 3D segmentation branch is implemented using the PointNet model. First, the input low-resolution 3D data is preprocessed, such as normalization, downsampling, etc., to adapt to the input requirements of the model; next, the PointNet model is used to extract the features of the point cloud data. PointNet processes each point independently, and then summarizes the features of each point to obtain a global feature representation; after feature extraction, fully connected layers or convolutional layers are used for segmentation prediction. These layers map global features to specific semantic segmentation results and generate 3D semantic segmentation results; finally, activation functions and loss functions are used to define the output layer to map the output of the network to the 3D semantic segmentation results.
[0026] Step S300: jointly training the 3D reconstruction branch and the 3D segmentation branch using the low-resolution 3D dataset, the high-resolution 3D dataset, and the 3D semantic segmentation result set to obtain a joint loss;
[0027] Specifically, the low-resolution 3D dataset, the high-resolution 3D dataset, and the 3D semantic segmentation result set are used for joint training, and the joint loss is obtained based on the joint loss function. The joint loss function is a loss function used to simultaneously optimize the 3D reconstruction branch and the 3D segmentation branch. The joint loss function consists of reconstruction loss, segmentation loss, and consistency loss, and the formula is L joint =L rec +L seg +L eonsis Among them, L rec represents the reconstruction loss, which is used to measure the reconstruction accuracy of the reconstruction branch; L seg represents the segmentation loss, which is used to measure the segmentation accuracy of the segmentation branch; L consis Represents the consistency loss, ensuring that the output results of the two branches are consistent with each other. Preferably, the weights for reconstruction loss, segmentation loss and consistency loss are set to δ rec , δ seg , δ consis, by adjusting the weights to balance the contributions of reconstruction loss, segmentation loss and consistency loss in the joint loss function to achieve the best joint training effect.
[0028] Step S400: when the joint loss is less than a preset loss threshold, acquiring target low-resolution 3D data;
[0029] Specifically, the joint loss is the loss value obtained through joint training, which can measure the performance of the joint training model in reconstruction and segmentation tasks. The preset loss threshold is a pre-set loss threshold. The performance of the model with a joint loss less than the threshold has reached the preset data processing requirements.
[0030] By comparing the joint loss and the preset loss threshold, we can determine whether the performance of the model in the reconstruction and segmentation tasks has reached the expected level. If the joint loss is less than the preset loss threshold, it means that the performance of the model has met the requirements and we can proceed to the next step. At this point, the model can be used for semantic segmentation tasks to obtain new low-resolution 3D data required for semantic segmentation tasks, which is the new 3D data to be processed.
[0031] Step S500: inputting the target low-resolution 3D data into the 3D dual-branch network model, and the 3D segmentation branch performs a forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result.
[0032] Specifically, the acquired target low-resolution 3D data is input into the 3D dual-branch network model, and the target low-resolution 3D data is forward operated by using the 3D segmentation branch therein to obtain the semantic segmentation result of the target.
[0033] First, the target low-resolution 3D data is preprocessed, such as normalization, cropping or padding, to ensure that the format and size of the data match the input requirements of the 3D segmentation branch. Then, the preprocessed target low-resolution 3D data is input into the 3D segmentation branch. In the 3D segmentation branch, a series of operations such as convolution, pooling and activation functions are used to extract features from the input data to help the network learn the local and global feature information of the target. In the process of feature extraction, convolution and pooling operations are used to upsample and downsample the feature map. Among them, the upsampling operation increases the resolution of the feature map to better capture the detailed information of the target; while the downsampling operation reduces the size of the feature map to extract higher-level semantic features. After the feature extraction and sampling operations are completed, the 3D segmentation branch uses operations such as convolution and fully connected layers to map the features to the corresponding semantic categories, that is, to assign each point in the feature map to a specific semantic category to achieve semantic segmentation of the target. Achieve high-precision semantic segmentation of low-resolution 3D data directly, improve the accuracy of 3D data semantic segmentation, and improve data processing efficiency.
[0034] Furthermore, the embodiment of the present application also includes:
[0035] Step S310: traversing the low-resolution 3D data set to obtain first low-resolution 3D data, and obtaining first high-resolution 3D data and first 3D semantic segmentation results in the high-resolution 3D data set and the 3D semantic segmentation result set respectively based on the first low-resolution 3D data;
[0036] Step S320: inputting the first low-resolution 3D data into the 3D reconstruction branch, obtaining first reconstructed 3D data, and calculating the reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data;
[0037] Step S330: inputting the first low-resolution 3D data into the 3D segmentation branch, obtaining first segmented 3D data, and calculating the segmentation loss between the first segmented 3D data and the first 3D semantic segmentation result;
[0038] Step S340: calculating the consistency loss between the first reconstructed 3D data and the first segmented 3D data;
[0039] Step S350: Add the reconstruction loss, the segmentation loss, and the consistency loss as the joint loss.
[0040] Specifically, by traversing the low-resolution 3D dataset, the low-resolution 3D data is obtained one by one. Then, according to the serial number of the low-resolution 3D data, the corresponding high-resolution 3D data and 3D semantic segmentation results are obtained from the high-resolution 3D dataset and the 3D semantic segmentation result set, respectively, to provide the required data basis for subsequent reconstruction and segmentation tasks.
[0041] The low-resolution 3D data is input into the 3D dual network model, which inputs the low-resolution 3D data into the 3D reconstruction branch and the 3D segmentation branch at the same time. The 3D reconstruction branch performs a series of convolution, deconvolution and activation functions on the input data to generate reconstructed 3D data, and then calculates the difference between the reconstructed 3D data and the high-resolution 3D data corresponding to the low-resolution 3D data through the mean square error (MSE) or structural similarity index (SSIM) to obtain the reconstruction loss. The 3D segmentation branch performs feature extraction and classification operations on the input data to generate segmented 3D data, and then calculates the difference between the segmented 3D data and the 3D semantic segmentation results corresponding to the low-resolution 3D data through the cross entropy loss function or the Dice loss function to obtain the segmentation loss.
[0042] Next, the consistency loss between the reconstructed 3D data obtained by the 3D reconstruction branch and the reconstructed 3D data obtained by the 3D segmentation branch of the low-resolution 3D data is calculated to evaluate the consistency between the reconstruction and segmentation tasks, providing guidance for subsequent optimization and improvement. After that, the reconstruction loss, segmentation loss, and consistency loss are normalized, and the three losses are added together to obtain the joint loss to evaluate the overall performance of the model. Minimization forces the network model to strike a balance between the reconstruction and segmentation tasks and maintain consistency, optimizing the 3D segmentation branch's ability to perform high-precision semantic segmentation on low-resolution 3D data.
[0043] Further, such as Figure 2 As shown, the embodiment of the present application also includes:
[0044] Step S321: the 3D reconstruction branch includes a first encoder and a first decoder;
[0045] Step S322: performing point cloud encoding on the first low-resolution 3D data based on the first encoder to obtain a point cloud feature vector;
[0046] Step S323: decoding the point cloud feature vector based on the first decoder to generate the first reconstructed 3D data;
[0047] Step S324: calculating the reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data by using a mean square error.
[0048] Specifically, the first encoder is a component in the 3D reconstruction branch, responsible for encoding the input low-resolution 3D data into point clouds. The encoding process uses PointNet++ to encode the point cloud, extract the global semantic information and spatial structure information of the point cloud, and output a low-dimensional and high-quality point cloud feature vector. The first decoder is another component in the 3D reconstruction branch, responsible for decoding the point cloud feature vector. The decoding process uses the FoldingNet algorithm to decode, predict the adjacent point relationship of the new point, and use the point cloud feature vector to decode the complete high-precision point cloud data. The reconstruction of the low-resolution 3D data can be achieved by inputting the low-resolution 3D data into the first encoder for encoding, and then inputting the encoded point cloud feature vector into the first decoder for decoding. The first reconstructed 3D data is the result obtained in the decoding process. It is the complete high-precision point cloud data decoded according to the point cloud feature vector. The reconstructed 3D data has a similar structure and shape to the input low-resolution 3D data.
[0049] Then, the first reconstructed 3D data and the first high-resolution 3D data are represented as corresponding matrices, assuming that they are R and H respectively. The difference between the two data is calculated by element-by-element subtraction to obtain a difference matrix D, each element of the difference matrix D is squared to obtain a square difference matrix SD, and the average value of the square difference matrix SD is calculated, that is, the mean square error Where n is the total number of elements in the matrix, and ∑SD represents the sum of all elements. The obtained MSE is the reconstruction loss L between the first reconstructed 3D data and the first high-resolution 3D data. rec , the smaller the value, the closer the reconstructed data is to the high-resolution data.
[0050] Further, such as Figure 3 As shown, the embodiment of the present application also includes:
[0051] Step S331: the 3D segmentation branch includes a second encoder and a second decoder;
[0052] Step S332: extracting features from the first low-resolution 3D data based on the second encoder to obtain a semantic feature vector;
[0053] Step S333: decoding the semantic feature vector based on the second decoder to generate the first segmented 3D data;
[0054] Step S334: calculating the segmentation loss of the first segmented 3D data and the first 3D semantic segmentation result by cross entropy loss.
[0055] Specifically, the second encoder is a component in the 3D segmentation branch, responsible for feature extraction of the first low-resolution 3D data. The encoding process uses U-Net to extract features, and gradually extracts the semantic features of the input data through multiple convolutional layers and pooling layers, and encodes it into a low-dimensional semantic feature vector. The second decoder is another component in the 3D segmentation branch, responsible for decoding the semantic feature vector and generating the first segmented 3D data. The decoding process uses FCN to decode, and decodes the semantic feature vector into a semantic segmentation result of the same dimension as the input data through deconvolution layers and upsampling operations, retaining the spatial information of the input data and generating pixel-level segmentation results.
[0056] The first segmented 3D data is 3D data with the same dimension as the input data, and each dimension represents the semantic category of a data point. Assume that the first segmented 3D data is P, and its dimension is (N, H, W, C), where N represents the number of data points, H and W represent the height and width of the data, and C represents the number of semantic categories. The first 3D semantic segmentation result is a true semantic segmentation result, which is an audited and accurate semantic segmentation result. Assume that the first 3D semantic segmentation result is Q, which has the same dimension as the first segmented 3D data. Use the softmax function to convert the numerical values of each dimension of the first segmented 3D data P and the first 3D semantic segmentation result Q into probability values to obtain a probability distribution form. For each dimension of each data point, calculate the cross entropy loss between their probability values in P and Q. Among them, the cross entropy loss L(Q, P) represents the difference between the first segmented 3D data and the first 3D semantic segmentation result, n represents the number of data points, P(i) represents the predicted probability value of the i-th data point in the first segmented 3D data, and Q(i) represents the true probability value of the i-th data point in the first 3D semantic segmentation result. By calculating the cross entropy between the true probability value Q(i) and the predicted probability value P(i) of each data point, and then summing the cross entropy losses of all data points, the final summation result is the segmentation loss L seg , indicating the degree of difference between the first segmented 3D data and the first 3D semantic segmentation result.
[0057] Furthermore, the embodiment of the present application also includes:
[0058] Step S341: construct a consistency loss function:
[0059]
[0060] Among them, L consisis the consistency loss value, N is the total number of points in the first reconstructed 3D data, label(rec, i) is the semantic label of the i-th point in the first reconstructed 3D data, and label(seg, i) is the semantic label of the i-th point in the first segmented 3D data;
[0061] Step S342: traverse each point in the first reconstructed 3D data to obtain a first reconstructed 3D point;
[0062] Step S343: Acquire the first spatial coordinates and the first semantic label of the first reconstructed 3D point;
[0063] Step S344: acquiring a second semantic label from the first segmented 3D data according to the first spatial coordinates;
[0064] Step S345: Calculate the consistency between the first semantic label and the second semantic label according to the consistency loss function to obtain the consistency loss.
[0065] Specifically, first, a consistency loss function is constructed to judge the consistency of reconstructed 3D data and segmented 3D data. It is used to measure the semantic consistency between the first reconstructed 3D data and the first segmented 3D data. Where N represents the total number of points in the first reconstructed 3D data, label(rec, i) is the semantic label of the i-th point in the first reconstructed 3D data, and label(seg, i) is the semantic label of the i-th point in the first segmented 3D data;
[0066] Then, the first reconstructed 3D data is a list containing N points, each point containing spatial coordinates and semantic labels. Traverse each point in the first reconstructed 3D data, for i from 0 to N-1, perform a loop traversal, and for each point, access the element in the first reconstructed 3D data list by index, that is, the i-th point. For the i-th point, access the element in the first reconstructed 3D data list to obtain its spatial coordinates and semantic labels. For example, according to different programming methods, if the first reconstructed 3D point is an object, use the properties or methods of the point object to obtain the spatial coordinates and semantic labels; if the first reconstructed 3D point is a tuple or array containing spatial coordinates and semantic labels, use the index to obtain the corresponding value.
[0067] Next, the first segmented 3D data is also a list of N points, each point containing spatial coordinates and a semantic label. Traverse each point in the first segmented 3D data, compare its spatial coordinates with the spatial coordinates of the i-th point in the first reconstructed 3D data, and if a point with matching spatial coordinates is found, obtain the semantic label of the point as the second semantic label. Subsequently, for each point, calculate the absolute value of the difference between the first semantic label and the second semantic label |label(rec, i)-label(seg, i)|. If the two labels are the same, the absolute value of the difference is 0, and if the two labels are different, the absolute value of the difference is 1. Accumulate and sum the absolute values of the differences of all points to obtain the total sum of the absolute values of the differences, and divide the total sum of the absolute values of the differences by the total number of points N to obtain the consistency loss value, which represents the degree of difference between the first semantic label and the second semantic label. The smaller the value, the more semantically consistent the two outputs are.
[0068] Furthermore, the embodiment of the present application also includes:
[0069] Step S610: when the joint loss is greater than or equal to the preset loss threshold, obtaining the number of training times of the current network parameters;
[0070] Step S620: If the number of training times is less than the iteration threshold, continue to perform joint training on the 3D reconstruction branch and the 3D segmentation branch according to the current network parameters;
[0071] Step S630: If the number of training times is greater than or equal to the iteration threshold, the current network parameters are adjusted, and the 3D reconstruction branch and the 3D segmentation branch are continuously jointly trained according to the adjusted network parameters.
[0072] Specifically, first, based on the comparison between the joint loss and the preset loss threshold, it is determined whether the number of training times of the current network parameters needs to be recorded. The preset loss threshold is a threshold set in advance and is used to determine whether the joint loss meets the requirements. If the joint loss is greater than or equal to the preset loss threshold, the number of training times of the current network parameters is obtained.
[0073] Then, determine whether the number of training times is less than the iteration threshold. The iteration threshold is a pre-set threshold used to control the number of training iterations. If the number of training times is less than the iteration threshold, it means that it is still within the range of iterative training. At this time, according to the current network parameters, that is, the parameters that have been trained, continue to jointly train the 3D reconstruction branch and the 3D segmentation branch to further optimize the performance of the model. If the number of training times is greater than or equal to the iteration threshold, it means that the preset number of iterations has been reached. In this case, adjust the current network parameters to use the new parameters to optimize the performance of the model. According to the adjusted network parameters, the 3D reconstruction branch and the 3D segmentation branch are jointly trained to further improve the accuracy and consistency of reconstruction and segmentation.
[0074] Furthermore, the embodiment of the present application also includes:
[0075] Step S631: obtaining a first loss threshold and a second loss threshold, wherein the first loss threshold is a loss judgment threshold of the 3D reconstruction branch, and the second loss threshold is a loss judgment threshold of the 3D segmentation branch;
[0076] Step S632: If the reconstruction loss is greater than or equal to the first loss threshold, calculate a first loss deviation;
[0077] Step S633: if the reconstruction loss is less than the first loss threshold, the first loss deviation is 0;
[0078] Step S634: If the segmentation loss is greater than or equal to the second loss threshold, calculate a second loss deviation;
[0079] Step S635: if the segmentation loss is less than the second loss threshold, the second loss deviation is 0;
[0080] Step S636: Compare the first loss deviation with the second loss deviation to obtain a maximum loss deviation, and adjust the network parameters of the corresponding branch according to the maximum loss deviation.
[0081] Specifically, first, obtain the first loss threshold and the second loss threshold. The first loss threshold is a threshold used to judge the loss of the 3D reconstruction branch, which is used to measure the accuracy of the reconstruction result; the second loss threshold is a threshold used to judge the loss of the 3D segmentation branch, which is used to measure the accuracy of the segmentation result. Then, judge the reconstruction loss and the first loss threshold. If the reconstruction loss is greater than or equal to the first loss threshold, it means that the accuracy of the reconstruction result does not meet the standard. At this time, the first loss deviation is obtained by subtracting the first loss threshold from the reconstruction loss, which is used to measure the difference between the reconstruction result and the expected result; if the reconstruction loss is less than the first loss threshold, it means that the accuracy of the reconstruction result has met the standard. At this time, the first loss deviation is 0, indicating that the reconstruction result is consistent with the expected result. At the same time, judge the segmentation loss and the second loss threshold. If the segmentation loss is greater than or equal to the second loss threshold, it means that the accuracy of the segmentation result does not meet the standard. At this time, the second loss deviation is obtained by subtracting the second loss threshold from the segmentation loss, which is used to measure the difference between the segmentation result and the expected result; if the segmentation loss is less than the second loss threshold, it means that the accuracy of the segmentation result has met the standard. At this time, the second loss deviation is 0, indicating that the segmentation result is consistent with the expected result.
[0082] Afterwards, the first loss deviation is compared with the second loss deviation to determine which deviation is larger, and the maximum loss deviation is determined based on the comparison result. If the first loss deviation is greater than the second loss deviation, the maximum loss deviation is the first loss deviation; if the second loss deviation is greater than or equal to the first loss deviation, the maximum loss deviation is the second loss deviation. Subsequently, based on the result of the maximum loss deviation, the branch that needs to be adjusted is determined. If the maximum loss deviation is the first loss deviation, the network parameters of the 3D reconstruction branch need to be adjusted; if the maximum loss deviation is the second loss deviation, the network parameters of the 3D segmentation branch need to be adjusted. According to the branch that needs to be adjusted, the network parameters of the branch are adjusted using the gradient descent method or the adaptive optimization algorithm. The adjusted network parameters will be used for subsequent joint training to further optimize the performance of the corresponding branch to make it more consistent with the expected results.
[0083] Compare the first loss deviation and the second loss deviation to find the largest loss deviation. The largest loss deviation indicates which branch, the reconstruction branch or the segmentation branch, has a larger difference in result from the expected result. According to the largest loss deviation, adjust the network parameters of the corresponding branch to further optimize the performance of the branch and make it more consistent with the expected result.
[0084] In summary, the 3D data processing method based on the dual network super-resolution algorithm provided in the embodiment of the present application has the following technical effects:
[0085] Obtain low-resolution 3D datasets, high-resolution 3D datasets, and 3D semantic segmentation result sets, where low-resolution 3D data, high-resolution 3D data, and 3D semantic segmentation results correspond one to one, providing a data basis for building a 3D dual-branch network model; build a 3D dual-branch network model, the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, where the 3D reconstruction branch and the 3D segmentation branch share model input data, providing support for high-precision semantic segmentation processing of low-resolution 3D data; through low-resolution 3D datasets, high-resolution 3D datasets, and 3D semantic The segmentation result set jointly trains the 3D reconstruction branch and the 3D segmentation branch to obtain the joint loss, providing judgment information for obtaining a 3D dual-branch network model that meets the requirements; when the joint loss is less than the preset loss threshold, the target low-resolution 3D data is obtained; the target low-resolution 3D data is input into the 3D dual-branch network model, and the 3D segmentation branch performs forward operation on the target low-resolution 3D data to obtain the target semantic segmentation result, and realizes high-precision semantic segmentation for the newly input target low-resolution 3D data, thereby achieving the technical effect of improving the accuracy of 3D data semantic segmentation and improving data processing efficiency.
[0086] Embodiment 2
[0087] Based on the same inventive concept as the 3D data processing method based on the dual network super-resolution algorithm in the aforementioned embodiment, Figure 4 As shown, the embodiment of the present application provides a 3D data processing system based on a dual network super-resolution algorithm, the system comprising:
[0088] A training data acquisition module 11 is used to acquire a low-resolution 3D data set, a high-resolution 3D data set, and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data, and the 3D semantic segmentation result correspond to each other one by one;
[0089] A network model construction module 12, used to construct a 3D dual-branch network model, wherein the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data;
[0090] A branch joint training module 13 is used to jointly train the 3D reconstruction branch and the 3D segmentation branch through the low-resolution 3D dataset, the high-resolution 3D dataset and the 3D semantic segmentation result set to obtain a joint loss;
[0091] A target data acquisition module 14, configured to acquire target low-resolution 3D data when the joint loss is less than a preset loss threshold;
[0092] The processing result acquisition module 15 is used to input the target low-resolution 3D data into the 3D dual-branch network model, and the 3D segmentation branch performs forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result.
[0093] Furthermore, the branch joint training module 13 includes the following execution steps:
[0094] Traversing the low-resolution 3D data set to obtain first low-resolution 3D data, and obtaining first high-resolution 3D data and a first 3D semantic segmentation result in the high-resolution 3D data set and the 3D semantic segmentation result set respectively based on the first low-resolution 3D data;
[0095] Inputting the first low-resolution 3D data into the 3D reconstruction branch to obtain first reconstructed 3D data, and calculating a reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data;
[0096] Inputting the first low-resolution 3D data into the 3D segmentation branch, obtaining first segmented 3D data, and calculating a segmentation loss between the first segmented 3D data and the first 3D semantic segmentation result;
[0097] calculating a consistency loss between the first reconstructed 3D data and the first segmented 3D data;
[0098] The reconstruction loss, the segmentation loss, and the consistency loss are added together as the joint loss.
[0099] Furthermore, the branch joint training module 13 also includes the following execution steps:
[0100] The 3D reconstruction branch includes a first encoder and a first decoder;
[0101] Performing point cloud encoding on the first low-resolution 3D data based on the first encoder to obtain a point cloud feature vector;
[0102] Decoding the point cloud feature vector based on the first decoder to generate the first reconstructed 3D data;
[0103] The reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data is calculated by a mean square error.
[0104] Furthermore, the branch joint training module 13 also includes the following execution steps:
[0105] The 3D segmentation branch includes a second encoder and a second decoder;
[0106] Performing feature extraction on the first low-resolution 3D data based on the second encoder to obtain a semantic feature vector;
[0107] Decoding the semantic feature vector based on the second decoder to generate the first segmented 3D data;
[0108] The segmentation loss of the first segmented 3D data and the first 3D semantic segmentation result is calculated by cross entropy loss.
[0109] Furthermore, the branch joint training module 13 also includes the following execution steps:
[0110] The consistency loss function is constructed as:
[0111]
[0112] Among them, L consis is the consistency loss value, N is the total number of points in the first reconstructed 3D data, label(rec, i) is the semantic label of the i-th point in the first reconstructed 3D data, and label(seg, i) is the semantic label of the i-th point in the first segmented 3D data;
[0113] Traversing each point in the first reconstructed 3D data to obtain a first reconstructed 3D point;
[0114] Obtaining a first spatial coordinate and a first semantic label of the first reconstructed 3D point;
[0115] Acquire a second semantic label from the first segmented 3D data according to the first spatial coordinate;
[0116] The consistency between the first semantic label and the second semantic label is calculated according to the consistency loss function to obtain the consistency loss.
[0117] Furthermore, the embodiment of the present application also includes a network parameter optimization module, which includes the following execution steps:
[0118] When the joint loss is greater than or equal to the preset loss threshold, obtaining the number of training times of the current network parameters;
[0119] If the number of training times is less than the iteration threshold, continuing to jointly train the 3D reconstruction branch and the 3D segmentation branch according to the current network parameters;
[0120] If the number of training times is greater than or equal to the iteration threshold, the current network parameters are adjusted, and the 3D reconstruction branch and the 3D segmentation branch continue to be jointly trained according to the adjusted network parameters.
[0121] Furthermore, the network parameter optimization module also includes the following execution steps:
[0122] Acquire a first loss threshold and a second loss threshold, wherein the first loss threshold is a loss judgment threshold of the 3D reconstruction branch, and the second loss threshold is a loss judgment threshold of the 3D segmentation branch;
[0123] If the reconstruction loss is greater than or equal to the first loss threshold, calculating a first loss deviation;
[0124] If the reconstruction loss is less than the first loss threshold, the first loss deviation is 0;
[0125] If the segmentation loss is greater than or equal to the second loss threshold, calculating a second loss deviation;
[0126] If the segmentation loss is less than the second loss threshold, the second loss deviation is 0;
[0127] The first loss deviation is compared with the second loss deviation to obtain a maximum loss deviation, and a network parameter of a corresponding branch is adjusted according to the maximum loss deviation.
[0128] Any step of the method described above can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor to implement any method in the embodiments of the present application, without any unnecessary restrictions.
[0129] Furthermore, the first or second mentioned above may not only represent an order relationship, but may also represent a specific concept, and / or refer to multiple elements that can be selected individually or in full. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.
Claims
1. A 3D data processing method based on a dual network super-resolution algorithm, characterized in that: The method comprises: Obtain a low-resolution 3D dataset, a high-resolution 3D dataset, and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data, and the 3D semantic segmentation result correspond one to one; Constructing a 3D dual-branch network model, wherein the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data; Jointly training the 3D reconstruction branch and the 3D segmentation branch through the low-resolution 3D dataset, the high-resolution 3D dataset, and the 3D semantic segmentation result set to obtain a joint loss; When the combined loss is less than a preset loss threshold, acquiring target low-resolution 3D data; The target low-resolution 3D data is input into the 3D dual-branch network model, and the 3D segmentation branch performs a forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result; The 3D reconstruction branch and the 3D segmentation branch are jointly trained by using the low-resolution 3D dataset, the high-resolution 3D dataset and the 3D semantic segmentation result set to obtain a joint loss, including: Traversing the low-resolution 3D data set to obtain first low-resolution 3D data, and obtaining first high-resolution 3D data and a first 3D semantic segmentation result in the high-resolution 3D data set and the 3D semantic segmentation result set respectively based on the first low-resolution 3D data; Inputting the first low-resolution 3D data into the 3D reconstruction branch to obtain first reconstructed 3D data, and calculating a reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data; Inputting the first low-resolution 3D data into the 3D segmentation branch, obtaining first segmented 3D data, and calculating a segmentation loss between the first segmented 3D data and the first 3D semantic segmentation result; calculating a consistency loss between the first reconstructed 3D data and the first segmented 3D data; Adding the reconstruction loss, the segmentation loss, and the consistency loss as the joint loss; The step of calculating the consistency loss of the first segmented 3D data of the first reconstructed 3D data comprises: The consistency loss function is constructed as: Among them, L consis is the consistency loss value, N is the total number of points in the first reconstructed 3D data, label(rec, i) is the semantic label of the i-th point in the first reconstructed 3D data, and label(seg, i) is the semantic label of the i-th point in the first segmented 3D data; Traversing each point in the first reconstructed 3D data to obtain a first reconstructed 3D point; Obtaining a first spatial coordinate and a first semantic label of the first reconstructed 3D point; Acquire a second semantic label from the first segmented 3D data according to the first spatial coordinate; Calculating the consistency of the first semantic label and the second semantic label according to the consistency loss function to obtain the consistency loss; Wherein, the method further comprises: When the joint loss is greater than or equal to the preset loss threshold, obtaining the number of training times of the current network parameters; If the number of training times is less than the iteration threshold, continuing to jointly train the 3D reconstruction branch and the 3D segmentation branch according to the current network parameters; If the number of training times is greater than or equal to the iteration threshold, adjusting the current network parameters, and continuing to jointly train the 3D reconstruction branch and the 3D segmentation branch according to the adjusted network parameters; If the number of training times is greater than or equal to the iteration threshold, the current network parameters are adjusted, including: Acquire a first loss threshold and a second loss threshold, wherein the first loss threshold is a loss judgment threshold of the 3D reconstruction branch, and the second loss threshold is a loss judgment threshold of the 3D segmentation branch; If the reconstruction loss is greater than or equal to the first loss threshold, calculating a first loss deviation; If the reconstruction loss is less than the first loss threshold, the first loss deviation is 0; If the segmentation loss is greater than or equal to the second loss threshold, calculating a second loss deviation; If the segmentation loss is less than the second loss threshold, the second loss deviation is 0; The first loss deviation is compared with the second loss deviation to obtain a maximum loss deviation, and a network parameter of a corresponding branch is adjusted according to the maximum loss deviation.
2. The method according to claim 1, characterized in that Inputting the first low-resolution 3D data into the 3D reconstruction branch to obtain first reconstructed 3D data, and calculating the reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data, including: The 3D reconstruction branch includes a first encoder and a first decoder; Performing point cloud encoding on the first low-resolution 3D data based on the first encoder to obtain a point cloud feature vector; Decoding the point cloud feature vector based on the first decoder to generate the first reconstructed 3D data; The reconstruction loss between the first reconstructed 3D data and the first high-resolution 3D data is calculated by a mean square error.
3. The method according to claim 1, characterized in that Inputting the first low-resolution 3D data into the 3D segmentation branch, obtaining first segmented 3D data, and calculating the segmentation loss between the first segmented 3D data and the first 3D semantic segmentation result, including: The 3D segmentation branch includes a second encoder and a second decoder; Performing feature extraction on the first low-resolution 3D data based on the second encoder to obtain a semantic feature vector; Decoding the semantic feature vector based on the second decoder to generate the first segmented 3D data; The segmentation loss of the first segmented 3D data and the first 3D semantic segmentation result is calculated by cross entropy loss.
4. A 3D data processing system based on a dual network super-resolution algorithm, characterized in that: A system for implementing a 3D data processing method based on a dual-network super-resolution algorithm as described in any one of claims 1 to 3, comprising: A training data acquisition module, wherein the training data acquisition module is used to acquire a low-resolution 3D data set, a high-resolution 3D data set, and a 3D semantic segmentation result set, wherein the low-resolution 3D data, the high-resolution 3D data, and the 3D semantic segmentation result correspond one to one; A network model construction module, wherein the network model construction module is used to construct a 3D dual-branch network model, wherein the 3D dual-branch network model includes a 3D reconstruction branch and a 3D segmentation branch, wherein the 3D reconstruction branch and the 3D segmentation branch share model input data; A branch joint training module, wherein the branch joint training module is used to jointly train the 3D reconstruction branch and the 3D segmentation branch through the low-resolution 3D dataset, the high-resolution 3D dataset and the 3D semantic segmentation result set to obtain a joint loss; A target data acquisition module, wherein the target data acquisition module is used to acquire target low-resolution 3D data when the joint loss is less than a preset loss threshold; A processing result acquisition module is used to input the target low-resolution 3D data into the 3D dual-branch network model, and the 3D segmentation branch performs a forward operation on the target low-resolution 3D data to obtain a target semantic segmentation result.