A method and device for extracting features from voxel data in real-time three-dimensional scientific computing

Abstract

The present invention relates to the intersection of deep learning and high-performance computing, and in particular to a method and device for extracting features from voxel data of real-time three-dimensional scientific computing. The method comprises: obtaining voxel data, the voxel data containing spatial data of a target three-dimensional object; inputting the voxel data into a first deep neural network, positioning the target three-dimensional object, and obtaining the center position and coordinate axis size information of the target three-dimensional object; cutting the voxel data according to the center position and coordinate axis size information to obtain voxel sub-data containing spatial data; performing SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object; performing point cloud processing on the approximate horizontal posture voxel data to obtain point cloud data; inputting the point cloud data into a second deep neural network to obtain feature data containing the category, posture and size of the target three-dimensional object.

Description

Technical Field

[0001] The present invention relates to the intersection of deep learning and high performance computing, and in particular to a method and device for extracting features from real-time three-dimensional scientific computing voxel data. Background Art

[0002] With the development of high-performance computing technology, various large-scale scientific computing methods have been widely used, generating a large amount of simulation result data.

[0003] For example, computational fluid dynamics simulation programs can simulate the operating laws of air and liquid flows, providing theoretical guidance for aerospace, automotive and other fields; molecular dynamics simulation programs can simulate and study the interaction laws between molecules for drug design; scientific computing programs based on phase field theory can predict the properties of new materials and lead to the discovery of new materials.

[0004] However, the three-dimensional data generated by these scientific computing programs is huge, which poses a challenge to the computer's storage space. If the calculation results in the simulation process can be feature extracted and processed, and only its key feature information is saved, the storage requirements can be greatly reduced.

[0005] Specifically, the inventor hopes to locate the structure or object in the three-dimensional result data, extract its posture, category, size and other features at different time steps, and process and analyze them in real time, thereby avoiding the storage of all the original calculation result data. At the same time, the feature information obtained in real time can also provide additional information for subsequent calculations and guide the calculation work. On the other hand, considering the need to run in coordination with scientific computing programs, the computational efficiency of the feature extraction method is also very important.

[0006] In short, studying real-time and efficient feature extraction methods for voxel data in three-dimensional scientific computing can not only reduce the storage pressure of computers, but also optimize the overall computing process, and has important application prospects. Summary of the invention

[0007] One or more embodiments of this specification describe a method and device for extracting features from real-time three-dimensional scientific computing voxel data, which can be connected to the output end of a scientific computing program to locate, infer, and extract features from target three-dimensional objects contained in the calculation results output by the scientific computing program in real time and efficiently.

[0008] According to a first aspect, a method for extracting features of a three-dimensional object is provided, which is applied to three-dimensional scientific computing voxel data, and the method comprises the following steps:

[0009] Acquire voxel data, wherein the voxel data includes spatial data of a target three-dimensional object;

[0010] Inputting the voxel data into a first deep neural network, positioning the target three-dimensional object, and obtaining the center position and coordinate axis size information of the target three-dimensional object;

[0011] According to the center position and the coordinate axis size information, the voxel data is clipped to obtain voxel sub-data containing the spatial data;

[0012] Performing SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object;

[0013] Performing point cloud processing on the approximately horizontal posture voxel data to obtain point cloud data;

[0014] The point cloud data is input into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

[0015] According to one embodiment, the first deep neural network is a CenterNet3D network, and the second deep neural network is a RotateNet network.

[0016] According to one embodiment, based on the CenterNet network, 2D convolution is replaced by 3D convolution to obtain the CenterNet3D network, and the network is trained by the following steps:

[0017] The training set is input into the CenterNet3D network for training, and after reaching a specified number of training rounds, the model parameters of the CenterNet3D network are saved.

[0018] According to one embodiment, the RotateNet network is obtained by the following steps:

[0019] The objects to be detected in the training set are cropped with random perturbations;

[0020] The cropped data is processed by SVD decomposition and point cloud conversion, and then input into the RotateNet network for training;

[0021] After reaching the specified number of training rounds, the model parameters of the RotateNet network are saved.

[0022] According to one embodiment, the training set includes voxel training data of the object to be detected and corresponding annotation labels;

[0023] The annotation label includes the center point coordinates of the object to be detected in the initial posture, the coordinate axis size in the corresponding horizontal plane coordinate system in the initial posture, the coordinate axis size in the real horizontal posture, the rotation matrix and the object category.

[0024] According to one embodiment, the voxel data is obtained by the following steps:

[0025] Compiling the code implementing the feature extraction method into a dynamic link library to obtain an in-situ feature extraction library, wherein the in-situ feature extraction library has a data input interface;

[0026] The data input interface is called to transfer the calculation result of the three-dimensional scientific computing program into the in-situ feature extraction library, wherein the calculation result includes the voxel data.

[0027] According to one embodiment, the method further comprises:

[0028] The OpenMP instructions are compiled into the in-situ feature extraction library. When the three-dimensional scientific computing program executes inter-process communication, the in-situ feature extraction library performs feature extraction of the voxel data to achieve parallel computing.

[0029] According to one implementation, the in-situ feature extraction library stores data in a memory during feature extraction, and writes the feature data back to the hard disk after feature extraction is completed.

[0030] According to one embodiment, the point cloud processing of the approximately horizontal posture voxel data includes: randomly selecting a number of points from all voxel points of the approximately horizontal posture voxel data to obtain their coordinate point sets.

[0031] According to a second aspect, the present invention further provides a device for extracting features of a three-dimensional object, acting on three-dimensional scientific computing voxel data, specifically comprising:

[0032] An acquisition module: configured to acquire voxel data, wherein the voxel data includes spatial data of a target three-dimensional object;

[0033] A target object positioning module: configured to input the voxel data into a first deep neural network, locate the target three-dimensional object, and obtain the center position and coordinate axis size information of the target three-dimensional object;

[0034] A cropping module: configured to crop the voxel data according to the center position and the coordinate axis size information to obtain voxel sub-data containing the spatial data;

[0035] SVD decomposition module: configured to perform SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object;

[0036] Point cloud module: configured to perform point cloud processing on the approximate horizontal posture voxel data to obtain point cloud data;

[0037] Rotation inference module: configured to input the point cloud data into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

[0038] In the prior art, although some related works have explored the extraction method of voxel features, it is still difficult to directly apply it to the voxel data generated by scientific computing programs. The method provided by the present invention can detect objects of different categories in a three-dimensional voxel space and further extract their posture, category and size information. Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] Firstly, the present invention improves and expands the existing target detection model so that it has the ability to locate three-dimensional objects and can also adapt to the characteristics of voxel data in scientific computing scenarios.

[0040] Secondly, in terms of three-dimensional object pose estimation, existing point cloud registration-based methods require a given initial point cloud, while voxel data in scientific computing is difficult to provide a suitable initial point cloud, and the result of directly applying the singular value decomposition method to estimate the pose is also difficult to control. The method provided by the present invention can accurately estimate the pose of voxel objects in scientific computing scenes in the absence of an initial point cloud.

[0041] Finally, the detection, classification and posture regression of three-dimensional objects at the same time will greatly increase the complexity of the algorithm, and the feature extraction of three-dimensional objects also needs to be coordinated with scientific computing programs in real time. Therefore, time complexity is also a key indicator that must be considered. The present invention designs an efficient computing pipeline and makes full use of computing components to achieve rapid and accurate extraction of multiple types of features of voxel objects in a limited time. It can be built into scientific computing programs to extract feature data of three-dimensional objects in real time, which helps to reduce computer storage pressure and guide subsequent calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A feature extraction system framework for real-time three-dimensional scientific computing voxel data is provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the CenterNet3D network model architecture provided by the present invention;

[0044] Figure 3 A schematic diagram of the RotateNet network model architecture provided by the present invention;

[0045] Figure 4 A schematic diagram of the end-to-end process of inferring the posture of a three-dimensional object provided by the present invention;

[0046] Figure 5A schematic flow chart of a feature extraction method for real-time three-dimensional scientific computing voxel data provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of a computing pipeline structure provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of a feature extraction device for real-time three-dimensional scientific computing voxel data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention proposes a method and device for extracting features from voxel data in real-time three-dimensional scientific computing. In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0050] As mentioned above, current 3D object detection is mainly concentrated in the field of autonomous driving. It mainly uses laser radar to scan surrounding objects to obtain point cloud data, and then detects objects in it, such as vehicles, pedestrians, and bicycles.

[0051] The mainstream 3D object detection methods can be divided into three categories: Voxel-based, Point-based, and a combination of the two. Although there are some methods that convert point cloud data into voxel data for 3D object detection, the voxelized point cloud data generated by LiDAR is very different from the voxel data generated by scientific computing.

[0052] For example, on one hand, the data generated by lidar is very sparse, and after being converted into voxel data, the amount of data is very small, while the voxel data generated by scientific computing is very dense.

[0053] On the other hand, in an autonomous driving environment, there are fewer object categories that need to be identified in three-dimensional target detection, while in the data generated by scientific computing, there may be many types of target three-dimensional objects.

[0054] On the other hand, in the autonomous driving scenario, the posture of the target three-dimensional object is relatively simple, generally only rotating relative to the ground, while in the data generated by scientific computing, the target three-dimensional object often has a more complex posture.

[0055] Therefore, when performing target three-dimensional object detection on data output by scientific computing programs, targeted model optimization is needed to enable the detection model to adapt to the characteristics of voxel data in scientific computing scenarios. At the same time, in the absence of an initial position point cloud, the posture of voxel objects in the scientific computing scene can be accurately estimated, thereby achieving category inference for multiple categories of target objects.

[0056] At the same time, in addition to being able to infer the category of the target object, the inventor also hopes to be able to extract the posture and size features of the target three-dimensional object generated by scientific computing. However, the current research on three-dimensional object feature extraction focuses more on the classification of the target three-dimensional object, while the rotation posture of the object is more of an enhancement to the training data, and is not detected and inferred as a result output like classification.

[0057] At present, the most widely studied method for extracting the posture of three-dimensional objects is point cloud registration, which is mainly to find the corresponding relationship between the source point cloud (source) and the template point cloud (template) to find the rotation matrix representing the posture. Among them, ICP and its improved methods are the most widely used. At the same time, with the emergence of PointNet, the method of extracting two point cloud features by deep learning and then finding the corresponding relationship has also been widely studied, such as PointNetLK, PCRNet, DeepCloset Point (DCP), and RPM-Net.

[0058] However, in scientific computing scenarios, these models cannot obtain the initial position point cloud. Therefore, the method of solving the transformation matrix by finding the corresponding relationship is not suitable for feature extraction of target three-dimensional objects in scientific computing.

[0059] There are also some related works attempting to directly estimate the pose of the target object without a starting position, but they can only estimate the pose of a single category of target objects and lack the ability to infer multiple object categories at the same time.

[0060] Therefore, in the present invention, it is necessary to design a feature extraction method for real-time three-dimensional scientific computing voxel data, which can still locate and infer the target object without the initial position point cloud. At the same time, the inference process can be built into the scientific computing program and run in coordination with the scientific computing program. By extracting features of a large amount of result data generated in the scientific computing program in real time, the full amount of simulation data that the scientific computing program originally needs to save is reduced to the feature data of the target three-dimensional object, thereby greatly reducing the storage pressure of the system.

[0061] The following describes in detail the specific implementation process of a feature extraction method for real-time three-dimensional scientific computing voxel data provided by the present invention under the above technical concept.

[0062] In one embodiment, the method is applied to voxel data generated by a scientific computing program to extract features of a target three-dimensional object. In other embodiments, the method can also be applied to voxel data generated by other types of computing programs to extract features of a target three-dimensional object contained in the voxel data, which will not be listed in this specification.

[0063] The following will first introduce the overall architecture of a real-time three-dimensional scientific computing voxel data feature extraction method proposed by the present invention in combination with the embodiments and drawings, then introduce the training of a deep neural network, and finally explain the process of feature extraction of three-dimensional scientific computing voxel data using the method proposed by the present invention.

[0064] Figure 1 A feature extraction system framework for real-time three-dimensional scientific computing voxel data according to an embodiment of the present invention is shown.

[0065] See also Figure 1 , in which the feature extraction method provided by the present invention is abstracted as a feature extraction library (shown as Insitu_feature_extract_library in the figure), which is connected to a scientific computing program (shown as Simulate_program in the figure), receives voxel data (shown as Voxel_data in the figure) generated by the scientific computing program, performs feature extraction processing on the voxel data, and returns the inference result (shown as extracted_feature in the figure) to the scientific computing program. The feature extraction process includes three stages:

[0066] The first stage (corresponding to Stage 1 in the figure): locate the target three-dimensional object contained in the voxel data.

[0067] In order to effectively locate the target three-dimensional object from the large-scale voxel data generated by scientific computing programs, the present invention further develops the CenterNet3D network model based on CenterNet, which is a single-level, anchor-free convolutional neural network that can be applied to three-dimensional object positioning. It determines the center position and size of the object by generating a key point heat map, thereby realizing the positioning of the three-dimensional object.

[0068] It is understandable that in other embodiments, a deep neural network with similar object positioning capabilities may be used as a basis and improved to obtain a network model capable of positioning three-dimensional objects, which is not specifically limited in this specification.

[0069] Figure 2 Figure 2 shows the schematic diagram of the CenterNet3D network model architecture. Figure 2In this network model, in order to process three-dimensional voxel data, the 2D convolution module in CenterNet is replaced by a 3D convolution module. At the same time, according to the characteristics of the technical problem to be solved by the present invention, the structure of the network is adjusted.

[0070] The entire network is mainly divided into two parts: the backbone network used to extract features of input data, and the detector that uses the features extracted by the backbone network to extract results from the input feature map data.

[0071] In the backbone network, ResNet34 is selected as the backbone network structure, which consists of multiple groups of residual modules (Res_block). The feature map will be downsampled once after a certain number of Res_blocks. The entire network will perform downsampling three times in total to obtain feature data at three different scales. Finally, the three scale features are fused and the results are used as the features of the input data.

[0072] This feature is sent to the subsequent detector to extract the center position of the corresponding target three-dimensional object (marked as Center in the figure) and the size information in the direction parallel to the coordinate axis (marked as hor_whd in the figure) from the input feature map.

[0073] At this point, the positioning of the target three-dimensional object in the voxel data output by the scientific computing program has been completed, and the center point information and corresponding coordinate axis size information of the target three-dimensional object have been obtained. This information will be used in further feature extraction in the next stage.

[0074] The second stage (corresponding to Stage 2 in the figure): cropping the located target three-dimensional object.

[0075] According to the center point information and corresponding coordinate axis size information of the target three-dimensional object obtained in the previous stage, the target three-dimensional object is clipped from the voxel data into a separate voxel space to obtain voxel sub-data (shown as N×3 in the figure). By clipping, the data processing in the subsequent steps can be concentrated on the spatial data of the target three-dimensional object.

[0076] The third stage (corresponding to Stage 3 in the figure): the point cloud data of the cropped target three-dimensional object is used to infer the posture and obtain the final output result.

[0077] In this embodiment, RotateNet will be used as a posture rotation regression network to perform posture inference on the target three-dimensional object. In other embodiments, other deep neural networks with similar capabilities can be used to perform posture inference on the target three-dimensional object, which are not given one by one in this specification.

[0078] Since the target 3D object obtained after cropping still has complex posture changes, directly inputting it into RotateNet will produce unstable feature results and affect the convergence of the model. Therefore, it is necessary to first perform SVD decomposition (Singular Value Decomposition) on the voxel sub-data of the target 3D object, and use the decomposition results to rotate the target 3D object to an approximately horizontal state to remove most of the posture information. In the figure, it is shown that the target 3D object with an initial posture of an approximately horizontal state and the initial posture rotation matrix R corresponding to the object are obtained from N×3 voxel sub-data after SVD processing. 3×3 .

[0079] Among all the voxel points of the obtained voxel data of the approximate horizontal posture, several points are randomly selected, and their coordinate value sets are obtained to complete the point cloud processing of the voxel sub-data, thereby obtaining the point cloud data of the target three-dimensional object with three-dimensional coordinate values.

[0080] Figure 3 Figure 2 shows the RotateNet network model architecture. Figure 3 ,The network model is mainly divided into two parts, namely the backbone network for ,acquiring the features of the input feature image, and the decoder for ,decoding the features to infer the category and posture information of the target ,three-dimensional object.

[0081] In an exemplary embodiment, the backbone network structure is similar to PointNet, which consists of 5 layers of multi-layered perceptrons. After the input feature image point cloud data set is processed by the backbone network, a 1024-dimensional feature vector will be obtained. After that, the feature vector will be sent to the decoder.

[0082] In one embodiment of the present invention, RotateNet has two decoders, one for classifying objects and the other for posture regression. Both decoders have the same structure and contain three fully connected layers, where the number of each fully connected layer is [1024, 1024, 512]. Finally, the classification fully connected layer will output the different probabilities that the target three-dimensional object belongs to N categories (shown as scores in the figure); the size fully connected layer will calculate the width, height, and depth of the target three-dimensional object, three size values ​​(shown as whd in the figure); the posture fully connected layer will obtain the 9 values ​​of the rotation matrix of the target three-dimensional object (shown as pose in the figure).

[0083] The end-to-end process of inferring the target 3D object pose is as follows Figure 4 As shown, by performing SVD decomposition on the voxel data Y of the target three-dimensional object, the voxel data R' of the target three-dimensional object in an approximately horizontal posture can be obtained. T*Y, where the rotation matrix is ​​R'. After the voxel data of the approximate horizontal posture is processed into point cloud, it is input into RotateNet, and the network's estimate E of the residual posture between the approximate horizontal posture and the true horizontal posture can be obtained. Combining E and R', the complete posture rotation matrix R of the target three-dimensional object is calculated. At the same time, RotateNet will also output the category (class) and size (whd) of the target three-dimensional object.

[0084] At this point, the target three-dimensional object can be located from the large-scale voxel data output by the scientific computing program, and feature data including the size, posture and category of the three-dimensional object can be extracted.

[0085] Based on the network model constructed above, the training data is loaded and the model is trained in order to obtain the optimized model parameters suitable for feature extraction of voxel data in three-dimensional scientific computing.

[0086] In an exemplary embodiment, a corresponding data set is prepared according to simulation data output by a target scientific computing program running on a device.

[0087] In one scenario of the above embodiment, part of the data in the data set is divided into a training set for training the network model, and the other part of the data is divided into a test set for testing network performance after the network training is completed. It is understandable that in other scenarios, the data set can also be divided into different categories according to different purposes, which is not specifically limited here.

[0088] The data in the dataset consists of two parts:

[0089] One part is the voxel data to be detected, which contains the spatial data of the target three-dimensional object. This part of data can come from the data actually calculated by the target scientific computing program, or it can come from artificial data with the same characteristics as the data generated by the target scientific computing program.

[0090] The other part is the corresponding label information, which includes the category, position, size and posture information of the target 3D object. This part of information is the feature data corresponding to the target 3D object in the voxel data obtained by any means (for example, manual annotation or pre-set). It specifically includes the following elements:

[0091] [class,center_x,center_y,center_z,w,h,d,hor_x,hor_y,hor_z,R_1,R_2,R_3,R_4,R_5,R_6,R_7,R_8,R_9]

[0092] Among them, class represents the category of the target three-dimensional object. center_x, center_y, center_z, respectively represent the coordinate values ​​of the center point of the target three-dimensional object in the three-dimensional space coordinate system under the condition that the lower left corner of the voxel data space is the origin. w, h, d, respectively represent the corresponding dimensions (lengths) of the x, y, and z coordinate axes in the three-dimensional space coordinate system of the target three-dimensional object in the initial posture. hor_x, hor_y, hor_z, respectively represent the corresponding dimensions (lengths) of the x, y, and z coordinate axes in the original coordinate system (horizontal plane coordinate system) of the target three-dimensional object in the initial posture. R1~R9, respectively correspond to the 9 values ​​of the rotation matrix of the target three-dimensional object arranged in rows.

[0093] At this point, the training data is ready. The following will explain the training methods of the CenterNet3D network and the RotateNet network.

[0094] According to one embodiment, before starting to train the CenterNet3D network, the data in the training set needs to be formatted to adapt to the input requirements of the network. The voxel data is converted into a numpy array. [center_x, center_y, center_z, hor_x, hor_y, hor_z] is selected from the label information data and converted into the result data format after the corresponding downsampling ratio according to the requirements of the CenterNet3D network as the supervision target.

[0095] It can be understood that the above-mentioned formatting of the training set data is a formatting method adopted in a specific scenario of an exemplary embodiment of the present invention. In other embodiments, it is necessary to perform matching formatting operations on the training data based on the requirements of the adopted network for input data, which is not specifically limited in this specification.

[0096] Set the training parameters, input the training data into the CenterNet3D network, and start training.

[0097] In one scenario of this embodiment, the training parameters of the CenterNet3D network are set to batch_size=12, initial learning_rate=1e-4, decay rate=0.95, input feature image size=128, and the optimizer uses the Adams optimizer. In other scenarios, different training parameters can be set according to the needs of the specific scenario, and are not specifically limited here.

[0098] After running to the specified number of training rounds, stop training and save the weight parameters of the CenterNet3D model obtained after training.

[0099] According to another embodiment, before starting the training of the RotateNet network, it is necessary to crop the target three-dimensional object from the voxel data according to the label information of the positioning of the target three-dimensional object in the training set (i.e., [center_x, center_y, center_z, hor_x, hor_y, hor_z]), simulating the cropping operation performed by CenterNet3D.

[0100] In one example, the positioning information of the target three-dimensional object used for cropping is disturbed, and the coordinate value of the center point is randomly disturbed between [0, 2] in the three coordinate axis directions. At the same time, the cropping error in the horizontal plane coordinate system direction is also increased by the perturbation error between [0, 2]. In other examples, different ranges of disturbances can be added in different dimensions, depending on the specific model training requirements, which are not limited here. The cropped voxel data is used as the training data of the RotateNet network model. After SVD decomposition and point cloud conversion processing, it is input into the model for training.

[0101] During the training process, RotateNet performs supervised learning based on a large amount of feature image data with rotation in the training set. The supervision target is the accurate rotation angle contained in the label information. RotateNet predicts the rotation angle of the target three-dimensional object based on the input feature image. By repeatedly adjusting the network parameters, the model gradually improves the accuracy of the rotation angle inference. At the same time, the RotateNet network also supervises the network training based on the object category and size information contained in the label information, gradually improving the accuracy of the category inference and size inference of the target three-dimensional object.

[0102] Set the training parameters and start training.

[0103] In one scenario of this embodiment, the training parameters of the RotateNet network are set to batch_size=32, initial learning_rate=1e-4, and decay rate=0.1. In other scenarios, different training parameters can be set according to the needs of specific scenarios, and are not specifically limited here.

[0104] In an example of this embodiment, in order to ensure that R1 to R9 output by RotateNet can form a rotation matrix, an additional restriction condition is added during network training to make the estimate E of the residual pose learned by the RotateNet network as orthogonal as possible.

[0105] After running to the specified number of training rounds, stop training and save the weight parameters of the RotateNet model obtained after training.

[0106] So far, a network model that can extract features from three-dimensional scientific computing voxel data has been obtained, and the training of the model has been completed.

[0107] In one embodiment, the model parameters and the code implementing the aforementioned feature extraction method are compiled into a dynamic link library to obtain an in-situ feature extraction library. The in-situ feature extraction library has a data input interface that can receive voxel data of features to be extracted. The in-situ feature extraction library is moved to a folder of a target scientific computing program, so that the target scientific computing program can pass in the voxel data generated by it by calling the data input interface of the in-situ feature extraction library, thereby realizing the in-situ feature extraction library built into the scientific computing program.

[0108] After completing the above feature extraction model preparation work, you can start to extract features from the voxel data generated by the three-dimensional scientific computing program. The following will describe a feature extraction method flow of real-time three-dimensional scientific computing voxel data in the inference stage according to an exemplary embodiment.

[0109] Figure 5 FIG. 2 shows a flow chart of a method for extracting features of voxel data in real-time three-dimensional scientific computing according to an exemplary embodiment of the present invention. Figure 5 , the method specifically comprises the following steps:

[0110] S501: Acquire voxel data, where the voxel data includes spatial data of a target three-dimensional object.

[0111] In an example of this embodiment, the code implementing the feature extraction method is compiled into an in-situ feature extraction library in the form of a dynamic link library and built into a scientific computing program.

[0112] The scientific computing program calls the data input interface of the in-situ feature extraction library and transfers the calculation results (including the voxel data of the target three-dimensional object to be extracted) to the in-situ feature extraction library.

[0113] Since the in-situ feature extraction library is built into the scientific computing program, when inputting voxel data, the scientific computing program only needs to pass the memory pointer where the calculation result is located to the data input interface. The in-situ feature extraction library can directly obtain the calculation result data from the memory and complete subsequent operations such as voxelization and downsampling.

[0114] S503: Input the voxel data into a first deep neural network, locate the target three-dimensional object, and obtain the center position and coordinate axis size information of the target three-dimensional object.

[0115] In an example of this embodiment, the first deep neural network uses a CenterNet3D network. The construction and training of the network have been described in detail in the specification and will not be repeated here. It is understandable that in other examples, other network models with equivalent functions for positioning three-dimensional objects can also be used, which are not listed one by one in this specification.

[0116] In this stage of processing, the CenterNet3D network will locate the target three-dimensional object in the input voxel data and output the corresponding result data, which contains [center_x, center_y, center_z, hor_x, hor_y, hor_z].

[0117] S505: Crop the voxel data according to the center position and the coordinate axis size information to obtain voxel sub-data containing the spatial data.

[0118] S507: Perform SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object.

[0119] S509: Performing point cloud processing on the approximate horizontal posture voxel data to obtain point cloud data.

[0120] Among all the voxel points of the obtained voxel data of the approximate horizontal posture, several points are randomly selected, and their coordinate value sets are obtained to complete the point cloud processing of the voxel sub-data, thereby obtaining the point cloud data of the target three-dimensional object with three-dimensional coordinate values.

[0121] S511: Input the point cloud data into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

[0122] In one example of this embodiment, the second deep neural network uses a RotateNet network. The construction and training of the network have been described in detail in the specification and will not be repeated here. It is understandable that in other examples, other network models with equivalent functions for inferring the posture of three-dimensional objects can also be used, which are not listed one by one in this specification.

[0123] In one example, the selected RotateNet model receives an input feature map of size 1024, so in the aforementioned steps, 1024 voxel points are randomly selected from the voxel data of the approximate horizontal posture, and their coordinate value sets are obtained (if there are less than 1024 voxel points, 0 is filled to supplement to 1024 points), and the point cloud processing of the voxel data is completed. In other examples, according to the requirements of the selected neural network for the input feature map, the randomly selected points can have different values, which are not specifically limited here.

[0124] In an example of this embodiment, during the RotateNet inference process, the residual pose estimate E obtained by network inference is unit orthogonalized. First, E is decomposed by SVD, and the singular value matrix ∑ obtained by the decomposition is replaced with the unit matrix, and then the left singular matrix U and the right singular matrix V are obtained. T , the unit orthogonalized residual posture E is obtained by the following formula.

[0125] E=U∑V T (1)

[0126] In this stage of processing, the RotateNet network will infer the posture, category, and size of the target three-dimensional object in the input point cloud data.

[0127] By combining the approximate horizontal posture voxel data of the target 3D object obtained by SVD decomposition in the previous step with the residual posture estimation obtained by the RotateNet network, a complete posture estimation of the target 3D object can be obtained. At the same time, the classification fully connected layer in the RotateNet network will infer the category of the target 3D object, and the size fully connected layer will calculate the size value of the target 3D object. Finally, the corresponding result data is output, which includes [class, w, h, d, R_1, R_2, R_3, R_4, R_5, R_6, R_7, R_8, R_9].

[0128] Combined with the positioning information of the target three-dimensional object output in the previous steps, the model completes the feature extraction of the spatial data of the target three-dimensional object contained in the voxel data output by the three-dimensional scientific computing program. The result data contains the positioning information, size information, posture information and category information of the target three-dimensional object.

[0129] In one example of this embodiment, the in-situ feature extraction library compiled by the above steps performs the above feature extraction steps. During the feature extraction process, the calculated data is stored in the memory, and after the feature extraction is completed, the feature data stored in the memory is written back to the hard disk.

[0130] In another exemplary embodiment, as the simulation scale of the scientific computing program continues to increase, the data scale of the calculation results generated by it also increases accordingly. In order to be able to calculate a larger data space, the communication volume and communication frequency between multiple processes of the scientific computing program will also continue to increase. During each process communication time, the computing component will be idle, thereby causing the overall computing efficiency of feature extraction to decrease. In response to this problem, the inventor designed a computing pipeline. In order to implement the pipeline, the OpenMP instruction is compiled into the in-situ feature extraction library. While the scientific computing program executes inter-process communication, the in-situ feature extraction library can perform feature extraction of the voxel data, realize parallel computing, and maximize the use of computing components.

[0131] Figure 6 FIG. 1 shows a schematic diagram of a computing pipeline structure provided according to an embodiment of the present invention. Figure 6 The pipeline integrates all the steps in the scientific computing process and the multi-scale feature extraction process. The multi-scale feature extraction consists of several small-scale feature extractions, where the small-scale feature extraction (referring to the feature extraction of the voxel data generated by the scientific computing program process in the current accelerator) is performed by the accelerator.

[0132] In each time step, the computational data generated by the accelerator computing process is extracted once. After multiple time steps, the data in multiple different processes are collected and merged through MPI to perform feature extraction at different scales, and the OpenMP mechanism can be used to parallelize feature extraction and inter-process communication.

[0133] like Figure 6 As shown in part (i), on the accelerator timeline, the scientific computing program performs computing operations (shown as simulatorn steps time boxes in the figure). During its execution, there are multiple inter-process communication time windows (shown as hexagons on the CPU timeline in the figure). In the time boxes of inter-process communication, on the accelerator timeline, there will be idle computing resources. By introducing the OpenMP mechanism, while communicating between processes, the accelerator can perform feature extraction calculations of the in-situ feature extraction library (shown as hexagons on the accelerator timeline in the figure), so that the computing components can be maximized and idle computing resources can be avoided.

[0134] In an example of this embodiment, when collecting data in multiple different processes through MPI, the simulation result data generated by different processes are first packaged and compressed (shown as pack in the figure) to reduce the amount of transmitted data, and then transmitted through MPI (shown as extractorMPI in the figure). After receiving the data, the in-situ feature extraction library needs to first perform an unpacking operation (shown as unpack in the figure) and then perform feature extraction calculations.

[0135] Figure 6 Part (iii) of the figure shows the parallel computing on the computing pipeline in a unit of time. It can be seen that while the CPU is executing the inter-process communication of the scientific computing program (shown as simulator MPI in the figure), the feature extraction calculation of the in-situ feature extraction library is executed in the accelerator (shown as extractor in the figure).

[0136] Through such a designed computing pipeline, on the one hand, computing hardware resources can be fully utilized. On the other hand, since the in-situ feature extraction library only needs to input the simulation results generated by the scientific computing program, and what needs to be passed in is the data pointer rather than the data itself, there is almost no impact on the computing performance of large-scale scientific computing applications.

[0137] In another embodiment of the present invention, a real-time three-dimensional scientific computing voxel data feature extraction device is provided, and the device can be deployed on any device or platform with computing and processing capabilities. Figure 7 A schematic diagram showing a device for extracting features from voxel data of real-time three-dimensional scientific computing according to an embodiment.

[0138] like Figure 7 As shown, the device 700 specifically includes:

[0139] Acquisition module 701: configured to acquire voxel data, wherein the voxel data includes spatial data of a target three-dimensional object;

[0140] The target object positioning module 702 is configured to input the voxel data into a first deep neural network, locate the target three-dimensional object, and obtain the center position and coordinate axis size information of the target three-dimensional object;

[0141] The cropping module 703 is configured to crop the voxel data according to the center position and the coordinate axis size information to obtain voxel sub-data containing the spatial data;

[0142] SVD decomposition module 704: configured to perform SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object;

[0143] Point cloud module 705: configured to perform point cloud processing on the approximately horizontal posture voxel data to obtain point cloud data;

[0144] The rotation inference module 706 is configured to input the point cloud data into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

[0145] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting features from a three-dimensional object, characterized in that: Applied to 3D scientific computing voxel data, including the following steps: Acquire voxel data, wherein the voxel data includes spatial data of a target three-dimensional object; Inputting the voxel data into a first deep neural network, positioning the target three-dimensional object, and obtaining the center position of the target three-dimensional object and the size information of the target three-dimensional object in the direction of the coordinate axis; According to the center position and the size information of the target three-dimensional object in the coordinate axis direction, the voxel data is clipped to obtain voxel sub-data containing the spatial data; Performing SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object; Performing point cloud processing on the approximate horizontal posture voxel data to obtain point cloud data; The point cloud data is input into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

2. The method according to claim 1, characterized in that The first deep neural network is a CenterNet3D network, and the second deep neural network is a Rotationnet network.

3. The method according to claim 2, characterized in that Based on the CenterNet network, the 2D convolution is replaced by the 3D convolution to obtain the CenterNet3D network, and the network is trained by the following steps: The training set is input into the CenterNet3D network for training, and after reaching a specified number of training rounds, the model parameters of the CenterNet3D network are saved.

4. The method according to claim 2, characterized in that: The Rotationnet network is obtained by following the steps below: The objects to be detected in the training set are cropped with random perturbations; The cropped data is processed by SVD decomposition and point cloud conversion, and then input into the Rotationnet network for training; After reaching the specified number of training rounds, the model parameters of the Rotationnet network are saved.

5. The method according to claim 3 or 4, characterized in that: The training set includes voxel training data of the object to be detected and corresponding annotation labels; The annotation label includes the center point coordinates of the object to be detected in the initial posture, the coordinate axis size in the corresponding horizontal plane coordinate system in the initial posture, the coordinate axis size in the real horizontal posture, the rotation matrix and the object category.

6. The method according to claim 1, characterized in that The voxel data is specifically obtained by the following steps: Compiling the code implementing the feature extraction method into a dynamic link library to obtain an in-situ feature extraction library, wherein the in-situ feature extraction library has a data input interface; The data input interface is called to transfer the calculation result of the three-dimensional scientific computing program into the in-situ feature extraction library, wherein the calculation result includes the voxel data.

7. The method according to claim 6, characterized in that The method further comprises: The OpenMP instructions are compiled into the in-situ feature extraction library. When the three-dimensional scientific computing program executes inter-process communication, the in-situ feature extraction library performs feature extraction of the voxel data to achieve parallel computing.

8. The method according to claim 6, characterized in that The method further comprises: The in-situ feature extraction library stores data in the memory during the feature extraction process, and writes the feature data back to the hard disk after the feature extraction is completed.

9. The method according to claim 1, characterized in that: The point cloud processing of the approximately horizontal posture voxel data includes: A number of points are randomly selected from all voxel points of the approximate horizontal posture voxel data to obtain their coordinate point sets.

10. A device for extracting features from a three-dimensional object, characterized in that: Acts on 3D scientific computing voxel data, including: An acquisition module: configured to acquire voxel data, wherein the voxel data includes spatial data of a target three-dimensional object; A target object positioning module: configured to input the voxel data into a first deep neural network, locate the target three-dimensional object, and obtain the center position of the target three-dimensional object and the size information of the target three-dimensional object in the direction of the coordinate axis; A clipping module: configured to clip the voxel data according to the center position and the size information of the target three-dimensional object in the coordinate axis direction to obtain voxel sub-data containing the spatial data; SVD decomposition module: configured to perform SVD decomposition on the voxel sub-data to obtain approximate horizontal posture voxel data of the target three-dimensional object; Point cloud module: configured to perform point cloud processing on the approximate horizontal posture voxel data to obtain point cloud data; Rotation inference module: configured to input the point cloud data into a second deep neural network to obtain feature data including the category, posture and size of the target three-dimensional object.

Images