3D target detection acceleration method for embedded computing platform and storage medium

Abstract

The invention discloses a 3D target detection acceleration method for an embedded computing platform, and the method comprises the steps: S1, reading and analyzing point cloud and image data captured by a vehicle-mounted laser radar and a camera, obtaining multi-modal data, and storing the multi-modal data in a GPU; step S2, the trained point cloud-image pre-fusion 3D target detection model is optimized through a TensorRT tool; s3, inputting the multi-modal data in the GPU into the TensorRT model, and outputting a 3D detection frame of an interested target; and S4, fusing the predicted target point cloud and the corresponding pixels of the image through a CUDA architecture, and finally generating a three-dimensional fusion detection result in the two-dimensional image. A computer program for executing the method is stored in the storage medium. The method has the advantages of being simple in principle, wide in application range, capable of improving detection efficiency and precision and the like.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of road target detection, and in particular to a 3D target detection acceleration method and storage medium for an embedded computing platform. Background Art

[0002] Road target detection technology is a key technology in the field of intelligent driving. Due to the diverse backgrounds and complex targets of road scenes, environmental perception and key target detection systems usually require high real-time performance and reliability. As for intelligent driving, the computing power and resources of intelligent driving's on-board embedded devices or edge computing platforms are fundamentally very limited. Therefore, how to optimize, accelerate and deploy deep learning network models on these platforms has become a research hotspot that both academia and industry are concerned about, and the demand for practical applications is becoming increasingly strong.

[0003] At present, most of the target detection methods used on embedded computing platforms are based on a single sensor, such as obtaining image data obtained by a vehicle-mounted camera. For example, the inference acceleration of the 2D target detection model is based on point cloud data obtained by a vehicle-mounted laser radar. Traditional point cloud filtering and clustering methods are generally used for 3D target detection. These methods do not fully utilize the advantages of each sensor and cannot achieve real-time target detection in all-weather and complex scenes.

[0004] The shortcomings of these traditional methods are:

[0005] 1. The inference speed of the 3D object detection model based on the fusion of image and point cloud data features cannot meet the real-time requirements;

[0006] 2. For each frame of fused data, the overall operating efficiency of the 3D target detection system on the vehicle embedded platform is low. Summary of the invention

[0007] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a 3D target detection acceleration method and storage medium for embedded computing platforms which has a simple principle, a wide range of applications, and can improve detection efficiency and accuracy.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A 3D object detection acceleration method for an embedded computing platform, comprising:

[0010] Step S1: Read and analyze the point cloud and image data captured by the vehicle-mounted laser radar and camera, obtain multimodal data and store it in the GPU;

[0011] Step S2: Optimize the trained point cloud-image front fusion 3D object detection model through the TensorRT tool;

[0012] Step S3: Input the multimodal data in the GPU into the TensorRT model and output the 3D detection box of the target of interest;

[0013] Step S4: The predicted target point cloud is fused with the corresponding pixels of the image through the CUDA architecture, and finally a three-dimensional fusion detection result is generated in the two-dimensional image.

[0014] As a further improvement of the method of the present invention: in the step S1, the point cloud and image data captured by the vehicle-mounted laser radar and camera are read and analyzed through the CDUA architecture to obtain multimodal data after time-space synchronization and store it in the GPU.

[0015] As a further improvement of the method of the present invention: the process of step S1 includes:

[0016] Step S101: capturing the original image and point cloud data of the vehicle-mounted sensor and storing them in the GPU memory;

[0017] Step S102: parsing the original image and point cloud data in parallel under the CUDA architecture to obtain time-synchronized multimodal data;

[0018] Step S103: preprocessing the image data using opencv under the CUDA architecture;

[0019] Step S104: pre-process the point cloud data under the CUDA architecture, and filter and select the point cloud data according to the image scale and GPU computing power.

[0020] As a further improvement of the method of the present invention: in the step S103, the preprocessing includes one or more of cropping, noise reduction, and normalization.

[0021] As a further improvement of the method of the present invention: in the step S2, the functional modules in the model that do not support TensorRT conversion need to be converted into ONNX by custom operators first, and then the corresponding custom plug-ins are written to complete the TensorRT model conversion.

[0022] As a further improvement of the method of the present invention: the process of step S2 includes:

[0023] Step S201: Reading a 3D object detection Pytorch model based on image and point cloud pre-fusion data;

[0024] Step S202: Setting a custom operator OP for the functional functions in the model that cannot be directly converted to ONNX;

[0025] Step S203: corresponding symbolic functions Symbolic are defined for the above functional functions respectively, the parameters of the symbolic functions are consistent with the original functions, and the return value of the symbolic function is the set operator OP;

[0026] Step S204: Call the Pytorch interface to complete the binding of the symbolic function and the original function. In the process of converting to the ONNX model, the symbolic function will be called instead of the original function to generate a model calculation graph in the form of setting OP;

[0027] Step S205: Read the converted ONNX model and related parameters;

[0028] Step S206: For the operator OP, write the corresponding custom plug-in function package in TensorRT, and add or rewrite the method attributes under the Custom class according to the functional requirements;

[0029] Step S207: Add a custom plug-in interface to the plug-in initialization source file, and complete the plug-in registration in the Cmake file to realize the correspondence between the ONNX custom operator OP and the custom plug-in;

[0030] Step S208: Compile the custom plug-in function package, generate the corresponding dynamic link library, and import it into the lib folder under the TensorRT installation directory.

[0031] Step S209: Parse the ONNX model through the TensorRT interface, obtain the relationship between network layers, and generate a TensorRT model for the vehicle-mounted target detection embedded platform;

[0032] Step S210: Parse the model for the first time. After building the engine, the model needs to be serialized and the engine needs to be saved locally. The local engine is directly read for deserialization.

[0033] Step S211: input corresponding multimodal data to obtain accelerated inference results.

[0034] As a further improvement of the method of the present invention: the process of step S3 includes:

[0035] Step S301: input data into the TensorRT model and output the target prediction box position information, category information, and confidence level;

[0036] Step S302: Calculate the image pixel position corresponding to the point cloud under the CUDA architecture according to the extrinsic matrix of the camera and the laser radar and the camera matrix.

[0037] As a further improvement of the method of the present invention: the process of step S4 includes:

[0038] Step S401: projecting the 3D target point cloud to a 2D image under the CUDA architecture to achieve fusion of the target point cloud and the corresponding image data;

[0039] Step S402: finally generating a 3D fusion detection result in the two-dimensional image, including the category, location information and confidence level of the target.

[0040] As a further improvement of the method of the present invention: in step S4, the three-dimensional fusion detection result includes category information and position information of the target.

[0041] The present invention further provides a storage medium, which can be read by a computer or a processor, and stores a computer program for executing any one of the above methods.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] 1. The 3D target detection acceleration method and storage medium for embedded computing platforms of the present invention have a simple principle, a wide range of applications, and can improve detection efficiency and accuracy. The present invention targets target detection requirements in actual road scenes, that is, a custom TensorRT plug-in for the bidirectional cascade feature extraction module in the pre-fusion target detection model to achieve model reasoning acceleration, and obtains the vehicle-mounted camera and lidar data for fusion, and uses the complementarity of multiple sensors to achieve a full sense of the environment.

[0044] 2. The 3D target detection acceleration method and storage medium for embedded computing platforms of the present invention realize real-time reading, parsing, fusion and other functions for vehicle-mounted cameras and lidar data based on the CUDA architecture. That is, the present invention optimizes and converts the image-point cloud fusion target detection model through the TensorRT plug-in to achieve reasoning acceleration; and for the non-neural network processing part of the entire target detection system, the reading, parsing and fusion of embedded platform data are realized based on the CUDA architecture, and the acceleration of data pre-processing and post-processing is completed, thereby ensuring the calculation accuracy and operation efficiency of the system deployed in the embedded platform.

[0045] 3. The 3D target detection acceleration method and storage medium for embedded computing platforms of the present invention define a TensorRT plug-in to accelerate model reasoning for the bidirectional cascade feature extraction module in the pre-fusion target detection model. The detection efficiency of the model on an embedded device supporting CUDA is increased by 60%, meeting the requirements of on-site real-time detection.

[0046] 4. The 3D target detection acceleration method and storage medium for embedded computing platforms of the present invention realize real-time reading, parsing, fusion and other functions for vehicle-mounted cameras and lidar data based on the CUDA architecture. The overall operating efficiency of the 3D target detection system on embedded devices supporting CUDA is improved by 50%, meeting the real-time operation requirements on site. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process of the present invention.

[0048] Figure 2 It is a schematic diagram of the design process of the custom OP operator in a specific application example of the present invention.

[0049] Figure 3 It is a schematic diagram of the design process of customizing the TensorRT plug-in in a specific application example of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, a 3D target detection acceleration method for an embedded computing platform of the present invention includes the following steps:

[0052] Step S1: Read and analyze the point cloud and image data captured by the vehicle-mounted laser radar and camera, obtain multimodal data and store it in the GPU;

[0053] Step S2: Optimize the trained point cloud-image front fusion 3D object detection model through the TensorRT tool; the TensorRT is a high-performance deep learning inference optimizer developed by NVIDIA, which includes a complete toolkit from model optimization, compilation to deployment;

[0054] Step S3: Input the multimodal data in the GPU into the TensorRT model and output the 3D detection box of the target of interest;

[0055] Step S4: The predicted target point cloud is fused with the corresponding pixels of the image through the CUDA architecture, and finally a three-dimensional fusion detection result is generated in the two-dimensional image.

[0056] In a specific application example, in step S4, the three-dimensional fusion detection result includes category information and location information of the target.

[0057] In a specific application example, in step S1, the point cloud and image data captured by the vehicle-mounted laser radar and camera are read and parsed through the CDUA architecture to obtain multimodal data after time and space synchronization and store it in the GPU; the CDUA is a computing unified device architecture, a GPU-based parallel computing platform and programming model developed by NVIDIA.

[0058] In a specific application example, the process of step S1 may include, according to actual needs:

[0059] Step S101: capturing the original image and point cloud data of the vehicle-mounted sensor and storing them in the GPU memory;

[0060] Step S102: parsing the original image and point cloud data in parallel under the CUDA architecture to obtain time-synchronized multimodal data;

[0061] Step S103: preprocessing the image data using opencv under the CUDA architecture; the preprocessing includes operations such as cropping, noise reduction, and normalization;

[0062] Step S104: pre-process the point cloud data under the CUDA architecture, and filter and select the point cloud data according to the image scale and GPU computing power.

[0063] In a specific application example, in step S2, the functional modules in the model that do not support TensorRT conversion need to be converted into ONNX by custom operators first, and then the corresponding custom plug-ins are written to complete the TensorRT model conversion.

[0064] In a specific application example, the process of step S2 may include, according to actual needs:

[0065] Step S201: Reading a 3D object detection Pytorch model based on image and point cloud pre-fusion data;

[0066] Step S202: Designing custom operators OP for the functional functions in the model that cannot be directly converted to ONNX, such as point cloud downsampling in the point cloud feature extraction network, three-neighbor distance index in the sphere, bilinear interpolation and other functions;

[0067] Step S203: corresponding symbolic functions Symbolic are defined for the above functional functions respectively, the parameters of the symbolic function are consistent with the original function, and the return value of the symbolic function is the customized operator OP;

[0068] Step S204: Call the Pytorch interface to complete the binding of the symbolic function and the original function. In the process of converting to the ONNX model, the symbolic function will be called instead of the original function to generate a model calculation graph in the form of a customized OP;

[0069] Step S205: Read the converted ONNX model and related parameters, including data frame processing batch, model inference accuracy, model input size, input and output nodes, etc.;

[0070] Step S206: For the previously customized operator OP, write the corresponding custom plug-in function package in TensorRT, and add or rewrite the method properties under the Custom class according to the functional requirements;

[0071] Step S207: Add a custom plug-in interface to the plug-in initialization source file, and complete the plug-in registration in the Cmake file to realize the correspondence between the ONNX custom operator OP and the custom plug-in;

[0072] Step S208: Compile the custom plug-in function package, generate the corresponding dynamic link library, and import it into the lib folder under the TensorRT installation directory.

[0073] Step S209: Parse the ONNX model through the TensorRT interface, obtain the relationship between network layers, optimize the model structure and generate a TensorRT model for the vehicle-mounted target detection embedded platform;

[0074] Step S210: After parsing the model for the first time and building the engine, the model needs to be serialized and the engine needs to be saved locally, thereby saving the subsequent model parsing and building time. The local engine can be directly read for deserialization;

[0075] Step S211: input corresponding multimodal data to obtain accelerated inference results.

[0076] In a specific application example, the process of step S3 may include, according to actual needs:

[0077] Step S301: input data into the TensorRT model and output the target prediction box position information, category information, and confidence level;

[0078] Step S302: Calculate the image pixel position corresponding to the point cloud under the CUDA architecture according to the extrinsic matrix of the camera and the laser radar and the camera matrix.

[0079] In a specific application example, the process of step S4 may include, according to actual needs:

[0080] Step S401: projecting the 3D target point cloud to a 2D image under the CUDA architecture to achieve fusion of the target point cloud and the corresponding image data;

[0081] Step S402: finally generating a 3D fusion detection result in the two-dimensional image, including the category, location information and confidence level of the target.

[0082] In this invention, a custom OP operator is designed. This invention aims at the various functions in the image and point cloud pre-fusion 3D object detection Pytorch model that cannot be directly converted into the ONNX model, and designs a custom OP operator to facilitate the conversion from the Pytorch model to the ONNX model. Figure 2 First, add the torch::RegisterOperators function required to register the operator after the corresponding function, add the corresponding parameters, and complete the registration of the OP operator; then write a CMakeLists file, specify the location of the external library that the function needs to call, compile the function source code through CMake, and generate a dynamic library file (generate a .so file under Linux system and a .dll file under Windows system); then use the torch.ops.load_library function to call the generated dynamic library file, and finally complete the conversion from the Pytorch model to the ONNX model.

[0083] In the present invention, a custom TensorRT plug-in is designed. For the functions in the image and point cloud pre-fusion 3D object detection Pytorch model that cannot be directly converted into the ONNX model, a custom OP operator is designed to realize the conversion from the Pytorch model to the ONNX model. Therefore, in the subsequent process of converting the ONNX model to the TensorRT model, it is necessary to customize the TensorRT plug-in for the specific functions of the above-mentioned functions to facilitate the conversion into the final TensorRT model. See Figure 3 First, for each function that cannot be directly converted into an ONNX model, write the corresponding custom plug-in function package in TensorRT, and add or rewrite each type in the plug-in function package according to the actual function needs of the function; then add the custom plug-in interface in the custom plug-in initialization source file (.cpp file), complete the plug-in registration in the CMakeLists file, and realize the correct correspondence between the above custom OP operator and the custom TensorRT plug-in; then compile the custom plug-in function package through CMake to generate the corresponding dynamic link library file (generate .so file under Linux system, generate .dll file under Windows system); finally, parse the previously generated ONNX model through the TensorRT interface, obtain the relationship between the various layers of the network inside the model, optimize the model structure and generate a TensorRT model for the embedded platform.

[0084] The present invention selects any multimodal (image and point cloud) fusion 3D target detection algorithm, uses image and point cloud data for training, and can obtain a model in Pytorch format. Use the model to perform 3D target detection, input image and point cloud data, and output the corresponding 3D target detection result. The present invention can convert the Pytorch format model into an ONNX format model in an embedded platform, and then convert it into a TensorRT format model. The final TensorRT model greatly improves the computing efficiency on the basis of functional consistency compared to the Pytorch model, and realizes the model reasoning acceleration of the 3D target detection task; at the embedded platform level, based on the CUDA architecture, the embedded platform data reading, parsing, fusion and other functions are realized, compared with not using the CUDA architecture, it can achieve a substantial speed-up of the embedded platform data pre-processing (input) and post-processing (output) tasks. By combining the speed-up of model reasoning and the speed-up of data pre- and post-processing, the acceleration of 3D target detection in an embedded computing platform is realized.

[0085] In other application examples, the original 3D object detection model type may also be selected as Caffe, uff, or other formats, which should also be within the protection scope of the present invention.

[0086] The 3D object detection acceleration method for embedded computing platforms of the present invention can be applied to any embedded computing device that supports the CUDA architecture.

[0087] The present invention further provides a storage medium, which can be read by a computer or a processor and stores a computer program for executing the above method.

[0088] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A 3D object detection acceleration method for embedded computing platforms, characterized in that: include: Step S1: Read and analyze the point cloud and image data captured by the vehicle-mounted laser radar and camera, obtain multimodal data and store it in the GPU; Step S2: Optimize the trained point cloud-image front fusion 3D object detection model through the TensorRT tool; Step S3: Input the multimodal data in the GPU into the TensorRT model and output the 3D detection box of the target of interest; Step S4: The predicted target point cloud is fused with the corresponding pixels of the image through the CUDA architecture, and finally a three-dimensional fusion detection result is generated in the two-dimensional image.

2. The 3D object detection acceleration method for embedded computing platforms according to claim 1, characterized in that: In step S1, the point cloud and image data captured by the vehicle-mounted laser radar and camera are read and analyzed through the CDUA architecture to obtain multimodal data after time-space synchronization and store it in the GPU.

3. The 3D object detection acceleration method for embedded computing platforms according to claim 2, characterized in that: The process of step S1 includes: Step S101: capturing the original image and point cloud data of the vehicle-mounted sensor and storing them in the GPU memory; Step S102: parsing the original image and point cloud data in parallel under the CUDA architecture to obtain time-synchronized multimodal data; Step S103: preprocessing the image data using opencv under the CUDA architecture; Step S104: pre-process the point cloud data under the CUDA architecture, and filter and select the point cloud data according to the image scale and GPU computing power.

4. The 3D object detection acceleration method for embedded computing platforms according to claim 3 is characterized in that: In step S103, the preprocessing includes one or more of cropping, noise reduction, and normalization.

5. The 3D object detection acceleration method for embedded computing platforms according to any one of claims 1 to 4, characterized in that: In step S2, for the functional modules in the model that do not support TensorRT conversion, custom operators are required to be converted into Onnx first, and then corresponding custom plug-ins are written to complete the TensorRT model conversion.

6. The 3D object detection acceleration method for embedded computing platforms according to any one of claims 1 to 4, characterized in that: The process of step S2 includes: Step S201: Reading a 3D object detection Pytorch model based on image and point cloud pre-fusion data; Step S202: Setting a custom operator OP for the functional functions in the model that cannot be directly converted to ONNX; Step S203: corresponding symbolic functions Symbolic are defined for the above functional functions respectively, the parameters of the symbolic functions are consistent with the original functions, and the return value of the symbolic function is the set operator OP; Step S204: Call the Pytorch interface to complete the binding of the symbolic function and the original function. In the process of converting to the ONNX model, the symbolic function will be called instead of the original function to generate a model calculation graph in the form of setting OP; Step S205: Read the converted ONNX model and related parameters; Step S206: For the operator OP, write the corresponding custom plug-in function package in TensorRT, and add or rewrite the method attributes under the Custom class according to the functional requirements; Step S207: Add a custom plug-in interface to the plug-in initialization source file, and complete the plug-in registration in the Cmake file to realize the correspondence between the ONNX custom operator OP and the custom plug-in; Step S208: Compile the custom plug-in function package, generate the corresponding dynamic link library, and import it into the lib folder under the TensorRT installation directory; Step S209: Parse the ONNX model through the TensorRT interface, obtain the relationship between network layers, and generate a TensorRT model for the vehicle-mounted target detection embedded platform; Step S210: Parse the model for the first time. After building the engine, the model needs to be serialized and the engine needs to be saved locally. The local engine is directly read for deserialization. Step S211: input corresponding multimodal data to obtain accelerated inference results.

7. The 3D object detection acceleration method for embedded computing platforms according to any one of claims 1 to 4, characterized in that: The process of step S3 includes: Step S301: input data into the TensorRT model and output the target prediction box position information, category information, and confidence level; Step S302: Calculate the image pixel position corresponding to the point cloud under the CUDA architecture according to the extrinsic matrix of the camera and the laser radar and the camera matrix.

8. The 3D object detection acceleration method for embedded computing platforms according to any one of claims 1 to 4, characterized in that: The process of step S4 includes: Step S401: projecting the 3D target point cloud to a 2D image under the CUDA architecture to achieve fusion of the target point cloud and the corresponding image data; Step S402: finally generating a 3D fusion detection result in the two-dimensional image, including the category, location information and confidence level of the target.

9. The 3D object detection acceleration method for embedded computing platforms according to any one of claims 1 to 4, characterized in that: In step S4, the three-dimensional fusion detection result includes category information and location information of the target.

10. A storage medium, which can be read by a computer or a processor, characterized in that: The storage medium stores a computer program for executing any one of the methods in claims 1-9.

Images