Target detection method, electronic device, medium and vehicle for autonomous driving
By voxelizing and feature extraction of the original point cloud data, combining sparse convolution networks and top view convolution networks, the problem of lidar being difficult to detect vulnerable road users is solved, and efficient detection of vehicles and vulnerable road users is achieved.
Patent Information
- Application Number
- CN202210556974.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-05-19
AI Technical Summary
In the prior art, target detection based on lidar is difficult to accurately detect targets of vulnerable road users (such as pedestrians and bicycles), resulting in the detection function of VRU targets in autonomous driving scenarios.
By voxelizing the original point cloud data, three-dimensional voxel features are obtained, and the sparse convolution network is input for feature extraction, the first and second sparse voxel features are obtained, and the downsampling process is performed respectively, combining the top view convolution network and tight The module performs feature extraction and fusion for detection of vehicles and disadvantaged road users.
It improves the detection effect of vulnerable road users, while ensuring the accuracy of vehicle detection and enhancing the detection ability of VRU targets.
Smart Images

Figure CN114943950B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target detection technology, and specifically provides a target detection method, electronic equipment, medium and vehicle for autonomous driving. Background Art
[0002] With the continuous development of advanced driver assistance functions, advanced driver assistance has also attracted more and more attention. At the same time, with the continuous improvement of the performance of sensors used on the vehicle side and the continuous advancement of information technology, the experience of driver assistance technology has also been further improved. With the advancement of technology, the requirements for target recognition functions in the driver assistance process are also constantly increasing. In the existing technology, due to the physical characteristics of perception sensors such as lidar, the detection of larger targets (such as vehicles, etc.) in the target detection technology based on lidar is more accurate, while for VRU (Vulnerable Road User) targets (such as pedestrians, bicycles, etc.), due to the relatively small size of VRU targets, VRU targets are more difficult to detect, and the detection function for VRU targets needs to be improved.
[0003] Accordingly, the art needs a new target detection model solution to solve the above problems. Summary of the invention
[0004] In order to overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least partially solve the problem of how to effectively detect VRU targets in an autonomous driving scenario.
[0005] In a first aspect, the present invention provides a target detection method for autonomous driving, comprising:
[0006] Voxelize the original point cloud data to obtain three-dimensional voxel features;
[0007] Inputting the three-dimensional voxel feature into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by M times and a second sparse voxel feature downsampled by N times, where M and N are natural numbers and M>N;
[0008] Inputting the first sparse voxel feature into a top view convolutional network to extract features to obtain features on the top view;
[0009] Inputting the features on the top view into a vehicle detection head to obtain vehicle information;
[0010] Inputting the second sparse voxel feature into a compacting module to perform compacting processing to obtain a compact feature;
[0011] Input the features on the top view into an upsampling module for upsampling to obtain upsampled features; fuse the compact features and the upsampled features to obtain fused features;
[0012] The fused features are input into a vulnerable road user detection head to obtain vulnerable road user information.
[0013] In a technical solution of the above-mentioned object detection method for autonomous driving, the step of inputting the second sparse voxel feature into a compacting module to perform compacting processing to obtain a compact feature includes:
[0014] The second sparse voxel feature is maximized in the depth dimension.
[0015] In a technical solution of the above-mentioned object detection method for autonomous driving, taking the maximum value of the second sparse voxel feature in the depth dimension includes:
[0016] Use ReduceMax operation on the second sparse voxel feature of (H, W, D, C) dimension to get the maximum value in D dimension to obtain a compact feature of (H, W, C) dimension; or
[0017] Use the Maxpooling operation on the second sparse voxel feature of the (H, W, D, C) dimension to take the maximum value in the D dimension to obtain a dense feature of the (H, W, C) dimension; or
[0018] The second sparse voxel features of (H, W, D, C) dimensions are stacked in the D dimension to obtain dense features of (H, W, C) dimensions.
[0019] Wherein, H and W represent the grid length and width of the top view convolutional network respectively, D represents the depth of the second voxel feature, and C represents the number of channels of the second voxel feature.
[0020] In a technical solution of the above-mentioned target detection method for autonomous driving, the step of inputting the three-dimensional voxel feature into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by M times and a second sparse voxel feature downsampled by N times includes:
[0021] The three-dimensional voxel features are input into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by 8 times and a second sparse voxel feature downsampled by 4 times.
[0022] In a technical solution of the above-mentioned target detection method for autonomous driving, the inputting the features on the top view into the upsampling module for upsampling to obtain upsampled features includes:
[0023] The features on the top view are input into the upsampling module for 2-fold upsampling to obtain upsampled features.
[0024] In a technical solution of the above-mentioned object detection method for autonomous driving, the sparse convolutional network, the top view convolutional network and the vehicle detection head constitute a subject detection network;
[0025] The compaction module, the upsampling module and the vulnerable road user detection head constitute a branch detection network.
[0026] The method further includes: using a training set to perform overall training on the main detection network and the branch detection network to obtain network parameters of the main detection network and network parameters of the branch detection network, respectively, wherein the training set includes training data containing vehicles and data containing vulnerable road users;
[0027] The branch detection network is trained separately using the data containing vulnerable road users in the training set.
[0028] In a technical solution of the above-mentioned target detection method for autonomous driving, the separately training of the branch detection network using the data of vulnerable road users in the training set includes:
[0029] Initializing the network parameters of the branch detection network obtained in the overall training;
[0030] Keeping the network parameters of the subject detection network obtained in the overall training unchanged;
[0031] The branch detection network is trained using data containing vulnerable road users.
[0032] In a second aspect, an electronic device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the autonomous driving target detection method described in any one of the technical solutions of the above-mentioned autonomous driving target detection method.
[0033] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the target detection method for autonomous driving described in any one of the technical solutions of the above-mentioned target detection method for autonomous driving.
[0034] In a fourth aspect, the present invention provides a vehicle comprising:
[0035] Vehicle-mounted laser radar, used to obtain raw point cloud data;
[0036] An electronic device according to the above electronic device technical solution.
[0037] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:
[0038] In the technical solution of the present invention, the three-dimensional voxel features obtained from the original point cloud data can be input into the sparse convolutional network, and the first sparse voxel features and the second sparse voxel features can be obtained by applying the sparse convolutional network for feature extraction, and the first sparse voxel features are input into the top view convolutional network for feature extraction to obtain the features on the top view. The vehicle detection head can obtain vehicle information based on the features on the top view, and the second sparse voxel features are compacted to obtain compact features, and the features on the top view are upsampled to obtain upsampled features, and the compact features and upsampled features are feature fused to obtain fused features. The vulnerable user detection head can obtain vulnerable road user information based on the fused features. Through the above configuration, the present invention can set detection heads for vehicles and vulnerable road users respectively according to their respective characteristics, and detect vehicle information and vulnerable road user information separately. While ensuring the vehicle detection effect, it can achieve better detection of vulnerable road user information, thereby improving the detection effect of vulnerable road users. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. Among them:
[0040] Figure 1 is a schematic flow chart of main steps of a target detection method for autonomous driving according to an embodiment of the present invention;
[0041] Figure 2 It is a block diagram of the main network structure components of a main detection network and a branch detection network according to an implementation method of an embodiment of the present invention;
[0042] Figure 3 It is a flowchart of the main steps of training a main detection network and a branch detection network according to an implementation of an embodiment of the present invention. DETAILED DESCRIPTION
[0043] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.
[0044] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.
[0045] In the field of autonomous driving, the target is generally detected by collecting the original point cloud data of the target through a laser radar, and extracting the target features by applying the original point cloud data through a neural network model. However, due to the physical characteristics of the laser radar, the laser radar generally collects and obtains sparse point cloud data of the target, and the sparse point cloud data will have uneven density. If a target detection model is used to obtain the features of the sparse point cloud data by iterating and clustering the sparse point cloud data obtained by the laser radar, multiple iterations and clustering operations will be required, and the amount of calculation will be large. In addition, due to the uneven density of the sparse point cloud data, the final performance of the target detection model will also be reduced. Therefore, the detection effect of larger targets such as vehicles through the neural network model is better, while VRU (Vulnerable Road User) targets are more difficult to detect. This will have a greater impact on the autonomous driving function. This field requires a new target detection method for autonomous driving to solve the above problems.
[0046] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a target detection method for autonomous driving according to an embodiment of the present invention. Figure 1 As shown, the target detection method for autonomous driving in the embodiment of the present invention mainly includes the following steps S101 to S107.
[0047] Step S101: voxelize the original point cloud data to obtain three-dimensional voxel features.
[0048] In this embodiment, the original point cloud data may be voxelized to obtain three-dimensional voxel features. Voxelization is to convert disordered original point cloud data into regular data that is easy to perform convolution operations.
[0049] In one implementation, the raw point cloud data may be acquired by a vehicle-mounted laser radar installed on the vehicle.
[0050] In one embodiment, the original point cloud data can be voxelized to obtain three-dimensional voxel features through the following steps: using a hash table-based method, first formulate a tensor of a certain size and initialize it to 0, set the size of the input voxel, then traverse the original point cloud data, and calculate which voxel the original point cloud data belongs to, while recording the coordinates of the voxel to which the original point cloud data belongs and the number of original point cloud data for each voxel, and finally obtain all voxels, the coordinates corresponding to the voxels, and the largest original point cloud data contained in each voxel, and use the average coordinates and number of channels of the point cloud in each voxel as the three-dimensional voxel feature.
[0051] Step S102: input the three-dimensional voxel feature into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by M times and a second sparse voxel feature downsampled by N times, where M and N are natural numbers and M>N.
[0052] In this embodiment, the three-dimensional voxel feature can be input into a sparse convolutional network (Sparse ConvNetwork) for high-dimensional sparse feature extraction to obtain high-dimensional sparse voxel features of different scales. Specifically, the first sparse voxel feature can be obtained by M-fold downsampling, and the second sparse voxel feature can be obtained by N-fold downsampling.
[0053] The sparse convolution network is a 3D sparse convolution network. The advantage of the 3D sparse convolution network is that it not only considers the 3D spatial information of the 3D voxel features, but also utilizes the sparsity of the 3D voxel features, so that the computational complexity of the feature extraction process is smaller. Those skilled in the art can set the values of M and N according to the needs of the actual application.
[0054] In one implementation, two convolution modules may be respectively set in the sparse convolutional network, wherein one convolution module implements M-fold downsampling of the three-dimensional voxel feature to obtain a first sparse voxel feature; and the other convolution module implements N-fold downsampling of the three-dimensional voxel feature to obtain a second sparse voxel feature.
[0055] In one implementation, the three-dimensional voxel feature can be input into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by 8 times and a second sparse voxel feature downsampled by 4 times, that is, M=8, N=4. In this case, the second sparse voxel feature has a higher resolution than the first sparse voxel feature.
[0056] Step S103: Input the first sparse voxel feature into the top view convolutional network to extract features to obtain features on the top view.
[0057] In this embodiment, the first sparse voxel feature can be input into the bird's eye view (BEV) convolutional network to perform feature extraction to obtain features on the top view. Generally speaking, when detecting a target, a convolutional neural network is often used to extract target features, and in a convolutional neural network, the low-level network generally has a small receptive field, and the resolution of extracted target features is high, which is suitable for detecting small target objects; while the high-level network's receptive field will gradually increase with the increase in downsampling or convolution times, but the resolution of extracted target features is low, which is more suitable for detecting large target objects. Therefore, in this embodiment, the first sparse voxel feature with a higher downsampling multiple is sent to the top view convolutional network to obtain features on the top view, thereby realizing target detection of large targets such as vehicles.
[0058] Step S104: Input the features on the top view into the vehicle detection head to obtain vehicle information.
[0059] In this embodiment, the features on the top view may be input into the vehicle detection head to obtain vehicle information.
[0060] In one implementation, the vehicle detection head may perform feature recognition on features on the top view through a region proposal network (RPN) to obtain vehicle information.
[0061] Step S105: The second sparse voxel feature is input into a compaction module for compaction processing to obtain a compact feature.
[0062] In this embodiment, since the second sparse voxel feature is compacted, the obtained compact feature is a dense feature relative to the second sparse voxel feature.
[0063] In one embodiment, the dimension of the second voxel feature can be expressed as (H, W, D, C), where H represents the network length of the top view convolutional network, W represents the network width of the top view convolutional network, D represents the depth of the second voxel feature, and C is the number of channels of the second voxel feature.
[0064] In one implementation, the maximum value of the second sparse voxel feature of the (H, W, D, C) dimension may be taken in the D dimension to obtain a compact feature.
[0065] For example, the second sparse voxel feature of (H, W, D, C) dimensions can be used to obtain the maximum value in D dimension using ReduceMax operation to obtain a dense feature of (H, W, C) dimensions. Here, ReduceMax refers to a function that obtains the maximum value in a specified dimension.
[0066] For another example, the second sparse voxel feature of (H, W, D, C) dimensions can be used to obtain the maximum value in D dimension by using the Maxpooling operation to obtain a dense feature of (H, W, C) dimensions. Maxpooling refers to the operation of dividing the feature into several small blocks of the same size (pooling size), taking only the maximum value of the specified dimension in each small block, and discarding other values.
[0067] For another example, the second sparse voxel features of (H, W, D, C) dimensions can be stacked in the D dimension to obtain a dense feature of (H, W, C) dimensions. That is, a fixed-size anchor point can be used in the D dimension, and the second sparse voxel features can be stacked at the anchor point to obtain a dense feature.
[0068] Step S106: input the features on the top view into the upsampling module for upsampling to obtain upsampled features; fuse the compact features and the upsampled features to obtain fused features.
[0069] In this embodiment, the features on the top view may be upsampled to obtain upsampled features, and the upsampled features may be fused with the compact features to obtain fused features. Since the fused features are obtained based on the upsampled features and the compact features, the fused features are high-resolution and contain multi-level features.
[0070] In one implementation, the features on the top view may be input into an upsampling module for 2-fold upsampling to obtain upsampled features.
[0071] In one implementation, an additive weighted average method may be applied to achieve feature fusion between upsampled features and compact features to obtain fused features.
[0072] In one implementation, a multiplication weighted average method may also be applied to achieve feature fusion between upsampled features and compact features to obtain fused features.
[0073] In one implementation, feature fusion between upsampled features and compact features may also be achieved through a concat operation to obtain fused features.
[0074] Step S107: input the fused features into the vulnerable road user detection head to obtain vulnerable road user information.
[0075] In this embodiment, the fused features obtained in step S106 are input into the vulnerable road user detection head, so that the vulnerable road users can be effectively detected. That is, since the fused features are high-resolution features and contain multi-level features, inputting the fused features into the vulnerable road user detection head can better detect vulnerable road users effectively. In one embodiment, vulnerable road users may include pedestrians, bicycles, and small objects in the distance in an autonomous driving environment.
[0076] Based on the above steps S101 to S107, the embodiment of the present invention can input the three-dimensional voxel features obtained according to the original point cloud data into the sparse convolutional network, and use the sparse convolutional network to perform feature extraction to obtain the first sparse voxel feature and the second sparse voxel feature respectively, and input the first sparse voxel feature into the top view convolutional network for feature extraction to obtain the features on the top view, and the vehicle detection head can obtain vehicle information according to the features on the top view, and compact the second sparse voxel feature to obtain a compact feature, and upsample the features on the top view to obtain an upsampled feature, and fuse the compact feature and the upsampled feature to obtain a fused feature, and the vulnerable user detection head can obtain vulnerable road user information according to the fused feature. Through the above configuration, the embodiment of the present invention can set detection heads for vehicles and vulnerable road users according to their respective characteristics, and detect vehicle information and vulnerable road user information separately, and while ensuring the vehicle detection effect, it can achieve better detection of vulnerable road user information, thereby improving the detection effect of vulnerable road users.
[0077] In one implementation of the present invention, see the attached Figure 2 and attached Figure 3 , Figure 2 It is a block diagram of the main network structure components of a main detection network and a branch detection network according to an implementation method of an embodiment of the present invention; Figure 3 FIG. 1 is a flow chart of the main steps of training the main detection network and the branch detection network according to an implementation of an embodiment of the present invention. Figure 2 As shown, in this embodiment, the sparse convolution network, the top view convolution network and the vehicle detection head can constitute a main body detection network; the compaction module, the upsampling module and the VRU detection head can constitute a branch detection network. Among them, the input of the sparse convolution network is the grid feature (three-dimensional voxel feature) obtained after the original three-dimensional point cloud is gridded (voxelized). The compact features obtained by the compaction module and the upsampling features obtained by the upsampling module are fused by the addition weighted average method. Figure 3 As shown, in addition to the above steps S101 to S107, the present invention may also include the following steps S108 and S109:
[0078] Step S108: using the training set to perform overall training on the main detection network and the branch detection network to obtain network parameters of the main detection network and network parameters of the branch detection network, respectively, wherein the training set includes training data containing vehicles and data containing vulnerable road users;
[0079] In this embodiment, the main detection network and the branch detection network can be trained as a whole using training data including vehicles and training sets including data of vulnerable road users to obtain network parameters of the main detection network and network parameters of the branch detection network.
[0080] Step S109: using the data of vulnerable road users in the training set to perform separate training on the branch detection network.
[0081] In this embodiment, after the overall training is completed, the data containing the vulnerable road users in the training set can be extracted to train the branch detection network separately. That is, using a two-stage training method, the main detection network and the branch detection network are first trained as a whole, and then in the fine-tuning stage, the branch detection network is trained separately using the data containing the vulnerable road users in the training set.
[0082] In one implementation, step S109 may further include:
[0083] Step S1091: Initializing the network parameters of the branch detection network obtained in the overall training;
[0084] Step S1092: keep the network parameters of the subject detection network obtained in the overall training unchanged;
[0085] Step S1093: training the branch detection network using data containing vulnerable road users.
[0086] In this embodiment, after the overall training of the main detection network and the branch detection network is completed, the network parameters of the main detection network can be kept unchanged, the network parameters of the branch detection network can be initialized, and the branch network can be trained separately using data containing vulnerable road users to obtain a trained branch network. In this way, without affecting the detection effect of the main detection network on vehicles, the branch detection network can be further effectively trained to detect vulnerable road users, so that the detection network composed of the main detection network and the branch detection network can have a better detection effect on vehicles and vulnerable road users.
[0087] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.
[0088] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.
[0089] Furthermore, the present invention also provides an electronic device. In an electronic device embodiment according to the present invention, the electronic device includes a processor and a storage device, the storage device can be configured to store a program for executing the target detection method of the autonomous driving of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the target detection method of the autonomous driving of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention.
[0090] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the target detection method for autonomous driving of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned target detection method for autonomous driving. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.
[0091] Furthermore, the present invention also provides a vehicle. In a vehicle embodiment according to the present invention, the vehicle may include a vehicle-mounted laser radar and the electronic device in the above electronic device embodiment. The vehicle-mounted laser radar is used to obtain raw point cloud data.
[0092] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.
[0093] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.
[0094] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A target detection method for autonomous driving, characterized in that: include: Voxelize the original point cloud data to obtain three-dimensional voxel features; Inputting the three-dimensional voxel feature into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by M times and a second sparse voxel feature downsampled by N times, where M and N are natural numbers and M>N; Inputting the first sparse voxel feature into a top view convolutional network to extract features to obtain features on the top view; Inputting the features on the top view into a vehicle detection head to obtain vehicle information; Inputting the second sparse voxel feature into a compacting module to perform compacting processing to obtain a compact feature; Input the features on the top view into an upsampling module for upsampling to obtain upsampled features; fuse the compact features and the upsampled features to obtain fused features; The fused features are input into a vulnerable road user detection head to obtain vulnerable road user information.
2. The method according to claim 1, characterized in that: The step of inputting the second sparse voxel feature into a compacting module to perform compacting processing to obtain a compact feature includes: The second sparse voxel feature is maximized in the depth dimension.
3. The method according to claim 2, characterized in that The step of maximizing the second sparse voxel feature in the depth dimension includes: Use ReduceMax operation on the second sparse voxel feature of (H, W, D, C) dimension to get the maximum value in D dimension to obtain a compact feature of (H, W, C) dimension; or Use the Maxpooling operation on the second sparse voxel feature of the (H, W, D, C) dimension to take the maximum value in the D dimension to obtain a dense feature of the (H, W, C) dimension; or The second sparse voxel features of (H, W, D, C) dimensions are stacked in the D dimension to obtain dense features of (H, W, C) dimensions. Wherein, H and W represent the grid length and width of the top view convolutional network respectively, D represents the depth of the second sparse voxel feature, and C represents the number of channels of the second sparse voxel feature.
4. The method according to claim 1, characterized in that: The step of inputting the three-dimensional voxel feature into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by M times and a second sparse voxel feature downsampled by N times includes: The three-dimensional voxel features are input into a sparse convolutional network for feature extraction to obtain a first sparse voxel feature downsampled by 8 times and a second sparse voxel feature downsampled by 4 times.
5. The method according to claim 1, characterized in that The inputting the features on the top view into the upsampling module for upsampling to obtain upsampling features includes: The features on the top view are input into the upsampling module for 2-fold upsampling to obtain upsampled features.
6. The method according to any one of claims 1 to 5, characterized in that The sparse convolutional network, the top view convolutional network and the vehicle detection head constitute a subject detection network; The compaction module, the upsampling module and the vulnerable road user detection head constitute a branch detection network. The method further includes: using a training set to perform overall training on the main detection network and the branch detection network to obtain network parameters of the main detection network and network parameters of the branch detection network, respectively, wherein the training set includes training data containing vehicles and data containing vulnerable road users; The branch detection network is trained separately using the data containing vulnerable road users in the training set.
7. The method according to claim 6, characterized in that The separately training the branch detection network using the data of vulnerable road users in the training set includes: Initializing the network parameters of the branch detection network obtained in the overall training; Keeping the network parameters of the subject detection network obtained in the overall training unchanged; The branch detection network is trained using data containing vulnerable road users.
8. An electronic device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by the processor to perform the method of any one of claims 1 to 7.
9. A storage medium storing a plurality of program codes, characterized in that: When the program code is executed, the method according to any one of claims 1 to 7 is implemented.
10. A vehicle, characterized in that: include: Vehicle-mounted laser radar, used to obtain raw point cloud data; The electronic device according to claim 8.
Citation Information
Patent Citations
Three-dimensional target detection method and device, computer equipment and storage medium
CN111199206A
Data fusion and target detection method, device and equipment
CN111860695A
Multi-sensor deep fusion 3D target detection method for automatic driving
CN113159151A
3D point cloud target detection method based on graph convolution
CN113705631A
Point cloud 3D target detection method based on key point multi-scale feature fusion
CN113706480A
