Method, apparatus, device and storage medium for target detection

Abstract

The application discloses a target detection method, device, equipment and storage medium, and belongs to the field of computer technology. The method comprises the following steps: acquiring image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is a multi-view vehicle driving environment image; feature extraction processing is respectively performed on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used for representing a target obstacle; feature fusion processing is performed on the voxel query feature information and the image feature information to obtain fusion feature information; and target detection processing is performed on the fusion feature information to obtain a target detection result. The application reduces the calculation amount and time consumption of target detection processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to the technical field of intelligent transportation and automatic driving, and specifically relates to a target detection method and device, equipment and storage medium. BACKGROUND

[0002] In the field of automatic driving, commonly used sensors include cameras and laser radars. Multiple cameras can capture image information around the vehicle body. Laser radars measure the distance of surrounding objects by emitting and receiving electromagnetic waves. The data obtained by laser radars is usually point cloud data, each point recording the position coordinates of the object and reflectivity information.

[0003] At present, the target detection scheme in the related art can splice all image features and all point cloud features in a bird's eye view, and then detect the target of interest based on the spliced features. However, the scheme in the related art has a large amount of calculation and high data processing time. SUMMARY

[0004] The present application provides a target detection method, device, equipment and storage medium, which solves the problem of large amount of calculation and high time consumption in detection processing. The technical solution is as follows:

[0005] In a first aspect, a target detection method is provided, which includes:

[0006] Obtaining image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is multi-view vehicle driving environment image;

[0007] Respectively performing feature extraction processing on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for representing target obstacles;

[0008] Performing feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information;

[0009] Performing target detection processing on the fusion feature information to obtain a target detection result.

[0010] In a possible implementation, performing feature extraction processing on the image data to obtain image feature information includes:

[0011] Using a preset image feature extraction network of a target detection model, performing down-sampling feature extraction processing on the image data according to preset sampling parameters to obtain a feature extraction processing result;

[0012] Based on the result of the feature extraction processing, the image feature information is obtained.

[0013] In a possible implementation, the point cloud data is subjected to feature extraction processing to obtain voxel query feature information, including:

[0014] Based on the point cloud data, the voxel point cloud data is obtained according to a preset voxel parameter;

[0015] The voxel point cloud data is subjected to feature extraction processing by using a voxel feature extraction network of a preset target detection model, to obtain a result of the feature extraction;

[0016] Based on the result of the feature extraction, the voxel query feature information is obtained.

[0017] In a possible implementation, the voxel query feature information is obtained based on the result of the feature extraction, including:

[0018] The result of the feature extraction is subjected to pruning processing to obtain a result of the feature extraction after pruning processing;

[0019] The voxel query feature information is obtained based on the result of the feature extraction after pruning processing.

[0020] In a possible implementation, the feature fusion processing is performed on the voxel query feature information and the image feature information to obtain fusion feature information, including:

[0021] The image feature information is subjected to screening processing based on the voxel query feature information by using a preset position calibration relationship, to obtain screened image feature information;

[0022] The voxel query feature information and the screened image feature information are subjected to feature fusion processing by using a feature fusion network of a preset target detection model, to obtain fusion feature information.

[0023] In a possible implementation, the target detection processing is performed on the fusion feature information to obtain a result of the target detection, including:

[0024] The target detection processing is performed on the fusion feature information by using a target detection network of a preset target detection model, to obtain a result of the target detection, and wherein

[0025] The result of the target detection includes at least one of class information of a target detection object, position coordinates of a detection box of the target detection object, and size information of the detection box of the target detection object.

[0026] In a second aspect, a model training method is provided, including:

[0027] obtain sample image data and sample point cloud data corresponding to the sample image data; the sample image data is a multi-view vehicle driving environment image;

[0028] respectively perform feature extraction processing on the sample image data and the sample point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle;

[0029] perform feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information;

[0030] perform target detection processing on the fusion feature information to obtain a target detection result;

[0031] update and train a target detection model to be trained based on the target detection result, the sample image data, and the sample image data to obtain a target detection model that has completed training.

[0032] In a third aspect, a device for target detection is provided, and the device includes:

[0033] an acquisition unit configured to acquire image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is a multi-view vehicle driving environment image;

[0034] an extraction unit configured to respectively perform feature extraction processing on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle;

[0035] a fusion unit configured to perform feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information;

[0036] a detection unit configured to perform target detection processing on the fusion feature information to obtain a target detection result.

[0037] In a fourth aspect, a device for model training is provided, and the device includes:

[0038] an acquisition unit configured to acquire sample image data and sample point cloud data corresponding to the sample image data; the sample image data is a multi-view vehicle driving environment image;

[0039] An extraction unit is configured to perform feature extraction on the sample image data and the sample point cloud data respectively to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle;

[0040] A fusion unit is configured to perform feature fusion on the voxel query feature information and the image feature information to obtain fusion feature information.

[0041] A detection unit is configured to perform target detection on the fusion feature information to obtain a target detection result.

[0042] A training unit is configured to perform update training on a target detection model to be trained based on the target detection result, the sample image data and the sample point cloud data, to obtain a target detection model completed with training.

[0043] In a fifth aspect, a computer-readable storage medium is provided, and the storage medium stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the method of the above aspects and any possible implementation manner.

[0044] In a sixth aspect, an electronic device is provided, and the electronic device comprises:

[0045] at least one processor; and

[0046] a memory connected with the at least one processor in communication; wherein

[0047] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the above aspects and any possible implementation manner.

[0048] In a seventh aspect, an autonomous vehicle is provided, and the autonomous vehicle comprises the electronic device as described above.

[0049] The technical scheme provided in the present application has at least the following beneficial effects:

[0050] According to the technical solution, on the one hand, the embodiment of the present application can obtain image data to be processed and point cloud data to be processed corresponding to the image data, the image data to be processed is a multi-view vehicle driving environment image, and then the image data and the point cloud data can be respectively subjected to feature extraction processing to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for representing a target obstacle, the voxel query feature information and the image feature information are subjected to feature fusion processing to obtain fusion feature information, the fusion feature information is subjected to target detection processing to obtain a target detection result, and since the image and the point cloud do not need to be converted into a bird's eye view, only the voxel feature information for representing the target obstacle and the image feature information are fused to perform target detection processing on the fused feature information to obtain the target detection result, the calculation amount of the target detection processing can be effectively reduced while ensuring the reliability of the target detection result, time consumption is reduced, and the reliability and efficiency of the target detection processing are ensured.

[0051] According to the technical solution, on the other hand, the embodiment of the present application can obtain sample image data and sample point cloud data corresponding to the sample image data, and then the sample image data and the sample point cloud data can be respectively subjected to feature extraction processing to obtain image feature information and voxel query feature information, the voxel query feature information is voxel feature information for representing a target obstacle, the voxel query feature information and the image feature information are subjected to feature fusion processing to obtain fusion feature information, the fusion feature information is subjected to target detection processing to obtain a target detection result, and based on the target detection result, the sample image data and the sample image data, a target detection model to be trained can be updated and trained to obtain a target detection model completed training, since the target detection model to be trained can be updated and trained based on the sample image data and the sample image data, and the fusion feature of the image feature information and the voxel feature information for representing the target obstacle, a target detection model with low latency and high detection accuracy can be obtained, and the detection performance and efficiency of the target detection model are improved.

[0052] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the embodiments in the present application shall fall within the scope of protection of the present application.

[0054] Figure 1 is a flow diagram of a target detection method provided by an embodiment of the present application;

[0055] Figure 2 is a flow diagram of a model training method provided by another embodiment of the present application;

[0056] Figure 3 is a schematic diagram of a target detection method provided by another embodiment of the present application;

[0057] Figure 4 is a schematic diagram of a target detection model in an application scenario of a target detection method provided by an embodiment of the present application;

[0058] Figure 5 is a structural block diagram of a target detection device provided by another embodiment of the present application;

[0059] Figure 6 is a structural block diagram of a model training device provided by another embodiment of the present application.

[0060] Figure 7 is a block diagram of an electronic device for implementing the target detection method and the model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present application will be described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to help in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0062] Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort shall fall within the scope of protection of the present application.

[0063] It should be noted that the terminal device involved in the embodiments of the present application can include, but is not limited to, a mobile phone, a personal digital assistant (PDA), a wireless handheld device, a tablet computer, and the like. The display device can include, but is not limited to, a personal computer, a television, and the like.

[0064] In addition, the term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the front and rear associated objects.

[0065] Please refer to Figure 1 which shows a flowchart of a target detection method provided by an embodiment of the present application. The target detection method can specifically include:

[0066] Step 101, obtaining image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is multi-view vehicle driving environment image.

[0067] Step 102, respectively performing feature extraction processing on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for characterizing a target obstacle.

[0068] Step 103, performing feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information.

[0069] Step 104, performing target detection processing on the fusion feature information to obtain a target detection result.

[0070] It should be noted that first, multi-view image data can be collected by a plurality of cameras on the vehicle, and point cloud data can be collected by a vehicle-mounted laser radar. Second, a time compensation algorithm can be used to align the multi-view image data and the point cloud data. Finally, based on the alignment result, the image data to be processed and the point cloud data to be processed corresponding to the image data are determined.

[0071] It should be noted that the target obstacle can be an object to be detected, i.e., a target object of interest. For example, in the scenario of an autonomous vehicle driving, the target obstacle can be other road users on the road, such as other motor vehicles, non-motor vehicles, pedestrians, and other objects driving on the road.

[0072] It should be noted that part or all of the execution subjects of steps 101-104 can be an application located at the local terminal, or can also be a functional unit such as a plug-in or a software development kit (Software Development Kit, SDK) arranged in the application located at the local terminal, or can also be a processing engine located in a server on the network side, or can also be a distributed system on the network side, for example, a processing engine or a distributed system in a target detection platform on the network side, and the present embodiment does not make special limitations thereon.

[0073] It can be understood that the application can be a native program (nativeApp) installed on the local terminal, or can also be a web program (webApp) of a browser on the local terminal, and the present embodiment does not make limitations thereon.

[0074] In this way, the image data to be processed and the point cloud data to be processed corresponding to the image data can be obtained, the image data to be processed is a multi-view vehicle driving environment image, and then the image data and the point cloud data can be subjected to feature extraction processing respectively to obtain image feature information and voxel query feature information, the voxel query feature information is voxel feature information for representing a target obstacle, the voxel query feature information and the image feature information are subjected to feature fusion processing to obtain fusion feature information, and the fusion feature information is subjected to target detection processing to obtain a target detection result. Since the image and the point cloud do not need to be converted into a bird's eye view, only the voxel feature information for representing a target obstacle and the image feature information are fused to perform target detection processing on the fused feature information to obtain a target detection result, the calculation amount of the target detection processing can be effectively reduced while ensuring the reliability of the target detection result, the time consumption is reduced, and the reliability and efficiency of the target detection processing are ensured.

[0075] Optionally, in one possible implementation manner of the present embodiment, in step 102, the image feature extraction network of the preset target detection model can be used to perform down-sampling feature extraction processing on the image data according to preset sampling parameters, to obtain a result of the feature extraction processing, and the image feature information can be obtained based on the result of the feature extraction processing.

[0076] In the present implementation manner, the image feature extraction network can include but is not limited to a backbone network such as ResNet50. The preset sampling parameters can be a preset sampling multiple.

[0077] In this way, the image feature extraction network of the preset target detection model can be used to directly extract features of the multi-angle images, more effective feature information of the multi-angle images can be obtained, and comprehensiveness and reliability of the image feature information are improved.

[0078] Optionally, in a possible implementation of the embodiment, in step 102, the voxel point cloud data can be obtained based on the point cloud data and according to preset voxel parameters, and then the voxel feature extraction network of the preset target detection model can be used to perform feature extraction on the voxel point cloud data to obtain a feature extraction result, so that the voxel query feature information can be obtained based on the feature extraction result.

[0079] In the implementation, the voxel feature extraction network can include, but is not limited to, a convolutional neural network such as a sparse convolutional neural network (SCNN). The preset voxel parameters can include a voxel size.

[0080] In a specific implementation process of the implementation, the point cloud data is voxelized according to the preset voxel size to obtain voxelized point cloud data, i.e., voxel point cloud data.

[0081] In another specific implementation process of the implementation, the feature extraction result can be pruned to obtain a pruned feature extraction result, and then the voxel query feature information can be obtained based on the pruned feature extraction result.

[0082] In the specific implementation process, the feature extraction result can be voxel feature information, i.e., voxel feature information of all point clouds.

[0083] In a case of the specific implementation process, the feature extraction result can be classified by using a preset classifier to obtain background voxel feature information and target obstacle voxel feature information, then the background voxel feature information can be filtered, and the target obstacle voxel point cloud feature information can be used as the voxel query feature information.

[0084] It can be understood that the background voxel feature information can be voxel feature information of point clouds representing a background. The target obstacle voxel feature information can be voxel feature information of point clouds representing a target obstacle object. The target obstacle can be a target object of interest.

[0085] Exemplarily, the result of the feature extraction can be classified by using a multi layer perception (MLP) to obtain background voxel point cloud feature information and target obstacle voxel feature information, and then the background voxel feature information can be filtered, that is, the background voxel feature information is pruned, and the target obstacle voxel feature information is taken as voxel query feature information.

[0086] In one case of the specific implementation process, a preset clustering algorithm is used to cluster the result of the feature extraction to obtain target obstacle voxel feature information, and then the target obstacle voxel feature information can be taken as the result of the feature extraction after the pruning processing.

[0087] Exemplarily, the result of the feature extraction can be clustered by using a K-means algorithm to obtain target obstacle voxel feature information, and then the target obstacle voxel feature information can be taken as voxel query feature information.

[0088] It can be understood that other existing methods can also be used to obtain target obstacle voxel feature information, that is, voxel query feature information, from the result of the feature extraction, which is not specifically limited here.

[0089] In this way, by pruning the result of the feature extraction, background voxel feature information representing background point cloud and target obstacle voxel feature information representing a target of interest can be distinguished, and the target obstacle voxel feature information is retained and taken as voxel query feature information. The voxel query feature information can be used for target detection in the subsequent process, which reduces the data processing amount of target detection, and further reduces the computation and time consumption.

[0090] It should be noted that the specific implementation process provided in the present implementation mode can be combined with the various specific implementation processes provided in the foregoing implementation modes to implement the target detection method of the present embodiment. For details, please refer to the related content in the foregoing implementation modes, which will not be repeated here.

[0091] Optionally, in one possible implementation mode of the present embodiment, in step 103, a preset position calibration relationship can be used to filter the image feature information based on the voxel query feature information to obtain filtered image feature information, and then a feature fusion network of a preset target detection model can be used to perform feature fusion processing on the voxel query feature information and the filtered image feature information to obtain fusion feature information.

[0092] In the implementation, the preset position calibration relationship can be a position relationship matrix of the vehicle-mounted laser radar and the camera. The position relationship matrix can represent a conversion relationship between coordinate systems of the camera and the laser radar. The position relationship matrix can be used for conversion between coordinate systems of the laser radar and the laser radar.

[0093] In one specific implementation process of the implementation, first, the preset position calibration relationship can be used to perform coordinate system conversion processing on the voxel query feature information, to obtain voxel query feature information in an image coordinate system. Second, the voxel query feature information in the image coordinate system and the image feature information can be matched based on the point cloud two-dimensional position information corresponding to the voxel query feature information and the position information corresponding to the image feature information, to obtain a matching result. Then, the image feature information matched with the voxel query feature information can be selected from the image feature information based on the matching result, and the selected image feature information matched with the voxel query feature information can be taken as filtered image feature information. Then, the feature fusion network of the preset target detection model can be used to perform feature fusion processing on the voxel query feature information and the filtered image feature information, to obtain fusion feature information.

[0094] Here, the filtered image feature information can be the image feature information matched with the voxel query feature information.

[0095] In one specific implementation process of the implementation, first, the preset position calibration relationship can be used to perform coordinate system conversion processing on the voxel query feature information, to obtain voxel query feature information in an image coordinate system. Second, the voxel query feature information in the image coordinate system and the image feature information can be matched based on the point cloud two-dimensional position information corresponding to the voxel query feature information and the position information corresponding to the image feature information, to obtain a matching result. Then, the image feature information matched with the voxel query feature information can be selected from the image feature information based on the matching result, and the selected image feature information matched with the voxel query feature information can be taken as filtered image feature information. Then, the feature fusion network of the preset target detection model can be used to perform feature fusion processing on the voxel query feature information and the filtered image feature information, to obtain fusion feature information.

[0096] In another specific implementation process of the implementation, the feature fusion network can be used to perform feature splicing processing on the voxel query feature information and the filtered image feature information based on the point cloud two-dimensional position information corresponding to the voxel query feature information and the position information corresponding to the image feature information, to obtain fusion feature information.

[0097] In the specific implementation process, the feature fusion network can include a feature fusion network based on a preset splicing algorithm.

[0098] For example, the voxel query feature information and the filtered image feature information can be directly spliced to obtain the fusion feature information.

[0099] Alternatively, the sum of the point cloud two-dimensional position information corresponding to the voxel query feature information and the position information corresponding to the image feature information can be calculated, and the voxel query feature information and the filtered image feature information can be spliced based on the calculated result to obtain the fusion feature information.

[0100] Alternatively, the average value of the point cloud two-dimensional position information corresponding to the voxel query feature information and the position information corresponding to the image feature information can be calculated, and the voxel query feature information and the screened image feature information are spliced based on the calculated result to obtain the fusion feature information.

[0101] For example, the voxel query feature information is 256-dimensional feature information, and the screened image feature information is 256-dimensional feature information, and the fusion feature information can be 512-dimensional feature information.

[0102] In this way, by converting the voxel query feature information of the point cloud data to the image dimension, the image feature information matched with the point cloud data can be quickly obtained from the image feature information to obtain the fusion feature information, and the calculation amount and time consumption of feature fusion are further reduced.

[0103] It should be noted that the specific implementation process provided in the present implementation mode can be combined with the various specific implementation processes provided in the foregoing implementation modes to achieve the target detection method of the present embodiment. For detailed description, please refer to the related content in the foregoing implementation modes, which will not be repeated here.

[0104] Optionally, in one possible implementation mode of the present embodiment, in step 104, the target detection network of the preset target detection model can be used to perform target detection processing on the fusion feature information to obtain the target detection result.

[0105] In the present implementation mode, the target detection result can include but is not limited to the category information of the target detection object, the position coordinates of the detection frame of the target detection object, and the size information of the detection frame of the target detection object.

[0106] It can be understood that the target detection network can be a network based on an existing target detection algorithm, which will not be specifically limited here.

[0107] It should be noted that the specific implementation process provided in the present implementation mode can be combined with the various specific implementation processes provided in the foregoing implementation modes to achieve the target detection method of the present embodiment. For detailed description, please refer to the related content in the foregoing implementation modes, which will not be repeated here.

[0108] Figure 2 is a flowchart of the model training method provided by another embodiment of the present application, as shown in Figure 2 .

[0109] Step 201, acquiring sample image data and sample point cloud data corresponding to the sample image data; the sample image data is a multi-view vehicle driving environment image.

[0110] Step 202, respectively, feature extraction processing is performed on the sample image data and the sample point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle.

[0111] Step 203, feature fusion processing is performed on the voxel query feature information and the image feature information to obtain fusion feature information.

[0112] Step 204, target detection processing is performed on the fusion feature information to obtain a target detection result.

[0113] Step 205, based on the target detection result, the sample image data and the sample image data, a target detection model to be trained is updated and trained to obtain a target detection model completed training.

[0114] It should be noted that the target obstacle can be an object to be detected, i.e., a target object of interest. For example, in the scenario of an autonomous vehicle driving, the target obstacle can be other traffic participants on the road, such as other motor vehicles, non-motor vehicles, pedestrians, and other objects driving on the road.

[0115] It should be noted that part or all of the execution subject of steps 201-205 can be an application located in the local terminal, or can also be a plug-in or software development kit (Software Development Kit, SDK) functional unit arranged in the application located in the local terminal, or can also be a processing engine located in the network side server, or can also be a distributed system located in the network side, such as a processing engine or a distributed system in the model training platform of the network side, etc. The present embodiment does not make special limitations.

[0116] It can be understood that the application can be a native application (native App) installed on the local terminal, or can also be a web application (web App) of a browser on the local terminal, and the present embodiment does not make limitations.

[0117] In this way, the sample image data and the sample point cloud data corresponding to the sample image data can be acquired, and then feature extraction processing can be performed on the sample image data and the sample point cloud data respectively to obtain image feature information and voxel query feature information, the voxel query feature information being voxel feature information used for representing a target obstacle, feature fusion processing is performed on the voxel query feature information and the image feature information to obtain fusion feature information, target detection processing is performed on the fusion feature information to obtain a target detection result, and the target detection model to be trained can be updated and trained based on the target detection result, the sample image data and the sample point cloud data to obtain a target detection model that has completed training. Since the target detection model to be trained can be updated and trained based on the sample image data and the sample point cloud data and the fusion feature of the image feature information and the voxel feature information used for representing a target obstacle, a target detection model with low latency and high detection accuracy can be obtained, and therefore the detection performance and efficiency of the target detection model are improved.

[0118] Optionally, in one possible implementation manner of the embodiment, the target detection model can include an image feature extraction network, a voxel feature extraction network, a feature fusion network and a target detection network.

[0119] In this implementation manner, the image feature extraction network can include but is not limited to a backbone network such as ResNet50. The voxel feature extraction network can include but is not limited to a convolutional neural network such as a sparse convolutional neural network (SCNN). The feature fusion network can include a feature fusion network based on a preset splicing algorithm.

[0120] It can be understood that the target detection method in the foregoing embodiments can be performed by using the target detection model that has completed training. Specifically, the image data to be processed and the point cloud data to be processed corresponding to the image data can be input into the target detection model that has completed training, and processed by the image feature extraction network, the voxel feature extraction network, the feature fusion network and the target detection network respectively to output a corresponding target detection result.

[0121] In order to better understand the method of the embodiments of the present application, the method of the embodiments of the present application will be described below in combination with the drawings and specific application scenarios.

[0122] Figure 3 is a flowchart of a target detection method provided by another embodiment of the present application, as shown in Figure 3As shown. In this embodiment, in this application scenario, the preset target detection model can include an image feature extraction network, a voxel feature extraction network, a feature fusion network and a target detection network, and the target detection method can include:

[0123] Step 301, obtaining sample image data and sample point cloud data corresponding to the sample image data.

[0124] Step 302, respectively performing feature extraction processing on the sample image data and the sample point cloud data to obtain image feature information and voxel query feature information.

[0125] In this embodiment, the voxel query feature information is voxel feature information for characterizing the target obstacle.

[0126] Step 303, performing feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information.

[0127] Step 304, performing target detection processing on the fusion feature information to obtain a target detection result.

[0128] Step 305, based on the target detection result, the sample image data and the sample image data, updating and training the target detection model to be trained to obtain a target detection model completed training.

[0129] Step 306, obtaining image data to be processed and point cloud data to be processed corresponding to the image data.

[0130] In this embodiment, the image data to be processed is multi-view vehicle driving environment image.

[0131] Step 307, using the image feature extraction network of the preset target detection model, performing down-sampling feature extraction processing on the image data according to the preset sampling parameter to obtain image feature information.

[0132] In this embodiment, the image feature extraction network can be a ResNet50 network.

[0133] For example, for image data collected by 6 cameras, the preset sampling parameter is 8, the input image data is 6x1920x1080x3, and the input image data can be processed by ResNet50 by 8 times down-sampling to obtain the feature information of the 6 camera image data as 6x240x135x3.

[0134] Here, 6 can represent 6 cameras, 1920x1080 and 240x135 can represent the pixels of the image data, and 3 can represent the three channels of the image RGB.

[0135] Step 308, based on the point cloud data, voxel point cloud data is obtained according to preset voxel parameters.

[0136] Step 309, using a voxel feature extraction network of a preset target detection model, the voxel point cloud data is processed for feature extraction to obtain voxel feature information.

[0137] In this embodiment, for example, first, for point cloud data covering a range of 80m x 80m x 8m in size, the size of each voxel can be 0.25m x 0.25m x 0.2m. The entire point cloud data space can be voxelized to obtain voxelized point cloud data. Second, a sparse convolution neural network (Sparse Convolution Network) is used to process the voxelized point cloud data for feature extraction to obtain the feature extraction result, i.e., voxel feature information.

[0138] Step 310, using a multi-layer perceptron, the voxel feature information is pruned to obtain voxel query feature information.

[0139] In this embodiment, the number of voxel feature information in the feature extraction result obtained by step 309 is large, which includes voxel feature information representing background information. Here, the voxel feature information can be input into a multi-layer perceptron for classification pruning to obtain voxel feature information of the target of interest, i.e., voxel query feature information (Voxelquery).

[0140] Here, the voxel query feature information can be voxel feature information representing the target obstacle.

[0141] Step 311, using a preset position calibration relationship, based on the voxel query feature information, the image feature information is screened to obtain screened image feature information.

[0142] In this embodiment, the preset position calibration relationship can be a preset position relationship matrix of the laser radar and the camera. The position relationship matrix can represent the conversion relationship between the coordinate systems of the camera and the laser radar. The calibration matrix can be used for the conversion between the coordinate systems of the laser radar and the laser radar.

[0143] Here, first, the preset position calibration relationship can be used to perform coordinate system conversion processing on the voxel query feature information, to obtain voxel query feature information in the image coordinate system. Second, based on the two-dimensional position information of the point cloud corresponding to the voxel query feature information and the position information corresponding to the image feature information, the voxel query feature information and the image feature information can be matched to obtain a matching result. Then, based on the matching result, the image feature information matched with the voxel query feature information is selected from the image feature information as filtered image feature information. Then, the feature fusion network of the preset target detection model can be used to perform feature fusion processing on the voxel query feature information and the filtered image feature information to obtain fusion feature information.

[0144] Step 312, using the feature fusion network of the preset target detection model, performing feature fusion processing on the voxel query feature information and the filtered image feature information to obtain fusion feature information.

[0145] Step 313, using the target detection network of the preset target detection model, performing target detection processing on the fusion feature information to obtain a target detection result.

[0146] In this embodiment, Figure 4 is a schematic diagram of a target detection model of an application scenario of a target detection method provided by an embodiment of the present application, as Figure 4 shown. First, the multi-angle image data collected by the 6-way camera and the point cloud data corresponding to the image data can be input into the target detection model. Second, on the one hand, the image data collected by the 6-way camera can be input into the image feature extraction network for 8 times down-sampling processing to obtain image feature information of the 6-way camera image data. On the other hand, the point cloud data is input into the voxel feature extraction network based on the sparse convolutional neural network to obtain extracted voxel feature information, and the voxel feature information is input into the classifier based on the multi-layer perception to perform pruning processing on the voxel feature information to obtain voxel query feature information (Voxel query) for representing target obstacles. Third, the center position coordinates of the Voxel query are projected onto the image dimension through the position relationship matrix of the radar and the camera, and based on the projection result, the image feature information corresponding to the Voxel query on the image is obtained, as Figure 4The red position part in the image feature information is shown. Again, splicing the Voxel query feature and the image feature information corresponding to the Voxel query on the image can obtain the fusion feature information of the two modalities. Finally, inputting the fusion feature information into the target detection network (head) can output the category of the target detection object corresponding to the position of each Voxel, the 3D detection box coordinates of the target detection object, the 3D detection box size of the target detection object, and the like, that is, the final target detection result can be obtained.

[0147] Based on the technical solutions of the embodiment, the target detection processing can be implemented by fusing the features of the two modalities based on only the region of interest, the commonly used idea of fusing the features in the bird's eye view (BEV) perspective can be abandoned, the features of the background region that is not concerned need not be fused and calculated, the unnecessary calculation amount is reduced, the calculation amount of data processing is reduced on the premise of maintaining the high accuracy of the multi-modal fusion target detection algorithm, and the time consumption of the entire target detection algorithm is optimized.

[0148] For each method embodiment described above, in order to simply describe, each method embodiment is described as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0149] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0150] Figure 5 The structure block diagram of the target detection device provided by one embodiment of the present application is shown in FIG. 5. Figure 5 The target detection device 500 of the embodiment can include an acquisition unit 501, an extraction unit 502, a fusion unit 503, and a detection unit 504. The acquisition unit 501 is configured to acquire image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is a multi-view vehicle driving environment image; the extraction unit 502 is configured to perform feature extraction processing on the image data and the point cloud data respectively to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for representing a target obstacle; the fusion unit 503 is configured to perform feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information; and the detection unit 504 is configured to perform target detection processing on the fusion feature information to obtain a target detection result.

[0151] It should be noted that part or all of the device for target detection in the embodiment can be an application located at the local terminal, or can also be a functional unit such as a plug-in or a software development kit (SDK) arranged in the application located at the local terminal, or can also be a processing engine located in a server at the network side, or can also be a distributed system at the network side, for example, a processing engine or a distributed system in a target detection platform at the network side, and the embodiment does not particularly limit this.

[0152] It can be understood that the application can be a native application (native App) installed on the local terminal, or can also be a web application (web App) of a browser on the local terminal, and the embodiment does not limit this.

[0153] Optionally, in a possible implementation manner of the embodiment, the extraction unit 502 can specifically be configured to perform down-sampling feature extraction processing on the image data according to preset sampling parameters by using an image feature extraction network of the preset target detection model, obtain a result of the feature extraction processing, and obtain the image feature information based on the result of the feature extraction processing.

[0154] Optionally, in a possible implementation manner of the embodiment, the extraction unit 502 can specifically be configured to obtain the voxel point cloud data according to preset voxel parameters based on the point cloud data, perform feature extraction processing on the voxel point cloud data by using a voxel feature extraction network of the preset target detection model, obtain a result of the feature extraction, and obtain the voxel query feature information based on the result of the feature extraction.

[0155] Optionally, in a possible implementation manner of the embodiment, the extraction unit 502 can specifically be configured to perform pruning processing on the result of the feature extraction to obtain a result of the feature extraction after the pruning processing, and obtain the voxel query feature information based on the result of the feature extraction after the pruning processing.

[0156] Optionally, in a possible implementation manner of the embodiment, the fusion unit 503 can specifically be configured to perform screening processing on the image feature information based on the voxel query feature information by using a preset position calibration relationship, to obtain screened image feature information, and perform feature fusion processing on the voxel query feature information and the screened image feature information by using a feature fusion network of the preset target detection model, to obtain fusion feature information.

[0157] Optionally, in a possible implementation of the embodiment, the detection unit 504 can be specifically configured to perform target detection processing on the fusion feature information by using a preset target detection model to obtain a target detection result, wherein the target detection result includes at least one of class information of a target detection object, position coordinates of a detection box of the target detection object, and size information of the detection box of the target detection object.

[0158] In the embodiment, the acquisition unit acquires image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is a multi-view vehicle driving environment image; the extraction unit is configured to perform feature extraction processing on the image data and the point cloud data respectively to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for representing a target obstacle; the fusion unit is configured to perform feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information, so that the detection unit can perform target detection processing on the fusion feature information to obtain a target detection result. Since the image and the point cloud do not need to be converted into a bird's eye view, only the voxel feature information for representing the target obstacle and the image feature information are fused to perform target detection processing on the fused feature information to obtain the target detection result, the reliability of the target detection result can be ensured, the computational complexity of the target detection processing can be effectively reduced, and the time consumption is reduced, thereby ensuring the reliability and efficiency of the target detection processing.

[0159] Figure 6 A structural block diagram of a model training apparatus provided by an embodiment of the application is shown in FIG. 6. Figure 6 The model training apparatus 600 of the embodiment can include an acquisition unit 601, an extraction unit 602, a fusion unit 603, a detection unit 604, and a training unit 605. The acquisition unit 601 is configured to acquire sample image data and sample point cloud data corresponding to the sample image data; the sample image data is a multi-view vehicle driving environment image; the extraction unit 602 is configured to perform feature extraction processing on the sample image data and the sample point cloud data respectively to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for representing a target obstacle; the fusion unit 603 is configured to perform feature fusion processing on the voxel query feature information and the image feature information to obtain fusion feature information; the detection unit 604 is configured to perform target detection processing on the fusion feature information to obtain a target detection result; and the training unit 605 is configured to update and train a target detection model to be trained based on the target detection result, the sample image data, and the sample image data to obtain a completed target detection model.

[0160] It should be noted that part or all of the device for model training in the embodiment can be an application located at the local terminal, or can also be a functional unit such as a plug-in or a software development kit (SDK) arranged in the application located at the local terminal, or can also be a processing engine located in a server at the network side, or can also be a distributed system at the network side, for example, a processing engine or a distributed system in a model training platform at the network side, and the embodiment does not particularly limit this.

[0161] It can be understood that the application can be a native application (nativeApp) installed on the local terminal, or can also be a web application (webApp) of a browser on the local terminal, and the embodiment does not limit this.

[0162] Optionally, in a possible implementation manner of the embodiment, the target detection model can include an image feature extraction network, a voxel feature extraction network, a feature fusion network, and a target detection network.

[0163] In the embodiment, the sample image data and the sample point cloud data corresponding to the sample image data are acquired by the acquisition unit, and then the sample image data and the sample point cloud data are subjected to feature extraction processing by the extraction unit respectively to obtain image feature information and voxel query feature information, the voxel query feature information being voxel feature information for representing a target obstacle, the voxel query feature information and the image feature information are subjected to feature fusion processing by the fusion unit to obtain fusion feature information, the fusion feature information is subjected to target detection processing by the detection unit to obtain a target detection result, and the training unit can update and train the target detection model to be trained based on the target detection result, the sample image data, and the sample image data, so that a target detection model with low latency and high detection accuracy can be obtained, and the detection performance and efficiency of the target detection model are improved.

[0164] In the technical solution of the application, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information such as images and attribute data of the user comply with relevant laws and regulations and do not violate public order and good customs.

[0165] According to the embodiments of the application, the application further provides an electronic device, a readable storage medium, and a computer program product.

[0166] According to embodiments of the present application, further provided is an autonomous vehicle including the provided electronic device, which can be a self-driving vehicle of L2 and above. For example, it can be a self-driving logistics vehicle, an autonomous logistics vehicle, etc.

[0167] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0168] As shown in Figure 7 The electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for the operation of the electronic device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0169] Various components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, a speaker, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0170] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the methods of object detection and the methods of model training. For example, in some embodiments, the methods of object detection and the methods of model training can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the methods of object detection and the methods of model training described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the methods of object detection and the methods of model training by any other suitable means, such as by means of firmware.

[0171] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0172] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0173] In the context of this application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a lined- up electrical connection, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0175] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically happening over the communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0177] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved, which are not limited herein.

[0178] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall fall within the protection scope of the present application.

Claims

1. A method of object detection, characterized in that, The method comprises: acquiring image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is multi-view vehicle driving environment image; respectively performing feature extraction processing on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information for characterizing a target obstacle; performing coordinate system conversion processing on the voxel query feature information using a preset position calibration relationship to obtain voxel query feature information in an image coordinate system; based on two-dimensional position information of the point cloud corresponding to the voxel query feature information and position information corresponding to the image feature information, performing matching processing on the voxel query feature information in the image coordinate system and the image feature information; based on the result of the matching processing, selecting image feature information matched with the voxel query feature information from the image feature information, and taking the selected image feature information matched with the voxel query feature information as filtered image feature information; based on the two-dimensional position information of the point cloud corresponding to the voxel query feature information and the position information corresponding to the image feature information, performing feature splicing processing on the voxel query feature information and the filtered image feature information using a feature fusion network of a preset target detection model to obtain fusion feature information; performing target detection processing on the fusion feature information to obtain a target detection result.

2. The method of claim 1, wherein, performing feature extraction processing on the image data to obtain image feature information, comprising: performing down-sampling feature extraction processing on the image data according to a preset sampling parameter using an image feature extraction network of a preset target detection model to obtain a feature extraction processing result; based on the feature extraction processing result, obtaining the image feature information.

3. The method of claim 1, wherein, performing feature extraction processing on the point cloud data to obtain voxel query feature information, comprising: based on the point cloud data, obtaining voxel point cloud data according to a preset voxel parameter; performing feature extraction processing on the voxel point cloud data using a voxel feature extraction network of a preset target detection model to obtain a feature extraction result; based on the feature extraction result, obtaining the voxel query feature information.

4. The method of claim 3, wherein, based on the feature extraction result, obtaining the voxel query feature information, comprising: performing pruning processing on the feature extraction result to obtain a pruned feature extraction result; based on the pruned feature extraction result, obtaining the voxel query feature information.

5. The method of claim 1, wherein, performing target detection processing on the fusion feature information using a target detection network of a preset target detection model to obtain a target detection result, wherein the target detection result comprises at least one of class information of a target detection object, position coordinates of a detection box of the target detection object, and size information of the detection box of the target detection object. The method comprises:

6. A method for model training, comprising: acquiring sample image data and sample point cloud data corresponding to the sample image data; the sample image data is multi-view vehicle driving environment image; ​ respectively, to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle; perform coordinate system conversion processing on the voxel query feature information by using a preset position calibration relationship to obtain voxel query feature information in an image coordinate system; perform matching processing on the voxel query feature information in the image coordinate system and the image feature information based on two-dimensional position information of the point cloud corresponding to the voxel query feature information and position information corresponding to the image feature information; and select image feature information matched with the voxel query feature information from the image feature information based on a result of the matching processing, and take the selected image feature information matched with the voxel query feature information as filtered image feature information; perform feature splicing processing on the voxel query feature information and the filtered image feature information based on the two-dimensional position information of the point cloud corresponding to the voxel query feature information and the position information corresponding to the image feature information by using a feature fusion network of a target detection model to be trained, to obtain fusion feature information; perform target detection processing on the fusion feature information to obtain a target detection result. update and train the target detection model to be trained based on the target detection result, the sample image data, and the sample point cloud data, to obtain a target detection model completed with training.

7. An apparatus for target detection, comprising: an acquisition unit configured to acquire image data to be processed and point cloud data to be processed corresponding to the image data; the image data to be processed is multi-view vehicle driving environment image data; an extraction unit configured to respectively perform feature extraction processing on the image data and the point cloud data to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used to represent a target obstacle; a fusion unit configured to perform coordinate system conversion processing on the voxel query feature information by using a preset position calibration relationship to obtain voxel query feature information in an image coordinate system; perform matching processing on the voxel query feature information in the image coordinate system and the image feature information based on two-dimensional position information of the point cloud corresponding to the voxel query feature information and position information corresponding to the image feature information; select image feature information matched with the voxel query feature information from the image feature information based on a result of the matching processing, and take the selected image feature information matched with the voxel query feature information as filtered image feature information; and perform feature splicing processing on the voxel query feature information and the filtered image feature information based on the two-dimensional position information of the point cloud corresponding to the voxel query feature information and the position information corresponding to the image feature information by using a feature fusion network of a target detection model to be trained, to obtain fusion feature information; a detection unit configured to perform target detection processing on the fusion feature information to obtain a target detection result.

8. An apparatus for model training, the apparatus comprising: The apparatus comprises: The acquisition unit is configured to acquire sample image data and sample point cloud data corresponding to the sample image data, wherein the sample image data is multi-view vehicle driving environment image data. The extraction unit is configured to perform feature extraction processing on the sample image data and the sample point cloud data respectively to obtain image feature information and voxel query feature information, wherein the voxel query feature information is voxel feature information used for representing a target obstacle. The fusion unit is configured to perform coordinate system conversion processing on the voxel query feature information based on a preset position calibration relationship to obtain voxel query feature information in an image coordinate system, perform matching processing on the voxel query feature information in the image coordinate system and the image feature information based on two-dimensional position information of the point cloud corresponding to the voxel query feature information and position information of the image feature information, select image feature information matched with the voxel query feature information from the image feature information based on a result of the matching processing, and take the selected image feature information matched with the voxel query feature information as filtered image feature information. The detection unit is configured to perform target detection processing on the fusion feature information to obtain a target detection result. The training unit is configured to perform update training on a target detection model to be trained based on the target detection result, the sample image data, and the sample point cloud data to obtain a target detection model completed with training.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-6.

11. An autonomous vehicle comprising the electronic device of claim 9.

Images