Three-dimensional target detection method and device, computer equipment and storage medium

Abstract

The invention relates to a three-dimensional target detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-detected image and point cloud depth information, and inputting the to-be-detected image and the point cloud depth information into a pre-trained target detection model; extracting two-dimensional image features in the to-be-detected image through the target detection model, and generating a target detection result of the target object according to the two-dimensional image features; generating three-dimensional occupation information of the target object according to the point cloud depth information through a target detection model; and generating a three-dimensional target detection result of the target object through the target detection model according to the target detection result and the three-dimensional occupation information. By adopting the method, the three-dimensional occupation information generated based on the point cloud depth information can be utilized to compensate the missing depth information in the two-dimensional image features, so that the detection precision of the three-dimensional target detection result is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a three-dimensional target detection method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of autonomous driving technology, 3D object detection is an important perception task. Through 3D object detection, autonomous vehicles can intelligently predict the location, size, or category of nearby 3D objects. Currently, the commonly used 3D object detection technologies are mainly divided into the following two types: vision-based monocular 3D object detection technology and radar-based 3D object detection technology.

[0003] Vision-based monocular 3D target detection technology is a technology that uses 2D images collected by a monocular camera (that is, a camera with only one camera) to detect 3D targets. Radar-based 3D target detection technology is a technology that uses point cloud data collected by high-cost radar equipment to detect 3D targets. Since vision-based monocular 3D target detection technology only requires one camera for hardware deployment, and its target detection accuracy is not affected by environmental factors such as rain and snow, it has shown great advantages compared to radar-based 3D target detection technology and has received widespread attention from industry and academia.

[0004] However, in actual application scenarios, due to the lack of depth information in two-dimensional images, it is difficult for vision-based monocular three-dimensional object detection technology to learn depth-related features, resulting in low depth prediction accuracy of current vision-based monocular three-dimensional object detection technology. Summary of the invention

[0005] Based on this, it is necessary to provide a high-precision three-dimensional target detection method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a three-dimensional target detection method, comprising:

[0007] Acquire an image to be detected and point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model;

[0008] Extracting two-dimensional image features in the image to be detected through the target detection model, and generating a target detection result of the target object according to the two-dimensional image features;

[0009] Generating three-dimensional occupancy information of the target object according to the point cloud depth information by the target detection model;

[0010] The target detection model generates a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information.

[0011] In one embodiment, the step of obtaining the image to be detected and the point cloud depth information includes:

[0012] Acquire an initial image captured by a monocular image acquisition device;

[0013] Using preset conversion parameters, the initial image is mapped to a virtual coordinate system corresponding to the monocular image acquisition device to obtain the image to be detected;

[0014] Determine the point cloud depth information according to the initial image.

[0015] In one embodiment, the pre-training method of the target detection model includes:

[0016] Acquire detection image samples and point cloud depth samples, and input the detection image samples and the point cloud depth samples into an initial target detection model;

[0017] Generating an initial detection result of an object sample according to the image feature sample of the detection image sample by the initial object detection model;

[0018] Generate a loss value corresponding to the initial detection result according to the point cloud depth sample by the initial target detection model;

[0019] The initial target detection model is trained iteratively by adjusting model parameters according to the loss value to obtain the target detection model.

[0020] In one embodiment, generating an initial detection result of an object sample according to an image feature sample of the detection image sample by the initial object detection model includes:

[0021] Performing multiple sampling processes on the detection image samples through the initial target detection model to obtain sampling feature maps at multiple scales;

[0022] Performing feature fusion on the sampling feature maps at multiple scales through the initial target detection model to obtain a fused feature map, and extracting image feature samples of the detection image samples from the fused feature map;

[0023] The initial target detection model uses the image feature samples of the detection image samples to match the object feature samples to generate the initial detection results of the object samples.

[0024] In one embodiment, the method further comprises:

[0025] Determine the sampling feature value of the target detection model in each channel according to the fusion feature map generated by the target detection model;

[0026] Performing pruning processing on the channel of the target detection model according to the sampled feature values ​​to obtain a processed pruned detection model;

[0027] According to the operating efficiency of the pruning detection model, the target detection model used in the edge device is determined.

[0028] In one embodiment, determining the target detection model used on the edge device according to the pruned target detection model includes:

[0029] Determine pixel features and pixel gradients extracted by the pruning detection model according to the fused feature map generated by the pruning detection model;

[0030] Generate a weight influence factor of the pixel point according to the pixel point feature and the pixel point gradient;

[0031] The pruning detection model is sparsely processed according to the weight influence factor to obtain a target detection model used in the edge device.

[0032] In a second aspect, the present application also provides a three-dimensional object detection device, comprising:

[0033] A data acquisition module is used to acquire the image to be detected and the point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model;

[0034] A feature extraction module, used to extract two-dimensional image features in the image to be detected through the target detection model, and generate a target detection result of the target object according to the two-dimensional image features;

[0035] An occupancy supervision module, configured to generate three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model;

[0036] A feature fusion module is used to generate a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model.

[0037] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the three-dimensional target detection method described in any one of the embodiments of the first aspect is implemented.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the three-dimensional target detection method described in any one of the embodiments in the first aspect above is implemented.

[0039] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the three-dimensional target detection method described in any one of the embodiments in the first aspect above.

[0040] The above-mentioned three-dimensional target detection method, device, computer equipment, storage medium and computer program product obtain the image to be detected and the point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model; extract the two-dimensional image features in the image to be detected through the target detection model, and generate the target detection result of the target object according to the two-dimensional image features; generate the three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model; generate the three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model, and can use the three-dimensional occupancy information generated based on the point cloud depth information to compensate for the missing depth information in the two-dimensional image features, thereby improving the detection accuracy of the three-dimensional target detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] FIG1 is a diagram of an application environment of a three-dimensional target detection method according to an embodiment;

[0043] FIG2 is a schematic diagram of a flow chart of a three-dimensional target detection method in one embodiment;

[0044] Figure 3 It is a schematic diagram of the process of obtaining the depth information of the image to be detected and the point cloud in one embodiment;

[0045] Figure 4 A schematic diagram of a process of pre-training a target detection model in one embodiment;

[0046] Figure 5 A schematic diagram of a process for generating an initial detection result in one embodiment;

[0047] Figure 6 A schematic diagram of a process of pruning a target detection model in one embodiment;

[0048] Figure 7 A schematic diagram of a process for weight sparsification of a pruning detection model in one embodiment;

[0049] Figure 8 is a schematic flow chart of a three-dimensional object detection method in another embodiment;

[0050] Fig. 9 A schematic diagram of a step of generating an image to be detected according to an embodiment;

[0051] Fig.10 A schematic diagram of a network structure of a target detection model in one embodiment;

[0052] Fig.11 is a structural block diagram of a three-dimensional object detection device in one embodiment;

[0053] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The three-dimensional target detection method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Specifically, a pre-trained target detection model can be pre-deployed in the server 104. The server 104 can obtain the image to be detected and the point cloud depth information collected or sent by the terminal 102 through communication interaction with the terminal 102, and input the image to be detected and the point cloud depth information into the pre-trained target detection model. The server 104 can extract the two-dimensional image features in the image to be detected through the target detection model, and generate the target detection result of the target object in the image to be detected according to the matching result of the two-dimensional image features and the features of the object samples learned during the training process. The server 104 can estimate the position of the target object in the three-dimensional space according to the point cloud depth information through the target detection model, thereby generating the three-dimensional occupancy information of the target object. The server 104 can generate the three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information of the target object through the target detection model. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0056] In an exemplary embodiment, as shown in FIG2 , a three-dimensional object detection method is provided, and the method is applied to the server 104 in FIG1 as an example for description, including the following steps S202 to S208. Among them:

[0057] Step S202: Obtain the image to be detected and the point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model.

[0058] The image to be detected may be obtained by an image acquisition device shooting the surrounding environment of the vehicle. The number of images to be detected may be one or more. Optionally, in some embodiments, the image to be detected may also be a continuous multi-frame image extracted from a video recorded by the vehicle-mounted image acquisition device.

[0059] The point cloud depth information can represent the distance in three-dimensional space between the object captured in the image to be detected (such as a vehicle, pedestrian, obstacle, etc.) and the image acquisition device that captured the image to be detected.

[0060] The pre-trained target detection model may be a neural network model trained using vision-based monocular three-dimensional target detection technology.

[0061] Specifically, the server can respond to a three-dimensional target detection request for an image to be detected, obtain the image to be detected sent by the terminal and the point cloud depth information corresponding to the image to be detected, or only receive multiple consecutive frames of images to be detected sent by the terminal, and extract the point cloud depth information corresponding to the image to be detected based on the multiple consecutive frames of images to be detected. The image to be detected and the point cloud depth information are input into the pre-trained target detection model to perform the subsequent steps S204-S208. Optionally, in some embodiments, before inputting the image to be detected and the point cloud depth information into the pre-trained target detection model, the server can also perform pre-processing operations such as filtering and denoising on the image to be detected and the point cloud depth information.

[0062] Step S204: extracting two-dimensional image features in the image to be detected through the target detection model, and generating a target detection result of the target object according to the two-dimensional image features.

[0063] The target object may refer to a three-dimensional object captured in the image to be detected.

[0064] The target detection results may include but are not limited to any one or more of the category to which the target object belongs (for example, pedestrian category, obstacle category, building category, vehicle category, etc.), the coordinates of the center point of the target object in the image to be detected, the actual size of the target object in three-dimensional space and depth information (that is, the distance between the target object and the image acquisition device that captures the image to be detected in three-dimensional space), angle information (that is, the Euler angle of the target object relative to the image acquisition device that captures the image to be detected), and the like.

[0065] Specifically, the server can perform feature extraction processing on the image to be detected through the target detection model, for example, by constructing a Histogram of Oriented Gradient (HOG) or extracting local features using a Local Binary Pattern (LBP) operator, thereby obtaining two-dimensional image features in the image to be detected. Matching is performed based on the two-dimensional image features and the features of the object samples learned during the model training process, thereby generating target detection results for the target object.

[0066] Step S206: Generate three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model.

[0067] The three-dimensional occupancy information may indicate whether a certain position in the three-dimensional space is occupied by the target object.

[0068] Specifically, the server can determine whether there is a target object at each pixel position in the three-dimensional space corresponding to the image to be detected based on the point cloud depth information through the target detection model, that is, whether the pixel position is occupied by the target object, thereby generating three-dimensional occupancy information corresponding to the target object in the image to be detected. For example, when the point cloud depth information is (X, Y, Z), the three-dimensional occupancy information can be generated as (X, Y, Z, 1) to indicate that the position (X, Y, Z) in the three-dimensional space is occupied by the target object.

[0069] Step S208: Generate a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model.

[0070] Specifically, the server can fuse the target detection result and three-dimensional occupancy information of the target object through the target detection model. For example, the three-dimensional occupancy information is used to correct the actual size and depth information of the target object in the three-dimensional space in the target detection result, and the corrected target detection result is used as the three-dimensional target detection result of the target object; or, the target detection result and the three-dimensional occupancy information can be directly spliced ​​to obtain the three-dimensional target detection result of the target object.

[0071] In the above-mentioned three-dimensional target detection method, by acquiring the image to be detected and the point cloud depth information, the image to be detected and the point cloud depth information are input into the pre-trained target detection model; the two-dimensional image features in the image to be detected are extracted by the target detection model, and the target detection result of the target object is generated according to the two-dimensional image features; the three-dimensional occupancy information of the target object is generated according to the point cloud depth information by the target detection model; the three-dimensional target detection result of the target object is generated according to the target detection result and the three-dimensional occupancy information by the target detection model, and the three-dimensional occupancy information can be used to compensate for the missing depth information in the two-dimensional image features, thereby improving the detection accuracy of the three-dimensional target detection result.

[0072] In the actual application scenarios of autonomous driving, autonomous vehicles usually use image acquisition devices installed around the vehicle body to collect images. However, there may be differences in the installation positions of image acquisition devices of different autonomous vehicles, which may cause the images collected by different autonomous vehicles to be deformed and cannot be applied to the same target detection model. Therefore, in order to solve the above-mentioned problem that the target detection model cannot be universal due to the differences in the installation positions of the image acquisition devices, this application proposes the following solution:

[0073] In an exemplary embodiment, Figure 3 As shown, step S202 may further include the following steps S302-S306:

[0074] Step S302, obtaining an initial image taken by a monocular image acquisition device.

[0075] Among them, the monocular image acquisition device can refer to a vehicle-mounted image acquisition device with only one camera.

[0076] The initial image may refer to an image directly imaged by a monocular image acquisition device, and may include a single image or a continuous plurality of frames of images.

[0077] Specifically, the server can obtain the initial image captured by the vehicle-mounted monocular image acquisition device for the vehicle surrounding environment through data communication with the vehicle-mounted terminal.

[0078] Step S304, using preset conversion parameters, the initial image is mapped to a virtual coordinate system corresponding to the monocular image acquisition device to obtain an image to be detected.

[0079] Among them, the preset conversion parameters can be used to represent the coordinate system conversion relationship between the imaging coordinate system of the monocular image acquisition device and the virtual coordinate system corresponding to the monocular image acquisition device.

[0080] The virtual coordinate system may represent a coordinate system that is independent of the installation position of the monocular image acquisition device.

[0081] Specifically, the server may pre-store a virtual coordinate system corresponding to the monocular image acquisition device and preset conversion parameters between the imaging coordinate system of the monocular image acquisition device and the virtual coordinate system. The server may use the preset conversion parameters to perform coordinate conversion processing on the initial image, and map the initial image to the virtual coordinate system corresponding to the monocular image acquisition device, thereby obtaining an image to be detected in the virtual coordinate system.

[0082] Optionally, in some embodiments, the vehicle-mounted monocular image acquisition device may include a camera at a preset position on the vehicle body, or multiple cameras at multiple positions on the vehicle body. When there are multiple monocular image acquisition devices, a virtual coordinate system can be set to correspond to the cameras at multiple positions according to the user's usage requirements in the actual application scenario, or multiple virtual coordinate systems can be set to correspond to multiple cameras one by one. The cameras on different autonomous driving vehicles can determine the corresponding virtual coordinate system based on the position of the camera relative to the autonomous driving vehicle. For example, the virtual coordinate systems corresponding to the cameras located on the left front of the vehicle can be set to coordinate system 1, and the virtual coordinate systems corresponding to the cameras located on the right front of the vehicle can be set to coordinate system 2, so as to solve the problem that the target detection model cannot be universal due to the differences in the installation positions of the cameras on different autonomous driving vehicles.

[0083] Step S306: determining point cloud depth information according to the initial image.

[0084] Specifically, the server can generate corresponding point cloud depth information based on the brightness, different viewing angles, luminosity, texture and other information of the pixels in the initial image. Alternatively, the distance between the object in the initial image and the monocular image acquisition device can be calculated based on the imaging parameters of the monocular image acquisition device to obtain the point cloud depth information. Alternatively, when the initial image is a continuous multi-frame image, the server can also use the position of the same object in different frame images and the movement speed of the autonomous driving vehicle to calculate the corresponding point cloud depth information.

[0085] In this embodiment, by acquiring an initial image captured by a monocular image acquisition device, and using preset conversion parameters to map the initial image to a virtual coordinate system corresponding to the monocular image acquisition device to obtain an image to be detected, the influence of the installation position differences of different monocular image acquisition devices on the image to be detected can be resolved, so that the target detection model provided by this application can be universal. By determining the point cloud depth information corresponding to the image to be detected based on the initial image, the missing depth information in the two-dimensional image to be detected can be compensated, which is conducive to improving the detection accuracy of subsequent three-dimensional target detection.

[0086] In an exemplary embodiment, Figure 4 As shown, the target detection model used in the above steps S202 to S208 can be pre-trained through the following steps S402-S408. Among them:

[0087] Step S402, obtaining detection image samples and point cloud depth samples, and inputting the detection image samples and point cloud depth samples into an initial target detection model.

[0088] Step S404: generating an initial detection result of the object sample according to the image feature sample of the detection image sample through the initial target detection model.

[0089] The detection image sample may represent sample data that has been annotated with data such as the category, center point position, three-dimensional size, depth information, angle information, etc. of the object sample. The point cloud depth sample may represent sample data obtained by annotating the depth information of the object sample in the detection image sample.

[0090] The object sample may represent a three-dimensional object in the detection image sample. The initial detection result may represent a detection result generated by the initial target detection model and related to data such as the category, center point position, three-dimensional size, depth information, angle information, etc. of the object sample.

[0091] Specifically, during the model training phase, the server can obtain labeled detection image samples and point cloud depth samples, and divide the detection image samples and point cloud depth samples into a training set and a validation set. The training set is first input into the initial target detection model for pre-training, and then the validation set is input into the pre-trained initial target detection model for validation. The image feature samples of the detection image samples in the validation set are extracted by the initial target detection model, and the image feature samples extracted from the validation set are matched with the object feature samples learned in the training set to generate the initial detection results of the object samples.

[0092] Step S406, generating a loss value corresponding to the initial detection result according to the point cloud depth samples through the initial target detection model.

[0093] Specifically, the server can obtain the difference between the point cloud depth sample and the annotation result of the verification set and the initial detection result generated in step S404 through the initial target detection model, and use the difference and the loss function preset by the initial target detection model to calculate the loss value corresponding to the initial detection result.

[0094] Step S408, adjusting the model parameters according to the loss value through the initial target detection model for iterative training to obtain the target detection model.

[0095] Specifically, the server can use the loss value of the initial target detection model to transfer back in the model network to adjust the model parameters, and repeat the operations in the above steps S402-S406 to iteratively train the initial target detection model after the model parameters are adjusted, until the loss value of the trained initial target detection model meets the preset threshold, or the number of iterative training meets the preset conditions, the training process ends, and the initial target detection model obtained at this time is used as the target detection model for actual application.

[0096] In this embodiment, point cloud depth samples are introduced into the vision-based target detection model training process to calculate the loss value of the initial detection result, and the loss value is used to back-propagate within the model to adjust the model parameters. The target detection model is obtained by iterative training. The point cloud depth samples can be used to provide supervision information to assist the model in learning deep features, which is beneficial to improving the detection accuracy of the target detection model.

[0097] In an exemplary embodiment, Figure 5 As shown, the above step S404 can also be implemented by the following steps S502-S506. Among them:

[0098] Step S502: Perform multiple sampling processes on the detection image samples through the initial target detection model to obtain sampling feature maps at multiple scales.

[0099] Specifically, the server can perform multiple sampling processes on the detection image samples using convolution kernels of different sizes through the initial target detection model to obtain sampling feature maps at multiple scales. Alternatively, the server can also use a bottom-up or top-down approach to process the sampling feature map obtained from each sampling process as input data for the next sampling process, thereby obtaining sampling feature maps at multiple scales from bottom-level features to high-level features.

[0100] Step S504, performing feature fusion on the sampling feature maps at multiple scales through the initial target detection model to obtain a fused feature map, and extracting image feature samples of the detection image samples from the fused feature map.

[0101] Among them, the fused feature map can represent the feature map of the output part of the input network of the initial target detection model.

[0102] Specifically, the server can perform feature fusion processing on the sampled feature maps at multiple scales through the initial target detection model, such as superimposing the pixel point features at the same position as the result of feature fusion, thereby obtaining a processed fused feature map. The fused feature map is subjected to feature extraction by the initial target detection model to obtain image feature samples of the detection image samples of the verification set in steps S402-S404.

[0103] Step S506: The initial target detection model is used to match the image feature samples of the detection image samples with the object feature samples to generate an initial detection result of the object sample.

[0104] Specifically, the server can use the initial target detection model to match the image feature samples of the detection image samples obtained in step S504 with the object feature samples learned by the model from the training set in steps S402-S404, thereby generating initial detection results for the object samples in the verification set.

[0105] In this embodiment, the detection image samples are sampled multiple times through the initial target detection model to obtain sampling feature maps at multiple scales, and feature fusion is performed on the sampling feature maps at multiple scales to obtain a fused feature map. The fused feature map is used to extract image feature samples of the detection image samples, and the image feature samples are used to generate initial detection results. It is possible to fuse the underlying global features and the high-level local features, so that the semantic information extracted by the model network is richer, which helps to improve the detection accuracy of the initial detection results.

[0106] At present, due to the large amount of data that needs to be processed by vision-based monocular 3D object detection technology, it is mostly used in large equipment such as industrial computers. However, most autonomous driving devices are equipped with low-end edge devices with low computing power, which leads to low detection efficiency of vision-based monocular 3D object detection technology in the field of autonomous driving.

[0107] In an exemplary embodiment, the present application solves the above technical problems. Figure 5 Based on the target detection model pre-training method shown in , a model optimization training method is also provided to enable the target detection model to run quickly and efficiently in edge devices. Figure 6 As shown, the following steps S602-S606 are included. Among them:

[0108] Step S602, determining the sampling feature value of the target detection model in each channel according to the fusion feature map generated by the target detection model.

[0109] Specifically, the server can input the detection image sample into the target detection model obtained by iterative training in step S408. The target detection model processes the detection image sample with reference to the generation method of the fusion feature map provided in the above steps S502-S504 to obtain the fusion feature map generated by the target detection model. The sampling feature values ​​of the pixel points in the fusion feature map generated by the target detection model under each channel of the target detection model are obtained.

[0110] Step S604, pruning the channels of the target detection model according to the sampled feature values ​​to obtain a processed pruned detection model.

[0111] Specifically, the server can compare the sampling feature values ​​of the same pixel point obtained in step S602 under each channel. Since the sampling feature value of the pixel point is linearly related to the contribution of the channel, the server can regard the channel with a smaller sampling feature value as a channel with a smaller contribution to the detection accuracy of the model and perform pruning processing to delete the channel with a smaller contribution, thereby improving the operating efficiency of the target detection model and obtaining a pruned detection model.

[0112] Step S606: Determine the target detection model used in the edge device according to the operating efficiency of the pruned detection model.

[0113] Specifically, the server can obtain the operating efficiency of the pruned detection model. When the operating efficiency is lower than the preset efficiency threshold, the pruned detection model is repeatedly executed multiple times in steps S602-S604 to perform iterative pruning until the operating efficiency of the pruned detection model after iterative pruning meets the preset efficiency threshold, and the pruned detection model after iterative pruning is used as the target detection model used in the edge device.

[0114] In this embodiment, the target detection model is pruned based on the contribution of the channel of the target detection model to obtain a pruned detection model, and the target detection model finally used in the edge device is determined according to the operating efficiency of the pruned detection model, so that the target detection model can be detected quickly and efficiently in the edge device, thereby improving the efficiency of three-dimensional target detection.

[0115] In an exemplary embodiment, Figure 7 As shown, this application Figure 6 Based on the implementation shown in , a method for weight sparseness of the pruning detection model to further improve the operation efficiency is also provided, and the above step S606 may also include the following steps S702-S706. Among them:

[0116] Step S702, according to the fusion feature map generated by the pruning detection model, determine the pixel features and pixel gradients extracted by the pruning detection model.

[0117] Specifically, the server can input the detection image sample into the pruning detection model, and the pruning detection model processes the detection image sample with reference to the generation method of the fused feature map provided in the above steps S502-S504 to obtain the fused feature map generated by the pruning detection model. The pixel point features and pixel point gradients of each pixel point are extracted from the fused feature map generated by the pruning detection model.

[0118] Step S704: generating a weight influence factor of the pixel point according to the pixel point feature and the pixel point gradient.

[0119] Specifically, the server may use a multiplication method to perform calculations on the pixel feature and the pixel gradient of the same pixel, and use the product of the pixel feature and the pixel gradient as a weight influencing factor of the pixel.

[0120] Step S706, performing sparse processing on the pruned detection model according to the weight influence factor to obtain a target detection model used in the edge device.

[0121] Specifically, the server can compare the weight influence factor of each pixel point, and determine the pixel point whose weight influence factor is less than the preset factor or the weight influence factor is less than the neighboring pixel point as the point to be thinned out. The pruning detection model is thinned out to delete the pixel point features and pixel point gradients of the point to be thinned out, reduce the amount of pixel features processed by the pruning detection model, and improve the operating efficiency of the processed pruning detection model. The pruning detection model after the sparse processing is used as the target detection model used in the edge device.

[0122] Optionally, in some embodiments, the server can also determine the comparison range of pixels based on the convolution kernel size of each network module in the pruning detection model. For example, when the convolution kernel size is 3x3, the weight influence factors of pixels within the 3x3 range can be compared to obtain the pixels that need to be sparse in the 3x3 range. The sparseness of different weight dimensions for different network modules of the pruning detection model can be achieved to achieve the fastest operating efficiency optimization in the shortest working time.

[0123] Optionally, in some other embodiments, the server may also repeatedly execute the operations of steps S702-S706 above to iteratively sparsely process the pruned detection model after the sparse processing until the operating efficiency of the model after the iterative sparse processing meets a preset efficiency threshold, and use the iteratively sparsely processed pruned detection model as the target detection model used in the edge device.

[0124] In this embodiment, by further performing weight thinning processing on the pruning detection model, the pixel feature data calculated by the pruning detection model can be reduced, and the operating efficiency of the model after subsequent thinning processing can be further improved.

[0125] In an exemplary embodiment, Figure 8 As shown, a three-dimensional target detection method based on the above embodiments is also provided, including the following steps S802-S810. Among them:

[0126] Step S802 , obtaining an initial image, using preset conversion parameters to map the initial image to a virtual coordinate system corresponding to the monocular image acquisition device to obtain an image to be detected, and determining point cloud depth information based on the initial image.

[0127] Specifically, the server can obtain the initial image taken by the vehicle-mounted monocular image acquisition device. Fig. 9 As shown, the initial image (i.e., the image at Fig. 9 The image before the coordinate system transformation on the left side) is mapped to the virtual coordinate system corresponding to the monocular image acquisition device to obtain the image to be detected (that is, the image located at Fig. 9 The server may refer to the method for obtaining the point cloud depth information provided in the above embodiment to process the initial image accordingly to obtain the point cloud depth information corresponding to the image to be detected.

[0128] Optionally, in some embodiments, there may be four monocular image acquisition devices on the vehicle. The server may establish a virtual coordinate system corresponding to each monocular image acquisition device, or establish a unified virtual coordinate system corresponding to multiple monocular image acquisition devices, according to the actual use requirements of the user. The coordinate system transformation of the initial image is realized by utilizing the transformation relationship between the imaging coordinate system of the monocular image acquisition device and the virtual coordinate system, that is, the preset transformation parameters, so as to eliminate the difference between the initial images caused by the difference in the installation position of the monocular image acquisition device, so that only one universal target detection model is required to realize three-dimensional target detection on different vehicles.

[0129] Step S804: input the image to be detected and the point cloud depth information into a pre-trained object detection model.

[0130] Specifically, in the model training phase, the server can obtain detection image samples and point cloud depth samples, and divide the detection image samples and point cloud depth samples into a training set and a validation set. The training set data is input into the initial target detection model so that the initial target detection model learns to obtain the object feature samples in the training set data, and then the validation set data is input into the initial target detection model.

[0131] The initial target detection model uses multiple convolution kernels of different sizes to perform multiple upsampling processes from bottom to top and multiple downsampling processes from top to bottom on the detection image samples of the validation set data, and obtains sampling feature maps at multiple scales, and fuses the underlying features and high-level features at multiple scales to obtain a fused feature map. The initial target detection model extracts image feature samples of the detection image samples from the fused feature map. The initial target detection model uses the image feature samples of the detection image samples of the validation set data to match the object feature samples in the training set data to generate the initial detection results of the object samples. The initial target detection model generates a loss value corresponding to the initial detection result based on the difference between the point cloud depth information and the initial detection result. The initial target detection model uses the back-propagation loss value to adjust the model parameters to iteratively train the initial target detection model to obtain the trained target detection model.

[0132] Input the detection image samples into the trained target detection model to obtain the fused feature map generated by the target detection model. Determine the sampling feature value of each pixel in each channel in the fused feature map generated by the target detection model. Determine the contribution of the corresponding channel to the target detection model based on the sampling feature value of the pixel, and prune the target detection model based on the contribution of the channel to eliminate channels with lower contribution to obtain a processed pruned detection model. According to the operating efficiency of the pruned detection model, repeat the above pruning operation to iteratively prune the pruned detection model to obtain a pruned detection model after iterative pruning.

[0133] Input the detection image sample into the pruned detection model after iterative pruning processing, and obtain the fused feature map generated by the pruned detection model after iterative pruning processing. The pixel features and pixel gradients of each pixel are extracted from the fused feature map by the pruned detection model after iterative pruning processing. The pixel features and pixel gradients of the same pixel are multiplied to obtain the weight influence factor of each pixel. The following operations are performed for different network modules of the pruned detection model after iterative pruning processing: determine the pixel sparse range according to the convolution kernel of the network module, compare the weight influence factor of each pixel within the pixel sparse range, and determine the points to be sparse based on the comparison results of the weight influence factors. Perform sparse processing on the pruned detection model after iterative pruning processing to eliminate the pixel features and pixel gradients of the points to be sparse within the pixel sparse range, and obtain the sparsed pruned detection model. Repeat the above-mentioned sparse processing operation, and perform iterative sparse processing on the pruned detection model after the sparse processing until the number of iterations meets the preset number threshold, or the model operation efficiency and model detection accuracy after the iteration meet the preset conditions, then complete the model training operation and obtain the target detection model used on the edge device.

[0134] Optionally, in some embodiments, Fig.10 As shown in the figure, the initial target detection model can include four parts: backbone network backbone, neck network neck, head network head and auxiliary supervision network auxiliary part.

[0135] The backbone network is used as the feature extraction part. The data sources processed are the detection image samples and point cloud depth samples. Common networks such as vovnet (a network structure that focuses on graphics processor computing and is energy-efficient) and dla (Deep Layer Aggregation, a deep aggregation network structure) can be used. The step lengths of different network layers relative to the original input image can be 8, 16, 32, or 64.

[0136] The neck network can adopt the feature extraction method of FPN (Feature Pyramid Networks) + PAN (Path Aggregation Network), and use the fusion of low-level features and high-level features to make the semantic information of the neck network richer, which helps to improve the detection accuracy.

[0137] The head network head as the output part can include two parts: a 3D detection head and a 3D occupancy output head. Among them, the 3D detection head can include four parts: a category detection head, which is responsible for outputting the category of the target object (i.e. Fig.10 cls in ); Two-dimensional center point detection head, which is responsible for outputting the center point position of the target object in the image to be detected (that is, Fig.10 (x, y) in the 3D information detection head, which is responsible for outputting the actual size and depth information of the target object in the 3D space (i.e. Fig.10 The angle detection head is responsible for outputting the Euler angle of the target object relative to the monocular image acquisition device (i.e. Fig.10 yaw, pitch, and roll).

[0138] The auxiliary part of the auxiliary supervision network can adopt the occupancy3D network (an occupancy supervision head, which can provide supervision information for the 3D detection head as occupancy supervision during the model training process, and can also output the 3D occupancy information during the model application process and merge it with the output result of the 3D detection head to improve the accuracy of the final 3D target detection result).

[0139] In some implementations, the auxiliary part of the auxiliary supervisory network may be trained first, and then the remaining network modules may be trained. In other implementations, the four networks may also be jointly trained.

[0140] Step S806, obtaining a fused feature map by fusing the sampled feature maps of the image to be detected at multiple scales through the target detection model, and generating a target detection result of the target object using the two-dimensional image features in the fused feature map.

[0141] Step S808: Generate three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model.

[0142] Step S810: Generate a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model.

[0143] Specifically, the server may perform multiple sampling processes on the image to be detected through the target detection model pre-trained in step S804 to obtain sampling feature maps of the image to be detected at multiple scales. The sampling feature maps at multiple scales are fused through the target detection model to obtain a fused feature map. The feature data of each pixel in the fused feature map is read to obtain the two-dimensional image features of the image to be detected. The target detection result of the target object is generated using the two-dimensional image features in the fused feature map. The point cloud depth information is processed through the target detection model to generate three-dimensional occupancy information of the target object. The target detection result and the three-dimensional occupancy information of the target object are fused through the target detection model to generate a three-dimensional detection result of the target object.

[0144] In this embodiment, by mapping the initial image to a virtual coordinate system, the effect of differences in the installation positions of the monocular image acquisition device on the image to be detected is resolved, so that the target detection model can be universal. By using point cloud depth samples to assist in the supervision of training the initial target detection model, a larger target detection model is obtained after training, which can ensure the detection accuracy of the target detection model. By pruning and weight sparsifying the target detection model to obtain a target detection model suitable for edge devices, the detection speed of the target detection model can be improved while ensuring the detection accuracy of the target detection model. By fusing the target detection result and the three-dimensional occupancy information to obtain a three-dimensional target detection result, the three-dimensional occupancy information can be used to compensate for the missing depth information in the image to be detected, thereby improving the detection accuracy of the three-dimensional target detection result.

[0145] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] Based on the same inventive concept, the embodiment of the present application also provides a three-dimensional object detection device for implementing the three-dimensional object detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more three-dimensional object detection device embodiments provided below can refer to the limitations of the three-dimensional object detection method above, and will not be repeated here.

[0147] In an exemplary embodiment, Fig.11 As shown, a three-dimensional object detection device 1100 is provided, including: a data acquisition module 1102, a feature extraction module 1104, an occupancy supervision module 1106 and a feature fusion module 1108, wherein:

[0148] The data acquisition module 1102 is used to acquire the image to be detected and the point cloud depth information, and input the image to be detected and the point cloud depth information into the pre-trained target detection model.

[0149] The feature extraction module 1104 is used to extract the two-dimensional image features in the image to be detected through the target detection model, and generate the target detection result of the target object according to the two-dimensional image features.

[0150] The occupancy supervision module 1106 is used to generate three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model.

[0151] The feature fusion module 1108 is used to generate a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model.

[0152] In an exemplary embodiment, the data acquisition module 1102 includes an image acquisition unit for acquiring an initial image taken by a monocular image acquisition device; a coordinate system conversion unit for mapping the initial image to a virtual coordinate system corresponding to the monocular image acquisition device using preset conversion parameters to obtain an image to be detected; and a depth acquisition unit for determining point cloud depth information based on the initial image.

[0153] In an exemplary embodiment, the three-dimensional object detection device 1100 also includes a model training module, which is used to obtain detection image samples and point cloud depth samples, and input the detection image samples and the point cloud depth samples into an initial object detection model; generate an initial detection result of the object sample according to the image feature samples of the detection image samples through the initial object detection model; generate a loss value corresponding to the initial detection result according to the point cloud depth samples through the initial object detection model; adjust the model parameters according to the loss value through the initial object detection model for iterative training to obtain the object detection model.

[0154] In an exemplary embodiment, the model training module is also used to perform multiple sampling processes on the detection image samples through the initial target detection model to obtain sampling feature maps at multiple scales; perform feature fusion on the sampling feature maps at multiple scales through the initial target detection model to obtain a fused feature map, and extract image feature samples of the detection image samples from the fused feature map; and use the image feature samples of the detection image samples to match the object feature samples through the initial target detection model to generate initial detection results of the object samples.

[0155] In an exemplary embodiment, the model training module is also used to determine the sampling feature values ​​of the target detection model in each channel based on the fusion feature map generated by the target detection model; prune the channels of the target detection model according to the sampling feature values ​​to obtain a processed pruned detection model; and determine the target detection model used on the edge device based on the operating efficiency of the pruned detection model.

[0156] In an exemplary embodiment, the model training module is also used to determine the pixel features and pixel gradients extracted by the pruning detection model based on the fusion feature map generated by the pruning detection model; generate the weight influence factor of the pixel based on the pixel features and pixel gradient; and perform sparse processing on the pruning detection model based on the weight influence factor to obtain the target detection model used on the edge device.

[0157] Each module in the above three-dimensional object detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0158] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as images to be detected, point cloud depth information, two-dimensional image features, target detection results, three-dimensional occupancy information, and three-dimensional target detection results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional target detection method is implemented.

[0159] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0160] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0162] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional target detection method, characterized in that: The method comprises: Acquire an image to be detected and point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model; Extracting two-dimensional image features in the image to be detected through the target detection model, and generating a target detection result of the target object according to the two-dimensional image features; Generating three-dimensional occupancy information of the target object according to the point cloud depth information by the target detection model; The target detection model generates a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information.

2. The method according to claim 1, characterized in that The obtaining of the image to be detected and the point cloud depth information includes: Acquire an initial image captured by a monocular image acquisition device; Using preset conversion parameters, the initial image is mapped to a virtual coordinate system corresponding to the monocular image acquisition device to obtain the image to be detected; Determine the point cloud depth information according to the initial image.

3. The method according to claim 1, characterized in that The pre-training method of the target detection model includes: Acquire detection image samples and point cloud depth samples, and input the detection image samples and the point cloud depth samples into an initial target detection model; Generating an initial detection result of an object sample according to the image feature sample of the detection image sample by the initial object detection model; Generate a loss value corresponding to the initial detection result according to the point cloud depth sample by the initial target detection model; The initial target detection model is trained iteratively by adjusting model parameters according to the loss value to obtain the target detection model.

4. The method according to claim 3, characterized in that: The generating an initial detection result of the object sample according to the image feature sample of the detection image sample by the initial target detection model includes: Performing multiple sampling processes on the detection image samples through the initial target detection model to obtain sampling feature maps at multiple scales; Performing feature fusion on the sampling feature maps at multiple scales through the initial target detection model to obtain a fused feature map, and extracting image feature samples of the detection image samples from the fused feature map; The initial target detection model uses the image feature samples of the detection image samples to match the object feature samples to generate the initial detection results of the object samples.

5. The method according to claim 4, characterized in that The method further comprises: Determine the sampling feature value of the target detection model in each channel according to the fusion feature map generated by the target detection model; Performing pruning processing on the channel of the target detection model according to the sampled feature values ​​to obtain a processed pruned detection model; According to the operating efficiency of the pruning detection model, the target detection model used in the edge device is determined.

6. The method according to claim 5, characterized in that The determining, according to the pruned target detection model, a target detection model used on the edge device includes: Determine pixel features and pixel gradients extracted by the pruning detection model according to the fused feature map generated by the pruning detection model; Generate a weight influence factor of the pixel point according to the pixel point feature and the pixel point gradient; The pruning detection model is sparsely processed according to the weight influence factor to obtain a target detection model used in the edge device.

7. A three-dimensional target detection device, characterized in that: The device comprises: A data acquisition module is used to acquire the image to be detected and the point cloud depth information, and input the image to be detected and the point cloud depth information into a pre-trained target detection model; A feature extraction module, used to extract two-dimensional image features in the image to be detected through the target detection model, and generate a target detection result of the target object according to the two-dimensional image features; An occupancy supervision module, configured to generate three-dimensional occupancy information of the target object according to the point cloud depth information through the target detection model; A feature fusion module is used to generate a three-dimensional target detection result of the target object according to the target detection result and the three-dimensional occupancy information through the target detection model.

8. The device according to claim 7, characterized in that The data acquisition module comprises: An image acquisition unit, used to acquire an initial image taken by a monocular image acquisition device; A coordinate system conversion unit, used to map the initial image to a virtual coordinate system corresponding to the monocular image acquisition device using preset conversion parameters to obtain an image to be detected; The depth acquisition unit is used to determine the point cloud depth information according to the initial image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images