Target detection method, device and equipment and computer readable storage medium

Abstract

The invention provides a target detection method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring to-be-detected image data and point cloud data corresponding to the image data, and respectively performing feature extraction on the image data and the point cloud data to correspondingly obtain a first image feature and a point cloud feature; performing depth prediction on the first image feature to obtain a depth feature, and splicing the depth feature and the first image feature to obtain a second image feature; performing projection processing on the second image feature and the point cloud feature to obtain a projected second image feature and a projected point cloud feature, and fusing the projected second image feature and the projected point cloud feature to obtain a fused feature; and performing target detection based on the fusion feature to obtain a target detection result. According to the invention, the accuracy of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target detection method, device, equipment and computer-readable storage medium. Background Art

[0002] The target detection technology based on the fusion of image and point cloud has important significance and wide application background in the fields of autonomous driving, robot navigation, intelligent monitoring, etc. Its main advantage lies in its ability to comprehensively utilize the characteristics of both image and point cloud data sources to improve the accuracy and robustness of target detection.

[0003] In the related art, when performing 3D target detection in the Bird's Eye View (BEV) space, the image features and point cloud features are directly fused, and then the fused features are input into the 3D target detection model. Since the two modal data have different degrees of difficulty in realizing 3D target detection, point clouds have a natural advantage over images in 3D target detection, and the target detection model that is finally trained does not give full play to the role of image features. At the same time, images are not sensitive to depth information, and image features cannot be accurately projected into the BEV space, resulting in low target detection accuracy. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, apparatus, device and computer-readable storage medium, which can improve the accuracy of target detection.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a method for detecting a target, which includes:

[0007] Collecting image data to be detected and point cloud data corresponding to the image data, performing feature extraction on the image data and the point cloud data respectively, and obtaining first image features and point cloud features correspondingly;

[0008] Performing depth prediction on the first image feature to obtain a depth feature, and concatenating the depth feature with the first image feature to obtain a second image feature;

[0009] Performing projection processing on the second image feature and the point cloud feature respectively to obtain a projected second image feature and a projected point cloud feature, and fusing the projected second image feature and the projected point cloud feature to obtain a fused feature;

[0010] Target detection is performed based on the fusion features to obtain a target detection result.

[0011] The present application provides a target detection device, including:

[0012] A feature extraction module, used to collect image data to be detected and point cloud data corresponding to the image data, perform feature extraction on the image data and the point cloud data respectively, and obtain first image features and point cloud features correspondingly;

[0013] A feature prediction module, configured to perform depth prediction on the first image feature to obtain a depth feature, and concatenate the depth feature with the first image feature to obtain a second image feature;

[0014] a feature fusion module, configured to perform projection processing on the second image feature and the point cloud feature respectively to obtain a projected second image feature and a projected point cloud feature, and fuse the projected second image feature and the projected point cloud feature to obtain a fused feature;

[0015] The target detection module is used to perform target detection based on the fusion features to obtain a target detection result.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the target detection method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the target detection method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the target detection method provided by the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By using the embodiment of the present application, the image data to be detected and the point cloud data corresponding to the image data are collected, and feature extraction is performed on the image data and the point cloud data respectively to obtain the first image feature and the point cloud feature respectively, and then the depth prediction is performed on the first image feature to obtain the depth feature, and the depth feature and the first image feature are spliced ​​to obtain the second image feature, and then the second image feature and the point cloud feature are projected respectively to obtain the projected second image feature and the projected point cloud feature, and the projected second image feature and the projected point cloud feature are fused to obtain the fused feature, and then target detection is performed based on the fused feature to obtain the target detection result. In this way, by performing depth prediction on the extracted first image feature to obtain the depth feature, and splicing the depth feature and the first image feature to obtain the second image feature, the second image feature carries depth information, which can improve the projection accuracy of the second image feature, and the target detection is performed in combination with the projected second image feature and the projected point cloud feature, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of an application mode of the target detection method provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3A It is a first flow chart of the target detection method provided in the embodiment of the present application;

[0026] Figure 3B is a second flow chart of the target detection method provided in an embodiment of the present application;

[0027] Figure 3C is a third flow chart of the target detection method provided in an embodiment of the present application;

[0028] Figure 4 It is a structural diagram of the training process of the target detection model provided in the embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of the reasoning process structure of the target detection model provided in the embodiment of the present application;

[0030] Figure 6 It is a structural diagram of a target detection model based on image and point cloud fusion provided in the related art;

[0031] Figure 7 It is a structural schematic diagram of an object detection model based on image and point cloud fusion provided in an embodiment of the present application;

[0032] Figure 8is a depth map of the sparse point cloud projection provided in the related art;

[0033] Fig. 9 It is a filled depth map provided in an embodiment of the present application.

[0034] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0042] 1) Image data: It is a collection of grayscale values ​​or color values ​​of each pixel represented by numerical values. In the fields of computer vision and image processing, image data is usually used to represent and store image content.

[0043] 2) Point cloud data: It is a three-dimensional coordinate information used to represent a set of points on the surface of an object or in space. Each point usually contains three coordinates, X, Y, and Z, to represent its position in three-dimensional space.

[0044] 3) Bird's Eye View (BEV): refers to the perspective of looking down on the scene from a high place, just like a bird observing the ground from the air.

[0045] 4) Batch Normalization (BN): It is a function that normalizes each small batch of data in the neural network by adjusting the mean and variance of the data to make it conform to the normal distribution.

[0046] 5) Two-dimensional convolution function: a function that performs convolution operations on a two-dimensional image.

[0047] 6) Non-linear activation function (Rectified Linear Unit, RELU): It is a function that introduces non-linear characteristics into the hidden layer of the neural network, enabling the network to learn and simulate more complex functions.

[0048] 7) Softmax function: It is an activation function applied to classification problems. In neural networks, the softmax function is usually used to convert the output of the last layer into a probability distribution and predict the category of the input data based on the probability.

[0049] The embodiments of the present application provide a target detection method, apparatus, device and computer-readable storage medium, which can improve the accuracy of target detection.

[0050] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0051] See also Figure 1 , Figure 1Schematic diagram of the application mode of the target detection method provided in the embodiment of the present application, for example, Figure 1 The invention involves a server 200, a network 300 and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0052] When performing target detection, the user triggers the target detection instruction on the terminal 400. The terminal 400 responds to the user's target detection instruction, collects the image data to be detected and the point cloud data corresponding to the image data, and sends the image data to be detected and the point cloud data corresponding to the image data and the target detection request to the server 200. The server 200 receives the image data to be detected and the point cloud data corresponding to the image data, and responds to the target detection request, respectively extracts features from the image data and the point cloud data, and obtains the first image feature and the point cloud feature. The first image feature is depth predicted to obtain the depth feature, and the depth feature and the first image feature are spliced ​​to obtain the second image feature. The second image feature and the point cloud feature are projected to obtain the projected second image feature and the projected point cloud feature, and the projected second image feature and the projected point cloud feature are fused to obtain the fused feature. Target detection is performed based on the fused feature to obtain the target detection result, and the target detection result is returned to the terminal 400. The terminal 400 receives the target detection result and displays it.

[0053] In some embodiments, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal 400 may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, vehicle-mounted terminal, aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0054] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0055] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0057] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0059] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0060] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0061] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A target detection device 455 stored in a memory 450 is shown, which may be software in the form of a program or plug-in, including the following software modules: a feature extraction module 4551, a feature prediction module 4552, a feature fusion module 4553, and a target detection module 4554. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0062] In some embodiments, the terminal or server can implement the target detection method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as cloud computing APP or instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0063] The target detection method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server device provided in the embodiment of the present application.

[0064] The target detection method provided by the embodiment of the present application is described below. As mentioned above, the electronic device implementing the target detection method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Next, the target detection method provided by the embodiment of the present application is described by taking the electronic device as a server as an example. Figure 3A , Figure 3A is a first flow chart of the target detection method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0065] In step 301, image data to be detected and point cloud data corresponding to the image data are collected, and feature extraction is performed on the image data and the point cloud data respectively to obtain first image features and point cloud features correspondingly.

[0066] Here, multi-view images of the target object to be detected at the same time are collected. The multi-view images include 6 camera images. The 6 camera images can be a front view image, a rear view image, a left front image, a left rear image, a right front image, and a right rear image. The image is converted into a digital signal, and the image is read into the memory. The image is preprocessed by image cropping to obtain image data. The image encoder network is used to extract features from the image data. The image encoder network includes a feature extraction network and a feature fusion network. The feature extraction network can be a deep residual network ResNet50, and the feature fusion network can be a neural network Transformer. The image data is recorded as a tensor [bs, N, 255, W, H], and the image data is input into the feature extraction network to extract multi-scale features. 8 samples are input as a batch, and each sample includes 6 camera images. Among them, bs is the batch size, N is the number of input images, 255 is the number of channels, and W and H are the width and height of the input image. The feature fusion network fuses the extracted multi-scale features. Feature fusion includes a series of processing such as upsampling and convolution on the multi-scale features to obtain a fusion result. The convolutional network layer is used to extract features from the fusion result to obtain the first image feature. The dimension of the first image feature is [N, 256, W / 8, H / 8], where N represents the number of camera images, 256 represents the number of channels, and W / 8 and H / 8 represent the width and height of the first image feature.

[0067] Point cloud data is collected by laser radar. Point cloud data refers to a set of vectors in a three-dimensional coordinate system, recorded in the form of points. Each point contains three-dimensional coordinates, and some points may have color information or reflection intensity information. The point cloud data corresponding to the timestamp of the image data to be detected is collected, that is, the image data to be detected and the point cloud data are collected for the target object in the same environment. The point cloud data is preprocessed by denoising, normalization, etc., and the point cloud data is converted into a format that can be processed by the neural network. Each point can be represented as a high-dimensional vector, where each dimension represents an attribute of the point, such as position, intensity, color, etc. A convolutional neural network (CNN) can be used to extract features from point cloud data. In the convolutional neural network, the point cloud data is subjected to convolution operations, pooling operations, and feature fusion processing to obtain point cloud features. The convolution operation is used to extract local features of the point cloud data, such as edges and textures. The pooling operation includes average pooling, which is used to reduce the size of the point cloud features while retaining the useful information of the point cloud data. Feature fusion includes obtaining point cloud features from multiple dimensions and fusing the point cloud features of multiple dimensions together.

[0068] Continue to refer Figure 3A In step 302, depth prediction is performed on the first image feature to obtain a depth feature, and the depth feature and the first image feature are concatenated to obtain a second image feature.

[0069] Here, depth prediction is to predict the depth of each feature point in the first image feature to obtain a predicted depth feature. The second image feature refers to an image feature with a depth feature, and the predicted depth feature and the first image feature are concatenated to obtain a second image feature with a depth feature.

[0070] In some embodiments, see Figure 3B , Figure 3B It is a second flow chart of the target detection method provided in an embodiment of the present application. Figure 3A In step 302 shown in the figure, "predicting the depth of the first image feature to obtain the depth feature" can be done by Figure 3B Steps 3021 to 3024 are implemented as described in detail below.

[0071] In step 3021, the image size of the image data and the feature size of the first image feature are determined, and an initial depth map is generated based on the image size.

[0072] Here, the characteristic size of the first image feature refers to the size of the first image feature, which can be expressed as the length × width in the first image feature. The initial depth map includes the depth values ​​corresponding to each initial pixel in different channels, and the initial pixel refers to each pixel constituting the initial depth map. A depth map is an image that represents the distance between each pixel in an image and the camera position, where the depth value of each pixel represents the distance from the corresponding pixel to the camera, usually in millimeters. Determine the image width W and image height H in the image size, and use multiple cameras to capture images from different perspectives at the same time, and use the triangulation principle to calculate the depth information of each pixel in the image. Combine the determined image size and the depth information of each pixel to generate an initial depth map.

[0073] In step 3022, the initial depth map is downsampled based on the feature size to obtain a sampled depth map.

[0074] Here, the sampled depth map includes the depth values ​​corresponding to multiple target pixels in different channels. The first image feature dimension is [N, 256, W / 8, H / 8], and the feature size of the first image feature is W / 8*H / 8. According to the feature size of the first image feature, the initial depth map is downsampled 8 times. The downsampling process includes a series of processes such as two-dimensional convolution function Conv2d, batch normalization function BN, and nonlinear activation function RELU to obtain the sampled depth map. The depth feature dimension corresponding to the sampled depth map is [N, 118, W / 8, H / 8], and the feature size of the depth feature is W / 8*H / 8, that is, the depth feature size is the same as the first image feature size.

[0075] For example, the size of the image data and the initial depth map are both 256*704, the first image feature dimension is [6, 256, 32, 88], and the feature size of the first image feature is 32*88. According to the feature size of the first image feature, the initial depth map is downsampled by 8 times to obtain a sampled depth map. The depth feature dimension corresponding to the sampled depth map is [6, 118, 32, 88], and the feature size of the depth feature is also 32*88.

[0076] In step 3023, for each target pixel, a depth value score corresponding to the depth value of each channel is determined, and the depth value corresponding to the maximum depth value score is determined as the predicted depth value of the target pixel.

[0077] After sampling, each target pixel in the depth map corresponds to multiple channels, and each channel corresponds to a depth value. For each target pixel, determine the number of channels and channel distances corresponding to the target pixel, divide the channel distances according to the number of channels, and obtain the depth value of each channel. The Softmax function is used to probabilistically output the depth value of each channel, obtain the depth value score corresponding to the depth value of each channel, and determine the target depth value corresponding to the maximum depth value score, and use the target depth value as the predicted depth value of the target pixel.

[0078] For example, the depth feature dimension corresponding to the sampled depth map is [6, 118, 32, 88]. The number of channels corresponding to each target pixel in the sampled depth map is 118, and the channel distance is 1m to 60m. The channel distance is divided into 118 parts according to the number of channels, and the interval of each channel distance is 0.5m. The depth value of the first channel is 1m, the depth value of the second channel is 1.5m, and the depth value of the third channel is 2m. And so on, the depth value of each channel is obtained. Determine the depth value score corresponding to the depth value of each channel. If the depth value corresponding to the maximum depth value score is 2m, the predicted depth value of the target pixel is 2m.

[0079] In step 3024, a depth feature is constructed based on the predicted depth value of the target pixel.

[0080] The spatial gradient of the predicted depth value of each target pixel is calculated to obtain the spatial rate of change of the predicted depth value, thereby constructing a depth feature containing the scene edge information.

[0081] In an embodiment of the present application, depth prediction is performed on the first image feature to obtain a depth feature, and the depth feature and the first image feature are subsequently spliced ​​to obtain a second image feature. The second image feature carries depth information and can improve the projection accuracy of the second image feature.

[0082] Continue to refer Figure 3AIn step 303, the second image features and the point cloud features are projected respectively to obtain projected second image features and projected point cloud features, and the projected second image features and the projected point cloud features are fused to obtain fused features.

[0083] Here, the projection processing is to project the second image features and point cloud features to the bird's eye view (BEV) space respectively. The bird's eye view space refers to the view space of overlooking the scene from a high place, which is collectively referred to as the BEV space below. The lift, split, and shoot algorithm, i.e., the LSS algorithm, can be used to project the second image features to the BEV space, and the depth of each feature point in the second image feature is estimated by using the camera's internal and external parameters, and each feature point is lifted to the corresponding position in the three-dimensional space. The lifted feature points are split along the camera optical axis to form a cone centered on the camera. The cone contains all feature points that may be visible in the BEV space, and the dimensions of the BEV space correspond to the height, width, and length of the cone. Finally, the feature points in the cone are projected onto the BEV plane along the camera optical axis. The Z-axis coordinates of the feature points are used as the height information in the BEV space, while the X- and Y-axis coordinates represent the two dimensions of the plane, and the projected second image features are obtained. Since the point cloud features are three-dimensional, when projecting the point cloud features into the BEV space, the point cloud features are directly compressed along the z-axis to obtain the projected point cloud features.

[0084] In some embodiments, see Figure 3C , Figure 3C It is a third flow chart of the target detection method provided in an embodiment of the present application. Figure 3A In step 303 shown in the figure, "the projected second image feature and the projected point cloud feature are fused to obtain a fused feature", which can be obtained by Figure 3C Steps 3031 to 3032 are implemented as described in detail below.

[0085] In step 3031, the projected second image features and the projected point cloud features are subjected to feature size adjustment to obtain target image features and target point cloud features having the same feature size.

[0086] After projecting the second image features and point cloud features into the BEV space respectively, the feature sizes of the projected second image features and the projected point cloud features will be different, which will cause the two features to be spatially misaligned when fused. A feature matching algorithm is used to determine the correspondence between the projected second image features and the projected point cloud features, and an alignment algorithm (such as the ICP algorithm) is used to adjust the projected point cloud features to align them with the projected second image features, thereby obtaining target image features and target point cloud features with the same feature sizes.

[0087] In step 3032, the target image features and the target point cloud features are spliced ​​to obtain fused features.

[0088] The target image features and target point cloud features are spliced ​​in the channel dimension and input into the fusion module based on the attention mechanism for processing. The fusion module performs global average pooling on the target image features and the target point cloud features respectively, and then both are processed through the fully connected layer to obtain the fused features.

[0089] In an embodiment of the present application, the projected second image features and the projected point cloud features are fused to obtain fused features. The fused features carry depth information, and subsequent target detection is performed based on the fused features, which can improve the accuracy of target detection.

[0090] Continue to refer Figure 3A In step 304, target detection is performed based on the fused features to obtain a target detection result.

[0091] Here, the fused features are input into the 3D target detection module to obtain the 3D target detection result. The 3D target detection result is to mark the target object in the image data to be detected in the form of a 3D detection frame. The annotation content includes the position of the 3D detection frame, the length, width and height of the 3D detection frame, the category of the 3D detection frame, etc.

[0092] In some embodiments, the target detection method provided by the embodiments of the present application can be implemented using a trained target detection model. The trained target detection model includes at least an image feature extraction module and a point cloud feature extraction module. Therefore, the image data and point cloud data can be respectively extracted by executing the following steps to obtain the first image feature and the point cloud feature respectively: obtain the trained target detection model; call the image feature extraction module in the trained target detection model to perform feature extraction on the image data to obtain the first image feature; call the point cloud feature extraction module in the trained target detection model to perform feature extraction on the point cloud data to obtain the point cloud feature.

[0093] Here, the image feature extraction module includes an image encoder network, and the image encoder network in the image feature extraction module is called to perform feature extraction processing on the image data to obtain the first image feature. The point cloud feature extraction module includes a convolutional neural network, and the convolutional neural network in the point cloud feature extraction module is called to perform feature extraction on the point cloud data to obtain the point cloud feature.

[0094] In the embodiment of the present application, the image data is subjected to feature extraction by the image feature extraction module in the trained target detection model, thereby improving the accuracy of extracting features of the image data. The point cloud data is subjected to feature extraction by the point cloud feature extraction module in the trained target detection model, thereby improving the accuracy of extracting features of the point cloud data.

[0095] In some embodiments, before obtaining the trained target detection model, the following steps can also be performed: obtain a sample data set and a target detection model to be trained, the sample data set including sample image data and three-dimensional annotation information corresponding to the sample image data; call the image feature extraction module in the target detection model to be trained to perform feature extraction on the sample image data to obtain a first sample image feature; perform two-dimensional target detection on the first sample image feature to obtain a two-dimensional target detection result; train the image feature extraction module based on the two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data to obtain a preliminary trained image feature extraction module, wherein the two-dimensional annotation information is determined using the three-dimensional annotation information; call the preliminary trained image feature extraction module to perform feature extraction on the sample image data to obtain a second sample image feature, and determine the three-dimensional target detection result based on the second sample image feature; train the target detection model to be trained using the three-dimensional target detection result and the three-dimensional annotation information to obtain a trained target detection model.

[0096] Here, the target detection model to be trained includes an image feature extraction module, and the image feature extraction module includes an image encoder network. The image encoder network in the image feature extraction module is called to perform feature extraction on the sample image data to obtain the first sample image feature. Performing two-dimensional target detection on the first sample image feature is to use a two-dimensional target detection model to perform target detection on the first sample image feature. The first sample image feature is input into the two-dimensional target detection model, and the two-dimensional target detection model includes multiple convolutional neural network layers. A rectangular detection frame that may contain the target object is generated by the two-dimensional target detection model. The target object category (for example: car, pedestrian, etc.) and the position of the rectangular detection frame in each rectangular detection frame are predicted according to the first sample image feature. Non-maximum suppression is performed on the predicted rectangular detection frame, and the rectangular detection frame with the highest confidence is retained to obtain a two-dimensional target detection result.

[0097] The 3D annotation information corresponding to the sample image data is the information obtained by manually annotating the sample image data with a 3D detection frame, marking the real position of the target object and the real category of the target object in the form of a 3D detection frame. The 3D annotation information corresponding to the sample image data is projected onto the sample image data through the camera extrinsic reference to obtain the 2D annotation information corresponding to the sample image data.

[0098] Based on the two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data, the value of the two-dimensional loss function is determined, and the model parameters of the image feature extraction module are updated according to the value of the two-dimensional loss function to train the image feature extraction module. The two-dimensional loss function can be a classification loss function, which represents the category loss of the rectangular detection box. The classification loss function can be a focal loss function FocalLoss, and the value of the two-dimensional loss function is calculated according to the focal loss function. The two-dimensional loss function can be a regression loss function, which represents the position loss of the rectangular detection box. The regression loss function can be a geometric loss function BBoxBEV, and the value of the two-dimensional loss function is calculated according to the geometric loss function. When the value of the two-dimensional loss function exceeds the preset threshold, the error signal of the feature extraction layer is determined based on the two-dimensional loss function, and the error signal is back-propagated in the image feature extraction module, and the model parameters of the image feature extraction module are updated during the propagation process to obtain a preliminarily trained image feature extraction module. The preliminarily trained image feature extraction module is called to perform feature extraction on the sample image data again to obtain the second sample image feature.

[0099] In an embodiment of the present application, by performing two-dimensional target detection on the first sample image feature, an image feature extraction module is trained based on the two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data, to obtain a preliminary trained image feature extraction module, and the preliminary trained image feature extraction module is called to extract the sample image feature again, thereby improving the accuracy of extracting the sample image feature.

[0100] In some embodiments, the target detection model to be trained also includes a depth prediction module, a point cloud feature extraction module and a three-dimensional target detection module, and the sample data set also includes sample point cloud data. The three-dimensional target detection result can be determined based on the second sample image feature by executing the following steps: calling the depth prediction module, performing depth prediction on the second sample image feature to obtain the predicted sample depth feature, calling the point cloud feature extraction module to perform feature extraction on the sample point cloud data to obtain the sample point cloud feature; splicing the sample depth feature and the second sample image feature to obtain the third sample image feature; projecting the third sample image feature and the sample point cloud feature respectively to obtain the projected third sample image feature and the projected sample point cloud feature, and fusing the projected third sample image feature and the projected sample point cloud feature to obtain the sample fusion feature; calling the three-dimensional target detection module to perform three-dimensional target detection on the sample fusion feature to obtain the three-dimensional target detection result.

[0101] Here, the depth prediction module is called to perform depth prediction on the second sample image feature. In the depth prediction module, the image size of the sample image data and the feature size of the second sample image feature are determined, and a sample initial depth map is generated based on the image size. The sample initial depth map includes the depth values ​​corresponding to each sample initial pixel in different channels. The sample initial depth map is downsampled based on the feature size to obtain a sample depth map after sampling. The sample depth map after sampling includes the depth values ​​corresponding to multiple sample target pixels in different channels. For each sample target pixel in the sampled sample depth map, the depth value score corresponding to the depth value of each channel is determined, and the depth value corresponding to the maximum depth value score is determined as the predicted depth value of the sample target pixel. Based on the predicted depth value of the sample target pixel, a predicted sample depth feature is obtained.

[0102] The point cloud feature extraction module includes a convolutional neural network. The convolutional neural network in the point cloud feature extraction module is called to extract features from the sample point cloud data to obtain sample point cloud features. The predicted sample depth features and the second sample image features are spliced ​​to obtain the third sample image features. The third sample image features and the sample point cloud features are projected into the BEV space respectively to obtain the projected third sample image features and the projected sample point cloud features. The projected third sample image features and the projected sample point cloud features are feature-sized to obtain sample target image features and sample target point cloud features with the same feature sizes. The sample target image features and the sample target point cloud features are spliced ​​to obtain sample fusion features.

[0103] The three-dimensional target detection module includes a three-dimensional target detection model. The three-dimensional target detection module is called to perform three-dimensional target detection on the sample fusion features, which is to use the three-dimensional target detection model to perform target detection on the sample fusion features. The sample fusion features are input into the three-dimensional target detection model. The three-dimensional target detection model can be a convolutional neural network or a deep learning model based on Transformer. The three-dimensional target detection model converts the sample fusion features into a two-dimensional BEV feature map. According to the two-dimensional BEV feature map, the target object category, the three-dimensional detection frame position and the three-dimensional detection frame size in the three-dimensional detection frame are predicted, and the predicted three-dimensional detection frame is non-maximum suppressed, and the three-dimensional detection frame with the highest confidence is retained to obtain the three-dimensional target detection result.

[0104] In an embodiment of the present application, by performing depth prediction on the second sample image feature to obtain a predicted sample depth feature, the sample depth feature and the second sample image feature are spliced ​​to obtain a third sample image feature, and the projected third sample image feature and the projected sample point cloud feature are fused to obtain a sample fusion feature, and three-dimensional target detection is performed on the sample fusion feature, which can improve the accuracy of three-dimensional target detection.

[0105] In some embodiments, the target detection model to be trained can be trained using the three-dimensional target detection results and the three-dimensional annotation information by executing the following steps to obtain a trained target detection model: determining a first loss value of the three-dimensional target detection module based on the three-dimensional target detection results and the three-dimensional annotation information; performing two-dimensional target detection on the second sample image feature to obtain an updated two-dimensional target detection result, and determining a second loss value of the image feature extraction module based on the updated two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data; projecting the sample point cloud data onto the sample image data to obtain a sample depth map, and determining the target sample depth features corresponding to the sample depth map, and determining a third loss value of the depth prediction module based on the predicted sample depth features and the target sample depth features; determining a total loss value of the target detection model to be trained based on the first loss value, the second loss value and the third loss value; and updating the model parameters of the target detection model to be trained based on the total loss value to obtain a trained target detection model.

[0106] Here, based on the three-dimensional target detection result and the three-dimensional annotation information, the first loss value of the three-dimensional target detection module is determined. The first loss value can be calculated by a classification loss function. The classification loss function represents the category loss of the three-dimensional detection frame. The classification loss function can be a focal loss function FocalLoss. The first loss value is calculated according to the focal loss function. The first loss value can also be calculated by a regression loss function. The regression loss function represents the position loss and size loss of the three-dimensional detection frame. The regression loss function can be a three-dimensional intersection over union loss function (Intersection Over Union ThreeDimensional, IOU3D). The first loss value is calculated according to the three-dimensional intersection over union loss function.

[0107] Perform two-dimensional target detection on the second sample image feature to obtain an updated two-dimensional target detection result, and determine the second loss value of the image feature extraction module based on the updated two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data. The second loss value can be calculated by a classification loss function, which can be a focal loss function FocalLoss, and the second loss value is calculated according to the focal loss function. The second loss value can also be calculated by a regression loss function, which can be a geometric loss function BBoxBEV, and the second loss value is calculated according to the geometric loss function.

[0108] The sample point cloud data is projected onto the sample image data to obtain a sample depth map, and the target sample depth feature corresponding to the sample depth map is determined. The predicted sample depth feature is upsampled so that the predicted sample depth feature after sampling has the same feature size as the target sample depth feature to determine the third loss value of the depth prediction module. The third loss value can be calculated by a classification loss function, which can be a cross entropy loss function CrossEntropy, and the third loss value is calculated according to the cross entropy loss function.

[0109] Based on the first loss value, the second loss value, and the third loss value, the total loss value of the target detection model to be trained is determined, and the model parameters of the target detection model to be trained are updated as a whole to obtain the trained target detection model. Different weights can be set for the first loss value, the second loss value, and the third loss value, and the first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value of the target detection model to be trained, and based on the total loss value, the model parameters of the target detection model to be trained are updated.

[0110] In an embodiment of the present application, the total loss value of the target detection model to be trained is determined based on the first loss value of the three-dimensional target detection module, the second loss value of the image feature extraction module and the third loss value of the depth prediction module. When the model parameters of the target detection model to be trained are updated based on the total loss value, the model parameters of the three-dimensional target detection module, the model parameters of the image feature extraction module and the model parameters of the depth prediction module are also updated at the same time to obtain the trained target detection model. When target detection is performed using the trained target detection model, the accuracy of target detection is improved.

[0111] In some embodiments, the sample point cloud data can be projected onto the sample image data to obtain a sample depth map by performing the following steps: projecting the sample point cloud data onto the sample image data to obtain an initial sample depth map; dividing the initial sample depth map into regions based on preset sizes to obtain multiple regional depth maps; for each regional depth map, determining the maximum depth value corresponding to the regional depth map, and updating the depth value of each pixel in the regional depth map to the maximum depth value to obtain an updated regional depth map; and constructing a sample depth map based on the multiple updated regional depth maps.

[0112] Here, the sample point cloud data is in the laser radar coordinate system, and the coordinate system of the sample image data is usually the camera coordinate system. According to the camera extrinsic matrix, the sample point cloud data is converted from the laser radar coordinate system to the camera coordinate system. After obtaining the sample point cloud coordinates in the camera coordinate system, the three-dimensional coordinates of the sample point cloud data are projected onto the two-dimensional sample image plane. Each sample point cloud data can find a corresponding pixel coordinate on the sample image plane. The pixel coordinate represents the projection of the sample point cloud data in the sample image data, and the initial sample depth map is obtained. The preset size can be N*N, where N is an integer greater than 1. The initial sample depth map is divided into N*N areas, and the maximum depth value corresponding to each area depth map is determined. For each area depth map, when the depth value of a pixel in the area depth map is 0, or the depth value of a pixel is not the maximum depth value, the depth value of the pixel is updated to the maximum depth value to obtain an updated area depth map, and multiple updated area depth maps are combined to form a sample depth map.

[0113] In an embodiment of the present application, for each regional depth map, the maximum depth value corresponding to the regional depth map is determined, and the depth value of each pixel in the regional depth map is updated to the maximum depth value. A sample depth map is constructed based on multiple updated regional depth maps. This can avoid the depth value of a certain pixel in the sample depth map being 0, so that when the loss value of the depth prediction module is determined based on the sample depth map, the loss value of the depth prediction module can be accurately calculated.

[0114] Example, reference Figure 4 , Figure 44 is a structural diagram of the training process of the target detection model provided by the embodiment of the present application. In the target detection model 401 to be trained, the sample fusion feature 402 is input into the three-dimensional target detection module 403 to obtain the three-dimensional target detection result. Based on the three-dimensional target detection result and the three-dimensional annotation information 404, the first loss value of the three-dimensional target detection module 403 is determined. The sample image data 405 is input into the image feature extraction module 406 for preliminary training to obtain the second sample image feature, and the second sample image feature is input into the two-dimensional target detection model 407 to obtain the updated two-dimensional target detection result. Based on the updated two-dimensional target detection result and the two-dimensional annotation information 408, the second loss value of the image feature extraction module 406 for preliminary training is determined. The depth prediction module 409 is called to perform depth prediction on the second sample image feature to obtain the predicted sample depth feature. Based on the predicted sample depth feature and the target sample depth feature 411 corresponding to the sample depth map, the third loss value of the depth prediction module 409 is determined. Based on the first loss value, the second loss value and the third loss value, the total loss value of the target detection model to be trained is determined. Based on the total loss value, the model parameters of the target detection model to be trained are updated to obtain the trained target detection model.

[0115] The following describes the reasoning process of the trained target detection model provided in the embodiment of the present application. Figure 5 , Figure 5 It is a structural diagram of the reasoning process of the target detection model provided in the embodiment of the present application. In the target detection model 501, the image feature extraction module 503 is called to perform feature extraction on the image data 502 to be detected to obtain the first image feature. The depth prediction module 504 is called to perform depth prediction on the first image feature to obtain the depth feature, and the depth feature and the first image feature are spliced ​​to obtain the second image feature. The point cloud feature extraction module 506 is called to perform feature extraction on the point cloud data 505 corresponding to the image data to obtain the point cloud feature. The second image feature and the point cloud feature are projected respectively to obtain the projected second image feature and the projected point cloud feature, and the projected second image feature and the projected point cloud feature are fused to obtain the fused feature. The three-dimensional target detection module 507 is called to perform target detection on the fused feature to obtain the target detection result.

[0116] In some embodiments, the target detection method provided by the embodiments of the present application can be applied in the field of autonomous driving. The cloud server collects the image data to be detected and the point cloud data corresponding to the image data, performs feature extraction on the image data and the point cloud data respectively, obtains the first image feature and the point cloud feature respectively, performs depth prediction on the first image feature to obtain the depth feature, and splices the depth feature and the first image feature to obtain the second image feature, performs projection processing on the second image feature and the point cloud feature respectively, obtains the projected second image feature and the projected point cloud feature, and fuses the projected second image feature and the projected point cloud feature to obtain the fused feature, performs target detection based on the fused feature, and obtains the target detection result. In this way, the cloud server obtains the depth feature by performing depth prediction on the extracted first image feature, splices the depth feature and the first image feature to obtain the second image feature. At this time, the second image feature carries depth information, which can improve the projection accuracy of the second image feature, and combines the projected second image feature and the projected point cloud feature for target detection, thereby improving the accuracy of target detection in autonomous driving.

[0117] The following describes an exemplary application of the target detection method provided in an embodiment of the present application in an autonomous driving scenario.

[0118] With the development of artificial intelligence, driverless taxis on urban roads and unmanned logistics vehicles in parks have all applied autonomous driving technology. One of the most important aspects of autonomous driving technology is how to perceive obstacles around the vehicle, and three-dimensional target detection plays an important role. Autonomous driving vehicles are equipped with various sensors, such as cameras and radars. The point cloud data generated by the radar can obtain the distance between the object in front and the vehicle, but cannot obtain texture information such as the color of the object. The image data generated by the camera can obtain the texture information of the object, but the distance between the object in front and the vehicle cannot be estimated using only two-dimensional images. Therefore, it is necessary to combine image data and point cloud data for target detection to give full play to their respective advantages.

[0119] Example, reference Figure 6 , Figure 6 It is a schematic diagram of the structure of the target detection model based on image and point cloud fusion provided in the related technology. Figure 6 The model structure in is used to explain the target detection method provided in the related art.

[0120] For example, a multi-view image 601 of 6 cameras is obtained, and the multi-view image 601 is extracted by an image feature extraction module 602 to obtain image features 603. The image feature extraction module 602 includes a ResNet50 network for extracting image features, or a SwinTransformer-Tiny network, etc., and then the image features 603 are projected in a bird's eye view (Bird's Eye View, BEV) space, hereinafter collectively referred to as BEV space. The algorithm for projection in the BEV space may be LSS (Lift, Splat, Shoot). After the image features 603 pass through the image projection module 604, image features 605 in the BEV space are obtained. At the same time, radar point cloud data 606 corresponding to the multi-view image 601 is obtained, and the radar point cloud data 606 passes through a point cloud feature extraction module 607 to obtain point cloud features 608. After the point cloud features 608 pass through a point cloud projection module 609, point cloud features 611 in the BEV space are obtained. The image feature 605 of the BEV space and the point cloud feature 611 of the BEV space are spliced ​​and passed through the BEV space encoding module 612 to obtain the fusion feature 613. The fusion feature 613 is input into the three-dimensional object detection module 614 to obtain the three-dimensional object detection result 615, which includes the length, width, height, center point and deflection angle of the three-dimensional detection box. The three-dimensional object detection result 615 and the three-dimensional annotation information 616 are subjected to loss calculation, and the entire object detection model is optimized according to the loss value.

[0121] The related technologies have the following problems in the target detection process:

[0122] In the 3D target detection based on the fusion of image and point cloud modalities, the image features and point cloud features are directly fused, and then the fused features are input into the 3D target detection model. Since the two modalities have different degrees of difficulty in realizing 3D target detection, the point cloud feature extraction model may be well trained, while the image feature extraction model may not have reached the fitting state. The target detection model that is finally trained does not give full play to the role of image features. At the same time, the image is not sensitive to depth information, and the image features cannot be accurately projected in the BEV space, resulting in low target detection accuracy.

[0123] In view of the problems existing in the related art, the present application embodiment proposes a target detection method, which includes the following improvements compared with the related art:

[0124] (1) The image feature extraction module is used to extract image features to obtain image features, and the image features are input into the two-dimensional target detection model to obtain the two-dimensional target detection results. The two-dimensional target detection results and the two-dimensional annotation information are subjected to a two-dimensional loss function calculation, and the image feature extraction module is optimized based on the two-dimensional loss function, thereby improving the accuracy of image feature extraction.

[0125] (2) Use the depth prediction module to perform depth prediction on the image features to obtain the predicted depth features. The predicted depth features and the depth map obtained by projecting the point cloud onto the image are used to calculate the depth loss function to obtain the depth loss value.

[0126] (3) Use the optimized image feature extraction module to extract features from the image again, input the extracted image features into the two-dimensional target detection model, and perform two-dimensional loss function calculation on the two-dimensional target detection results and the two-dimensional annotation information again to obtain a two-dimensional loss value. Concatenate the first image features and the depth features to obtain the target image features, project the target image features and the point cloud features in the BEV space, fuse the target image features and the point cloud features in the BEV space to obtain fused features, perform three-dimensional target detection based on the fused features, and perform three-dimensional loss function calculation on the three-dimensional target detection results and the three-dimensional annotation information to obtain a three-dimensional loss value. Based on the weights set for the two-dimensional loss value, the depth loss value, and the three-dimensional loss value, calculate the total loss value, and optimize the entire target detection model based on the total loss value, thereby improving the performance of the target detection model and thus improving the accuracy of target detection.

[0127] Example, reference Figure 7 , Figure 7 Schematic diagram of the structure of the target detection model based on image and point cloud fusion provided in the embodiment of the present application. Figure 7 The model structure in is used to explain the target detection method provided in the embodiment of the present application.

[0128] For example, a multi-view image 701 of 6 cameras is obtained, and the multi-view image 701 is extracted by an image feature extraction module 702 to obtain an image feature 703. The image feature 703 is input into a two-dimensional target detection model 717 to obtain a two-dimensional target detection result. The two-dimensional target detection result and the two-dimensional annotation information 718 are subjected to a two-dimensional loss calculation, and the image feature extraction module 702 is optimized to obtain a preliminarily trained image feature extraction module. The two-dimensional loss can be a classification loss or a regression loss. The classification loss includes a focal loss function FocalLoss, which represents the category loss of a rectangular detection box. The regression loss includes a geometric loss function BBoxBEVL1, a three-dimensional intersection-over-union loss function IOU3D, etc. The regression loss represents the position loss of the rectangular detection box. The geometric loss function calculates the loss between the true position of the rectangular detection box and the predicted position of the rectangular detection box. The three-dimensional intersection-over-union loss function calculates the loss between the width and height of the true rectangular detection box and the width and height of the predicted rectangular detection box. In the process of target detection, the image feature extraction module that has been initially trained is used to extract features from the image again, and the extracted image features are used for subsequent detection. The optimized image feature extraction module can better focus on the foreground information in the image.

[0129] When the image feature 703 is projected into the BEV space, that is, when the image feature 703 is input into the image projection module 704, the depth of each feature point is predicted in the depth prediction module 719. It is assumed that the feature dimension of the image feature 703 is [6, 256, 32, 88], where 6 represents 6 surround images, 256 represents the number of channels, and 32 and 88 represent the width and height of the feature. Initialize a depth map with the same size as the image. The image size can be set to 256*704, and the size of the depth map is also set to 256*704. According to the feature size of the image feature, the depth map is downsampled 8 times, and a series of downsampling is performed by combining the two-dimensional convolution layer function Conv2d, the batch normalization function BN, and the activation function ReLU to obtain a sampled depth map. The sampled depth map includes the depth values ​​corresponding to multiple pixels in different channels. The depth feature dimension corresponding to the sampled depth map is [6, 118, 32, 88], where 118 represents the number of channels, and each channel represents a distance value. For example, the distance of 1m to 60m is divided into 118 distance values ​​according to 0.5m. The first distance value is 1m, the second distance value is 1.5m, and so on. Then, each distance value is processed by the softmax activation function to obtain the score corresponding to each distance value, and the distance value corresponding to the maximum score is determined as the predicted depth value of each pixel, thereby constructing a predicted depth feature, and the predicted depth feature is upsampled to obtain an adjusted depth feature 720. The adjusted depth feature 720 and the depth map 721 are subjected to depth loss calculation. The depth loss can be a classification loss, and the classification loss can be a cross entropy loss function to optimize the depth prediction module 719. The optimized depth prediction module is used to perform depth prediction on the image features to obtain predicted depth features, and the image features and the predicted depth features are spliced ​​to obtain target image features. After the target image features pass through the image projection module 704, the image features 705 of the BEV space are obtained.

[0130] Example, reference Figure 8 , Figure 8 It is a depth map of sparse point cloud projection provided in the related art. The point cloud data is projected onto the image to obtain a depth map 721 to optimize the depth prediction module 719. The depth map obtained by projecting the sparse point cloud onto the image is also sparse, that is, some areas in the depth map have depth values, and some areas have depth values ​​of 0, which will interfere with the optimization of the depth prediction module.

[0131] Example, reference Fig. 9 , Fig. 9The filled depth map is provided in the embodiment of the present application. The depth map is divided into N*N regions, and then the maximum depth value corresponding to each region is determined, the maximum depth value is determined as the actual depth value corresponding to each region, and the actual depth value is used to fill the region without the depth value to obtain the filled depth map.

[0132] Continue to refer Figure 7 , the radar point cloud data 706 is passed through the point cloud feature extraction module 707 to obtain the point cloud feature 708, and the point cloud feature 708 is passed through the point cloud projection module 709 to obtain the point cloud feature 711 of the BEV space. The image feature 705 of the BEV space and the point cloud feature 711 of the BEV space are spliced, and after passing through the BEV space encoding module 712, a fusion feature 713 is obtained. The fusion feature 713 is input into the three-dimensional target detection module 714 to obtain the three-dimensional target detection result 715, and the three-dimensional target detection result 715 and the three-dimensional annotation information 716 are used to perform three-dimensional loss calculation to obtain a three-dimensional loss value. The image is extracted again using the preliminarily trained image feature extraction module, and the extracted image features are input into the two-dimensional target detection model, and the two-dimensional target detection results and the two-dimensional annotation information are used to perform two-dimensional loss function calculation to obtain a two-dimensional loss value. The depth prediction module is used to perform depth prediction on the extracted image features to obtain the predicted depth features. The predicted depth features and the depth map obtained by projecting the point cloud on the image are used to calculate the depth loss function to obtain the depth loss value. Based on the weights set for the two-dimensional loss value, the depth loss value, and the three-dimensional loss value, the total loss value is calculated, and the entire object detection model is optimized based on the total loss value.

[0133] In the above-mentioned application scenario of target detection in autonomous driving, the image data is subjected to feature extraction by using the optimized image feature extraction module to obtain image features, thereby improving the accuracy of image feature extraction. The image features are subjected to depth prediction to obtain depth features, and the image features and depth features are concatenated to obtain target image features, which are then projected into the BEV space to improve the accuracy of image feature projection into the BEV space. The target image features and point cloud features are fused in the BEV space to obtain fused features, and then target detection is performed based on the fused features, thereby improving the accuracy of target detection.

[0134] The following further describes an exemplary structure of the target detection device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2As shown, the software modules stored in the target detection device 455 of the memory 450 may include: a feature extraction module 4551, which is used to collect the image data to be detected and the point cloud data corresponding to the image data, and perform feature extraction on the image data and the point cloud data respectively to obtain the first image feature and the point cloud feature respectively; a feature prediction module 4552, which is used to perform depth prediction on the first image feature to obtain the depth feature, and splice the depth feature and the first image feature to obtain the second image feature; a feature fusion module 4553, which is used to perform project processing on the second image feature and the point cloud feature respectively to obtain the projected second image feature and the projected point cloud feature, and fuse the projected second image feature and the projected point cloud feature to obtain the fused feature; a target detection module 4554, which is used to perform target detection based on the fused feature to obtain the target detection result.

[0135] In some embodiments, the feature prediction module 4552 is also used to determine the image size of the image data and the feature size of the first image feature, generate an initial depth map based on the image size, the initial depth map includes the depth values ​​corresponding to each initial pixel in different channels; downsample the initial depth map based on the feature size to obtain a sampled depth map, the sampled depth map includes the depth values ​​corresponding to multiple target pixels in different channels; for each target pixel, determine the depth value score corresponding to the depth value of each channel, and determine the depth value corresponding to the maximum depth value score as the predicted depth value of the target pixel; based on the predicted depth value of the target pixel, construct a depth feature.

[0136] In some embodiments, the feature fusion module 4553 is also used to adjust the feature size of the projected second image features and the projected point cloud features to obtain target image features and target point cloud features with the same feature size; and to splice the target image features and the target point cloud features to obtain fused features.

[0137] In some embodiments, the feature extraction module 4551 is also used to obtain a trained target detection model; call the image feature extraction module in the trained target detection model to perform feature extraction on the image data to obtain a first image feature; call the point cloud feature extraction module in the trained target detection model to perform feature extraction on the point cloud data to obtain a point cloud feature.

[0138] In some embodiments, the feature extraction module 4551 is also used to obtain a sample data set and a target detection model to be trained before obtaining a trained target detection model, the sample data set including sample image data and three-dimensional annotation information corresponding to the sample image data; call the image feature extraction module in the target detection model to be trained to perform feature extraction on the sample image data to obtain a first sample image feature; perform two-dimensional target detection on the first sample image feature to obtain a two-dimensional target detection result; train the image feature extraction module based on the two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data to obtain a preliminary trained image feature extraction module, the two-dimensional annotation information being determined using the three-dimensional annotation information; call the preliminary trained image feature extraction module to perform feature extraction on the sample image data to obtain a second sample image feature, and determine the three-dimensional target detection result based on the second sample image feature; train the target detection model to be trained using the three-dimensional target detection result and the three-dimensional annotation information to obtain a trained target detection model.

[0139] In some embodiments, the target detection model to be trained also includes a depth prediction module, a point cloud feature extraction module and a three-dimensional target detection module. The sample data set also includes sample point cloud data. The feature extraction module 4551 is also used to call the depth prediction module to perform depth prediction on the second sample image feature to obtain the predicted sample depth feature, and call the point cloud feature extraction module to perform feature extraction on the sample point cloud data to obtain the sample point cloud feature; splice the sample depth feature and the second sample image feature to obtain the third sample image feature; project the third sample image feature and the sample point cloud feature respectively to obtain the projected third sample image feature and the projected sample point cloud feature, and fuse the projected third sample image feature and the projected sample point cloud feature to obtain the sample fusion feature; call the three-dimensional target detection module to perform three-dimensional target detection on the sample fusion feature to obtain the three-dimensional target detection result.

[0140] In some embodiments, the feature extraction module 4551 is also used to determine a first loss value of the three-dimensional target detection module based on the three-dimensional target detection result and the three-dimensional annotation information; perform two-dimensional target detection on the second sample image feature to obtain an updated two-dimensional target detection result, and determine the second loss value of the image feature extraction module based on the updated two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data; project the sample point cloud data onto the sample image data to obtain a sample depth map, and determine the target sample depth feature corresponding to the sample depth map, and determine the third loss value of the depth prediction module based on the predicted sample depth feature and the target sample depth feature; determine the total loss value of the target detection model to be trained based on the first loss value, the second loss value and the third loss value; based on the total loss value, update the model parameters of the target detection model to be trained to obtain the trained target detection model.

[0141] In some embodiments, the feature extraction module 4551 is also used to project the sample point cloud data onto the sample image data to obtain an initial sample depth map; divide the initial sample depth map into regions based on preset sizes to obtain multiple regional depth maps; for each regional depth map, determine the maximum depth value corresponding to the regional depth map, and update the depth value of each pixel in the regional depth map to the maximum depth value to obtain an updated regional depth map; and construct a sample depth map based on multiple updated regional depth maps.

[0142] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device performs the above-mentioned method of the embodiment of the present application.

[0143] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3A The target detection method is shown.

[0144] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0145] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0146] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0147] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0148] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Collecting image data to be detected and point cloud data corresponding to the image data, performing feature extraction on the image data and the point cloud data respectively, and obtaining first image features and point cloud features correspondingly; Performing depth prediction on the first image feature to obtain a depth feature, and concatenating the depth feature with the first image feature to obtain a second image feature; Performing projection processing on the second image feature and the point cloud feature respectively to obtain a projected second image feature and a projected point cloud feature, and fusing the projected second image feature and the projected point cloud feature to obtain a fused feature; Target detection is performed based on the fusion features to obtain a target detection result.

2. The method according to claim 1, characterized in that The performing depth prediction on the first image feature to obtain a depth feature includes: Determine an image size of the image data and a feature size of the first image feature, and generate an initial depth map based on the image size, wherein the initial depth map includes depth values ​​corresponding to each initial pixel point in different channels; Downsampling the initial depth map based on the feature size to obtain a sampled depth map, where the sampled depth map includes depth values ​​corresponding to a plurality of target pixels in different channels; For each target pixel, determine the depth value score corresponding to the depth value of each channel, and determine the depth value corresponding to the maximum depth value score as the predicted depth value of the target pixel; The depth feature is constructed based on the predicted depth value of the target pixel.

3. The method according to claim 1, characterized in that The step of fusing the projected second image feature and the projected point cloud feature to obtain a fused feature includes: Adjusting the feature sizes of the projected second image features and the projected point cloud features to obtain target image features and target point cloud features having the same feature sizes; The target image features and the target point cloud features are spliced ​​to obtain fusion features.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting features of the image data and the point cloud data respectively to obtain first image features and point cloud features accordingly includes: Get the trained target detection model; Calling the image feature extraction module in the trained object detection model to perform feature extraction on the image data to obtain the first image feature; The point cloud feature extraction module in the trained target detection model is called to perform feature extraction on the point cloud data to obtain the point cloud features.

5. The method according to claim 4, characterized in that Before obtaining the trained target detection model, the method further includes: Acquire a sample data set and a target detection model to be trained, wherein the sample data set includes sample image data and three-dimensional annotation information corresponding to the sample image data; Calling the image feature extraction module in the target detection model to be trained to perform feature extraction on the sample image data to obtain a first sample image feature; performing two-dimensional target detection on the first sample image feature to obtain a two-dimensional target detection result; Based on the two-dimensional target detection result and the two-dimensional annotation information corresponding to the sample image data, training the image feature extraction module to obtain a preliminarily trained image feature extraction module, wherein the two-dimensional annotation information is determined using the three-dimensional annotation information; Calling the preliminarily trained image feature extraction module to perform feature extraction on the sample image data to obtain second sample image features, and determining a three-dimensional object detection result based on the second sample image features; The target detection model to be trained is trained using the three-dimensional target detection result and the three-dimensional annotation information to obtain a trained target detection model.

6. The method according to claim 5, characterized in that The target detection model to be trained further includes a depth prediction module, a point cloud feature extraction module and a three-dimensional target detection module, the sample data set further includes sample point cloud data, and the three-dimensional target detection result is determined based on the second sample image feature, including: Calling the depth prediction module to perform depth prediction on the second sample image feature to obtain predicted sample depth features, and calling the point cloud feature extraction module to perform feature extraction on the sample point cloud data to obtain sample point cloud features; Concatenate the sample depth feature and the second sample image feature to obtain a third sample image feature; Performing projection processing on the third sample image feature and the sample point cloud feature respectively to obtain projected third sample image features and projected sample point cloud features, and fusing the projected third sample image features and the projected sample point cloud features to obtain sample fusion features; The three-dimensional object detection module is called to perform three-dimensional object detection on the sample fusion features to obtain a three-dimensional object detection result.

7. The method according to claim 6, characterized in that The method of training the target detection model to be trained by using the three-dimensional target detection result and the three-dimensional annotation information to obtain a trained target detection model includes: Determining a first loss value of the three-dimensional object detection module based on the three-dimensional object detection result and the three-dimensional annotation information; Performing two-dimensional object detection on the second sample image feature to obtain an updated two-dimensional object detection result, and determining a second loss value of the image feature extraction module based on the updated two-dimensional object detection result and the two-dimensional annotation information corresponding to the sample image data; Projecting the sample point cloud data onto the sample image data to obtain a sample depth map, determining a target sample depth feature corresponding to the sample depth map, and determining a third loss value of the depth prediction module based on the predicted sample depth feature and the target sample depth feature; Determine a total loss value of the object detection model to be trained based on the first loss value, the second loss value, and the third loss value; Based on the total loss value, the model parameters of the target detection model to be trained are updated to obtain a trained target detection model.

8. The method according to claim 7, characterized in that The step of projecting the sample point cloud data onto the sample image data to obtain a sample depth map includes: Projecting the sample point cloud data onto the sample image data to obtain an initial sample depth map; Dividing the initial sample depth map into regions based on a preset size to obtain a plurality of regional depth maps; For each regional depth map, determining a maximum depth value corresponding to the regional depth map, and updating the depth value of each pixel in the regional depth map to the maximum depth value, to obtain an updated regional depth map; The sample depth map is constructed based on the multiple updated regional depth maps.

9. A target detection device, characterized in that: The device comprises: A feature extraction module is used to collect image data to be detected and point cloud data corresponding to the image data, and perform feature extraction on the image data and the point cloud data respectively to obtain first image features and point cloud features correspondingly; A feature prediction module, configured to perform depth prediction on the first image feature to obtain a depth feature, and concatenate the depth feature with the first image feature to obtain a second image feature; a feature fusion module, configured to project the second image feature and the point cloud feature respectively to obtain a projected second image feature and a projected point cloud feature, and fuse the projected second image feature and the projected point cloud feature to obtain a fused feature; The target detection module is used to perform target detection based on the fusion features to obtain a target detection result.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the target detection method described in any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the target detection method according to any one of claims 1 to 8 is implemented.

Images