Three-dimensional target detection method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a three-dimensional target detection method, an electronic device and a storage medium. The method comprises: obtaining a target image; inputting the target image into a three-dimensional target detection network to obtain a three-dimensional detection result of the target image, the three-dimensional target detection network being configured to: obtain semantic features and depth features of the target image; obtain a view cone feature of the target image based on the semantic features and the depth features; construct a plurality of corresponding view cone feature pixels in the view cone feature into corresponding bird's eye view angle feature channels to obtain a bird's eye view angle feature of the target image; and obtain the three-dimensional detection result of the target image based on the bird's eye view angle feature.

Description

Technical Field

[0001] This application relates to the field of computer vision, specifically to three-dimensional target detection methods, electronic devices, and storage media. Background Technology

[0002] Current 3D object detection networks, such as CaDDN, first convert the features of the image used for 3D object detection into features related to the view frustum space when performing object detection. Then, they sample or pool the features related to the view frustum space to obtain features related to the bird's-eye view. They then use the features related to the bird's-eye view to perform object detection and obtain the 3D detection result.

[0003] However, the computational cost of sampling or pooling features related to the view frustum is high, resulting in a slow speed of obtaining features related to the bird's-eye view and thus a slow speed of obtaining 3D detection results. Current 3D object detection networks cannot be applied to devices with high requirements for obtaining 3D detection results, such as autonomous vehicles. Summary of the Invention

[0004] This application provides a three-dimensional target detection method, an electronic device, and a storage medium.

[0005] This application provides a three-dimensional target detection method, including:

[0006] Acquire the target image;

[0007] The target image is input into a 3D target detection network to obtain the 3D detection result of the target image. The 3D target detection network is configured as follows:

[0008] Obtain the semantic and depth features of the target image;

[0009] Based on the semantic features and the depth features, the frustum features of the target image are obtained;

[0010] The corresponding multiple frustum feature pixels in the frustum feature are constructed into corresponding bird's-eye view feature channels to obtain the bird's-eye view features of the target image.

[0011] Based on the bird's-eye view features, the three-dimensional detection results of the target image are obtained.

[0012] This application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described three-dimensional target detection method.

[0013] This application provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the above-described three-dimensional target detection method.

[0014] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the above-described three-dimensional target detection method.

[0015] The 3D target detection method provided in this application embodiment acquires a target image; inputs the target image into a 3D target detection network to obtain the 3D detection result of the target image. The 3D target detection network is configured to: acquire semantic features and depth features of the target image; obtain frustum features of the target image based on the semantic features and depth features of the target image; construct corresponding bird's-eye view feature channels from multiple frustum feature pixels in the frustum features to obtain bird's-eye view features of the target image; and obtain the 3D detection result of the target image based on the bird's-eye view features. It eliminates the need for sampling or pooling features related to the frustum space, thus allowing for faster acquisition of bird's-eye view features and 3D detection results. The 3D target detection method provided in this application embodiment can be applied to devices with high speed requirements for obtaining 3D detection results. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] Figure 1 A flowchart of the three-dimensional target detection method provided in an embodiment of this application is shown;

[0018] Figure 2 A structural block diagram of the three-dimensional target detection device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Figure 1A flowchart of a three-dimensional target detection method provided in an embodiment of this application is shown. The method includes:

[0022] Step 101: Obtain the target image.

[0023] In this application, the target image can be captured by a camera mounted on a vehicle, such as an autonomous vehicle.

[0024] Step 102: Input the target image into the 3D target detection network to obtain the 3D detection result of the target image.

[0025] In this application, a 3D object detection network is configured to acquire semantic features and depth features of a target image; obtain the frustum features of the target image based on the semantic features and depth features of the target image; construct corresponding multiple frustum feature pixels in the frustum features of the target image into corresponding bird's-eye-view (BEV) feature channels to obtain the bird's-eye-view (BEV) features of the target image; and obtain the 3D detection result of the target image based on the BEV features of the target image.

[0026] In this application, the semantic features of the target image have multiple semantic feature channels, and each semantic feature channel is a feature map.

[0027] The number of semantic feature channels in the semantic features of a target image can be referred to as the number of semantic feature channels of the target image.

[0028] The height dimension of the semantic feature channel of the target image is: the number of rows in the semantic feature channel of the target image.

[0029] The width dimension of the semantic feature channel of the target image is: the number of columns in the semantic feature channel of the target image.

[0030] For each semantic feature channel in the semantic features of the target image, the number of semantic feature pixels in that semantic feature channel is: the height dimension of the semantic feature channel of the target image * the width dimension of the semantic feature channel of the target image.

[0031] The height dimension of the semantic feature channel of the target image can be 16, the width dimension of the semantic feature channel of the target image can be 44, the number of semantic feature channels of the target image can be 256, and each semantic feature channel of the target image can have 16*44 semantic feature pixels.

[0032] In this application, when obtaining the semantic features of a target image, the target image can be input into a convolutional neural network, such as ResNet or MobileNet, for extracting the semantic features of the image, and the convolutional neural network for extracting the semantic features of the image outputs the semantic features of the target image.

[0033] In this application, the depth features of the target image have multiple depth feature channels, and each depth feature channel is a feature map.

[0034] The number of depth feature channels in the depth features of a target image can be referred to as the number of depth feature channels of the target image.

[0035] Each depth feature channel of the target image corresponds to a different preset depth, and the number of depth feature channels of the target image is the same as the number of preset depths.

[0036] The preset depth can have 112 channels, and the preset depth can have 112 depth feature channels.

[0037] The height dimension of the depth feature channel of the target image is the number of rows in the depth feature channel of the target image. The height dimension of the depth feature channel of the target image is the same as the height dimension of the semantic feature channel of the target image.

[0038] The width dimension of the depth feature channel of the target image is the number of columns in the depth feature channel of the target image. The width dimension of the depth feature channel of the target image is the same as the width dimension of the semantic feature channel of the target image.

[0039] For each depth feature channel in the depth features of the target image, the number of depth feature pixels in that depth feature channel is: the height dimension of the depth feature channel of the target image * the width dimension of the depth feature channel of the target image.

[0040] The height dimension of the depth feature channel of the target image can be 16, the width dimension of the depth feature channel of the target image can be 44, the number of depth feature channels of the target image can be 112, and each depth feature channel of the depth feature of the target image has 16*44 depth feature pixels.

[0041] In this application, when acquiring the depth features of a target image, the depth features of the target image can be obtained based on the intrinsic parameters of the camera that acquired the target image, the semantic features of the target image, and a first prior relation. The first prior relation represents the association between the depth features of a given image and the intrinsic parameters of the camera that acquired the given image and the semantic features of the given image.

[0042] In this application, the depth feature channel is denoted as D. The position of a depth feature pixel is (Dr, Ds, Dt), which indicates that the depth feature pixel is the depth feature pixel in the s-th row of the r-th depth feature channel and the depth feature pixel is the depth feature pixel in the t-th column of the r-th depth feature channel.

[0043] In this application, a deep feature channel D r The position of (D) in r D s D t The depth feature pixels can be: the semantic feature pixels corresponding to the depth feature pixels in the corresponding D r The probability at the preset depth.

[0044] The semantic feature channel is denoted as F, and the position of a semantic feature pixel is (F o F p F q The position of the semantic feature pixel indicates that the semantic feature pixel is the semantic feature pixel in the p-th row of the o-th semantic feature channel and the semantic feature pixel is the semantic feature pixel in the q-th column of the o-th semantic feature channel.

[0045] Let n be the number of channels for the semantic features of the target image, and let n be the number of channels for the semantic features at position (D). r D s D t The depth feature pixel has a position (F1, F) s F t The semantic feature pixels of ) have positions (F2, F s F t The semantic feature pixel of ) has a position of (F n F s F t The semantic feature pixels of ) all correspond to the depth feature pixels.

[0046] In this application, when the view frustum features of a target image are obtained based on the semantic features and depth features of the target image, the view frustum features of the target image can be obtained according to the depth features, semantic features, and a second prior relation. The second prior relation represents the association between the view frustum features of a given image and the semantic features and depth features of a given image.

[0047] In this application, the view frustum feature of the target image has multiple view frustum feature pixels.

[0048] The corresponding frustum feature pixels in the target image frustum feature are obtained by weighting the corresponding semantic feature pixels using the probabilities of the corresponding semantic feature pixels at the corresponding preset depth.

[0049] For any semantic feature pixel and any preset depth, the semantic feature pixel is weighted by the probability of the semantic feature pixel at the preset depth to obtain a view frustum feature pixel. The obtained view frustum feature pixel can be called the related view frustum feature pixel of the semantic feature pixel at the preset depth.

[0050] For any semantic feature pixel and any preset depth, the semantic feature pixel can be weighted by the probability of the semantic feature pixel at the preset depth: multiply the probability of the semantic feature pixel at the preset depth by the semantic feature pixel itself, and the product of the probability of the semantic feature pixel at the preset depth and the semantic feature pixel can be used as a related frustum feature pixel of the semantic feature pixel at the preset depth.

[0051] For each semantic feature pixel in any semantic feature channel, the probability of that semantic feature pixel at each preset depth is used to obtain multiple corresponding frustum feature pixels.

[0052] In this application, the number of channels for the depth features of the target image is the same as the number of channels for the preset depth.

[0053] The number of frustum feature pixels in the frustum feature is: the number of channels of the semantic features of the target image * the number of channels of the depth features of the target image * the height dimension of the semantic feature channels of the target image * the width dimension of the semantic feature channels of the target image.

[0054] In this application, the height dimension of the semantic feature channel of the target image is the same as the height dimension of the depth feature channel of the target image, and the width dimension of the semantic feature channel of the target image is the same as the width dimension of the depth feature channel of the target image.

[0055] The number of channels for the semantic features of the target image can be 256, the number of channels for the depth features of the target image can be 112, the height dimension of the semantic feature channels of the target image can be 16, the width dimension of the semantic feature channels of the target image can be 44, and the number of frustum feature pixels in the frustum features can be 256*112*16*44.

[0056] In this application, the bird's-eye view features of the target image include: multiple bird's-eye view feature channels. Each bird's-eye view feature channel is a feature map, and each bird's-eye view feature channel has multiple bird's-eye view feature pixels.

[0057] In this application, the height dimension of the bird's-eye view feature channel of the target image is the number of rows in the bird's-eye view feature channel of the target image, and the height dimension of the bird's-eye view feature channel of the target image is the same as the number of channels of the depth feature of the target image.

[0058] The width dimension of the bird's-eye view feature channel of the target image is: the number of columns in the bird's-eye view feature channel of the target image. The width dimension of the bird's-eye view feature channel of the target image is the same as the width dimension of the semantic feature channel of the target image.

[0059] The number of bird's-eye view feature channels in the bird's-eye view feature of a target image can be referred to as the number of channels of the bird's-eye view feature of the target image.

[0060] The number of channels for the bird's-eye view features of the target image is: the number of channels for the semantic features of the target image * the height dimension of the semantic feature channels of the target image.

[0061] The height dimension of the semantic feature channels of the target image can be 16, the width dimension of the semantic feature channels of the target image can be 44, and the number of semantic feature channels of the target image can be 256.

[0062] The height dimension of the depth feature channel of the target image can be 16, the width dimension of the depth feature of the target image can be 44, and the number of channels of the depth feature of the target image can be 112.

[0063] The height dimension of the bird's-eye view feature channel of the target image can be 112, the width dimension of the bird's-eye view feature channel of the target image can be 44, and the number of channels of the bird's-eye view feature of the target image can be 4096.

[0064] In this application, multiple corresponding frustum feature pixels in the frustum feature of the target image are constructed as corresponding bird's-eye view feature channels to obtain the bird's-eye view features of the target image.

[0065] For each frustum feature pixel in the frustum feature of the target image, the position of the frustum feature pixel in the bird's-eye view feature can be determined. The position of the frustum feature pixel in the bird's-eye view feature indicates which bird's-eye view feature channel the frustum feature pixel belongs to, which row in the corresponding bird's-eye view feature channel the frustum feature pixel belongs to, and which column in the corresponding bird's-eye view feature channel the frustum feature pixel belongs to.

[0066] For each frustum feature pixel in the target image's frustum feature, after determining its position in the bird's-eye view feature, all frustum feature pixels belonging to the same bird's-eye view feature channel can be identified based on their positions. All frustum feature pixels belonging to the same bird's-eye view feature channel constitute a corresponding bird's-eye view feature channel. Thus, all bird's-eye view feature channels of the target image can be obtained, resulting in the bird's-eye view feature of the target image.

[0067] The following example illustrates the process of determining the position of a view frustum feature pixel in the bird's-eye view feature. The process of determining the positions of other view frustum feature pixels in the bird's-eye view feature is similar:

[0068] The height dimension of the semantic feature channel of the target image is the number of rows in the semantic feature channel of the target image. The height dimension of the semantic feature channel of the target image is denoted as h, the number of semantic feature channels of the target image is denoted as n, the bird's-eye view feature is denoted as B, and the view frustum feature is denoted as V.

[0069] For a view frustum feature pixel V(c, i, j, d) k V(c, i, j, d) k ) is the position (F) c F i F j The semantic feature pixels at a preset depth d k Relevant frustum feature pixels;

[0070] c is one of 1, 2...n, and has a position (F c F i F j The semantic feature pixels of the c-th semantic feature channel are the semantic feature pixels that belong to the i-th row and the j-th column.

[0071] d k For the k-th preset depth among all preset depths, V(c, i, j, d) k By utilizing the position of (F) c F i F j The semantic feature pixels of ) in d k The probability pair has a position (F) c F i F j The semantic feature pixels are weighted to obtain the result;

[0072] V(c, i, j, d) k ) belongs to the j-th row of the Bx-th bird's-eye view feature channel, x = h*(c-1)+i;

[0073] V(c, i, j, d) k It belongs to the kth column of the Bx-th bird's-eye view feature channel.

[0074] In this application, the three-dimensional detection results of the target image are obtained based on the bird's-eye view features of the target image.

[0075] In this application, the 3D target detection network includes a detector, which may be called CenterPointHead, which can convert the bird's-eye view features of the target image into the input vector of the detector. Each bird's-eye view feature channel of the target image is a component of the input vector of the detector. The input vector of the detector is input into the detector, and the detector outputs the 3D detection result of the target image.

[0076] The 3D detection results of the target image include: the detection results for each detected target in the target image. The detection results for each detected target can include: the type of the detected target, the x-axis coordinates of the center point of the 3D bounding box of the detected target in 3D space, the y-axis coordinates of the center point of the 3D bounding box of the detected target in 3D space, the z-axis coordinates of the center point of the 3D bounding box of the detected target in 3D space, the length of the 3D bounding box of the detected target, the width of the 3D bounding box of the detected target, and the height of the 3D bounding box of the detected target. Detected targets include pedestrians, vehicles, obstacles, etc. The 3D bounding box of the detected target is the bounding box that encloses the detected target in 3D space.

[0077] In some embodiments, constructing a corresponding plurality of frustum feature pixels in the frustum features of the target image into a corresponding bird's-eye view feature channel to obtain the bird's-eye view features of the target image includes: performing a bird's-eye view feature channel construction operation on each semantic feature channel of the semantic features of the target image respectively. For each semantic feature channel, the bird's-eye view feature channel construction operation for that semantic feature channel includes: for each row in the semantic feature channel, constructing a bird's-eye view feature channel corresponding to that row. Constructing a bird's-eye view feature channel corresponding to that row includes: for each semantic feature pixel in that row, constructing a row of bird's-eye view feature channels corresponding to that row from the related frustum feature pixels of that semantic feature pixel at each preset depth.

[0078] The height dimension of the semantic feature channel of the target image is the number of rows in the semantic feature channel of the target image. For each semantic feature channel, the bird's-eye view feature channel is obtained by constructing the bird's-eye view feature channel of the semantic feature channel. The height dimension of the semantic feature channel of the target image is then obtained.

[0079] The bird's-eye view feature channel construction operation is performed on each semantic feature channel of the target image. The number of bird's-eye view feature channels is: the number of semantic feature channels of the target image * the height dimension of the semantic feature channels of the target image.

[0080] In this application, the number of channels for the semantic features of the target image can be 256, the height dimension of the bird's-eye view feature channels of the target image can be 112, and the number of channels for the bird's-eye view features of the target image can be 4096.

[0081] The following example illustrates the process of constructing a bird's-eye view feature channel corresponding to a row of a semantic feature channel. The process of constructing a bird's-eye view feature channel corresponding to other rows of other semantic feature channels is similar:

[0082] The height dimension of the semantic feature channels of the target image is denoted as h, the number of semantic feature channels of the target image is denoted as n, the number of preset depths is denoted as m, and the bird's-eye view features are denoted as B.

[0083] c is one of 1, 2...n. For the i-th row in semantic feature channel c, the bird's-eye view feature channel corresponding to the i-th row in semantic feature channel c is: the Bx-th bird's-eye view feature channel, x = h*(c-1)+i, where i is one of 1, 2...h.

[0084] The position (F) in semantic feature channel c c F i F j The semantic feature pixels of ) are: the semantic feature pixels in the c-th semantic feature channel that belong to the i-th row and the j-th column, and will have the position (F c F i F j The semantic feature pixels of ) are constructed into the j-th row of the Bx-th bird's-eye view feature channel by the relevant frustum feature pixels at each preset depth, that is, the position (F) c F i F j The semantic feature pixels of ) in the relevant view frustum feature pixels of d1 will have the position (F) c F i F j The semantic feature pixels of ) in the relevant frustum feature pixels of d2... will have the position (F c F i F j The semantic feature pixels of ) in d m The relevant view frustum feature pixels are constructed as the j-th row in the Bx-th bird's-eye view feature channel;

[0085] The j-th row in the Bx-th bird's-eye view feature channel includes: the position (F) c F i F j The semantic feature pixels of ) are the relevant frustum feature pixels at each preset depth;

[0086] Having position (F) c F i F j The semantic feature pixels at a preset depth d k The relevant view frustum feature pixels belong to the k-th column of the Bx-th bird's-eye view feature channel, where k is one of 1, 2...m, and the preset depth d k This is the k-th preset depth among all preset depths. The order of the preset depths is obtained by sorting all preset depths from smallest to largest.

[0087] In some embodiments, obtaining a 3D detection result of a target image based on its bird's-eye view features includes: processing the bird's-eye view features of the target image using a unit for processing bird's-eye view features to obtain processed bird's-eye view features, wherein the unit for processing bird's-eye view features includes multiple convolutional layers; converting the processed bird's-eye view features into an input vector of a detector in a 3D target detection network; and inputting the input vector of the detector into the detector to obtain the 3D detection result of the target image output by the detector.

[0088] In this application, the number of convolutional layers in the unit used to process bird's-eye view features is denoted as n. The k-th convolutional layer in the unit is connected to the (k-1)-th convolutional layer, where k is one of 2, 3, ..., n. The input of the first convolutional layer in the unit used to process bird's-eye view features is the bird's-eye view feature, and the output of the last convolutional layer in the unit used to process bird's-eye view features is the processed bird's-eye view feature. The processed bird's-eye view feature is converted into the input vector of the detector in the 3D object detection network, and each channel in the processed bird's-eye view feature is a component of the detector's input vector. The input vector of this detector is input into the detector, and the detector outputs the 3D detection result of the target image.

[0089] In some embodiments, obtaining semantic and depth features of a target image includes: processing the intrinsic parameters of the camera acquiring the target image using a fully connected layer to obtain a weight vector, wherein each weight in the weight vector corresponds to a semantic feature channel in the semantic features of the target image; for each semantic feature channel of the semantic features of the target image, weighting the semantic feature channel using the weight corresponding to that semantic feature channel to obtain a weighted semantic feature channel; inputting all the obtained weighted semantic feature channels into a unit for extracting depth features to obtain the depth features of the target image output by the unit for extracting depth features, wherein the unit for extracting depth features includes multiple convolutional layers.

[0090] In this application, a fully connected layer can be used to process the intrinsic parameters of the camera that acquires the target image to obtain a weight vector. Each component of the weight vector is a weight, and each weight corresponds to a semantic feature channel in the semantic features of the target image.

[0091] The semantic features of the target image can have 256 semantic feature channels, and the dimension of the weight vector can be 256.

[0092] For each semantic feature channel of the target image, the semantic feature channel is weighted using the weight corresponding to that semantic feature channel to obtain a weighted semantic feature channel. This weighting process includes: multiplying each semantic feature pixel in the semantic feature channel by the weight corresponding to that semantic feature channel to obtain a weighted semantic feature pixel for each semantic feature pixel; and replacing each semantic feature pixel in the semantic feature channel with its weighted semantic feature pixel to obtain the weighted semantic feature channel. The weighted semantic feature channel includes the weighted semantic feature pixel of each semantic feature pixel in the semantic feature channel.

[0093] All weighted semantic feature channels can be input into a unit for extracting depth features. The unit for extracting depth features includes multiple convolutional layers and outputs the depth features of the target image.

[0094] The number of convolutional layers in the unit used for extracting depth features is denoted as n. The k-th convolutional layer in the unit used for extracting depth features is connected to the (k-1)-th convolutional layer, where k is one of 2, 3, ..., n. The input of the first convolutional layer in the unit used for extracting depth features is all the weighted semantic feature channels obtained, and the last convolutional layer outputs the depth features of the target image.

[0095] In some embodiments, obtaining the frustum features of the target image based on the semantic and depth features of the target image includes: taking the outer product of the depth features and the semantic features of the target image to obtain the frustum features of the target image.

[0096] Please refer to Figure 2 The diagram illustrates the structural block diagram of the three-dimensional target detection device provided in an embodiment of this application. The three-dimensional target detection device includes: an acquisition unit 201 and a detection unit 302.

[0097] The acquisition unit 201 is configured to acquire the target image;

[0098] The detection unit 202 is configured to input the target image into a three-dimensional target detection network to obtain a three-dimensional detection result of the target image. The three-dimensional target detection network is configured to: acquire semantic features and depth features of the target image; obtain frustum features of the target image based on the semantic features and the depth features; construct corresponding multiple frustum feature pixels in the frustum features into corresponding bird's-eye view feature channels to obtain bird's-eye view features of the target image; and obtain a three-dimensional detection result of the target image based on the bird's-eye view features.

[0099] In some embodiments, constructing a corresponding plurality of frustum feature pixels in the frustum features into a corresponding bird's-eye view feature channel to obtain the bird's-eye view features of the target image includes: performing a bird's-eye view feature channel construction operation on each semantic feature channel of the semantic features respectively. The bird's-eye view feature channel construction operation on the semantic feature channel includes: for each row in the semantic feature channel, constructing a bird's-eye view feature channel corresponding to the row. Constructing a bird's-eye view feature channel corresponding to the row includes: for each semantic feature pixel in the row, constructing the related frustum feature pixels of the semantic feature pixel at each preset depth into a row of the bird's-eye view feature channel corresponding to the row.

[0100] In some embodiments, obtaining the three-dimensional detection result of the target image based on the bird's-eye view features includes: processing the bird's-eye view features using a unit for processing bird's-eye view features to obtain processed bird's-eye view features, wherein the unit for processing bird's-eye view features includes multiple convolutional layers; converting the processed bird's-eye view features into the input vector of the detector; and inputting the input vector of the detector into the detector to obtain the three-dimensional detection result output by the detector.

[0101] In some embodiments, obtaining the semantic features and depth features of the target image includes: processing the intrinsic parameters of the camera acquiring the target image using a fully connected layer to obtain a weight vector, wherein each weight in the weight vector corresponds to a semantic feature channel in the semantic features of the target image; for each semantic feature channel of the semantic features of the target image, weighting the semantic feature channel using the weight corresponding to that semantic feature channel to obtain a weighted semantic feature channel; inputting all the obtained weighted semantic feature channels into a unit for extracting depth features to obtain the depth features of the target image output by the unit for extracting depth features, wherein the unit for extracting depth features includes multiple convolutional layers.

[0102] In some embodiments, obtaining the frustum features of the target image based on the semantic features and the depth features includes: taking the outer product of the depth features and the semantic features to obtain the frustum features of the target image.

[0103] In some embodiments, the 3D object detection network is applied to autonomous vehicles.

[0104] This application also provides an electronic device that may be configured with one or more processors; and a memory for storing one or more programs, which may include instructions for performing the operations described in the above embodiments. When the one or more programs are executed by the one or more processors, the one or more processors perform the instructions for performing the operations described in the above embodiments of the three-dimensional target detection method.

[0105] This application also provides a storage medium, which may be included in an electronic device or exist independently, not assembled into an electronic device. The storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the operations described in the embodiments of the three-dimensional target detection method.

[0106] It should be noted that the storage medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, including but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with a message execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signaling media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with a message execution system, apparatus, or device. Program code included on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable messages for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer messages.

[0108] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical embodiments formed by specific combinations of the above-described technical features, but should also cover other technical embodiments formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical embodiments formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A target detection method, characterized in that, The method includes: Acquire the target image; The target image is input into a 3D target detection network to obtain the 3D detection result of the target image. The 3D target detection network is configured as follows: Obtain the semantic and depth features of the target image; Based on the semantic features and the depth features, the frustum features of the target image are obtained; The corresponding multiple frustum feature pixels in the frustum feature are constructed into corresponding bird's-eye view feature channels to obtain the bird's-eye view features of the target image. Based on the bird's-eye view features, the three-dimensional detection results of the target image are obtained; Each depth feature channel of the target image corresponds to a different preset depth; Specifically, constructing corresponding multiple frustum feature pixels from the frustum features into corresponding bird's-eye view feature channels to obtain the bird's-eye view features of the target image includes: For each semantic feature channel of the semantic features, a bird's-eye view feature channel construction operation is performed. The bird's-eye view feature channel construction operation for the semantic feature channels includes: For each row in the semantic feature channel, a bird's-eye view feature channel corresponding to the row is constructed. Constructing the bird's-eye view feature channel corresponding to the row includes: for each semantic feature pixel in the row, constructing the relevant frustum feature pixels of the semantic feature pixel at each preset depth as a row of the bird's-eye view feature channel corresponding to the row. The height dimension of the semantic feature channel of the target image is the number of rows in the semantic feature channel of the target image. For each semantic feature channel, the bird's-eye view feature channel is obtained by performing a bird's-eye view feature channel construction operation on the semantic feature channel. The number of bird's-eye view feature channels obtained by performing a bird's-eye view feature channel construction operation on each semantic feature channel of the target image is: the number of semantic feature channels of the target image * the height dimension of the semantic feature channel of the target image.

2. The method according to claim 1, characterized in that, Based on the bird's-eye view features, the three-dimensional detection results of the target image include: The bird's-view features are processed using a unit for processing bird's-view features to obtain processed bird's-view features. The unit for processing bird's-view features includes multiple convolutional layers. The processed bird's-eye view features are converted into input vectors for the detectors in the 3D target detection network; The input vector of the detector is input into the detector to obtain the three-dimensional detection result of the target image output by the detector.

3. The method according to claim 1, characterized in that, Obtaining the semantic and depth features of the target image includes: The intrinsic parameters of the camera that acquires the target image are processed using a fully connected layer to obtain a weight vector, wherein each weight in the weight vector corresponds to a semantic feature channel in the semantic features of the target image. For each semantic feature channel of the target image, the semantic feature channel is weighted using the weight corresponding to that semantic feature channel to obtain the weighted semantic feature channel. All the weighted semantic feature channels obtained are input into the unit for extracting depth features, and the depth features of the target image are obtained by the unit for extracting depth features. The unit for extracting depth features includes multiple convolutional layers.

4. The method according to claim 1, characterized in that, Based on the semantic features and the depth features, the frustum features of the target image are obtained as follows: The depth features and the semantic features are multiplied by an outer product to obtain the frustum features of the target image.

5. The method according to any one of claims 1-4, characterized in that, The 3D object detection network is applied to autonomous vehicles.

6. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the method of any one of claims 1-5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-5.

Images