Three-dimensional target detection method, electronic device and computer storage medium

Abstract

The embodiment of the present application provides a three-dimensional target detection method, an electronic device and a computer storage medium, wherein the three-dimensional target detection method includes: obtaining multiple environmental images of different fields of view for the same three-dimensional environment scene, and extracting multi-scale features corresponding to each environmental image; based on the extracted multi-scale features corresponding to each environmental image, generating a bird's-eye view feature map corresponding to the three-dimensional environment scene in a polar coordinate system; performing three-dimensional target detection on the bird's-eye view feature map in a polar coordinate system to obtain a three-dimensional target detection result. Through the embodiment of the present application, the accuracy and efficiency of three-dimensional target detection are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image detection technology, and in particular to a three-dimensional target detection method, electronic equipment, and computer storage medium. Background Art

[0002] In the fields of unmanned driving and robotics, the ability to perceive the environment is an important ability to ensure their normal operation. To achieve correct perception of the environment, three-dimensional object detection is the basic function of environmental perception. Through three-dimensional object detection, the position and category of objects in the three-dimensional scene can be predicted, and the size and orientation of the objects can be determined. Furthermore, based on this, operational tasks such as driving prediction and path planning can be achieved.

[0003] An important branch of 3D object detection is 3D object detection based on visual images. Currently, a more commonly used 3D object detection based on visual images uses depth prediction, which converts image pixels into pseudo-lidar data and then uses point cloud 3D object detection algorithms for detection. However, this method has high requirements for the accuracy of depth prediction. Inaccurate depth prediction will significantly reduce the performance of the detector, resulting in inaccurate detection results. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a three-dimensional object detection solution to at least partially solve the above-mentioned problem.

[0005] According to a first aspect of an embodiment of the present application, a three-dimensional target detection method is provided, comprising: acquiring multiple environmental images with different fields of view for the same three-dimensional environment scene, and extracting multi-scale features corresponding to each environmental image; generating a bird's-eye view feature map in a polar coordinate system corresponding to the three-dimensional environment scene based on the extracted multi-scale features corresponding to each environmental image; performing three-dimensional target detection in a polar coordinate system on the bird's-eye view feature map to obtain a three-dimensional target detection result.

[0006] According to the second aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.

[0007] According to a third aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0008] According to the solution provided by the embodiment of the present application, firstly, multiple environmental images of different fields of view are used as the detection basis. Under different fields of view, the information of multiple environmental images can complement each other, so as to avoid the situation that the information of one or some objects is not sufficient (such as unclear image or missing information of some objects in the image or the image of the object is too small, etc.) under a single field of view, which affects the subsequent information processing and target detection effect. Secondly, each environmental image corresponds to multi-scale features, so that the features corresponding to the multiple environmental images are richer and more comprehensive, providing richer information and better data basis for subsequent target detection. Furthermore, whether it is a bird's-eye view feature map generated based on the multi-scale features of multiple environmental images, or a three-dimensional target detection based on the bird's-eye view feature map, it is performed in a polar coordinate system, which avoids the information loss caused by the need to sample to the Cartesian coordinate system in the traditional three-dimensional target detection method, so that the detection result is more accurate. Furthermore, as can be seen from the above, the three-dimensional target detection solution provided in the embodiment of the present application does not need to convert image pixels into pseudo lidar data, nor does it need to adopt a depth prediction method. Therefore, it avoids the high-precision data requirements and high detection costs caused by the need for high-precision data for depth prediction, effectively reduces the implementation cost of three-dimensional target detection, and avoids the problem of inaccurate detection caused by low data accuracy when using depth prediction, thereby improving the accuracy and efficiency of three-dimensional target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0010] Figure 1A A schematic diagram of an exemplary system applicable to the embodiment of the present application;

[0011] Figure 1B A schematic diagram of an exemplary three-dimensional object detection model structure applicable to the embodiment of the present application;

[0012] Figure 2A A flowchart of a three-dimensional object detection method according to an embodiment of the present application;

[0013] Figure 2B for Figure 2A A schematic diagram of a three-dimensional target detection result in a polar coordinate system in the illustrated embodiment;

[0014] Figure 2C for Figure 2A A schematic diagram of an example scenario in the illustrated embodiment;

[0015] Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0017] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0018] Figure 1A An exemplary system applicable to the embodiment of the present application is shown. As shown in Figure 1, the system 100 may include a cloud service end 102, a communication network 104 and / or one or more user devices 106, and Figure 1 shows multiple user devices.

[0019] The cloud service end 102 can be any appropriate device for storing information, data, programs and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud service end clusters, etc. In some embodiments, the cloud service end 102 can perform any appropriate function. For example, in some embodiments, the cloud service end 102 can be used to perform three-dimensional target detection based on multiple environmental images with different fields of view. As an optional example, in some embodiments, the cloud service end 102 can be used to generate a multi-scale bird's-eye view feature map in a polar coordinate system corresponding to a three-dimensional environmental scene based on multi-scale features corresponding to multiple environmental images with different fields of view; and then, perform three-dimensional target detection in a polar coordinate system based on the multi-scale bird's-eye view feature map. As another example, in some embodiments, the cloud service end 102 can be used to send the three-dimensional target detection result to the user device, or send the result of downstream job task processing based on the three-dimensional target detection result to the user device.

[0020] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0021] User device 106 may include any one or more user devices that are suitable for interacting with a user and capable of collecting environmental images of different fields of view. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include a vehicle, an aircraft, a robot, and / or any other suitable type of user device with an autonomous driving function.

[0022] In an optional embodiment, the three-dimensional object detection provided in the embodiment of the present application can be implemented based on an exemplary three-dimensional object detection model structure, such as Figure 1B shown.

[0023] As can be seen from the figure, the 3D object detection model structure includes a feature map extraction part, an encoder and a decoder. Among them, the feature map extraction part is mainly used to extract multi-scale features of the environment image to obtain the corresponding multi-scale feature map. Exemplarily, this part can be implemented in the form of image feature extraction BACKBONE (backbone network) + FPN (feature map pyramid network), where BACKBONE can be implemented using any appropriate network structure, including but not limited to CNN network and ResNET network.

[0024] The encoder is based on the multiple multi-scale feature maps extracted by the feature map extraction part, and generates corresponding vectors that can further characterize the image features and characterization vectors of the vertical scan lines of the image after corresponding encoding processing. Exemplarily, the encoder can be implemented using an encoder with a Transformer structure. In the embodiment of the present application, for example, an encoder with a Deformal DETR structure is used for implementation.

[0025] The decoder generates a bird's-eye view feature map in a polar coordinate system based on the above vector output by the encoder, performs three-dimensional target detection in a polar coordinate system based on the bird's-eye view feature map, and outputs the three-dimensional target detection result in the polar coordinate system. Exemplarily, the decoder can be implemented using a decoder with a Transformer structure. In the embodiment of the present application, for example, a decoder with a Deformal DETR structure is implemented.

[0026] Below, based on the above system and model structure description, the three-dimensional target detection solution provided in the embodiment of the present application is described through an example.

[0027] Reference Figure 2A , shows a step flow chart of a three-dimensional target detection method according to an embodiment of the present application.

[0028] The three-dimensional object detection method comprises the following steps:

[0029] Step S202: Acquire multiple environment images with different fields of view for the same three-dimensional environment scene, and extract multi-scale features corresponding to each environment image.

[0030] In the embodiment of the present application, the three-dimensional environment scene can be a real environment at any physical location where the image acquisition device is located. Multiple environmental images with different fields of view can be acquired by multiple image acquisition devices such as multiple cameras, or can be acquired by the same image acquisition device with multiple image acquisition units such as a multi-eye camera, etc., usually multiple environmental images with different fields of view at the same time. Of course, the same time is not the same as the indiscriminate time, and multiple environmental images with different fields of view acquired within the allowable error time can also be applied to the solution of the embodiment of the present application.

[0031] In the field of image processing, multiple environmental images with different fields of view refer to multiple environmental images of the same three-dimensional environment scene taken from different angles. For example, multiple images of the driving environment of an autonomous vehicle taken at the same time by a multi-camera mounted on the autonomous vehicle.

[0032] For each environment image, feature extraction can be performed. In the embodiment of the present application, a multi-scale feature extraction method is adopted, that is, feature extraction of different scales is performed on each environment image in order to obtain more and more comprehensive effective information in the environment image. Among them, the multi-scale feature extraction method can be implemented by those skilled in the art in accordance with actual needs in an appropriate manner. Figure 1BWhen the model shown in is used, it can be obtained through the feature map extraction part of the model. That is, each environment image passes through the BACKBONE and FPN of the feature map extraction part to obtain multi-scale features. Because in the machine learning model, the features are usually mapped to the form of feature maps, the multi-scale features can also be called multi-scale feature maps.

[0033] Exemplarily, assuming that there are 5 environment images with different fields of view, each environment image corresponds to 5 multi-scale feature maps, then all environment images will correspond to 5*5=25 multi-scale feature maps.

[0034] It should be noted that in the embodiments of the present application, unless otherwise specified, "plurality", "multiple" and other numbers related to "multiple" all mean two or more.

[0035] Step S204: Generate a bird's-eye view feature map in a polar coordinate system corresponding to the three-dimensional environment scene based on the multi-scale features corresponding to each of the multiple environment images extracted.

[0036] After obtaining all the multi-scale feature maps corresponding to multiple environmental images, the corresponding bird's-eye view feature map can be generated based on these multi-scale feature degrees, and the generated bird's-eye view feature map is also a multi-scale bird's-eye view feature map. In the embodiment of the present application, the bird's-eye view feature map is also a feature map used to characterize image features, but it can correspond to a bird's-eye view of a three-dimensional environmental scene, or it can be said that it is a feature map corresponding to the bird's-eye view. Generally speaking, a bird's-eye view can be considered as a stereoscopic image drawn from a certain point high above, looking down at the undulations of the ground, based on the principle of perspective, using a high viewpoint perspective method.

[0037] In order to realize the conversion from the feature map of the environment image to the bird's-eye view feature map, in a feasible way, for each of the multiple environment images, the pixels in the feature map corresponding to each scale feature can be independently converted into a bird's-eye view feature map based on the polar coordinate system corresponding to the scale feature according to one pixel column corresponding to one polar coordinate ray; and then, according to the bird's-eye view feature map corresponding to each scale feature of each environment image, a bird's-eye view feature map in the polar coordinate system corresponding to the three-dimensional environment scene can be generated. Thus, the generated bird's-eye view feature map can fully extract effective multi-scale information, and can provide richer and more comprehensive feature information for subsequent three-dimensional target detection, so as to improve the effect of subsequent large and small target detection in the three-dimensional environment scene.

[0038] When converting an environment image to a bird's-eye view, it is usually assumed that there is a 1-1 correspondence between the vertical scan lines of the image and the rays passing through the camera position in the bird's-eye view. Based on this, this conversion can be regarded as a sequence-to-sequence conversion. In this case, since multi-scale features can represent spatial features, the spatial features can be converted from the image to the bird's-eye view through the encoder-decoder. For example, in an encoder-decoder based on the attention mechanism, the multi-scale feature map is input to the encoder, and the encoder obtains the information of the vertical scan lines of the image based on the multi-scale feature map, and creates a "memory" representation vector based on this information. After being input to the decoder, the representation vector will be decoded into the polar ray of the bird's-eye view, based on which the bird's-eye feature map is generated.

[0039] In order to enable the multi-scale feature maps corresponding to the above-mentioned multiple environmental images with different fields of view to effectively fuse information in the polar coordinate system and improve the subsequent three-dimensional target detection effect, in a feasible method, deformable attention calculation can be performed on the bird's-eye view feature map corresponding to each scale feature corresponding to each environmental image, and a bird's-eye view feature map after feature fusion is obtained according to the calculation result; according to the bird's-eye view feature map after feature fusion corresponding to each scale feature corresponding to each environmental image, a bird's-eye view feature map corresponding to the three-dimensional environmental scene in the polar coordinate system is generated.

[0040] In a specific example, multiple environmental images are respectively processed through BACKBONE and FPN to obtain a multi-scale feature map corresponding to each environmental image. Then, each scale feature map is independently converted to a bird's-eye view feature map in a sequence-to-sequence manner by an encoder-decoder using a transformer structure according to the principle that one image column (i.e., pixel column) corresponds to one polar ray (polar coordinate ray), thereby obtaining multi-scale bird's-eye view feature map features. It can be seen that in the scheme of the embodiment of the present application, environmental images of different fields of view are converted to the same bird's-eye view range, that is, the bird's-eye view in the polar coordinate system of different fields of view corresponds to the same BEV (bird's-eye view) space.

[0041] When using Figure 1BWhen the encoder-decoder using the Deformable DETR structure is shown in , the above process can be implemented as follows: each pixel on the feature map of each scale is regarded as a query. For one of the pixels, it is assumed to be the current pixel to be processed, and all the remaining pixels can be regarded as values ​​(including pixels on feature maps of different scales). In order to avoid excessive calculation, the encoder using the Deformable DETR structure first uses a feature map to generate several offsets from the query in the form of deformable conv, and then uses the pixel of the feature map corresponding to the offset index as the value, and then uses MLP (multi-layer perceptron) to generate attention, that is, the result of the attention algorithm, for value weighted summation. Based on the final weighted summation results of each feature map, a vector that can represent the image features and a representation vector of the image vertical scan line information can be obtained. Because the offset is generated by MLP, it is relatively flexible, and a larger offset can be generated near and a smaller offset can be generated far away; or different offsets can be generated on feature maps of different scales. As a result, feature extraction is more flexible, so the subsequent three-dimensional object detection performance is also better.

[0042] Then, the representation vector output by the encoder is input into the decoder, which decodes it into a bird's-eye view polar ray and generates a bird's-eye view feature map based on it.

[0043] Step S206: Perform three-dimensional target detection on the bird's-eye view feature image in a polar coordinate system to obtain a three-dimensional target detection result.

[0044] Since information is lost when transferring information from the environment image to the bird's-eye view in a Cartesian coordinate system, the three-dimensional object detection result is affected. Therefore, in the embodiment of the present application, three-dimensional object detection is performed directly in a polar coordinate system to avoid the problem of information loss.

[0045] To this end, in a feasible manner, this step can be implemented as follows: based on the bird's-eye view feature map, predict the position of the object in the polar coordinate system; determine the corresponding features of the object in the bird's-eye view feature map according to the position of the object; based on the features, detect the object for describing the properties of the object in three-dimensional space; and obtain the three-dimensional target detection result of the object according to the object position and the detection result. Among them, the detection of the properties of the object in the three-dimensional space at least includes: the detection of the size of the object in the polar coordinate system, the speed detection of the object relative to the polar coordinate system, and the direction detection. Correspondingly, the detection results at least include: the size of the object in the polar coordinate system, the speed and direction of the object relative to the polar coordinate system, etc.

[0046] Among them, in order to obtain more accurate features in the bird's-eye view feature map corresponding to the object position, in a feasible method, the candidate features of the predicted object position in the bird's-eye view feature map can be first determined; then the candidate features can be subjected to deformable convolution processing, and the features corresponding to the object position in the bird's-eye view feature map can be obtained according to the processing results.

[0047] Further optionally, in order to gradually reduce the error of object position prediction, when predicting the object position in the polar coordinate system based on the bird's-eye view feature map, the decoder can also predict the object position in the polar coordinate system based on the bird's-eye view feature map. The decoder includes multiple attention layers, each of which predicts the object position in the polar coordinate system based on the bird's-eye view feature map, wherein each subsequent attention layer predicts the position deviation based on the center point of the object position output by the previous attention layer to correct the object position predicted by the previous attention layer, and the correction result of the last attention layer is used as the object position prediction result predicted by the decoder.

[0048] In a specific example, the detection input is a multi-scale bird's-eye view feature map in a polar coordinate system and a learnable parameter object query. Then, self-attention and cross-attention are performed between different object queries. When performing cross-attention, for each query, MLP first generates reference points (reference object position points in the polar coordinate system), and then another MLP generates an offset (i.e., position deviation) to locate the value of the reference points on the multi-scale bird's-eye view map in the polar coordinate system. The total number of reference points is M×num_points. M represents the number of scales of the multi-scale, and num_points represents the number of reference object position points on each scale feature map. After that, the attention calculation result is also generated by this query to perform a weighted summation of the previously located values. After the above processing is performed on the bird's-eye view feature maps of all scales, a multi-dimensional vector of a single object in the polar coordinate system will be predicted, for example, it can be a 10-dimensional vector, as the three-dimensional target detection result in the polar coordinate system. For example:

[0049] (ρ, θ, z, log w, log h, log l, sinΔθ ori , cosΔθ ori , v θ , v ρ )

[0050] Among them, ρ, θ represent the position of the object, Z represents the height coordinate of the object, W, h, l represent the length, width, and height of the object respectively, and Δθ ori Indicates the relative orientation of the object, v θ , v ρ They represent the tangential velocity and radial velocity of the object respectively.

[0051] A three-dimensional target detection result in the above polar coordinate system is as follows Figure 2B As shown in the figure, the meanings of the various parameters in the figure are as described above.

[0052] In the above process, each attention layer of the decoder will perform three-dimensional target prediction, and the prediction results will be calculated into the loss function. The reference point (reference object position point) input to the first layer is generated by objectquery, and the reference point after the first layer is the center point of the object position predicted by the previous layer. When predicting the center point of the object, each layer only predicts the offset based on the reference point, and then corrects the reference point to obtain the coordinates of the object center point. In this way, cascade detection of three-dimensional targets is achieved, which can gradually reduce the error of object position prediction.

[0053] Further optionally, in order to facilitate the processing of downstream tasks, the three-dimensional target detection in the polar coordinate system can be performed on the bird's-eye view feature map. After the three-dimensional target detection result in the polar coordinate system is obtained, the three-dimensional target detection result in the polar coordinate system is converted into a three-dimensional target detection result in the Cartesian coordinate system. The specific implementation of the transformation from the polar coordinate system to the Cartesian coordinate system can be implemented by those skilled in the art in any appropriate manner according to actual needs, and the embodiments of the present application are not limited to this.

[0054] The following is an example of a specific scenario to illustrate the above process. Figure 2C shown.

[0055] Figure 2C In the figure, it is assumed that a multi-camera is provided in the autonomous driving vehicle, and a plurality of environmental images of the physical environment in which the autonomous driving vehicle is located are simultaneously captured by the multi-camera at the same time, and the plurality of environmental images have different fields of view.

[0056] The multiple environment images with different fields of view are input as follows: Figure 1BThe three-dimensional target detection model shown in the figure first extracts the multi-scale features of each environmental image through its feature map extraction part, and generates a multi-scale feature map corresponding to each environmental image. Then, all the multi-scale feature maps corresponding to all environmental images are input into the encoder, and the encoder performs feature extraction and encoding again to obtain vectors that can better represent the features of the environmental image and the representation vectors that can represent the vertical scanning lines of the image. Then, these vectors output by the encoder are input into the decoder, and the decoder converts and generates corresponding multi-scale bird's-eye view feature maps (i.e., bird's-eye view feature maps of multiple different scales) in the polar coordinate system based on these vectors in the manner that one pixel column corresponds to one polar coordinate ray. Then, the decoder performs three-dimensional target detection in the polar coordinate system based on the multi-scale bird's-eye view feature map, including: first predicting the position of the object, then obtaining the features in the corresponding bird's-eye view feature map based on the position of the object, and then predicting the size of the object, the speed relative to the polar coordinate system, the direction, etc. based on these features, and finally outputting the three-dimensional target detection result in the polar coordinate system. Further, the three-dimensional target detection result in the polar coordinate system can be converted into the three-dimensional target detection result in the Cartesian coordinate system. Based on the three-dimensional target detection result in the Cartesian coordinate system, driving planning is performed for the autonomous driving vehicle.

[0057] It can be seen that through this embodiment, firstly, multiple environmental images of different fields of view are used as the detection basis. Under different fields of view, the information of multiple environmental images can complement each other, avoiding the situation that the information of one or some objects is not sufficient under a single field of view (such as the information of some objects is missing in the unclear image or the image of the object is too small, etc.), which affects the subsequent information processing and target detection effect. Secondly, each environmental image corresponds to multi-scale features, so that the features corresponding to the multiple environmental images are richer and more comprehensive, providing richer information and a better data basis for subsequent target detection. Furthermore, whether it is a bird's-eye view feature map generated based on the multi-scale features of multiple environmental images, or a three-dimensional target detection based on the bird's-eye view feature map, it is performed in a polar coordinate system, which avoids the information loss caused by the need to sample to the Cartesian coordinate system in the traditional three-dimensional target detection method, making the detection result more accurate. Furthermore, as can be seen from the above, the three-dimensional target detection solution provided by the present embodiment does not need to convert image pixels into pseudo lidar data, nor does it need to adopt a depth prediction method. Therefore, it avoids the high-precision data requirements and high detection costs caused by the need for high-precision data for depth prediction, effectively reducing the implementation cost of three-dimensional target detection, and avoids the problem of inaccurate detection caused by low data accuracy when using depth prediction, thereby improving the accuracy and efficiency of three-dimensional target detection.

[0058] Reference Figure 3 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0059] like Figure 3 As shown, the electronic device may include: a processor (processor) 302 , a communication interface (Communications Interface) 304 , a memory (memory) 306 , and a communication bus 308 .

[0060] in:

[0061] The processor 302 , the communication interface 304 , and the memory 306 communicate with each other via the communication bus 308 .

[0062] The communication interface 304 is used to communicate with other electronic devices or servers.

[0063] The processor 302 is used to execute the program 310, and specifically can execute the relevant steps in the above-mentioned three-dimensional object detection method embodiment.

[0064] Specifically, the program 310 may include program codes, which include computer operation instructions.

[0065] The processor 302 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0066] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0067] The program 310 may be specifically used to enable the processor 302 to execute operations corresponding to the three-dimensional object detection method described in the aforementioned method embodiment.

[0068] The specific implementation of each step in the program 310 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0069] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to the three-dimensional object detection method in the above method embodiment.

[0070] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0071] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0072] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0073] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A three-dimensional target detection method, comprising: Acquire multiple environmental images of different fields of view for the same three-dimensional environmental scene, and extract multi-scale features corresponding to each environmental image; Based on the extracted multi-scale features corresponding to each environmental image, a bird's-eye view feature map corresponding to the three-dimensional environmental scene in a polar coordinate system is generated; Perform three-dimensional target detection in a polar coordinate system on the bird's-eye view feature image to obtain a three-dimensional target detection result.

2. The method according to claim 1, wherein: The step of generating a bird's-eye view feature map corresponding to the three-dimensional environment scene in a polar coordinate system based on the extracted multi-scale features corresponding to each environment image comprises: For any scale feature corresponding to any environment image, obtain the pixels in the feature map corresponding to the scale feature, and convert and generate a bird's-eye view feature map corresponding to the scale feature according to one pixel column corresponding to one polar coordinate ray; According to the bird's-eye view feature map corresponding to each scale feature corresponding to each environmental image, a bird's-eye view feature map in a polar coordinate system corresponding to the three-dimensional environmental scene is generated.

3. The method according to claim 2, wherein: The step of generating a bird's-eye view feature map corresponding to the three-dimensional environment scene in a polar coordinate system according to the bird's-eye view feature map corresponding to each scale feature corresponding to each environment image includes: Perform deformable attention calculation on the bird's-eye view feature map corresponding to each scale feature corresponding to each environment image, and obtain the bird's-eye view feature map after feature fusion according to the calculation result; A bird's-eye view feature map in a polar coordinate system corresponding to the three-dimensional environment scene is generated according to the bird's-eye view feature map obtained by fusion of features corresponding to each scale feature corresponding to each environment image.

4. The method according to any one of claims 1 to 3, wherein: The performing three-dimensional target detection in a polar coordinate system on the bird's-eye view feature image to obtain a three-dimensional target detection result includes: Based on the bird's-eye view feature map, predict the position of the object in a polar coordinate system; Determine, according to the position of the object, a feature corresponding to the object in the bird's-eye view feature map; According to the features, detecting the object for describing the properties of the object in three-dimensional space; A three-dimensional target detection result of the object is obtained according to the object position and the detection result.

5. The method according to claim 4, wherein: Determining the feature corresponding to the object in the bird's-eye view feature map according to the position of the object includes: Determining candidate features of the predicted object position in the bird's-eye view feature map; The candidate features are subjected to deformable convolution processing, and features corresponding to the object position in the bird's-eye view feature map are obtained according to the processing results.

6. The method according to claim 4, wherein: The detection of the attribute of the object in the three-dimensional space at least includes: detection of the size of the object in the polar coordinate system, and detection of the speed and direction of the object relative to the polar coordinate system.

7. The method according to claim 4, wherein: The method of predicting the position of an object in a polar coordinate system based on the bird's-eye view feature map includes: The decoder predicts the object position in a polar coordinate system based on the bird's-eye view feature map, wherein the decoder includes multiple attention layers, each attention layer predicts the object position in a polar coordinate system based on the bird's-eye view feature map, wherein each subsequent attention layer predicts the position deviation based on the center point of the object position output by the previous attention layer to correct the object position predicted by the previous attention layer, and the correction result of the last attention layer is used as the object position prediction result predicted by the decoder.

8. The method according to any one of claims 1 to 3, wherein: The performing three-dimensional target detection in a polar coordinate system on the bird's-eye view feature image to obtain a three-dimensional target detection result includes: Performing three-dimensional target detection in a polar coordinate system on the bird's-eye view feature image to obtain a three-dimensional target detection result in the polar coordinate system; The three-dimensional target detection result in the polar coordinate system is converted into a three-dimensional target detection result in a Cartesian coordinate system.

9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images