Method for determining aerial view feature, target detection method, device and electronic equipment

Abstract

The embodiment of the present disclosure discloses a bird's eye view feature determination method, a target detection method, a device and an electronic device, wherein the method comprises: determining first image features corresponding to each of at least one perspective based on a to-be-processed image corresponding to each of the perspective; determining a three-dimensional bird's eye view position encoding based on a pre-obtained two-dimensional bird's eye view position encoding and bird's eye view content encoding; and determining a bird's eye view feature corresponding to the at least one perspective based on the first image features corresponding to each of the perspective and the three-dimensional bird's eye view position encoding. The embodiment of the present disclosure realizes accurate determination of the bird's eye view feature without relying on camera parameters based on the enhanced three-dimensional bird's eye view position encoding, thereby improving the accuracy and stability of the processing result of the related task based on the bird's eye view feature.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method for determining features of a bird's-eye view, a method for object detection, an apparatus, and an electronic device. Background Technology

[0002] In the field of autonomous driving, images captured by multiple cameras on a vehicle using vision technology can be used to determine bird's eye view (BEV) features. These BEV features are then used for tasks such as object detection and semantic segmentation, making this an emerging research focus. In related technologies, coordinate system transformation is usually performed by combining camera parameters to determine the bird's eye view features from the image features. However, the determined bird's eye view features are easily affected by camera parameters (such as the influence of extrinsic noise), resulting in poor robustness of the obtained bird's eye view features, which in turn leads to poor accuracy and stability of the processing results of the corresponding tasks. Summary of the Invention

[0003] To address the technical problems mentioned above, such as the poor accuracy and stability of task processing results due to the influence of camera parameters on bird's-eye view features, this disclosure is proposed. Embodiments of this disclosure provide a method for determining bird's-eye view features, a target detection method, an apparatus, and an electronic device.

[0004] According to one aspect of the present disclosure, a method for determining bird's-eye view features is provided, comprising: determining first image features corresponding to each of the at least one viewpoint based on images to be processed corresponding to each of the viewpoints; determining a three-dimensional bird's-eye view position code based on a pre-obtained two-dimensional bird's-eye view position code and a bird's-eye view content code; and determining bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view position code.

[0005] According to another aspect of the present disclosure, a target detection method is provided, comprising: determining first image features corresponding to each of the at least one viewpoint based on the images to be processed corresponding to each of the viewpoints; determining a three-dimensional bird's-eye view position code based on a pre-obtained two-dimensional bird's-eye view position code and a bird's-eye view content code; determining bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view position code; and determining a target detection result based on the bird's-eye view features using a detection head network.

[0006] According to another aspect of the present disclosure, an apparatus for determining bird's-eye view features is provided, comprising: a first processing module, configured to determine first image features corresponding to each of the at least one viewpoint based on images to be processed corresponding to each of the viewpoints; a second processing module, configured to determine a three-dimensional bird's-eye view position code based on a pre-obtained two-dimensional bird's-eye view position code and a bird's-eye view content code; and a third processing module, configured to determine bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view position code.

[0007] According to another aspect of the present disclosure, a target detection apparatus is provided, comprising: a fourth processing module, configured to determine first image features corresponding to each of the at least one viewpoint based on images to be processed corresponding to each of the viewpoints; a fifth processing module, configured to determine a three-dimensional bird's-eye view position code based on a pre-obtained two-dimensional bird's-eye view position code and a bird's-eye view content code; a sixth processing module, configured to determine bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view position code; and a seventh processing module, configured to determine a target detection result based on the bird's-eye view features using a detection head network.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the above embodiments of the present disclosure.

[0010] Based on the bird's-eye view feature determination method, target detection method, apparatus, and electronic device provided in the above embodiments of this disclosure, a three-dimensional bird's-eye view position code is determined based on two-dimensional bird's-eye view position coding and bird's-eye view content coding, and bird's-eye view features are determined based on first image features and the three-dimensional bird's-eye view position code. This disclosure, based on enhanced three-dimensional bird's-eye view position coding, achieves accurate determination of bird's-eye view features without relying on camera parameters, thereby improving the accuracy and stability of related task processing results based on bird's-eye view features.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is an exemplary application scenario of the method for determining bird's-eye view features provided in this disclosure;

[0014] Figure 2 This is a flowchart illustrating a method for determining bird's-eye view features provided in an exemplary embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram of a lidar coordinate system provided in an exemplary embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating a method for determining bird's-eye view features provided in another exemplary embodiment of this disclosure;

[0017] Figure 5 This is a flowchart illustrating a method for determining bird's-eye view features provided in yet another exemplary embodiment of this disclosure;

[0018] Figure 6 This is a flowchart illustrating a method for determining bird's-eye view features provided in yet another exemplary embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the network structure of a view-aware attention network provided in an exemplary embodiment of this disclosure;

[0020] Figure 8 This is a flowchart illustrating step 2021 provided in an example embodiment of this disclosure;

[0021] Figure 9 This is a flowchart illustrating step 20215 provided in an exemplary embodiment of this disclosure;

[0022] Figure 10 This is a schematic diagram of the structure of a location-aware enhancement network provided in an exemplary embodiment of this disclosure;

[0023] Figure 11 This is a schematic flowchart of a target detection method provided in an exemplary embodiment of this disclosure;

[0024] Figure 12 This is a schematic diagram of the structure of a bird's-eye view feature determination device provided in an exemplary embodiment of this disclosure;

[0025] Figure 13This is a schematic diagram of the structure of a device for determining bird's-eye view features provided in another exemplary embodiment of this disclosure;

[0026] Figure 14 This is a schematic diagram of the structure of the second processing unit 5032 provided in an exemplary embodiment of the present disclosure;

[0027] Figure 15 This is a schematic diagram of the structure of the third processing unit 5021 provided in an exemplary embodiment of this disclosure;

[0028] Figure 16 This is a schematic diagram of the target detection device provided in an example embodiment of the present disclosure;

[0029] Figure 17 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0030] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0031] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0032] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0033] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0034] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0035] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0036] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0037] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0038] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0039] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0040] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0041] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0042] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0043] This disclosure outlines

[0044] In the process of realizing this disclosure, the inventors discovered that in the field of autonomous driving, images collected from multiple cameras on a vehicle based on vision technology can determine bird's eye view (BEV) features. Then, tasks such as object detection and semantic segmentation based on bird's eye view features have become an emerging research focus. In related technologies, coordinate system transformation is usually combined with camera parameters to determine the bird's eye view features from image features. The determined bird's eye view features are easily affected by camera parameters (such as the influence of extrinsic noise), resulting in poor robustness of the obtained bird's eye view features, which in turn leads to poor accuracy and stability of the processing results of the corresponding tasks.

[0045] Exemplary Overview

[0046] Figure 1 This is an exemplary application scenario of the method for determining bird's-eye view features provided in this disclosure.

[0047] In the target detection scenario of autonomous driving, images acquired by image sensors from each viewpoint (viewpoint 1 to viewpoint m, where m is a positive integer) (or images acquired after certain preprocessing) can be used as images to be processed, obtaining images to be processed corresponding to m viewpoints respectively. Using the bird's-eye view feature determination method of this disclosure, the first image features corresponding to each viewpoint can be determined based on the images to be processed corresponding to each viewpoint. For example, a feature extraction network obtained through pre-training can be used to extract features from each image to be processed to obtain the first image features corresponding to each viewpoint. The two-dimensional bird's-eye view position encoding can be the two-dimensional position encoding of the XY plane with Z=0 in the lidar coordinate system corresponding to the bird's-eye view. This two-dimensional bird's-eye view position encoding can be learned during training based on the initial two-dimensional bird's-eye view position encoding. The bird's-eye view content encoding is the content encoding in the bird's-eye view space, representing the content information of the corresponding position on the bird's-eye view. This bird's-eye view content encoding can also be learned during training based on the initial bird's-eye view content encoding. Based on the two-dimensional bird's-eye view location encoding and content encoding, a three-dimensional bird's-eye view location encoding can be determined. This three-dimensional bird's-eye view location encoding is a three-dimensional location encoding in the bird's-eye view space, enhancing the location information compared to the two-dimensional encoding. By interacting with the first image features corresponding to each viewpoint and the enhanced three-dimensional bird's-eye view location encoding, bird's-eye view features can be accurately determined. This conversion from multi-view image features to bird's-eye view features can be achieved without relying on camera parameters, effectively avoiding the influence of camera parameters on bird's-eye view features. This significantly improves the accuracy and robustness of bird's-eye view features. Furthermore, object detection based on these bird's-eye view features can effectively improve the accuracy and stability of object detection results. The detection head network can be configured according to actual needs; for example, it can be a multilayer perceptron (MLP). This disclosure does not impose any limitations.

[0048] Exemplary methods

[0049] Figure 2 This is a flowchart illustrating a method for determining bird's-eye view features according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 2 As shown, it includes the following steps:

[0050] Step 201: Based on the images to be processed corresponding to each viewpoint in at least one viewpoint, determine the first image features corresponding to each viewpoint.

[0051] In this context, "viewpoint" refers to the camera's (or viewpoint's) perspective. For example, in an autonomous driving scenario, each camera on the vehicle corresponds to a viewpoint. The number of viewpoints can be set according to actual needs. For instance, if a vehicle has a front-view camera, a left front camera, a left rear camera, a right front camera, a right rear camera, and a rear-view camera, there are a total of 6 viewpoints; the specific number is not limited. The image captured by each camera serves as the image to be processed for its corresponding viewpoint, or the image captured by the camera can be pre-processed before being used as the image to be processed; the specific number can be set according to actual needs. Preprocessing can be set according to actual needs, such as image enhancement or cropping, to ensure that the image to be processed meets the input requirements of the feature extraction network, so that the feature extraction network can determine the first image features corresponding to each viewpoint. The feature extraction network can be any feasible network, such as a ResNet50-based backbone network, which can extract multi-scale features from the image to be processed, using the lowest-scale features as the first image features; the specific number can be set according to actual needs. The feature extraction network can be a feature extraction network in a pre-trained model, such as a pre-trained object detection model, semantic segmentation model, etc.; the specific number is not limited.

[0052] Step 202: Determine the location code of the three-dimensional bird's-eye view based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code.

[0053] The two-dimensional bird's-eye view location code can be a two-dimensional location code in the XY plane (Z=0) of the lidar coordinate system corresponding to the bird's-eye view. It represents the two-dimensional location information of the bird's-eye view, including location information in the X and Y directions. This two-dimensional bird's-eye view location code can be learned during training based on an initial two-dimensional bird's-eye view location code. The bird's-eye view content code is a content code in the bird's-eye view space, representing the content information of the corresponding location on the bird's-eye view. This content code can also be learned during training based on an initial bird's-eye view content code. Based on these two-dimensional and content codes, a three-dimensional bird's-eye view location code can be determined. The three-dimensional bird's-eye view location code is a three-dimensional location code in the bird's-eye view space, which enhances the location information compared to the two-dimensional bird's-eye view location code.

[0054] For example, Figure 3 This is a schematic diagram of a lidar coordinate system provided in an exemplary embodiment of this disclosure. On the vehicle, the origin of the lidar coordinate system is the center of the lidar, the forward direction of the vehicle is the positive Y-axis direction of the lidar coordinate system, the rightward direction of the vehicle's width is the positive X-axis direction of the lidar coordinate system, and the upward direction perpendicular to the XY plane is the positive Z-axis direction of the lidar coordinate system. The lidar coordinate system is a grid coordinate system, with each grid corresponding to a certain coordinate region in the real physical world.

[0055] For example, the location encoding of a two-dimensional bird's-eye view can be represented as H b ×W b ×C p H represents the feature size of the location encoding in a two-dimensional bird's-eye view. b Indicates the height and W of the bird's-eye view. b Indicates the width of the bird's-eye view, C p Indicates the number of channels. Q p It is a learnable tensor, acquired during training. The content encoding of a bird's-eye view can be represented as... C s This indicates the number of channels used for encoding the content of the bird's-eye view. A pre-trained location-aware enhancement network is used to infer the location encoding of the 2D bird's-eye view, allowing the inference of a reference height corresponding to each 2D location. This reference height is the coordinate value along the Z-axis in the LiDAR coordinate system, used to enhance the 2D bird's-eye view location encoding. By encoding the reference height and combining it with the bird's-eye view content encoding, the enhanced 3D bird's-eye view location encoding is obtained.

[0056] Steps 201 and 202 are not in any particular order.

[0057] Step 203: Based on the first image features corresponding to each viewpoint and the three-dimensional bird's-eye view position code, determine the bird's-eye view features corresponding to at least one viewpoint.

[0058] The bird's-eye view feature is an overall BEV image feature extracted and unified into the LiDAR coordinate system based on image features from each viewpoint and 3D bird's-eye view position encoding. For example, image features from six viewpoints on the vehicle are unified into the BEV coordinate system to obtain complete BEV image features of a certain area around the vehicle. In this embodiment, the conversion from the first image features of each viewpoint to the bird's-eye view feature is achieved based on 3D bird's-eye view position encoding, without relying on camera parameters.

[0059] For example, the first image features corresponding to each viewpoint and the three-dimensional bird's-eye view position code can be used as input to a pre-trained viewpoint-aware attention network. The viewpoint-aware attention network enables attentional interaction between the first image features corresponding to each viewpoint and the three-dimensional bird's-eye view position code, thereby obtaining bird's-eye view features. The specific viewpoint-aware attention network can be set according to actual needs.

[0060] The method for determining bird's-eye view features provided in this embodiment determines the three-dimensional bird's-eye view location code based on the two-dimensional bird's-eye view location code and the bird's-eye view content code. Then, it determines the bird's-eye view features based on the first image features and the three-dimensional bird's-eye view location code. Based on the enhanced three-dimensional bird's-eye view location code, it can accurately determine the bird's-eye view features without relying on camera parameters, obtain more robust bird's-eye view features, and thus improve the accuracy and stability of the processing results of related tasks based on bird's-eye view features.

[0061] Figure 4 This is a flowchart illustrating a method for determining bird's-eye view features provided in another exemplary embodiment of this disclosure.

[0062] In one optional example, step 203 may specifically include the following steps:

[0063] Step 2031: Determine the bird's-eye view embedding features based on the three-dimensional bird's-eye view location encoding and bird's-eye view content encoding.

[0064] Optionally, the location encoding and content encoding of the 3D bird's-eye view can be fused in a certain way to obtain the bird's-eye view embedding features. The specific fusion method can be set according to actual needs, such as fusion by concat, and there is no specific limitation.

[0065] In an optional example, in order to better separate the roles of 3D bird's-eye view location encoding and bird's-eye view content encoding in subsequent inference processes, the 3D bird's-eye view location encoding and bird's-eye view content encoding can be fused by concatenation, thereby further improving the robustness and accuracy of bird's-eye view features.

[0066] Step 2032: Based on the first image features and bird's-eye view embedding features corresponding to each viewpoint, the bird's-eye view features are determined using a pre-trained viewpoint perception attention network.

[0067] The view-aware attention network is a visual Transformer-based encoding and decoding network, which includes a self-attention network and a cross-attention network. The first image features and the bird's-eye view embedded features interact through the cross-attention network in the view-aware attention network to infer the bird's-eye view feature map. The specific structure of the view-aware attention network can be set according to actual needs, and this embodiment does not limit it.

[0068] This disclosure achieves feature interaction between first image features and bird's-eye view embedded features through a view-aware attention network, thereby enabling the extraction of bird's-eye view features from first image features from various viewpoints based on 3D bird's-eye view position encoding and bird's-eye view content encoding. This achieves the determination of bird's-eye view features independent of camera parameters, avoids the influence of camera parameters on bird's-eye view features, and further improves the accuracy of bird's-eye view features.

[0069] Figure 5 This is a flowchart illustrating a method for determining bird's-eye view features provided in yet another exemplary embodiment of this disclosure.

[0070] In an optional example, the viewpoint-aware attention network includes an encoder and a decoder. Step 2032, based on the first image features corresponding to each viewpoint and the bird's-eye view embedding features, utilizes the pre-trained viewpoint-aware attention network to determine the bird's-eye view features, including:

[0071] Step 20321: Based on the first image features corresponding to each viewpoint, use the encoder to determine the first encoding result corresponding to each viewpoint.

[0072] The encoder in the Visual Transformer's encoder-decoder network can include at least one encoder block. The first image features corresponding to each viewpoint are encoded by the encoder to obtain the first encoding result corresponding to each viewpoint. Each encoder block in the Visual Transformer's encoder-decoder network typically includes a self-attention network. Each first image feature undergoes a self-attention operation in the encoder, and is combined with other networks (such as layer normalization networks, feedforward networks, etc.) to complete the encoding. The specific configuration can be set according to actual needs.

[0073] Step 20322: Based on the first encoding results and bird's-eye view embedding features corresponding to each viewpoint, the bird's-eye view features are determined using a decoder.

[0074] The decoder is the decoder in the encoding / decoding network of the visual Transformer. The decoder can include at least one decoder block, each typically including a cross-attention network, and may also include other related networks, such as layer normalization networks, feedforward networks, etc., depending on actual needs. The first encoding results corresponding to each viewpoint interact with the bird's-eye view embedded features through cross-attention in the decoder, decoding the bird's-eye view features from the first encoding results corresponding to each viewpoint.

[0075] This disclosure obtains first encoding results for each viewpoint by encoding the first image features corresponding to each viewpoint using an encoder. Then, based on the bird's-eye view embedding features, the decoder decodes each first encoding result to obtain the bird's-eye view features. Since the bird's-eye view embedding features include enhanced three-dimensional bird's-eye view position encoding and bird's-eye view content encoding, the accuracy of the bird's-eye view features obtained by decoding is further improved.

[0076] Figure 6 This is a flowchart illustrating a method for determining bird's-eye view features provided in yet another exemplary embodiment of this disclosure.

[0077] In an optional example, the encoder includes a self-attention network; step 20321, based on the first image features corresponding to each viewpoint, uses the encoder to determine the first encoding result corresponding to each viewpoint, including:

[0078] 1. For each viewpoint corresponding to the first image feature, based on the first image feature and the image position code corresponding to the first image feature, determine the first query tensor, the first key tensor and the first value tensor corresponding to the first image feature.

[0079] The image position encoding can be implemented using any feasible encoding method, such as absolute position encoding, relative position encoding, etc., and this disclosure does not limit it. The first query tensor, the first key tensor, and the first value tensor can be obtained by mapping according to the corresponding mapping rules. For example, for the first query tensor, the first image features can be mapped to the first query tensor based on the first query mapping matrix, and the specifics will not be elaborated further.

[0080] 2. Based on the first query tensor, the first key tensor, and the first value tensor, a self-attention network is used to determine the self-attention result corresponding to the first image feature.

[0081] The self-attention network is a network based on the self-attention mechanism. Specifically, it performs self-attention operations on the first query tensor and the first key tensor to obtain the first weight, and then performs a weighted summation on the first value tensor based on the first weight to obtain the self-attention result corresponding to the first image feature. The specific principle of the self-attention mechanism will not be elaborated further.

[0082] 3. Based on the self-attention results, determine the encoding result corresponding to the first image feature.

[0083] The encoder includes at least one encoder block, and each encoder block includes a self-attention network as well as other related networks, such as layer normalization networks and feedforward networks. Therefore, after obtaining the self-attention result, it is also necessary to determine the final encoding result corresponding to the first image feature based on the self-attention network and using other related networks in the encoder other than the self-attention network.

[0084] 4. The encoding results corresponding to each first image feature are used as the first encoding results corresponding to each viewpoint.

[0085] In this context, the first image feature of each viewpoint can obtain its corresponding encoding result. Therefore, the encoding result corresponding to each first image feature can be used as the first encoding result corresponding to each viewpoint.

[0086] This disclosure achieves self-attention of image features during the encoder stage, making it easier to capture long-range interdependent features in the image and thus contributing to more accurate and effective inference of bird's-eye view features.

[0087] In an optional example, the decoder includes a cross-attention network; step 20322, based on the first encoding results corresponding to each viewpoint and the bird's-eye view embedding features, utilizes the decoder to determine the bird's-eye view features, including:

[0088] a. Based on the preset window segmentation rules, determine the preset number of windows corresponding to the embedded features of the bird's-eye view.

[0089] The preset window segmentation rules can be obtained in advance through certain methods, such as by comparing the model performance and effects of various segmentation methods. The specific rules can be set according to actual needs, and this disclosure does not impose any limitations. The preset window segmentation rules include the number of windows (i.e., the preset number) for segmenting the bird's-eye view embedding features, and the segmentation method. The segmentation method indicates how the bird's-eye view embedding features are segmented into the preset number of windows, such as uniformly segmenting the bird's-eye view embedding features into 4 windows. The specific rules can be set according to actual needs.

[0090] For example, based on the different areas in the bird's-eye view, the bird's-eye view embedding features are divided into four windows: left rear (BL) window, left front (FL) window, right rear (BR) window, and right front (FR) window. Each window includes the bird's-eye view embedding features corresponding to the window area.

[0091] b. For each window in a preset number of windows, determine the second query tensor based on that window.

[0092] Specifically, the second query tensor can be derived by mapping the bird's-eye view embedding features of the window into the second query tensor based on the second query mapping matrix.

[0093] c. Based on the first encoding results corresponding to each viewpoint in the viewpoint group corresponding to the window, determine the second key tensor and the second value tensor.

[0094] The view group corresponding to each window is a pre-determined view group that includes at least one view, meaning the view group for each window is determined based on all views. For example, for the four windows mentioned above, the view group corresponding to the left front (FL) window includes the front view, left front view, and left rear view; the view group corresponding to the right front (FR) window includes the front view, right front view, and right rear view; the view group corresponding to the left rear (BL) window may include the rear view, left rear view, and left front view; the view group corresponding to the right rear (BR) window may include the rear view, right rear view, and right front view; and so on. Specific settings can be configured according to actual needs. After determining the view group corresponding to each window, the second key tensor and second value tensor used for cross-attention are determined based on the first encoding results corresponding to each view in the view group. This achieves view-based cross-attention between the first encoding results of each view and the bird's-eye view embedded features. By dividing the window based on view, each bird's-eye view grid does not need to interact with the encoding results of all views, thus effectively reducing inference time and resource consumption and improving inference speed.

[0095] d. Based on the second query tensor, the second key tensor, and the second value tensor, use a cross-attention network to determine the cross-attention result corresponding to the window.

[0096] Since the second query tensor is determined based on the bird's-eye view embedding features of the window, and the second key tensor and the second value tensor are determined based on the first encoding result corresponding to the first image features of the view group corresponding to the window, the attention operation based on the second query tensor, the second key tensor and the second value tensor is a cross-attention operation. That is, the interaction between the bird's-eye view embedding features of each window and the first encoding result of its corresponding view group can obtain the cross-attention result corresponding to the window.

[0097] e. Based on the cross-attention result corresponding to the window, determine the decoding result corresponding to the window;

[0098] Since a decoder typically includes one or more decoder blocks, and each decoder block includes a cross-attention network and other related networks, after obtaining the cross-attention result corresponding to the window, it is necessary to use other related networks to process the cross-attention result. After processing all decoder blocks, the decoding result corresponding to the window is obtained.

[0099] f. Based on the decoding results corresponding to each window, determine the features of the bird's-eye view.

[0100] Since each window corresponds to a portion of the bird's-eye view, the decoding result for each window is the bird's-eye view feature of that window. Therefore, the decoding results for each window together constitute the overall bird's-eye view feature.

[0101] This disclosure achieves cross-attention based on view prior by dividing the embedded features of the bird's-eye view into windows and setting relevant view groups for each window. On the one hand, it can effectively reduce the number of views interacting with the window, thereby effectively reducing inference time and resource consumption. On the other hand, since the view groups set for the window can be views that are more relevant to the window (for example, the right front view is basically completely unrelated to the left rear window, so the view group corresponding to the left rear window does not need to include the right front view), the cross-attention calculation is also more efficient and more conducive to model convergence.

[0102] In one optional example, Figure 7This is a schematic diagram of the network structure of a view-aware attention network provided in an exemplary embodiment of this disclosure. In this diagram, Encoder×1 represents an encoder consisting of encoder blocks in one image, Decoder×6 represents a decoder consisting of decoder blocks in six images, BEV Embedding represents bird's-eye view embedding features, Image PosEmbedding represents image position encoding, Image Feature represents a first image feature, CAM_BL represents a left rear view, CAM_FR represents a right front view, CAM_B represents a rear view, CAM_FL represents a left front view, Q1 represents a first query tensor, K1 represents a first key tensor, V1 represents a first value tensor, Self Attention represents a self-attention network, Add&LayerNorm represents an addition and layer normalization network, Feed Forward represents a feedforward network, Q2 represents a second query tensor, K2 represents a second key tensor, V2 represents a second value tensor, Cross Attention represents a cross-attention network, BEV Representation represents bird's-eye view features, and c represents concat (stitching). Taking a viewpoint CAM_BL as an example, the first query tensor Q1 is determined based on the first image feature corresponding to the viewpoint. The first key tensor K1 and the first value tensor V1 are determined based on the first image feature and image position embedding corresponding to the viewpoint. The first query tensor Q1, the first key tensor K1, and the first value tensor V1 complete the self-attention operation in the encoder's self-attention network to obtain the self-attention result corresponding to the viewpoint. The self-attention result is added to the first image feature by Add&LayerNorm and then normalized to obtain the first normalized result. The first normalized result is then processed by Feed Forward and Add&LayerNorm to obtain the first encoding result corresponding to the viewpoint. The first encoding results of the three views included in view group 1 correspond to the BL window of BEV Embedding. The BL window is used to determine the second query tensor Q2. The first encoding results of each view in view group 1 are concatenated with the image position encoding to determine the second key tensor K2. The first encoding results of each view in view group 1 are also used to determine the second value tensor V2. Q2, K2, and V2 complete cross-attention in the cross-attention network to realize the interaction between the first encoding results of each view in view group 1 and the bird's-eye view embedding features, and obtain the cross-attention result corresponding to the window. The cross-attention result is passed through the subsequent network of the decoder block and the subsequent decoder block to finally obtain the decoding result of the window. Based on the decoding results of each window, the bird's-eye view features (BEV Representation) are obtained.This is only an exemplary network structure diagram. In practical applications, the specific network structure of the encoder and decoder in the view perception attention network can be based on any Transformer structure and implemented according to the actual needs of this disclosure. It is not limited to the specific structure shown in the figure above, and can be set according to actual needs.

[0103] In an optional example, step 202, which determines the three-dimensional bird's-eye view location code based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code, includes:

[0104] Step 2021: Based on the two-dimensional bird's-eye view location encoding and bird's-eye view content encoding, the three-dimensional bird's-eye view location encoding is determined using a pre-trained location-aware enhancement network.

[0105] The location-aware augmentation network is used to predict the reference height of each location based on the location encoding of the two-dimensional bird's-eye view. The height encoding is obtained by performing location encoding on the reference height, and location-related information is extracted from the content encoding of the bird's-eye view to update the height encoding. The updated height encoding is then combined with the two-dimensional bird's-eye view location encoding to obtain the three-dimensional bird's-eye view location encoding.

[0106] This disclosure enhances the location encoding of a two-dimensional bird's-eye view through a location-aware augmentation network to obtain a three-dimensional bird's-eye view location encoding, thereby improving the accuracy and effectiveness of the three-dimensional bird's-eye view location encoding and thus improving the accuracy of bird's-eye view features.

[0107] Figure 8 This is a flowchart illustrating step 2021 provided in an example embodiment of this disclosure.

[0108] In an optional example, step 2021, based on the two-dimensional bird's-eye view location encoding and bird's-eye view content encoding, utilizes a pre-trained location-aware enhancement network to determine the three-dimensional bird's-eye view location encoding, including:

[0109] Step 20211: Based on the bird's-eye view content encoding, the first feedforward network in the location-aware augmentation network is used to determine the first intermediate result.

[0110] The first feedforward network may include an input layer, a hidden layer, and an output layer. The specific structure of each layer can be set according to actual needs. For example, the number of neurons in each layer and the number of hidden layers can be set according to actual needs. The details will not be elaborated here.

[0111] Step 20212: Based on the location encoding of the two-dimensional bird's-eye view, the second feedforward network in the location-aware enhancement network is used to determine the second intermediate result.

[0112] The second feedforward network is similar to the first feedforward network and can be set according to actual needs, so it will not be elaborated here.

[0113] Step 20213: Based on the second intermediate result, the third intermediate result is determined using the first activation function in the location-aware augmentation network.

[0114] The first activation function can be set according to actual needs, such as the Sigmoid activation function, and there is no specific limitation.

[0115] Step 20214: Based on the third intermediate result, determine the reference height, which is the coordinate value of the Z-axis direction in the lidar coordinate system corresponding to the bird's-eye view.

[0116] The reference height can be obtained by normalizing (Norm) the third intermediate result, that is, scaling the third intermediate result obtained by the activation function to a preset height range (z). min ,z max For example, in the third intermediate result obtained by the Sigmoid activation function, the value corresponding to each position is a value in the range (0,1). This value is then scaled to a preset height range (z). min ,z max Within ) is used as a reference height.

[0117] Step 20215: Determine the three-dimensional bird's-eye view location code based on the reference height, the two-dimensional bird's-eye view location code, and the first intermediate result.

[0118] After obtaining the reference height, a positional encoding can be performed on the reference height to obtain a height code. Combining the height code, the positional code of the 2D bird's-eye view, and the first intermediate result, the positional code of the 3D bird's-eye view is determined. The positional encoding of the reference height can be implemented using sinusoidal positional encoding, which can be set according to actual needs. For example, the height code is multiplied by the corresponding element of the first intermediate result to obtain the updated height code. The updated height code is then added to the 2D bird's-eye view positional code to obtain the 3D bird's-eye view positional code.

[0119] For example, the location encoding of a two-dimensional bird's-eye view is represented as follows: Then refer to height Z ref Represented as:

[0120] Z ref = Norm(Sigmoid(FFN(Q p )))

[0121] Where FFN represents the second feedforward network, Sigmoid is the activation function, and Norm represents normalization.

[0122] For reference height Zref Perform position encoding to obtain the height encoding Q. ref It is expressed as follows:

[0123] Q ref =Sinusoidal(Z) ref )

[0124] Sinusoidal represents the sinusoidal position code.

[0125] The content of the bird's-eye view is coded as follows: The location-related information is extracted from the content encoding of the bird's-eye view, and the first intermediate result M is represented as follows:

[0126] M = FFN(Q) c )

[0127] Here, FFN represents the first feedforward network.

[0128] Based on the first intermediate result, height encoding, and 2D bird's-eye view location encoding, the 3D bird's-eye view location encoding Q is determined. ep It is expressed as follows:

[0129] Q ep =add(M·Q) ref Q p )

[0130] Among them, M·Q ref Indicate M and Q ref Element-wise multiplication, add(M·Q) ref Q p ) represents M·Q ref and Q p Add them together.

[0131] This disclosure extracts location-related information (first intermediate result) from the content encoding of a bird's-eye view through a first feedforward network, predicts a reference height based on the two-dimensional bird's-eye view location encoding, and determines the three-dimensional bird's-eye view location encoding by combining the first intermediate result, the two-dimensional bird's-eye view location encoding, and the reference height, thereby further improving the accuracy of bird's-eye view features.

[0132] Figure 9 This is a flowchart illustrating step 20215 provided in an exemplary embodiment of this disclosure.

[0133] In an optional example, step 20215, which determines the 3D bird's-eye view location code based on the reference height, the 2D bird's-eye view location code, and the first intermediate result, includes:

[0134] A. Perform position encoding on the reference height to obtain the first height encoding.

[0135] The position encoding of the reference height can be achieved using sinusoidal position encoding.

[0136] B. Determine the second height code based on the first height code and the first intermediate result.

[0137] The first height code and the first intermediate result are matrices or tensors of the same dimension. Specifically, the first height code includes height encoding information corresponding to each position in the two-dimensional bird's-eye view, and the first intermediate result includes position-related information corresponding to each position in the two-dimensional bird's-eye view. The result of multiplying the corresponding elements of the first height code and the first intermediate result is the second height code. Element-wise multiplication means multiplying the element values ​​at the same positions in the two matrices or tensors, resulting in a second height code with the same dimension as the first height code.

[0138] C. Determine the location code of the three-dimensional bird's-eye view based on the second altitude code and the two-dimensional bird's-eye view location code.

[0139] Specifically, the second altitude code is added to the two-dimensional bird's-eye view location code to obtain the enhanced three-dimensional bird's-eye view location code.

[0140] For example, Figure 10 This is a schematic diagram of the structure of a position-aware enhancement network provided in an exemplary embodiment of this disclosure. In the diagram, Position-Aware Enhancement represents the position-aware enhancement network, Feed Forward-1 represents the first feedforward network, Feed Forward-2 represents the second feedforward network, Norm represents the normalization network, Sigmoid represents the activation function, and Sinusoidal Embedding represents sinusoidal position encoding. This indicates element-wise multiplication, where 'a' represents 'add' (addition). The bird's-eye view content is encoded in Q. c After Feed Forward-1, the first intermediate result M and the two-dimensional bird's-eye view location code Q are obtained. p The second intermediate result is obtained through Feed Forward-2. This second intermediate result is then processed by Sigmoid to obtain the third intermediate result. This third intermediate result is then activated by Sigmoid to obtain a third intermediate result. Finally, this third intermediate result is normalized using Norm to obtain the reference height Z. ref Using Sinusoidal Embedding to reference height Z ref Perform position encoding to obtain the first height encoding Q. ref First height encoding Q ref Multiplying the corresponding element of the first intermediate result M yields the second height code and the two-dimensional bird's-eye view location code Q. p Add them together to obtain the location code Q of the 3D bird's-eye view.ep .

[0141] In one optional example, a position-aware enhancement network and a viewpoint-aware attention network, combined with the corresponding task's head network (such as the detection head network for object detection, the head network for speech segmentation, etc.), constitute the complete model's network structure. A trained model is obtained through training. During training, the position-aware enhancement network can be trained using either supervised or unsupervised methods, depending on the specific needs. For supervised training, the height label Z can be used as the basis for training. r The training reference height Z of the network output ref_l Supervision is conducted based on the high-level label Z. r With training reference height Z ref_l The height loss is determined and used as part of the overall network loss to adjust the network parameters, thereby achieving supervised learning of the reference height.

[0142] This disclosure extracts location-related information (first intermediate result) from the bird's-eye view content encoding through a first feedforward network, which is used to update the height encoding of the reference height predicted based on the two-dimensional bird's-eye view location encoding, thereby further enhancing the bird's-eye view location information based on the bird's-eye view content encoding, so as to further improve the accuracy of the bird's-eye view features.

[0143] The embodiments or optional examples disclosed above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit it.

[0144] The method for determining any of the bird's-eye view features provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, the method for determining any of the bird's-eye view features provided in this disclosure can be executed by a processor, such as by a processor executing the method for determining any of the bird's-eye view features mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0145] Figure 11 This is a schematic flowchart of an exemplary embodiment of the target detection method provided in this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 11 As shown, it includes the following steps:

[0146] Step 301: Based on the images to be processed corresponding to each of the at least one viewpoint, determine the first image features corresponding to each of the viewpoints.

[0147] Step 302: Determine the location code of the three-dimensional bird's-eye view based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code.

[0148] Step 303: Based on the first image features corresponding to each of the aforementioned viewpoints and the three-dimensional bird's-eye view location code, determine the bird's-eye view features corresponding to the at least one viewpoint.

[0149] The specific operations of steps 301-302 can be found in steps 201-203 of the aforementioned embodiment and their further details, and will not be repeated here.

[0150] Step 304: Based on the bird's-eye view features, use the detection head network to determine the target detection result.

[0151] The detection head network can adopt any feasible network structure, which is not limited in this disclosure. For example, a multilayer perceptron (MLP) can be used. The target detection result can include 3D detection box information and its corresponding type. The 3D detection box information can specifically include the position parameters of the 3D detection box, which can be set according to actual needs, as long as the position of the 3D detection box can be determined. The 3D detection box can be a 3D detection box in the world coordinate system or a 3D detection box in the vehicle coordinate system; there is no specific limitation.

[0152] Any of the target detection methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the target detection methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the target detection methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0153] Exemplary device

[0154] Figure 12 This is a schematic diagram of a device for determining bird's-eye view features provided in an exemplary embodiment of this disclosure. The device in this embodiment can be used to implement corresponding bird's-eye view feature determination method embodiments of this disclosure, such as... Figure 12 The device shown includes: a first processing module 501, a second processing module 502, and a third processing module 503.

[0155] The first processing module 501 is used to determine the first image features corresponding to each of the at least one viewpoints based on the images to be processed corresponding to each of the at least one viewpoints; the second processing module 502 is used to determine the three-dimensional bird's-eye view position code based on the pre-obtained two-dimensional bird's-eye view position code and bird's-eye view content code; the third processing module 503 is used to determine the bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the at least one viewpoints and the three-dimensional bird's-eye view position code.

[0156] Figure 13 This is a schematic diagram of the structure of a device for determining bird's-eye view features provided in another exemplary embodiment of this disclosure.

[0157] In an optional example, the third processing module 503 includes: a first processing unit 5031 and a second processing unit 5032.

[0158] The first processing unit 5031 is used to determine the bird's-eye view embedding features based on the three-dimensional bird's-eye view location encoding and the bird's-eye view content encoding;

[0159] The second processing unit 5032 is used to determine the bird's-eye view features based on the first image features and the bird's-eye view embedding features corresponding to each of the aforementioned viewpoints, using a pre-trained viewpoint awareness attention network.

[0160] Figure 14 This is a schematic diagram of the structure of the second processing unit 5032 provided in an exemplary embodiment of the present disclosure.

[0161] In an optional example, the view-aware attention network includes an encoder and a decoder, and the second processing unit 5032 includes: a first processing subunit 50321 and a second processing subunit 50322.

[0162] The first processing subunit 50321 is used to determine the first encoding result corresponding to each of the first image features based on each of the first image features, using the encoder; the second processing subunit 50322 is used to determine the bird's-eye view feature based on the first encoding result corresponding to each of the first image features and the bird's-eye view embedding feature, using the decoder.

[0163] In an optional example, the encoder includes a self-attention network; the first processing subunit 50321 is specifically used for:

[0164] For each of the first image features corresponding to a given viewpoint, based on the first image feature and the image position encoding corresponding to the first image feature, a first query tensor, a first key tensor, and a first value tensor corresponding to the first image feature are determined; based on the first query tensor, the first key tensor, and the first value tensor, the self-attention network is used to determine the self-attention result corresponding to the first image feature; based on the self-attention result, the encoding result corresponding to the first image feature is determined; and the encoding results corresponding to each of the first image features are used as the first encoding results corresponding to each of the given viewpoints.

[0165] In an optional example, the decoder includes a cross-attention network; the second processing subunit 50322 is specifically used for:

[0166] Based on a preset window segmentation rule, a preset number of windows corresponding to the bird's-eye view embedding features are determined; for each of the preset number of windows, a second query tensor is determined based on the window; based on the first encoding results corresponding to each view in the view group corresponding to the window, a second key tensor and a second value tensor are determined; based on the second query tensor, the second key tensor, and the second value tensor, the cross-attention network is used to determine the cross-attention result corresponding to the window; based on the cross-attention result corresponding to the window, the decoding result corresponding to the window is determined; based on the decoding results corresponding to each window, the bird's-eye view features are determined.

[0167] In an optional example, the second processing module 502 includes: a third processing unit 5021, used to determine the three-dimensional bird's-eye view location code based on the two-dimensional bird's-eye view location code and the bird's-eye view content code, using a pre-trained location-aware enhancement network.

[0168] In one optional example, Figure 15 This is a schematic diagram of the structure of a third processing unit 5021 provided in an exemplary embodiment of the present disclosure. In this example, the third processing unit 5021 includes: a third processing subunit 50211, a fourth processing subunit 50212, a fifth processing subunit 50213, a sixth processing subunit 50214, and a seventh processing subunit 50215.

[0169] The third processing subunit 50211 is used to determine a first intermediate result based on the bird's-eye view content encoding and using the first feedforward network in the position-aware enhancement network; the fourth processing subunit 50212 is used to determine a second intermediate result based on the two-dimensional bird's-eye view position encoding and using the second feedforward network in the position-aware enhancement network; the fifth processing subunit 50213 is used to determine a third intermediate result based on the second intermediate result and using the first activation function in the position-aware enhancement network; the sixth processing subunit 50214 is used to determine a reference height based on the third intermediate result, wherein the reference height is the coordinate value of the Z-axis direction in the lidar coordinate system corresponding to the bird's-eye view; and the seventh processing subunit 50215 is used to determine the three-dimensional bird's-eye view position encoding based on the reference height, the two-dimensional bird's-eye view position encoding, and the first intermediate result.

[0170] In an optional example, the seventh processing subunit 50215 is specifically used to: perform position encoding on the reference height to obtain a first height encoding; determine a second height encoding based on the first height encoding and the first intermediate result; and determine the three-dimensional bird's-eye view position encoding based on the second height encoding and the two-dimensional bird's-eye view position encoding.

[0171] Figure 16 This is a schematic diagram of the target detection apparatus provided in an example embodiment of this disclosure. The apparatus in this embodiment can be used to implement corresponding target detection method embodiments of this disclosure, such as... Figure 16 The device shown includes: a fourth processing module 601, a fifth processing module 602, a sixth processing module 603, and a seventh processing module 604.

[0172] The fourth processing module 601 is used to determine the first image features corresponding to each of the at least one viewpoints based on the images to be processed corresponding to each of the at least one viewpoints; the fifth processing module 602 is used to determine the three-dimensional bird's-eye view position code based on the pre-obtained two-dimensional bird's-eye view position code and bird's-eye view content code; the sixth processing module 603 is used to determine the bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the at least one viewpoints and the three-dimensional bird's-eye view position code; the seventh processing module 604 is used to determine the target detection result based on the bird's-eye view features using a detection head network.

[0173] In this embodiment, the specific structures of the fourth processing module 601, the fifth processing module 602, and the sixth processing module 603 are the same as those of the first processing module 501, the second processing module 502, and the third processing module 503 in the aforementioned embodiments. Therefore, the specific operation of each module in this embodiment can be found in the corresponding modules and their detailed structures in the aforementioned embodiments, and will not be repeated here.

[0174] Exemplary electronic devices

[0175] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0176] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the method for determining bird's-eye view features as described in any of the above embodiments of the present disclosure, or implements the target detection method as described in any of the above embodiments of the present disclosure.

[0177] Figure 17 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0178] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0179] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0180] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0181] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0182] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0183] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0184] Of course, for the sake of simplicity, Figure 17 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0185] Exemplary computer program products and computer-readable storage media

[0186] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0187] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0188] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0189] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0190] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0191] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0192] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0193] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0194] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0195] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0196] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for determining features in a bird's-eye view, comprising: Based on the images to be processed corresponding to each of the at least one viewpoint, determine the first image features corresponding to each of the at least one viewpoint; Based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code, the three-dimensional bird's-eye view location code is determined; the two-dimensional bird's-eye view location code represents the two-dimensional location information of the bird's-eye view; the bird's-eye view content code represents the content information of the corresponding location on the bird's-eye view; the three-dimensional bird's-eye view location code is a three-dimensional location code in the bird's-eye view space; Based on the first image features corresponding to each of the aforementioned viewpoints and the three-dimensional bird's-eye view location code, the bird's-eye view features corresponding to the at least one viewpoint are determined. The step of determining the three-dimensional bird's-eye view location code based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code includes: Based on the two-dimensional bird's-eye view location code and the bird's-eye view content code, the three-dimensional bird's-eye view location code is determined using a pre-trained location-aware enhancement network. The location-aware enhancement network is used to determine the height code of each location based on the location code of the two-dimensional bird's-eye view, and to determine the location code of the three-dimensional bird's-eye view based on the height code and the content code of the bird's-eye view.

2. The method of claim 1, wherein, The step of determining the bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the aforementioned viewpoints and the three-dimensional bird's-eye view location encoding includes: Based on the location encoding and content encoding of the 3D bird's-eye view, the embedding features of the bird's-eye view are determined; Based on the first image features and the bird's-eye view embedding features corresponding to each of the aforementioned viewpoints, the bird's-eye view features are determined using a pre-trained viewpoint awareness attention network.

3. The method of claim 2, wherein, The viewpoint-aware attention network includes an encoder and a decoder. The step of determining the bird's-eye view features based on the first image features corresponding to each viewpoint and the bird's-eye view embedding features, using the pre-trained viewpoint-aware attention network, includes: Based on the first image features corresponding to each of the aforementioned viewpoints, the encoder is used to determine the first encoding result corresponding to each of the aforementioned viewpoints. Based on the first encoding result corresponding to each of the aforementioned viewpoints and the bird's-eye view embedding features, the decoder is used to determine the bird's-eye view features.

4. The method of claim 3, wherein, The encoder includes a self-attention network; the step of determining the first encoding result corresponding to each of the first image features corresponding to each of the viewpoints using the encoder includes: For each of the first image features corresponding to the viewpoint, based on the first image feature and the image position code corresponding to the first image feature, determine the first query tensor, the first key tensor, and the first value tensor corresponding to the first image feature; Based on the first query tensor, the first key tensor, and the first value tensor, the self-attention network is used to determine the self-attention result corresponding to the first image feature. Based on the self-attention result, determine the encoding result corresponding to the first image feature; The encoding results corresponding to each of the first image features are used as the first encoding results corresponding to each of the viewpoints.

5. The method of claim 4, wherein, The decoder includes a cross-attention network; the step of determining the bird's-eye view features using the decoder based on the first encoding results corresponding to each of the aforementioned viewpoints and the bird's-eye view embedding features includes: Based on preset window segmentation rules, a preset number of windows corresponding to the bird's-eye view embedding features are determined; For each of the predetermined number of windows, a second query tensor is determined based on that window; Based on the first encoding result corresponding to each view in the view group corresponding to the window, the second key tensor and the second value tensor are determined. Based on the second query tensor, the second key tensor, and the second value tensor, the cross-attention network is used to determine the cross-attention result corresponding to the window; Based on the cross-attention result corresponding to the window, determine the decoding result corresponding to the window; Based on the decoding results corresponding to each of the windows, the bird's-eye view features are determined.

6. The method of claim 5, wherein, The step of determining the 3D bird's-eye view location code based on the 2D bird's-eye view location code and the bird's-eye view content code, using a pre-trained location-aware enhancement network, includes: Based on the encoding of the bird's-eye view content, the first intermediate result is determined using the first feedforward network in the location-aware enhancement network; Based on the location encoding of the two-dimensional bird's-eye view, the second intermediate result is determined using the second feedforward network in the location-aware enhancement network; Based on the second intermediate result, a third intermediate result is determined using the first activation function in the location-aware enhancement network; Based on the third intermediate result, a reference height is determined, which is the coordinate value of the Z-axis direction in the lidar coordinate system corresponding to the bird's-eye view; Based on the reference height, the two-dimensional bird's-eye view location code, and the first intermediate result, the three-dimensional bird's-eye view location code is determined.

7. The method according to claim 6, wherein, The step of determining the three-dimensional bird's-eye view location code based on the reference height, the two-dimensional bird's-eye view location code, and the first intermediate result includes: The reference height is position-encoded to obtain a first height code; Based on the first height code and the first intermediate result, determine the second height code; The location code of the three-dimensional bird's-eye view is determined based on the second height code and the two-dimensional bird's-eye view location code.

8. A target detection method, comprising: Based on the images to be processed corresponding to each of the at least one viewpoint, determine the first image features corresponding to each of the at least one viewpoint; Based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code, the three-dimensional bird's-eye view location code is determined; the two-dimensional bird's-eye view location code represents the two-dimensional location information of the bird's-eye view; the bird's-eye view content code represents the content information of the corresponding location on the bird's-eye view; the three-dimensional bird's-eye view location code is a three-dimensional location code in the bird's-eye view space; Based on the first image features corresponding to each of the aforementioned viewpoints and the three-dimensional bird's-eye view location code, the bird's-eye view features corresponding to the at least one viewpoint are determined. Based on the bird's-eye view features, a detection head network is used to determine the target detection result; The step of determining the three-dimensional bird's-eye view location code based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code includes: Based on the two-dimensional bird's-eye view location code and the bird's-eye view content code, the three-dimensional bird's-eye view location code is determined using a pre-trained location-aware enhancement network. The location-aware enhancement network is used to determine the height code of each location based on the location code of the two-dimensional bird's-eye view, and to determine the location code of the three-dimensional bird's-eye view based on the height code and the content code of the bird's-eye view.

9. A device for determining features in a bird's-eye view, comprising: The first processing module is used to determine the first image features corresponding to each of the at least one viewpoints based on the images to be processed corresponding to each of the at least one viewpoints. The second processing module is used to determine the three-dimensional bird's-eye view location code based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code; the two-dimensional bird's-eye view location code represents the two-dimensional location information of the bird's-eye view; the bird's-eye view content code represents the content information of the corresponding location on the bird's-eye view; the three-dimensional bird's-eye view location code is a three-dimensional location code in the bird's-eye view space; The third processing module is used to determine the bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view location code. The second processing module includes: The third processing unit is used to determine the three-dimensional bird's-eye view location code based on the two-dimensional bird's-eye view location code and the bird's-eye view content code, using a pre-trained location-aware enhancement network. The location-aware enhancement network is used to determine the height code of each location based on the location code of the two-dimensional bird's-eye view, and to determine the location code of the three-dimensional bird's-eye view based on the height code and the content code of the bird's-eye view.

10. A target detection device, comprising: The fourth processing module is used to determine the first image features corresponding to each of the at least one viewpoint based on the images to be processed corresponding to each of the viewpoints. The fifth processing module is used to determine the three-dimensional bird's-eye view location code based on the pre-obtained two-dimensional bird's-eye view location code and bird's-eye view content code; the two-dimensional bird's-eye view location code represents the two-dimensional location information of the bird's-eye view; the bird's-eye view content code represents the content information of the corresponding location on the bird's-eye view; the three-dimensional bird's-eye view location code is a three-dimensional location code in the bird's-eye view space; The sixth processing module is used to determine the bird's-eye view features corresponding to the at least one viewpoint based on the first image features corresponding to each of the viewpoints and the three-dimensional bird's-eye view position code. The seventh processing module is used to determine the target detection result based on the bird's-eye view features and using a detection head network; The fifth processing module includes: The third processing unit is used to determine the three-dimensional bird's-eye view location code based on the two-dimensional bird's-eye view location code and the bird's-eye view content code, using a pre-trained location-aware enhancement network. The location-aware enhancement network is used to determine the height code of each location based on the location code of the two-dimensional bird's-eye view, and to determine the location code of the three-dimensional bird's-eye view based on the height code and the content code of the bird's-eye view.

11. A computer-readable storage medium storing a computer program for performing the method for determining bird's-eye view features according to any one of claims 1-7, or the computer program for performing the target detection method according to claim 8.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining bird's-eye view features as described in any one of claims 1-7, or to execute the instructions to implement the target detection method as described in claim 8.

Images