Methods for determining features in bird's-eye view images, image processing methods, apparatus and equipment

Abstract

This disclosure provides a method, image processing method, apparatus, and device for determining bird's-eye view features. The method includes: determining first image features corresponding to each viewpoint based on images to be processed corresponding to each viewpoint in at least one viewpoint; performing a cross-attention operation based on local windows based on each first image feature and a first bird's-eye view query feature to obtain a first attention result; determining a second bird's-eye view query feature based on the first attention result; performing a cross-attention operation based on local and global windows based on the second bird's-eye view query feature and each first image feature to obtain a second attention result; and determining target bird's-eye view features corresponding to at least one viewpoint based on the second attention result. This disclosure significantly reduces the computational complexity of the cross-attention operation through window segmentation, and effectively improves the accuracy and effectiveness of bird's-eye view features through local and global interactions.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method for determining features of a bird's-eye view, an image processing method, an apparatus, and a device. Background Technology

[0002] In fields such as autonomous driving and assisted driving, images captured by multiple cameras on a vehicle using vision technology can be used to determine bird's-eye view (BEV) features. These BEV features are then used for tasks such as object detection and semantic segmentation, making this an emerging research focus. Related technologies typically use attention mechanisms to convert image features into BEV features. However, these attention mechanisms are usually based on global attention, resulting in high computational complexity and significant challenges in deployment on the vehicle side. Summary of the Invention

[0003] To address the aforementioned technical issues of high computational complexity, embodiments of this disclosure provide a method, image processing method, apparatus, and device for determining bird's-eye view features. Based on local and global window partitioning, it enables local and global interaction between bird's-eye view query features and image features, effectively reducing the computational complexity of determining bird's-eye view features, improving processing efficiency, and facilitating deployment on vehicle-side devices.

[0004] The first aspect of this disclosure provides a method for determining bird's-eye view features, comprising: determining first image features corresponding to each of the at least one viewpoints based on images to be processed corresponding to each of the viewpoints; performing a cross-attention operation based on local windows based on each of the first image features and a first bird's-eye view query feature to obtain a first attention result; determining a second bird's-eye view query feature based on the first attention result; performing a cross-attention operation based on local and global windows based on the second bird's-eye view query feature and each of the first image features to obtain a second attention result; and determining target bird's-eye view features corresponding to the at least one viewpoint based on the second attention result.

[0005] A second aspect of this disclosure provides an image processing method, comprising: acquiring images to be processed corresponding to each of at least one viewpoint; determining target bird's-eye view features based on the images to be processed corresponding to each of the viewpoints; and determining an image processing result based on the target bird's-eye view features and a prediction head network; wherein the target bird's-eye view features are obtained based on the method for determining bird's-eye view features provided in any of the above embodiments.

[0006] A third aspect of this disclosure provides an apparatus for determining bird's-eye view features, comprising: a first processing module, configured to determine first image features corresponding to each of the at least one viewpoints based on images to be processed corresponding to each of the viewpoints; a second processing module, configured to perform a cross-attention operation based on local windows based on each of the first image features and a first bird's-eye view query feature to obtain a first attention result; a third processing module, configured to determine a second bird's-eye view query feature based on the first attention result; a fourth processing module, configured to perform a cross-attention operation based on local and global windows based on the second bird's-eye view query feature and each of the first image features to obtain a second attention result; and a fifth processing module, configured to determine target bird's-eye view features corresponding to the at least one viewpoint based on the second attention result.

[0007] A fourth aspect of this disclosure provides an image processing apparatus, comprising: an image acquisition module for acquiring images to be processed corresponding to each of at least one viewpoint; a bird's-eye view feature determination module for determining target bird's-eye view features based on the images to be processed corresponding to each of the viewpoints; and a prediction module for determining an image processing result based on the target bird's-eye view features and a prediction head network; wherein the target bird's-eye view features are obtained based on the bird's-eye view feature determination method provided in any of the above embodiments.

[0008] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing a method for determining bird's-eye view features as described in any of the above embodiments of this disclosure, or for performing an image processing method as described in any of the above embodiments of this disclosure.

[0009] A sixth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method for determining bird's-eye view features as described in any of the above embodiments of this disclosure, or to execute the image processing method as described in any of the above embodiments of this disclosure.

[0010] A seventh aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs either the method for determining bird's-eye view features provided in any of the above embodiments of this disclosure, or the image processing method described in any of the above embodiments of this disclosure.

[0011] Based on the bird's-eye view feature determination method, image processing method, apparatus, and device provided in the above embodiments of this disclosure, by performing a cross-attention operation based on local windows on the first image features and the first bird's-eye view query features corresponding to each viewpoint, local interaction between the first bird's-eye view query features and the first image features can be achieved. Based on the first attention result obtained from the local interaction, the second bird's-eye view query features are determined. Then, based on the second bird's-eye view query features and each of the first image features, a cross-attention operation based on local and global windows is performed to achieve global interaction between the second bird's-eye view query features and the first image features. Based on the second attention result obtained from the global interaction, the target bird's-eye view features are obtained. On the one hand, window segmentation can greatly reduce the computational complexity of determining bird's-eye view features from multi-view image features. On the other hand, through local and global interaction, both local and global features of the first image features are taken into account, thereby effectively improving the accuracy and effectiveness of the determined bird's-eye view features. Attached Figure Description

[0012] Figure 1 This is an exemplary application scenario of the method for determining bird's-eye view features provided in this disclosure;

[0013] Figure 2 This is a flowchart illustrating a method for determining bird's-eye view features provided in an exemplary embodiment of this disclosure;

[0014] Figure 3 This is a flowchart illustrating a method for determining bird's-eye view features provided in another exemplary embodiment of this disclosure;

[0015] Figure 4 This is a schematic diagram of partial window segmentation provided in an exemplary embodiment of this disclosure;

[0016] Figure 5 This is a schematic diagram illustrating the global window segmentation principle provided in an exemplary embodiment of this disclosure;

[0017] Figure 6 This is a schematic diagram of the network architecture of a bird's-eye view feature determination model provided in an exemplary embodiment of this disclosure;

[0018] Figure 7 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0019] Figure 8 This is a schematic diagram of the structure of a bird's-eye view feature determination device provided in an exemplary embodiment of this disclosure;

[0020] Figure 9 This is a schematic diagram of the structure of a device for determining bird's-eye view features provided in another exemplary embodiment of this disclosure;

[0021] Figure 10This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0022] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0025] This disclosure outlines

[0026] In developing this disclosure, the inventors discovered that in fields such as autonomous driving and assisted driving, bird's-eye view features can be determined from images captured by multiple cameras on a vehicle using visual technology. These bird's-eye view features are then used for tasks such as object detection and semantic segmentation, making this an emerging research focus. Related technologies typically use attention mechanisms to determine bird's-eye view features from image features. However, these attention mechanisms are usually based on global attention, resulting in high computational complexity and significant challenges in deployment on the vehicle side.

[0027] Exemplary Overview

[0028] Figure 1 This is an exemplary application scenario of the method for determining bird's-eye view features provided in this disclosure.

[0029] In the target detection scenario of autonomous driving, images acquired by image sensors at various viewpoints (e.g., viewpoint 1 to viewpoint m, where m is a positive integer) on the vehicle (or images acquired after certain preprocessing) can be used as images to be processed, obtaining images to be processed corresponding to m viewpoints respectively. Using the bird's-eye view feature determination method of this disclosure (executed in the bird's-eye view feature determination device), first image features corresponding to each viewpoint can be determined based on the images to be processed corresponding to each viewpoint. For example, a pre-trained feature extraction network can be used to extract features from each image to be processed, obtaining first image features corresponding to each viewpoint. Based on each first image feature and the first bird's-eye view query feature, a cross-attention operation based on local windows is performed to obtain a first attention result. Based on the first attention result, a second bird's-eye view query feature is determined. Based on the second bird's-eye view query feature and each first image feature, a cross-attention operation based on local and global windows is performed to obtain a second attention result. Based on the second attention result, target bird's-eye view features corresponding to at least one viewpoint are determined, and then target detection results can be obtained using a detection head network based on the target bird's-eye view features. The detection head network can be configured according to actual needs. For example, the detection head network can be a multilayer perceptron (MLP), a convolutional network, etc., and this disclosure does not limit it. The cross-attention operation based on local windows refers to segmenting the bird's-eye view query features and the first image features into local windows, and performing cross-attention operations within the corresponding local windows to achieve local information interaction between the bird's-eye view query features and the image features. The cross-attention operation based on local and global windows refers to performing cross-attention operations between the local windows of the bird's-eye view query features and the corresponding global windows of the first image features to achieve global information interaction between the bird's-eye view query features and the image features. On the one hand, window segmentation can greatly reduce the computational complexity of determining bird's-eye view features from multi-view image features; on the other hand, through local and global interaction, both local and global features of the first image features are taken into account, thereby effectively improving the accuracy and effectiveness of the determined bird's-eye view features, and thus improving the accuracy of the target detection results.

[0030] The method for determining bird's-eye view features provided in this embodiment is not limited to the target detection scenario described above, but can also be applied to semantic segmentation, target tracking, and other scenarios. No specific scenario is limited.

[0031] Exemplary methods

[0032] Figure 2 This is a flowchart illustrating a method for determining bird's-eye view features according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 2 As shown, it includes the following steps:

[0033] Step 201: Based on the images to be processed corresponding to each viewpoint in at least one viewpoint, determine the first image features corresponding to each viewpoint.

[0034] Here, "viewpoint" refers to the camera's (or viewpoint's) perspective. For example, in an autonomous driving scenario, each camera on the vehicle corresponds to one viewpoint. The number of viewpoints can be set according to actual needs. For instance, if a vehicle has a front-view camera, a left front-view camera, a left rear-view camera, a right front-view camera, a right rear-view camera, and a rear-view camera, there are a total of 6 viewpoints; the specific number is not limited. The image captured by each camera serves as the image to be processed for its corresponding viewpoint, or the image captured by the camera can be preprocessed before being used as the image to be processed, depending on actual needs. Preprocessing can be set according to actual needs, such as image enhancement or cropping, to ensure that the image to be processed meets the input requirements of the feature extraction network, so that the feature extraction network can determine the first image features corresponding to each viewpoint.

[0035] In some optional embodiments of this disclosure, the feature extraction network can be any implementable network, such as a backbone network based on ResNet50, U-NET, etc., to extract multi-scale features of the image to be processed. The multi-scale features or features at any scale (e.g., the lowest scale feature) can be used as the first image feature, and the specific configuration can be determined according to actual needs. The feature extraction network is a feature extraction network in a pre-trained model, such as a pre-trained object detection model, semantic segmentation model, etc., and is not specifically limited.

[0036] Step 202: Based on each first image feature and the first bird's-eye view query feature, perform a cross-attention operation based on a local window to obtain the first attention result.

[0037] Among them, the cross-attention operation based on local windows refers to dividing the first bird's-eye view query features and the first image features into local windows, and performing cross-attention operations within the corresponding local windows to realize local information interaction between the first bird's-eye view query features and the first image features.

[0038] In some optional embodiments of this disclosure, the first bird's-eye view query feature can be an initialized bird's-eye view query feature or a bird's-eye view query feature obtained through previous updates. For example, based on the second attention result obtained in step 204 below, a new bird's-eye view query feature can be determined as the first bird's-eye view query feature, and step 202 and subsequent steps can be repeated to iteratively update the bird's-eye view query feature multiple times.

[0039] In some optional embodiments of this disclosure, the bird's-eye view query feature may refer to the query feature under the vehicle coordinate system, which is a coordinate system with the center of the rear axle of the vehicle as the origin, the length direction of the vehicle as the vertical axis (x-axis), and the width direction of the vehicle as the horizontal axis (y-axis).

[0040] In some optional embodiments of this disclosure, the cross-attention operation can be implemented based on a cross-attention network. Examples include single-head cross-attention networks, multi-head cross-attention networks, and so on.

[0041] In some optional embodiments of this disclosure, the first attention result may include the attention result corresponding to each local window of the first bird's-eye view query feature. For example, the first bird's-eye view query feature is divided into M local windows, and the feature in each local window is called the window query feature. The first image feature is also divided into M local windows, and the feature in each local window is called the window image feature. Then, the window query feature and the window image feature have a one-to-one correspondence, so that the window query feature of each local window has the window image feature of the corresponding local window. When performing cross-attention operation, the corresponding window query feature and the window image feature perform cross-attention operation in the local window to obtain the attention result corresponding to each local window, thereby realizing local feature interaction and effectively reducing the computational complexity of cross-attention operation.

[0042] Step 203: Based on the first attention result, determine the second bird's-eye view query features.

[0043] In some optional embodiments of this disclosure, the second bird's-eye view query feature can have the same dimension as the first bird's-eye view query feature, which is equivalent to the query feature updated by the first bird's-eye view query feature through a local window cross-attention operation. Since the first attention result is obtained based on the local window cross-attention operation, the window is segmented and the data structure needs to be transformed during the acquisition process to enable interaction within the local window during the cross-attention operation. Therefore, the data structure of the obtained first attention result needs to undergo window merging and data structure transformation, and can also undergo other processing, such as certain neural network processing, to obtain the second bird's-eye view query feature that is consistent with the first bird's-eye view query feature.

[0044] Step 204: Based on the query features of the second bird's-eye view and the features of each first image, perform cross-attention operation based on local and global windows to obtain the second attention result.

[0045] The cross-attention operation based on local and global windows refers to performing a cross-attention operation between each local window of the second bird's-eye view query feature and the corresponding global window of the first image feature. This enables the interaction of global information between the second bird's-eye view query feature and the first image feature, thereby facilitating a more accurate and effective conversion of multi-view image features to a bird's-eye view perspective. The global window refers to rearranging features from different local windows in the first image feature into the same window using certain pixel rearrangement rules. This ensures that the features within each global window include features from different local windows, thus converging global information into multiple global windows for interaction with the second bird's-eye view query feature, achieving the interaction of global information between the second bird's-eye view query feature and the first image feature.

[0046] Step 205: Based on the second attention result, determine the target bird's-eye view features corresponding to at least one viewpoint.

[0047] In some optional embodiments of this disclosure, the second attention result is similar to the first attention result and may include the attention result corresponding to each local window of the second bird's-eye view query feature. Then, the updated bird's-eye view query feature can be obtained as the target bird's-eye view feature through window merging, data structure transformation, neural network processing, etc.

[0048] In some optional embodiments of this disclosure, a third bird's-eye view query feature can be determined based on the second attention result, and then the third bird's-eye view query feature can be used as the first bird's-eye view query feature. Steps 202-204 and the step of determining the third bird's-eye view query feature based on the second attention result can be repeated. After multiple iterations, the finally obtained third bird's-eye view query feature can be used as the target bird's-eye view feature, thereby further improving the accuracy and effectiveness of the obtained target bird's-eye view feature.

[0049] The method for determining bird's-eye view features provided in this embodiment achieves local interaction between the first bird's-eye view query features and the first image features by performing cross-attention operations based on local windows on the first image features and the first bird's-eye view query features corresponding to each viewpoint. Based on the first attention result obtained from the local interaction, the second bird's-eye view query features are determined. Then, based on the second bird's-eye view query features and each first image feature, cross-attention operations based on local and global windows are performed to achieve global interaction between the second bird's-eye view query features and the first image features. Based on the second attention result obtained from the global interaction, the target bird's-eye view features are obtained. On the one hand, the window segmentation can greatly reduce the computational complexity of determining bird's-eye view features from multi-view image features. On the other hand, through local and global interactions, both local and global features of the first image features are taken into account, thereby effectively improving the accuracy and effectiveness of the determined bird's-eye view features. In addition, this disclosure can obtain accurate and effective bird's-eye view features with at least two cross-attention operations, while the cross-attention based on Transformer in related technologies usually requires six cross-attention operations. It can be seen that this disclosure can further reduce the complexity of the model structure and facilitate deployment and application on the vehicle side.

[0050] Figure 3 This is a flowchart illustrating a method for determining bird's-eye view features provided in another exemplary embodiment of this disclosure.

[0051] In some optional embodiments, step 202, which involves performing a local window-based cross-attention operation based on each first image feature and the first bird's-eye view query feature to obtain a first attention result, may specifically include the following steps:

[0052] Step 2021: Based on the query features of the first bird's-eye view and the segmentation rules of the first local window, determine the query features of the first window corresponding to the first number of local windows.

[0053] In some optional embodiments of this disclosure, the first local window segmentation rule may include a bird's-eye view query feature segmentation rule and an image feature segmentation rule. The bird's-eye view query feature segmentation rule may include window segmentation-related information such as the window size and number of windows corresponding to the bird's-eye view query feature, used to determine the specific segmentation method of the bird's-eye view query feature. The first local window segmentation rule may also include data structure transformation rules, so that the obtained first window query feature can meet the data structure requirements of the cross-attention network for window cross-attention operations. Similarly, the image feature segmentation rule may include window segmentation-related information such as the window size and number of windows corresponding to the image feature, used to determine the specific segmentation method of the image feature, and may include data structure transformation rules, etc., which can be set according to actual needs. The number of local windows segmented by the first bird's-eye view query feature is the same as the number of local windows segmented by the first image feature, so as to facilitate local interaction between corresponding local windows.

[0054] For example, the data structure of the first bird's-eye view query feature is represented as B*C*H*W, where B represents the batch size, C represents the number of feature channels, and H*W represents the feature size (height and width) of each channel. The data structure of the first image feature is represented as B*N*C*h*w, where N represents the number of views, B represents the batch size, C represents the number of feature channels, and h*w represents the feature size (height and width) of the first image feature for each view. When the first image feature is a multi-scale image feature, the image feature at each scale can be represented as B*N*C*hi*wi, where hi*wi represents the image feature at the i-th scale. The data structure after local window segmentation of the first bird's-eye view query features can be represented as B*C*n*H1*W1, where n represents the number of local windows and H1*W1 represents the size of the local window, for example, a 3*3 window. Each H*W feature map of the original B*C*H*W first bird's-eye view query features is segmented into n H1*W1 feature maps. This requires data structure transformation, sequentially changing to B*C*n*S→B*n*S*C→B*n*1*S*C→B*n*N*S*C, to obtain the first window query features. S is the value of H1*W1, which is the length of the vector flattened from the H1*W1 feature map. For example, if H1*W1 is 3*3, then S=9. This means that the feature maps within the local window need to be converted into vectors to facilitate subsequent cross-attention operations. The addition of N dimensions is to ensure that the query features in the first window correspond to the image features from N viewpoints. This can be achieved by copying, for example, copying the intermediate data of B*n*1*S*C into N identical copies, resulting in a B*n*N*S*C result. The final query features in the first window can be represented as B*n NS*C matrices, where NS represents flattening and merging the N and S dimensions into one dimension. This converts each N*S*C three-dimensional tensor into an NS*C two-dimensional matrix, facilitating subsequent cross-attention operations. For example, if N=5 and S=9, the N S*C matrices in each N*S*C three-dimensional tensor can be concatenated along the S dimension to achieve merging, resulting in a 45*C matrix, i.e., a 45-row, C-column matrix.

[0055] In some optional embodiments of this disclosure, the first window query feature may also be a feature that has not undergone data structure transformation, that is, a feature after only local window segmentation, such as the segmented B*C*n*H1*W1 feature mentioned above, which is used as the first window query feature, and the data structure is transformed during the subsequent cross-attention operation.

[0056] Step 2022: For each first image feature, based on the first local window segmentation rule, determine the first window image features that correspond to each first window query feature in the first image feature.

[0057] Specifically, the segmentation method corresponding to the first image feature can be determined based on the first local window segmentation rule, so that the first image feature can be segmented according to the segmentation method to obtain the local window corresponding to the first image feature.

[0058] In some optional embodiments of this disclosure, the features within each local window are transformed by a data structure and used as the first window image features of that local window.

[0059] In some optional embodiments of this disclosure, the features within each local window can be directly used as the features of the first window image. This allows for data structure transformation during subsequent cross-attention operations, which can be set according to actual needs.

[0060] For example, the first image feature is represented as B*N*C*h*w, and the first window image feature obtained by its segmentation and data structure transformation is represented as B*n*N*L*C, where n represents the number of local windows, and L, similar to S above, represents the length of the local window when flattened into a vector. The obtained first window image feature can be represented as B*n NL*C matrices, where NL is similar to NS above and will not be elaborated here. Since the size of the first bird's-eye view query feature and the size of the first image feature can be different, and both have the same number of local windows, L and S can be different. For example, if the H*W of the first bird's-eye view query feature is 4*4, and it is segmented into 4 local windows, the length S of each local window can be 4. If the h*w of the first image feature is 2*4, and it is segmented into 4 local windows, the length of each local window can be 2.

[0061] For example, Figure 4 This is a schematic diagram of partial window segmentation provided in an exemplary embodiment of this disclosure. The first bird's-eye view query feature is a 4*4 feature, and the partial window size is 2*2. By segmenting the partial window, four partial windows are obtained, each represented by a different grayscale.

[0062] In some optional embodiments of this disclosure, the first local window segmentation rule may include the correspondence between local windows of bird's-eye view query features and local windows of image features. Based on this, the first window image features corresponding to each first window query feature in the first image features can be determined.

[0063] In some optional embodiments of this disclosure, the correspondence between the query features of the first window and the image features of the first window can also be determined based on the local window sequence obtained by segmentation. The specific correspondence can be set according to actual needs.

[0064] Step 2023: Based on the query features of each first window and the image features of the first window corresponding to each query feature, perform cross-attention operation to obtain the first attention result.

[0065] In this cross-attention operation, the image features of each first window are used to determine the key and value tensors, while the query features of each first window are used to determine the query tensor. This allows cross-attention operations to be implemented based on the query, key, and value tensors. The key (K), value (V), and query (Q) tensors are used for the cross-attention operation. The key and value tensors can be obtained from the image features of each first window through different linear mappings. For example, the key tensor is obtained through linear mapping weight Wk, and the value tensor is obtained through linear mapping weight Wv. The query tensor is obtained based on the query features of each first window and the linear mapping weight Wq. Wk, Wv, and Wq are all learnable weights. During the cross-attention operation, the attention distribution (i.e., weight tensor) corresponding to the value tensor is determined based on the query and key tensors. The attention distribution then weights the value tensor to obtain the first attention result.

[0066] This embodiment of the disclosure achieves local window segmentation of the first bird's-eye view query features and the first image features through local window segmentation rules, and obtains first window query features and first window image features suitable for local window cross attention, thereby performing cross attention operation within the local window to realize local information interaction between bird's-eye view query features and image features.

[0067] In some optional embodiments of this disclosure, step 2021, which determines the first window query features corresponding to a first number of local windows based on the first bird's-eye view query features and the first local window segmentation rules, includes:

[0068] Based on the first local window segmentation rule, the size of the first local window corresponding to the first bird's-eye view query feature is determined; according to the size of the first local window, the first bird's-eye view query feature is segmented into the first window query features corresponding to the first number of local windows respectively.

[0069] The size of the first local window can be set according to actual needs. For example, it can be represented as H1*W1. The first bird's-eye view query feature is divided into a first number of H1*W1 local windows, and the first window query feature corresponding to the local window is determined according to the features within each local window.

[0070] In some optional embodiments of this disclosure, the first window query feature can be a feature transformed according to the aforementioned data structure transformation, or a feature without data structure transformation. For features without data structure transformation, data structure transformation can be performed before subsequent cross-attention operations to transform them into a data structure suitable for cross-attention operations within a local window. The specific transformation can be set according to actual needs.

[0071] This embodiment helps to accurately and effectively segment the first bird's-eye view query feature by determining the size of the first local window corresponding to the first bird's-eye view query feature.

[0072] In some optional embodiments of this disclosure, step 2022, which determines the first window image features corresponding to each first window query feature based on the first local window segmentation rule, includes:

[0073] Based on the first local window segmentation rule, the size of the second local window corresponding to the first image feature is determined; according to the second local window size, the first image feature is segmented into first window image features corresponding to each first window query feature.

[0074] Specifically, if the size of the first image feature is inconsistent with the size of the first bird's-eye view query feature, then the size of the second local window is inconsistent with the size of the first local window, so that the number of local windows segmented by the first image feature is the same as the number of local windows segmented by the first bird's-eye view feature. The principle of local window segmentation of the first image feature is similar to that of the aforementioned first bird's-eye view query feature, and will not be repeated here.

[0075] This embodiment can achieve accurate and effective local window segmentation of the first image feature by determining the size of the second local window corresponding to the first image feature.

[0076] In some optional embodiments of this disclosure, step 2023, which involves performing a cross-attention operation based on the query features of each first window and the corresponding image features of the first window to obtain a first attention result, includes:

[0077] Based on the query features of each first window, determine the first query tensor; based on the image features of each first window, determine the first key tensor and the first value tensor; based on the first query tensor, the first key tensor, and the first value tensor, perform a cross-attention operation to obtain the first attention result.

[0078] If the first window query feature is a feature suitable for window cross-attention interaction after data structure transformation, then the first window query feature can be mapped through a linear network layer to obtain the first query tensor Q1. If the first window query feature has only undergone window segmentation without data structure transformation, then the aforementioned data structure transformation can be performed on each first window query feature to obtain the transformed window query feature, and then mapped through a linear network layer to obtain the first query tensor Q1. Similarly, the first window image features can be mapped through two linear network layers respectively, or by first transforming the data structure and then mapping through a linear network layer to obtain the first key tensor K1 and the first value tensor V1.

[0079] In some optional embodiments of this disclosure, after determining the first query tensor Q1, the first key tensor K1, and the first value tensor V1, a cross-attention operation can be performed on the first query tensor Q1, the first key tensor K1, and the first value tensor V1 based on a cross-attention network layer to obtain a first attention result. Specifically, a weight tensor can be determined first based on the first query tensor and the first key tensor, and then a weighted sum can be performed on the first value tensor based on the weight tensor to obtain the first attention result.

[0080] For example, taking the single-head cross-attention operation as an example, the query features of each first window after data structure transformation are represented as B*n NS*C matrices, and the first image features after data structure transformation are represented as B*n NL*C matrices. Then, the data structure of the first query tensor Q1 determined based on each first window query feature is also a B*n NS*C matrix, and the data structures of the first key tensor K1 and the first value tensor V1 are B*n NL*C matrices. The weight tensor a determined based on Q1 and K1 is represented as follows:

[0081]

[0082] Where T represents the transpose of the matrix, Softmax() represents normalization, and Q1K1 T This means that B*n NS*C matrices are multiplied by the transposes of B*n NL*C matrices respectively to obtain B*n NS*NL matrices. The final weight tensor a data structure is a weight matrix of B*n NS*NL matrices.

[0083] For example, if B=1, the number of local windows n=4, the number of viewpoints N=5, S=4, L=2, then NS=20, NL=10, and the obtained a includes four 20*10 weight matrices.

[0084] The weight matrix is ​​used to weight the first-value tensor to obtain the first attention result b, as shown below:

[0085] b = aV1

[0086] In this case, the V1 data structure consists of B*n NL*C matrices. The weight matrix of each NS*NL in a is multiplied by the corresponding NL*C matrix in V1 to obtain the data structure of b, which consists of B*n NS*C matrices.

[0087] In some optional embodiments of this disclosure, for multi-head cross-attention operations, a split-head operation can also be performed on the first query tensor Q1, the first key tensor K1, and the first value tensor V1. This split-head operation can be achieved through data structure transformation. Taking the first query tensor Q1 as an example, the data structure transformation process is: B*n*N*S*C→B*n*N*S*d*e→B*d*n*N*S*e→Bd*n*NS*e, where d represents the number of heads and e represents the number of channels in each head. That is, the number of feature channels C is divided into d heads, resulting in a matrix of Bd*n NS*e. The split-head principle for the first key tensor K1 and the first value tensor V1 is similar to that for the first query tensor Q1, and will not be repeated here. Through the split-head operation, multi-head parallel computation can be achieved to improve the computational efficiency of the cross-attention operation. The cross-attention operation for each head is consistent with the single-head cross-attention operation described above, and will not be repeated here.

[0088] In some optional embodiments of this disclosure, after obtaining the first attention result, in order to obtain the second bird's-eye view query features, the data structure of the first attention result can be transformed into a B*C*H*W structure. Furthermore, the second bird's-eye view query features can be obtained through a certain neural network. The specific settings can be configured according to actual needs.

[0089] This embodiment converts each first window query feature into a first query tensor and each first window image feature into a first key tensor and a first value tensor, enabling the interaction between the first window query features and the first window image features within a local window through cross-attention operations, thereby effectively reducing the computational load of cross-attention operations.

[0090] In some optional embodiments of this disclosure, the first attention result includes the attention result corresponding to each query feature of the first window.

[0091] Step 203, based on the first attention result, determines the second bird's-eye view query features, including:

[0092] Step 2031: Merge the attention results corresponding to the query features of each first window into windows to obtain the first merged result.

[0093] Window merging can be achieved through data structure transformations, linear mappings, and other methods.

[0094] In some optional embodiments of this disclosure, taking a single-head cross-attention operation as an example, the first attention result is represented as the aforementioned B*n NS*C matrix, or as B*n*N*S*C. The data structure transformation of the window merging process is: B*n*N*S*C → B*N*n*S*C. Then, it is mapped through a linear network layer to obtain a mapping result with a data structure of B*N*n*S*C. The mean is calculated along the N-dimensional plane, resulting in another data structure of B*n*S*C. The S-dimensional vector is transformed into H1*W1, yielding B*n*H1*W1*C. n*H1*W1 is merged into H*W, resulting in B*H*W*C. Finally, B*H*W*C is transformed into B*C*H*W, thus obtaining the first merging result. The meanings of each symbol are explained in the foregoing examples.

[0095] In practical applications, the specific data structure transformation process is not limited to the process in the example above, as long as the required transformation result can be achieved.

[0096] In some optional embodiments of this disclosure, for multi-head cross-attention operations, the first attention result obtained may include the attention results corresponding to each of the multiple heads, and then the multiple heads can be merged through data structure transformation, followed by window merging. Alternatively, in the multi-head cross-attention network, the transformation of multiple heads can be achieved directly through data structure transformation. The specific transformation can be set according to actual needs, and this disclosure does not limit it.

[0097] Step 2032: Based on the first merging result, the second bird's-eye view query features are determined using the first feedforward neural network.

[0098] The first feedforward neural network can be a multi-layered feedforward neural network, specifically including an input layer, one or more hidden layers, and an output layer. For example, the first feedforward neural network may include two cascaded linear network layers, with a ReLU activation layer between the two linear network layers, which can be configured according to actual needs.

[0099] This embodiment merges windows after local window cross-attention operations, which facilitates subsequent cross-attention operations based on local and global windows. This allows the bird's-eye view query features updated through local interaction to interact with global information of image features for further updates, continuously improving the accuracy and effectiveness of the bird's-eye view query features.

[0100] In some optional embodiments of this disclosure, step 204, which involves performing a cross-attention operation based on local and global windows to obtain a second attention result, based on the second bird's-eye view query features and each of the first image features, includes:

[0101] Step 2041: Based on the second bird's-eye view query features and the second local window segmentation rules, determine the second window query features corresponding to the second number of local windows respectively.

[0102] The second local window segmentation rule can be the same as or different from the first local window segmentation rule, depending on actual needs. However, the segmentation principle is similar to the aforementioned first bird's-eye view query feature, and will not be repeated here.

[0103] Step 2042: For each first image feature, based on the global window segmentation rules, determine the global window image features corresponding to each second window query feature in the first image feature.

[0104] The global window segmentation rules can include the size of the global window, the number of windows, and the method for determining global features within the global window. The method for determining global features can include one or more of the following: pixel rearrangement, mapping relationships between pixel positions in each global window and pixel positions in the first image feature, etc. The specific method can be set according to actual needs, as long as it can achieve the goal of representing features from different local windows within the same global window. Pixel rearrangement can include swapping some rows and columns in the first image feature to achieve pixel rearrangement. After rearrangement, window segmentation is performed, thus representing features from different local windows in the original first image feature within the same global window. The mapping relationship between global window pixel positions and first image feature pixel positions is similar to the pixel rearrangement effect. For example, for each global window, a correspondence can be established between each pixel position and a pixel position in the first image feature. Therefore, when performing global window segmentation, the features of each pixel position in the first image feature can be directly placed into the corresponding pixel position in the corresponding global window based on the correspondence to obtain the global window image features corresponding to each global window.

[0105] For example, Figure 5 This is a schematic diagram of the global window segmentation principle provided in an exemplary embodiment of this disclosure. Taking a 4*4 feature as an example, the global window size is 2*2. Through global window segmentation, four 2*2 global windows are obtained. Different gray levels represent features from different local windows, so that each global window includes features from four local windows, thereby reflecting the global information of the first image feature in each global window.

[0106] Step 2043: Based on the query features of each second window and the global window image features corresponding to each query feature of the second window, perform cross-attention operation to obtain the second attention result.

[0107] In this process, the number of local windows segmented by the second bird's-eye view query feature is the same as the number of global windows segmented by the first image feature, so as to achieve a one-to-one correspondence between local windows and global windows. This enables the second window query feature of each local window of the second bird's-eye view query feature to interact with the corresponding global window image feature in each first image feature during the cross-attention operation.

[0108] This embodiment performs global window segmentation on the first image features to obtain global window image features that interact with the corresponding second window query features. This further enables the interaction of global information between bird's-eye view query features and image features, helping to obtain more accurate and effective bird's-eye view query features.

[0109] In some optional embodiments of this disclosure, step 2041, which determines the second window query features corresponding to a second number of local windows based on the second bird's-eye view query features and the second local window segmentation rules, includes:

[0110] Based on the second local window segmentation rule, the size of the third local window corresponding to the second bird's-eye view query feature is determined; according to the size of the third local window, the second bird's-eye view query feature is segmented into the second window query features corresponding to the second number of local windows respectively.

[0111] The acquisition of query features in the second window is similar to that in the first window, and will not be described in detail here.

[0112] In some optional embodiments of this disclosure, step 2042, which determines the global window image features corresponding to each second window query feature in the first image feature based on global window segmentation rules, includes:

[0113] Based on the global window segmentation rules, the global window size and the pixel recombination method of the global window are determined; according to the global window size and the pixel recombination method, the global window image features corresponding to each second window query feature in the first image feature are determined.

[0114] The pixel reorganization method of the global window (i.e., the pixel rearrangement method mentioned above) can be set according to actual needs, as long as it can reflect the global information of the first image feature in the global window. For example, the first image feature can be divided into local windows first, and then the pixels in each local window can be reorganized, such as by pixel swapping, to obtain the global window image features corresponding to each global window. Alternatively, pixel reorganization can be performed first, and then window segmentation can be performed based on the reorganized image features to obtain the global window image features. See [example for details]. Figure 5 The principle of global window segmentation is shown.

[0115] This embodiment achieves global window segmentation through pixel recombination, thereby displaying the global information of the first image feature in the global window for interaction with the query feature in the second window. This helps to realize the interaction between the global information of the bird's-eye view feature and the image feature, further improving the accuracy and effectiveness of the bird's-eye view query feature.

[0116] In some optional embodiments of this disclosure, determining the global window image features corresponding to each second window query feature in the first image feature according to the global window size and pixel recombination method includes:

[0117] The second number of global windows corresponding to the first image feature are determined based on the global window size; for each global window, the target pixels corresponding to each pixel position in the first image feature are determined based on the pixel recombination method; the global window image feature corresponding to the global window is determined based on the target pixels corresponding to each pixel position in the first image feature; and the global window image feature corresponding to each second window query feature is determined according to the correspondence rules between the second number of local windows and the second number of global windows.

[0118] The pixel recombination method represents the transformation of the original pixel arrangement of the first image feature. For example, swapping the i-th row with the j-th row and swapping the i-th column with the j-th column achieves pixel recombination. The recombined pixels have a one-to-one correspondence with the original pixels of the first image feature. Therefore, based on the pixel recombination method, the target pixels corresponding to each pixel position in the global window in the first image feature can be determined. For example, see [link to relevant documentation]. Figure 5 Pixel position 11 in global window 1 corresponds to pixel a11 in the first image feature, pixel position 12 in global window 1 corresponds to pixel a13 in the first image feature, and so on.

[0119] In some optional embodiments of this disclosure, step 2043, which involves performing a cross-attention operation based on the query features of each second window and the global window image features corresponding to each second window query feature, to obtain the second attention result, includes:

[0120] Based on the query features of each second window, determine the second query tensor; based on the image features of each global window, determine the second key tensor and the second value tensor; based on the second query tensor, the second key tensor, and the second value tensor, perform a cross-attention operation to obtain the second attention result.

[0121] The determination of the second query tensor, the second key tensor, and the second value tensor is similar to that of the first query tensor, the first key tensor, and the first value tensor mentioned above. The principle of cross-attention operation can be found in the aforementioned embodiments, and will not be repeated here.

[0122] This embodiment converts each second window query feature into a second query tensor and each global window image feature into a second key tensor and a second value tensor for cross-attention operation, thereby facilitating the interaction of global information between window query features and image features based on a cross-attention network.

[0123] In some optional embodiments of this disclosure, the second attention result includes the attention result corresponding to each second window query feature.

[0124] Step 205, based on the second attention result, determines at least one target bird's-eye view feature corresponding to a viewpoint, including:

[0125] Step 2051: Merge the attention results corresponding to the query features of each second window to obtain the second merged result.

[0126] The specific operation of this step is similar to that of step 2031 mentioned above, and will not be repeated here.

[0127] Step 2052: Based on the second merging result, the target bird's-eye view features are determined using the second feedforward neural network.

[0128] The second feedforward neural network is similar to the first feedforward neural network mentioned above, and will not be described in detail here.

[0129] In some optional embodiments of this disclosure, a second feedforward neural network can be used to process the second merging result to obtain a third bird's-eye view query feature, and the third bird's-eye view query feature can be used as the target bird's-eye view feature.

[0130] In some optional embodiments of this disclosure, step 2052, based on the second merging result, utilizes a second feedforward neural network to determine the target bird's-eye view features, including:

[0131] Based on the second merging result, a third bird's-eye view query feature is determined using a second feedforward neural network; the third bird's-eye view query feature is used as the first bird's-eye view query feature, and the step of performing cross-attention operation based on local windows based on each first image feature and the first bird's-eye view query feature is repeated to obtain the first attention result; in response to the number of repeated executions reaching a preset threshold, the third bird's-eye view query feature is used as the target bird's-eye view feature.

[0132] The preset threshold number of repetitions can be set according to actual needs. For example, it can be set to 2 times, 3 times, etc. After obtaining the third bird's-eye view query feature, the third bird's-eye view query feature is used as the first bird's-eye view query feature, and then steps 202-204 are repeated to obtain the second attention result. Based on the second attention result, the second feedforward neural network is used to determine the third bird's-eye view query feature. If the current number of repetitions reaches the preset threshold number, the third bird's-eye view query feature can be used as the target bird's-eye view feature. If the preset threshold number is not reached, the aforementioned process can be repeated for the third bird's-eye view query feature until the number of repetitions reaches the preset threshold number, and the target bird's-eye view feature is obtained.

[0133] This embodiment obtains target bird's-eye view features by repeating the process multiple times, which can further improve the accuracy and effectiveness of target bird's-eye view features, thereby helping to further improve the accuracy of subsequent target detection, semantic segmentation and other tasks.

[0134] In some optional embodiments of this disclosure, when the first image feature is a multi-scale image feature (each scale image feature includes image features from multiple perspectives), one or any number of scale image features are used to interactively fuse with the bird's-eye view query feature. For cases using multiple scales, the image features at each scale (data structure B*N*C*hi*wi) can be sequentially used as the first image feature, and the first bird's-eye view query feature can be updated multiple times to obtain the final target bird's-eye view query feature. For example, multi-scale image features include image features at five scales, such as 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 scales of the image to be processed. By using the 1 / 32 and 1 / 64 scale image features to interact and fuse with the bird's-eye view query features, the 1 / 32 scale image features can be used as the first image feature. The third bird's-eye view query feature can be obtained by repeating the aforementioned steps multiple times. The updated third bird's-eye view query feature can then be used as the first bird's-eye view query feature. The 1 / 64 scale image features can be used as the first bird's-eye view feature. The above step 202 can be repeated once or multiple times to obtain the third bird's-eye view query feature. The third bird's-eye view query feature can be used as the target bird's-eye view feature. By continuously updating the bird's-eye view query feature through the local and global interaction of multi-scale image features, the accuracy and effectiveness of the target bird's-eye view feature can be further improved.

[0135] In some optional embodiments of this disclosure, Figure 6This is a schematic diagram of the network architecture of a model for determining bird's-eye view features provided in an exemplary embodiment of this disclosure. The network structure within the dashed box can be called a spatial fusion module, used for the fusion of multi-view image spaces. Through the segmentation of local and global windows, it efficiently fuses multi-view image features into the bird's-eye view, obtaining high-quality bird's-eye view features. This spatial fusion module can be stacked to achieve multiple spatial fusions, thereby obtaining more accurate and effective target bird's-eye view features. Feature fusion module × K represents K stacked feature fusion modules. The specific value can be set according to actual needs. Alternatively, multiple feature fusion modules can perform spatial fusion for image features at multiple scales respectively.

[0136] It should be noted that the bird's-eye view feature determination model disclosed herein needs to be trained in advance. During training, it can be combined with specific image processing tasks, such as object detection tasks, semantic segmentation tasks, etc. By connecting the spatial fusion module with the prediction head network of the corresponding task (such as detection head network, semantic segmentation prediction head network, etc.), supervised training can be performed based on the task labels of the image processing task to obtain the trained spatial fusion module.

[0137] The method for determining bird's-eye view features provided in this disclosure can achieve more accurate and effective bird's-eye view features with less computation by interactively fusing local and global information from bird's-eye view query features with multi-view, multi-scale image features. Through training, the spatial fusion module of this disclosure can effectively determine the target bird's-eye view features with at least two cross-attention operations (i.e., one spatial fusion module). Compared with the existing Transformer (which typically requires 6 cross-attention operations), this can greatly reduce the complexity of the network structure and further reduce the computational complexity, making it more conducive to deployment and application on terminals (e.g., vehicles).

[0138] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0139] The method for determining any of the bird's-eye view features provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, the method for determining any of the bird's-eye view features provided in this disclosure can be executed by a processor, such as by a processor executing the method for determining any of the bird's-eye view features mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0140] Figure 7 This is a schematic flowchart of an exemplary embodiment of the image processing method provided in this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 7 As shown, the method includes the following steps:

[0141] Step 301: Obtain the images to be processed corresponding to each viewpoint in at least one viewpoint.

[0142] Step 302: Determine the bird's-eye view features of the target based on the images to be processed corresponding to each viewpoint;

[0143] Step 303: Determine the image processing result based on the target bird's-eye view features and the prediction head network;

[0144] The target bird's-eye view features are obtained based on the bird's-eye view feature determination method provided in any of the above embodiments. The prediction head network can be configured according to the actual image processing task requirements. For example, the prediction head network may include at least one of the head networks for target detection, semantic segmentation, and target tracking tasks, etc., without specific limitations. Correspondingly, the image processing results may include at least one of the image processing results such as target detection results, semantic segmentation results, and target tracking results.

[0145] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0146] Exemplary device

[0147] Figure 8 This is a schematic diagram of a device for determining bird's-eye view features provided in an exemplary embodiment of this disclosure. The device in this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 8 The device shown includes: a first processing module 501, a second processing module 502, a third processing module 503, a fourth processing module 504, and a fifth processing module 505.

[0148] The first processing module 501 is used to determine the first image features corresponding to each viewpoint based on the images to be processed corresponding to each viewpoint in at least one viewpoint.

[0149] The second processing module 502 is used to perform cross-attention operation based on local windows based on each first image feature and the first bird's-eye view query feature to obtain the first attention result.

[0150] The third processing module 503 is used to determine the second bird's-eye view query features based on the first attention result.

[0151] The fourth processing module 504 is used to perform cross-attention operations based on local and global windows based on the query features of the second bird's-eye view and the features of each first image to obtain the second attention result.

[0152] The fifth processing module 505 is used to determine the target bird's-eye view features corresponding to at least one viewpoint based on the second attention result.

[0153] Figure 9 This is a schematic diagram of the structure of a device for determining bird's-eye view features provided in another exemplary embodiment of this disclosure.

[0154] In some optional embodiments of this disclosure, the second processing module 502 includes:

[0155] The first determining unit 5021 is used to determine the first window query features corresponding to a first number of local windows based on the first bird's-eye view query features and the first local window segmentation rules.

[0156] The second determining unit 5022 is used to determine, for each first image feature, the first window image feature corresponding to each first window query feature based on the first local window segmentation rule.

[0157] The first processing unit 5023 is used to perform cross-attention operation based on the query features of each first window and the image features of the first window corresponding to each query feature of the first window, to obtain the first attention result.

[0158] In some optional embodiments of this disclosure, the first determining unit 5021 is specifically used for:

[0159] Based on the first local window segmentation rule, the size of the first local window corresponding to the first bird's-eye view query feature is determined; according to the size of the first local window, the first bird's-eye view query feature is segmented into the first window query features corresponding to the first number of local windows respectively.

[0160] In some optional embodiments of this disclosure, the second determining unit 5022 is specifically used for:

[0161] Based on the first local window segmentation rule, the size of the second local window corresponding to the first image feature is determined; according to the second local window size, the first image feature is segmented into first window image features corresponding to each first window query feature.

[0162] In some optional embodiments of this disclosure, the first processing unit 5023 is specifically used for:

[0163] Based on the query features of each first window, determine the first query tensor; based on the image features of each first window, determine the first key tensor and the first value tensor; based on the first query tensor, the first key tensor, and the first value tensor, perform a cross-attention operation to obtain the first attention result.

[0164] In some optional embodiments of this disclosure, the first attention result includes the attention result corresponding to each query feature of the first window.

[0165] The third processing module 503 includes:

[0166] The first window merging unit 5031 is used to merge the attention results corresponding to the query features of each first window to obtain the first merging result.

[0167] The second processing unit 5032 is used to determine the second bird's-eye view query features based on the first merging result and using the first feedforward neural network.

[0168] In some optional embodiments of this disclosure, the fourth processing module 504 includes:

[0169] The third determining unit 5041 is used to determine the second window query features corresponding to the second number of local windows based on the second bird's-eye view query features and the second local window segmentation rules.

[0170] The fourth determining unit 5042 is used to determine, for each first image feature, the global window image features corresponding to each second window query feature in the first image feature based on the global window segmentation rules.

[0171] The third processing unit 5043 is used to perform cross-attention operation based on the query features of each second window and the global window image features corresponding to each query feature of the second window, to obtain the second attention result.

[0172] In some optional embodiments of this disclosure, the third determining unit 5041 is specifically used for:

[0173] Based on the second local window segmentation rule, the size of the third local window corresponding to the second bird's-eye view query feature is determined; according to the size of the third local window, the second bird's-eye view query feature is segmented into the second window query features corresponding to the second number of local windows respectively.

[0174] In some optional embodiments of this disclosure, the fourth determining unit 5042 is specifically used for:

[0175] Based on the global window segmentation rules, the global window size and the pixel recombination method of the global window are determined; according to the global window size and the pixel recombination method, the global window image features corresponding to each second window query feature in the first image feature are determined.

[0176] In some optional embodiments of this disclosure, the fourth determining unit 5042 is specifically used for:

[0177] The second number of global windows corresponding to the first image feature are determined based on the global window size; for each global window, the target pixels corresponding to each pixel position in the first image feature are determined based on the pixel recombination method; the global window image feature corresponding to the global window is determined based on the target pixels corresponding to each pixel position in the first image feature; and the global window image feature corresponding to each second window query feature is determined according to the correspondence rules between the second number of local windows and the second number of global windows.

[0178] In some optional embodiments of this disclosure, the third processing unit 5043 is specifically used for:

[0179] Based on the query features of each second window, determine the second query tensor; based on the image features of each global window, determine the second key tensor and the second value tensor; based on the second query tensor, the second key tensor, and the second value tensor, perform a cross-attention operation to obtain the second attention result.

[0180] In some optional embodiments of this disclosure, the second attention result includes the attention result corresponding to each second window query feature.

[0181] The fifth processing module 505 includes:

[0182] The second window merging unit 5051 is used to merge the attention results corresponding to the query features of each second window to obtain the second merging result.

[0183] The fourth processing unit 5052 is used to determine the target bird's-eye view features based on the second merging result and using the second feedforward neural network.

[0184] In some optional embodiments of this disclosure, the fourth processing unit 5052 is specifically used for:

[0185] Based on the second merging result, a third bird's-eye view query feature is determined using a second feedforward neural network; the third bird's-eye view query feature is used as the first bird's-eye view query feature, and the step of performing cross-attention operation based on local windows based on each first image feature and the first bird's-eye view query feature is repeated to obtain the first attention result; in response to the number of repeated executions reaching a preset threshold, the third bird's-eye view query feature is used as the target bird's-eye view feature.

[0186] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement the corresponding image processing method embodiments of this disclosure. Figure 10 The apparatus shown includes: an image acquisition module 410, a bird's-eye view feature determination module 420, and a prediction module 430.

[0187] The image acquisition module 410 is used to acquire the images to be processed corresponding to each viewpoint in at least one viewpoint.

[0188] The bird's-eye view feature determination module 420 is used to determine the target bird's-eye view features based on the images to be processed corresponding to each viewpoint.

[0189] Prediction module 430 is used to determine the image processing result based on the target bird's-eye view features and the prediction head network;

[0190] The target bird's-eye view features are obtained based on the bird's-eye view feature determination method provided in any of the above embodiments.

[0191] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0192] Exemplary electronic devices

[0193] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 11 and a memory 12.

[0194] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0195] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of this disclosure described above.

[0196] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0197] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0198] The output device 14 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0199] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0200] Exemplary computer program products and computer-readable storage media

[0201] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0202] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0203] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0204] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0205] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0206] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for determining features in a bird's-eye view, comprising: Based on the images to be processed corresponding to each of the at least one viewpoint, determine the first image features corresponding to each of the at least one viewpoint; Based on each of the first image features and the first bird's-eye view query features, a cross-attention operation based on local windows is performed to obtain a first attention result; the cross-attention operation based on local windows refers to dividing the first bird's-eye view query features and the first image features into local windows and performing a cross-attention operation within the corresponding local windows. Based on the first attention result, determine the second bird's-eye view query features; Based on the second bird's-eye view query features and each of the first image features, a cross-attention operation based on local and global windows is performed to obtain a second attention result; wherein, the cross-attention operation based on local and global windows refers to performing a cross-attention operation on each local window of the second bird's-eye view query features and the corresponding global window of the first image features; the global window is obtained by rearranging the features of different local windows in the first image features into the same window through pixel rearrangement rules; Based on the second attention result, the target bird's-eye view features corresponding to the at least one viewpoint are determined.

2. The method according to claim 1, wherein, The step of performing a cross-attention operation based on local windows based on each of the first image features and the first bird's-eye view query features to obtain the first attention result includes: Based on the first bird's-eye view query features and the first local window segmentation rules, determine the first window query features corresponding to a first number of local windows respectively; For each of the first image features, based on the first local window segmentation rule, determine the first window image features that correspond to each of the first window query features in the first image feature; Based on the query features of each first window and the image features of the first window corresponding to each query feature, a cross-attention operation is performed to obtain the first attention result.

3. The method according to claim 2, wherein, The step of performing a cross-attention operation based on the query features of each first window and the corresponding image features of the first window to obtain the first attention result includes: Based on the query features of each of the first windows, determine the first query tensor; Based on the features of each of the first window images, determine the first key tensor and the first value tensor; Based on the first query tensor, the first key tensor, and the first value tensor, a cross-attention operation is performed to obtain the first attention result.

4. The method according to claim 2, wherein, The step of determining the first window query features corresponding to a first number of local windows based on the first bird's-eye view query features and the first local window segmentation rules includes: Based on the first local window segmentation rule, the size of the first local window corresponding to the first bird's-eye view query feature is determined; Based on the size of the first local window, the first bird's-eye view query feature is segmented into the first window query features corresponding to the first number of local windows; and / or, For each of the first image features, based on the first local window segmentation rule, the step of determining the first window image features corresponding to each of the first window query features includes: Based on the first local window segmentation rule, the size of the second local window corresponding to the first image feature is determined; Based on the second local window size, the first image feature is segmented into first window image features corresponding to each first window query feature.

5. The method according to claim 2, wherein, The first attention result includes the attention result corresponding to each query feature of the first window; The step of determining the second bird's-eye view query features based on the first attention result includes: The attention results corresponding to the query features of each first window are merged into windows to obtain the first merged result; Based on the first merging result, the second bird's-eye view query features are determined using a first feedforward neural network.

6. The method according to claim 1, wherein, The step of performing a cross-attention operation based on local and global windows to obtain a second attention result, based on the second bird's-eye view query features and each of the first image features, includes: Based on the second bird's-eye view query features and the second local window segmentation rules, determine the second window query features corresponding to the second number of local windows respectively; For each of the first image features, based on the global window segmentation rules, determine the global window image features that correspond to each of the second window query features in the first image feature; Based on the query features of each second window and the global window image features corresponding to each query feature of the second window, a cross-attention operation is performed to obtain the second attention result.

7. The method according to claim 6, wherein, The step of performing a cross-attention operation based on the query features of each second window and the global window image features corresponding to each query feature of the second window to obtain the second attention result includes: Based on the query features of each second window, determine the second query tensor; Based on the global window image features, determine the second key tensor and the second value tensor; Based on the second query tensor, the second key tensor, and the second value tensor, a cross-attention operation is performed to obtain the second attention result.

8. The method according to claim 6, wherein, The step of determining the second window query features corresponding to a second number of local windows based on the second bird's-eye view query features and the second local window segmentation rules includes: Based on the second local window segmentation rule, determine the size of the third local window corresponding to the second bird's-eye view query feature; Based on the third local window size, the second bird's-eye view query feature is segmented into the second window query features corresponding to the second number of local windows; and / or, The step of determining the global window image features corresponding to each of the second window query features in the first image feature based on the global window segmentation rule includes: Based on the global window segmentation rules, the global window size and the pixel recombination method of the global window are determined; Based on the global window size and the pixel recombination method, determine the global window image features in the first image features that correspond to each of the second window query features.

9. The method according to claim 8, wherein, The step of determining the global window image features corresponding to each of the second window query features in the first image features based on the global window size and the pixel recombination method includes: The second number of global windows corresponding to the first image feature is determined based on the global window size; For each global window, based on the pixel recombination method, the target pixels corresponding to the positions of each pixel in the global window in the first image feature are determined respectively; Based on the target pixels corresponding to the pixel positions in the global window in the first image feature, the global window image feature corresponding to the global window is determined. Based on the correspondence rules between the second number of local windows and the second number of global windows, the global window image features corresponding to the query features of each of the second windows are determined.

10. The method according to claim 6, wherein, The second attention result includes the attention result corresponding to each query feature of the second window; The step of determining the target bird's-eye view features corresponding to the at least one viewpoint based on the second attention result includes: The attention results corresponding to the query features of each second window are merged to obtain a second merged result. Based on the second merging result, the target bird's-eye view features are determined using a second feedforward neural network.

11. The method according to claim 10, wherein, The step of determining the target bird's-eye view features based on the second merging result using a second feedforward neural network includes: Based on the second merging result, the third bird's-eye view query features are determined using the second feedforward neural network; The third bird's-eye view query feature is used as the first bird's-eye view query feature. The step of performing cross-attention operation based on local windows based on each of the first image features and the first bird's-eye view query features to obtain the first attention result is repeated. In response to the number of repeated executions reaching a preset threshold, the third bird's-eye view query feature is used as the target bird's-eye view feature.

12. An image processing method, comprising: Obtain the images to be processed corresponding to each of the at least one viewpoint; Based on the images to be processed corresponding to each of the aforementioned viewpoints, the bird's-eye view features of the target are determined. Based on the target bird's-eye view features and the prediction head network, the image processing result is determined; The target bird's-eye view features are obtained based on the method for determining bird's-eye view features according to any one of claims 1-11.

13. A device for determining features in a bird's-eye view, comprising: The first processing module is used to determine the first image features corresponding to each of the at least one viewpoints based on the images to be processed corresponding to each of the at least one viewpoints. The second processing module is used to perform a cross-attention operation based on local windows based on each of the first image features and the first bird's-eye view query features to obtain a first attention result; the cross-attention operation based on local windows refers to dividing the first bird's-eye view query features and the first image features into local windows and performing a cross-attention operation within the corresponding local windows. The third processing module is used to determine the second bird's-eye view query features based on the first attention result; The fourth processing module is used to perform a cross-attention operation based on local and global windows based on the second bird's-eye view query features and each of the first image features to obtain a second attention result; wherein, the cross-attention operation based on local and global windows refers to performing a cross-attention operation on each local window of the second bird's-eye view query features and the corresponding global window of the first image features; the global window is obtained by rearranging the features of different local windows in the first image features into the same window through pixel rearrangement rules; The fifth processing module is used to determine the target bird's-eye view features corresponding to the at least one viewpoint based on the second attention result.

14. An image processing apparatus, comprising: The image acquisition module is used to acquire the images to be processed corresponding to each of the at least one viewpoint; The bird's-eye view feature determination module is used to determine the target bird's-eye view features based on the images to be processed corresponding to each of the aforementioned viewpoints. The prediction module is used to determine the image processing result based on the target bird's-eye view features and the prediction head network; The target bird's-eye view features are obtained based on the method for determining bird's-eye view features according to any one of claims 1-11.

15. A computer-readable storage medium storing a computer program for performing at least one of the bird's-eye view feature determination method according to any one of claims 1-11 and the image processing method according to claim 12.

16. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement at least one of the bird's-eye view feature determination method according to any one of claims 1-11 and the image processing method according to claim 12.

Images