Long-time fusion method and device for multi-view three-dimensional target detection, board card and storage medium

Abstract

The invention discloses a long-time fusion method and device for multi-view three-dimensional target detection, a board card and a storage medium. The apparatus may be included in a computing processing device of a combined processing device, which may include one or more data processing devices. The combined processing device can also comprise an interface device and other processing devices. And the computing processing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the equipment and the other processing devices and used for storing data of the equipment and the other processing devices. By utilizing the scheme of the invention, the long-term information can be fully and accurately utilized, the long-term fusion capability of the cyclic fusion paradigm is improved, and effective delay and memory consumption are kept at the same time.

Description

Technical Field

[0001] This application generally relates to the field of 3D target detection technology. More specifically, this application relates to a long-term fusion method and device for multi-view 3D target detection, as well as a board and computer-readable storage medium. Background Technology

[0002] In camera-based bird's-eye view ("BEV") 3D object detection, long-term fusion methods are commonly used to improve the detection of occluded targets. Existing long-term fusion methods can be broadly categorized into parallel fusion and cyclic fusion. Parallel fusion, in particular, learns the visual features of the current frame while simultaneously caching and updating features from multiple historical frames. That is, in the parallel fusion paradigm, all historical frames contribute equally to the gradient of the fusion module. This causes computational and storage overhead to increase with the length of historical frame features. Since superior performance requires longer historical information, the parallel fusion paradigm faces a dilemma between effectiveness and efficiency in deployment.

[0003] In contrast, recurrent fusion encodes all historical features as a single recurrent feature for caching and updating, thus significantly reducing inference latency and memory consumption. However, during the training of a bird's-eye view detector for 3D object detection, only the most recent frames contribute to the gradient of the fusion module in the recurrent fusion paradigm. This hinders the temporal fusion module in the recurrent fusion paradigm from learning fusion patterns from long-term temporal information from all historical frames. Furthermore, in scenarios such as autonomous driving, due to the coarse-grained nature of the BEV (Baby Electric Vehicle) mesh, the recurrent fusion paradigm aligns features only within a single BEV mesh during multiple historical feature alignments, rather than aligning the target as a whole. This may result in different targets being fused on a single BEV mesh, confusing the identification of the same target, leading to semantic ambiguity and ultimately inaccurate target detection results.

[0004] In view of this, there is an urgent need to provide a long-term fusion scheme for multi-view 3D target detection, so as to make full and accurate use of long-term time information, improve the long-term fusion capability of the cyclic fusion paradigm, and at the same time maintain effective latency and memory consumption. Summary of the Invention

[0005] In order to at least address one or more of the technical problems mentioned above, this application proposes a long-term fusion scheme for multi-view 3D target detection in several aspects.

[0006] In a first aspect, this application provides a long-term fusion method for multi-view 3D target detection, comprising: acquiring multi-view 2D images of a target scene under a data frame and obtaining a self-motion matrix between adjacent frames; based on the multi-view 2D images under the data frame and the self-motion matrix, using a bird's-eye view detector to extract first bird's-eye view features of all historical frames and second bird's-eye view features of the current frame; performing initial long-term fusion of the first bird's-eye view features and the second bird's-eye view features to obtain an initial fusion result, and backpropagating gradients to all historical frames based on the 3D target detection results of the initial fusion result to update the initial weight coefficients of the bird's-eye view detector; performing feature alignment of the first bird's-eye view features and the second bird's-eye view features based on the updated weight coefficients and the self-motion matrix; and performing final long-term fusion based on the aligned first bird's-eye view features and the second bird's-eye view features to obtain a final fusion result.

[0007] In a second aspect, this application provides a method for multi-view 3D target detection, comprising: inputting the final fusion result obtained from one or more embodiments of the first aspect to the bird's-eye view detector, and sequentially performing feature decoding via the feature decoding unit of the bird's-eye view detector and 3D target detection via the task head unit to obtain the final 3D target detection result.

[0008] In a third aspect, this application provides a long-term fusion device for multi-view 3D target detection, comprising: a processor; and a memory storing program instructions for long-term fusion of multi-view 3D target detection, wherein when the program instructions are executed by the processor, the long-term fusion device implements one or more embodiments of the aforementioned first aspect.

[0009] In a fourth aspect, this application provides an apparatus for multi-view 3D target detection, comprising: a processor; and a memory storing program instructions for multi-view 3D target detection, wherein when the program instructions are executed by the processor, the apparatus implements the embodiments in the second aspect described above.

[0010] In a fifth aspect, this application provides a board, including: the long-term fusion device described in the third aspect; or the device described in the fourth aspect.

[0011] In a sixth aspect, this application provides a computer-readable storage medium storing computer-readable instructions for long-term fusion of multi-view 3D target detection, which, when executed by one or more processors, implement one or more embodiments of the first aspect described above; or storing computer-readable instructions for multi-view 3D target detection, which, when executed by one or more processors, implement embodiments of the second aspect described above.

[0012] The long-term fusion scheme for multi-view 3D target detection provided above, in this embodiment, is to perform initial fusion using the first bird's-eye view features of all historical frames extracted from the multi-view 2D images under the data frame and the self-motion matrix between adjacent frames, and the second bird's-eye view features of the current frame. The gradient is then backpropagated to all historical frames based on the 3D target detection result of the initial fusion. This ensures that all historical frames contribute to the gradient. Furthermore, this embodiment aligns the first and second bird's-eye view features using updated weight coefficients and the self-motion matrix, and then performs final long-term fusion based on the aligned first and second bird's-eye view features. That is, this embodiment considers the target's self-motion matrix to learn fine-grained semantic changes from it, reducing semantic differences during inter-frame alignment. Based on this, long-term temporal information can be fully and accurately utilized, improving the long-term fusion capability of the cyclic fusion paradigm while maintaining effective latency and memory consumption. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0014] Figure 1 This is an exemplary schematic diagram illustrating the comparison of detection accuracy under different time fusion paradigms;

[0015] Figure 2 This is an exemplary schematic diagram illustrating the alignment of historical features in an autonomous driving scenario;

[0016] Figure 3 This diagram shows a structural schematic of a board 300 according to an embodiment of this application;

[0017] Figure 4 This is a structural diagram showing the combined processing device 400 in chip 301 of this embodiment;

[0018] Figure 5 A schematic diagram of the internal structure of the computing device 401 is shown;

[0019] Figure 6 A schematic diagram of the internal structure of processor core 506 is shown;

[0020] Figure 7 This diagram illustrates a scenario where one processor core wants to write data to a processor core in another cluster.

[0021] Figure 8 This illustration shows a design diagram of the hardware and software architecture in one embodiment of this application;

[0022] Figure 9 This is an exemplary flowchart illustrating a long-term fusion method 900 for multi-view 3D target detection according to an embodiment of this application;

[0023] Figure 10 This is an exemplary schematic diagram illustrating a long-term fusion overall method for multi-view 3D target detection according to an embodiment of this application;

[0024] Figure 11 This is an exemplary schematic diagram illustrating the correlation between video visibility and mAP according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0029] 3D object detection technology using only cameras poses a significant challenge for autonomous driving, but has gained widespread attention in recent years due to its low-cost deployment. Previous research in 3D object detection has primarily focused on transforming 2D features into a unified 3D space. Depending on whether explicit 3D representation is used, these transformation methods can be categorized into query-based and bird's-eye view-based (“BEV”) approaches. Query-based methods typically define multiple object queries in 3D space to collect features from multi-view image features. For example, DETR3D queries the image coordinates of the target to obtain sample features, allowing for direct prediction of 3D bounding boxes based on these features. In some implementations, historical information is enhanced using technologies such as DETR4D and Sparse4D. Positional information of 3D coordinates is encoded into image features using technologies such as PETR to facilitate the detection of 3D objects similar to DETR.

[0030] Currently, methods that learn temporal information based on bird's-eye view ("BEV") representations have shown significant improvements. Specifically, typical BEVDet multi-view 3D object detection involves converting 2D features into BEV features based on the Lift-Splat-Shoot paradigm. BEVDepth significantly improves the conversion process through explicit depth supervision, and BEVFormer further introduces a novel attention mechanism to collect 2D features from the BEV mesh. Furthermore, BEVFormer proposes a temporal self-attention mechanism, extending BEVDet to a temporal version by fusing historical BEV features into a cyclic paradigm and integrating previous BEV features through a concatenated operation based on BEVDet4D (i.e., a parallel paradigm). For example, the aforementioned method is extended by increasing the number of previous frames to 16 frames using SOLOFusion, resulting in significant advantages. BEVFormerv2 can switch to the same parallel paradigm as SOLOFusion.

[0031] As described in the background section above, in bird's-eye view ("BEV") 3D object detection, long-term fusion methods can include parallel fusion and cyclic fusion. Parallel fusion faces a dilemma between effectiveness and efficiency in deployment, while cyclic fusion can significantly reduce inference latency and memory consumption.

[0032] For example, in a demonstrative scenario, when using 16 historical frames, the cyclic fusion paradigm offers a 6x speedup and a 12x reduction in memory usage compared to the parallel fusion paradigm. However, in cyclic fusion, only the most recent frames contribute to the gradient of the fusion module, thus failing to fully utilize long-term temporal information and parallel fusion. Consequently, its accuracy stops improving within a certain number of historical frames. Figure 1 As shown.

[0033] Figure 1 This is an exemplary schematic diagram illustrating the comparison of detection accuracy under different time fusion paradigms. For example... Figure 1 Figure (a) shows the mAP of three temporal fusion methods under a certain number of historical frames. The horizontal axis represents the number of historical frames, and the vertical axis represents ΔmAP. The three curves in the figure, from top to bottom, correspond to the long-term fusion method of the embodiment of this application (Ours), the parallel fusion method based on SOLOFusion, and the cyclic fusion method based on VideoBEV. It can be understood that mAP is used to measure the performance of the object detection algorithm. It is obtained by comprehensively weighting the average accuracy of detection of all categories, which can comprehensively consider the performance of the model on different categories, thereby more comprehensively evaluating the accuracy and recall of the object detection model.

[0034] Figure 1 Figure (b) shows the mAP of various temporal fusion methods in frames per second (FPS), a performance comparison based on the nuScenes validation set. It's important to understand that all the aforementioned methods use a ResNet50 pre-trained on ImageNet as the backbone, with an input size of 256×704. Furthermore, the frames per second were measured on a single RTX3090 with PyTorch fp32.

[0035] As shown in the figure, compared with the parallel fusion method based on SOLOFusion, the cyclic fusion method based on VideoBEV benefits less from using historical frames. This phenomenon becomes more pronounced when more than four historical frames are used (corresponding to historical frame 7.5).

[0036] As can be seen from the above, this is due to the following two shortcomings of the cyclic fusion paradigm: (1) Gradient contribution. In the cyclic fusion paradigm, only the most recent frame contributes to the gradient of the fusion module, making it impossible for the temporal fusion module in the cyclic fusion paradigm to learn the long-term temporal information of all historical frames. (2) Semantic ambiguity. When aligning historical features multiple times, the cyclic fusion paradigm only aligns features in a single BEV grid. This may result in different targets being fused on a single BEV grid, thus confusing the identification of the same target and leading to semantic ambiguity for the same target.

[0037] Figure 2 This is an exemplary schematic diagram illustrating the alignment of historical features in an autonomous driving scenario. For example... Figure 2Figure (a) illustrates the alignment of target vehicles within a coarse-grained grid for pure electric vehicles. For example, when the ego vehicle shown in the upper left grid turns right, its corresponding BEV feature changes from Bt-1 to Bt, as shown within the rectangular dashed box in the lower left grid. However, when aligning Bt-1 with Bt, alignment is only performed on features within each grid. For example, as shown within the rectangular dashed box in the upper right grid, the front, body, and rear of the target vehicle are aligned separately. This alignment method only considers the alignment position and ignores the direction, which can lead to inaccurate conversion of direction information within the grid. This may result in different targets being merged on a single BEV grid, leading to semantic ambiguity for the same target, for example... Figure 2 As shown in Figure (b).

[0038] Figure 2 Figure (b) illustrates the semantic ambiguity. Taking the target vehicle corresponding to rectangles A and B as an example, rectangle A represents the BEV feature changes of Bt-1, which have been converted to be aligned with Bt and decoded into bounding boxes. Rectangle B represents the actual bounding box of Bt. It is important to understand that for static targets, the orientations of rectangles A and B do not match. As shown in the figure, due to the coarse granularity of the BEV mesh, the existing cyclic fusion method during temporal alignment suffers from semantic ambiguity, which can lead to incorrect target transformation orientations in actual 3D target detection scenarios.

[0039] It is understandable that aligning historical frames with the current frame in 3D object detection is crucial for temporal fusion based on BEV (Browser Emissions). As an example, suppose the BEV feature of a historical frame is denoted as B... j The alignment process can then be represented by the following formula:

[0040]

[0041] in, P represents the BEV feature of the j-th frame in the ego coordinates of the i-th frame. j，i Let represent the self-motion matrix between frame j and frame i, and ζ represent the self-motion transformation function. Based on this formula (1), the parallel fused representation of frame i can be obtained:

[0042]

[0043] in, This represents the fused BEV features of the i-th frame. Let W represent the temporal fusion function, and W represent the weight parameters. Similarly, the cyclic fusion paradigm for the i-th frame can be expressed as:

[0044]

[0045] Based on the above formula (3), the gradient of the parameters in the time fusion function during parallel fusion can be calculated:

[0046]

[0047] In cyclic fusion, by using B i-1 Separation allows the gradient of W in the cyclic fusion process to be written as:

[0048]

[0049] As can also be seen from the above formulas (4) and (5), in the cyclic fusion paradigm, only the current BEV feature B is considered. i A historical BEV feature This directly affects the gradient of W, while in the parallel fusion paradigm, all historical BEV features with a window size of k contribute to the gradient of W.

[0050] Furthermore, in the parallel fusion paradigm, a series of 2D view features are directly projected into the current 3D BEV feature space, avoiding temporal alignment in the 3D space. For example, suppose the historical view 2D view features at time step j are denoted as C. j The formula for projecting it onto the BEV feature at time step i is:

[0051]

[0052] Among them, I j and L j Let represent the camera's intrinsic and extrinsic parameters, respectively. This additional two-dimensional spatial view transformation can effectively align features between frames, thereby improving the performance of parallel fusion. However, since two-dimensional features cannot be obtained during recursive fusion, this method is not feasible for recursive fusion. This difference is another factor contributing to the performance gap between cyclic fusion and parallel fusion methods.

[0053] Based on this, embodiments of this application provide a long-term fusion method for multi-view 3D object detection, which enables all historical frames to contribute gradients by backpropagating gradients to all historical frames. Specifically, the backpropagation gradient of embodiments of this application can be expressed as the following formula:

[0054]

[0055] Here, k represents the total number of historical frames used in the training iterations. Based on this, each historical BEV feature can contribute equally to the gradient of the temporal fusion module, similar to the parallel paradigm.

[0056] As mentioned earlier, calculating the backpropagation of the entire network along the time dimension places significant demands on GPU memory, as shown in Table 1.

[0057] Table 1 Memory usage during training

[0058]

[0059] OOM indicates insufficient memory on an A100-40G with FP32 precision. As shown in Setting 1 of Table 1, a memory overflow problem can occur even with only 2 frames if all modules, including the image encoder, perform backpropagation along the time dimension. However, long-term frames must be incorporated into the training process to effectively learn temporal information. Therefore, a balance needs to be struck between GPU memory usage (determined by the modules involved in backpropagation) and the length of the time series.

[0060] Therefore, this embodiment of the application also learns fine-grained semantic changes from the self-motion matrix by considering the self-motion transformation of the target. This allows for the full and accurate utilization of long-term temporal information, reducing semantic differences during inter-frame alignment and thus improving the long-term fusion capability of the cyclic fusion paradigm while maintaining effective latency and memory consumption. For example, as shown in Setting 2 in Table 1, this embodiment of the application can be trained on a single A100-40G GPU with FP32 precision and a batch size of 8, effectively utilizing up to 16 consecutive frames.

[0061] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0062] Figure 3 This diagram illustrates the structure of a board 300 according to an embodiment of this application. Figure 3 As shown, board 300 includes a chip (or "processing chip") 301, which is a system-on-chip (SoC) integrating one or more combined processing devices. These combined processing devices are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 300 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0063] Chip 301 is connected to external device 303 via external interface device 302. External device 303 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 303 to chip 301 via external interface device 302. The calculation results from chip 301 can be transmitted back to external device 303 via external interface device 302. Depending on the application scenario, external interface device 302 may have different interface forms, such as a PCIe interface.

[0064] The board 300 also includes a storage device 304 for storing data, which includes one or more memory cells 305. The storage device 304 is connected to and transmits data with the controller 306 and the chip 301 via a bus. The controller 306 in the board 300 is configured to regulate the state of the chip 301. Therefore, in one application scenario, the controller 306 may include a microcontroller (MCU).

[0065] Figure 4 This is a structural diagram illustrating the combined processing device 400 in chip 301 of this embodiment. (As shown) Figure 4 As shown, the combined processing device 400 includes a computing device 401, an interface device 402, a processing device 403, and a DRAM 404.

[0066] The computing device 401 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 403 through the interface device 402 to jointly complete the user-specified operations.

[0067] Interface device 402 is used to transmit data and control commands between computing device 401 and processing device 403. For example, computing device 401 can obtain input data from processing device 403 via interface device 402 and write it to on-chip storage device of computing device 401. Further, computing device 401 can obtain control commands from processing device 403 via interface device 402 and write them to on-chip control cache of computing device 401. Alternatively or optionally, interface device 402 can also read data from storage device of computing device 401 and transmit it to processing device 403.

[0068] The processing device 403, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 401. Depending on the implementation, the processing device 403 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 401 of this application can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 401 and the processing device 403 are considered together, they are considered to form a heterogeneous multi-core structure.

[0069] DRAM 404 is used to store data to be processed. It is DDR memory, typically 16G or larger, and is used to store data in computing device 401 and / or processing device 403.

[0070] Figure 5 A schematic diagram of the internal structure of computing device 401 is shown. Computing device 401 is used to process input data such as computer vision, speech, natural language processing, and data mining. The computing device 401 in the diagram adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 401 includes multiple clusters, and each cluster includes multiple processor cores, which can be used to execute the tasks specified in this application. In other words, computing device 401 is constructed in a hierarchical structure of system-on-a-chip, clusters, and processor cores.

[0071] From the perspective of system-on-a-chip hierarchy, such as Figure 5 As shown, the computing device 401 includes an external storage controller 501, a peripheral communication module 502, an on-chip interconnect module 503, a synchronization module 504, and multiple clusters 505.

[0072] There can be multiple external storage controllers 501; two are shown as an example in the figure. These controllers are used to access external storage devices, such as those issued by the processor core, in response to access requests from the processor core. Figure 4The DRAM 404 in the chip allows the computing device 401 to read data from external sources or write data to external sources. The peripheral communication module 502 receives control signals from the processing device 403 via the interface device 402, initiating the computing device 401 to execute tasks. The on-chip interconnect module 503 connects the external storage controller 501, the peripheral communication module 502, and multiple clusters 505, enabling data and control signal transmission between modules. The synchronization module 504 is a Global Barrier Controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 505 are the computing core of the computing device 401. Four are shown exemplary in the figure; however, with hardware development, the computing device 401 of this application may also include 8, 16, 64, or even more clusters 505.

[0073] From the perspective of cluster hierarchy, such as Figure 5 As shown, each cluster 505 includes multiple processor cores (IPU cores) 506 and one memory core (MEM core) 507.

[0074] Four processor cores 506 are shown in the figure as an example, but this application does not limit the number of processor cores 506. Its internal architecture is as follows: Figure 6 As shown, each processor core 506 includes three main modules: a control module 51, an arithmetic module 52, and a storage module 53.

[0075] The control module 51 coordinates and controls the operation of the computation module 52 and the storage module 53 to complete the deep learning task. It includes an instruction fetch unit (IFU) 611 and an instruction decode unit (IDU) 612. The instruction fetch unit 611 fetches instructions from the processing device 403, and the instruction decode unit 612 decodes the fetched instructions and sends the decoding result as control information to the computation module 52 and the storage module 53.

[0076] The computation module 52 includes a vector operation unit 621 and a matrix operation unit 622. The vector operation unit 621 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 622 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0077] Storage module 53 is used to store or move related data, including Neuron RAM (NRAM) 631, Weight RAM (WRAM) 632, Input / Output Direct Memory Access (IODMA) 633, and Move Direct Memory Access (MVDMA) 634. NRAM 631 is used to store input, output data, and intermediate results for computation by processor core 506; WRAM 632 is used to store the weights of the deep learning network; IODMA 633 controls the memory access of NRAM 631 / WRAM 632 and DRAM 404 through broadcast bus 509; MVDMA 634 controls the memory access of NRAM 631 / WRAM 632 and SRAM 508.

[0078] Back Figure 5 The storage core 507 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 506, and performing communication between cluster 505 and DRAM 404, communication between clusters 505, and communication between processor cores 506. In other embodiments, the storage core 507 has scalar operation capabilities and is used to perform scalar operations.

[0079] Storage core 507 includes a shared memory unit (SRAM) 508, a broadcast bus 509, a Cluster Direct Memory Access (CDMA) module 510, and a Global Direct Memory Access (GDMA) module 511. SRAM 508 acts as a high-performance data relay station. Data multiplexed between different processor cores 506 within the same cluster 505 does not need to be obtained from DRAM 404 by each processor core 506 individually. Instead, it is relayed between processor cores 506 via SRAM 508. Storage core 507 only needs to quickly distribute the multiplexed data from SRAM 508 to multiple processor cores 506, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0080] Broadcast bus 509, CDMA 510, and GDMA 511 are used to perform communication between processor cores 506, communication between clusters 505, and data transfer between cluster 505 and DRAM 404, respectively. These will be explained separately below.

[0081] The broadcast bus 509 is used to complete high-speed communication between the processor cores 506 within the cluster 505. In this embodiment, the broadcast bus 509 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 508 to several specific processor cores 506. Broadcast is a communication method that transmits a piece of data from SRAM 508 to all processor cores 506, and is a special case of multicast.

[0082] CDMA 510 is used to control SRAM 508 access between different clusters 505 within the same computing device 401. Figure 7 This diagram illustrates the operation of CDMA510 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0083] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0084] Back Figure 5The GDMA 511, in conjunction with the external memory controller 501, controls memory access from the SRAM 508 of the cluster 505 to the DRAM 404, or reads data from the DRAM 404 into the SRAM 508. As described above, communication between the DRAM 404 and the NRAM 631 or WRAM 632 can be achieved through two channels. The first channel is a direct connection between the DRAM 404 and the NRAM 631 or WRAM 632 via the IODAM 633; ​​the second channel involves first transmitting data between the DRAM 404 and SRAM 508 via the GDMA 511, and then transmitting data between the SRAM 508 and the NRAM 631 or WRAM 632 via the MVDMA 634. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 404 and the NRAM 631 or WRAM 632 may be more efficient via the second channel. The embodiments of this application can select the data transmission channel according to their own hardware conditions.

[0085] In other embodiments, the functions of GDMA 511 and IODMA 633 can be integrated into the same component. For ease of description, this application treats GDMA 511 and IODMA 633 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those of this application falls within the scope of protection of this application. Furthermore, the functions of GDMA 511, IODMA 633, CDMA 510, and MVDMA 634 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those of this application falls within the scope of protection of this application.

[0086] The above combination Figures 3-7 The hardware architecture and internal structure of this application have been described in detail. It is understood that the above description is merely exemplary and not restrictive. Depending on different application scenarios and hardware specifications, those skilled in the art can also make changes to the board (or artificial intelligence device) and its internal structure, and these changes still fall within the protection scope of this application. In addition to... Figures 3-7 The hardware architecture shown in this application also involves a software and hardware architecture, which will be described below.

[0087] Figure 8The diagram illustrates a hardware and software architecture design according to an embodiment of this application. As shown in the diagram, the hardware and software architecture in this embodiment may include an AI processor 801, a driver and operating system 802, a compiler and programming language 803, a library 804, a framework layer 805, and an application layer 806. It is understood that this hardware and software architecture can be applied to the artificial intelligence computing system or heterogeneous computing platform of this application.

[0088] Specifically, the AI ​​processor 801 (which may be included, for example, in the board described below in conjunction with the accompanying drawings) incorporates both computational and data handling optimizations in its hardware design. To this end, it employs customized computational units to accelerate computation and on-chip memory to accelerate data handling, thereby achieving extremely high performance and energy efficiency. Furthermore, to support various algorithm optimizations, the AI ​​processor 801 can have customized computational units and instruction sets, where the instruction set can provide computational instructions of different granularities (scalar, vector, and / or matrix). Moreover, considering factors such as algorithm memory access characteristics, hardware cost, and verification difficulty, on-chip memory can be used, and data handling can be optimized. In practice, the AI ​​processor of this application can achieve speeds tens of times faster than mainstream GPUs (Graphics Processing Units).

[0089] The driver and operating system 802 are primarily responsible for scheduling tasks on the AI ​​processor 801. This scheduling operation can include, for example, scheduling based on task priority, communication and synchronization between multiple devices. For the compiled program, the operating system and driver can schedule and execute the task to be performed on a specific processor, including but not limited to the following operations: allocating and releasing device memory, enabling data transfer between devices, maintaining task queues, and scheduling tasks according to priority to achieve synchronization and cooperation between multiple devices.

[0090] The compiler and programming language 803 can be an assembly language developed for the instruction set of the AI ​​processor 801. In applications, it can translate deep learning operators developed for the AI ​​processor 801 into combinations of processor instructions, enabling efficient use of the AI ​​processor 801. In some application scenarios, the compiler can be used to optimize the compilation process by executing intermediate expression stages.

[0091] Library 804 may include runtime library 814 and machine learning library 824. In one implementation scenario, the aforementioned library 804 can use the instruction set of AI processor 801 and perform partial optimizations based on the instruction set of AI processor 801 to improve the running speed of operators. Runtime library 814 may be a high-performance operator library specifically developed for AI processor 801, and it can be used to complete the interaction between general-purpose processors and artificial intelligence processors. Furthermore, runtime library 814 can also provide a set of interfaces for artificial intelligence processors. As for machine learning library 824, it can be used to accelerate various machine learning or deep learning algorithms on artificial intelligence processors. Specifically, machine learning library 824 can provide a set of efficient, general-purpose, flexible and scalable programming interfaces. Its upper-layer machine learning applications can directly adopt the programming interfaces of various programming frameworks (such as PyTorch, TensorFlow, Caffe, MXNet, etc.), or they can directly program using the interface provided by machine learning library 824. In addition, the machine learning library 824 of this application can be easily called by hardware platforms, while runtime library 814 can implement some basic and commonly used operators, such as convolution, pooling and other operations.

[0092] Framework layer 805 can add encapsulation for operators developed for AI processors, primarily encapsulating operators from runtime library 814. In addition, framework layer 805 can modify related task scheduling or memory management components. In one application scenario, framework layer 805 can adopt the architecture of frameworks such as PyTorch and TensorFlow.

[0093] Figure 9 This is an exemplary flowchart illustrating a long-term fusion method 900 for multi-view 3D object detection according to an embodiment of this application. Figure 9 As shown, in step S901, multi-view two-dimensional images of the data frame in the target scene are acquired, and the self-motion matrix between adjacent frames is obtained. In some embodiments, the target scene includes, but is not limited to, autonomous driving, machine vision (e.g., robotics), and virtual reality. Taking autonomous driving as an example, two-dimensional images of six views in the data frame can typically be acquired using acquisition devices such as cameras or radar.

[0094] It is understood that the aforementioned self-motion matrix between adjacent frames can describe the dynamic behavior of a 3D target, including, for example, changes in position or orientation. In some embodiments, the self-motion matrix may include, but is not limited to, parameters such as the acceleration and turning angle of the 3D target. In some implementation scenarios, the aforementioned self-motion matrix can be acquired by acquisition devices such as radar.

[0095] Next, in step S902, based on the multi-view 2D images and self-motion matrix under the data frame, a bird's-eye view detector is used to extract the first bird's-eye view features of all historical frames and the second bird's-eye view features of the current frame. In some embodiments, the bird's-eye view detector may include at least an image encoding module, a view transformation module, and a long-term fusion module. In some implementation scenarios, the image features corresponding to the multi-view 2D images under the data frame are extracted by using the image encoding module, and the corresponding image features are converted into initial bird's-eye view features under each data frame by the view transformation module. Then, based on the initial bird's-eye view features, self-motion matrix, and initial weight coefficients under each data frame, the long-term fusion module is used to extract the first bird's-eye view features of all historical frames and the second bird's-eye view features of the current frame.

[0096] That is, by inputting multi-view 2D images and self-motion matrices from data frames into a bird's-eye view detector, image features are first extracted by an image encoding module in the bird's-eye view detector. Then, the image features are converted into bird's-eye view features by a view transformation module. Finally, a long-term fusion module extracts the first bird's-eye view features of all historical frames and the second bird's-eye view features of the current frame. In some embodiments, the aforementioned image encoding module may be, for example, a convolutional network model, and the aforementioned view transformation module may be, for example, a Transform model. In other embodiments, the aforementioned long-term fusion module may include at least a linear transformation unit and a convolutionally gated recurrent unit, whereby the linear transformation unit is used for self-coordinate transformation between adjacent frames, and the convolutionally gated recurrent unit is used for fusing bird's-eye view features between adjacent frames.

[0097] Specifically, firstly, based on the initial bird's-eye view features of each historical frame, the self-motion matrix between adjacent historical frames, and the initial weight coefficients, a linear transformation unit is used to perform a first self-coordinate transformation on the initial bird's-eye view features of each historical frame to obtain the transformed bird's-eye view features of each historical frame. Then, a convolutional gated recurrent unit is used to fuse the transformed bird's-eye view features of each historical frame with the initial bird's-eye view features of the next historical frame to extract the first bird's-eye view features of all historical frames.

[0098] That is, this application embodiment performs self-coordinate transformation by adding an internal grid to achieve accurate semantic recognition. Specifically, this application embodiment linearly simulates the aforementioned transformation. In an exemplary scenario, the aforementioned linear transformation can be expressed as the following formula:

[0099]

[0100] Among them, W p represents a linear parameter. In some embodiments, it is assumed that different channels in the projection feature share the same motion embedding, thus W p It can be decoupled as follows:

[0101]

[0102] in, This represents the self-movement embedding from timestamp j to timestamp i. The weights represent learnable values, i.e., weight coefficients in the context of this application's embodiments, and ⊙ represents the Hadamard product. In some implementation scenarios, given the self-motion matrix P... j，i The above E j，i It can be based on P j，i The transformed high-dimensional vector:

[0103] E j，i =σ(ψ(P) j，i (10)

[0104] Where σ represents the sigmoid activation function, and ψ represents a multilayer perceptron ("MLP"). Specifically, the linear transformation unit described above in this application embodiment can be represented by the following formula:

[0105]

[0106] For the fusion operation, this embodiment employs a convolutional gated recurrent unit (“ConvGRU”), thus the entire long-term fusion module of this embodiment is denoted as TA-GRU (e.g., Figure 10 (As shown). Specifically, the long-term fusion module of this application embodiment can be decomposed into the following formula:

[0107]

[0108] Where [.,.] denotes the concatenation of feature maps, W z W z W o This represents a two-dimensional convolution kernel. Based on embodiments of this application, the TA-GRU can learn fine-grained semantic changes from the self-motion matrix, thereby eliminating erroneous directions.

[0109] In other embodiments, a second self-coordinate transformation can be performed based on the initial bird's-eye view features of the current frame, the self-motion matrix between the current frame and historical frames, and the initial weight coefficients to obtain the second bird's-eye view features of the current frame. Similar to the first self-coordinate transformation described above, the second bird's-eye view features of the current frame are obtained through a linear transformation unit.

[0110] Based on the first bird's-eye view features of all historical frames extracted above and the second bird's-eye view features of the current frame, in step S903, the first and second bird's-eye view features are initially fused over a long period to obtain an initial fusion result. The gradient is then backpropagated to all historical frames based on the 3D target detection result of the initial fusion result to update the initial weight coefficients of the bird's-eye view detector. Specifically, the gradient backpropagation to all historical frames can be implemented using the above formula (7). In some embodiments, the above convolutional gated recurrent unit can be used to initially fuse the first and second bird's-eye view features over a long period to obtain the initial fusion result.

[0111] In some embodiments, the bird's-eye view detector further includes a feature decoding unit and a task head unit. In some implementations, the feature decoding unit performs feature decoding on the initial fusion result to obtain a decoding result, which is then input to the task head unit for 3D target detection to obtain a 3D target detection result. Further, the gradient is backpropagated to all historical frames based on the 3D target detection result of the initial fusion result to obtain the gradient corresponding to each historical frame, so as to update the initial weight coefficients of the bird's-eye view detector based on the sum of the gradients corresponding to each historical frame.

[0112] Further, in step S904, the first bird's-eye view feature and the second bird's-eye view feature are aligned according to the updated weight coefficients and the self-motion matrix. In some embodiments, the first bird's-eye view feature and the second bird's-eye view feature are aligned by performing a first self-coordinate transformation on the first bird's-eye view feature using the above-mentioned linear transformation unit based on the first bird's-eye view feature, the updated weight coefficients, and the self-motion matrix.

[0113] Finally, at step S905, a final long-term fusion is performed based on the aligned first and second bird's-eye view features to obtain the final fusion result. In some embodiments, a convolutional gated recurrent unit is used to perform a final long-term fusion of the aligned first and second bird's-eye view features to obtain the final fusion result.

[0114] As described above, this application's embodiments employ a novel long-term fusion framework. This framework fuses the bird's-eye view features from all historical frames with the bird's-eye view features of the current frame, and then backpropagates gradients to all historical frames. This ensures that all historical frames contribute gradients and can learn subtle self-motion transformations, reducing semantic differences during inter-frame alignment. Based on this, long-term temporal information can be fully and accurately utilized, improving the long-term fusion capability of the cyclic fusion paradigm. Furthermore, by fusing all historical frame data, this application's embodiments require focusing only on the historical frame data saved in the previous frame during 3D object detection, avoiding an increase in the length of historical frame features and thus maintaining effective latency and memory consumption.

[0115] In some embodiments, the final fusion result obtained by the above-described long-term fusion method can be input to a bird's-eye view detector, and then sequentially decoded by the feature decoding unit of the bird's-eye view detector and detected as a three-dimensional target by the task head unit to obtain the final three-dimensional target detection result. Based on this, a more accurate three-dimensional target detection result can be obtained.

[0116] Figure 10 This is an exemplary schematic diagram illustrating a long-term fusion architecture for multi-view 3D object detection according to an embodiment of this application. For example... Figure 10 As shown in the figure, the bird's-eye view detector in this embodiment may include an image encoding module 1001, a view transformation module 1002, a long-term fusion module 1003, a feature decoding unit 1004, and a task head unit 1005. The long-term fusion module (i.e., the aforementioned TA-GRU) 1003 may include a linear transformation unit 1006 and a convolutional gated recurrent unit 1007. In the implementation scenario, the multi-view two-dimensional images (e.g., exemplarily shown in the figure) under the data frame of the target scene are... t-3 ,I t-2 ,I t-1 ,I t In the bird's-eye view detection process, which involves obtaining the self-motion matrix between adjacent frames, image features are extracted via image encoding module 1001. These features are then converted into bird's-eye view features via view transformation module 1002, such as the bird's-eye view feature B corresponding to each historical frame. t-3 B t-2 B t-1 B t .

[0117] Next, the long-term fusion module 1003 extracts the first bird's-eye view features of all historical frames and the second bird's-eye view features of the current frame. The specific fusion operation is shown within the dashed box in the figure. Based on the initial bird's-eye view features of each historical frame, the self-motion matrix between adjacent historical frames, and the initial weight coefficients, the linear transformation unit 1006 performs a first self-coordinate transformation on the initial bird's-eye view features of each historical frame to obtain the transformed bird's-eye view features of each historical frame. Specifically, this includes based on the self-transformation matrix P... j，i High-dimensional vector E is obtained by performing grid feature sampling. j，i Self-motion embedding is achieved based on weight coefficients, and the first self-coordinate transformation is performed through the linear transformation unit 1006 to obtain the transformed bird's-eye view features. See formulas (9)-(11) above.

[0118] Furthermore, the bird's-eye view features transformed from each historical frame are fused with the initial bird's-eye view features of the next historical frame using a convolutional gated recurrent unit 1007 to extract the first bird's-eye view features of all historical frames. Specifically, refer to formula (12) above to obtain the first bird's-eye view features of all historical frames through fusion. As an example, the bird's-eye view features transformed from the historical frame corresponding to t-3 are... Initial bird's-eye view feature B of the next historical frame t-2 Obtain bird's-eye view features Furthermore, the bird's-eye view features corresponding to t-2 Initial bird's-eye view feature B corresponding to t-1 t-1 By fusing the data, we can obtain the first bird's-eye view features of all historical frames.

[0119] First bird's-eye view features based on all the historical frames obtained above The second bird's-eye view feature B of the current frame t By performing initial fusion, the initial fusion result can be obtained. In some implementation scenarios, the feature decoding unit 1004 performs feature decoding on the initial fusion result to obtain a decoding result. This decoding result is then input to the task head unit 1005 for 3D object detection to obtain a 3D object detection result. Further, the gradient of the 3D object detection result from the initial fusion result is backpropagated to all historical frames (e.g., the arrow shown by the dashed line in the figure) to obtain the gradient corresponding to each historical frame. The initial weight coefficients of the bird's-eye view detector are updated based on the sum of the gradients corresponding to each historical frame. The first bird's-eye view feature and the second bird's-eye view feature are aligned using the updated weight coefficients and the self-motion matrix. Finally, a long-term fusion is performed based on the aligned first and second bird's-eye view features to obtain the final fusion result. The final fusion result is then input into the bird's-eye view detector of this embodiment to obtain the final 3D object detection result.

[0120] The solution implemented in this application allows all historical frames to contribute gradients and learn subtle self-motion transformations, reducing semantic differences during inter-frame alignment and fully and accurately utilizing long-term temporal information to improve the long-term fusion capability of the cyclic fusion paradigm. This results in accurate fusion outcomes, leading to precise 3D object detection.

[0121] In one exemplary scenario, this application embodiment trains, validates, and tests 1000 collected driving scenarios by dividing them into 700, 150, and 150 scenarios. Each scenario lasts approximately 20 seconds. In each scenario, six cameras are used to capture a complete 360-degree field of view. Furthermore, in the aforementioned dataset, 23 object categories and precise 3D bounding boxes are labeled for each scenario at a frequency of 2Hz. Evaluation is conducted using 10 categories: cars, trucks, buses, trailers, construction vehicles, pedestrians, motorcycles, bicycles, guardrails, and traffic cones. The following metrics are used in the evaluation: mean average accuracy (“mAP”), nuScenes detection score (“NDS”), mean translation error (“ATE”), mean scale error (“ASE”), mean orientation error (“AOE”), mean velocity error (“AVE”), and mean attribute error (“AAE”).

[0122] Specifically, this application's embodiments are based on experiments conducted using BEVDet4D-Depth, with the same learning rate, optimizer, and data augmentation. ResNet50, ResNet101, ConNeXt-B, and V2-99 were used as backbone networks. Results for the ResNet50 and ResNet101 models on the aforementioned validation set with pre-trained weights on ImageNet and NullImages are presented, and performance on the test set for the V299 model initialized from DD3D and the ConvNeXt-B model pre-trained on COCO is given. In the comparison process, the StreamPETR training settings were followed, with the training epochs extended to 60, and 2D auxiliary supervision was used for fair comparison.

[0123] To achieve real-time inference speed, this application also optimizes the BEVDet4D-Depth baseline through several key modifications, including, for example, reducing the number of depth bins, eliminating the ASPP module to minimize latency related to view transitions, and merging the six detection tasks performed by CenterHead into one task to significantly reduce post-processing time. Specific comparison results are shown in Tables 2 and 3 below.

[0124] Table 2 shows the comparison results of 3D detection on the validation set.

[0125]

[0126] In the table, ED represents external data, indicating the use of pre-trained weights from an external dataset.

[0127] Table 3 shows the comparison results of 3D detection on the validation set.

[0128]

[0129] The asterisk (*) indicates that the test time increment is used during the test and that future frames are not used in any method.

[0130] Based on the above comparison results, it can be seen that the bird's-eye view detector (i.e., RecurrentBEV) of this application embodiment, using ResNet50 as the backbone, achieves a 1.3% higher mAP and 0.9% higher NDS on the validation set with an input size of 256×704 than the previous state-of-the-art StreamPETR method. Furthermore, this application embodiment also employs a ResNet101 backbone pre-trained with null images and expands the input size to 512×1408. Compared to StreamPETR with the same settings, it achieves a 2.4% higher mAP and a 2.0% higher NDS. The RecurrentBEV of this application embodiment achieves significant improvements in mAP and mATE metrics, indicating an improvement in the detection of occluded objects.

[0131] Furthermore, this embodiment of the application also follows the settings in StreamPETR, using V2-99 and ConvNeXt-B as the backbone, with an input size of 640×1600 for testing. Specifically, when using V299 as the backbone, the RecurrentBEV of this embodiment achieves 1.6% higher mAP and 1.3% higher NDS than the previously state-of-the-art Sparse4Dv2 method. When the backbone is extended to ConvNeXt-B, the RecurrentBEV of this embodiment also outperforms other methods.

[0132] In some embodiments, this application also compares the long-term fusion method of this application with the baselines of parallel fusion paradigms and existing cyclic fusion methods. Specifically, the RecurrentBEV method of this application is 2.5 times faster and has a 42% lower memory footprint than the common SOLOFusion-based parallel fusion paradigm. Compared to existing cyclic fusion methods, the RecurrentBEV method of this application requires only an additional 33MB of memory, while improving the average precision (mAP) from 0.396 to 0.445. For example, as shown in Table 4.

[0133] Table 4 Comparison of Time and Memory in Inference

[0134]

[0135] In some embodiments, this application also conducts experiments on the composition of RecurrentBEV, the receptive field of TA-GRU, and the length of frame sequences during training. In this scenario, ablation experiments were performed using a ResNet-50 backbone. In Exp1, ConvGRU was used as the temporal fusion module, achieving performance similar to VideoBEV. Following Exp2 and Exp3, backpropagation of historical BEV features was implemented during training. In Exp3 and Exp4, the TA-GRU of this application was used instead of the temporal fusion module. Based on this, compared to the baseline, the overall components (BP and TA-GRU) achieved significant performance improvements, such as mAP +1.7%, NDS +1.6%, and mAOE -4.7%, as shown in Table 5.

[0136] Table 5 Ablation status of each component

[0137]

[0138] Here, BP represents backpropagation of historical BEV features, and TG represents TA-GRU. Specifically, TA-GRU demonstrates its ability to enhance mAP and mAOE metrics, regardless of whether backpropagation of historical BEV features is enabled, indicating its consistent ability to alleviate semantic ambiguity for the same object. Furthermore, the mean orientation error (mAOE) increases significantly, suggesting that alignment of historical BEV features facilitates internal mesh transformation, effectively mitigating semantic ambiguity by improving object orientation. For example, as shown in Exp. 4, performance is affected when historical BEV features are separated, highlighting the importance of backpropagation in the embodiments of this application. However, during training, relying solely on backpropagation of historical BEV features yields limited improvement. This indicates that a proper temporal fusion module can effectively perform backpropagation, and the combination of these two components can achieve a synergistic effect exceeding the sum of their individual contributions.

[0139] In some embodiments, this application also relates to experiments on the kernel size of convolutions in TA-GRU, as shown in Table 6, for example.

[0140] Table 6 Sensitive Field Ablation

[0141]

[0142] As shown in the table, both excessively large and excessively small convolutional kernels negatively impact performance. If the kernel is too large, its fusion module will collect noise information, while if the kernel is too small, it will lead to mismatches in moving objects. Preferably, a 5x5 kernel size can be used in the TA-GRU.

[0143] In some embodiments, the present application also uses BEVDet-Depth without temporal fusion as a baseline and applies the long-term fusion method of the present application to different training sequence lengths in Table 6.

[0144] Table 7 Training sequence length

[0145]

[0146] As shown in the table, single-frame fusion significantly enhances velocity prediction ("mAVE"). Fusion of 4 to 8 frames primarily enhances the localization metrics mAP and mATE, improving them by 8.5% and 12.7%, respectively. Further increasing the frame count to 16 frames only provides limited performance improvement. Preferably, the bird's-eye view detector of this embodiment can be trained on 8 frames.

[0147] Furthermore, this embodiment of the application also performs target occlusion analysis on the aforementioned validation set. Specifically, firstly, a "visible object" is defined as an object whose overall visibility exceeds 60%. Next, assuming an object is classified as a "distinct object" in any frame of the video, it is considered visible throughout the entire video. Finally, the overall visibility of the video is obtained by dividing the number of visible objects by the total number of objects. For example... Figure 11 As shown.

[0148] Figure 11 This is an exemplary schematic diagram illustrating the correlation between video visibility and mAP according to an embodiment of this application. Figure 11 As shown in Figure (a), the horizontal axis represents the data frame length, the vertical axis represents visibility on the left and mAP on the right. The video visibility and mAP show the same trend, indicating that the improvement in mAP in long-term fusion stems from the detection of occluded objects. Visibility can be understood as the proportion of different targets in the video, expressed as a percentage of the total number of objects. The Pearson correlation coefficient between visibility and mAP is 0.977. Figure 11 Figure (b) shows the mATE metrics for objects at different locations in the validation set. The horizontal axis represents the target distance, and the vertical axis represents the mATE. The data below each bar represents the result of a single-frame baseline, while the data outside the bars represents the results of the embodiments described in this application. As can be seen from the figure, objects between 25 and 30 meters improved the mATE metric by 24%, demonstrating the advantage of long-term fusion in accurately locating distant objects.

[0149] Based on the foregoing description, those skilled in the art will understand that this application also discloses a device including a processor and a memory. Specifically, the memory can store program instructions for long-term fusion of multi-view 3D object detection, which, when executed by the processor, implement the functionality of this application. Figure 9 The method steps described herein. In some embodiments, the memory may also store program instructions for multi-view 3D object detection, which, when executed by a processor, implement the method for multi-view 3D object detection of this application.

[0150] Furthermore, since the solution of this application can be implemented through computational program instructions, this application also discloses a computer-readable storage medium or computer program product storing a computer program / instructions for long-term fusion of multi-view 3D target detection, thereby achieving the combination of... Figure 9 The described method steps. In some embodiments, computer-readable instructions for multi-view 3D object detection are stored thereon to implement the method for multi-view 3D object detection of this application.

[0151] The solution of this application has been described in detail above with reference to the accompanying drawings. Depending on the application scenario, the equipment or apparatus of this application may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The equipment or apparatus of this application can also be applied to fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare.

[0152] Furthermore, the devices or apparatuses of this application can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminals. In one or more embodiments, the high-power devices or apparatuses according to the scheme of this application can be applied to cloud devices (e.g., cloud servers), while the low-power devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling, and collaborative work of end-to-cloud or cloud-edge-end integration.

[0153] It should be noted that, for the sake of brevity, this application describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this application is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this application, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this application can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this application. In addition, depending on the solution, the description of some embodiments in this application also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this application can also be referred to the relevant descriptions of other embodiments.

[0154] In terms of specific implementation, based on the disclosure and teachings of this application, those skilled in the art will understand that the several embodiments disclosed in this application can also be implemented in other ways not disclosed herein. For example, regarding the various units in the device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0155] In this application, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this application. Additionally, in some scenarios, multiple units in the embodiments of this application may be integrated into one unit or each unit may exist physically independently.

[0156] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the solution of this application is embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this application. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0157] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory ("RRAM"), dynamic random access memory ("DRAM"), static random access memory ("SRAM"), enhanced dynamic random access memory ("EDRAM"), high bandwidth memory ("HBM"), hybrid memory cube ("HMC"), ROM, and RAM, etc.

[0158] The foregoing can be better understood in accordance with the following terms:

[0159] Clause A1, A long-term fusion method for multi-view 3D object detection, comprising:

[0160] Acquire multi-view 2D images of the target scene under data frames and obtain the self-motion matrix between adjacent frames;

[0161] Based on the multi-view two-dimensional images under the data frame and the self-motion matrix, the first bird's-eye view feature of all historical frames and the second bird's-eye view feature of the current frame are extracted using a bird's-eye view detector.

[0162] The first bird's-eye view feature and the second bird's-eye view feature are initially fused over a long period of time to obtain an initial fusion result. The gradient is then backpropagated to all historical frames based on the three-dimensional target detection result of the initial fusion result to update the initial weight coefficients of the bird's-eye view detector.

[0163] The first and second bird's-eye view features are aligned based on the updated weight coefficients and the self-motion matrix; and

[0164] The final fusion result is obtained by performing a long-term fusion based on the aligned first and second bird's-eye view features.

[0165] Clause A2, the long-term fusion method according to Clause A1, wherein the bird's-eye view detector includes at least an image encoding module, a view transformation module, and a long-term fusion module, and based on the multi-view two-dimensional images under the data frame and the self-motion matrix, the extraction of first bird's-eye view features of all historical frames and second bird's-eye view features of the current frame using the bird's-eye view detector includes:

[0166] The image encoding module is used to extract image features corresponding to the multi-view two-dimensional images under the data frame;

[0167] The view transformation module is used to convert the corresponding image features into initial bird's-eye view features for each data frame; and

[0168] Based on the initial bird's-eye view features of each data frame, the self-motion matrix, and the initial weight coefficients, the long-term fusion module is used to extract the first bird's-eye view features of all historical frames and the second bird's-eye view features of the current frame.

[0169] Clause A3, the long-term fusion method according to Clause A2, wherein the long-term fusion module includes at least a linear transformation unit and a convolutional gated recurrent unit, and the linear transformation unit is used for self-coordinate transformation between adjacent frames, and the convolutional gated recurrent unit is used for fusion of bird's-eye view features between adjacent frames.

[0170] Clause A4, the long-term fusion method according to Clause A3, wherein the first bird's-eye view features of historical frames are extracted by the following operations:

[0171] Based on the initial bird's-eye view features of each historical frame, the self-motion matrix between adjacent historical frames, and the initial weight coefficients, the linear transformation unit performs a first self-coordinate transformation on the initial bird's-eye view features of each historical frame to obtain the transformed bird's-eye view features of each historical frame; and

[0172] The convolutional gated recurrent unit is used to fuse the transformed bird's-eye view features of each historical frame with the initial bird's-eye view features of the next historical frame to extract the first bird's-eye view features of all historical frames.

[0173] Clause A5, the long-term fusion method according to Clause A4, wherein the second bird's-eye view feature of the current frame is extracted by the following operations:

[0174] A second self-coordinate transformation is performed based on the initial bird's-eye view features of the current frame, the self-motion matrix between the current frame and historical frames, and the initial weight coefficients to obtain the second bird's-eye view features of the current frame.

[0175] Clause A6. The long-term fusion method according to Clause A3, wherein the initial long-term fusion of the first bird's-eye view feature and the second bird's-eye view feature to obtain the initial fusion result includes:

[0176] The first bird's-eye view feature and the second bird's-eye view feature are initially fused over a long period of time using the convolutional gated recurrent unit to obtain an initial fusion result.

[0177] Clause A7. The long-term fusion method according to Clause A2, wherein the bird's-eye view detector further includes a feature decoding unit and a task head unit, and the three-dimensional target detection result is obtained through the following operations:

[0178] The feature decoding unit is used to perform feature decoding on the initial fusion result to obtain a decoding result; and

[0179] The decoding result is input into the task head unit for three-dimensional target detection to obtain the three-dimensional target detection result.

[0180] Clause A8, the long-term fusion method according to Clause A7, wherein the gradient is backpropagated to all historical frames based on the 3D object detection results of the initial fusion result to update the initial weight coefficients of the bird's-eye view detector includes:

[0181] Based on the 3D target detection results of the initial fusion result, the gradient is backpropagated to all historical frames to obtain the gradient corresponding to each historical frame; and

[0182] The initial weight coefficients of the bird's-eye view detector are updated based on the sum of gradients corresponding to each historical frame.

[0183] Clause A9. The long-term fusion method according to Clause A3, wherein feature alignment of the first bird's-eye view feature and the second bird's-eye view feature based on the updated weight coefficients and the self-motion matrix includes:

[0184] Based on the first bird's-eye view feature, the updated weight coefficients, and the self-motion matrix, the first bird's-eye view feature is subjected to a first self-coordinate transformation using the linear transformation unit to align the first bird's-eye view feature with the second bird's-eye view feature.

[0185] Clause A10, the long-term fusion method according to Clause A9, wherein the final long-term fusion is performed based on the aligned first and second bird's-eye view features to obtain the final fusion result includes:

[0186] The convolutional gated recurrent unit is used to perform a final long-term fusion of the aligned first bird's-eye view features and the second bird's-eye view features to obtain the final fusion result.

[0187] Clause A11, A method for multi-view 3D target detection, comprising:

[0188] The final fusion result obtained by the long-term fusion method according to any one of Clauses A1-A10 is input to the bird's-eye view detector, and then the feature decoding unit of the bird's-eye view detector performs feature decoding and the task head unit performs 3D target detection in sequence to obtain the final 3D target detection result.

[0189] Clause A12, A long-term fusion device for multi-view 3D target detection, comprising:

[0190] Processor; and

[0191] A memory storing program instructions for long-term fusion of multi-view 3D target detection, which, when executed by the processor, cause the long-term fusion apparatus to implement the long-term fusion method according to any one of clauses A1-A10.

[0192] Clause A13, A device for multi-view 3D target detection, comprising:

[0193] Processor; and

[0194] A memory containing program instructions for multi-view 3D target detection, which, when executed by the processor, cause the device to implement the method described in accordance with clause A11.

[0195] Clause A14, a board comprising: a long-term fusion device as described in Clause A12; or a device as described in Clause A13.

[0196] Clause A15. A computer-readable storage medium storing computer-readable instructions for long-term fusion of multi-view 3D object detection, which, when executed by one or more processors, implement the long-term fusion method as described in any one of Clauses A1-A10; or storing computer-readable instructions for multi-view 3D object detection, which, when executed by one or more processors, implement the method as described in Clause A11.

[0197] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A long-term fusion method for multi-view three-dimensional target detection, comprising: collecting multi-view two-dimensional images under a data frame in a target scene and obtaining a self-motion matrix between adjacent frames; extracting first aerial view features of all historical frames and second aerial view features of a current frame using an aerial view detector based on the multi-view two-dimensional images under the data frame and the self-motion matrix; performing initial long-term fusion on the first aerial view features and the second aerial view features to obtain an initial fusion result, and back-propagating a gradient to all historical frames according to a three-dimensional target detection result of the initial fusion result to update initial weight coefficients of the aerial view detector; aligning the first aerial view features and the second aerial view features according to the updated weight coefficients and the self-motion matrix; and performing final long-term fusion based on the aligned first aerial view features and the second aerial view features to obtain a final fusion result. 2.The long-term fusion method of claim 1, wherein the aerial view detector at least comprises an image encoding module, a view conversion module and a long-term fusion module, and the extracting first aerial view features of all historical frames and second aerial view features of a current frame using an aerial view detector based on the multi-view two-dimensional images under the data frame and the self-motion matrix comprises: extracting image features corresponding to the multi-view two-dimensional images under the data frame using the image encoding module; converting the corresponding image features into initial aerial view features under each data frame using the view conversion module; and extracting first aerial view features of all historical frames and second aerial view features of a current frame using the long-term fusion module based on the initial aerial view features under each data frame, the self-motion matrix and the initial weight coefficients. 3.The long-term fusion method of claim 2, wherein the long-term fusion module at least comprises a linear transformation unit and a convolutional gated recurrent unit, and the linear transformation unit is used for self-coordinate transformation between adjacent frames, and the convolutional gated recurrent unit is used for fusion of aerial view features between adjacent frames. 4.The long-term fusion method of claim 3, wherein the first aerial view features of historical frames are extracted by: performing first self-coordinate transformation on the initial aerial view features of each historical frame using the linear transformation unit based on the initial aerial view features of each historical frame, a self-motion matrix between adjacent historical frames and the initial weight coefficients to obtain transformed aerial view features of each historical frame; and fusing the transformed aerial view features of each historical frame with initial aerial view features of a next historical frame using the convolutional gated recurrent unit to extract the first aerial view features of all historical frames. 5.The long-term fusion method of claim 4, wherein the second aerial view features of a current frame are extracted by: performing second self-coordinate transformation based on initial aerial view features of the current frame, a self-motion matrix between the current frame and historical frames and the initial weight coefficients to obtain the second aerial view features of the current frame. ​ ​ 6. The long-term fusion method of claim 3, wherein the initial long-term fusion of the first aerial view feature and the second aerial view feature to obtain an initial fusion result comprises: initially fusing the first aerial view feature and the second aerial view feature using the convolutional gated recurrent unit to obtain an initial fusion result.

7. The long-term fusion method of claim 2, wherein the aerial view detector further comprises a feature decoding unit and a task head unit, and the three-dimensional object detection result is obtained by: feature decoding the initial fusion result using the feature decoding unit to obtain a decoded result; and inputting the decoded result into the task head unit for three-dimensional object detection to obtain the three-dimensional object detection result.

8. The long-term fusion method of claim 7, wherein the backpropagating gradients from the three-dimensional object detection result of the initial fusion result to all historical frames to update initial weight coefficients of the aerial view detector comprises: backpropagating gradients from the three-dimensional object detection result of the initial fusion result to all historical frames to obtain gradients corresponding to each historical frame; and updating the initial weight coefficients of the aerial view detector based on a sum of the gradients corresponding to each historical frame.

9. The long-term fusion method of claim 3, wherein the feature alignment of the first aerial view feature and the second aerial view feature according to the updated weight coefficients and the ego-motion matrix comprises: performing a first ego-coordinate transformation on the first aerial view feature using the linear transformation unit based on the first aerial view feature, the updated weight coefficients, and the ego-motion matrix to align the first aerial view feature and the second aerial view feature.

10. The long-term fusion method of claim 9, wherein the final long-term fusion of the aligned first aerial view feature and the second aerial view feature to obtain a final fusion result comprises: performing a final long-term fusion of the aligned first aerial view feature and the second aerial view feature using the convolutional gated recurrent unit to obtain the final fusion result.

11. A method for multi-view three-dimensional object detection, comprising: inputting a final fusion result obtained by the long-term fusion method of any one of claims 1-10 into the aerial view detector, and sequentially performing feature decoding via a feature decoding unit of the aerial view detector and three-dimensional object detection via a task head unit to obtain a final three-dimensional object detection result.

12. A long-term fusion device for multi-view three-dimensional object detection, comprising: a processor; and a memory having stored therein program instructions for long-term fusion for multi-view three-dimensional object detection, which when executed by the processor, cause the long-term fusion device to implement the long-term fusion method of any one of claims 1-10.

13. A device for multi-view three-dimensional object detection, comprising: a processor; and a memory having stored therein program instructions for multi-view three-dimensional object detection, which when executed by the processor, cause the device to implement the method of claim 11. ​ ​ ​ 14. A board card comprising: The long temporal fusion device according to claim 12; Or the device according to claim 13.

15. A computer readable storage medium having stored thereon computer readable instructions for long temporal fusion for multi-view three-dimensional object detection, which, when executed by one or more processors, implement the long temporal fusion method according to any one of claims 1-10; or having stored thereon computer readable instructions for multi-view three-dimensional object detection, which, when executed by one or more processors, implement the method according to claim 11.

Images