Multi-view 3D target detection method and device based on space-time alignment and storage medium

Abstract

The invention discloses a multi-view 3D target detection method and device based on space-time alignment and a storage medium, and belongs to the field of target detection. The method comprises the following steps: receiving a multi-view image of a current frame; re-projecting a 2D detection result and a 3D detection result from at least one historical frame to each view image plane of the current frame based on a camera geometrical relationship through a space-time target prior propagator, and generating a space-time target prior image; fusing the space-time target prior image and the multi-view image features of the current frame through a space-time geometric encoder, and outputting a space-time geometric feature map; performing 2D target detection on the space-time geometric feature map to obtain a 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain corresponding depth information; based on the 2D detection result and the depth information, constructing a 3D query of the target center through a target center query generator; and inputting the 3D query into a 3D detection head, and outputting a 3D target detection result of the current frame.

Description

Multi-view 3D target detection method, device, and storage medium based on spatiotemporal alignment Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a multi-view 3D target detection method, apparatus and storage medium based on spatiotemporal alignment. Background Technology

[0002] 3D object detection aims to identify objects and estimate their precise location and size in 3D space. It is a fundamental component in numerous fields, such as autonomous driving, robotic manipulation, and intelligent navigation. In recent years, look-around 3D detection has shown great potential because it relies solely on camera-based 2D input and has significantly lower computational costs compared to radar-based methods. Existing methods typically employ a query-driven detection framework that follows DETR3D. These methods usually initialize object queries and propagate them between frames to encode temporal context. While this temporal query propagation brings performance improvements, their strong dependence on 2D image planar feature sampling makes it difficult for 3D detectors to perform well in complex scenes where 2D detectors excel.

[0003] Figure 1 illustrates an existing multi-view 3D object detection framework based on 2D detection. Referring to Figure 1, existing look-around 3D object detectors typically initialize high-confidence queries using 2D information from the current frame, while using historically propagated 3D features from multiple past frames as priors for joint detection. However, this high dependence on 2D cues inevitably leads to spatiotemporal inconsistencies between 2D and 3D representations; that is, existing 2D-to-3D pipelines suffer from spatiotemporal discrepancies due to over-reliance on 2D detection. Specifically, 2D features lack sufficient spatial location information, resulting in limited spatial retrieval capabilities during 3D inference. Furthermore, insufficient temporal interaction in existing methods often leads to target omissions under occlusion conditions.

[0004] To address this issue, recent research has introduced 2D-to-3D pipelines, utilizing 2D detectors as reliable geometric priors to optimize or initialize 3D queries. Specifically, methods such as MV2D and QAF2D project RoI-aligned 2D features into 3D space using camera parameters. However, this design typically assigns higher confidence to 3D queries initialized from 2D information, while neglecting the importance of historical 3D cues. Therefore, these methods inherit inherent limitations in both spatial and temporal modeling.

[0005] From a spatial perspective, 3D space contains additional geometric dimensions (such as depth, scale, and orientation) that are not explicitly modeled by 2D features. Therefore, 3D queries derived solely from 2D representations lack sufficient geometric foundation, leading to spatial discrepancies and suboptimal localization accuracy after optimization. This problem is further exacerbated by the continued dominance of 2D features in subsequent query optimization. From a temporal perspective, these methods do not fully utilize the temporal dynamics encoded in historical 3D detections. When a target is occluded in the current frame, the model cannot effectively utilize accumulated 3D positional cues to maintain focus on the target, resulting in temporal inconsistencies and target omissions under occlusion conditions, as shown in Figure 1.

[0006] To address the technical problems of existing surround-view 3D target detectors, which rely excessively on 2D detection information, resulting in inaccuracies in the spatial dimension and breaks in the temporal dimension between 2D and 3D feature representations, leading to decreased positioning accuracy, missed detection of occluded targets, and inconsistencies in spatiotemporal perception in complex dynamic scenes, no effective solution has yet been proposed. Summary of the Invention

[0007] The embodiments of this disclosure provide a multi-view 3D target detection method, apparatus and storage medium based on spatiotemporal alignment, so as to at least solve the technical problems existing in the prior art where the existing surround-view 3D target detectors are overly dependent on 2D detection information, resulting in inaccuracy of 2D and 3D feature representations in the spatial dimension and breakage in the temporal dimension, which in turn leads to a decrease in the positioning accuracy of 3D target detection in complex dynamic scenes, missed detection of occluded targets and inconsistency of spatiotemporal perception.

[0008] According to one aspect of the present disclosure, a multi-view 3D target detection method based on spatiotemporal alignment is provided, comprising: receiving a multi-view image of a current frame, wherein the multi-view image is a set of images covering a preset panoramic range simultaneously acquired by multiple cameras with different viewpoints; generating a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometric relationships using a pre-trained spatiotemporal target prior propagator; fusing the spatiotemporal target prior map and the multi-view image features of the current frame using a pre-trained spatiotemporal geometric encoder to output a spatiotemporal geometric feature map; performing 2D target detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain corresponding depth information; constructing a 3D query for the target center using a pre-trained target center query generator based on the 2D detection result and the depth information; inputting the 3D query into a preset 3D detection head and outputting the 3D target detection result of the current frame.

[0009] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0010] According to another aspect of the present disclosure, a multi-view 3D target detection device based on spatiotemporal alignment is also provided, comprising: a multi-view image receiving module, configured to receive multi-view images of the current frame, wherein the multi-view images are a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different viewpoints; a spatiotemporal target prior map generation module, configured to generate a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using a pre-trained spatiotemporal target prior propagator; and a spatiotemporal geometric feature map generation module, configured to generate a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry; and a spatiotemporal geometric feature map generation module, configured to generate a spatiotemporal geometric feature map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry. The system comprises: a spatiotemporal target prior map and multi-view image features of the current frame, and an encoder that integrates these features to output a spatiotemporal geometric feature map; a 2D detection result and depth information determination module that performs 2D target detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and performs depth estimation on the multi-view image of the current frame to obtain the corresponding depth information; a 3D query construction module that constructs a 3D query for the target center based on the 2D detection result and the depth information using a pre-trained target center query generator; and a 3D target detection result determination module that inputs the 3D query into a preset 3D detection head and outputs the 3D target detection result of the current frame.

[0011] According to another aspect of the present disclosure, a multi-view 3D target detection device based on spatiotemporal alignment is also provided, comprising: a processor; and a memory connected to the processor, configured to provide the processor with instructions for processing the following steps: receiving a multi-view image of the current frame, wherein the multi-view image is a set of images covering a preset panoramic range simultaneously acquired by multiple cameras with different viewpoints; generating a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using a pre-trained spatiotemporal target prior propagator; fusing the spatiotemporal target prior map and the multi-view image features of the current frame using a pre-trained spatiotemporal geometry encoder; performing 2D target detection on the spatiotemporal geometry feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain corresponding depth information; constructing a 3D query for the target center using a pre-trained target center query generator based on the 2D detection result and the depth information; inputting the 3D query into a preset 3D detection head and outputting the 3D target detection result of the current frame.

[0012] This application first receives a multi-view image of the current frame, which is a set of images simultaneously acquired by multiple cameras from different perspectives, covering a preset panoramic range, providing basic observation data for subsequent cross-view spatiotemporal alignment and 3D inference. Then, through a pre-trained spatiotemporal target prior propagator, the 2D and 3D detection results from at least one historical frame are reprojected onto the view image planes of the current frame based on camera geometry, generating a spatiotemporal target prior map. This explicitly injects historically verified, geometrically reliable spatial structure and temporal dynamic information into the 2D perception layer of the current frame, providing a structured prior for establishing cross-modal and cross-temporal feature consistency. Next, through a pre-trained spatiotemporal geometry encoder, the spatiotemporal target prior map is fused with image features extracted from the multi-view image of the current frame, and depth-aware geometric information is incorporated to output a spatiotemporal geometric feature map. This achieves depth interaction and alignment between historical priors and current observations at the feature level, enhancing the model's ability to perceive complex spatial structures (such as depth and direction). Subsequently, 2D object detection is performed on the spatiotemporal geometric feature map to obtain accurate 2D detection results for the current frame. Simultaneously, depth estimation is performed on the multi-view image of the current frame to obtain corresponding depth information, thus providing two key, interconnected perceptual outputs for the geometric enhancement from 2D to 3D. Next, based on the 2D detection results and depth information, a pre-trained object center query generator is used to back-project the 2D bounding box center and depth values ​​to construct a 3D query for the object center. This generates a geometrically sound 3D query with a clear spatial location, effectively overcoming the inherent ambiguity of directly inferring 3D position from 2D appearance features. Finally, the 3D query is input into a preset 3D detection head, decoded, and the final 3D object detection result for the current frame is output, completing the end-to-end inference from multi-view 2D images to accurate 3D detection. Therefore, this application proposes a unified framework that integrates heterogeneous 2D and 3D information through explicit spatiotemporal alignment, enhances geometric perception, and utilizes historical information to recover occluded targets. This significantly improves the localization accuracy, target recall, and spatiotemporal perception consistency of surround-view 3D detection in dynamic and occluded scenarios. Furthermore, it addresses the technical problems existing in current surround-view 3D target detectors, which rely excessively on 2D detection information, leading to inaccuracies in the spatial dimension and breaks in the temporal dimension of 2D and 3D feature representations. This, in turn, causes a decrease in localization accuracy, missed detection of occluded targets, and spatiotemporal perception in complex dynamic scenes. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the accompanying drawings: Figure 1 is a framework diagram of existing 2D detection-based multi-view 3D object detection; Figure 2 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of this disclosure; Figure 3 is a flowchart of the spatiotemporal alignment-based multi-view 3D object detection method according to Embodiment 1 of this application; Figure 4 is a framework diagram of the spatiotemporal alignment-based multi-view 3D object detection method according to Embodiment 1 of this application; Figure 5 is a framework diagram of the spatiotemporal object prior propagator according to Embodiment 1 of this application; Figure 6 is a framework diagram of the spatiotemporal geometry encoder according to Embodiment 1 of this application; Figure 7 is a framework diagram of STUR3D according to Embodiment 1 of this application; Figure 8 is a qualitative detection result diagram of multi-view images and BEV space on the nuScenes validation set according to Embodiment 1 of this application; Figure 9 is a schematic diagram of the spatiotemporal alignment-based multi-view 3D object detection device according to Embodiment 2 of this application; Figure 10 is a schematic diagram of the spatiotemporal alignment-based multi-view 3D object detection device according to Embodiment 3 of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0015] Example 1 According to this embodiment, a method embodiment of a multi-view 3D target detection method based on spatiotemporal alignment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0016] The method embodiments provided in this example can be executed in a server or similar computing device. Figure 2 shows a hardware structure block diagram of a computing device for implementing a spatiotemporally aligned multi-view 3D object detection method. As shown in Figure 2, the computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may include a display, a keyboard, and a cursor control device connected to the input / output interface. Those skilled in the art will understand that the structure shown in Figure 2 is merely illustrative and does not limit the structure of the aforementioned electronic device. For example, the computing device may include more or fewer components than shown in Figure 2, or have a different configuration than shown in Figure 2.

[0017] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this application. The data processing circuit can be embodied, in whole or in part, as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit can be a single, independent processing module, or it can be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0018] The memory can be used to store software programs and modules of application software, such as the program instruction / data storage device corresponding to the spatiotemporal alignment-based multi-view 3D object detection method in this embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the spatiotemporal alignment-based multi-view 3D object detection method of the aforementioned application. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0019] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0020] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0021] It should be noted that, in some alternative embodiments, the computing device shown in FIG2 above may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be pointed out that FIG2 is merely one example of a specific embodiment and is intended to illustrate the types of components that may be present in the aforementioned computing device.

[0022] Under the above operating environment, according to the first aspect of this embodiment, a multi-view 3D target detection method based on spatiotemporal alignment is provided. Figure 3 shows a flowchart of the method, and Figure 4 shows the overall architecture of the method. Referring to Figures 3 and 4, the method includes: S301: receiving a multi-view image of the current frame, wherein the multi-view image is a set of images covering a preset panoramic range, synchronously acquired by multiple cameras from different perspectives; S302: generating a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometric relationships using a pre-trained spatiotemporal target prior propagator; S303: using a pre-trained... The spatiotemporal geometric encoder fuses the spatiotemporal target prior map with the multi-view image features of the current frame to output a spatiotemporal geometric feature map; S304: 2D target detection is performed on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and depth estimation is performed on the multi-view image of the current frame to obtain the corresponding depth information; S305: Based on the 2D detection result and the depth information, a 3D query of the target center is constructed through a pre-trained target center query generator; S306: The 3D query is input into a preset 3D detection head to output the 3D target detection result of the current frame.

[0023] In this embodiment, the multi-view image of the current frame is first received. This multi-view image is a set of images covering a preset panoramic range, simultaneously captured by multiple cameras from different perspectives (corresponding to step S301). Specifically, multiple cameras mounted on a platform (such as an autonomous vehicle) capture synchronized images from different directions at the same timestamp. These images together constitute a 360-degree observation of the surrounding environment. This provides the model with comprehensive visual input to support subsequent cross-view spatiotemporal information fusion and 3D spatial reasoning.

[0024] Then, using a pre-trained spatiotemporal target prior propagator, the 2D and 3D detection results from at least one historical frame are reprojected onto the image planes of each view of the current frame based on camera geometry, generating a spatiotemporal target prior map (corresponding to step S302). Specifically, using the high-confidence detection information confirmed in historical frames (including their 2D positions in the image and their 3D positions in three-dimensional space), combined with camera parameters and system self-motion information, these historical detection boxes are precisely projected onto each image view of the current frame. Thus, the stable spatial structure and temporal evolution clues implicit in the historical data are explicitly transformed into structured priors on the image plane of the current frame, providing spatiotemporal context to guide the perception of the current frame.

[0025] Secondly, a spatiotemporal geometric encoder, pre-trained, fuses the spatiotemporal target prior map with the multi-view image features of the current frame to output a spatiotemporal geometric feature map (corresponding to step S303). Specifically, depth estimation can be performed on the current frame image to obtain the depth distribution information of the scene. Subsequently, by combining the coordinate and depth information of the image itself, a feature representation containing geometric relationships is constructed. Then, through a depth-aware attention mechanism, the constructed geometric features, the visual features of the current frame image, and the spatiotemporal target prior map generated in the previous step are deeply fused and interacted. Thus, an enhanced feature representation (i.e., a spatiotemporal geometric feature map) that contains rich appearance texture, precise geometric structure (such as depth and positional relationships), and incorporates historical priors is generated, laying the foundation for subsequent accurate target detection.

[0026] Subsequently, 2D target detection is performed on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and depth estimation is performed on the multi-view image of the current frame to obtain the corresponding depth information (corresponding to step S304). Specifically, the operation is performed directly on the enhanced feature map that integrates spatiotemporal and geometric information to predict the two-dimensional bounding boxes, categories, and corresponding feature vectors of each target in the current frame image. At the same time, the original multi-view image is processed in parallel to output the depth value corresponding to each pixel or region. Thus, the accurate target localization in the image coordinates of the current frame (2D detection result) and a key dimension information (depth) in three-dimensional space are obtained simultaneously. The two together constitute the necessary and complementary clues for upgrading from two-dimensional image space to three-dimensional physical space.

[0027] Next, based on the 2D detection results and the depth information, a 3D query for the target center is constructed using a pre-trained target center query generator (corresponding to step S305). Specifically, for each 2D detection box, its feature vector is extracted, and the depth value corresponding to its center point in the depth map is read. Using the camera's imaging model (intrinsic parameters), the image center coordinates (u, v) and the depth value (d) are combined to calculate the preliminary center position of the target in 3D space. Then, this geometric position information is fused with the target's feature vector to form a 3D query vector centered on the target, containing semantics and preliminary geometric position. Thus, an initial query with a clear geometric basis that can be directly used for 3D space decoding is generated, effectively avoiding the ambiguity and errors caused by blindly guessing the 3D position from only 2D features.

[0028] Finally, the 3D queries are input into a preset 3D detection head, which outputs the 3D target detection results for the current frame (corresponding to step S306). Specifically, the 3D detection head (usually a Transformer-based decoder) receives a set of 3D queries generated in the previous step. This 3D detection head uses iterative self-attention and cross-attention mechanisms to interact with the global spatiotemporal geometric feature map, progressively optimizing the 3D position, size, orientation, and velocity attributes of the target represented by each query. Thus, it ultimately outputs the 3D bounding boxes of all detected targets in the current frame, completing the end-to-end prediction from multi-view 2D image input to 3D detection results.

[0029] As described in the background section, existing look-around 3D object detectors typically initialize high-confidence queries using 2D information from the current frame, while using historically propagated 3D features from multiple past frames as priors for joint detection. However, this heavy reliance on 2D cues inevitably leads to spatiotemporal inconsistencies between 2D and 3D representations; that is, existing 2D-to-3D pipelines suffer from spatiotemporal discrepancies due to over-reliance on 2D detection. Specifically, 2D features lack sufficient spatial location information, resulting in limited spatial retrieval capabilities during 3D inference. Furthermore, insufficient temporal interaction in existing methods often leads to target omissions in occluded scenarios.

[0030] In view of this, this application first receives a multi-view image of the current frame, which is a set of images covering a preset panoramic range, synchronously acquired by multiple cameras from different perspectives, providing basic observation data for subsequent cross-view spatiotemporal alignment and 3D inference. Then, through a pre-trained spatiotemporal target prior propagator, the 2D and 3D detection results from at least one historical frame are reprojected onto the view image planes of the current frame based on camera geometry, generating a spatiotemporal target prior map. This explicitly injects historically verified, geometrically reliable spatial structure and temporal dynamic information into the 2D perception layer of the current frame, providing a structured prior for establishing cross-modal and cross-temporal feature consistency. Next, through a pre-trained spatiotemporal geometry encoder, the spatiotemporal target prior map is fused with image features extracted from the multi-view image of the current frame, and depth-aware geometric information is incorporated to output a spatiotemporal geometric feature map. This achieves depth interaction and alignment between historical priors and current observations at the feature level, enhancing the model's ability to perceive complex spatial structures (such as depth and direction). Subsequently, 2D object detection is performed on the spatiotemporal geometric feature map to obtain accurate 2D detection results for the current frame. Simultaneously, depth estimation is performed on the multi-view image of the current frame to obtain corresponding depth information, thus providing two key, interconnected perceptual outputs for the geometric enhancement from 2D to 3D. Next, based on the 2D detection results and depth information, a pre-trained object center query generator is used to back-project the 2D bounding box center and depth values ​​to construct a 3D query for the object center. This generates a geometrically sound 3D query with a clear spatial location, effectively overcoming the inherent ambiguity of directly inferring 3D position from 2D appearance features. Finally, the 3D query is input into a preset 3D detection head, decoded, and the final 3D object detection result for the current frame is output, completing the end-to-end inference from multi-view 2D images to accurate 3D detection. Therefore, this application proposes a unified framework that integrates heterogeneous 2D and 3D information through explicit spatiotemporal alignment, enhances geometric perception, and utilizes historical information to recover occluded targets. This significantly improves the localization accuracy, target recall, and spatiotemporal perception consistency of surround-view 3D detection in dynamic and occluded scenarios. Furthermore, it addresses the technical problems existing in current surround-view 3D target detectors, which rely excessively on 2D detection information, leading to inaccuracies in the spatial dimension and breaks in the temporal dimension of 2D and 3D feature representations. This, in turn, causes a decrease in localization accuracy, missed detection of occluded targets, and spatiotemporal perception in complex dynamic scenes.

[0031] Optionally, the spatiotemporal target prior propagator includes a 2D detection propagation branch and a 3D detection projection branch set in parallel. Furthermore, the operation of reprojecting the 2D and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using the pre-trained spatiotemporal target prior propagator to generate a spatiotemporal target prior map includes: extracting the appearance features of 2D and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch respectively, and reprojecting the 2D and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix; encoding the category information of each reprojected detection box through an encoder and fusing it with the extracted corresponding appearance features to generate object embeddings for each detection box; injecting each object embedding into the spatial mask of the corresponding projection box region of the feature map to generate semantic prior maps based on the output of the 2D detection propagation branch and the output of the 3D detection projection branch respectively, which together serve as the spatiotemporal target prior map.

[0032] In this embodiment, considering that in multi-view temporal 3D object detection, the detection results verified in the previous frame usually have geometric reliability and semantic stability, the core idea of ​​the spatiotemporal object prior propagator (hereinafter referred to as STOPP) is not to rely on implicit feature propagation or global fusion, but to explicitly reuse reliable historical detection results and transform them into structured spatiotemporal object priors on the current frame image plane through temporal propagation, projection, and semantic injection. This design enables the detector to focus on consistent foreground regions across time and suppress background noise. As shown in Figure 4, STOPP includes a 2D detection propagation branch and a 3D detection projection branch set in parallel. The 2D detection propagation branch performs temporal transformation using depth information from the prior frame, while the 3D detection projection branch reprojects the target localization onto the 2D plane, thereby achieving cross-modal consistency and eliminating localization differences.

[0033] Let there be a set of high-confidence 2D detection results for historical frames (frame t-1). With 3D inspection result set They are defined as follows: (1) (2) Where M represents the number of 2D bounding boxes and N represents the number of 3D bounding boxes. It is the i-th 2D bounding box, whose position is constrained within a specific image region. It is the j-th 3D bounding box. , , Here are the center coordinates, w, h, and l represent the width, length, and height of the 3D bounding box, and θ is the yaw angle of the 3D bounding box. , It is the speed of the 3D bounding box. It is the category label of the i-th 2D bounding box. It is the category label of the j-th 3D bounding box. ∈ , represents the RoI alignment appearance feature of the i-th 2D bounding box. ∈ , represents the RoI alignment appearance feature of the j-th 3D bounding box; where y represents a fixed length.

[0034] Furthermore, the operation of the spatiotemporal target prior propagator to generate the spatiotemporal target prior map specifically includes the following four stages: Stage 1: Feature extraction.

[0035] Specifically, given a historical frame (i.e., frame t-1), the multi-view feature map C, and These represent the number of channels, height, and width of the feature map, respectively. For each bounding box in both branches, the RoIAlign operation is used to extract its appearance features. : (3) For each 3D detection box, a geometric projection operator is also required. (i.e., the camera projection matrix) projects it onto the image plane of the historical frames to obtain the 2D projection area. Then, RoIAlign is executed in this area to extract its appearance features. : (4) Among them, This represents the projection matrix of the LiDAR onto the image, which combines the camera's intrinsic and extrinsic parameters at frame t-1.

[0036] Phase Two: Temporal Geometric Alignment and Reprojection.

[0037] This stage aims to align historical detection boxes with the current frame (frame t). That is, to maintain temporal consistency, historical 3D and 2D detection boxes are aligned with the current frame using self-motion and camera parameters.

[0038] For the 2D detection propagation branch: utilize the predicted depth map from historical frames. Sample each 2D detection box ( Depth value at the center Combining the self-motion matrix from frame t-1 to frame t A pseudo-3D center is constructed and time-propagated and reprojected onto the current frame image plane to obtain the reprojected 2D detection box. : (5) Among them, This is the projection matrix of the LiDAR to the image obtained by combining the intrinsic and extrinsic parameters of the camera at frame t.

[0039] For the 3D detection projection branch: directly utilize the self-motion matrix. and camera projection matrix , 3D detection box Transform and project the image onto the current frame plane to obtain the reprojected 3D detection box. : (6) Stage 3: Object semantic encoding.

[0040] Specifically, for each time-aligned projection bounding box (including those from the 2D detection propagation branch and the 3D detection projection branch), its category information is encoded using the encoder CatEnc and fused with the appearance features extracted in stage one to generate a compact object embedding. (7) Among them, Represents the appearance characteristics of object o. Embedded into the generated object.

[0041] It should be noted that all projected bounding boxes follow the same visibility principle, discarding bounding boxes outside the field of view.

[0042] Phase 4: Generation of the spatiotemporal target prior map.

[0043] Specifically, embedding for each object In its corresponding projection frame area Above, at a size of Constructing spatial prior masks on feature maps To embed the object into the area covered by the injection mask: (8) Where (i, j) represents the spatial location on the feature map.

[0044] Subsequently, the above operations are performed on all objects processed by the 2D detection propagation branch and the 3D detection projection branch, respectively, and a unified semantic prior graph of multi-view objects is generated through normalization aggregation: (9) Where ⊗ represents the cross product, The general formula for generating semantic prior maps of multi-view objects is given. This unified process equation (7), (8), and (9) is applied to obtain prior maps of two types of spatial-temporal objects. and These correspond to the semantic prior map output by the 2D detection propagation branch and the semantic prior map output by the 3D detection projection branch, respectively. Together, they constitute the spatiotemporal target prior map, which is used to guide subsequent detection.

[0045] Thus, the spatiotemporal target prior propagator explicitly establishes a geometric consistency association between historical 2D / 3D detections and the current frame image plane through the complete computational process defined by formulas (1)-(9), and transforms the semantic and appearance information in historical detections into spatially structured prior feature maps. This mechanism not only provides rich context for the current frame, enhancing the model's ability to perceive occluded targets, but also effectively bridges the gap between 2D and 3D representations through unified representation on the image plane, laying an alignment foundation for subsequent fusion and detection.

[0046] It should be noted that the Spatiotemporal Target Prior Propagator (STOPP) proposed in this application not only suppresses background interference and achieves efficient target localization, but also uses time cues to mitigate the effects of occlusion and eliminate the differences between 2D and 3D detection information, thus achieving cross-modal alignment.

[0047] Figure 5 illustrates an overview of the principle of the spatiotemporal target prior propagator. Referring to Figure 5, the spatiotemporal target prior propagator injects spatial representations from 3D space and rich semantic context into the current 2D features through 3D-to-2D and 2D-to-2D temporal interactions, enhancing the consistency between 2D-based queries and 3D spatial representations. Simultaneously, historical detection results guide the model to focus on targets that may be occluded.

[0048] Optionally, the operation of extracting the appearance features of 2D and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch respectively, and reprojecting the 2D and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix, includes: performing the following steps through the 2D detection propagation branch: for each 2D detection box in the 2D detection results of the historical frame, extracting the appearance features of the 2D detection box from the historical frame image features using the RoIAlign operation; sampling the depth of the center point of the 2D detection box from the corresponding historical frame depth map. Depth value; Based on the center point depth value, combined with the self-motion matrix, the 2D detection box is lifted and propagated into a pseudo-3D center point located in the current frame coordinate system, and the pseudo-3D center point is reprojected onto the current frame image plane; The following steps are performed through the 3D detection projection branch: For each 3D detection box in the 3D detection results of the historical frame, the appearance features of the 3D detection box are extracted from the historical frame image features using the RoIAlign operation; The self-motion matrix and camera projection matrix are applied to transform the 3D detection box to the current frame coordinate system and project it onto each view image plane of the current frame.

[0049] In this embodiment, the mathematical essence of the steps performed by the 2D detection propagation branch is the process described by the above formula (5). This process first enhances 2D pixels into 3D points through depth sampling and backprojection, then achieves temporal motion compensation through the self-motion matrix, and finally reprojects it onto the current frame image plane. This quantitative transformation ensures the accurate transmission of historical 2D information in space and time. Furthermore, the core of the steps performed by the 3D detection projection branch is the operation described by formula (6). This operation directly performs rigid body transformation and perspective projection on the detection box that already has a three-dimensional geometric definition, thereby mapping it onto the multi-view image of the current frame.

[0050] Thus, the two branches, through different geometric paths (2D→3D→2D and 3D→3D→2D), both achieve strict spatial alignment between historical detection information and the current frame image plane in precise mathematical forms (corresponding to formulas (5) and (6) above, respectively). The 2D detection propagation branch focuses on reconstructing the target position using single-point depth and is sensitive to depth estimation; the 3D detection projection branch relies on the overall geometric accuracy of the historical 3D bounding box. The complementarity of the two is fused through a unified prior graph generation framework (formulas (7)-(9) above), jointly enhancing the robustness and integrity of the spatiotemporal context.

[0051] Optionally, the spatiotemporal geometry encoder includes a depth estimation network, a depth-aware geometric attention module, and an adaptive threshold gate. Furthermore, the operation of fusing the spatiotemporal target prior map with the multi-view image features of the current frame using the pre-trained spatiotemporal geometry encoder to output a spatiotemporal geometric feature map includes: estimating the depth of the multi-view image of the current frame using the depth estimation network to obtain depth distribution information, and generating a geometric prior based on the depth distribution information and image plane coordinates; fusing the geometric prior, the multi-view image features of the current frame, the semantic prior map output by the 2D detection propagation branch, and the semantic prior map output by the 3D detection projection branch using the depth-aware geometric attention module; and filtering the fused features using the adaptive threshold gate to output the spatiotemporal geometric feature map.

[0052] In this embodiment, as shown in Figure 6, the spatiotemporal geometric encoder (STGE) aims to deeply integrate multi-view image features with spatiotemporal target priors through depth-aware geometric modeling and cross-modal attention fusion, so as to output an enhanced, geometrically aware spatiotemporal geometric feature map, which improves the localization and occlusion recovery effect and provides a high-quality representation for subsequent 2D detection and 3D query generation.

[0053] Specifically, the operation process of the Spatiotemporal Geometric Encoder (STGE) includes the following core steps: Step 1: Depth estimation and geometric prior generation.

[0054] Specifically, given a multi-view image of the current frame t, it is first processed by a lightweight depth estimation network (DepthNet). This network uses the features of the multi-view image... As input, the system regresses the depth value and probability distribution of each pixel, and then performs attention-based temporal depth aggregation to finally generate a pixel-by-pixel depth map. (Training is supervised by LiDAR depth signals). Based on depth maps. By combining the inherent pixel coordinates (u, v) of the image, two geometric priors are generated: an image planar coordinate prior and a depth distance prior. These two priors are combined into a geometric prior tensor x, with the shape of... The geometric prior x encodes the three-dimensional spatial relationships at each location in the image.

[0055] Step 2: Depth-perception geometric attention fusion.

[0056] This step is the core of the Spatiotemporal Geometric Encoder (STGE), aiming to deeply fuse geometric priors, visual features, and spatiotemporal priors. A depth-aware geometric attention mechanism (i.e., the depth-aware geometric attention module GeoAtten) is employed, and its computation process is defined as follows: (10) Where Q, K, V are query, key and value matrices obtained by linear projection of input features, ⊙ represents element-wise multiplication, and β∈ (0,1) is a learnable decay rate parameter used to control the influence of geometric prior x on attention weights.

[0057] This attention mechanism is applied in parallel to two feature paths: one is the multi-view image feature of the current frame. The other approach is the combined spatiotemporal target prior, namely the 2D semantic prior generated by STOPP. With 3D semantic prior graph The two feature streams are superimposed. After initial processing by independent convolutional layers, they are input into their respective GeoAtten modules: (11) (12) Thus, the geometrically enhanced image features are obtained. with prior features .

[0058] Step 3: Adaptive threshold filtering and feature map generation.

[0059] Specifically, to further highlight the foreground target region and suppress background noise, this application introduces an Adaptive Threshold Gate (ATG). This module utilizes two learnable parameters γ (controlling the threshold offset) and τ (controlling the mask smoothness) to generate a spatial mask based on the fused features, which is used to weight and emphasize high-confidence regions. The final spatiotemporal geometric feature map... It is generated by the following formula: (13) Where, σ It is the Sigmoid function.

[0060] Thus, the spatiotemporal geometric encoder achieves deep and adaptive fusion of multi-source information through the complete computational process defined by formulas (10)-(13). Its innovation lies in: 1) Depth-guided geometric modeling: By explicitly estimating the depth and constructing geometric priors, crucial three-dimensional spatial structure information is injected into 2D features, alleviating the ambiguity of pure 2D representation in dimensions such as depth and scale; 2) Cross-modal attention fusion: By utilizing a learnable geometric attention mechanism, the correlation between image features and spatiotemporal prior features is dynamically established, realizing the organic unity of semantics, appearance, and historical geometric context at the feature level; 3) Adaptive spatial selection: By using an adaptive threshold gate to perform soft spatial filtering on the fused features, the model can focus on regions more likely to contain targets, enhancing the discriminative power and robustness of the feature map. The final output spatiotemporal geometric feature map is a high-quality feature representation with rich appearance details, accurate geometric structure perception, and strong spatiotemporal context, laying a solid foundation for subsequent accurate 2D detection and reliable 3D query generation.

[0061] Optionally, the operation of performing 2D object detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain the corresponding depth information includes: inputting the spatiotemporal geometric feature map into a 2D detection head, predicting 2D bounding boxes, the class confidence corresponding to the 2D bounding boxes, and the object embedding corresponding to the 2D bounding boxes as the 2D detection result; and processing the multi-view image of the current frame through an independent depth estimation network to regress pixel-level depth values ​​to form the depth information.

[0062] In this embodiment, two parallel subtasks are included: decoding 2D detection results from enhanced spatiotemporal geometric features, and estimating dense depth information from the original image.

[0063] First, regarding 2D object detection: the spatiotemporal geometric feature map generated by the spatiotemporal geometric encoder is input into a 2D detection head. This detection head is typically a lightweight network based on convolutional or Transformer structures. Its core function is to parse dense features that fuse spatiotemporal context and geometric priors, and output a series of 2D detection proposals on each view. For each detected object, the detection head will simultaneously predict three key outputs: 1) a 2D bounding box parameter (usually center coordinates, width, and height) to locate the object's position in the image; 2) a class confidence vector representing the probability that the box belongs to each predefined class (e.g., vehicle, pedestrian); 3) a high-dimensional object embedding vector, which is derived from... The corresponding regions are extracted and further encoded to encapsulate the target's visual appearance, semantic category, and spatiotemporal context. All these outputs together constitute the accurate and information-rich 2D detection results for the current frame, providing direct observational evidence for subsequent geometric upscaling from 2D to 3D.

[0064] Secondly, regarding depth estimation: Depth information is acquired through a dedicated depth estimation network (DepthNet). This network takes the raw multi-view image of the current frame or primary image features extracted by the backbone network as input. Its design goal is to establish a mapping from 2D images to 3D geometry, specifically by regressing the depth value for each pixel or feature location. During training, the network is supervised by sparse but accurate ground truth depth values ​​generated from LiDAR point cloud projections, learning to predict a dense depth map. This process can be formally represented as: given the input image I, network parameters θ, the depth estimation network outputs a depth map. =DepthNet(I;θ). The resulting depth information is a two-dimensional data field aligned with the input image space, containing relative or absolute distances between points in the scene. Although a depth estimation network was already used in the STGE module to generate geometric priors, the network used to generate the depth information here can be the same network sharing weights, or it can be a structurally identical but independently trained branch to ensure that specialized and reliable depth values ​​are provided for subsequent 3D query generation.

[0065] Thus, this step achieves the decoupling and refinement of perceptual information. On one hand, it utilizes highly fused and enhanced spatiotemporal geometric feature maps to perform accurate 2D perception tasks, achieving results superior to those obtained solely from raw image features. On the other hand, it performs parallel low-level estimation of the scene's 3D structure, generating dense depth information. These two outputs—2D detection results representing "what is and where" and depth information representing "how far"—constitute the two necessary and complementary modalities for subsequent 3D querying of the target center, providing a crucial data interface for the entire method to achieve accurate 3D spatial reasoning.

[0066] Optionally, the target center query generator includes a feature extraction module and a coordinate backprojection fusion module; and the operation of constructing a 3D query of the target center based on the 2D detection results and the depth information using a pre-trained target center query generator includes: using the feature extraction module to extract the object embedding and image coordinates corresponding to each 2D detection box in the 2D detection results; taking the center point of the 2D detection box as the target center and obtaining the depth value corresponding to the target center from the depth information; using the coordinate backprojection fusion module, using camera intrinsic parameters, backprojecting the image coordinates and corresponding depth value of the target center to three-dimensional space to obtain the candidate three-dimensional center coordinates corresponding to the target center, and fusing the object embedding and the candidate three-dimensional center coordinates to generate the 3D query of the target center.

[0067] In this embodiment, the core function of the target center query generator (hereinafter referred to as OQG) is to combine the accurate 2D detection results in the image coordinate system with geometric depth information, and generate a target center 3D query with clear three-dimensional spatial orientation through explicit geometric back projection and feature fusion, so as to provide high-quality initialization input for the subsequent 3D detection head.

[0068] Specifically, the operation process of the Target Center Query Generator (OQG) strictly follows the principle of combining geometric constraints and feature learning. Its detailed steps are as follows: Step 1: Input and feature acquisition.

[0069] The input to OQG consists of two parts: 1) the 2D detection result of the current frame obtained from the aforementioned steps, wherein for each detected 2D bounding box, its bounding box parameters, class confidence, and a high-dimensional object embedding have been obtained simultaneously. ;2) Depth information D of the current frame tFirst, the feature extraction module extracts two key pieces of information from each 2D detection bounding box: one is the object embedding of the box, which encodes the appearance and semantics of the target; the other is the image coordinates of the box, typically represented by the center point coordinates (u,v) of its bounding box, which is defined as the projection of the target's center onto the image. Simultaneously, the object embedding... Using the camera's intrinsic parameter matrix Generate coarse position code , where M represents the number of 2D bounding boxes.

[0070] Step 2: Depth Association and 3D Coordinate Candidate Generation. For each target center defined by image coordinates (u,v), query the depth information D. t The depth value d at the corresponding location is then used by the coordinate back-projection fusion module to determine the depth value d at the camera's intrinsic parameter matrix. Perform backprojection calculations to transform the image coordinates (u,v) and depth value d into candidate 3D center coordinates in the camera coordinate system. .

[0071] Step 3: Optimize the location encoding generation. This involves optimizing the coarse location encoding... With candidate 3D center coordinates By fusing through a linear layer, a more accurate optimized positional code is generated: (14) Among them, The final optimized positional encoding serves two purposes. First, reliable 2D detection results provide correction clues for pseudo-point clouds. Second, with the help of positional encoding, optimized 3D points can be generated, thereby suppressing error propagation into the 3D detection pipeline.

[0072] Step 4: 3D query vector generation. Specifically, this involves obtaining optimized positional encoding. Then, OQG embeds it with the object corresponding to the 2D detection box. These are combined to form a 3D query for the i-th target. The location encoding is included. As the positional encoding component of this 3D query, it provides the decoder with a precise initial anchor point in three-dimensional space; object embedding This serves as the content feature component of the 3D query, carrying both the visual appearance and semantic information of the target. This combination method enables the 3D query to possess both clear geometric orientation and rich recognition features, thus becoming a high-quality input for subsequent 3D decoders to refine bounding boxes and perform attribute regression.

[0073] Thus, the target center query generator achieves a reliable conversion from 2D perception to 3D structured query through a coherent and parsable process of "extracting object embeddings and coordinates → depth back-projection to obtain candidate 3D centers → linear fusion to generate optimized position codes → combination to construct a complete 3D query". The core advantages of this design are: explicit use of geometric constraints, directly back-projecting image coordinates and depth values ​​into three-dimensional coordinates through the camera model, providing interpretable and accurate geometric initialization for the query, fundamentally avoiding the spatial ambiguity common in end-to-end regression; feature and position depth fusion, linearly fusing semantically rich object embeddings and accurate candidate three-dimensional coordinates through the above formula (14), so that the generated optimized position codes have both semantic consistency and geometric accuracy; clear and efficient structure, the whole process is based on deterministic geometric calculations and lightweight linear transformations, without the need for complex iterations, ensuring the real-time performance of the query generation stage. Finally, the generated 3D query set serves as the direct input to the 3D decoder, providing it with a superior iterative starting point, significantly improving the 3D detection accuracy and stability of the model in occlusion, scale changes and long-distance scenes.

[0074] In summary, this application proposes a spatiotemporally aligned multi-view 3D object detection architecture (hereinafter referred to as STUR3D), which unifies the look-around 3D detection framework of 2D and 3D spatiotemporal representations through rich cross-frame and cross-modal information exchange. First, a spatiotemporal object prior propagator (STOPP) is introduced, which performs 2D-to-2D and 3D-to-2D temporal propagation. By reprojecting historical 2D and 3D detection results onto the current image plane, STOPP generates explicit spatiotemporal object priors, guiding the 2D detector to learn geometric perception representations while suppressing noise or inconsistent proposals. This enables the detector to focus on occluded targets in the current frame and reduce false alarms by utilizing stable historical evidence. Second, a spatiotemporal geometric encoder (STGE) is developed to further inject temporal and geometric cues into the 2D detection head. Utilizing a depth-aware geometric attention mechanism, STGE fuses historical 2D and 3D detection features with current frame multi-view image features and filters out unreliable foreground assumptions. This encoding process narrows the representation gap between 2D and 3D feature spaces, facilitating the extraction of discriminative representations required for accurate 3D localization in complex environments. Furthermore, an Object Center Query Generator (OQG) is introduced, leveraging depth-guided positional encoding to elevate optimized 2D detection results into object-centered 3D queries. By injecting pseudo-depth cues estimated from the current frame, OQG enhances the geometric basis of the generated 3D queries and mitigates the inherent spatial biases of relying solely on appearance-based 2D features. These components collectively constitute a unified spatiotemporal alignment pipeline, harmonizing 2D and 3D representations, improving inter-frame temporal coherence, and significantly enhancing robustness in challenging dynamic and occluded scenes. Figure 7 illustrates the STUR3D detection framework.

[0075] This application evaluates the proposed spatiotemporally aligned multi-view 3D object detection method on the large-scale nuScenes dataset, a widely used benchmark for multi-view 3D object detection in autonomous driving. It contains 1000 scenes, divided into 700 for training, 150 for validation, and 150 for testing, each scene approximately 20 seconds long and labeled at a frequency of 2Hz. The dataset provides 32-line LiDAR point clouds and six camera views covering 360 degrees. Following the official protocol, this application will report mAP and NDS, as well as mATE, mASE, mAOE, maAVE, and mAAE for detailed analysis.

[0076] This application follows previous methods, using ResNet50, ResNet101, and V2-99 as backbone networks. For supervised depth, LiDAR points are projected onto the camera view to form a downsampled ×16 sparse depth map, and DepthNet is trained on effective pixels using masked regression loss. Depth prediction is supervised by LiDAR during training but runs in vision-only mode during inference. All models are trained on four NVIDIA GeForce RTX A6000 GPUs using AdamW with a cosine annealing strategy and a learning rate of 4×1000. The batch size is 8. The model in this application is trained for 90 epochs on the nuScenes validation set and 60 epochs on the test set. CBGS, test-time augmentation, or future frames are not used. This application retains the first 128 2D detections and the first 256 3D detections per frame. In addition, this application initializes the learnable attenuation parameter β and the gating parameters γ and τ to 0.5 and provides an ablation study on this initialization. For the geometric prior x, this application generates a single-scale tensor and performs inference on the segmented set following Equation (10).

[0077] This application compares the proposed STUR3D with existing state-of-the-art multi-view 3D object detectors on the nuScenes validation and test sets. As shown in Table 1, for the nuScenes validation set, with a ResNet50 backbone network pre-trained on nuImages and an input image size of 704×256, STUR3D achieves 48.6% mAP and 57.9% NDS performance, which is 1.6% higher than the state-of-the-art method OPEN and 1.4% higher than NDS, and 3.7% higher than the baseline StreamPETR of this application and 2.9% higher than NDS. With a ResNet101 backbone network pre-trained on nuImages and an input image size of 1408×512, STUR3D achieves 53.1% mAP and 61.3% NDS performance. STUR3D outperforms the state-of-the-art method Sparse4Dv2 by 1.0% in mAP and 0.5% in NDS, becoming the new state-of-the-art method on the nuScenes validation set. Compared to the baseline, STUR3D in this application shows a significant performance improvement, with a 2.7% increase in mAP and a 2.1% increase in NDS.

[0078] Table 1

[0079] Table 1 shows a comparison with previous state-of-the-art multi-view 3D detectors on the nuScenes validation set. For a fair comparison, this application reproduces the baseline method under the same settings as the method in this application. † indicates perspective view pre-training benefiting from nuImages. This indicates the use of CBGS training, which will extend 1 epoch to 4.5 epochs. § indicates the use of future frames.

[0080] As shown in Table 6 below, STUR3D achieved 52.8% mAP and 57.8% NDS using only camera input and without any LiDAR supervision during training. Since no LiDAR is ever used as input during inference, this table only compares whether LiDAR signals were used for training supervision.

[0081] Table 2 presents the detection results on the nuScenes test set and compares the proposed framework with previous state-of-the-art methods. STUR3D achieves 57.9% mAP and 64.6% NDS performance using a V2-99 backbone network and an input image size of 640×1600. STUR3D outperforms the baseline StreamPETR by 2.8% mAP and 1.0% NDS, and outperforms the state-of-the-art method DVPE by 0.6% mAP and 0.2% NDS.

[0082] Table 2

[0083] Table 2 shows the comparison on the nuScenes test set. No other techniques (such as CBGS, extended test time) were used in the method of this application during training and testing.

[0084] To evaluate the effectiveness of STUR3D, this application conducted a series of ablation studies on the nuScenes validation set, using V2-99 as the backbone network and input image size of 320×800. STUR3D was retrained for 24 epochs, and the results are summarized in Table 3. This application observes that each introduced component brings progressive improvements. First, the proposed model (b) using only the STOPP module improves mAP by +1.6%, demonstrating the role of temporal prior propagation in stabilizing target representation. Integrating STGE (c) further improves mAP by +0.5% and NDS by +0.9%, highlighting the importance of geometrically perceptual consistency among multi-view features. Introducing OQG alone (d) results in an improvement of +2.0% in mAP and +2.1% in NDS, highlighting the importance of target-centric query generation with deep priors for stronger initialization. When combining STOPP and OQG (e), the system benefits from both temporal and geometric priors, achieving an improvement of +3.9% in mAP and +3.4% in NDS compared to the baseline. Similarly, pairing STGE with OQG (f) further optimizes the depth-guided geometry, improving the spatial basis of 3D queries. Finally, the complete STUR3D framework achieves 53.0% mAP and 61.2% NDS, which is +4.8% mAP and +4.1% NDS higher than the baseline, confirming that the integration of all proposed components leads to a significant improvement in multi-view 3D object detection.

[0085] Table 3

[0086] Table 3 presents the ablation studies of each component in STUR3D on the nuScenes validation set. STOPP, STGE, and OQG represent the spatiotemporal target prior propagator, spatiotemporal geometric encoder, and target center query generator, respectively.

[0087] The runtime and accuracy of the nuScenes value set. "STOPP" indicates the use of the STOPP module. "3Dgen." indicates the existence of a 3D query generator, where STUR3D uses OQG.

[0088] Table 4 shows the runtime and accuracy on the nuScenes validation set. This application also indicates whether the STOPP module was used and whether a 3D query generator was present; STUR3D uses OQG.

[0089] STUR3D achieved 53.0% mAP and 61.2% NDS at 7.8 FPS. Compared to MV2D (6.4 FPS, 47.2% mAP, 56.0% NDS) and DVPE (6.2 FPS, 50.5% mAP, 56.6% NDS), STUR3D offers comparable throughput while delivering significant performance improvements: +5.8% higher mAP and +5.2% higher NDS than MV2D, and +2.5% higher mAP and +4.6% higher NDS than DVPE. Compared to QAF2D (3.9 FPS), STUR3D is approximately twice as fast and more accurate. Although OPEN achieved the highest throughput at 10.3 FPS, STUR3D offered the best mAP and reached 61.2% on NDS, comparable to OPEN, demonstrating a good balance between accuracy and efficiency.

[0090] Table 4

[0091] The STOPP module exhibits strong plug-and-play characteristics. As shown in Table 4, integrating it into DVPE achieves a speed of 5.6 FPS, with mAP and NDS improving by +0.3% and +0.3%, respectively. Integrating it into OPEN results in a total speed of 8.7 FPS, mAP at 52.4%, and NDS at 60.7%. These results demonstrate that STUR3D's spatiotemporal prior propagation mechanism is a transferable module, and the overall design significantly improves detection accuracy and temporal consistency while ensuring inference efficiency.

[0092] As shown in Table 5, this application analyzes the impact of different detection inputs and the number of cached historical feature maps in STOPP. Removing 2D or 3D detection results (BC) leads to a significant decrease in mAP and NDS, indicating that the joint use of 2D and 3D priors is crucial for accurate propagation. Increasing the number of cached frames from 1 to 2 (F) maintains comparable accuracy, while expanding to 4 (G) does not bring further gains due to accumulated temporal noise.

[0093] Table 5

[0094] Table 5 shows the ablation results of the STOPP detection and the number of time-buffered frames.

[0095] As shown in Table 6, the proposed STGE consistently outperforms the linear encoder and MLP encoder across all metrics, demonstrating stronger geometric feature modeling capabilities. Under LiDAR supervision, STGE achieves a significant improvement in overall detection quality, with NDS increasing by +3.4%, reflecting improved 3D positioning accuracy.

[0096] Table 6

[0097] Table 6 compares other deep coding methods on the nuScenes validation set. LiDAR supervision indicates whether LiDAR signals are used as supervision during training; all models do not use LiDAR input during inference.

[0098] Figure 8 shows the qualitative detection results on the nuScenes validation set for multi-view images and the BEV space. On the multi-view images, the 3D predicted bounding boxes of different methods are displayed in different colors. Blue represents the baseline, green represents QAF2D, and red represents the predictions of this application. The yellow bounding boxes on the BEV map represent the ground truth values.

[0099] As shown in Figure 8, the visualization results on the nuScenes validation set demonstrate that STUR3D achieves significant improvements over the baseline and QAF2D in various challenging scenarios. In the right front and right back images, STUR3D achieves significantly higher localization accuracy, with predicted bounding boxes more closely aligned with ground truth values ​​in both image and BEV space. In the left front image, STUR3D exhibits stronger detection capabilities, successfully identifying targets completely missed by the baseline. In the front and back images, STUR3D effectively recovers severely occluded or partially hidden targets, while the baseline and QAF2D fail. This improvement stems from the spatiotemporal propagation structure, which utilizes information from the previous frame to maintain temporal consistency and recover occluded targets. These results validate that STUR3D not only improves detection completeness and localization accuracy but also demonstrates superior robustness and stability in occluded environments.

[0100] Therefore, the proposed spatiotemporally aligned multi-view 3D object detection architecture (STUR3D) is a spatiotemporally unified representation learning framework for camera-based 3D object detection. By propagating aligned spatiotemporally historical 3D and 2D detection results to the image plane through Stopp, injecting geometric and depth-aware cues using STGE, and elevating optimized 2D detection results to a 3D query for object centers through OQG, STUR3D explicitly aligns heterogeneous 2D and 3D spatiotemporal representations, guiding the 2D detector to extract the representations required for 3D detection. Extensive experiments on the nuScenes benchmark demonstrate that STUR3D significantly outperforms robust query-based baselines across various backbone networks and input resolutions, achieving 57.9% mAP and 64.6% NDS on the test set, while maintaining a good accuracy-efficiency tradeoff and robustness to occlusion. Ablation studies further validate the contributions of each component, the plug-and-play nature of Stopp, and the effectiveness of the depth-aware geometric modeling proposed in this application under LiDAR supervision and visual-only depth settings.

[0101] Furthermore, referring to FIG2, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0104] Figure 9 illustrates a spatiotemporally aligned multi-view 3D target detection device according to this embodiment, which corresponds to the method described in Embodiment 1. Referring to Figure 9, the device includes: a multi-view image receiving module 910, used to receive multi-view images of the current frame, wherein the multi-view images are a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different viewpoints; a spatiotemporal target prior map generation module 920, used to reproject 2D and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using a pre-trained spatiotemporal target prior propagator, to generate a spatiotemporal target prior map; and a spatiotemporal geometric feature map generation module 930, used to fuse the spatiotemporal target prior using a pre-trained spatiotemporal geometric encoder. The system uses the multi-view image features of the current frame to output a spatiotemporal geometric feature map; a 2D detection result and depth information determination module 940 is used to perform 2D target detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and to perform depth estimation on the multi-view image of the current frame to obtain the corresponding depth information; a 3D query construction module 950 is used to construct a 3D query of the target center based on the 2D detection result and the depth information through a pre-trained target center query generator; and a 3D target detection result determination module 960 is used to input the 3D query into a preset 3D detection head and output the 3D target detection result of the current frame.

[0105] Optionally, the spatiotemporal target prior propagator includes a 2D detection propagation branch and a 3D detection projection branch set in parallel; and the spatiotemporal target prior map generation module 920 is specifically used for: extracting the appearance features of 2D detection boxes and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch respectively, and reprojecting the 2D detection boxes and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix; encoding the category information of each detection box after reprojection through an encoder, and fusing it with the extracted corresponding appearance features to generate object embeddings for each detection box; injecting each object embedding into the spatial mask of the corresponding projection box region of the feature map, and generating semantic prior maps based on the output of the 2D detection propagation branch and the output of the 3D detection projection branch respectively, which together serve as the spatiotemporal target prior map.

[0106] It should be noted that the spatiotemporal alignment-based multi-view 3D target detection device provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0107] According to this embodiment, the first step is to receive a multi-view image of the current frame. This multi-view image is a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different perspectives, providing basic observation data for subsequent cross-view spatiotemporal alignment and 3D inference. Then, through a pre-trained spatiotemporal target prior propagator, the 2D and 3D detection results from at least one historical frame are reprojected onto the view image planes of the current frame based on camera geometry, generating a spatiotemporal target prior map. This explicitly injects historically verified, geometrically reliable spatial structure and temporal dynamic information into the 2D perception layer of the current frame, providing a structured prior for establishing cross-modal and cross-temporal feature consistency. Next, through a pre-trained spatiotemporal geometry encoder, the spatiotemporal target prior map is fused with image features extracted from the multi-view image of the current frame, and depth-sensing geometric information is incorporated to output a spatiotemporal geometric feature map. This achieves depth interaction and alignment between historical priors and current observations at the feature level, enhancing the model's ability to perceive complex spatial structures (such as depth and direction). Subsequently, 2D object detection is performed on the spatiotemporal geometric feature map to obtain accurate 2D detection results for the current frame. Simultaneously, depth estimation is performed on the multi-view image of the current frame to obtain corresponding depth information, thus providing two key, interconnected perceptual outputs for the geometric enhancement from 2D to 3D. Next, based on the 2D detection results and depth information, a pre-trained object center query generator is used to back-project the 2D bounding box center and depth values ​​to construct a 3D query for the object center. This generates a geometrically sound 3D query with a clear spatial location, effectively overcoming the inherent ambiguity of directly inferring 3D position from 2D appearance features. Finally, the 3D query is input into a preset 3D detection head, decoded, and the final 3D object detection result for the current frame is output, completing the end-to-end inference from multi-view 2D images to accurate 3D detection. Therefore, this application proposes a unified framework that integrates heterogeneous 2D and 3D information through explicit spatiotemporal alignment, enhances geometric perception, and utilizes historical information to recover occluded targets. This significantly improves the localization accuracy, target recall, and spatiotemporal perception consistency of surround-view 3D detection in dynamic and occluded scenarios. Furthermore, it addresses the technical problems existing in current surround-view 3D target detectors, which rely excessively on 2D detection information, leading to inaccuracies in the spatial dimension and breaks in the temporal dimension of 2D and 3D feature representations. This, in turn, causes a decrease in localization accuracy, missed detection of occluded targets, and spatiotemporal perception in complex dynamic scenes.

[0108] Figure 10 illustrates a spatiotemporally aligned multi-view 3D target detection device according to this embodiment, which corresponds to the method described in Embodiment 1. Referring to Figure 10, the device includes: a processor 1010; and a memory 1020 connected to the processor 1010, used to provide the processor 1010 with instructions to process the following steps: receiving a multi-view image of the current frame, wherein the multi-view image is a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different viewpoints; and using a pre-trained spatiotemporal target prior propagator, reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry, to generate a spatiotemporal target prior propagation. The process involves: verifying the spatiotemporal target prior image; fusing the spatiotemporal target prior image with the multi-view image features of the current frame using a pre-trained spatiotemporal geometric encoder to output a spatiotemporal geometric feature map; performing 2D target detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain the corresponding depth information; based on the 2D detection result and the depth information, constructing a 3D query for the target center using a pre-trained target center query generator; inputting the 3D query into a preset 3D detection head to output the 3D target detection result of the current frame.

[0109] It should be noted that the spatiotemporal alignment-based multi-view 3D target detection device provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0110] According to this embodiment, the first step is to receive a multi-view image of the current frame. This multi-view image is a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different perspectives, providing basic observation data for subsequent cross-view spatiotemporal alignment and 3D inference. Then, through a pre-trained spatiotemporal target prior propagator, the 2D and 3D detection results from at least one historical frame are reprojected onto the view image planes of the current frame based on camera geometry, generating a spatiotemporal target prior map. This explicitly injects historically verified, geometrically reliable spatial structure and temporal dynamic information into the 2D perception layer of the current frame, providing a structured prior for establishing cross-modal and cross-temporal feature consistency. Next, through a pre-trained spatiotemporal geometry encoder, the spatiotemporal target prior map is fused with image features extracted from the multi-view image of the current frame, and depth-sensing geometric information is incorporated to output a spatiotemporal geometric feature map. This achieves depth interaction and alignment between historical priors and current observations at the feature level, enhancing the model's ability to perceive complex spatial structures (such as depth and direction). Subsequently, 2D object detection is performed on the spatiotemporal geometric feature map to obtain accurate 2D detection results for the current frame. Simultaneously, depth estimation is performed on the multi-view image of the current frame to obtain corresponding depth information, thus providing two key, interconnected perceptual outputs for the geometric enhancement from 2D to 3D. Next, based on the 2D detection results and depth information, a pre-trained object center query generator is used to back-project the 2D bounding box center and depth values ​​to construct a 3D query for the object center. This generates a geometrically sound 3D query with a clear spatial location, effectively overcoming the inherent ambiguity of directly inferring 3D position from 2D appearance features. Finally, the 3D query is input into a preset 3D detection head, decoded, and the final 3D object detection result for the current frame is output, completing the end-to-end inference from multi-view 2D images to accurate 3D detection. Therefore, this application proposes a unified framework that integrates heterogeneous 2D and 3D information through explicit spatiotemporal alignment, enhances geometric perception, and utilizes historical information to recover occluded targets. This significantly improves the localization accuracy, target recall, and spatiotemporal perception consistency of surround-view 3D detection in dynamic and occluded scenarios. Furthermore, it addresses the technical problems existing in current surround-view 3D target detectors, which rely excessively on 2D detection information, leading to inaccuracies in the spatial dimension and breaks in the temporal dimension of 2D and 3D feature representations. This, in turn, causes a decrease in localization accuracy, missed detection of occluded targets, and spatiotemporal perception in complex dynamic scenes.

[0111] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-view 3D target detection method based on spatiotemporal alignment, characterized in that, include: Receive the multi-view image of the current frame, wherein the multi-view image is a set of images covering a preset panoramic range, which are simultaneously acquired by multiple cameras with different perspectives; A pre-trained spatiotemporal target prior propagator reprojects 2D and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry, generating a spatiotemporal target prior map. A pre-trained spatiotemporal geometry encoder fuses the spatiotemporal target prior map with the multi-view image features of the current frame, outputting a spatiotemporal geometry feature map. 2D target detection is performed on the spatiotemporal geometry feature map to obtain the 2D detection results of the current frame, and depth estimation is performed on the multi-view images of the current frame to obtain corresponding depth information. Based on the 2D detection results and the depth information, a pre-trained target center query generator constructs a 3D query for the target center. The 3D query is input into a preset 3D detection head, outputting the 3D target detection results of the current frame.

2. The method according to claim 1, characterized in that, The spatiotemporal target prior propagator includes a parallel 2D detection propagation branch and a 3D detection projection branch. The operation of reprojecting 2D and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using the pre-trained spatiotemporal target prior propagator to generate a spatiotemporal target prior map includes: extracting the appearance features of 2D and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch respectively, and reprojecting the 2D and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix; encoding the category information of each reprojected detection box through an encoder and fusing it with the extracted corresponding appearance features to generate object embeddings for each detection box; injecting each object embedding into the spatial mask of the corresponding projection box region of the feature map to generate semantic prior maps based on the output of the 2D detection propagation branch and the 3D detection projection branch respectively, which together serve as the spatiotemporal target prior map.

3. The method according to claim 2, characterized in that, The operation of extracting the appearance features of 2D and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch respectively, and reprojecting the 2D and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix, includes: performing the following steps through the 2D detection propagation branch: for each 2D detection box in the 2D detection results of the historical frame, extracting the appearance features of the 2D detection box from the historical frame image features using the RoIAlign operation; sampling the depth value of the center point of the 2D detection box from the corresponding historical frame depth map. Based on the center point depth value and combined with the self-motion matrix, the 2D detection box is lifted and propagated into a pseudo-3D center point located in the current frame coordinate system, and the pseudo-3D center point is reprojected onto the current frame image plane; the following steps are performed through the 3D detection projection branch: for each 3D detection box in the 3D detection results of the historical frame, the appearance features of the 3D detection box are extracted from the historical frame image features using the RoIAlign operation; the self-motion matrix and camera projection matrix are applied to transform the 3D detection box to the current frame coordinate system and project it onto each view image plane of the current frame.

4. The method according to claim 2, characterized in that, The spatiotemporal geometric encoder includes a depth estimation network, a depth-aware geometric attention module, and an adaptive threshold gate. The operation of fusing the spatiotemporal target prior map with the multi-view image features of the current frame using the pre-trained spatiotemporal geometric encoder to output a spatiotemporal geometric feature map includes: estimating the depth of the multi-view image of the current frame using the depth estimation network to obtain depth distribution information, and generating a geometric prior based on the depth distribution information and image plane coordinates; fusing the geometric prior, the multi-view image features of the current frame, the semantic prior map output by the 2D detection propagation branch, and the semantic prior map output by the 3D detection projection branch using the depth-aware geometric attention module; and filtering the fused features using the adaptive threshold gate to output the spatiotemporal geometric feature map.

5. The method according to claim 1, characterized in that, The operation of performing 2D object detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain the corresponding depth information includes: inputting the spatiotemporal geometric feature map into a 2D detection head to predict 2D bounding boxes, the class confidence corresponding to the 2D bounding boxes, and the object embedding corresponding to the 2D bounding boxes as the 2D detection result; and processing the multi-view image of the current frame through an independent depth estimation network to regress pixel-level depth values ​​to form the depth information.

6. The method according to claim 1, characterized in that, The target center query generator includes a feature extraction module and a coordinate back-projection fusion module. Furthermore, the operation of constructing a 3D query of the target center based on the 2D detection results and the depth information using a pre-trained target center query generator includes: using the feature extraction module to extract the object embedding and image coordinates corresponding to each 2D detection box in the 2D detection results; using the center point of the 2D detection box as the target center and obtaining the depth value corresponding to the target center from the depth information; using the coordinate back-projection fusion module, using camera intrinsic parameters, back-projecting the image coordinates and corresponding depth value of the target center to three-dimensional space to obtain the candidate three-dimensional center coordinates corresponding to the target center, and fusing the object embedding and the candidate three-dimensional center coordinates to generate the 3D query of the target center.

7. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, a processor performs the method according to any one of claims 1 to 6.

8. A multi-view 3D target detection device based on spatiotemporal alignment, characterized in that, include: A multi-view image receiving module is used to receive multi-view images of the current frame, wherein the multi-view images are a set of images covering a preset panoramic range, synchronously acquired by multiple cameras with different viewpoints; a spatiotemporal target prior map generation module is used to generate a spatiotemporal target prior map by reprojecting 2D detection results and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using a pre-trained spatiotemporal target prior propagator; a spatiotemporal geometric feature map generation module is used to output a spatiotemporal geometric feature map by fusing the spatiotemporal target prior map with the multi-view image features of the current frame using a pre-trained spatiotemporal geometric encoder; a 2D detection result and depth information determination module is used to perform 2D target detection on the spatiotemporal geometric feature map to obtain the 2D detection result of the current frame, and to perform depth estimation on the multi-view image of the current frame to obtain the corresponding depth information; The 3D query construction module is used to construct a 3D query of the target center based on the 2D detection results and the depth information using a pre-trained target center query generator; the 3D target detection result determination module is used to input the 3D query into a preset 3D detection head and output the 3D target detection result of the current frame.

9. The multi-view 3D target detection device according to claim 8, characterized in that, The spatiotemporal target prior propagator includes a 2D detection propagation branch and a 3D detection projection branch set in parallel. The spatiotemporal target prior map generation module is specifically used for: extracting the appearance features of 2D and 3D detection boxes from historical frames through the 2D detection propagation branch and the 3D detection projection branch, respectively; reprojecting the 2D and 3D detection boxes onto the current frame image plane based on camera geometry and self-motion matrix; encoding the category information of each reprojected detection box through an encoder and fusing it with the extracted corresponding appearance features to generate object embeddings for each detection box; injecting each object embedding into the spatial mask of the corresponding projection box region of the feature map, generating semantic prior maps based on the output of the 2D detection propagation branch and the output of the 3D detection projection branch, respectively, which together serve as the spatiotemporal target prior map.

10. A multi-view 3D target detection device based on spatiotemporal alignment, characterized in that, include: processor; The system includes a memory connected to the processor, for providing the processor with instructions to process the following steps: receiving a multi-view image of the current frame, wherein the multi-view image is a set of images covering a preset panoramic range, simultaneously acquired by multiple cameras from different perspectives; generating a spatiotemporal target prior map by reprojecting 2D and 3D detection results from at least one historical frame onto the view image planes of the current frame based on camera geometry using a pre-trained spatiotemporal target prior propagator; fusing the spatiotemporal target prior map with the multi-view image features of the current frame using a pre-trained spatiotemporal geometry encoder; performing 2D target detection on the spatiotemporal geometry feature map to obtain the 2D detection result of the current frame, and performing depth estimation on the multi-view image of the current frame to obtain corresponding depth information; constructing a 3D query for the target center using a pre-trained target center query generator based on the 2D detection result and the depth information; inputting the 3D query into a preset 3D detection head and outputting the 3D target detection result of the current frame.

Images