End-to-end bev-aware information processing method and electronic device

Abstract

The application discloses an end-to-end Bev perception information processing method and an electronic device. The end-to-end Bev perception information processing method comprises the following steps: performing three-dimensional detection on current picture information corresponding to a camera angle pre-set on a main body to obtain a candidate combination of three-dimensional frames; performing two-dimensional detection on the current picture information to obtain position information corresponding to each target in the current picture information; obtaining time sequence information between a target instance of a previous frame of the current picture information and the current picture information; obtaining initial query information of the current picture based on the candidate combination of the three-dimensional frames, the position information and the time sequence information; and obtaining an end-to-end Bev perception information processing result based on the initial query information. The embodiment of the application can simplify the calculation complexity.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and more specifically, relates to an end-to-end Bev-aware information processing method and electronic device. Background Technology

[0002] Autonomous vehicles require environmental perception. Generally, autonomous vehicles can perceive objects in their surroundings based on bird's-eye view images obtained from surround-view cameras. Bird's-eye view perception is a core technology in autonomous driving environmental perception. It's a technique that transforms image information captured by multiple sensors (such as cameras) around the vehicle into a unified, top-down three-dimensional coordinate system for understanding and representation. To accurately perceive the environment, autonomous vehicles may be equipped with multiple image sensors, and then process the images obtained from these sensors. However, this often involves large image processing volumes and complex computations. Summary of the Invention

[0003] The purpose of this application is to provide an end-to-end Bev-aware information processing method and electronic device that can simplify computational complexity.

[0004] In a first aspect, embodiments of this application provide an end-to-end Bev perception information processing method, comprising: performing three-dimensional detection based on current image information corresponding to the viewpoint of a camera pre-set on the subject, to obtain candidate combinations of three-dimensional bounding boxes; performing two-dimensional detection based on the current image information to obtain position information corresponding to each target in the current image information; obtaining the target instance of the previous frame of the current image information and the temporal sequence information between the target instance and the current image information; obtaining initial query information of the current image based on the candidate combinations of three-dimensional bounding boxes, the position information, and the temporal sequence information; and obtaining the end-to-end Bev perception information processing result based on the initial query information.

[0005] In one implementation, the step of performing 3D detection on the current image information based on the pre-set viewpoint of the camera on the subject to obtain candidate combinations of 3D bounding boxes includes: inputting the image information corresponding to the pre-set viewpoint of the camera on the subject into a monocular 3D detection branch of a 3D image processing model; obtaining a combination of 3D bounding boxes corresponding to the viewpoint output by the monocular 3D detection branch; and determining the candidate combination of 3D bounding boxes based on the combination of 3D bounding boxes.

[0006] In one implementation, the step of performing two-dimensional detection based on the current image information to obtain the position information corresponding to each target in the current image information includes: inputting the current image information into a monocular two-dimensional detection branch of a three-dimensional image processing model; obtaining a combination of two-dimensional boxes output by the monocular two-dimensional detection branch; and determining the position information corresponding to each target in the current image information based on the combination of the two-dimensional boxes, the pitch angle of the camera pre-set on the subject, and the relative position of the camera pre-set on the subject.

[0007] In one implementation, obtaining the target instance of the previous frame of the current image information and the temporal information between the target instance and the current image information includes: obtaining the target instance of the previous frame of the current image information output by the backbone network of the 3D graph processing model; determining the pose change information between the previous frame and the current frame of the subject based on sensor information; and obtaining the temporal information based on the pose change information and preset velocity information.

[0008] In one implementation, obtaining initial query information for the current screen based on the candidate combination of the 3D bounding boxes, the position information, and the temporal information includes: deduplicating the candidate combination of the 3D bounding boxes, the position information, and the temporal information; fusing the deduplicated candidate combination of the 3D bounding boxes, the deduplicated position information, and the deduplicated temporal information to obtain fused information; and obtaining initial query information for the current screen based on the fused information and a pre-set uniformly distributed set of background anchor boxes.

[0009] In one implementation, obtaining the initial query information of the current screen based on the fusion information and a pre-set uniformly distributed set of background anchor frames includes: obtaining a preset number of candidate anchor frames based on the fusion information; and combining the candidate anchor frames and the set of background anchor frames to obtain the initial query information of the current screen.

[0010] In one implementation, obtaining the information processing result of the bird's-eye view perception map based on the initial query information includes: inputting the initial query information into the decoding branch of a 3D map processing model; obtaining correction information of the initial query information output by the decoding branch, wherein the correction information is determined based on the predicted category of the initial query information; and obtaining the end-to-end Bev perception processing result based on the correction information.

[0011] In one embodiment, the three-dimensional graph processing model includes a monocular three-dimensional branch and a monocular two-dimensional branch, wherein the monocular three-dimensional branch and the monocular two-dimensional branch share the backbone network and decoding branch of the three-dimensional graph processing model.

[0012] Secondly, embodiments of this application provide an end-to-end Bev perception information processing device, comprising: a candidate combination module, used to perform three-dimensional detection based on current image information corresponding to the viewpoint of a camera pre-set on the subject, to obtain candidate combinations of three-dimensional bounding boxes; a position information module, used to perform two-dimensional detection based on the current image information, to obtain position information corresponding to each three-dimensional bounding box in the current image information; a timing information module, used to obtain the timing information between the target instance of the previous frame of the current image information and the current image information; an initial query information module, used to obtain initial query information of the current image based on the candidate combinations of three-dimensional bounding boxes, the position information, and the timing information; and a processing result module, used to obtain end-to-end Bev perception information processing results based on the initial query information.

[0013] Thirdly, embodiments of this application also provide an electronic device, the electronic device comprising: a processor and a memory; the memory being used to store a program for the electronic device to execute the method provided in any embodiment of this application, and to store data involved in implementing the method provided in any embodiment of this application; the processor being configured to execute the program stored in the memory.

[0014] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method provided in any embodiment of this application.

[0015] The method provided in this application embodiment can process and filter the information obtained from the query before obtaining the end-to-end Bev perception information processing results. It can effectively simplify the original query information generated by the camera, reduce the amount of data of the image information processed by the model, and improve the information processing speed without reducing the query accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the method flow provided in the embodiments of this application; Figure 2 This is a schematic diagram of a three-dimensional graph processing model structure in one embodiment of this application. Detailed Implementation

[0018] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0019] Bird's-eye view perception based on surround-view cameras is crucial for environmental understanding in autonomous driving, and 3D detectors (such as models based on the DETR architecture) have attracted attention due to their flexibility. However, the methods for 3D detectors to detect images have certain problems. For example, when 3D detectors detect images, computational redundancy and inefficient initialization occur: models typically initialize a large number (e.g., 900) of 3D bounding boxes or queries randomly or uniformly in the bird's-eye view space. The vast majority of bounding boxes are located in background areas without targets, causing the decoder (especially the attention module) to perform a large amount of redundant computation, resulting in high inference latency. At the same time, random initialization makes the model converge slowly and has low recall for distant and small targets. Although some studies have attempted to introduce LiDAR priors, this goes against the low-cost principle of pure vision systems.

[0020] Therefore, embodiments of this application provide an end-to-end Bev-aware information processing method, referring to... Figure 1 As shown, the steps include the following.

[0021] Step S11: Perform 3D detection based on the current image information corresponding to the camera's viewpoint pre-set on the subject to obtain candidate combinations of 3D bounding boxes.

[0022] In this embodiment, the subject can be a vehicle, robot, drone, industrial equipment, or other similar device. A camera pre-installed on the subject refers to a vision sensor fixedly mounted on the subject, and the camera's parameters have been calibrated with the subject's coordinate system. The camera's parameters typically include intrinsic and extrinsic parameters. Intrinsic parameters may include focal length, principal point, and distortion coefficients, while extrinsic parameters may include the camera's rotation matrix and translation vector relative to the subject.

[0023] The current image information corresponding to the viewpoint can refer to the image data captured by the camera at the current moment, as well as additional information related to the corresponding viewpoint, such as: RGB (red, green, blue) images, grayscale images, depth maps, timestamps, camera pose during the subject's movement, etc.

[0024] 3D detection refers to inferring information about a target in three-dimensional space from two-dimensional images or multimodal data. This information includes the target's position (x, y, z), size (w, h, l), orientation (yaw / rotation), and category (car, pedestrian, etc.). Possible implementation methods include monocular 3D detection, binocular / multi-view reconstruction, vision and depth fusion, and Bird's Eye View (BEV) projection methods to perform 3D detection on image information.

[0025] A 3D bounding box can include a bounding box used to represent the target in 3D space. A 3D bounding box can include information such as its center point, size, and orientation. Candidate combinations refer to a set of multiple possible 3D bounding boxes generated during the 3D detection process, which have not yet undergone final filtering (such as NMS or confidence filtering), including multi-scale candidates, multi-location candidates, and multi-class candidates.

[0026] Step S12: Perform two-dimensional detection based on the current image information to obtain the position information corresponding to each target in the current image information.

[0027] In this embodiment, 2D detection refers to the process of identifying a target on an image plane and outputting the target's 2D location. This can include: generating a bounding box at the target's 2D location, generating a category label for the target, and generating confidence scores for the bounding box or category label. 2D detection can be implemented using anchor-box based networks or non-anchor-box based networks. Anchor-box based networks can include residual convolutional neural networks, while non-anchor-box based networks can include YOLO (you only look once), CenterNet, etc.

[0028] The target in the image information can refer to a physical entity or region in the image that has a clear semantic category, is complete and independent, and needs to be identified and located by the perception system. The location information corresponding to each target can refer to the location information of each three-dimensional target mapped in the two-dimensional image.

[0029] Step S13: Obtain the target instance of the previous frame of the current frame information and the timing information between it and the current frame information.

[0030] In this embodiment, the target instance in the previous frame of the current image information refers to the temporal information between the target instance in the previous frame and the current image information. This temporal information can refer to the temporal information between the target instance in the previous frame and the target instance in the current image information. The temporal information between the target instance in the previous frame and the current image information can be obtained based on the motion patterns of the subject. The target instance may include: a unique ID (identifier), information about the 3D bounding box, category, and motion state. The 3D bounding box information may include position, size, and orientation, and the motion state may include speed, acceleration, etc.

[0031] Temporal information can include the temporal and motion relationships between the previous frame and the current frame. Further, temporal information can include: time intervals, camera / subject pose changes, and target motion. Target motion can include displacement, velocity prediction information, etc. Candidate combinations of 3D bounding boxes can include a set of high-confidence 3D bounding boxes.

[0032] Step S14: Based on the candidate combinations of the three-dimensional bounding boxes, the location information, and the temporal information, obtain the initial query information for the current screen.

[0033] In this embodiment, the candidate combinations, position information, and timing information of the 3D bounding boxes can correspond to multiple targets, and the targets corresponding to all candidate combinations, position information, and timing information of the 3D bounding boxes may be duplicated. Obtaining initial query information for the current screen based on the candidate combinations, position information, and timing information can include: obtaining query information corresponding to all non-duplicated targets based on the targets corresponding to the candidate combinations, position information, and timing information, as initial query information. The initial query information can include at least one of the 3D bounding boxes, position information, and timing information, or it can generate information indicating at least one target based on the 3D bounding boxes, position information, and timing information, and use the target-indicating information as the initial query information.

[0034] Step S15: Based on the initial query information, obtain the end-to-end Bev-aware information processing results.

[0035] In possible implementations, the initial query information can also be expressed in vector form. A Bird's-Eye Perception (BEV) Feature Map refers to a representation that maps the perception results to a top-down view. A BEV Feature Map can include a coordinate system and representation content. The coordinate system can be ground plane coordinates, and the representation content can include target distribution, occupancy, semantic information, etc. End-to-end BEV perception refers to using a unified deep learning model, directly inputting raw sensor data (the current image information from the aforementioned cameras) and outputting the perception results in a bird's-eye view space.

[0036] The method provided in this application performs 3D and 2D detection on the current image information, and associates 2D position information, 3D bounding box candidate combinations, and temporal information with the previous frame. This allows for correction of 3D targets under image observation constraints, reducing depth estimation errors caused by single 3D detection and improving target localization accuracy and detection precision. By introducing target instances from the previous frame and temporal information between the current image and the previous frame, temporal modeling of the target in the current frame is performed, achieving cross-frame information fusion. This effectively reduces jitter and jumps in detection results, improving the continuity and stability of target detection. By constructing initial query information and processing the bird's-eye view based on this initial query information, target-related feature information can be extracted from the bird's-eye view space in a query-driven manner, avoiding redundant calculations on the entire bird's-eye view, thereby improving computational efficiency and reducing resource consumption.

[0037] The method provided in this application embodiment can process and filter the information obtained from the query before obtaining the end-to-end Bev perception information processing results. It can effectively simplify the original query information generated by the camera, reduce the amount of data of the image information processed by the model, and improve the information processing speed without reducing the query accuracy.

[0038] In one implementation, the step of performing 3D detection on the current image information based on the pre-set viewpoint of the camera on the subject to obtain candidate combinations of 3D bounding boxes includes: inputting the image information corresponding to the pre-set viewpoint of the camera on the subject into a monocular 3D detection branch of a 3D image processing model; obtaining a combination of 3D bounding boxes corresponding to the viewpoint output by the monocular 3D detection branch; and determining the candidate combination of 3D bounding boxes based on the combination of 3D bounding boxes.

[0039] In a possible implementation, the 3D bounding boxes can include 3D proposals output by the monocular 3D detection branch. In the context of the monocular 3D detection branch, a 3D proposal refers to a candidate object hypothesis with complete 3D information generated by the network, which can be represented as B_mono3d. In implementation, the model obtains the top N1 3D proposals from each viewpoint priority ranking from the monocular 3D branch, performs 3D non-maximum suppression (3D NMS) processing after coordinate transformation, and retains N2 proposals to form the geometric prior anchor box set A_3d (i.e., the combination of 3D bounding boxes). N1 can be 10-100, further, N1 can be 50, N2 can be 100-200, further, N2 can be 150. The IoU (Intersection over Union) threshold for 3D non-maximum suppression processing can be 0.1-0.9, further, the IoU threshold can be 0.5.

[0040] In one implementation, the step of performing two-dimensional detection based on the current image information to obtain the position information corresponding to each target in the current image information includes: inputting the current image information into a monocular two-dimensional detection branch of a three-dimensional image processing model; obtaining a combination of two-dimensional boxes output by the monocular two-dimensional detection branch; and determining the position information corresponding to each target in the current image information based on the combination of the two-dimensional boxes, the pitch angle of the camera pre-set on the subject, and the relative position of the camera pre-set on the subject.

[0041] The monocular 2D detection branch includes a lightweight FCOS (Fully Convolutional One-Stage Object Detection) head connected after the FPN (Feature Pyramid Network) features of each viewpoint. The FCOS outputs a 2D detection box b_2d. For high-confidence boxes, the midpoint of its bottom edge is taken as the image observation of the "target-ground contact point". Camera calibration provides the camera's pitch angle (how much the camera tilts downward) and the geometric relationship between the camera and the ground. Therefore, a spatial ray can be drawn from the camera's optical center through this pixel and intersected with the "ground plane assumption" (or the depth constraint provided by sparse depth) to obtain the ground position point P of the target in the ego (egocentric coordinate system). Then, using the location point P as the bottom center of the 3D bounding box (the height direction can be raised to a reasonable height using the category size prior / average size), given a default or a small number of discrete orientation candidates, one or a small number of 3D candidate boxes are generated; after summarizing the candidate boxes generated from all perspectives, lightweight deduplication is performed and sorted by score, and the prior anchor box set A_2d (i.e., the combination of two-dimensional boxes) is measured.

[0042] In one implementation, obtaining the target instance of the previous frame of the current image information and the temporal information between the target instance and the current image information includes: obtaining the target instance of the previous frame of the current image information output by the backbone network of the 3D graph processing model; determining the pose change information between the previous frame and the current frame of the subject based on sensor information; and obtaining the temporal information based on the pose change information and preset velocity information.

[0043] In a possible implementation, the high-confidence target instance I_prev (e.g., cache limit instance num_temp_instances=128) output by the main branch in the previous frame is read from the cache. First, the pose change of the vehicle from the previous frame to the current frame is obtained using IMU / odometry, and motion compensation is performed on each 3D bounding box in I_prev (mapping its coordinates from the previous frame to the ego coordinate system of the current frame). Then, combined with the target motion estimation, a constant velocity model is used to extrapolate the target center position in time to obtain the predicted 3D proposal set for the current frame, forming the temporal prior anchor box set A_temp. Here, A_temp only participates in the construction of A_init as an initialization candidate prior to reduce random initialization and does not replace the detection / refinement results of the current frame.

[0044] In one implementation, obtaining initial query information for the current screen based on the candidate combination of the 3D bounding boxes, the position information, and the temporal information includes: deduplicating the candidate combination of the 3D bounding boxes, the position information, and the temporal information; fusing the deduplicated candidate combination of the 3D bounding boxes, the deduplicated position information, and the deduplicated temporal information to obtain fused information; and obtaining initial query information for the current screen based on the fused information and a pre-set uniformly distributed set of background anchor boxes.

[0045] In one possible embodiment, when acquiring fusion information, the combination, position information, and temporal information of 3D bounding boxes can be deduplicated and fused. Based on a preset source confidence weight, for example, the source confidence weight could be: A_temp (temporal information) > A_3d (combination of 3D bounding boxes) > A_2d (position information corresponding to 2D bounding boxes), and the confidence of the 3D bounding boxes and the confidence of the 2D bounding boxes, the fused candidate anchor boxes are sorted. Then, the top-K (e.g., K=256) high-quality candidate anchor boxes are selected and combined with a small number (e.g., 128) evenly distributed background exploration anchor boxes to form the final initialization anchor box set A_init, whose total number num_anchor is strictly controlled at a low level (e.g., 384). Then, the initialization anchor box set A_init is used as the initial query for the main 3D bird's-eye view perception decoding branch; the decoding branch predicts the class confidence for each initialization anchor box and regresses the correction amount of its 3D bounding box parameters, thereby refining and updating the anchor box position / size / orientation in A_init, and outputting the final detection result for the current frame. Then, the data is filtered according to the confidence threshold (e.g., confidence > 0.25), and the Top-N instances (e.g., N=128) are selected according to the confidence to form I_curr and update the cache for use in generating I_prev in the next frame, so that num_temp_instances is kept at a low level (e.g., 128).

[0046] In one implementation, obtaining the initial query information of the current screen based on the fusion information and a pre-set uniformly distributed set of background anchor frames includes: obtaining a preset number of candidate anchor frames based on the fusion information; and combining the candidate anchor frames and the set of background anchor frames to obtain the initial query information of the current screen.

[0047] In one embodiment, obtaining the processing result of the 3D graph based on the initial query information includes: inputting the initial query information into the decoding branch of the 3D graph processing model; obtaining correction information of the initial query information output by the decoding branch, wherein the correction information is determined based on the predicted category of the initial query information; and obtaining the processing result of the 3D graph based on the correction information.

[0048] In one embodiment, the three-dimensional graph processing model includes a monocular three-dimensional branch and a monocular two-dimensional branch, wherein the monocular three-dimensional branch and the monocular two-dimensional branch share the backbone network and decoding branch of the three-dimensional graph processing model.

[0049] This application proposes an end-to-end Bev-aware information processing method integrating "training-inference". It constructs a multi-task collaborative training framework and implements a dynamic sparse prior initialization strategy based on the diverse and high-quality intermediate results generated during inference within this framework. The two are interdependent and mutually reinforcing: collaborative training ensures reliable priors for each branch, while sparse initialization fully utilizes these priors to improve inference efficiency, forming a positive closed loop.

[0050] like Figure 2 As shown, the three-dimensional graph processing model in this application embodiment includes: a shared multi-view image feature extraction backbone network (equivalent to...) Figure 2 The shared backbone ResNet-101+FPN shown; a 3D bird's-eye view perception decoding branch of the backbone (equivalent to Figure 2 The main BEV decoding branch shown in the diagram); and an auxiliary monocular 3D detection branch (equivalent to...). Figure 2 The monocular 3D detection branch shown); an auxiliary monocular 2D detection branch (equivalent to...) Figure 2 (The monocular 2D detection branch is shown in the diagram). Furthermore, the 3D graph processing model includes a caching module, which can store preprocessing vectors (I_prev) to provide historical context information during the processing of the current frame.

[0051] Figure 2The illustrated 3D graph processing model achieves hierarchical feature sharing and specialization: the deep features extracted by the backbone network are shared by all branches, but a lightweight dedicated adaptation layer is designed for each auxiliary branch. The shared features come from the deep multi-scale features extracted by the shared backbone (ResNet-101+FPN); each branch connects to a task-specific module / detection head with independent parameters (backbone BEV decoding branch, monocular 3D mini-network, monocular 2D detection branch) after sharing features. First, a common representation is formed through the shared backbone, and then the task differences are satisfied through the branch-specific modules. During the training phase, the proposed consistency-guided joint loss function is adjusted to suppress multi-task conflicts and shift the optimization focus to the main task and consistency terms. The monocular 2D detection branch can use the FCOS (Fully Convolutional One-Stage Object Detection) head. The monocular 3D detection branch can be used to detect 3D anchors as geometric prior information. The monocular 2D detection branch can be used to detect 2D anchors as metric prior information. The caching module can be used to cache temporal anchor boxes (A_temp) as temporal prior information. 3D anchor boxes, 2D anchor boxes, and temporal anchor boxes are fused, deduplicated, and sorted to obtain the top K initial anchor boxes (A_init), which include background exploration information. Information from the shared backbone output is input into the backbone BEV decoding branch, along with the initial anchor boxes, to obtain the current vector I_curr. The cache is then updated using the current vector I_curr. Based on the results from the monocular 3D detection branch, the monocular 2D detection branch, and the backbone BEV decoding branch, the total loss value is obtained.

[0052] In one example of this application, the loss function of the 3D graph processing model includes a consistency-guided joint loss function: the total loss function is defined as: L_total = L_bev + α × L_mono3d + β × L_mono2d + γ × L_consistency. Where: L_bev is the 3D detection loss of the main branch, which can include: L1, GIoU, and focal loss under Hungarian matching. That is, when calculating the 3D detection loss for the main branch, Hungarian matching is used to match the model's guessed boxes with the actual boxes one-to-one; then, for these paired boxes, three different losses—L1 loss (L1 norm loss), GIoU (Generalized Intersection over Union) loss, and Focal loss—are used to measure the difference between them. L_mono3d is the detection loss for the monocular 3D branch. L_mono2d is the detection loss for the monocular 2D branch. L_consistency is the consistency constraint loss, which forces different branches to maintain spatial consistency in their predictions of the same physical target. For example, when constraining the consistency of the 2D monocular branch, the 3D bounding boxes predicted by the main branch and matched with the ground truth are projected onto each viewpoint, and the L1 loss is calculated with the 2D bounding boxes predicted by the monocular 2D branch at that viewpoint that have the largest IoU. When constraining the consistency of the 3D monocular branch, the B_mono3d coordinates are transformed to ego, and then paired with the 3D bounding boxes of the main branch according to the largest 3D IoU. The L1 loss is calculated for differences in position / size / heading and then summarized.

[0053] Meanwhile, 3D detectors face challenges in multi-task collaborative training when detecting images. Introducing auxiliary tasks (such as monocular 2D / 3D detection) has become a trend to improve model robustness. However, simply adding multiple detector heads to a network for independent training leads to conflicting network optimization directions, chaotic feature learning, and fails to ensure that the auxiliary tasks effectively improve the performance of the main task. Designing a unified joint training framework that enables beneficial collaboration among multiple tasks at the shared feature level, rather than simple parallel processing, is a pressing issue.

[0054] In this embodiment, during the initial training of the 3D graph processing model, the auxiliary task loss weights (α, β) are increased to guide the backbone network to quickly learn basic geometric and semantic features. As training progresses, according to the weight scheduling criteria (see the weight scheduling under the collaborative training process in the specific implementation), the weights of the main task loss L_bev and the consistency loss γ are gradually increased, shifting the focus of network optimization towards improving the final 3D perception accuracy and multi-task prediction consistency. Through dynamic sparse initialization, the number of core queries / anchor boxes processed by the decoder is reduced by more than 50% (e.g., from 900 to 384), while the number of instances propagated across frames is significantly reduced (e.g., from 600 to 128). This directly leads to a significant reduction in the computational complexity and memory usage of core modules such as attention, significantly improving inference speed and making it more suitable for vehicle deployment.

[0055] In one example of this application, the 3D image processing model includes: a shared backbone, a main branch, a monocular 3D branch, and a monocular 2D branch. The shared backbone uses ResNet-101 (Residual Network 101) and a feature pyramid network as a shared feature extractor, with the input being a 6-view panoramic image. The main branch employs a BEV decoder based on deformable attention, outputting 3D detection results. The monocular 3D branch connects a small network after the FPN features of each viewpoint to predict the depth distribution, 3D bounding box size, and orientation of each pixel, obtaining 3D proposals through keypoint association. The monocular 2D branch connects a lightweight FCOS head after the FPN features of each viewpoint, outputting 2D detection boxes.

[0056] During the training phase of the 3D graph processing model, L_bev uses the standard Set Prediction Loss. L_mono3d uses keypoint heatmap loss and size regression loss. L_mono2d uses FCOS loss. L_consistency is configured to, for each 3D box predicted by the main branch and matched to the ground truth, project it onto each view and calculate the L1 loss with the 2D box predicted by the monocular 2D branch in that view that has the largest IoU.

[0057] In this example, the 3D graph processing model can be configured to train for a total of 24 epochs. The first 8 epochs are set to (α=1.0, β=0.5, γ=0.1); epochs 9-16 are adjusted to (α=0.5, β=0.2, γ=0.3); and the last 8 epochs are adjusted to (α=0.2, β=0.1, γ=0.5). This scheduling guides the network to gradually transition from learning basic tasks to optimizing collaborative consistency.

[0058] During the runtime of the 3D graph model, the target total number of anchor boxes per frame is set to num_anchor=384, and the number of cached instances is num_temp_instances=128. Based on the above 3D graph processing model, the end-to-end Bev perception information processing method in this application example includes: obtaining the Top-50 proposals from each viewpoint from the monocular 3D branch based on the image information obtained by the camera from each viewpoint, converting them and performing 3D NMS (IoU threshold 0.5), retaining about 150; obtaining 2D boxes with confidence > 0.4 from the monocular 2D branch, performing ground inverse projection using the calibrated camera pitch angle, generating about 100 3D proposals; reading 128 instances from the previous frame from the cache, performing motion compensation using a constant velocity model and IMU data, generating 128 proposals; merging all proposals (about 378); performing lightweight 3D IoU deduplication (threshold 0.8); and applying the score S = 0.3 × source weight + The data is sorted by category confidence score of 0.7, with source weights of 1.0 for A_temp, 0.8 for A_3d, and 0.6 for A_2d. The top 256 proposals are selected and supplemented with 128 uniform background anchor boxes to form 384 initial query results. The BEV decoder processes these 384 queries and outputs the results. Results with confidence scores > 0.25 are filtered, and the Top-128 are updated to the cache based on their confidence scores. These results are used to generate the information processing results for the bird's-eye view perception map.

[0059] The 3D graph detection model provided in this application, validated on the nuScenes test set, shows a 42% improvement in inference speed (FPS) compared to the baseline model (without co-training, randomly initialized with 900 queries), a 38% reduction in decoder module FLOPs, and no change in mAP. However, it achieves a 6.2% improvement in recall for small targets (pedestrians, motorcycles). The key performance indicator AMOTA (tracking accuracy) is improved by 4.5%, and the number of training epochs required to achieve the same validation set accuracy is reduced by approximately 30%, resulting in improvements in efficiency, accuracy, and convergence.

[0060] The method provided in this application improves the accuracy of image processing: the A_3d and A_2d priors from the image effectively cover distant and small targets, improving recall; the temporal prior A_temp enhances tracking stability. Convergence is accelerated: the initialization position is closer to the real target, significantly shortening the model training convergence time. Furthermore, model feature learning is enhanced: the consistency loss in the collaborative training framework forces the network to learn a unified feature representation robust to viewpoint and task changes, improving the model's generalization ability and achieving comprehensive optimization of perceptual performance, forming a positive design loop: the training framework provides high-quality, multi-source prior information for inference initialization; while the efficiency of sparse initialization verifies and depends on the collaborative effect of multi-task training. The organic combination of these two aspects constitutes a complete high-performance system design scheme from training to inference.

[0061] This application embodiment also provides an end-to-end Bev perception information processing device, including: a candidate combination module, used to perform three-dimensional detection based on current image information corresponding to the viewpoint of a camera on the subject, and obtain candidate combinations of three-dimensional bounding boxes; a position information module, used to perform two-dimensional detection based on the current image information, and obtain position information corresponding to each three-dimensional bounding box in the current image information; a timing information module, used to obtain the timing information between the target instance of the previous frame of the current image information and the current image information; an initial query information module, used to obtain initial query information of the current image based on the candidate combinations of three-dimensional bounding boxes, the position information, and the timing information; and a processing result module, used to obtain the end-to-end Bev perception information processing result based on the initial query information.

[0062] The apparatus provided in this application embodiment can implement any one of the end-to-end Bev-aware information processing methods in this application embodiment.

[0063] This application also provides an electronic device, which includes a processor and a memory; the memory is used to store a program for the electronic device to execute the method provided in any embodiment of this application, and to store data involved in implementing the method provided in any embodiment of this application; the processor is configured to execute the program stored in the memory.

[0064] The embodiments of the present invention described above are combinations of elements and features of the invention. Unless otherwise stated, elements or features may be considered optional. Individual elements or features may be practiced without combination with other elements or features. Furthermore, embodiments of the invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the invention may be rearranged. Some constructions of any embodiment may be included in another embodiment and may be replaced by corresponding constructions of another embodiment. It will be apparent to those skilled in the art that claims that are not explicitly referenced in the appended claims may be combined to form embodiments of the invention, or may be included as new claims in modifications made after the submission of this invention.

[0065] In firmware or software configuration, the embodiments of the present invention can be implemented in the form of modules, processes, functions, etc. Software code can be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and can send data to and receive data from the processor via various known means.

[0066] The various aspects of the systems and methods described herein can be implemented as functions programmable into any type of circuit, including programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), programmable array logic (PAL) devices, electronically programmable logic and memory devices, standard cell-based devices, and application-specific integrated circuits (ASICs). Other possibilities for implementing these aspects of the system include: microcontrollers with memory, such as electronically erasable programmable read-only memory (EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, these aspects of the system can be embodied in microprocessors with software-based circuit emulation, discrete logic (sequential and combinational), custom devices, fuzzy (neural) logic, quantum devices, and any combination of the various device types described above. Of course, underlying device technologies can be provided in various component types, such as metal-oxide-semiconductor field-effect transistor (MOSFET) technologies such as complementary metal-oxide-semiconductor (CMOS), bipolar technologies such as emitter-coupled logic (ECL), polymer technologies (e.g., silicon conjugated polymers and metal conjugated polymer metal structures), hybrid analog and digital, etc.

[0067] The various functions or processes disclosed herein can be described, based on their behavior, register transfers, logic components, transistors, geometric layouts, and / or other characteristics, as data and / or instructions embodied in various computer-readable media. Computer-readable media that may contain such formatted data and / or instructions include, but are not limited to, various forms of non-volatile storage media (e.g., optical, magnetic, or semiconductor storage media) and carrier waves, which can be used to transmit such formatted data and / or instructions via wireless, optical, or wired signal media, or any combination thereof. Such data and / or instructions can be processed by a processing entity (e.g., one or more processors) upon receipt by any of various circuits (e.g., a computer).

[0068] The above description of the illustrated embodiments of the systems and methods is not intended to be exhaustive or to limit the systems and methods to the precise forms disclosed. While specific embodiments and examples of system components and methods have been described herein for illustrative purposes, those skilled in the art will understand that various equivalent modifications can be made within the scope of the systems, components, and methods. The teachings of the systems and methods provided herein can be applied to other processing systems and methods, and are not limited to those described above.

[0069] Those skilled in the art will understand that various changes and / or modifications can be made to the invention illustrated in particular embodiments without departing from the spirit or scope of the broad description of the invention. Therefore, these embodiments are to be considered illustrative rather than restrictive in all respects. Furthermore, the invention includes any combination of features described with respect to different embodiments (including those in the abstract section), even if such feature or combination of features is not expressly specified in the claims or the detailed description of these embodiments.

[0070] Generally, the terminology used in the following claims should not be construed as limiting the systems and methods to the specific embodiments disclosed in the specification and claims, but should be interpreted as encompassing all processing systems operating under the claims. Therefore, the systems and methods are not limited by this disclosure, but their scope is determined entirely by the claims.

[0071] Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprising,” “including,” and “including” should be interpreted in a encompassing sense, not in an exclusive or exhaustive sense; that is, in the sense of “including but not limited to.” The use of singular or plural words also includes both singular and plural, respectively. Furthermore, “this article,” “in the following,” “above,” “below,” and words with similar meanings refer to the application as a whole, and not to any particular part of the application. When the word “or” is used to refer to a list of two or more items, the word “or” includes all of the following interpretations: any item in the list, all items in the list, and any combination of items in the list.

[0072] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0073] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An end-to-end Bev-aware information processing method, characterized in that, include: Three-dimensional detection is performed based on the current image information corresponding to the camera's viewpoint pre-set on the subject to obtain candidate combinations of three-dimensional bounding boxes; Two-dimensional detection is performed based on the current image information to obtain the position information corresponding to each target in the current image information; Obtain the target instance of the previous frame of the current frame information and the timing information between the target instance and the current frame information; Based on the candidate combinations of the three-dimensional bounding boxes, the location information, and the temporal information, the initial query information of the current screen is obtained; Based on the initial query information, the end-to-end Bev-aware information processing results are obtained.

2. The method according to claim 1, characterized in that, The step of performing 3D detection on the current image information based on the pre-set viewpoint of the camera on the subject to obtain candidate combinations of 3D bounding boxes includes: The image information corresponding to the camera viewpoint of the subject is pre-set and input into the monocular 3D detection branch of the 3D image processing model; Obtain the combination of 3D bounding boxes corresponding to the viewpoint output by the monocular 3D detection branch; Based on the combination of the three-dimensional boxes, candidate combinations of the three-dimensional boxes are determined.

3. The method according to claim 1, characterized in that, The step of performing two-dimensional detection based on the current image information to obtain the position information corresponding to each target in the current image information includes: The current image information is input into the monocular two-dimensional detection branch of the three-dimensional image processing model; Obtain the combination of the two-dimensional boxes output by the monocular two-dimensional detection branch; Based on the combination of the two-dimensional frames, the pitch angle of the camera pre-set on the subject, and the relative position of the camera pre-set on the subject, the position information corresponding to each target in the current image information is determined.

4. The method according to claim 1, characterized in that, The timing information between the target instance of the previous frame and the current frame information obtained from the current frame information includes: Obtain the target instance of the previous frame from the current image information output by the backbone network of the 3D image processing model; Based on sensor information, determine the pose change information of the subject between the previous frame and the current frame; The timing information is obtained based on the pose change information and the preset velocity information.

5. The method according to claim 1, characterized in that, The initial query information for the current frame is obtained based on the candidate combinations of the 3D bounding box, the location information, and the temporal information, including: The candidate combinations of the 3D bounding boxes, the position information, and the temporal information are deduplicated, and the candidate combinations of the deduplicated 3D bounding boxes, the deduplicated position information, and the deduplicated temporal information are fused to obtain fused information. Based on the fused information and a pre-set set of evenly distributed background anchor frames, the initial query information for the current screen is obtained.

6. The method according to claim 5, characterized in that, The initial query information for the current screen is obtained based on the fused information and a pre-set set of evenly distributed background anchor frames, including: Based on the fused information, a preset number of candidate anchor boxes are obtained; The candidate anchor boxes and the set of background anchor boxes are combined to obtain the initial query information of the current screen.

7. The method according to claim 5, characterized in that, The process of obtaining end-to-end Bev-aware information processing results based on the initial query information includes: The initial query information is input into the decoding branch of the 3D graph processing model; Obtain correction information for the initial query information output by the decoding branch, wherein the correction information is determined based on the predicted category of the initial query information; Based on the corrected information, the end-to-end Bev-aware processing result is obtained.

8. The method according to claims 2-4, 7, characterized in that, The 3D graph processing model includes a monocular 3D branch and a monocular 2D branch, and the monocular 3D branch and the monocular 2D branch share the backbone network and decoding branch of the 3D graph processing model.

9. An end-to-end Bev sensing information processing device, characterized in that, include: The candidate combination module is used to perform 3D detection based on the current image information corresponding to the viewpoint of the camera on the subject, and obtain candidate combinations of 3D bounding boxes. The location information module is used to perform two-dimensional detection based on the current image information to obtain the location information corresponding to each three-dimensional box in the current image information; The timing information module is used to obtain the timing information between the target instance of the previous frame of the current frame information and the current frame information; The initial query information module is used to obtain the initial query information of the current screen based on the candidate combination of the three-dimensional bounding box, the position information, and the temporal information. The processing result module is used to obtain end-to-end Bev-aware information processing results based on the initial query information.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store a program for the electronic device to execute the method as described in any one of claims 1-9, and to store data related to implementing the method as described in any one of claims 1-9; The processor is configured to execute programs stored in the memory.

Images