Detection, detection model training method and device, equipment, medium and product

Abstract

This invention provides a training method, detection method, apparatus, device, medium, and product for a detection model. The method includes: acquiring and processing multiple frames of environmental data frame by frame; firstly, extracting bird's-eye view features from the T-th frame data using a feature processing network; then fusing these features with the fusion features of the (T-1)-th frame in the cache to generate the fusion features for the T-th frame; updating the cache; and finally, outputting the detection result based on the fusion features through an object detection network. This scheme employs iterative temporal feature fusion, requiring only the fusion features of one frame to be cached, thus fixing resource overhead and avoiding the linear correlation between computational resources and temporal length found in traditional temporal feature fusion methods. Furthermore, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly reducing memory overhead during the actual vehicle deployment phase.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a detection method, a detection model training method, a detection device, a detection model training device, an electronic device, a readable storage medium, and a computer program product. Background Technology

[0002] In the field of intelligent driving, obstacle detection is one of the key technologies. Existing bird's-eye view obstacle detection solutions are mainly divided into two categories: single-frame-based detection models and temporal fusion detection models. Among them, temporal bird's-eye view detection models, by leveraging temporal information, can capture dynamic information, enhance the understanding of object dynamic behavior and temporary occlusion, and improve the accuracy and reliability of detection results. They can also better understand the scene context and its evolution, making them particularly effective for complex scenes and situations where inference from a single frame is difficult. Furthermore, temporal models integrate information from multiple frames, mitigating the impact of poor single-frame environmental data quality. For example, by observing the coherent movement of objects across multiple frames, they can compensate for occlusion problems in a single frame, exhibiting stronger robustness. Simultaneously, temporal models are naturally suitable for object tracking, directly estimating the velocity and direction of motion of objects, which is extremely useful for dynamic scene analysis and behavior prediction.

[0003] However, time-series models also have disadvantages. They require processing multiple frames of data, leading to high computational costs and increased consumption of computing and storage resources. In the real-vehicle deployment phase, existing time-series models need to cache multiple frames of time-series features. Under the condition that resources in real vehicles are limited, the deployment problem of such models becomes particularly prominent. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a detection method, a training method for a detection model, an apparatus, an electronic device, a readable storage medium, and a computer program product, so as to solve the problems of high computational cost and large consumption of computing and storage resources in time-series fusion detection models.

[0005] To address the above problems, this invention provides a detection method, wherein the detection model includes a feature processing network and a target detection network, and the method includes:

[0006] Acquire multi-frame environmental data;

[0007] For each frame of the multi-frame environmental data, the feature processing network performs the following operations:

[0008] Extract the corresponding bird's-eye view features from the environmental data of frame T; where T is a positive integer;

[0009] The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame;

[0010] Update the fusion feature in the cache to the fusion feature of the T-th frame;

[0011] The target detection network processes the fused features and outputs the detection results.

[0012] Optionally, fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame includes:

[0013] The bird's-eye view features of the T-th frame and the fusion features of the (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features;

[0014] The splicing features are subjected to dimensionality reduction processing to obtain the fused features of the Tth frame.

[0015] Optionally, the step of stitching the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame along the channel dimension to obtain the stitched features includes:

[0016] The fused features of the (T-1)th frame are aligned to the coordinate system of the Tth frame through coordinate system transformation to obtain the aligned fused features of the (T-1)th frame.

[0017] The bird's-eye view features of the T-th frame and the fused features of the aligned (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features.

[0018] Optionally, extracting the corresponding bird's-eye view features from the environmental data of frame T includes:

[0019] Feature extraction is performed on the environmental data of the Tth frame to obtain a two-dimensional feature map;

[0020] The two-dimensional feature map is converted into the three-dimensional bird's-eye view feature.

[0021] Optionally, the detection model is an obstacle detection model based on a bird's-eye view. The step of processing the fused features through the target detection network and outputting the detection result includes:

[0022] The fused features of the T-th frame are input into the target detection network of the obstacle detection model;

[0023] The target detection network processes the fusion features of the T-th frame to determine the detection result; wherein the detection result includes obstacle information.

[0024] The present invention also provides a method for training a detection model, the detection model comprising a feature processing network and an object detection network, the method comprising:

[0025] Acquire training data; wherein the training data includes multi-frame sample environment data;

[0026] For each frame of the multi-frame sample environment data, the feature processing network performs the following operations:

[0027] Extract the corresponding bird's-eye view features from the environmental data of the Tth frame sample;

[0028] The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame;

[0029] Update the fusion feature in the cache to the fusion feature of the T-th frame;

[0030] Based on the fusion features, the detection model is trained to obtain a trained detection model.

[0031] Optionally, the multi-frame sample environmental data is environmental data acquired sequentially from the same video stream in chronological order.

[0032] The present invention also provides a detection device, wherein the detection model includes a feature processing network and a target detection network, and the device includes:

[0033] The data acquisition module is used to acquire multiple frames of environmental data.

[0034] For each frame of the multi-frame environmental data, the feature processing network performs the following operations:

[0035] The feature extraction module is used to extract the corresponding bird's-eye view features from the environmental data of the T-th frame; where T is a positive integer;

[0036] The feature fusion module is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame.

[0037] A cache update module is used to update the fusion features in the cache to the fusion features of the T-th frame;

[0038] The detection module is used to process the fused features through the target detection network and output the detection results.

[0039] Optionally, the feature fusion module includes:

[0040] The stitching submodule is used to stitch the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame according to the channel dimension to obtain the stitched features;

[0041] The dimensionality reduction submodule is used to perform dimensionality reduction processing on the splicing features to obtain the fused features of the T-th frame.

[0042] Optionally, the splicing submodule includes:

[0043] The transformation unit is used to align the fused features of the (T-1)th frame to the coordinate system of the Tth frame through coordinate system transformation, so as to obtain the aligned fused features of the (T-1)th frame.

[0044] The stitching unit is used to stitch the bird's-eye view features of the Tth frame with the fused features of the aligned T-1th frame according to the channel dimension to obtain the stitched features.

[0045] Optionally, the feature extraction module includes:

[0046] An extraction submodule is used to extract features from the environmental data of the Tth frame to obtain a two-dimensional feature map;

[0047] The conversion submodule is used to convert the two-dimensional feature map into the three-dimensional bird's-eye view features.

[0048] Optionally, the detection model is an obstacle detection model based on a bird's-eye view, and the detection module includes:

[0049] The input submodule is used to input the fused features of the T-th frame into the target detection network of the obstacle detection model;

[0050] The result determination submodule is used to process the fusion features of the T-th frame through the target detection network to determine the detection result; wherein the detection result includes obstacle information.

[0051] The present invention also provides a training apparatus for a detection model, the detection model including a feature processing network and an object detection network, the apparatus comprising:

[0052] A data acquisition module is used to acquire training data; wherein, the training data includes multi-frame sample environment data;

[0053] For each frame of the multi-frame sample environment data, the feature processing network performs the following operations:

[0054] The feature extraction module is used to extract the corresponding bird's-eye view features from the environmental data of the T-th frame sample; where T is a positive integer;

[0055] The feature fusion module is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame.

[0056] A cache update module is used to update the fusion features in the cache to the fusion features of the T-th frame;

[0057] The model training module is used to train the detection model based on the fused features to obtain the trained detection model.

[0058] Optionally, the multi-frame sample environmental data is environmental data acquired sequentially from the same video stream in chronological order.

[0059] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0060] Memory, used to store computer programs;

[0061] The processor, when executing a program stored in memory, implements the method described above.

[0062] This invention also discloses a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the method described above.

[0063] This invention also discloses a computer program product that, when run on an electronic device, causes a processor to execute the steps in the method described above.

[0064] According to an embodiment of the present invention, multiple frames of environmental data are acquired. For each frame of the multi-frame environmental data, the following operations are performed through the feature processing network: extracting the corresponding bird's-eye view features from the T-th frame of environmental data; fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame; and updating the fusion features in the cache to the fusion features of the T-th frame. Finally, the fusion features are processed through the target detection network to output the detection results. This scheme adopts iterative temporal feature fusion, which only requires caching the fusion features of one frame, fixing the resource overhead and avoiding the linear correlation between computational resources and temporal length in the original temporal feature fusion method. At the same time, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly saving memory overhead in the actual vehicle deployment stage. Attached Figure Description

[0065] Figure 1 A flowchart illustrating the steps of a detection method according to an embodiment of the present invention is shown.

[0066] Figure 2 A schematic diagram of the iterative temporal feature fusion scheme is shown;

[0067] Figure 3 A flowchart illustrating the steps of a training method for a detection model according to an embodiment of the present invention is shown.

[0068] Figure 4 A schematic diagram of the video streaming training process is shown;

[0069] Figure 5 A structural block diagram of a detection device embodiment provided by one embodiment of the present invention is shown;

[0070] Figure 6 The diagram shows a structural block diagram of an embodiment of a training device for a detection model provided by an embodiment of the present invention;

[0071] Figure 7 A structural block diagram of an electronic device for detecting or training a detection model is shown according to an exemplary embodiment. Detailed Implementation

[0072] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The detection method in this embodiment of the invention can be executed by electronic devices such as smart vehicle systems and robots. For example, the detection method can be applied to smart vehicle systems or any other applicable hardware and software. This embodiment of the invention does not limit this application.

[0074] The training method of the detection model in the embodiments of the present invention can be executed by electronic devices such as server clusters and model training machines. For example, the training method of the detection model can be applied to server clusters or other applicable hardware and software. The embodiments of the present invention do not limit this.

[0075] Reference Figure 1 The diagram illustrates a flowchart of a detection method provided by an embodiment of the present invention. The detection model includes a feature processing network and a target detection network, and may specifically include the following steps:

[0076] Step 101: Acquire the collected multi-frame environmental data.

[0077] In this embodiment of the invention, multi-frame environmental data refers to multiple frames of data continuously collected from the environment, which may include image data or other types of sensor data, such as lidar data, millimeter-wave radar data, etc. This data is continuous in time and can reflect dynamic changes in the environment.

[0078] In this embodiment of the invention, multiple frames of environmental data are collected for detection. These data contain spatiotemporal information about the environment, providing a foundation for subsequent feature extraction and fusion. Typically, this data is collected synchronously by multiple sensors on a vehicle-mounted or robotic platform.

[0079] For example, while driving, an autonomous vehicle uses surround-view cameras to collect 10 frames of images per second, forming multiple frames of environmental data.

[0080] For each frame of the multi-frame environmental data, the following operations are performed through the feature processing network:

[0081] In this embodiment of the invention, the detection model can be divided into two main parts: a feature processing part and a target detection part, wherein the feature processing part is referred to as the feature processing network and the target detection part is referred to as the target detection network.

[0082] The feature processing network is primarily responsible for extracting and fusing features from the input data to generate fused features that fully represent environmental information. It includes two key steps: feature extraction and feature fusion.

[0083] The object detection network uses fused features generated by the feature processing network to detect objects. Its main responsibility is to identify and locate targets (such as obstacles) in the environment based on the information in the fused features and output the detection results.

[0084] This is a cyclical operation, meaning that each frame of acquired environmental data undergoes a series of subsequent processing steps. The environmental data sequence is processed frame by frame, and historical information is iteratively fused to improve the detection accuracy and robustness of the current frame. Feature extraction, fusion, and cache updates are performed independently for each frame, rather than batch processing the entire sequence, simulating a real-time inference scenario. The processing of the current frame depends on the fusion result of the previous frame, achieving dynamic information accumulation and transfer.

[0085] For example, the 10 frames of environmental data acquired previously will be processed sequentially from frame 1 to frame 10.

[0086] Step 102: Extract the corresponding bird's-eye view features from the environmental data of frame T.

[0087] In this embodiment of the invention, bird's eye view (BEV) features refer to features extracted from environmental data that can reflect the three-dimensional spatial information of the scene. Typically, the original environmental data is transformed into bird's eye view features through a specific view transformation.

[0088] In this embodiment of the invention, the specific implementation of extracting the corresponding bird's-eye view features from the environmental data of frame T can include various methods, and this embodiment of the invention does not limit this. Wherein, T is a positive integer.

[0089] For example, for image data, a convolutional neural network is used to extract two-dimensional feature maps, and for point cloud data, three-dimensional voxel features are generated through voxelization. The two-dimensional or three-dimensional features are mapped to a unified bird's-eye view coordinate system to form a three-dimensional tensor (height, width, and channel), which is the bird's-eye view feature.

[0090] In one optional embodiment of the present invention, a specific implementation of extracting corresponding bird's-eye view features from the environmental data of the Tth frame may include: extracting features from the environmental data of the Tth frame to obtain a two-dimensional feature map; and converting the two-dimensional feature map into a three-dimensional bird's-eye view feature through view transformation.

[0091] Using specific algorithms or models, information representing the characteristics of the T-th frame of environmental data is extracted. In deep learning, models such as Convolutional Neural Networks (CNNs) are commonly used for feature extraction. A two-dimensional feature map refers to the extracted features being represented in a two-dimensional form, typically a matrix, where each element represents a feature value.

[0092] The environmental data of frame T is input into the feature extraction model, and high-level features are extracted from the data through multi-layer convolution, pooling and other operations of the model.

[0093] For example, suppose there is an autonomous driving scenario where the environmental data in frame T is a road image. A pre-trained convolutional neural network is used to extract features from this image, resulting in a two-dimensional feature map. This two-dimensional feature map contains important features of the image, such as lane lines and vehicle outlines.

[0094] View transformation refers to the process of converting a two-dimensional feature map into a three-dimensional feature map, usually achieved through specific transformation algorithms or models, such as geometric transformations and 3D reconstruction. 3D bird's-eye view features refer to the three-dimensional features obtained from the two-dimensional feature map, which can reflect the objects and scene structures in the environment from a bird's-eye view perspective.

[0095] Converting two-dimensional feature maps into three-dimensional bird's-eye view features can better capture the spatial position and geometric relationships of objects, providing richer information for subsequent feature fusion and obstacle detection.

[0096] For example, a view transformation algorithm can be used to convert a two-dimensional feature map into a three-dimensional bird's-eye view feature. This algorithm maps each pixel in the two-dimensional feature map to three-dimensional space based on the camera's intrinsic and extrinsic parameters, forming a three-dimensional bird's-eye view feature. This three-dimensional feature can more intuitively reflect the position of objects and scene structure on the road, such as vehicle positions and lane directions.

[0097] View transformation reduces the impact of factors such as perspective changes and occlusion, enhancing the model's robustness under different environments and conditions. This means the model can maintain stable performance even when facing complex scenes. The conversion from 2D feature maps to 3D bird's-eye views enriches the representation of features, enabling the model to understand environmental information from different angles. This helps improve the model's adaptability to complex scenes and enhances overall detection performance.

[0098] Step 103: Fuse the bird's-eye view features of frame T with the fusion features of frame T-1 in the cache to obtain the fusion features of frame T.

[0099] In this embodiment of the invention, a cache is used to store the fused features, and its contents are updated after each frame is processed. When processing frame T, the cache stores the fused features of frame T-1. During subsequent inference or training, the fused features of the previous frame stored in the cache are reused.

[0100] In this embodiment of the invention, initially, the fusion features in the cache can be set to predefined initial values ​​or random values ​​to ensure the stability of the fusion process. Alternatively, when processing the first frame, its bird's-eye view features can be directly extracted without fusion, and these features can be stored in the cache as the initial fusion features.

[0101] In this embodiment of the invention, by fusing features from frame T and frame (T-1), information can be accumulated over time, enhancing the model's understanding of dynamic scenes and its ability to capture object motion. Any applicable feature fusion method can be used, and this embodiment of the invention does not impose any limitations on this. The fused features are denoted as the fused features of frame T.

[0102] In one optional embodiment of the present invention, a specific implementation of fusing the bird's-eye view features of frame T with the fusion features of frame T-1 in the cache to obtain the fusion features of frame T may include: splicing the bird's-eye view features of frame T with the fusion features of frame T-1 according to the channel dimension to obtain spliced ​​features; and performing dimensionality reduction processing on the spliced ​​features to obtain the fusion features of frame T.

[0103] Channel concatenation refers to joining two feature maps along the channel dimension to form a new feature map. The channel dimension is a dimension of the feature map, typically representing the number of feature channels in the map. The concatenated feature map, denoted as the concatenated feature map, contains all the channel information from the original two feature maps, providing a more comprehensive representation of the input data's features.

[0104] The bird's-eye view features of frame T are integrated with the fused features of frame T-1. By stitching together the features along the channel dimension, a richer feature representation is formed, providing a foundation for subsequent dimensionality reduction and feature fusion.

[0105] Before concatenating, it's necessary to ensure that the two feature maps have the same dimensions in spatial dimensions (height and width). If the dimensions differ, adjustments can be made using methods such as upsampling or downsampling. Then, the two feature maps are concatenated along the channel dimension to form a new feature map.

[0106] For example, suppose the size of the bird's-eye view feature in frame T is 400×400×256, and the size of the fused feature in frame T-1 is 400×400×256. By stitching along the channel dimensions, the size of the resulting stitched feature is 400×400×512.

[0107] Convolutional layers are the core components of Convolutional Neural Networks (CNNs). They extract local features by sliding convolutional kernels across the input data. Dimensionality reduction refers to reducing the number of channels in the feature map through convolutional operations, thereby reducing computational complexity and extracting higher-level feature representations.

[0108] By using convolutional layers to reduce the dimensionality of the concatenated high-dimensional features, computational complexity is reduced, while higher-level feature representations are extracted, enhancing the model's understanding of dynamic scenes and its ability to capture object motion.

[0109] Convolutional layers can be used to reduce the number of channels in the concatenated features to a suitable value. For example, if the number of channels in the concatenated features is 512, it can be reduced to 256 using convolutional layers.

[0110] For example, assuming the size of the stitched feature is 400×400×512, it is reduced to 256 channels by a 1x1 convolutional layer, and the size of the fused feature of the Tth frame is 400×400×256.

[0111] By concatenating and dimensionality-reducing the bird's-eye view features of the current frame with the fused features of the previous frame, the model can acquire richer spatiotemporal information, thereby improving the detection accuracy of targets such as obstacles. Feature concatenation and dimensionality reduction can reduce the impact of factors such as changes in viewpoint and occlusion, enhancing the model's robustness under different environments and conditions. Through efficient feature concatenation and dimensionality reduction, this method can meet real-time requirements while ensuring detection accuracy, making it suitable for application scenarios requiring rapid response. Channel-dimensional concatenation and convolutional dimensionality reduction enrich the representation of features, enabling the model to understand environmental information from different perspectives and improving its adaptability to complex scenes.

[0112] In one optional embodiment of the present invention, a specific implementation of stitching the bird's-eye view feature of frame T and the fusion feature of frame T-1 along the channel dimension to obtain the stitched feature may include: aligning the fusion feature of frame T-1 to the coordinate system of frame T through coordinate system transformation to obtain the aligned fusion feature of frame T-1; and stitching the bird's-eye view feature of frame T and the aligned fusion feature of frame T-1 along the channel dimension to obtain the stitched feature.

[0113] Coordinate system transformation refers to converting data from one coordinate system to another to ensure spatial alignment and consistency. Through coordinate system transformation, two feature maps are made to have the same coordinate system in spatial dimensions, facilitating subsequent feature stitching or fusion operations.

[0114] The fused features from the previous frame are transformed from their original coordinate system to the coordinate system of the current frame to ensure spatial consistency between the two feature maps. This step is crucial for accurate feature fusion, as only features in the same coordinate system can be correctly stitched and fused.

[0115] Coordinate system transformation methods can utilize various geometric transformations, such as translation, rotation, and scaling, to transform feature maps from one coordinate system to another. Typically, transformation parameters need to be estimated based on the relative motion or transformation relationship between the two frames. This can be achieved through methods such as feature matching and optical flow estimation.

[0116] For example, suppose the fused features of frame T-1 are in coordinate system A, while the bird's-eye view features of frame T are in coordinate system B. By estimating the transformation parameters (such as translation vector and rotation angle) from coordinate system A to coordinate system B, the fused features of frame T-1 are transformed to coordinate system B, resulting in the aligned fused features of frame T-1.

[0117] Then, the bird's-eye view features of the current frame are integrated with the fused features of the previous frame after alignment. By splicing through the channel dimension, a richer feature representation is formed, which provides a foundation for subsequent dimensionality reduction and feature fusion.

[0118] Coordinate system transformation and feature stitching can reduce the impact of factors such as viewpoint changes and occlusion, enhancing the robustness of the model under different environments and conditions. Through efficient coordinate system transformation and feature stitching, this method can meet real-time requirements while ensuring detection accuracy, making it suitable for applications requiring rapid response.

[0119] Step 104: Update the fusion feature in the cache to the fusion feature of frame T.

[0120] In this embodiment of the invention, the fused features of the currently calculated T-th frame are stored for use in the processing of the next frame. The caching mechanism ensures that the fused features of the previous frame can be obtained when processing the next frame, thereby maintaining the continuity of feature fusion. Through a rolling caching mechanism, only the latest fused features are retained to reduce memory usage.

[0121] For example, after processing frame T, the fused features of frame T are stored in a buffer. When feature fusion is needed in the next frame (frame T+1), the buffer will contain the fused features of frame T.

[0122] Step 105: Process the fused features through the object detection network and output the detection results.

[0123] In this embodiment of the invention, a trained detection model, the target detection network, can identify and locate targets (e.g., obstacles) in an image based on the input fused features, and output information such as the target's position and category as the detection result.

[0124] For example, after the fused features of frame T are input into the detection head (i.e., the target detection network) of the detection model, the position and speed of vehicles and pedestrians in three-dimensional space in frame T are output.

[0125] In this embodiment of the invention, the feature processing network outputs fused features. It can process the fused features of each frame and output detection results, or it can process the fused features at intervals of several frames and output detection results, or it can process the fused features of a set frame and output detection results. Specifically, any applicable implementation method can be adopted, and this embodiment of the invention does not limit it.

[0126] In one optional embodiment of the present invention, the detection model is an obstacle detection model based on a bird's-eye view. A specific implementation of processing the fused features through a target detection network and outputting the detection result may include: inputting the fused features of the Tth frame into the target detection network of the obstacle detection model; processing the fused features of the Tth frame through the target detection network to determine the detection result; wherein the detection result includes obstacle information.

[0127] A bird's-eye view-based obstacle detection model is a trained model capable of identifying and locating obstacles in an environment based on multiple frames of environmental data. Fusion features are fed as input to the obstacle detection model's target detection network, enabling the model to identify and locate obstacles based on these features.

[0128] The target detection network is a sub-network of the obstacle detection model, specifically designed for target identification and localization based on the input feature map.

[0129] The detection results include obstacle information. This obstacle information includes the obstacle's location, type, size, etc., typically represented by a bounding box or mask, or any other applicable information; this embodiment of the invention does not impose any limitations on this.

[0130] By leveraging the computational power and learned knowledge of the object detection network, and based on the fused features of the input, the network outputs specific information about obstacles in the environment.

[0131] The object detection network analyzes the fused features of the input to identify the obstacle features contained within. Based on the analysis results, it generates information such as the location and category of the obstacles.

[0132] For example, an object detection network analyzes the fused features of the input to identify obstacles such as vehicles and pedestrians, and generates bounding boxes and category labels for each obstacle. The output might include information such as the vehicle's location coordinates, size, and confidence score.

[0133] By leveraging spatiotemporal information from fused features, object detection networks can more accurately identify and locate obstacles, reducing false positives and false negatives. The fused features contain information from multiple frames of environmental data, making the model more stable and reliable in complex scenes and changing environments. This method ensures detection accuracy while meeting real-time requirements, making it suitable for applications requiring rapid response, such as autonomous driving.

[0134] For example, such as Figure 2The diagram illustrates the iterative temporal feature fusion scheme. Image Frame Input (Img Frames): The process begins with inputting image frames at multiple timestamps, from t = T-4 to t = T. Image Encoder: Each image frame is processed by the image encoder to extract image features. 2D to 3D Conversion: The features extracted by the image encoder are converted from 2D to 3D to obtain bird's-eye view features (BEV features). BEV Temporal Feature Fusion: The BEV features of the current frame are fused with the fused BEV features of the previous frame to obtain new fused BEV features. BEV Encoder: The fused BEV features are further processed by the BEV encoder. 3D Head: Finally, the obstacle prediction results are output through the 3D head (i.e., the object detection network). In the iterative temporal feature fusion scheme, the temporal features of historical frames are continuously updated iteratively. At frame t = T-1, the fused feature (Fusion BEV feature) includes the BEV feature from frame t = T-2; similarly, the fused feature from frame t = T-2 also includes the BEV information from frame t = T-3. Taking n = 5 as an example, conventional temporal training methods require extracting image and BEV features from the panoramic images at 5 timestamps. In the BEV feature temporal fusion module, the BEV features from frame T-5 are first transformed to frame T-4 using a coordinate system transformation, and then concatenated with the BEV features extracted from frame T-4 along the channel dimension. Next, the channel dimension is reduced through convolution operations to obtain the fused feature from frame T-4, which is then output. This process is performed frame by frame until frame T.

[0135] During the deployment phase, this iterative temporal scheme only needs to cache the fused features of frame T-1, reducing memory usage to only 1 / 5 of the previous method, significantly saving real-vehicle resources. Meanwhile, the temporal features of frame T-1 theoretically contain information from all historical frames, enhancing the model's expressive power and robustness.

[0136] According to an embodiment of the present invention, multiple frames of environmental data are acquired. For each frame of the multi-frame environmental data, the following operations are performed through a feature processing network: extracting the corresponding bird's-eye view features from the T-th frame of environmental data; fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame; and updating the fusion features in the cache to the fusion features of the T-th frame. Finally, the fusion features are processed through an object detection network to output the detection results. This scheme adopts iterative temporal feature fusion, which only requires caching the fusion features of one frame, fixing the resource overhead and avoiding the linear correlation between computational resources and temporal length in the original temporal feature fusion method. At the same time, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly saving memory overhead in the actual vehicle deployment stage.

[0137] Reference Figure 3 The diagram illustrates a flowchart of a training method for a detection model according to an embodiment of the present invention. The detection model includes a feature processing network and an object detection network, and may specifically include the following steps:

[0138] Step 201: Obtain training data; wherein, the training data includes multi-frame sample environment data;

[0139] In this embodiment of the invention, training data refers to a dataset used to train the detection model, which contains multiple samples, each of which provides the information required for the model to learn.

[0140] In this embodiment of the invention, the training data includes multi-frame sample environment data. Multi-frame sample environment data refers to multiple frames of data samples continuously collected from the environment, used by the model to learn environmental features and target patterns. The training data also includes data such as real labels.

[0141] In this embodiment of the invention, a dataset for training the detection model is acquired. This dataset contains spatiotemporal information about the environment, which is used for model learning and optimization. The data source can be real-time video streams from cameras, pre-recorded video files, or data collected by other sensors. After acquiring the data, preprocessing may be required, such as adjusting the data size and normalizing the data values, to adapt to the model's input requirements.

[0142] For example, suppose there is a scenario where a vehicle's camera captures road conditions at a rate of 30 frames per second. Ten consecutive frames are acquired as training data, which will then serve as input data for subsequent model training.

[0143] In an optional embodiment of the present invention, the multi-frame sample environmental data is environmental data acquired sequentially from the same video stream in chronological order.

[0144] A video stream refers to continuous video data, which can be a real-time video stream or a pre-recorded video file. Multiple frames of sample environmental data are continuously acquired from the same video stream in chronological order to ensure the continuity of the data over time. This ensures the temporal continuity of the training data, enabling the model to learn the dynamic changes in the environmental data and the motion patterns of objects.

[0145] For example, suppose there is a scenario where a vehicle's camera captures road conditions at a rate of 30 frames per second. 1800 frames are continuously acquired from a 1-minute video stream as training data, which will then serve as input data for subsequent model training.

[0146] During the real-vehicle deployment phase, the temporal features of frame T-1 are updated iteratively to include information from all historical frames. However, the window size during the training phase is fixed at n=5, leading to inconsistencies between the training and inference phases. To address this, a video-based streaming approach was developed. This approach uses multi-frame sample environmental data, which is continuously acquired temporally from the same video stream. Training is performed on a video-by-video basis, ensuring data continuity across both video and temporal dimensions. This approach caches temporal fusion features during training. In the fusion phase, the cached fusion features from the previous frame are directly fused with the bird's-eye view features of the current frame, and the cache is updated for the next iteration. Each iteration only needs to extract the bird's-eye view features of the current frame, saving redundant computation of image feature extraction during window training and aligning with real-vehicle streaming inference.

[0147] For example, such as Figure 4 The diagram illustrates the video streaming training process. Image Frame Input (Img Frames): Input the image frame with the current timestamp, t=T. Feature Extractor: Extracts features from the current frame. BEV Feature: Obtains the BEV feature for the current timestamp. BEV Feature Fusion: Fuses the BEV feature of the current frame with the cached BEV feature of the previous frame to obtain a new fused BEV feature. Cached Fusion BEV Feature: Updates the cached fused BEV feature to the fused BEV feature of the current frame for use in the next frame.

[0148] For each frame in the multi-frame sample environment data, the following operations are performed through the feature processing network:

[0149] Step 202: Extract the corresponding bird's-eye view features from the environmental data of the T-th frame sample.

[0150] In this embodiment of the invention, converting two-dimensional sample environmental data into three-dimensional bird's-eye view features can better capture the spatial position and geometric relationships of objects, providing richer information for subsequent feature fusion. Here, T is a positive integer.

[0151] Step 203: Fuse the bird's-eye view features of frame T with the fusion features of frame T-1 in the cache to obtain the fusion features of frame T.

[0152] In this embodiment of the invention, the specific implementation of this step can be found in the description of the foregoing embodiments, and will not be repeated here.

[0153] Step 204: Update the fusion feature in the cache to the fusion feature of frame T.

[0154] In this embodiment of the invention, the specific implementation of this step can be found in the description of the foregoing embodiments, and will not be repeated here.

[0155] Step 205: Based on the fusion features, train the detection model to obtain the trained detection model.

[0156] In this embodiment of the invention, the fused features corresponding to the environmental data of each frame are input into the target detection network of the detection model. The target detection network of the detection model calculates and outputs the detection results based on the input fused features, including information such as the target's location and category.

[0157] In this embodiment of the invention, a loss function is used to measure the difference between the model output and the true labels marked in the training data, and the backpropagation algorithm is used to optimize the model parameters and minimize the loss function value.

[0158] For example, the fused features corresponding to multiple frames of sample environmental data are input into a target detection network of an obstacle detection model based on a bird's-eye view. The target detection network calculates and outputs the detection results of obstacles in each frame based on these fused features, such as the obstacle's coordinates, category, and confidence level. By comparing the model output with the true labels in the training data, the loss function value is calculated, and the model parameters are adjusted using the backpropagation algorithm to optimize model performance.

[0159] According to an embodiment of the present invention, training data containing multiple frames of sample environment data is obtained. For each frame of sample environment data, the following operations are performed through a feature processing network: extracting the corresponding bird's-eye view features from the T-th frame of sample environment data; fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame; updating the fusion features in the cache to the fusion features of the T-th frame. Finally, based on all fusion features, a detection model is trained to obtain a trained model. This scheme adopts iterative temporal feature fusion, which only requires caching the fusion features of one frame, fixing the resource overhead and avoiding the linear correlation between computational resources and temporal length in the original temporal feature fusion method. At the same time, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly saving memory overhead in the actual vehicle deployment stage.

[0160] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0161] Reference Figure 5 The diagram illustrates a structural block diagram of a detection device embodiment provided by another embodiment of the present invention. The detection model includes a feature processing network and a target detection network, and may specifically include the following modules:

[0162] Data acquisition module 301 is used to acquire multi-frame environmental data.

[0163] For each frame of the multi-frame environmental data, the following operations are performed through the feature processing network:

[0164] Feature extraction module 302 is used to extract corresponding bird's-eye view features from the environmental data of frame T; where T is a positive integer;

[0165] The feature fusion module 303 is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame.

[0166] The cache update module 304 is used to update the fusion features in the cache to the fusion features of the T-th frame;

[0167] The detection module 305 is used to process the fused features through the target detection network and output the detection results.

[0168] Optionally, the feature fusion module includes:

[0169] The stitching submodule is used to stitch the bird's-eye view features of frame T with the fusion features of frame T-1 along the channel dimension to obtain the stitched features.

[0170] The dimensionality reduction submodule is used to reduce the dimensionality of the stitched features to obtain the fused features of the Tth frame.

[0171] Optionally, the splicing submodule includes:

[0172] The transformation unit is used to align the fused features of frame T-1 to the coordinate system of frame T through coordinate system transformation, so as to obtain the aligned fused features of frame T-1.

[0173] The stitching unit is used to stitch the bird's-eye view features of frame T with the fused features of the aligned frame T-1 along the channel dimension to obtain the stitched features.

[0174] Optionally, the feature extraction module includes:

[0175] The extraction submodule is used to extract features from the environmental data of the Tth frame to obtain a two-dimensional feature map.

[0176] The transformation submodule is used to convert two-dimensional feature maps into three-dimensional bird's-eye view features.

[0177] Optionally, the detection model is an obstacle detection model based on a bird's-eye view, and the detection module includes:

[0178] The input submodule is used to input the fused features of the Tth frame into the target detection network of the obstacle detection model;

[0179] The result determination submodule is used to process the fusion features of the T-th frame through the target detection network to determine the detection result, which includes obstacle information.

[0180] According to an embodiment of the present invention, multiple frames of environmental data are acquired. For each frame of the multi-frame environmental data, the following operations are performed through a feature processing network: extracting the corresponding bird's-eye view features from the T-th frame of environmental data; fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame; and updating the fusion features in the cache to the fusion features of the T-th frame. Finally, the fusion features are processed through an object detection network to output the detection results. This scheme adopts iterative temporal feature fusion, which only requires caching the fusion features of one frame, fixing the resource overhead and avoiding the linear correlation between computational resources and temporal length in the original temporal feature fusion method. At the same time, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly saving memory overhead in the actual vehicle deployment stage.

[0181] Reference Figure 6The diagram illustrates a structural block diagram of a training device embodiment for a detection model according to another embodiment of the present invention. The detection model includes a feature processing network and an object detection network, and may specifically include the following modules:

[0182] The data acquisition module 401 is used to acquire training data, which includes multi-frame sample environment data.

[0183] For each frame in the multi-frame sample environment data, the following operations are performed through the feature processing network:

[0184] Feature extraction module 402 is used to extract corresponding bird's-eye view features from the environmental data of the T-th frame sample; where T is a positive integer;

[0185] The feature fusion module 403 is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame.

[0186] The cache update module 404 is used to update the fusion features in the cache to the fusion features of the T-th frame;

[0187] The model training module 405 is used to train the detection model based on fused features to obtain the trained detection model.

[0188] Optionally, the multi-frame sample environment data is environment data acquired sequentially from the same video stream in time order.

[0189] According to an embodiment of the present invention, training data containing multiple frames of sample environment data is obtained. For each frame of sample environment data, the following operations are performed through a feature processing network: extracting the corresponding bird's-eye view features from the T-th frame of sample environment data; fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame; updating the fusion features in the cache to the fusion features of the T-th frame. Finally, based on all fusion features, a detection model is trained to obtain a trained model. This scheme adopts iterative temporal feature fusion, which only requires caching the fusion features of one frame, fixing the resource overhead and avoiding the linear correlation between computational resources and temporal length in the original temporal feature fusion method. At the same time, iterative temporal features can represent all historical information, enhancing the model's expressive power and robustness, and significantly saving memory overhead in the actual vehicle deployment stage.

[0190] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0191] Figure 7This is a structural block diagram illustrating an electronic device 700 for detecting or training a detection model, according to an exemplary embodiment. For example, the electronic device 700 may be a vehicle-mounted computer, a computer, a digital broadcasting terminal, a messaging device, a game console, a medical device, a fitness device, etc.

[0192] Reference Figure 7 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0193] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0194] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0195] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0196] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0197] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0198] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0199] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD environmental data sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0200] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0201] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0202] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0203] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform a detection method. The detection model includes a feature processing network and a target detection network, and the method includes:

[0204] Acquire multi-frame environmental data;

[0205] For each frame of the multi-frame environmental data, the feature processing network performs the following operations:

[0206] Extract the corresponding bird's-eye view features from the environmental data of frame T; where T is a positive integer;

[0207] The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame;

[0208] Update the fusion features in the cache to the fusion features of the T-th frame;

[0209] The target detection network processes the fused features and outputs the detection results.

[0210] Optionally, fusing the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame includes:

[0211] The bird's-eye view features of the T-th frame and the fusion features of the (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features;

[0212] The splicing features are subjected to dimensionality reduction processing to obtain the fused features of the Tth frame.

[0213] Optionally, the step of stitching the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame along the channel dimension to obtain the stitched features includes:

[0214] The fused features of the (T-1)th frame are aligned to the coordinate system of the Tth frame through coordinate system transformation to obtain the aligned fused features of the (T-1)th frame.

[0215] The bird's-eye view features of the T-th frame and the fused features of the aligned (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features.

[0216] Optionally, extracting the corresponding bird's-eye view features from the environmental data of frame T includes:

[0217] Feature extraction is performed on the environmental data of the Tth frame to obtain a two-dimensional feature map;

[0218] The two-dimensional feature map is converted into the three-dimensional bird's-eye view feature.

[0219] Optionally, the detection model is an obstacle detection model based on a bird's-eye view, and the target detection network outputs detection results based on the fusion features corresponding to the multi-frame environmental data, including:

[0220] The fused features of the T-th frame are input into the target detection network of the obstacle detection model;

[0221] The target detection network processes the fusion features of the T-th frame to determine the detection result; wherein the detection result includes obstacle information.

[0222] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to execute a training method for a detection model, the detection model including a feature processing network and an object detection network, the method comprising:

[0223] Acquire training data; wherein the training data includes multi-frame sample environment data;

[0224] For each frame of the multi-frame sample environment data, the feature processing network performs the following operations:

[0225] Extract the corresponding bird's-eye view features from the environmental data of the T-th frame sample; where T is a positive integer;

[0226] The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame;

[0227] Update the fusion features in the cache to the fusion features of the T-th frame;

[0228] Based on the fusion features, the detection model is trained to obtain a trained detection model.

[0229] Optionally, the multi-frame sample environmental data is environmental data acquired sequentially from the same video stream in chronological order.

[0230] A computer program product, when run on an electronic device, causes a processor to execute one or more steps of the detection methods described in the embodiments of the present invention.

[0231] A computer program product, when run on an electronic device, causes a processor to execute steps in the training method of one or more detection models described in the embodiments of the present invention.

[0232] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0233] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0235] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0236] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0237] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0238] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0239] The foregoing has provided a detailed description of a detection method, a detection model training method, a detection device, a detection model training device, an electronic device, a readable storage medium, and a computer program product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A detection method based on a detection model, characterized in that, The detection model includes a feature processing network and an object detection network, and the method includes: Acquire multi-frame environmental data; For each frame of the multi-frame environmental data, the feature processing network performs the following operations: Extract the corresponding bird's-eye view features from the environmental data of frame T; where T is a positive integer; The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame; Update the fusion feature in the cache to the fusion feature of the T-th frame; The target detection network processes the fused features and outputs the detection results.

2. The method according to claim 1, characterized in that, The step of fusing the bird's-eye view features of frame T with the fusion features of frame T-1 in the cache to obtain the fusion features of frame T includes: The bird's-eye view features of the T-th frame and the fusion features of the (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features; The splicing features are subjected to dimensionality reduction processing to obtain the fused features of the Tth frame.

3. The method according to claim 2, characterized in that, The step of stitching together the bird's-eye view features of frame T and the fusion features of frame T-1 along the channel dimension to obtain the stitched features includes: The fused features of the (T-1)th frame are aligned to the coordinate system of the Tth frame through coordinate system transformation to obtain the aligned fused features of the (T-1)th frame. The bird's-eye view features of the T-th frame and the fused features of the aligned (T-1)-th frame are stitched together according to the channel dimension to obtain the stitched features.

4. The method according to any one of claims 1-3, characterized in that, The extraction of corresponding bird's-eye view features from the environmental data of frame T includes: Feature extraction is performed on the environmental data of the Tth frame to obtain a two-dimensional feature map; The two-dimensional feature map is converted into the three-dimensional bird's-eye view feature.

5. The method according to any one of claims 1-4, characterized in that, The detection model is an obstacle detection model based on a bird's-eye view. The target detection network processes the fused features and outputs detection results, including: The fused features of the T-th frame are input into the target detection network of the obstacle detection model; The target detection network processes the fusion features of the T-th frame to determine the detection result; wherein the detection result includes obstacle information.

6. A method for training a detection model, characterized in that, The detection model includes a feature processing network and an object detection network, and the method includes: Acquire training data; wherein the training data includes multi-frame sample environment data; For each frame of the multi-frame sample environment data, the feature processing network performs the following operations: Extract the corresponding bird's-eye view features from the environmental data of the T-th frame sample; where T is a positive integer; The bird's-eye view features of the Tth frame are fused with the fusion features of the (T-1)th frame in the cache to obtain the fusion features of the Tth frame; Update the fusion feature in the cache to the fusion feature of the T-th frame; Based on the fusion features, the detection model is trained to obtain a trained detection model.

7. The method according to claim 6, characterized in that, The multi-frame sample environment data is environment data continuously acquired from the same video stream in chronological order.

8. A detection device, characterized in that, The method includes: The data acquisition module is used to acquire multiple frames of environmental data. For each frame of the multi-frame environmental data, the feature processing network performs the following operations: The feature extraction module is used to extract the corresponding bird's-eye view features from the environmental data of the T-th frame; where T is a positive integer; The feature fusion module is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame. A cache update module is used to update the fusion features in the cache to the fusion features of the T-th frame; The detection module is used to process the fused features through the target detection network and output the detection results.

9. A training device for a detection model, characterized in that, The device includes: A data acquisition module is used to acquire training data; wherein, the training data includes multi-frame sample environment data; For each frame of the multi-frame sample environment data, the feature processing network performs the following operations: The feature extraction module is used to extract the corresponding bird's-eye view features from the environmental data of the T-th frame sample; where T is a positive integer; The feature fusion module is used to fuse the bird's-eye view features of the T-th frame with the fusion features of the (T-1)-th frame in the cache to obtain the fusion features of the T-th frame. A cache update module is used to update the fusion features in the cache to the fusion features of the T-th frame; The model training module is used to train the detection model based on the fused features to obtain the trained detection model.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

11. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-7.

12. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the processor to execute the steps of the method according to any one of claims 1-7.

Images