Beverage feature fusion method, electronic device, storage medium and program product

Abstract

One or more embodiments of the specification provide a BEV feature fusion method, an electronic device, a storage medium and a program product, the method comprising: a BEV feature fusion method, characterized by comprising: acquiring a BEV feature of a vehicle, and navigation-related information associated with the BEV feature, the BEV feature being generated according to driving environment data of the vehicle; generating a plurality of navigation heat maps based on the navigation-related information; channel splicing the plurality of navigation heat maps to obtain a multi-channel navigation heat map, and fusing the multi-channel navigation heat map with the BEV feature to obtain a fused BEV feature.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of intelligent driving, and more particularly to a BEV feature fusion method, electronic device, storage medium, and program product. Background Technology

[0002] In intelligent driving scenarios, BEV (Bird's-Eye-View) features have become a core means of fusing multi-source sensor data and achieving a unified representation of the environment. To improve the quality of predictive planning and meet requirements such as accurately matching vehicle trajectory with navigation intent, safely avoiding dynamic obstacles, and ensuring a comfortable and smooth driving experience, navigation information needs to be effectively integrated into BEV features to provide sufficient environmental and path guidance for subsequent planning. However, the fusion effect of navigation information and BEV features in existing technologies is still not ideal, and the industry has an urgent need for higher-quality fused BEV features.

[0003] In related technologies, navigation heatmaps are typically generated based on navigation path information and then directly fused with BEV features. However, as an information carrier, the single-channel representation of a navigation heatmap can only convey a limited amount of path information due to the single information type it is generated from. This makes it difficult to simultaneously address the representational needs of navigation paths in different scenarios, considering factors such as location distribution and path direction. Consequently, the fused BEV features lack sufficient environmental perception capabilities in complex driving scenarios, affecting the accuracy and robustness of subsequent prediction and planning. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to the first aspect of this specification, a BEV feature fusion method is proposed, comprising: Obtain the vehicle's BEV characteristics and navigation-related information associated with the BEV characteristics, which are generated based on the vehicle's driving environment data; Multiple navigation heatmaps are generated based on the navigation-related information. The multiple navigation heatmaps are stitched together to obtain a multi-channel navigation heatmap, and the multi-channel navigation heatmap is fused with the BEV features to obtain the fused BEV features.

[0005] According to a second aspect of this specification, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor performs the steps of the method described in the first aspect by executing the executable instructions.

[0006] According to a third aspect of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0007] According to a fourth aspect of this specification, a computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] As described in the above embodiments, this specification generates multiple navigation heatmaps based on navigation-related information, stitches them together, and then fuses the resulting multi-channel navigation heatmaps with BEV features. This allows the fusion process to simultaneously introduce multi-dimensional navigation semantic information into the BEV features, rather than being limited to a single information type. Compared to the traditional single-channel heatmap, which can only convey limited path information, the multi-channel heatmap structure of this method can carry a richer combination of navigation features. This enables the fused BEV features to have a more comprehensive environmental perception capability in complex driving scenarios, thereby providing a more reliable path guidance foundation for subsequent vehicle prediction and planning tasks. Attached Figure Description

[0009] Figure 1 This is a schematic flowchart illustrating a BEV feature fusion method according to an embodiment disclosed in this specification; Figure 2 This is a schematic diagram of a BEV feature fusion system shown in the embodiments disclosed in this specification; Figure 3 This is a flowchart illustrating another BEV feature fusion method shown in the embodiments disclosed in this specification; Figure 4 This is a schematic structural diagram of an electronic device shown in the embodiments disclosed in this specification; Figure 5 This is a block diagram of a BEV feature fusion device shown in the embodiments disclosed in this specification. Detailed Implementation

[0010] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0011] Figure 1 This is a flowchart illustrating an exemplary embodiment of a BEV feature fusion method as shown in this specification. Figure 1 As shown, the method may include the following steps: Step S102: Obtain the vehicle's BEV features and navigation-related information associated with the BEV features, wherein the BEV features are generated based on the vehicle's driving environment data.

[0012] In the solution described in this specification, the vehicle's BEV (Battery Electric Vehicle) characteristics and the navigation-related information associated with these characteristics can first be obtained. The BEV characteristics are a spatial representation obtained by performing a bird's-eye view model of the vehicle's surrounding environment, and can be generated based on the aforementioned vehicle driving environment data. The aforementioned navigation-related information refers to various types of information related to the vehicle's navigation task, which can be obtained from in-vehicle navigation systems, high-precision map interfaces, or cloud navigation services. This specification does not restrict the specific methods of obtaining these information, and the specific details will be further explained below.

[0013] The aforementioned navigation-related information and BEV features are spatially correlated, meaning they correspond to the same BEV space. In other words, a point on the navigation path corresponds to the environmental representation of its spatial location in the BEV features, eliminating the need for additional coordinate transformations or spatial alignment. This spatial consistency provides the technical basis for subsequent channel-dimensional fusion operations between the generated navigation heatmap and BEV features.

[0014] The aforementioned driving environment data can originate from sensors mounted on the vehicle itself, third-party devices in the driving environment, or a combination of both. Depending on the data source and acquisition device, the driving environment data can include various modalities, such as camera image data, LiDAR point cloud data, and millimeter-wave radar ranging data. For example, when the data originates from the vehicle itself, environmental information around the vehicle can be collected through sensors such as the vehicle-mounted forward-facing camera, panoramic camera, LiDAR, and millimeter-wave radar. When the data originates from third-party devices, perceptual information about the vehicle's driving environment can be obtained through roadside monitoring cameras, roadside LiDAR, and perception data shared with other connected vehicles. Furthermore, some environmental semantic information can also be directly obtained through vehicle-to-everything (V2X) communication; these data are essentially generated based on the collected and processed driving environment data. In short, this specification does not restrict the specific source or acquisition method of the driving environment data, as long as it can generate BEV characteristics representing the state of the vehicle's surrounding environment and corresponding environmental semantic information.

[0015] For the aforementioned BEV features, the driving environment data upon which they are based can be preprocessed to improve their quality. For example, the raw sensor data collected by the vehicle may contain noise interference, spatiotemporal inconsistencies, and occasional frame drops, which can affect the representational capability of BEV features. Therefore, before or during the generation of BEV features, preprocessing techniques such as spatiotemporal alignment, noise filtering, and inter-frame compensation can be used to optimize the driving environment data. The BEV features generated after preprocessing can exhibit different representational forms depending on the data quality and sensor configuration: when the number of sensor types meets preset conditions or the data quality is high, dense BEV features can be constructed, which continuously cover each grid area under the BEV's perspective with high-resolution feature vectors in the spatial dimension, clearly representing environmental elements such as lane line edges, static obstacle outlines, traffic participant positions, and their subtle movement trends; when data quality is limited or the sensor configuration is low, BEV features may only retain sparse keypoints or discrete instantiated objects, exhibiting a sparse representational form. Of course, regardless of the representational form used, the final optimized BEV features can be applied to the coding methods described later in this specification.

[0016] The following uses BEV characteristics (hereinafter referred to as) For example, its tensor dimension can be represented as B×T×C×H×W, which can be regarded as Figure 2 One implementation of BEV feature 210 is given, where B is the batch size, T is the number of time frames, C is the number of feature channels, and H and W are the spatial dimensions of the BEV feature map, respectively. This BEV feature, as a temporal BEV feature, contains complete information about the static environment and dynamic objects within a preset time period.

[0017] It should be noted beforehand that the above Figure 2 This is a schematic diagram illustrating a BEV feature fusion system according to an embodiment of this specification. The system is used to perform the BEV feature fusion method described in any of the foregoing embodiments. Specifically, the system may include an acquisition module, a heatmap generation module, a channel stitching module, a scene perception and smoothing module, and an SE attention enhancement module. It is understood that... Figure 2The functions of each module have been broken down and refined, therefore the corresponding modules are not directly labeled in the diagram. Specifically, the acquisition module acquires the vehicle's BEV features and associated navigation-related information, as well as the vehicle's location information mentioned below; the heatmap generation module generates multiple navigation heatmaps based on the navigation-related information and the aforementioned vehicle location information; the channel stitching module stitches the generated multiple navigation heatmaps together to obtain a multi-channel navigation heatmap; the scene perception and smoothing module identifies driving scenarios and dynamically adjusts the Gaussian kernel parameters to smooth the stitched multi-channel navigation heatmap; and the SE attention enhancement module recalibrates the channels of the fused BEV features based on navigation command information. Each of these modules can be deployed in the vehicle's onboard computing platform, with the processor executing the corresponding program instructions to implement the module's functions. In actual deployment, these modules can be integrated into the same computing unit or distributed across different processing units and work collaboratively; this specification does not restrict the specific deployment method. Through the collaborative operation of these modules, the system can effectively fuse navigation information with BEV features, outputting enhanced BEV features, providing a more comprehensive feature base for subsequent vehicle prediction and planning tasks.

[0018] Based on this, such as Figure 2 The BEV feature preprocessing operation for BEV feature 210 is shown in the figure. The preprocessing process may include the following three steps, wherein: The first step is to align the vehicle's center coordinate system and determine the vehicle's real-time position. Mapped to the center pixel position of the BEV feature map The first step is to unify the spatial reference of features across frames and eliminate inter-frame positional fluctuations caused by vehicle motion. The second step is to perform temporal normalization on the feature values ​​and map them to the [-1,1] interval through linear transformation to eliminate numerical scale differences between different frames and accelerate model convergence. The third step is to zero-fill invalid regions at the edges of the BEV feature map to avoid interference from invalid edge pixels in subsequent processing.

[0019] After the above preprocessing steps, the preprocessed time-series BEV features are output. (hereinafter referred to as) The dimensions remain B×T×C×H×W. The preprocessed BEV features are optimized in terms of spatial alignment, numerical stability, and edge handling, providing a higher-quality feature foundation for subsequent heatmap generation and fusion.

[0020] Next, let's return to the introduction of navigation-related information, which can be used to indicate the driving guidance requirements that a vehicle needs to follow to reach its destination from its current location. In subsequent steps, at least some of the navigation-related information can also be converted into a heat map for spatial alignment and fusion with BEV features.

[0021] The aforementioned navigation-related information can include navigation path location information, navigation path curvature information, navigation instruction information, and continuous navigation state information. Specifically, navigation path location information describes the geometric distribution of the navigation path in space; navigation path curvature information describes the degree of curvature of the navigation path at different locations; navigation instruction information, in discrete form, instructs the vehicle to perform the driving behavior on the current road segment, such as going straight, turning left, or turning right; and continuous navigation state information is used to characterize continuously changing indicators related to the current navigation task, such as speed limits and remaining distance. These discrete navigation instructions and continuous navigation state information, from different dimensions, together constitute the complete semantics of navigation-related information, providing diverse information sources for further enhancement of BEV features after subsequent fusion.

[0022] For ease of explanation, the navigation path location information and navigation path curvature information mentioned above can be further categorized in this specification as information related to the global navigation path, hereinafter referred to as global navigation path-related information. Together, they describe the geometric distribution and steering characteristics of the navigation path in space. The navigation command information and navigation continuous state information mentioned above can be categorized as information related to local constraints, hereinafter referred to as local constraint-related information. These two types of information describe the constraints imposed on vehicle behavior by the current navigation task, respectively, in the form of discrete commands and continuous numerical values. In summary, the essential difference between these two types of information lies in their semantic nature: global navigation path-related information focuses on the spatial morphology description of the path, while local constraint-related information focuses on the real-time constraints on vehicle driving behavior.

[0023] In one embodiment, navigation-related information can be preprocessed separately based on the differences between the global navigation path-related information and the local constraint-related information to convert it into a structured tensor form, providing standardized input for heatmap generation and even fusion in subsequent steps.

[0024] Specifically, the global navigation path information in the above navigation-related information can be recorded as the original navigation path. It may include, for example, a sequence of navigation path point coordinates corresponding to the navigation path location information mentioned above, and curvature values ​​at each path point corresponding to the navigation path curvature information mentioned above; local constraint information may be denoted as the original local constraint. It may include, for example, discrete steering commands corresponding to the aforementioned navigation command information, and continuously changing indicators such as speed limits and remaining distance corresponding to the aforementioned continuous navigation status information.

[0025] In one embodiment, for the original navigation path in navigation-related information Preprocessing operations can be performed, corresponding to Figure 2 The preprocessing steps for navigation-related information 230 are as follows. Specifically, the original navigation path can be downsampled to... Find the critical path points and calculate the curvature at each path point. Where R is the turning radius of the path at that point, and for straight path segments, its curvature can be set to zero. Then, the coordinates (x, y), heading angle θ, and curvature k of each path point are integrated into an enhanced global path feature. (hereinafter referred to as) Its dimension can be represented as B× ×4, where each row corresponds to a feature vector [x, y, θ, k] for a path point. This enhanced global path feature describes the geometry and turning characteristics of the navigation path in space in a structured data format, providing... Figure 2 The generation of the “Navigation Position Heatmap and Navigation Curvature Heatmap”232 provides a direct data foundation.

[0026] In another embodiment, for the encoding of local constraint information, discrete steering commands such as left turn, straight ahead, and right turn can be one-hot encoded and mapped to corresponding discrete vectors; continuous constraints such as speed limits and remaining distances are normalized and mapped to the [0,1] interval. Then, the encoded discrete command vectors are concatenated with the normalized continuous constraint vectors to obtain the local navigation constraint vector. (hereinafter referred to as) Its dimension can be represented as ,For example When the value is 5, it can be considered to include 3D steering command encoding and 2D continuous constraint encoding. In summary, this local navigation constraint vector comprehensively describes the constraints imposed on vehicle behavior by the current navigation task in a numerical form, and can be considered as... Figure 2 The local navigation constraint information 234 in the data provides necessary constraint guidance information for the subsequent channel recalibration of the fused BEV features and the generation of enhanced BEV features.

[0027] In other words, the above-mentioned enhanced global path features and local navigation constraint vector Together, they constitute the preprocessed navigation-related information, providing a structured input basis for the generation of multiple subsequent navigation heatmaps and channel recalibration based on navigation commands.

[0028] Step S104: Generate multiple navigation heatmaps based on the navigation-related information.

[0029] As described in the previous step, after obtaining the aforementioned navigation-related information, multiple navigation heatmaps can be generated based on this information. Each navigation heatmap represents the distribution of navigation information in the BEV space from a different perspective. By generating multiple navigation heatmaps, the various spatial semantic information required for the navigation task can be described more comprehensively.

[0030] As mentioned above, the navigation-related information may include navigation path location information and navigation path curvature information. Accordingly, the solution in this specification can generate a navigation location heatmap based on the navigation path location information, and generate a navigation curvature heatmap based on the navigation path curvature information.

[0031] On the one hand, the aforementioned navigation location heatmap can be used to characterize the spatial distribution of navigation paths in the BEV space. Specifically, each path point on the navigation path can be mapped to the spatial coordinate system corresponding to the BEV features, and a preset core weight value can be assigned to the corresponding pixel position. Furthermore, the smoothing process described below can create a continuously varying weight distribution around the path. On the other hand, the aforementioned navigation curvature heatmap is used to characterize the steering intensity of the navigation path at different positions in the BEV space. Specifically, the curvature value of each point on the navigation path can be calculated, such as the reciprocal of the turning radius. This curvature value is then normalized and mapped to the corresponding BEV pixel position, with areas exhibiting higher curvature corresponding to higher weight values.

[0032] In this way, the navigation location heatmap provides a spatial reference for the navigation path, while the navigation curvature heatmap highlights the steering semantics of curved areas within the navigation path, allowing the navigation path location information and path curvature information to be integrated into the subsequent fusion process. Thus, the navigation location heatmap and navigation curvature heatmap spatially represent navigation information from two dimensions: path geometric distribution and path steering features, providing different levels of navigation guidance information for BEV features.

[0033] In one embodiment, the generation of the navigation location heatmap and navigation curvature heatmap can be based on the enhanced global path features obtained after the preprocessing described above. This can be achieved by first enhancing global path features, specifically for navigation location heatmaps. The path point coordinates (x, y) are mapped to the BEV pixel coordinate system to obtain the corresponding pixel positions, and a preset weight value is assigned to each position to obtain the navigation position heatmap. For the navigation curvature heatmap, enhanced global path features can be extracted. The normalized curvature value k of each path point is mapped to the corresponding BEV pixel position. Regions with greater curvature receive higher weight values, ultimately generating a navigation curvature heatmap that highlights the turning semantics of curves. Thus, as... Figure 2 As shown in the “Navigation Position Heatmap and Navigation Curvature Heatmap” 232, the two provide structured navigation spatial information for the subsequent generation of fused BEV features 242 from the two dimensions of path geometric distribution and path turning features, respectively.

[0034] It's worth noting that, in addition to generating multiple navigation heatmaps based on navigation-related information, vehicle location information can also be further incorporated, corresponding to... Figure 2 The self-driving navigation positioning heatmap 220. Specifically, the BEV feature fusion system first obtains the vehicle's self-driving location information, which is associated with the aforementioned BEV features, meaning both correspond to the same BEV space. Based on this, at least one navigation heatmap can be generated based on navigation-related information, such as the aforementioned navigation location heatmap and / or the aforementioned navigation curvature heatmap. Simultaneously, a self-driving navigation positioning heatmap is generated based on the self-driving location information, corresponding to... Figure 2 The vehicle navigation positioning heatmap 222 is used as an additional navigation heatmap, and together with other navigation heatmaps generated based on navigation-related information, it constitutes the above-mentioned multiple navigation heatmaps.

[0035] It should be noted that the aforementioned vehicle location information is not part of the navigation-related information. Navigation-related information originates from navigation systems or map services and describes the driving route the vehicle should follow and related guidance requirements; while vehicle location information describes the vehicle's current position within the BEV space. The two differ in their information source and semantic attributes. This specification logically distinguishes between the two; however, in practical implementation, the methods for obtaining vehicle location information and navigation-related information can be the same or share some data sources. For example, vehicle location information can be obtained through onboard positioning systems such as Global Positioning System (GPS), Inertial Measurement Unit (IMU), odometers, visual odometers, or laser SLAM, or through high-precision map interfaces or cloud positioning services. These acquisition methods can overlap with the sources of navigation-related information. This specification does not restrict the specific methods for obtaining vehicle location information.

[0036] Of course, after obtaining the above vehicle location information and before generating the vehicle navigation positioning heatmap, the vehicle location information itself can be preprocessed in the same way as the above navigation-related information.

[0037] In one embodiment, the aforementioned vehicle position information can be recorded as the vehicle state, which includes key state parameters of the vehicle during its driving process: the vehicle's lateral position in the global coordinate system. , y-axis position ,speed acceleration and heading angle The preprocessing process may include the following steps: First, perform spatial coordinate alignment, aligning the vehicle coordinates... Mapping to the BEV vehicle's central coordinate system and fixing it at the origin (0,0), the vehicle's current position is used as the spatial reference benchmark. This ensures that the spatial position represented by the preprocessed vehicle position information is always centered on the vehicle, thus maintaining consistency with the spatial benchmark of the aforementioned BEV features. Secondly, continuous state parameters such as velocity and acceleration are normalized and scaled to the [0,1] interval to match the numerical scale of the continuous constraint vectors in the preprocessed navigation-related information. Finally, the normalized velocity, acceleration, and heading angle are integrated to generate the vehicle state tensor. Its dimension can be represented as B× ,For example When =4, it is equivalent to including the origin position marker (0,0) and velocity. acceleration and heading angle After the above preprocessing, the vehicle state tensor carries key information such as the vehicle's position, attitude, and motion state in the BEV space in the form of a structured multidimensional array, and therefore can be directly used to generate the subsequent vehicle navigation and positioning heatmap.

[0038] When generating a vehicle navigation positioning heatmap based on the aforementioned vehicle location information, a heatmap with a preset spatial distribution, such as a Gaussian distribution, can be generated in the BEV space, centered on the pixel location corresponding to the vehicle. This results in the vehicle's location and its surrounding area receiving higher weight values, while the weight of areas farther away from the vehicle gradually decreases. This vehicle navigation positioning heatmap can clearly define the vehicle's spatial reference position in the BEV space, enabling the BEV features to maintain a clear anchorage of the vehicle's spatial position while perceiving environmental information during subsequent fusion processes.

[0039] In one embodiment, the generation of the aforementioned vehicle navigation and positioning heatmap can be based on the vehicle state tensor obtained after the preprocessing described above. To achieve this, the heatmap is generated using the center pixel of the BEV feature map as the origin of the vehicle's position and the Gaussian distribution formula.

[0040] Thus, the aforementioned vehicle navigation positioning heatmap, as an additional navigation heatmap, can work in conjunction with navigation location heatmaps and navigation curvature heatmaps generated based on navigation-related information to jointly constitute a spatial semantic description of the navigation task from multiple perspectives, including vehicle positioning, path geometry, and steering features.

[0041] Step S106: The multiple navigation heatmaps are spliced ​​together to obtain a multi-channel navigation heatmap, and the multi-channel navigation heatmap is fused with the BEV features to obtain the fused BEV features.

[0042] Finally, the previously generated navigation heatmaps are concatenated by channel to form a multi-channel navigation heatmap. This multi-channel navigation heatmap is then fused with BEV features to obtain the fused BEV features. Through channel concatenation, the multi-dimensional information from multiple navigation heatmaps is unified into a single tensor structure, which is then fused with the acquired BEV features. This results in fused BEV features that retain the original environmental perception information while incorporating rich navigation guidance information, providing a more comprehensive feature base for subsequent vehicle prediction and planning tasks.

[0043] Taking the scenario where the navigation location heatmap, navigation curvature heatmap, and vehicle navigation positioning heatmap have all been generated separately as an example, specifically, each of these three heatmaps describes the spatial semantic information required for the navigation task from different dimensions: the navigation location heatmap provides the spatial reference for the navigation path, the navigation curvature heatmap highlights the steering semantics of the curve area, and the vehicle navigation positioning heatmap clarifies the vehicle's positioning reference in the BEV space. Based on this, the three heatmaps can be stitched together along the channel dimension; this step can be considered as... Figure 2 The operation of "striking multiple navigation heatmaps by channel" in the middle outputs a multi-channel navigation heatmap. (hereinafter referred to as) Its dimensions can be represented as B×3×H'×W', where H' and W' are the spatial dimensions of the heatmap, corresponding to the spatial dimensions of the BEV feature map, and the so-called "3" corresponds to the number of channels of the three heatmaps mentioned above.

[0044] It is worth mentioning that this specification first concatenates multiple single-channel navigation heatmaps along the channel dimension into a multi-channel tensor, and then fuses the multi-channel navigation heatmap with BEV features. This method avoids the increased complexity of the fusion operation and the lack of a unified tensor structure to support the information from each heatmap during the fusion process, which are caused by fusing multiple single-channel heatmaps with BEV features separately. This allows the navigation heatmap and BEV features to be integrated in a unified and efficient manner while maintaining alignment along the channel dimension.

[0045] Furthermore, before fusing the aforementioned multi-channel navigation heatmap with BEV features, the multi-channel navigation heatmap can be smoothed, i.e., corresponding to... Figure 2 The "smoothing of multi-channel navigation heatmaps" operation optimizes the spatial weight distribution of the heatmaps to better suit the needs of actual driving scenarios. Several examples are provided below to illustrate various smoothing methods.

[0046] In one embodiment, the heatmap generation can employ a binary representation method. Specifically, after mapping navigation path points to the BEV space, the pixel positions where the path is located are directly assigned a preset non-zero weight value, such as 1, while the weights of other non-path regions are reset to zero, thereby forming a binary heatmap that only marks the path positions. This binary heatmap can mark the geometric position of the navigation path in the BEV space in the simplest way, providing a clear path position reference for subsequent fusion operations. This method is simple to implement, has low computational overhead, and is suitable for scenarios where high heatmap accuracy is not required or computational resources are limited.

[0047] In another embodiment, considering that the binarized representation in the above embodiments only has a hard distinction of "presence or absence of a path" and lacks a spatial transition region, the heatmap is not sensitive to changes in the surrounding environment. In narrow-channel scenarios such as congestion and parking, the binarized representation cannot reflect the spatial relationship between the path and surrounding obstacles, and the lack of a smooth transition between path information and surrounding environment information is not conducive to the perception of environmental context in the subsequent fusion process. To address this, this embodiment can use fixed Gaussian kernel parameters to smooth the multi-channel navigation heatmap. Specifically, after generating the multi-channel navigation heatmap, the heatmap can be convolved and smoothed with a Gaussian kernel with a fixed standard deviation σ, so that the surrounding area of ​​the path forms a continuously gradually changing weight distribution. This method introduces a spatial gradient, enabling the corresponding heatmap to have a certain perception ability of the surrounding environment of the path. The weights gradually decrease from the center of the path to both sides, providing richer spatial guidance information for the subsequent fusion process.

[0048] In another embodiment, while the smoothing method using fixed Gaussian kernel parameters in the above embodiments improves the spatial transition problem of binarized representation, its Gaussian kernel standard deviation σ is a fixed value and does not adjust with changes in the driving scenario. This results in the following: in narrow-lane scenarios such as congestion and parking, the fixed Gaussian kernel coverage is too wide, which can easily introduce redundant information from the surrounding area and interfere with core path perception; in open scenarios such as highways and long straight roads, the fixed Gaussian kernel coverage is too narrow, which cannot fully capture dynamic traffic participants and static environmental elements around the path, resulting in insufficient safety redundancy.

[0049] To address the aforementioned issues, this specification proposes a scene-adaptive dynamic Gaussian kernel smoothing mechanism. Specifically, the scene type of the current driving scenario of the vehicle is first identified, and the corresponding Gaussian kernel parameter, i.e., the Gaussian kernel standard deviation σ, is determined based on the scene type. Then, the multi-channel navigation heatmap is smoothed based on this Gaussian kernel parameter to obtain a smoothed multi-channel navigation heatmap.

[0050] The aforementioned scenario types can be categorized based on the spatial openness of the driving scenario, and the value of the Gaussian kernel parameter is positively correlated with the spatial openness. When the vehicle is in a narrow passage scenario with low spatial openness, a smaller standard deviation σ is selected to make the heatmap coverage relatively concentrated and reduce redundant information interference; when the vehicle is in an open scenario with high spatial openness, a larger standard deviation σ is selected to appropriately expand the heatmap coverage to capture more environmental information and provide more sufficient safety redundancy for the subsequent fusion process. As for the specific rules for classifying spatial openness, they can be dynamically configured according to actual needs. For example, a classification threshold can be set based on indicators such as the width of the passable area, obstacle density, or number of lanes, and the total number of scenario types after classification can be determined. This specification does not impose any restrictions on this.

[0051] For example, after identifying the driving scenario corresponding to the vehicle, a corresponding scenario type flag can be output, indicating the current scenario category of the vehicle. Depending on the spatial openness of different scenarios, different levels of Gaussian kernel standard deviation σ can be preset: for scenarios with high spatial openness, such as highways or long straight roads, a larger standard deviation σmax, such as 3 to 5, can be used to expand the heatmap coverage and fully capture dynamic traffic participants and static environmental elements around the path. For narrow passage scenarios with low spatial openness, such as congestion or parking, a smaller standard deviation σmin, such as 1 to 2, can be used to reduce the heatmap coverage and minimize interference from redundant information on the core path perception. For scenarios with moderate spatial openness, such as ordinary urban roads, a median value σmid, such as 2 to 3, can be used to achieve a balance between coverage and accuracy. After determining the standard deviation σ corresponding to the current scenario type, this dynamically configured standard deviation σ can be reapplied to the Gaussian smoothing process of the multi-channel navigation heatmap, thereby outputting a scenario-adapted multi-channel navigation heatmap. (hereinafter referred to as) ), corresponding to Figure 2 The scene-adapted multi-channel navigation heatmap 240 is shown in the figure. Through the above method, the three heatmaps can automatically adjust their spatial coverage under different driving scenarios, achieving dynamic adaptation between heatmap representation and scene features. Of course, Gaussian smoothing can be replaced by other smoothing methods such as mean filtering; simply replace the corresponding Gaussian kernel parameters with the parameters required by the corresponding smoothing method. This specification does not impose any limitations on this, but for the convenience of subsequent embodiments, Gaussian smoothing will be used by default below.

[0052] Therefore, through the scene-adaptive dynamic Gaussian kernel smoothing mechanism in the above embodiments, the navigation heatmap can maintain a reasonable coverage range in driving scenarios with different spatial characteristics. This avoids information redundancy interference in narrow passage scenarios and makes up for insufficient safety redundancy in open scenarios, thereby improving the overall representation capability of the multi-channel navigation heatmap after fusion with BEV features.

[0053] It should be noted that the above smoothing process can also be performed independently on each navigation heatmap before channel stitching. However, for the same vehicle, each navigation heatmap faces the same driving scenario at the same time, and the Gaussian kernel parameters determined based on this scenario are also consistent. In this case, if smoothing is performed independently on each heatmap, multiple Gaussian convolution operations need to be repeated. Therefore, in the above embodiments, this specification preferentially chooses to first stitch the navigation heatmaps together, and then perform unified smoothing on the stitched multi-channel navigation heatmaps, thereby avoiding the redundant calculation overhead caused by smoothing each heatmap independently, and improving the overall processing efficiency while obtaining the same smoothing effect. Of course, those skilled in the art can flexibly adjust the execution order of the smoothing process according to actual needs, and this specification does not impose specific restrictions on this.

[0054] It is worth mentioning that before fusing the multi-channel navigation heatmap with the BEV features, further time-series encoding processing can be performed on the preprocessed time-series BEV features, i.e., corresponding to... Figure 2 The spatiotemporal coding operation is used to extract local spatiotemporal correlation information from BEV features. In one embodiment, the preprocessed time-series BEV features can be... The data is processed sequentially using convolution and Transformer operations. Convolution extracts local spatial features within the BEV features of each frame, while Transformer models the temporal dependencies between different frames and performs spatial downsampling in the process. After this processing, the temporally fused BEV features are output. Its temporal dimension information has been effectively integrated and can be regarded as Figure 2 The encoded BEV feature 212 provides a more compact temporal feature representation for subsequent stitching and fusion with multi-channel navigation heatmaps. It should also be noted that the temporal encoding method combining convolution and Transformer described above is merely an exemplary implementation provided in this specification. Besides this method, those skilled in the art can use other network structures capable of capturing temporal dependencies to implement the above temporal encoding process, such as 3D convolution, recurrent neural networks, or variations thereof. This specification does not impose any limitations on this approach.

[0055] Regarding the fusion process of the multi-channel navigation heatmap and the BEV features, in one embodiment, the above-mentioned channel stitching and fusion process can be further implemented in the following way: The multi-channel navigation heatmap adapted to the above scenario can be... Repeat T times along the temporal dimension to make it consistent with the aforementioned temporally fused BEV features in the temporal dimension. To maintain consistency, the repeated multi-channel navigation heatmap and the time-series fused BEV features are then stitched together along the channel dimension to obtain the fused features. It can be regarded as Figure 2 The fused BEV features 242. Through the above method, the multi-dimensional navigation spatial information such as path location, path curvature and vehicle positioning carried by the multi-channel navigation heatmap are uniformly integrated with the environmental perception temporal information carried by the temporal fusion BEV features in the channel dimension.

[0056] In addition to obtaining the aforementioned fused BEV features, the solution in this specification can further enhance the fused BEV features by utilizing the navigation command information in the aforementioned navigation-related information. This allows the fused BEV features to respond more accurately to navigation command guidance, thereby improving the accuracy of subsequent vehicle execution tasks such as predictive planning. Specifically, local navigation constraint information can be determined based on the aforementioned navigation command information and mapped to a high-dimensional constraint vector matching the channel dimensions of the fused BEV features. Then, based on this high-dimensional constraint vector, the channel recalibration of the fused BEV features is performed using the SE (Squeeze-and-Excitation) attention mechanism. That is, the response of each channel of the fused BEV features is recalibrated to obtain enhanced BEV features, which can be considered as... Figure 2 Enhanced BEV features 244.

[0057] The process of determining local navigation constraint information based on navigation instruction information can be further specified as the following steps: After obtaining the local navigation constraint information, it can be mapped into a high-dimensional constraint vector that matches the channel dimension of the fused BEV features through linear transformations or other methods. Subsequently, based on this high-dimensional constraint vector, the channels of the fused BEV features are recalibrated using the SE attention mechanism.

[0058] In one embodiment, the above-mentioned channel recalibration process can be further implemented in the following way: First, the aforementioned local navigation constraint vector is... The linear transformation maps to the fusion features. High-dimensional constraint vectors for channel-dimensional matching; simultaneously, for fused features Global average pooling is performed to obtain global statistical information for each channel. Then, the high-dimensional constraint vector is concatenated with the global statistical information and input into a multilayer perceptron. After activation by the Sigmoid function, the weight coefficients corresponding to each channel are generated. Finally, these weight coefficients can be applied to the fused features using Hadamard product. On each channel, differentiated weighting is implemented for different channels to output enhanced BEV characteristics. In the above process, linear transformation can be implemented using fully connected layers, and global average pooling can be replaced by alternative methods such as global max pooling. The number of layers and nodes of the multilayer perceptron can also be configured according to actual needs, and this specification does not impose any restrictions on this.

[0059] Through the aforementioned channel recalibration process, feature channels closely related to the current navigation command will be given higher weights, while feature channels unrelated to or weakly related to the navigation task corresponding to the vehicle will be suppressed. This enables the enhanced BEV features to respond more accurately to the guidance of navigation commands, providing more targeted feature representations for subsequent prediction and planning tasks.

[0060] It should be noted that the SE attention mechanism described above is only an exemplary implementation provided in this specification. Those skilled in the art can select other channel attention mechanisms or spatial attention mechanisms to replace it according to actual needs. For example, the SE attention mechanism can be replaced with a convolutional block attention module or an efficient channel attention module. This specification does not impose any restrictions on this.

[0061] Figure 3 This is a flowchart illustrating another BEV feature fusion method as shown in an exemplary embodiment of this specification. Figure 3 As shown, the method may include the following steps: Step S302: Obtain the vehicle's BEV characteristics and related navigation information.

[0062] In one embodiment, the acquisition module of the BEV feature fusion system can collect driving environment data from onboard sensor systems, such as a forward-facing camera, a panoramic camera, and a lidar system. After preprocessing the data, including spatiotemporal alignment and noise filtering, BEV features are generated based on the preprocessed data. The dimensions of these features can be represented as B×T×C×H×W. Simultaneously, the acquisition module can obtain navigation-related information associated with the BEV features from a navigation system or map service. This navigation-related information includes navigation path location information and navigation path curvature information, both of which correspond to the same BEV space as the BEV features.

[0063] Step S304: Generate a navigation location heatmap based on the navigation path location information.

[0064] In one embodiment, the acquisition module of the BEV feature fusion system can collect driving environment data from onboard sensor systems, such as a forward-looking camera, a panoramic camera, and a lidar. After preprocessing the data, including spatiotemporal alignment and noise filtering, BEV features are generated based on the preprocessed data. These BEV features are spatial representations obtained by modeling the vehicle's surrounding environment from a bird's-eye view. Their dimensions can be represented as B×T×C×H×W, where B is the batch size, T is the number of time-series frames (assumed to be 5 to 10 frames), C is the number of feature channels (assumed to be 256), and H and W are the spatial dimensions of the BEV feature map (assumed to be 400×200). Simultaneously, the acquisition module obtains navigation-related information associated with the BEV features from a navigation system or map service. This navigation-related information includes navigation path location information and navigation path curvature information, both of which correspond to the same BEV space as the BEV features.

[0065] Step S306: Generate a navigation curvature heatmap based on the navigation path curvature information.

[0066] In one embodiment, the heatmap generation module of the above system can map the coordinates (x, y) of each path point in the navigation path location information to the BEV pixel coordinate system, assign a preset core weight value of "1" to the corresponding pixel position, and reset the weights of the remaining areas around the path to zero, forming a navigation location heatmap that represents the spatial distribution of the navigation path. This navigation location heatmap marks the geometric position of the navigation path in the BEV space in the simplest way, providing a spatial reference for subsequent operations.

[0067] Step S308: The above multiple navigation heatmaps are spliced ​​together to obtain a multi-channel navigation heatmap.

[0068] In one embodiment, the channel stitching module of the above system stitches the navigation position heatmap generated in step S304 with the navigation curvature heatmap generated in step S306 along the channel dimension. Since each heatmap is a single-channel two-dimensional tensor with a size of H'×W', corresponding to the size of the BEV feature map space, the stitched image forms a two-channel multi-channel navigation heatmap, the dimension of which can be represented as B×2×H'×W'. Through the above stitching operation, the path position information and path curvature information are uniformly integrated into the same tensor structure.

[0069] Step S310: Smooth the multi-channel navigation heatmap to obtain a smoothed multi-channel navigation heatmap.

[0070] In one embodiment, the scene perception and smoothing module of the above system can identify the scene type of the current driving scenario and determine the corresponding Gaussian kernel standard deviation σ based on the scene type. For example, in a high-speed scenario with a high degree of openness, a larger σ, such as 3 to 5, is used. Subsequently, the determined Gaussian kernel is used to perform convolutional smoothing on the multi-channel navigation heatmap obtained in step S308, causing the weight distribution in the heatmap to gradually transition from the path center to both sides, resulting in a smoothed multi-channel navigation heatmap. Through this smoothing process, the spatial coverage of the multi-channel navigation heatmap matches the spatial characteristics of the current driving scenario.

[0071] Step S312: The smoothed multi-channel navigation heatmap is fused with the BEV features to obtain the fused BEV features.

[0072] In one embodiment, the system channel stitching module stitches the smoothed multi-channel navigation heatmap obtained in step S310 with the BEV features obtained in step S302 along the channel dimension. Specifically, the multi-channel navigation heatmap can be repeated T times along the temporal dimension to ensure consistency with the BEV features in the temporal dimension, and then the two can be stitched along the channel dimension to obtain fused features. Through the above fusion operation, the BEV features retain original environmental perception information such as lane lines, obstacles, and traffic participants, while also incorporating multi-dimensional navigation guidance information such as navigation path position and path curvature that have undergone scene adaptive smoothing. The obtained fused BEV features can be used for subsequent vehicle prediction and planning tasks, providing a more comprehensive feature foundation for trajectory planning.

[0073] Figure 4 This is a schematic structural diagram of an electronic device according to an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile storage, and may also include other necessary hardware. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it, forming a BEV feature fusion device at the logical level. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0074] Figure 5 This specification illustrates a block diagram of a BEV feature fusion device through embodiments. Please refer to... Figure 5 The device includes: The sample acquisition unit 502 is used to acquire the BEV features of the vehicle and navigation-related information associated with the BEV features, wherein the BEV features are generated based on the vehicle's driving environment data. Model processing unit 504 is used to generate multiple navigation heatmaps based on the navigation-related information; The model update unit 506 is used to perform channel stitching on the multiple navigation heatmaps to obtain a multi-channel navigation heatmap, and to fuse the multi-channel navigation heatmap with the BEV features to obtain the fused BEV features.

[0075] Optionally, the navigation-related information includes navigation path location information and navigation path curvature information; the model processing unit 504 is specifically used for: A navigation location heatmap is generated based on the navigation path location information, and a navigation curvature heatmap is generated based on the navigation path curvature information.

[0076] Optionally, the model processing unit 504 is specifically used for: Obtain the vehicle's location information, which is associated with the BEV features; At least one navigation heatmap is generated based on the navigation-related information, and a vehicle navigation positioning heatmap is generated based on the vehicle location information as an additional navigation heatmap.

[0077] Optionally, the device further includes: The heatmap smoothing unit is used to identify the scene type of the driving scenario in which the vehicle is currently located; Based on the scene type, determine the Gaussian kernel parameters; Based on the Gaussian kernel parameters, the multi-channel navigation heatmap is smoothed to obtain a smoothed navigation heatmap.

[0078] Optionally, the scene type is classified based on the spatial openness of the driving scene; the value of the Gaussian kernel parameter is positively correlated with the spatial openness.

[0079] Optionally, the navigation-related information includes navigation instruction information; the device further includes: A feature recalibration unit is used to determine local navigation constraint information based on the navigation instruction information; The local navigation constraint information is mapped into a high-dimensional constraint vector that matches the dimension of the fused BEV feature channel. Based on the high-dimensional constraint vector, the channel recalibration of the fused BEV features is performed through the SE attention mechanism to obtain enhanced BEV features.

[0080] Optionally, the navigation-related information further includes navigation continuous state information; the navigation continuous state information is used to characterize continuously changing indicators related to the vehicle's navigation task; the feature recalibration unit is specifically used for: The discrete navigation instructions in the navigation instruction information are encoded to obtain a discrete instruction vector; The navigation continuous state information is normalized to obtain a continuous state vector; The discrete command vector is concatenated with the continuous state vector to form the local navigation constraint information.

[0081] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0082] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0083] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0084] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0085] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0086] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0087] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0088] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0089] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0090] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0091] Therefore, specific embodiments of the subject matter have been described. Furthermore, the processes depicted in the figures are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0092] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A BEV feature fusion method, characterized in that, include: Obtain the vehicle's BEV characteristics and navigation-related information associated with the BEV characteristics, which are generated based on the vehicle's driving environment data; Multiple navigation heatmaps are generated based on the navigation-related information. The multiple navigation heatmaps are stitched together to obtain a multi-channel navigation heatmap, and the multi-channel navigation heatmap is fused with the BEV features to obtain the fused BEV features.

2. The method according to claim 1, characterized in that, The navigation-related information includes navigation path location information and navigation path curvature information; The generation of multiple navigation heatmaps based on the navigation-related information includes: A navigation location heatmap is generated based on the navigation path location information, and a navigation curvature heatmap is generated based on the navigation path curvature information.

3. The method according to claim 1, characterized in that, The generation of multiple navigation heatmaps based on the navigation-related information includes: Obtain the vehicle's location information, which is associated with the BEV features; At least one navigation heatmap is generated based on the navigation-related information, and a vehicle navigation positioning heatmap is generated based on the vehicle location information as an additional navigation heatmap.

4. The method according to claim 1, characterized in that, The method further includes: Identify the scene type of the current driving scenario of the vehicle; Based on the scene type, determine the Gaussian kernel parameters; Based on the Gaussian kernel parameters, the multi-channel navigation heatmap is smoothed to obtain a smoothed navigation heatmap.

5. The method according to claim 4, characterized in that, The scene type is classified based on the spatial openness of the driving scene; the value of the Gaussian kernel parameter is positively correlated with the spatial openness.

6. The method according to claim 1, characterized in that, The navigation-related information includes navigation instruction information; the method further includes: Local navigation constraint information is determined based on the navigation instruction information; The local navigation constraint information is mapped into a high-dimensional constraint vector that matches the dimension of the fused BEV feature channel. Based on the high-dimensional constraint vector, the channel recalibration of the fused BEV features is performed through the SE attention mechanism to obtain enhanced BEV features.

7. The method according to claim 6, characterized in that, The navigation-related information also includes navigation continuous state information; the navigation continuous state information is used to characterize continuously changing indicators related to the vehicle's navigation task; determining local navigation constraint information based on the navigation instruction information includes: The discrete navigation instructions in the navigation instruction information are encoded to obtain a discrete instruction vector; The navigation continuous state information is normalized to obtain a continuous state vector; The discrete command vector is concatenated with the continuous state vector to form the local navigation constraint information.

8. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images