A prediction method, readable storage medium, program product and vehicle-mounted device

Abstract

The application relates to the field of intelligent driving, in particular to an obstacle trajectory prediction method, a storage medium, a program product and a vehicle-mounted device. The obstacle trajectory prediction method comprises the following steps: acquiring image features of N images of an environment where a vehicle is located at a first moment and a current BEV feature, wherein the current BEV feature is obtained based on the image features of the N images and / or point cloud data of the environment where the vehicle is located, N is an integer greater than or equal to 1; obtaining first information based on the image features of the N images and the current BEV feature, historical image features of the environment where the vehicle is located at at least one historical moment and historical BEV features of the environment where the vehicle is located at at least one historical moment, wherein the first information comprises a motion trajectory of at least one obstacle in the environment where the vehicle is located after the first moment. In this way, the motion trajectory of the obstacle can be accurately predicted.

Description

Technical Field

[0001] This application relates to the field of driver assistance systems, and in particular to a prediction method, a readable storage medium, a program product, and an in-vehicle device. Background Technology

[0002] Traditional driver assistance systems typically utilize independent perception and prediction models to detect and predict the position of dynamic obstacles in real time. Specifically, the perception model identifies information such as the obstacle's position, speed, yaw angle, and size based on sensor data, and then sends this information to the prediction model. The prediction model can then predict the obstacle's future trajectory based on this information.

[0003] However, since the predictive model relies on information provided by the perception model to predict the future trajectory of the obstacle, if the information provided by the perception model is inaccurate, the predictive model may not be able to accurately predict the trajectory of the obstacle, which may lead to decision-making errors in the assisted driving vehicle and increase safety risks. Summary of the Invention

[0004] This application provides a prediction method, a readable storage medium, a program product, and an in-vehicle device, which enables the determination of prediction results without relying on obstacle perception results, but can directly determine more accurate obstacle-related perception and prediction results based on obstacle-related image features and BEV features.

[0005] In a first aspect, this application provides a prediction method applied to an in-vehicle device on a vehicle. The method includes: acquiring image features of N images of the environment in which the vehicle is located at a first moment and current BEV features, wherein the current BEV features are obtained based on the image features of the N images and / or point cloud data of the environment in which the vehicle is located, and N is an integer greater than or equal to 1; obtaining first information based on the image features of the N images and the current BEV features, historical image features of the environment in which the vehicle is located at at least one historical moment, and historical BEV features of the environment in which the vehicle is located at at least one historical moment, wherein the first information includes the motion trajectory of at least one obstacle in the environment in which the vehicle is located after the first moment.

[0006] In this embodiment of the application, first information can be obtained by combining the image features of N images with the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment. The first information includes the trajectory of at least one obstacle in the vehicle's environment after the first moment.

[0007] Thus, compared to traditional driver assistance systems that first use a perception model to obtain the obstacle's pose, which contains errors and makes it impossible to accurately predict the obstacle's trajectory based on that pose, this application's embodiment directly inputs N images corresponding to the vehicle's environment at the same moment, current BEV features, and historical BEV features from at least one historical moment into the perception prediction network. The perception prediction network then predicts the obstacle's future trajectory using these features, without relying on perception results (such as the obstacle's pose), making the predicted obstacle's future trajectory more accurate. It should be noted that the perception prediction network can refer to the first model mentioned in this application's embodiment.

[0008] It should be noted that the N images can be images captured by different cameras of the in-vehicle equipment at the first moment, showing the surrounding environment of the vehicle. These different cameras can include forward-looking narrow-angle, forward-looking wide-angle, panoramic (front / rear / left / right), and rear-view images, etc. It should be understood that each camera captures an image corresponding to a different perspective, and at least one of the N images can include at least one obstacle. The image features of the N images can be obtained by extracting features from the N images using a backbone network and a neck network; these image features are also known as multi-image features. Furthermore, the point cloud data can be a set of three-dimensional coordinates captured by the in-vehicle equipment's LiDAR and / or millimeter-wave radar at the first moment, showing the surrounding environment of the vehicle.

[0009] In one possible implementation of the first aspect described above, obtaining first information based on image features of N images and current BEV features, historical image features of the vehicle's environment at at least one historical moment, and historical BEV features of the vehicle's environment at at least one historical moment includes: inputting image features of N images and current BEV features, historical image features of the vehicle's environment at at least one historical moment, and historical BEV features of the vehicle's environment at at least one historical moment into a first model to obtain first information, wherein the first information further includes the pose of at least one obstacle at a first moment, and the pose of at least one obstacle includes the three-dimensional position, size, orientation angle, and velocity of at least one obstacle.

[0010] Thus, compared to traditional driver assistance systems that obtain obstacle poses by inputting multiple image features into a perception model, the pose error is large. However, in the embodiments of this application, by inputting N images corresponding to the vehicle's environment at the same moment, the current BEV features (image-point cloud fusion), and at least one historical BEV feature from a historical moment, the perceived obstacle pose is more accurate.

[0011] Based on the above, the prediction method provided in this application combines N image features from the same moment with the current BEV features (image-point cloud fusion) and historical BEV features, inputting them into a lightweight first model. By fusing point cloud information of the vehicle's environment, more accurate perception results (pose) and prediction results (future trajectory) for each obstacle can be obtained, avoiding the accumulation of pose errors in the traditional "perception first, prediction second" approach. In addition, the lightweight first model requires less computational power.

[0012] In one possible implementation of the first aspect above, the current BEV feature is obtained in the following way: the BEV feature corresponding to N images is obtained based on the image features of N images, and the BEV feature corresponding to the point cloud data is obtained based on the point cloud data; the current BEV feature is obtained based on the BEV feature corresponding to N images and the BEV feature corresponding to the point cloud data.

[0013] In some embodiments of this application, the current BEV feature can be the BEV feature obtained by fusing the BEV features corresponding to N images and the BEV features corresponding to point cloud data, followed by dimensionality reduction processing.

[0014] In other embodiments of this application, the current BEV feature can be the BEV features corresponding to N images.

[0015] In one possible implementation of the first aspect above, the current BEV feature is obtained based on the BEV features corresponding to N images and the BEV features corresponding to point cloud data, including: performing feature concatenation and dimensionality reduction processing on the BEV features corresponding to N images and the BEV features corresponding to point cloud data to obtain the current BEV feature.

[0016] In one possible implementation of the first aspect described above, first information is obtained based on image features of N images and current BEV features, historical image features of the vehicle's environment at at least one historical moment, and historical BEV features of the vehicle's environment at at least one historical moment. This includes: obtaining first instance features based on image features of N images and current BEV features, wherein the first instance features include semantic and positional features of at least one obstacle at a first moment; and obtaining first information based on the first instance features, historical instance features at least one historical moment, and camera parameters, wherein the historical instance features include semantic and positional features of at least one obstacle at the historical moment corresponding to the historical instance features, the N images and the historical images corresponding to the historical image features are acquired based on at least one camera, and the camera parameters include camera parameters of at least one camera.

[0017] It should be noted that semantic features can be used to characterize information such as the category of obstacles (e.g., pedestrians, bicycles, cars, traffic cones, etc.) and appearance (e.g., color, edges, texture, and type), helping to identify and classify different types of obstacles. Location features can be used to characterize information such as the spatial location of obstacles, helping to determine the location of obstacles.

[0018] In this embodiment, historical image features of the vehicle's environment at at least one historical moment and historical BEV features of the vehicle's environment at at least one historical moment can be directly read from the time-series cache of the in-vehicle device. The historical moment can be any moment before the first moment. It should be noted that the historical BEV features of the vehicle's environment at at least one historical moment can be obtained based on image features of at least one historical moment and / or historical point cloud data of the vehicle's environment at at least one historical moment.

[0019] It is understood that camera parameters may include internal parameters, focal length and resolution, external parameters and rotation and translation parameters, etc., and this application does not specifically limit them.

[0020] In one possible implementation of the first aspect above, obtaining a first instance feature based on the image features of N images and the current BEV feature includes: performing feature fusion on the image features of N images and the current BEV feature to obtain a first fused feature, and obtaining the semantic features and positional features of at least one obstacle at a first moment based on the first fused feature; performing feature fusion on the semantic features and positional features of at least one obstacle at the first moment to obtain the first instance feature of at least one obstacle; wherein the feature fusion includes at least one of summation, splicing and attention-based fusion.

[0021] In one possible implementation of the first aspect above, obtaining first information based on the first instance feature, the historical instance feature of at least one historical moment, and camera parameters includes: performing feature alignment processing on at least one historical instance feature and the first instance feature based on the camera parameters; performing fusion processing on the aligned at least one historical instance feature and the first instance feature to obtain a joint instance feature of at least one obstacle, wherein the fusion processing includes at least one of summation, splicing, and attention-based fusion; determining the motion trajectory of each obstacle after the first moment from multiple preset trajectory types based on the joint instance feature of each obstacle; wherein, when the obstacle is a vehicle, the trajectory type includes at least one of straight, left turn, right turn, braking, and reversing; and when the obstacle is a pedestrian or animal, the trajectory type includes at least one of straight, left turn, right turn, and backward movement.

[0022] In this embodiment, the motion trajectory of each obstacle after the first moment is determined from multiple preset trajectory types based on the joint instance features of each obstacle. This can refer to the vehicle-mounted device mentioned in this embodiment inputting the joint instance features of each obstacle into the first model, and then the vehicle-mounted device can use the first model to output first information based on the joint instance features of each obstacle.

[0023] In one possible implementation of the first aspect above, determining the motion trajectory of each obstacle after a first moment from multiple preset trajectory types based on the joint instance features of each obstacle includes: for any first obstacle among at least one obstacle, determining the confidence levels of the motion trajectory of the first obstacle after the first moment based on the joint instance features of the first obstacle for multiple preset trajectory types; and taking the motion trajectory with the highest confidence level among the multiple preset trajectory types as the motion trajectory of the first obstacle after the first moment.

[0024] In this embodiment, the confidence levels of the first obstacle's trajectory after a first moment are determined based on the joint instance features of the first obstacle, which are then classified as multiple preset trajectory types. The trajectory with the highest confidence among the multiple preset trajectory types is taken as the first obstacle's trajectory after the first moment. This corresponds to the vehicle-mounted device mentioned in this embodiment inputting the joint instance features of each obstacle into a first model. The vehicle-mounted device can then use the first model to determine the confidence levels of the first obstacle's trajectory after a first moment, based on the joint instance features of each obstacle, and then take the trajectory with the highest confidence among the multiple preset trajectory types as the first obstacle's trajectory after the first moment, outputting this as first information.

[0025] In some embodiments of this application, the motion trajectory with the highest confidence can be composed of multiple trajectory points (such as 30 trajectory points). Multiple trajectory points can represent the predicted motion trajectory points of the first obstacle after the first moment. In addition, the motion trajectory points after the first moment can refer to the motion trajectory points of the first time period after the first moment. The first time period can be, for example, 6 seconds. This application does not make specific limitations on this.

[0026] Secondly, this application also provides an in-vehicle device, comprising: at least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, the in-vehicle device causes the in-vehicle device to perform the methods mentioned in the first aspect and any possible implementation thereof.

[0027] Thirdly, this application also provides a vehicle that may include the vehicle-mounted equipment provided in the second aspect above.

[0028] Fourthly, this application also provides a readable storage medium storing instructions that, when executed on an in-vehicle device, cause the in-vehicle device to perform the methods mentioned in the first aspect and any possible implementation thereof.

[0029] Fifthly, this application also provides a program product that, when run on an in-vehicle device, causes the in-vehicle device to perform the methods mentioned in the first aspect and any possible implementation thereof.

[0030] The beneficial effects of the second to fifth aspects mentioned above can be referred to the relevant descriptions in the first aspect and any possible implementation of the first aspect, which will not be repeated here. Attached Figure Description

[0031] Figure 1 According to an embodiment of this application, a schematic diagram of perception and prediction based on mutually independent perception and prediction models is shown;

[0032] Figure 2 According to an embodiment of this application, a flowchart of a prediction method is shown;

[0033] Figure 3 According to an embodiment of this application, a schematic diagram of a BEV coordinate system mesh is shown;

[0034] Figure 4 According to an embodiment of this application, a schematic diagram of a process for fusing image features of N images and current BEV features is shown;

[0035] Figure 5 According to an embodiment of this application, a process diagram of a prediction method that can be implemented through an in-vehicle device is shown;

[0036] Figure 6 According to an embodiment of this application, a schematic diagram of the structure of a prediction module is shown;

[0037] Figure 7 According to an embodiment of this application, a schematic diagram of a model training method is shown;

[0038] Figure 8 According to an embodiment of this application, a functional framework diagram of a vehicle 100 is shown. Detailed Implementation

[0039] The illustrative embodiments of this application include, but are not limited to, a prediction method, a readable storage medium, a program product, and an in-vehicle device. In the prediction method provided by the embodiments of this application, the prediction result does not rely on the perception result of the obstacle to determine the prediction result; instead, it can directly determine the more accurate perception result and prediction result corresponding to the obstacle based on the obstacle-related image features and BEV features.

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] For ease of understanding, some of the terms and related technologies used in this application are explained below.

[0042] 1. Bird's-eye view technology

[0043] Bird's-eye view (BEV) technology is used to convert 3D environmental data collected by different sensors into a top-down 2D grid. This enables perception systems to perform tasks such as obstacle detection and trajectory prediction as if processing 2D images. For example, in the field of driver assistance systems, BEV technology can generate a panoramic view from above the vehicle based on spatial information around the vehicle, thus providing a comprehensive view of the surrounding environment, including the front, rear, left, and right sides. This allows driver assistance systems to better understand the vehicle's surroundings and improve the accuracy of perception and prediction.

[0044] As mentioned earlier, traditional driver assistance systems typically utilize independent perception and prediction models to detect and predict the position of dynamic obstacles in real time. However, since the prediction model relies on information provided by the perception model (such as the obstacle's position, speed, yaw angle, and size) to predict its future trajectory, if the information provided by the perception model contains errors, the prediction model may be unable to accurately predict the obstacle's trajectory. This could lead to decision-making errors in the driver assistance vehicle and increase safety risks.

[0045] For example, Figure 1 A schematic diagram is shown illustrating perception and prediction based on mutually independent perception and prediction models.

[0046] like Figure 1 As shown, images 1, 2, and 3, taken from different perspectives in the vehicle's environment by cameras 10, 20, and 30 of the vehicle-mounted device, are presented. Images 1, 2, and 3 may include at least one identical obstacle. A neural network is used to extract features from images 1, 2, and 3, respectively, yielding image features for image 1, image features for image 2, and image features for image 3 (hereinafter referred to as multi-image features). The vehicle-mounted device can input these multi-image features into a perception model to determine the perception result, such as the obstacle's position, speed, yaw angle, and size. Then, the vehicle-mounted device can use a prediction model to predict the trajectory of the at least one identical obstacle after the current moment based on the perception result.

[0047] In the above process, the input of the prediction model is the output of the perception model. If the output of the perception model is inaccurate, the prediction model will be unable to accurately predict the trajectory of the obstacle after the current moment.

[0048] To address the aforementioned problems, this application provides a prediction method applicable to in-vehicle devices. Specifically, the method includes: First, the in-vehicle device acquires image features of N images of the vehicle's environment at a first moment and current BEV features, wherein the current BEV features are obtained by the in-vehicle device based on the image features of the N images and / or point cloud data of the vehicle's environment. Then, the in-vehicle device can obtain first information based on the image features of the N images, the current BEV features, historical image features of the vehicle's environment at at least one historical moment, and historical BEV features of the vehicle's environment at at least one historical moment. The first information includes the trajectory of at least one obstacle in the vehicle's environment after the first moment.

[0049] Thus, compared to traditional driver assistance systems that first use a perception model to obtain the obstacle's pose, where the perceived pose has accumulated errors, making it impossible for the prediction model to accurately predict the obstacle's trajectory based on that pose, the in-vehicle device in this embodiment directly inputs N images corresponding to the vehicle's environment at the same moment, the current BEV features, and at least one historical BEV feature from a historical moment into the perception prediction network. The perception prediction network (as an example of the first model) directly predicts the obstacle's trajectory after the first moment based on the image features and BEV features (image-point cloud fusion), without relying on perception results (such as the obstacle's pose), making the predicted future trajectory of the obstacle more accurate.

[0050] It should be understood that the N images can be one or more. For any one of the one or more obstacles, the N images can include information about that one obstacle, or a portion of the N images can include information about that one obstacle.

[0051] It should be understood that the historical BEV features of the vehicle's environment at at least one historical moment can be obtained by the on-board equipment based on the historical image features of N images at at least one historical moment and / or the historical point cloud data of the vehicle's environment. The specific determination process can be compared with the determination process of the image features of N images at the first moment, the current BEV features and the BEV features corresponding to the point cloud data exemplified in the embodiments of this application, which will not be repeated here.

[0052] In some embodiments of this application, the vehicle-mounted device can obtain first information by inputting the image features of N images and the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment into a first model. The first information also includes the pose of at least one obstacle at a first moment. The pose of at least one obstacle includes the three-dimensional position, size, orientation angle, and velocity of at least one obstacle.

[0053] Thus, compared to traditional driver assistance systems that obtain obstacle poses by inputting multiple image features into a perception model, the cumulative error of this pose is large. In the embodiments of this application, the on-board device inputs N images corresponding to the vehicle's environment at the same time, the current BEV features (image-point cloud fusion), and at least one historical BEV feature from a historical time into the first model, making the perceived obstacle pose more accurate.

[0054] It should be noted that the prediction method provided in this application embodiment can be applied to telematics devices, vehicle control units (VCUs), and advanced driver assistance systems (ADAS), etc.

[0055] The following is combined with Figure 2 The flowchart illustrating the prediction method is provided to illustrate the prediction method in the embodiments of this application. It should be understood that... Figure 2 In the prediction method shown, the executing entity for each process can be an onboard device. For example... Figure 2 As shown, specifically:

[0056] S201: Obtain the image features of N images of the environment in which the vehicle is located at the first moment and the current BEV features, wherein the current BEV features are obtained based on the image features of the N images and / or the point cloud data of the environment in which the vehicle is located.

[0057] In some embodiments of this application, the N images can be images of the vehicle's surrounding environment captured by different cameras of the in-vehicle device at a single moment. These different cameras may include forward-facing narrow-angle, forward-facing wide-angle, panoramic (front / rear / left / right), and rear-facing cameras, etc. It should be understood that each camera captures an image corresponding to a different perspective, and at least one of the N images may contain the same obstacle.

[0058] After acquiring N images, the vehicle-mounted device can use a backbone network and a neck network to extract features from the N images, resulting in image features for each image, i.e., multi-image features. It should be understood that the backbone network can be ResNet-18, ResNet-50, Mobile Net V2 / V3, Efficient Net B0-B4, or a Vision Transformer (ViT), etc. The neck network can be a Feature Pyramid Network (FPN), etc. This application does not specifically limit the method of image feature extraction.

[0059] In some embodiments of this application, after the vehicle-mounted device acquires the image features of N images, it can use inverse perspective mapping (IPM) and / or a BEV former network to project the image features of the N images onto the BEV coordinate system, thereby obtaining the BEV features corresponding to the N images in the BEV coordinate system.

[0060] It should be noted that the BEV features corresponding to the N images in the BEV coordinate system can be arranged in (c1, h1, w1), where c1 represents the number of channels (i.e., feature dimension), h1 represents the height (i.e., the number of rows of features in the height direction), and w1 represents the width (i.e., the number of columns of features in the width direction). For example, the BEV features corresponding to the N images captured by the forward-looking narrow-angle camera in the BEV coordinate system would be (64, 384, 256), and so on. It should be understood that c1 can characterize the semantic attributes of the image features in the BEV coordinate system grid, such as color, edges, texture, category, and confidence level, etc.

[0061] In some embodiments of this application, the point cloud data can be a set of three-dimensional coordinates collected by a vehicle-mounted device such as a LiDAR and / or millimeter-wave radar from the vehicle's surrounding environment at a single moment. The vehicle-mounted device can use a point pillar network to transform the point cloud data to the BEV coordinate system, thereby obtaining the BEV features corresponding to the point cloud data in the BEV coordinate system. Specifically, the point pillar network is a 3D detection network that transforms point cloud data to the BEV coordinate system to obtain the corresponding BEV features.

[0062] It should be noted that the BEV features corresponding to the point cloud data in the BEV coordinate system can be arranged in (c2, h2, w2), where c2 represents the number of channels (i.e., feature dimension), h2 represents the number of features of the vehicle (self) in the front-rear direction, and w2 represents the number of features of the self in the left-right direction. For example, the BEV features corresponding to the point cloud data in the BEV coordinate system are (64, 384, 256), which can represent dividing the ground around the vehicle into (h2, w2) = (384, 256) BEV grids. There are 384 grids along the front-rear direction (h2 direction), each 0.2m long, for a total length of 76.8m; and 256 grids along the left-right direction (w2 direction), each 0.2m long, for a total width of 12.8m. It should be understood that c2 can characterize the geometric properties of the BEV features corresponding to the point cloud data in the BEV coordinate system grid, such as height and reflection intensity.

[0063] For example, Figure 3 A schematic diagram of a BEV coordinate system grid is shown. (For example...) Figure 3 As shown, the horizontal direction (w2 direction) represents the left-right direction of the vehicle, with 256 grids, each 0.2m, totaling 12.8m (not shown). The vertical direction (h2 direction) represents the front-back direction of the vehicle, with 384 grids, each 0.2m, totaling 76.8m (not shown). Each grid corresponds to a 0.2m × 0.2m ground cube. Each grid stores a multi-dimensional feature vector, representing the geometric properties (height, density, and intensity, etc.) of all laser point clouds within that cube. It should be understood that... Figure 3 The grid area occupied by the vehicle is only an illustrative example. In some embodiments, the grid area occupied by the vehicle may be larger, and this application does not limit this.

[0064] In some embodiments of this application, after obtaining the BEV features corresponding to N images and the BEV features corresponding to point cloud data, the vehicle-mounted device can obtain a fourth BEV feature based on the BEV features corresponding to N images and the BEV features corresponding to point cloud data, and then perform dimensionality reduction processing on the fourth BEV feature to obtain the current BEV feature.

[0065] It should be noted that the fourth BEV feature is obtained by the in-vehicle device through feature fusion of the BEV features corresponding to N images and the BEV features corresponding to point cloud data. Feature fusion methods can include, but are not limited to, concatenation, element-wise addition, or attention-based fusion. For example, the in-vehicle device can concatenate the BEV features corresponding to N images and the BEV features corresponding to point cloud data along the channel dimension to obtain the fourth BEV feature. Alternatively, the in-vehicle device can input the BEV features corresponding to N images and the BEV features corresponding to point cloud data into an attention network respectively, obtain their respective attention weights, and then perform weighted fusion of the BEV features corresponding to N images and the BEV features corresponding to point cloud data according to the attention weights to obtain the fourth BEV feature.

[0066] Furthermore, the in-vehicle device can perform dimensionality reduction on the fourth BEV feature using a 3×3 convolutional kernel and a specific stride, or based on an attention mechanism, to obtain the current BEV feature. The current BEV feature has the same feature dimension as the BEV features corresponding to the N images and the BEV features corresponding to the point cloud data. The feature dimension of the fourth BEV feature is typically larger than that of the current BEV feature, the BEV features corresponding to the N images, and the BEV features corresponding to the point cloud data. Based on the above, the current BEV feature can simultaneously include image features (i.e., the BEV features corresponding to the N images) and point cloud features (i.e., the BEV features corresponding to the point cloud data). It should be noted that in other embodiments of this application, the current BEV feature can also be obtained by performing dimensionality reduction on the current BEV feature using convolutional kernels of other sizes (such as 1×1, 5×5, etc.) and other strides; this application does not specifically limit this.

[0067] For example, the BEV feature corresponding to N images is (64,384,256), and the BEV feature corresponding to the point cloud data is also (64,384,256). By concatenating the BEV feature (64,384,256) of the N images and the BEV feature (64,384,256) of the point cloud data, we obtain a fourth BEV feature (128,384,256). Then, we perform dimensionality reduction on the fourth BEV feature (128,384,256) to obtain the current BEV feature (64,384,256).

[0068] In other embodiments of this application, the current BEV feature may also be the BEV features corresponding to N images.

[0069] In other words, the current BEV feature can be obtained by fusing the BEV features corresponding to N images with the BEV features corresponding to point cloud data, followed by dimensionality reduction. Alternatively, the current BEV feature can be the BEV features corresponding to N images.

[0070] S202: Based on the image features of N images and the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment, first information is obtained, wherein the first information includes the motion trajectory of at least one obstacle in the vehicle's environment after the first moment.

[0071] In some embodiments of this application, the in-vehicle device can directly read historical image features of the vehicle's environment at at least one historical moment and historical BEV features of the vehicle's environment at at least one historical moment from the time-series cache. The historical moment can be any moment before the first moment. It should be noted that the historical BEV features of the vehicle's environment at at least one historical moment can be obtained based on image features of at least one historical moment and / or historical point cloud data of the vehicle's environment at at least one historical moment.

[0072] In some embodiments of this application, the image features of N images can be fused with the current BEV feature to obtain a first fused feature.

[0073] For example, Figure 4 A schematic diagram of a process for fusing image features from N images with the current BEV feature is shown.

[0074] refer to Figure 4Given the image features of N images and the current BEV features, the onboard device can use either the BEV features corresponding to the N images or a fusion of the BEV features corresponding to the N images and the BEV features corresponding to the point cloud data. The current BEV features can be clustered using algorithms such as k-means clustering (or hierarchical clustering, Gaussian mixture models, GMM) to obtain at least one anchor box. The anchor box can use a predefined geometric shape (usually a cube or rectangle) to detect the position and size of potential target objects in the object detection task. The onboard device can use at least one anchor box to sample the image features of the N images and the current BEV features respectively. Each anchor box can include q three-dimensional sampling points. A sampling operator (gridsample) can project the q three-dimensional sampling points of each anchor box onto the image features of the N images and the current BEV features to determine q two-dimensional sampling points. Then, bilinear interpolation is performed on the q two-dimensional sampling points to obtain the first sampling features corresponding to the image features of the N images and the second sampling features corresponding to the current BEV features. The vehicle-mounted device can obtain the first instance feature by fusing the first sampled feature and the second sampled feature.

[0075] It should be noted that an obstacle may include at least one anchor frame, which can be a cube, where q > 0 and q is a positive integer, such as q = 7, representing the center points of the 6 faces and 1 body center of the anchor frame. Each point is a three-dimensional sampling point (xi, yi, zi), where xi represents the vehicle's position in the forward / backward direction in the BEV coordinate system, yi represents the vehicle's position in the forward / backward direction in the BEV coordinate system, and zi represents the vehicle's position in the up / down direction in the BEV coordinate system. In other embodiments of this application, q may also be 8, etc., and this application does not specifically limit it in this way.

[0076] In some embodiments of this application, after obtaining the first fused feature, the vehicle-mounted device can extract the semantic and positional features of at least one obstacle at a first moment from the first fused feature, and fuse the semantic and positional features of the at least one obstacle at the first moment to obtain the first instance feature of at least one obstacle. That is, the first instance feature of at least one obstacle at the first moment may include the semantic and positional features of at least one obstacle at the first moment. Feature fusion may include at least one of summation, concatenation, and attention-based fusion. It should be understood that the first instance feature may refer to a single fixed-length vector obtained by summing / concatenating / fusing the semantic vector (e.g., used to represent category, appearance, etc.) extracted from the first fused feature with the position vector (e.g., center, size, orientation, etc.) on the same obstacle at the same moment, on the same obstacle.

[0077] In some embodiments of this application, after obtaining the first instance features of at least one obstacle at a first moment, the vehicle-mounted device can input the first instance features, the historical instance features of at least one obstacle at at least one historical moment, and camera parameters into a first model to obtain first information. The historical instance features can be obtained by fusing historical image features of the environment in which the vehicle is located at at least one obstacle at at least one historical moment with historical BEV features of the environment in which the vehicle is located at at least one historical moment. Therefore, the historical instance features can include semantic features and positional features of at least one obstacle at the historical moment corresponding to the historical instance features. It should be understood that semantic features can be used to characterize information such as the obstacle's category (e.g., pedestrian, bicycle, car, traffic cone, etc.) and appearance (e.g., color, edge, texture, and category, etc.), helping to identify and classify different types of obstacles. Positional features can be used to characterize the spatial position of the obstacle, helping to determine the obstacle's location. In other embodiments of this application, the positional features can also be pose features, which can be used to characterize the obstacle's spatial position and geometric dimensions, etc., and this application does not specifically limit this. Furthermore, the historical images corresponding to the N images and historical image features can be acquired based on at least one camera, and the camera parameters include the camera parameters of at least one camera. At least one camera can refer to any of the different cameras mentioned above.

[0078] It should be noted that the feature alignment process between at least one historical instance feature and the first instance feature using camera parameters involves projecting the anchor frame center point stored in the historical instance feature from the camera coordinates to the vehicle coordinates to obtain the BEV mesh at the historical moment. Then, the BEV mesh is translated to the BEV mesh at the first moment using the vehicle pose increment (Δx, Δy, Δθ) from the historical moment to the first moment, ensuring that the historical instance feature and the first instance feature describe the same ground area. This allows the aligned historical instance feature and the first instance feature to be fused to obtain a joint instance feature of at least one obstacle. It should be noted that the fusion process can include at least one of summation, stitching, and attention-based fusion.

[0079] In some embodiments of this application, the vehicle-mounted device can initialize multimodal embedding features, which can characterize multiple preset trajectory types. It should be noted that when the obstacle is a vehicle, the multiple preset trajectory types can include at least one of straight, left turn, right turn, braking, and reversing. When the obstacle is a pedestrian or animal, the multiple preset trajectory types include at least one of straight, left turn, right turn, and reversing. In other embodiments of this application, the multiple preset trajectory types can be adaptively adjusted for obstacles with different attributes; this application does not specifically limit the multiple preset trajectory types.

[0080] In some embodiments of this application, for any first obstacle among at least one obstacle, the vehicle-mounted device can determine the confidence level of the first obstacle's motion trajectory after the first moment as a plurality of preset trajectory types based on the joint instance features of the first obstacle, and then take the motion trajectory with the highest confidence level among the plurality of preset trajectory types as the motion trajectory of the first obstacle after the first moment.

[0081] The specific process may include: taking at least one obstacle as the first obstacle as an example, it should first be noted that the joint instance feature can be obtained by the vehicle-mounted device by fusing the first instance feature of the first obstacle at the first moment and the historical instance feature of the first obstacle at at least one historical moment. After obtaining the joint instance feature of the first obstacle and the multimodal embedding feature representing multiple preset trajectory types, the vehicle-mounted device can sum the joint instance feature and the multimodal embedding feature to obtain the multimodal embedding feature fused with the joint instance feature. Furthermore, based on the fact that the multimodal embedding feature can represent multiple preset trajectory types, the vehicle-mounted device can use a linear decoder to perform a fully connected multi-layer perceptron (MLP) on the multimodal embedding feature of the joint instance feature, that is, decode the multimodal embedding feature fused with the joint instance feature to obtain the confidence of multiple preset trajectory types, and select the motion trajectory with the highest confidence among the multiple preset trajectory types as the motion trajectory of the first obstacle after the first moment.

[0082] It should be noted that, in other embodiments of this application, the vehicle-mounted device may also utilize recurrent neural networks (RNNs), convolutional neural networks (CNNs), graph neural networks (GNNs), and hybrid models to process the multimodal embedding features of the joint instance features, thereby obtaining the confidence scores of multiple preset trajectory types, and selecting the motion trajectory with the highest confidence score among the multiple preset trajectory types as the motion trajectory of the first obstacle after the first moment. That is, this application does not specifically limit the method for determining the confidence scores of multiple preset trajectory types based on the multimodal embedding features of the joint instance features.

[0083] It should be noted that the trajectory with the highest confidence can consist of multiple trajectory points (such as 30 trajectory points), and these multiple trajectory points can represent the predicted trajectory points of the first obstacle after the first moment. In addition, the trajectory points after the first moment can refer to the trajectory points of the first time period after the first moment, such as 6 seconds, etc. This application does not make specific limitations in this regard.

[0084] Based on the above, the vehicle-mounted device can obtain first information by inputting the image features of N images and the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment into the first model. The first information includes the motion trajectory of at least one obstacle in the vehicle's environment after the first moment.

[0085] In some embodiments of this application, the vehicle-mounted device can also input the image features of N images and the current BEV features, the historical image features of the vehicle's environment at at least one historical moment and the historical BEV features of the vehicle's environment at at least one historical moment into the first model to obtain the first information output by the first model. The first information may also include the pose of at least one obstacle at the first moment. The pose of at least one obstacle includes the three-dimensional position, size, orientation angle and velocity of at least one obstacle.

[0086] Based on the above, the prediction method provided in this application combines N image features from the same moment with the current BEV features (image-point cloud fusion), along with historical image features and historical BEV features, and inputs them into a lightweight first model. This yields accurate perception results (pose) and prediction results (future trajectory) for each obstacle, avoiding the accumulation of pose errors in the traditional "perception first, prediction second" approach. Furthermore, the embodiments in this application incorporate point cloud information about the vehicle's environment, resulting in more accurate perception and prediction results. The lightweight first model also requires less computational power, saving computing resources for onboard equipment.

[0087] To more intuitively understand the prediction method provided in this application, Figure 5 According to an embodiment of this application, a schematic diagram of a prediction method that can be implemented through an in-vehicle device is provided. (Reference) Figure 5 The specific process may include:

[0088] First, the vehicle-mounted device can obtain N images through the vehicle's cameras. Image 1 can be the image obtained by the vehicle's camera 10 (such as a forward-facing narrow-angle camera) at the first moment (e.g., time t) after capturing and processing the image of the vehicle's surrounding environment. For example, the vehicle-mounted device can resize the initial image captured by camera 10 by 1 / 4, reducing the width and height of image 1 to 1 / 4 of its original size, resulting in image 1 [1*3*512*960], where "1" represents the number of images, "3" represents the RGB color channels, and "512*960" represents the reduced resolution.

[0089] Image 2 can be a set of images (corresponding to front / rear / left / right / rear-view perspectives) obtained by the vehicle's camera 20 (such as a panoramic + rear-view camera) capturing the vehicle's surrounding environment at the first moment and then processing the images. For example, resizing the initial set of images captured by the camera 20 by 1 / 2 reduces the width and height of the set of images to half of their original values, resulting in the set of images [5*3*512*960], where "5" represents the number of images, "3" represents the RGB color channels, and "512*960" represents the reduced resolution.

[0090] Image N can be the image obtained by the vehicle's camera N0 (such as a forward-facing wide-angle camera) capturing the vehicle's surrounding environment at the first moment and then processing the image. For example, the initial image captured by camera N0 is resized by 1 / 4, reducing the width and height of image N to 1 / 4 of the original, resulting in image N[1*3*512*960], where "1" represents the number of images, "3" represents the RGB color channels, and "512*960" represents the reduced resolution.

[0091] Given N images, the onboard device can extract features from these N images to obtain image features, and then transform these image features into the BEV coordinate system to obtain the corresponding BEV features for each of the N images in the BEV coordinate system. Furthermore, the onboard device can obtain point cloud data through the vehicle's LiDAR and / or millimeter-wave radar. Point cloud data can be obtained by the radar collecting data about the vehicle's environment at a specific moment. With the point cloud data obtained, the onboard device can transform it into the BEV coordinate system to obtain the corresponding BEV features for the point cloud data in the BEV coordinate system.

[0092] The onboard equipment can obtain the current BEV feature from the BEV features corresponding to N images and / or the BEV features corresponding to point cloud data. It should be understood that, on the one hand, the current BEV feature can be the BEV features corresponding to N images. On the other hand, the current BEV feature can be the BEV feature obtained by fusing the BEV features corresponding to N images and the BEV features corresponding to point cloud data, followed by dimensionality reduction processing.

[0093] After obtaining the current BEV features, the vehicle-mounted device can obtain a first instance feature of at least one obstacle based on the image features of N images and the current BEV features. Specifically, the vehicle-mounted device can sample the image features of the N images and the current BEV features respectively, obtaining a first sampled feature corresponding to the image features of the N images and a second sampled feature corresponding to the current BEV features. Then, the vehicle-mounted device can fuse the first sampled feature and the second sampled feature to obtain a first fused feature. After obtaining the first fused feature, the vehicle-mounted device can extract the semantic and positional features of at least one obstacle at a first moment from the first fused feature, and fuse the semantic and positional features of at least one obstacle at the first moment to obtain the first instance feature of at least one obstacle.

[0094] Next, the onboard device can directly read the historical instance features of the vehicle's environment at at least one historical moment (e.g., time t-1) through the time-series cache. These historical instance features are obtained by fusing historical image features of at least one obstacle in the vehicle's environment at at least one historical moment with historical BEV features of the vehicle's environment at at least one historical moment. Therefore, the historical instance features include the semantic and positional features of at least one obstacle at the historical moment corresponding to the historical instance features. Furthermore, by sending camera parameters, the first instance features of at least one obstacle at the first moment (time t), and the historical instance features at at least one historical moment (e.g., time t-1) to the feature interaction module (sparse4D), the feature interaction module can receive the data and output joint instance features.

[0095] It should be understood that camera parameters can be used to align the first instance features and historical instance features to facilitate fusion processing and obtain the joint instance features of at least one obstacle at the first time step (time t). It should be noted that the feature interaction module can send the joint instance features of at least one obstacle at the first time step (time t) to the timing cache module of the onboard device for storage.

[0096] Furthermore, the onboard device can send the joint instance features of at least one obstacle at a first time (time t) to a first model, and determine first information (pose + trajectory) through the first model. The first model includes a perception module and a prediction module. On one hand, the first information can include the pose of at least one obstacle at the first time, which includes the three-dimensional position, size, orientation angle, and velocity of at least one obstacle. It should be understood that the perception module can use a sensing head, i.e., a lightweight detection head, to regress the joint instance features to obtain the pose of at least one obstacle, which can include the three-dimensional position, size, orientation angle, and velocity. On the other hand, the first information can include the trajectory of at least one obstacle in the vehicle's environment after the first time. It should be understood that the first model can use the prediction module to process the joint instance features to obtain the trajectory of at least one obstacle after the first time.

[0097] The following is combined with Figure 6 The schematic diagram of a prediction module illustrates the process of determining the motion trajectory.

[0098] refer to Figure 6 First, the prediction module can be based on Figure 5 The sparse4D algorithm obtains joint instance features, from which semantic and positional features are extracted. The prediction module then sums the semantic and positional features to obtain a combined joint instance feature. Next, the prediction module initializes multimodal embeddings, which can represent multiple preset motion trajectory types. For example, when the obstacle is a vehicle, the multimodal trajectory type can include at least one of straight, left turn, right turn, braking, and reversing; when the obstacle is a pedestrian or animal, the multimodal trajectory type includes at least one of straight, left turn, right turn, and backward movement. It should be understood that obtaining the joint instance features and initializing the multimodal embeddings can be performed simultaneously.

[0099] After initializing the multimodal embedding features, the prediction module can sum the joint instance features (which fuse semantics and location) and the multimodal embedding features to obtain the multimodal embedding features fused with the joint instance features. Next, the prediction module can perform fully connected processing on the multimodal embedding features fused with the joint instance features to obtain a multimodal motion trajectory set and a multimodal motion trajectory confidence score set. For example, the multimodal motion trajectory set is [1, 5, 384, 30, 2], where "1" represents a feature dimension placeholder, "5" represents the types of motion trajectory, "384" represents the number of obstacles, "30" represents the number of trajectory points in each motion trajectory type, and "2" represents the coordinates (x, y) of each trajectory point. The multimodal motion trajectory type confidence feature set is [1, 5, 384, 1], where "5" represents the five types of motion trajectory, "384" represents the number of obstacles, and "1" represents the confidence score for each corresponding motion trajectory type. Next, the prediction module performs matrix alignment between the multimodal motion trajectory type feature set and the multimodal motion trajectory type confidence feature set, and selects the motion trajectory with the highest confidence score among multiple preset trajectory types for each obstacle as the motion trajectory of each obstacle after the first moment. In this way, the prediction module can determine the precise motion trajectory of each obstacle after the first moment.

[0100] This application also provides a model training method, which specifically includes:

[0101] First, the onboard device can acquire training sample data, which may include at least image features of N images of the vehicle's environment at at least one first moment and current BEV features. The current BEV features are obtained based on the image features of the N images and / or point cloud data of the vehicle's environment. Then, the onboard device can input the training sample data into the model to be trained to obtain first information about at least one obstacle at at least one first moment. This first information includes the pose of at least one obstacle at the first moment and its trajectory after the first moment. Furthermore, the onboard device can determine the lane line and roadside BEV positions based on the current BEV features.

[0102] Then, the onboard device can acquire reference information of at least one obstacle at at least at a first moment. The reference information includes the reference pose of at least one obstacle at at least at the first moment and its trajectory after at least the first moment, as well as the lane line and roadside BEV reference positions. Then, based on the first information and the reference information, the lane line and roadside BEV positions, and the lane line and roadside BEV reference positions, the onboard device can determine the loss function of the model to be trained. The loss function characterizes the model output error.

[0103] Furthermore, the in-vehicle device can iteratively train the model to be trained based on the loss function. It should be understood that the model to be trained may refer to the first model mentioned in the foregoing embodiments.

[0104] In this way, the vehicle-mounted equipment can use the lane lines and roadside BEV reference positions as ground truth labels during the model training phase, and use them together with the reference information that is also used as ground truth labels to determine the loss function and train the model to be trained. This allows the model to output more accurate perception and prediction results during inference, namely the pose and future trajectory of obstacles.

[0105] For example, Figure 7 A schematic diagram of a model training method is shown.

[0106] refer to Figure 7 First, the onboard device inputs training sample data into the model to be trained. The model can output primary information (obstacle pose + trajectory) based on the training sample data. Furthermore, the onboard device can determine the lane line and roadside BEV positions using the current BEV features in the model. Then, the onboard device can determine the loss function of the model based on the primary information (obstacle pose and trajectory), reference information (obstacle reference pose and trajectory), lane line and roadside BEV positions, and lane line and roadside BEV reference positions. Finally, the onboard device can iteratively train the model based on this loss function, enabling the model to output more accurate primary information during inference. This primary information can include perception results (such as obstacle pose) and prediction results (such as the obstacle's future trajectory).

[0107] It should be noted that the lane line & roadside BEV positions and lane line & roadside BEV reference positions can be used only during the model training process rather than the inference stage. This allows the model to operate without relying on high-precision maps during inference, and enables the model to output more accurate perception and prediction results while requiring less computational power for inference.

[0108] It is understood that the methods mentioned in the embodiments of this application can be applied to in-vehicle equipment in vehicles. Figure 8 This is a possible structural schematic diagram of a vehicle 100 provided in an embodiment of this application.

[0109] like Figure 8 As shown, the functional framework of vehicle 100 may include various subsystems, such as Figure 8The diagram shows a sensor system 110, a control system 120, one or more peripheral devices 130 (one is shown as an example), a power supply 140, and an onboard device 150. Optionally, the vehicle 100 may also include other functional systems, such as an engine system that powers the vehicle 100, etc., which are not limited herein. Optionally, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 100 can be interconnected via wired or wireless means.

[0110] The sensor system 110 may include several detection devices that can sense the measured information and convert the sensed information into electrical signals or other desired forms of information output according to a certain rule. Figure 8 As shown, these detection devices may include a global positioning system (GPS) 111, a vehicle speed sensor 112, an inertial measurement unit (IMU) 113, etc., and this application does not limit them.

[0111] The Global Positioning System 111 is a system that uses GPS positioning satellites to perform real-time positioning and navigation globally. In this application, the Global Positioning System 111 can be used to achieve real-time positioning of the vehicle 100 and provide the geographical location information of the vehicle 100. The vehicle speed sensor 112 is used to detect the vehicle speed of the vehicle 100. The inertial measurement unit 113 may include a combination of an accelerometer and a gyroscope, and is a device for measuring the angular rate and acceleration of the vehicle 100.

[0112] The control system 120 may include a steering unit 121 and a braking unit 122, etc.

[0113] Steering unit 121 can represent a system for adjusting the direction of travel of vehicle 100, and may include, but is not limited to, a steering wheel or other structural device for adjusting or controlling the direction of travel of vehicle 100. Braking unit 122 can represent a system for slowing down the speed of vehicle 100, and may also be referred to as a vehicle braking system. It may include, but is not limited to, a brake controller, a reducer, or other structural device for slowing down vehicle 100. In practical applications, braking unit 122 can use friction to slow down the tires of vehicle 100, thereby slowing down the speed of vehicle 100.

[0114] Peripheral device 130 may include several components, such as Figure 8 The diagram shows a communication system 131, a touchscreen 132, a user interface 133, etc. The communication system 131 is used to enable network communication between the vehicle 100 and other devices besides the vehicle.

[0115] In practical applications, the communication system 131 can use wireless communication technology or wired communication technology to realize network communication between the vehicle 100 and other devices. The wired communication technology can refer to communication between the vehicle 100 and other devices via network cables or optical fibers. The wireless communication technology includes, but is not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technology, etc.

[0116] The touchscreen 132 can be used to detect operation commands on the touchscreen 132. For example, the user can perform touch operations on the content data displayed on the touchscreen 132 according to actual needs to achieve the corresponding function, such as playing music, video, or other multimedia files. The user interface 133 can specifically be a touch panel, used to detect operation commands on the touch panel. The user interface 133 can also be a physical button or a mouse. The user interface 133 can also be a display screen, used to output data and display images or data. Optionally, the user interface 133 can also be at least one device belonging to the category of peripheral devices, such as a touchscreen, microphone, and speaker.

[0117] Several functions of vehicle 100 are controlled and implemented by on-board equipment 150. On-board equipment 150 may include multiple processors such as processor 151, chassis domain controller (CDC) 152, mobility domain controller (MDC) 153, telematics box (T-BOX) 154, as well as memory 155 (also referred to as storage device) and gateway 156. In practical applications, the memory 155 may be located inside or outside the on-board equipment 150, for example, as a cache in vehicle 100, etc., and this application does not limit this.

[0118] Among them, processor 151, CDC 152, MDC 153, and T-BOX 154 can be used to run relevant programs or instructions corresponding to programs stored in memory 155 to realize the corresponding functions of vehicle 100, such as the function of calling vehicle camera.

[0119] Memory 155 may include volatile memory, such as RAM; it may also include non-volatile memory, such as ROM, flash memory, HDD, or SSD; or it may include a combination of the above types of memory. Memory 155 can be used to store a set of program code or instructions corresponding to program code, so that processor 151 can call the program code or instructions stored in memory 155 to implement the corresponding functions of vehicle 100. This function includes, but is not limited to, […]. Figure 8 The illustrated vehicle functional framework diagram shows some or all of the functions. In this application, the memory 155 can store a set of program codes for sensing and predicting obstacles. The processors 151, CDC 152, MDC 153, and T-BOX 154 can call the program codes to control the vehicle 100 to perform the methods illustrated in this application.

[0120] Optionally, in addition to storing program code or instructions, the memory 155 may also store information such as road maps, driving routes, and sensor data. The on-board device 150 can be combined with other components in the vehicle functional framework diagram, such as sensors in the sensor system and GPS, to realize the relevant functions of the vehicle 100. For example, the on-board device 150 can control the driving direction or speed of the vehicle 100 based on data input from the sensor system 110; this application does not impose limitations on this.

[0121] This application provides a vehicle that may include the aforementioned on-board equipment.

[0122] This application provides a readable storage medium on which instructions are stored. When the instructions are executed on an in-vehicle device, the in-vehicle device can perform the technical solutions described in the above embodiments.

[0123] This application provides a program product that, when run on a device, causes the in-vehicle device to execute the technical solutions described above.

[0124] It is understood that the various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0125] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0126] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. The mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0127] The above describes the possible hardware structures of electronic devices. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0128] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0129] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0130] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A method of prediction, characterized by, The method includes: The image features of N images of the environment in which the vehicle is located at the first moment and the current BEV features are obtained, wherein the current BEV features are obtained based on the image features of the N images and / or the point cloud data of the environment in which the vehicle is located, and N is an integer greater than or equal to 1. Based on the image features of the N images and the current BEV features, historical image features of the vehicle's environment at at least one historical moment, and historical BEV features of the vehicle's environment at at least one historical moment, first information is obtained, wherein the first information includes the trajectory of at least one obstacle in the vehicle's environment after the first moment; wherein... The first information is obtained based on the image features of the N images, the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment, including: A first instance feature is obtained based on the image features of the N images and the current BEV feature. The first instance feature includes the semantic features and positional features of the at least one obstacle at the first moment. Based on the first instance features, the historical instance features of the at least one historical moment, and camera parameters, the first information is obtained, wherein the historical instance features include the semantic features and positional features of the at least one obstacle at the historical moment corresponding to the historical instance features; The first instance feature is obtained based on the image features of the N images and the current BEV feature, including: The image features of the N images and the current BEV features are fused to obtain a first fused feature, and the semantic features and positional features of the at least one obstacle at the first moment are obtained based on the first fused feature; The semantic and positional features of the at least one obstacle at the first moment are fused to obtain the first instance feature of the at least one obstacle.

2. The method according to claim 1, characterized in that, The first information is obtained based on the image features of the N images, the current BEV features, the historical image features of the vehicle's environment at at least one historical moment, and the historical BEV features of the vehicle's environment at at least one historical moment. This information includes: The first information is obtained through the first model, based on the image features of the N images, the current BEV features, and the historical image features and historical BEV features of the vehicle's environment at at least one historical moment. The first information also includes the pose of the at least one obstacle at the first moment, and the pose of the at least one obstacle includes the three-dimensional position, size, orientation angle and velocity of the at least one obstacle.

3. The method according to claim 1, characterized in that, The current BEV characteristics are obtained in the following way: Based on the image features of the N images, BEV features corresponding to the N images are obtained, and based on the point cloud data, BEV features corresponding to the point cloud data are obtained. The current BEV feature is obtained based on the BEV features corresponding to the N images and the BEV features corresponding to the point cloud data.

4. The method according to claim 3, characterized in that, The process of obtaining the current BEV feature based on the BEV features corresponding to the N images and the BEV features corresponding to the point cloud data includes: The BEV features corresponding to the N images and the BEV features corresponding to the point cloud data are subjected to feature concatenation and dimensionality reduction processing to obtain the current BEV features.

5. The method according to claim 3, characterized in that, The N images and the historical images corresponding to the historical image features are based on acquisitions from at least one camera, and the camera parameters include the camera parameters of at least one camera.

6. The method according to claim 1, characterized in that, Feature fusion can be any one of summation, concatenation, or attention-based fusion.

7. The method according to claim 6, characterized in that, The process of obtaining the first information based on the first instance features, the historical instance features of at least one historical moment, and camera parameters includes: Based on the camera parameters, the at least one historical instance feature is aligned with the first instance feature, and the aligned at least one historical instance feature and the first instance feature are fused to obtain the joint instance feature of the at least one obstacle. The fusion process is any one of summation, splicing, and attention-based fusion. Based on the joint instance features of each obstacle, the motion trajectory of each obstacle after the first moment is determined from multiple preset trajectory types; Wherein, when the obstacle is a vehicle, the trajectory type includes at least one of going straight, turning left, turning right, braking, and reversing; when the obstacle is a pedestrian or animal, the trajectory type includes at least one of going straight, turning left, turning right, and reversing.

8. The method according to claim 7, characterized in that, The determination of the motion trajectory of each obstacle after the first moment from multiple preset trajectory types based on the joint instance features of each obstacle includes: For any first obstacle among the at least one obstacle, the confidence levels of the motion trajectory of the first obstacle after the first moment are determined based on the joint instance features of the first obstacle, respectively, for the multiple preset trajectory types. The trajectory with the highest confidence among the multiple preset trajectory types is taken as the trajectory of the first obstacle after the first moment.

9. A vehicle-mounted device, characterized in that, include: At least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, causing the vehicle-mounted device to perform the method as described in any one of claims 1 to 8.

10. A vehicle, characterized in that, The vehicle includes the on-board equipment as described in claim 9.

11. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on the vehicle-mounted device, cause the vehicle-mounted device to perform the method of any one of claims 1 to 8.

12. A program product, characterized in that, When the program product is run on an in-vehicle device, it causes the in-vehicle device to perform the method of any one of claims 1 to 8.

Images