Trajectory prediction method, electronic device, vehicle, medium and program product

Abstract

The application relates to the technical field of vehicle driving, in particular to a trajectory prediction method, an electronic device, a vehicle, a medium and a program product. In the method, a first prediction trajectory of a moving obstacle and a target environment image captured by a target vehicle are acquired, the first prediction trajectory is determined based on a BEV feature map of an environment where the moving obstacle is located, the first prediction trajectory comprises a plurality of trajectory points, and the target environment image comprises a front view image, the front view image comprises a local area corresponding to at least one trajectory point in the plurality of trajectory points; local image features corresponding to the local area of each trajectory point in the target environment image are determined; and based on the local image features, a second prediction trajectory of the moving obstacle is predicted. It is known that the local image features comprise environment features corresponding to a region that the moving obstacle may reach in the future, the vehicle can expand the perception of long-distance traffic elements based on the local image features, and then a more accurate trajectory is predicted.

Description

Technical Field

[0001] This application relates to the field of vehicle driving technology, and in particular to a trajectory prediction method, electronic device, vehicle, medium, and program product. Background Technology

[0002] During driving, the vehicle can predict the future trajectory of other vehicles based on information about moving and static objects in the surrounding environment, and assist the current vehicle in determining its driving strategy.

[0003] In some embodiments, the current vehicle encodes bird's-eyeview (BEV) feature data based on collected sensor data. The BEV feature data includes multi-dimensional information on moving objects (vehicles, pedestrians, etc.) and the static environment (e.g., lane lines, road boundaries, traffic signs, etc.), providing a unified representation for trajectory prediction. Furthermore, based on the BEV feature map, the vehicle can determine the features of other vehicles driving in the same environment that are related to their historical trajectories (e.g., vehicle type, speed, and direction of travel), thereby predicting the trajectories of other vehicles based on their features. This allows the current vehicle to determine its driving strategy based on the predicted trajectories of other vehicles.

[0004] However, the perception information obtained by the vehicle based on the BEV feature map is limited, and it cannot perceive information such as lane information, signs, and traffic lights at a distance. This leads to problems such as the predicted trajectory of other vehicles not conforming to the lane or the driving speed not matching the speed limit information. As a result, the current vehicle may not slow down in time or may slow down unnecessarily when it is based on the predicted trajectory of other vehicles, affecting the ride comfort. Summary of the Invention

[0005] This application provides a trajectory prediction method, electronic device, vehicle, medium, and program product to avoid the problem of inaccurate trajectory prediction of moving obstacles.

[0006] In a first aspect, embodiments of this application provide a trajectory prediction method, which includes: acquiring a first predicted trajectory of a moving obstacle and a target environment image captured by a target vehicle, wherein the first predicted trajectory is determined based on a BEV feature map of the environment in which the moving obstacle is located, the first predicted trajectory includes multiple trajectory points, the target environment image includes a front view image, and the front view image includes a local region corresponding to at least one of the multiple trajectory points; determining the local image features corresponding to the local region of each trajectory point in the target environment image; and predicting a second predicted trajectory of the moving obstacle based on the local image features.

[0007] Understandably, the first predicted trajectory is a coarse trajectory point. Since the forward view image includes a local region corresponding to at least one of the multiple trajectory points, meaning the forward view image contains a significant amount of environmental information about the moving obstacle, after obtaining the first predicted trajectory, by determining the local image features corresponding to the local region of each trajectory point in the target environment image, the environmental features corresponding to the area that the moving obstacle's future trajectory may reach can be extracted. Based on the local image features, the vehicle can expand its perception of distant traffic elements, thereby predicting a more accurate trajectory for the moving obstacle based on the extracted local image features.

[0008] In some possible implementations of the first aspect described above, the trajectory points of the first predicted trajectory have vehicle trajectory point coordinates in the vehicle coordinate system; and the local image features corresponding to the local region of each trajectory point in the target environment image are determined, including: extracting image features from the target environment image to obtain a forward-looking feature image; determining the image trajectory point coordinates of the trajectory point in the forward-looking feature image; determining the local region of the trajectory point from the forward-looking feature image based on the image trajectory point coordinates and the region range parameter; and obtaining local image features based on the image features in the local region.

[0009] In some possible implementations of the first aspect above, the target environment image is captured by the target vehicle using a forward-looking camera; and determining the coordinates of the trajectory points in the forward-looking feature image includes: obtaining the coordinates of the vehicle trajectory points in the vehicle coordinate system; and performing coordinate system transformation on the coordinates of the vehicle trajectory points according to the camera parameters of the forward-looking camera to obtain the coordinates of the image trajectory points in the image coordinate system of the forward-looking feature image.

[0010] In some possible implementations of the first aspect above, local image features are obtained based on image features in the local region, including: extracting local image features from the local region using bilinear interpolation.

[0011] In some possible implementations of the first aspect described above, obtaining the first predicted trajectory of the moving obstacle and the target environment image captured by the target vehicle includes: obtaining a BEV feature map of the environment in which the moving obstacle is located; matching BEV features related to the historical trajectory of the moving obstacle in the BEV feature map based on the query features of the moving obstacle; and inputting the BEV features related to the historical trajectory of the moving obstacle into a first prediction model to obtain the first predicted trajectory.

[0012] In some possible implementations of the first aspect above, the second predicted trajectory of the moving obstacle is predicted based on local image features, including: predicting the second predicted trajectory based on the vehicle trajectory point coordinates of the trajectory point, the local image features of the trajectory point, and the BEV features related to the historical trajectory of the moving obstacle.

[0013] In some possible implementations of the first aspect described above, a second predicted trajectory is predicted based on the vehicle trajectory coordinates of the trajectory point, the local image features of the trajectory point, and the BEV features related to the historical trajectory of the moving obstacle. This includes: mapping the vehicle trajectory coordinates of the trajectory point to a trajectory point position encoding vector that matches the dimension of the BEV features related to the historical trajectory of the moving obstacle; fusing the local image features of the trajectory point into the BEV features related to the historical trajectory of the moving obstacle based on the trajectory point position encoding vector to obtain updated BEV features related to the historical trajectory of the moving obstacle; and inputting the updated BEV features related to the historical trajectory of the moving obstacle into the second prediction model to obtain the second predicted trajectory.

[0014] Secondly, embodiments of this application also provide an electronic device, including: a memory for storing instructions; and at least one processor for executing the instructions to cause the electronic device to implement the trajectory prediction method proposed in the first aspect and any possible implementation of the first aspect.

[0015] Thirdly, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the trajectory prediction method proposed in the first aspect and any possible implementation thereof.

[0016] Fourthly, embodiments of this application also provide a computer program product that, when run on a device, causes the device to execute the trajectory prediction method proposed in the first aspect and any possible implementation thereof.

[0017] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be referenced to the beneficial effects corresponding to the first aspect and any possible implementation of the first aspect. Attached Figure Description

[0018] Figure 1 According to an embodiment of this application, a schematic diagram of a scenario in which a vehicle 100 is driven is shown;

[0019] Figure 2 According to some embodiments of this application, a schematic flowchart of a trajectory prediction method is shown;

[0020] Figure 3 According to an embodiment of this application, a flowchart for obtaining a first predicted trajectory is shown;

[0021] Figure 4 According to an embodiment of this application, a flowchart for determining local image features is shown;

[0022] Figure 5 According to some embodiments of this application, a flowchart for obtaining a second predicted trajectory is shown;

[0023] Figure 6 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 10' is shown;

[0024] Figure 7 According to some embodiments of this application, a structural schematic diagram of a vehicle 100 is shown. Detailed Implementation

[0025] The illustrative embodiments of this application include, but are not limited to, trajectory prediction methods, electronic devices, vehicles, media, and program products.

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Figure 1 According to some embodiments of this application, a schematic diagram of a vehicle 100 driving at an asymmetric intersection is shown. During the driving process, when passing through the intersection, the vehicle 100 can collect sensor data such as images to obtain information about surrounding traffic and road elements. Feature extraction and encoding are performed on the image sensor information to obtain a bird's-eye view (BEV) feature map. Based on the features extracted from the BEV feature map related to other vehicles and their historical trajectories, the trajectory of other vehicles corresponding to the vehicle 100 when passing through the intersection is predicted, thus obtaining... Figure 1 The trajectory 01 enables vehicle 100 to determine its driving strategy based on the trajectory 01.

[0028] For example, after vehicle 100 obtains the trajectory 01 of vehicle 200, it predicts its own trajectory as follows: Figure 1 Trajectory 02 in the image. Based on trajectory 01 corresponding to other vehicles and trajectory 02 corresponding to our own vehicle, we determine that at the current driving speed... Figure 1 The collision at point C will cause vehicle 100 to slow down prematurely.

[0029] However, due to the millions of grid points in the dense BEV feature map during the prediction process, the computing power of the onboard equipment limits the ability of vehicle 100 to effectively perceive long-distance traffic element information from the BEV feature map. For example, vehicle 100 cannot effectively extract the fact that a distant intersection is asymmetrical from the dense BEV feature map, resulting in a clearly incorrect predicted trajectory 01. In reality, vehicle 200 will not drive according to trajectory 01 and will not pass point C; therefore, vehicle 100 does not need to slow down in advance.

[0030] In other words, due to incorrect trajectory prediction, the vehicle may need to decelerate unnecessarily, affecting the passenger experience and comfort.

[0031] Therefore, to solve the above problems, this application proposes a trajectory prediction method. In this method, a first predicted trajectory of a moving obstacle and a target environment image captured by a target vehicle are acquired. The first predicted trajectory is determined based on the BEV feature map of the environment in which the moving obstacle is located. The first predicted trajectory includes multiple trajectory points. The target environment image includes a front view image, which includes a local region corresponding to at least one of the multiple trajectory points. The local image features corresponding to the local region of each trajectory point in the target environment image are determined. Based on the local image features, a second predicted trajectory of the moving obstacle is predicted.

[0032] Understandably, the first predicted trajectory is a coarse trajectory point. Since the forward view image includes a local region corresponding to at least one of the multiple trajectory points, meaning the forward view image contains a significant amount of environmental information about the moving obstacle, after obtaining the first predicted trajectory, by determining the local image features corresponding to the local region of each trajectory point in the target environment image, the environmental features corresponding to the area that the moving obstacle's future trajectory may reach can be extracted. Based on the local image features, the vehicle can expand its perception of distant traffic elements, thereby predicting a more accurate trajectory for the moving obstacle based on the extracted local image features.

[0033] Figure 2 According to some embodiments of this application, a schematic diagram of a trajectory prediction method is shown. The process is illustrated using an electronic device as the execution subject. It is understood that the electronic device in the embodiments of this application can include, but is not limited to, any device capable of processing model data, including but not limited to mobile phones, in-vehicle systems, terminals in self-driving cars, computers, servers, etc. The specific steps are as follows:

[0034] S201, acquire a first predicted trajectory of the moving obstacle and a target environment image captured by the target vehicle, wherein the first predicted trajectory is determined based on the BEV feature map of the environment in which the moving obstacle is located, the first predicted trajectory includes multiple trajectory points, and the target environment image includes a front view image, the front view image including a local area corresponding to at least one of the multiple trajectory points.

[0035] Understandably, moving obstacles can include other vehicles, moving pedestrians, moving signs, etc.

[0036] In some embodiments, a BEV feature map of the environment in which the moving obstacle is located is obtained; based on the query features of the moving obstacle, BEV features related to the historical trajectory of the moving obstacle are matched in the BEV feature map; the BEV features related to the historical trajectory of the moving obstacle are input into a first prediction model to obtain a first predicted trajectory. It is understood that the BEV feature map can be obtained by fusing images from multiple perspectives captured by the target vehicle.

[0037] Understandably, the placement of the cameras on the target vehicle directly determines the field of view and coverage area of ​​the image acquisition. Therefore, different camera placements result in images captured from different perspectives. Front-view cameras are primarily installed inside the windshield or near the rearview mirror, mainly used to capture images of the environment directly in front of and to the sides of the vehicle. Based on focal length and field of view, front-view cameras are mainly divided into two categories: narrow-angle front-view cameras and wide-angle front-view cameras. Rear-view cameras are mainly installed in the center of the trunk lid, above the license plate, or on the rear bumper, primarily used to capture images of the environment directly behind and to the sides of the vehicle. Panoramic cameras typically exist in a multi-camera combination, installed on the left front fender, right front fender, left rear fender, and right rear fender, achieving 360° full coverage of the vehicle's surrounding environment through multi-view collaborative acquisition. Correspondingly, the acquired images from multiple perspectives can include, but are not limited to, panoramic images, rear images, and front images. Front images can include narrow-angle front images and wide-angle front images.

[0038] For example, Figure 3 According to an embodiment of this application, a flowchart illustrating the process of obtaining a first predicted trajectory is shown. It is understood that the first predicted trajectory is a coarsely predicted trajectory.

[0039] In some implementations, the BEV feature map of the environment where the moving obstacle is located can be obtained as follows: First, acquire the circumferential view image, rear view image, front narrow-angle image, and front wide-angle image captured by the target vehicle. Then, input the circumferential view image and the rear view image into the same backbone network 1 and neck network 1, which are mainly composed of convolutional layers, to encode the circumferential view image and the rear view image into multi-scale, multi-dimensional circumferential view image features and rear view image features. Similarly, input the front narrow-angle image into a backbone network 2 and neck network 2, which are mainly composed of convolutional layers, to encode the front narrow-angle image into multi-scale, multi-dimensional front narrow-angle image features. Finally, input the front wide-angle image into a backbone network 3 and neck network 3, which are mainly composed of convolutional layers, to encode the front wide-angle image into multi-scale, multi-dimensional front wide-angle image features. Feature fusion is performed between image features of the same but different scales and dimensions to obtain fused image features. This fused image feature integrates features from different scales and dimensions, including panoramic, rear, narrow-angle, and wide-angle images. These panoramic, rear, narrow-angle, and wide-angle features are then input into an image BEV fusion network, transforming the multi-view 2D image features into unified 3D BEV features, thus obtaining a BEV feature map of the environment containing the moving obstacle.

[0040] In some implementations, matching BEV features related to the historical trajectory of a moving obstacle from its query features in the BEV feature map can be achieved as follows: Obtain a set of candidate anchor boxes for the moving obstacle obtained through clustering, and transform this set into query features of the moving obstacle using a feature mapping module. Then, input the query features of the moving obstacle and the BEV feature map into an attention mechanism (Transformer) model, which outputs BEV features related to the historical trajectory of the moving obstacle.

[0041] For example, fully connected layers, convolutional layers, and other structures can be used to encode the coordinates, size, shape, and other information of the anchor frame into a fixed-dimensional query feature vector. A similarity weight between the query features and each region of the BEV feature map is calculated based on an attention mechanism model. Multi-scale features of the sampled moving obstacle are then weighted and fused based on these similarity weights to obtain the high-dimensional features of the moving obstacle (i.e., BEV features related to the historical trajectory of the moving obstacle).

[0042] Understandably, the high-dimensional features of a moving obstacle basically imply its physical and semantic features. For example, when the moving obstacle is another vehicle, its high-dimensional features include the vehicle type, size, location on the road, speed, and orientation of that other vehicle.

[0043] In some implementations, the BEV features related to the historical trajectory of the moving obstacle are input into the first prediction model to obtain the first predicted trajectory. This can be achieved by inputting the extracted high-dimensional features of the moving obstacle (i.e., the BEV features of the moving obstacle related to its historical trajectory) into a multilayer perceptron (MLP) (i.e., as the prediction model). The multilayer perceptron (MLP) decodes the first predicted trajectory of the moving obstacle.

[0044] Understandably, the first predicted trajectory is generated based on the global features of the BEV feature map, resulting in a relatively low-precision trajectory. Furthermore, the predicted first trajectory can provide the vehicle's perception with a general trajectory direction or range, which can be used subsequently to determine the local image features corresponding to the local region of each trajectory point from the target environment image.

[0045] S202, determine the local image features corresponding to the local region of each trajectory point in the target environment image.

[0046] In some embodiments, image features can be extracted from the target environment image to obtain a forward-looking feature image. Furthermore, if the trajectory points of the first predicted trajectory have vehicle trajectory point coordinates in the vehicle coordinate system, the image trajectory point coordinates in the forward-looking feature image are determined; based on the image trajectory point coordinates and the region range parameter, a local region of the trajectory point is determined from the forward-looking feature image; and based on the image features in the local region, local image features are obtained.

[0047] Understandably, since the target environment image includes a forward-looking image, image features at the forward-looking angle can be extracted from the target environment image to obtain a forward-looking feature image. Then, based on a coarse first predicted trajectory, local image features of the areas where the moving obstacle may potentially reach in the future can be sampled from the forward-looking feature image.

[0048] For example, the target environment image can be input into the same backbone network and neck network, which are mainly composed of convolutional layers, to obtain encoded multi-scale and multi-dimensional front view image features. The encoded multi-scale and multi-dimensional front view image features are then fused to obtain a front view feature image.

[0049] Understandably, the target environment image is captured by a forward-looking camera on the target vehicle. In this case, the coordinate system of the vehicle trajectory points can be transformed based on the camera parameters of the forward-looking camera to obtain the coordinates of the image trajectory points in the image coordinate system of the forward-looking feature image. Then, based on the coordinates of the image trajectory points and region range parameters (e.g., coordinate offset parameters), the local region of the trajectory points is determined from the forward-looking feature image; based on the image features in the local region, local image features are obtained. For example, bilinear interpolation can be used to extract local image features from the local region.

[0050] For example, refer to Figure 4 The diagram shows a flowchart for obtaining local image features. As shown, the first predicted trajectory includes trajectory points P_0, P_1, ..., P_n. In the vehicle coordinate system, the coordinates of trajectory point P_0 are (x0, y0, z0), the coordinates of trajectory point P_1 are (x1, y1, z1), ..., the coordinates of trajectory point P_n are (xn, yn, zn). Through the intrinsic and extrinsic parameters of the forward-looking camera, the coordinates of trajectory point P_0 (x0, y0, z0), trajectory point P_1 (x1, y1, z1), ..., trajectory point P_n (xn, yn, zn) are transformed and mapped to the forward-looking feature image to obtain the coordinates in the image coordinate system, which are (x0', y0'), (x1', y1'), ..., (xn', yn').

[0051] Understandably, image coordinate points can be used as sampling points. Based on coordinate offset coefficients, the sampling regions (i.e., local regions of the trajectory point) surrounding the sampling point in the forward-looking feature images of various dimensions (such as different scales and semantic channels) are determined. Features of areas where obstacles may pass through in the future are extracted from these sampling regions using bilinear interpolation. Since the sampling regions of forward-looking feature maps of different dimensions may not completely overlap, and single-dimensional sampling can only capture local, partial information, the features obtained after sampling need to be fused using a multilayer perceptron (MLP) to obtain complete local information about the trajectory point. For example, for trajectory point P_0, sampling is performed from three forward-looking feature maps of different dimensions (corresponding to fine-grained contour features, medium-grained texture features, and coarse-grained environmental features, respectively), resulting in one 128-dimensional sparse feature vector for each, i.e., three independent 128-dimensional features. These three 128-dimensional vectors are concatenated according to channel dimensions and fed into the MLP model, fusing to output a single 128-dimensional feature vector, which is the local image feature corresponding to the trajectory point.

[0052] S203, based on local image features, predicts the second predicted trajectory of the moving obstacle.

[0053] In some embodiments, a second predicted trajectory is predicted based on the vehicle trajectory point coordinates of the trajectory point, the local image features of the trajectory point, and the BEV features associated with the historical trajectory of the moving obstacle.

[0054] In some implementations, the vehicle trajectory coordinates of the trajectory points are mapped to trajectory point location encoding vectors that match the BEV feature dimensions related to the historical trajectory of the moving obstacle; based on the trajectory point location encoding vectors, the local image features of the trajectory points are fused into the BEV features related to the historical trajectory of the moving obstacle to obtain updated BEV features related to the historical trajectory of the moving obstacle; the updated BEV features related to the historical trajectory of the moving obstacle are input into the second prediction model to obtain the second predicted trajectory.

[0055] Understandably, after obtaining the local image features from the trajectory points of each moving obstacle, it is necessary to inject the spatial location information of the trajectory points into the BEV features related to the historical trajectory of the moving obstacle and the local image features of the trajectory points extracted from the front view image. This allows the local image features of the trajectory points to be fused into the BEV features related to the historical trajectory of the moving obstacle, resulting in updated BEV features related to the historical trajectory of the moving obstacle. At this point, the updated BEV features related to the historical trajectory of the moving obstacle incorporate environmental features of the areas the moving obstacle may reach in the future. This supplements the BEV features with a wider range of environmental information, such as the structure of distant lanes, overhead signs, and traffic light information, thereby enabling more accurate trajectory prediction.

[0056] For example, Figure 5 This diagram illustrates a specific process for predicting a second predicted trajectory of a moving obstacle based on local image features. The process primarily involves inputting the coordinates of vehicle trajectory points from different time periods (i.e., the coarse trajectory in the diagram), BEV features related to the historical trajectory of the moving obstacle (i.e., the BEV features of the moving obstacle in the diagram), and local image features of the trajectory points into a multi-head attention network. The network outputs updated BEV features related to the historical trajectory of the moving obstacle (i.e., the updated high-dimensional features of the moving obstacle in the diagram). These updated BEV features are then input into a multilayer perceptron (MLP) (i.e., as the second prediction model) to predict the second predicted trajectory (i.e., the fine trajectory).

[0057] Specifically, the coordinates of vehicle trajectory points can be mapped to a uniformly distributed feature space to obtain a position encoding vector. The BEV features related to the historical trajectory of the moving obstacle are used as the query feature vector Q of the moving obstacle. Then, the position encoding vector is added element-wise to this query feature vector Q to obtain the summed query feature of the moving obstacle. Furthermore, the position encoding vector is embedded into the local image features of the trajectory points, and a key (K) vector and a value (V) vector are obtained based on the embedded local image features. A multi-head cross-attention module can perform attention encoding on the summed query feature of the moving obstacle and the embedded local image features of the trajectory points to obtain the updated query feature of the moving obstacle, i.e., the updated BEV features related to the historical trajectory of the moving obstacle.

[0058] Understandably, injecting BEV features (i.e., moving obstacle query features) related to the historical trajectory of a moving obstacle and local image features of the trajectory point into the spatial location information of the trajectory point allows the multi-head attention network model to distinguish the spatial orientation and relative distance of different trajectory points, thereby accurately understanding the spatial dependency between the trajectory point and the forward-looking features (i.e., the local image features of the trajectory point). This enables the moving obstacle query features to further focus on a wider range of distant lane structures, aerial signs, traffic light information, etc., from the forward-looking features.

[0059] Understandably, the embodiments of this application can be understood as dividing the trajectory prediction method into two stages. The first stage is to predict a first predicted trajectory, and the second stage is to use the first predicted trajectory to obtain a second predicted trajectory. This allows the second predicted trajectory to sample features corresponding to the areas where the moving obstacle may potentially reach in the future from the forward-looking feature image based on the coarse trajectory obtained in the first stage. This results in obtaining the geometric features of the far-end lane and the semantic information of traffic lights and signs from the forward-looking feature image. While ensuring computational efficiency, the BEV paradigm is also optimized. By performing cross-attention calculation on the historically related BEV features of the moving obstacle and the local image features obtained from the forward-looking image, the shortcomings of insufficient accuracy in perceiving far-end local features and aerial elements based on the BEV feature map are compensated for. This enhances the physical constraints of static maps and traffic regulations on the predicted trajectory, effectively improving the self-consistency between the predicted trajectory and lane lines and traffic regulations.

[0060] This application also provides a schematic diagram of the structure of an electronic device 10'. The electronic device 10' may include at least one processor that executes instructions stored in a non-transitory computer-readable medium such as memory. The processor may be any conventional processor, such as a commercially available central processing unit (CPU). Optionally, the processor may be a special-purpose device such as an ASIC or other hardware-based processor. The memory may contain instructions (e.g., program logic) that can be executed by the processor to perform various functions of the vehicle 100, including the functions of the embodiments described above.

[0061] Next, combine Figure 6 The structure of electronic device 10' will be described.

[0062] like Figure 6 As shown, the electronic device 10' includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic unit 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) device 105. Wherein:

[0063] Processor 101 may include one or more processing units, such as a central processing unit, graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, field-programmable gate array (FPGA), neural network processing unit (NPU), etc., or a processing module or circuit that may include one or more single-core or multi-core processors. In some embodiments, the CPU may be used to optimize the neural network model to be run, and the NPU may be used to run the neural network model to be run. Processor 101 is used to execute any of the trajectory prediction methods described above.

[0064] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 102 can be used to store data provided by the aforementioned different services, such as sensor data, image data, or video data, and can also be used to store instructions for the trajectory prediction methods provided in the aforementioned embodiments.

[0065] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card. In other embodiments, the non-volatile memory 103 may be used to store instructions for the trajectory prediction methods provided in the foregoing embodiments.

[0066] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 10' to implement the trajectory prediction method provided in the embodiments of this application.

[0067] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 10', thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 10', for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 10' may communicate with other devices through the communication interface 104, for example, the electronic device 10' may obtain relevant data from other devices through the communication interface 104.

[0068] Input / output (I / O) device 105 can be an input device such as a keyboard or mouse, and an output device such as a monitor. Users can interact with electronic device 10' through input / output (I / O) device 105.

[0069] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface to other modules of the electronic device 10'. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface to the system memory 102 and the non-volatile memory 103.

[0070] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for the system control logic unit 106 to form a system-on-chip (SoC).

[0071] Understandable. Figure 6 The structure of the electronic device 10' shown is merely an example. In other embodiments, the electronic device 10' may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0072] It is understood that the trajectory prediction method mentioned in the embodiments of this application can be applied to electronic devices in vehicle 100. Figure 7 This is a schematic diagram of a possible functional framework of a vehicle 100 provided in an embodiment of this application.

[0073] like Figure 7 As shown, the functional framework of vehicle 100 may include various subsystems, such as the sensor system 10, control system 20, one or more peripheral devices 30 (one is shown as an example), power supply 40, and computer system 50. Optionally, vehicle 100 may also include other functional systems, such as an engine system that provides power to vehicle 100, etc., which are not limited herein. The sensor system 10 may include several detection devices that can sense the measured information and convert the sensed information into electrical signals or other required forms of information output according to a certain rule.

[0074] As shown in the figure, these detection devices may include a Global Positioning System (GPS) 11, a vehicle speed sensor 12, an Inertial Measurement Unit (IMU) 13, etc., and this application is not limited thereto. The GPS 11 is a system that uses GPS positioning satellites to perform real-time positioning and navigation globally. In this application, the GPS 11 can be used to achieve real-time positioning of the vehicle 100, providing the vehicle 100's geographical location information. The vehicle speed sensor 12 is used to detect the vehicle speed of the vehicle 100. The inertial measurement unit 13 may include a combination of an accelerometer and a gyroscope, and is a device for measuring the angular rate and acceleration of the vehicle 100. For example, during the movement of the vehicle 100, the inertial measurement unit can measure the changes in the vehicle's position and angle based on the vehicle's inertial acceleration, such as measuring the vehicle's acceleration and angular rate.

[0075] The control system 20 may include a steering unit 21, a braking unit 22, etc. The steering unit 21 may represent a system for adjusting the direction of travel of the vehicle 100, and may include, but is not limited to, a steering wheel or other structural devices for adjusting or controlling the direction of travel of the vehicle 100. The braking unit 22 may represent a system for slowing down the vehicle 100, and may also be referred to as the vehicle 100 braking system. It may include, but is not limited to, a brake controller, a reducer, or other structural devices for slowing down the vehicle 100. In practical applications, the braking unit 22 may utilize friction to slow down the tires of the vehicle 100, thereby slowing down the vehicle 100's speed.

[0076] Peripheral device 30 may include several components, such as the communication system 31, touch screen 32, user interface 33, etc., as shown in the figure. The communication system 31 is used to enable network communication between vehicle 100 and other devices besides vehicle 100. In practical applications, the communication system 31 can employ wireless communication technology or wired communication technology to achieve network communication between vehicle 100 and other devices. This wired communication technology can refer to communication between vehicle 100 and other devices via network cable or fiber optic cable, etc. This wireless communication technology includes, but is not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technology, etc. The touchscreen 32 can be used to detect operation commands displayed on the touchscreen 32. For example, the user can perform touch operations on the content data displayed on the touchscreen 32 according to actual needs to achieve the corresponding function, such as playing music, video, or other multimedia files. User interface 33 may specifically be a touch panel for detecting operation commands on the touch panel. User interface 33 may also be a physical button or a mouse. User interface 33 may also be a display screen for outputting data and displaying images or data. Optionally, user interface 33 may also be at least one device belonging to the category of peripheral devices, such as a touch screen, microphone, and speaker.

[0077] Several functions of the vehicle 100 are controlled and implemented by the computer system 50. The computer system 50 may include multiple processors such as a general-purpose processor 51, a CDC 52, an MDC 53, a T-BOX 54, as well as a memory 55 (also referred to as a storage device) and a gateway 56.

[0078] In practical applications, the memory 55 can be located either inside or outside the computer system 50, such as as a cache in the vehicle 100; this application does not impose any limitations on this. The general-purpose processor 51 can be, for example, a graphics processing unit (GPU). The general-purpose processor 51, CDC 52, MDC 53, and T-BOX 54 can be used to run relevant programs or corresponding instructions stored in the memory 55 to implement the corresponding functions of the vehicle 100.

[0079] In some implementations, the general-purpose processor 51 can acquire a first predicted trajectory of the moving obstacle and a target environment image captured by the target vehicle. The first predicted trajectory is determined based on the BEV feature map of the environment in which the moving obstacle is located. The first predicted trajectory includes multiple trajectory points, and the target environment image includes a front view image, which includes a local region corresponding to at least one of the multiple trajectory points. Local image features corresponding to the local region of each trajectory point in the target environment image are determined. Based on the local image features, a second predicted trajectory of the moving obstacle is predicted. This can be used to avoid the problem of inaccurate trajectory prediction for moving obstacles.

[0080] The memory 55 may include volatile memory, such as RAM; it may also include non-volatile memory, such as ROM, flash memory, HDD, or SSD; or it may include a combination of the above types of memory. The memory 55 can be used to store a set of program code or instructions corresponding to program code, so that the general-purpose processor 51 can call the program code or instructions stored in the memory 55 to implement the corresponding functions of the vehicle 100. This function includes, but is not limited to, […]. Figure 7 The schematic diagram of the functional framework of vehicle 100 shown includes some or all of the functions. In this application, memory 55 can store a set of program code for controlling vehicle 100. General-purpose processor 51, CDC 52, MDC 53, and T-BOX 54 can call this program code to control vehicle 100 to perform vehicle network switching.

[0081] Optionally, in addition to storing program code or instructions, memory 55 may also store road maps, driving lines, or other structural devices used to adjust or control the direction of travel of vehicle 100.

[0082] It should be noted that the above Figure 7 This is merely a schematic diagram of one possible functional framework for vehicle 100. In practical applications, vehicle 100 may include more or fewer systems or components, and this application is not limiting. Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0083] It should be understood that Figure 7 The functional framework structure of the vehicle 100 shown is only an example. In other embodiments, the vehicle 100 may include more or fewer modules, which is not limited herein.

[0084] This application also provides a computer program product that, when executed on a device, enables the device to implement the methods provided in the foregoing embodiments. For example, it can implement methods such as... Figure 2 The trajectory prediction method shown.

[0085] This application also provides a computer-readable storage medium storing one or more programs, which, when executed by a device, cause the device to implement the methods provided in the foregoing embodiments. For example, implementing... Figure 2 The trajectory prediction method shown.

[0086] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0087] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0088] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0089] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon by one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0090] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0091] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0092] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0093] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A trajectory prediction method, characterized in that, The method includes: The first predicted trajectory of a moving obstacle and a target environment image captured by a target vehicle are obtained. The first predicted trajectory is determined based on the BEV feature map of the environment in which the moving obstacle is located. The first predicted trajectory includes multiple trajectory points. The target environment image includes a front view image, and the front view image includes a local region corresponding to at least one of the multiple trajectory points. The local image features corresponding to the local region of each trajectory point in the target environment image are determined; Based on the local image features, a second predicted trajectory of the moving obstacle is predicted; Wherein, the trajectory points of the first predicted trajectory have the coordinates of vehicle trajectory points in the vehicle coordinate system; and The step of determining the local image features corresponding to the local region of each trajectory point in the target environment image includes: Extract image features from the target environment image to obtain a forward-looking feature image; Determine the coordinates of the trajectory point in the forward-looking feature image; Based on the coordinates of the trajectory points in the image and the region range parameters, the local region of the trajectory points is determined from the forward-looking feature map; The local image features are obtained based on the image features in the local region.

2. The trajectory prediction method according to claim 1, characterized in that, The target environment image was captured by the target vehicle using a forward-facing camera; and Determining the coordinates of the trajectory point in the forward-looking feature image includes: Obtain the coordinates of the vehicle trajectory point in the vehicle coordinate system; Based on the camera parameters of the forward-looking camera, the coordinate system of the vehicle trajectory point coordinates is transformed to obtain the image trajectory point coordinates in the image coordinate system of the forward-looking feature image.

3. The trajectory prediction method according to claim 1, characterized in that, The process of obtaining the local image features based on the image features in the local region includes: The local image features are extracted from the local region using bilinear interpolation.

4. The trajectory prediction method according to claim 1, characterized in that, The acquisition of the first predicted trajectory of the moving obstacle and the target environment image captured by the target vehicle includes: Obtain the BEV feature map of the environment in which the moving obstacle is located; Based on the query features of the moving obstacle, BEV features related to the historical trajectory of the moving obstacle are matched in the BEV feature map; The BEV features associated with the historical trajectory of the moving obstacle are input into the first prediction model to obtain the first predicted trajectory.

5. The trajectory prediction method according to claim 4, characterized in that, The step of predicting the second predicted trajectory of the moving obstacle based on the local image features includes: The second predicted trajectory is obtained based on the vehicle trajectory point coordinates of the trajectory point, the local image features of the trajectory point, and the BEV features related to the historical trajectory of the moving obstacle.

6. The trajectory prediction method according to claim 5, characterized in that, The step of predicting the second predicted trajectory based on the vehicle trajectory point coordinates, local image features of the trajectory point, and BEV features related to the historical trajectory of the moving obstacle includes: The vehicle trajectory point coordinates of the trajectory point are mapped to a trajectory point position encoding vector that matches the BEV feature dimension related to the historical trajectory of the moving obstacle. Based on the trajectory point position encoding vector, the local image features of the trajectory point are fused into the BEV features related to the historical trajectory of the moving obstacle to obtain the updated BEV features related to the historical trajectory of the moving obstacle. The updated BEV features associated with the historical trajectory of the moving obstacle are input into the second prediction model to obtain the second predicted trajectory.

7. An electronic device, characterized in that, include: Memory, used to store instructions; At least one processor is configured to execute the instructions to cause the electronic device to implement the trajectory prediction method of any one of claims 1-6.

8. A vehicle, characterized in that, include: Memory, used to store instructions; At least one processor is configured to execute the instructions to cause the vehicle to implement the trajectory prediction method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the trajectory prediction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, When the computer program product is run on the device, it causes the device to execute the trajectory prediction method according to any one of claims 1 to 6.

Images