A path planning method and device

Abstract

A path planning method and device, the path planning method comprising: obtaining perception data (210); inputting the perception data into an encoding network to output scene features (202), the scene features comprising first features and second features; then inputting the first features into a first decoder to output a first heat flow map implicit field (203); then inputting the second features and the first heat flow map implicit field into a second decoder to output a second heat flow map implicit field (204), the first heat flow map implicit field and the second heat flow map implicit field representing probabilities of a first device and at least one second device appearing at various positions within a preset time period; and determining a driving path of the first device according to the first heat flow map implicit field and the second heat flow map implicit field (205). The accuracy and consistency of interaction prediction results of a self vehicle and other vehicles are improved.

Description

A path planning method and apparatus Technical Field

[0001] This application relates to the field of intelligent driving, and more particularly to a path planning method and apparatus. Background Technology

[0002] In the field of smart devices, planning and prediction networks are typically used to predict the behavioral decisions of smart devices. These networks are usually trained using a multi-head, multi-task training model. Each head's task in the network is usually a sub-task in the overall planning process, and these sub-tasks are interconnected. Although a multi-task training framework can support a single network learning multiple tasks, the consistency between tasks is poor. Inconsistencies between multiple tasks will lead to the fragmentation of the entire prediction and planning solution, making it impossible to use them uniformly.

[0003] For example, in an existing intelligent driving solution, historical information about the vehicle and other vehicles, traffic lights, and high-precision maps are input. A graph neural network (GNN) is used to process and encode the features of the vehicle, other vehicles, and road topology. In the decoding stage, the encoded features of the vehicle and other vehicles are hierarchically decoded into a heatmap over a fixed time period, and path completion is performed using a fully connected network. Regarding multimodal processing, the multimodal data of the vehicle and other vehicles are rearranged and combined, and modality fusion is performed using cross-attention to obtain the vehicle and other vehicle modalities. However, this solution has a relatively simplistic approach to planning the vehicle's modality and cannot develop decision-making schemes that are more suitable for real-world scenarios.

[0004] Therefore, how to plan more suitable behavioral decisions for smart devices has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a path planning method and apparatus for using a heat flow graph to represent the full future planning and control information of the vehicle and other vehicles, and outputting the implicit fields of other vehicles based on the implicit fields of the vehicle. Therefore, the predicted implicit fields of other vehicles are more accurate and reasonable, and the interaction prediction results of the vehicle and other vehicles are more accurate and consistent.

[0006] In view of the above, in a first aspect, this application provides a path planning method, comprising: firstly, acquiring perception data, which includes data collected by at least one sensor in a first device, such as image, video, or point cloud data; then inputting the perception data into an encoding network to output scene features, which include a first feature and a second feature, wherein the first feature is a feature of the first device, and the second feature is a feature of at least one second device in the scene; then inputting the first feature into a first decoder to output a first heatmap implicit field, which represents the probability of the first device appearing at each location within a preset time period; then inputting the second feature and the first heatmap implicit field into a second decoder to output a second heatmap implicit field, which represents the probability of at least one second device appearing at each location within a preset time period; and determining the driving path of the first device based on the first heatmap implicit field and the second heatmap implicit field.

[0007] In this embodiment, a heatmap can be used to represent the positional changes of the first device and the second device within a preset time period. Furthermore, when outputting the implicit heatmap field of the second device, the features of the first device can be used as input, allowing the second decoder to refer to the implicit heatmap field of the first device and the features of the second device, taking into account the relative positional changes of the first device and the second device in space, thereby outputting a more consistent implicit heatmap field of the first device and the implicit heatmap field of the second device, and thus outputting a more accurate travel path of the first device.

[0008] In one possible implementation, the aforementioned method further includes: acquiring a query representation, the query representation including at least one corresponding representation of location or time, the location being the spatiotemporal location of the first device corresponding to the sensing data, such as spatial coordinates or temporal coordinates; the aforementioned inputting the first feature into the first decoder and outputting the first heat flow map implicit field includes: inputting the first feature and the representation feature into the first decoder and outputting the first heat flow map implicit field.

[0009] Therefore, in the embodiments of this application, when outputting the implicit field of the first thermal flow map, the spatiotemporal coordinates can be used as a query for decoding, thereby querying the location of the first device under multiple spatiotemporal coordinates and outputting a more continuous implicit field of the thermal flow map.

[0010] In one possible implementation, the aforementioned first decoder includes a first query decoding layer; the first query decoding layer is used to decode based on the first feature and the query representation to obtain the first heat flow graph implicit field. In this application embodiment, a query decoding layer can be set to interact with the features and query representations of the first device, thereby outputting the heat flow graph implicit field containing each query spatiotemporal coordinate.

[0011] In one possible implementation, the aforementioned inputting the second feature and the first heatmap implicit field to the second decoder and outputting the second heatmap implicit field includes: inputting the first heatmap implicit field and the second feature to the second decoder and outputting the second heatmap implicit field.

[0012] In this embodiment of the application, a second decoder can be used to decode the implicit field of the thermal flow graph of the first device and the features of the second device. The second decoding layer can be used to identify the possible mutual influence between the first device and the second device, so as to obtain a more accurate implicit field of the second thermal flow graph.

[0013] In one possible implementation, the aforementioned second decoder includes an interaction encoding layer and a second query decoding layer. The interaction encoding layer is used to interact with the second feature and the first heatmap implicit field to obtain the interaction feature. The second query decoding layer is used to decode based on the interaction feature and the query representation to obtain the second heatmap implicit field. In this embodiment, an interaction encoding layer is set to encode the possible interactions between the first device and the second device, so that the implicit heatmap field of the second device with better consistency with the implicit heatmap field of the first device can be output subsequently, so as to plan a more accurate path for the first device.

[0014] In one possible implementation, the aforementioned determination of the travel path of the first device based on the implicit field of the first thermal flow map and the implicit field of the second thermal flow map includes: determining at least one first path of the first device based on the implicit field of the first thermal flow map; determining at least one second path corresponding to each first path based on the implicit field of the second thermal flow map; and determining the travel path based on the at least one first path and the at least one second path.

[0015] In this embodiment, a path can be planned for the first device based on the implicit fields of the first and second thermal flow maps with better consistency. This allows for consideration of the possible paths of the second device under each path of the first device, fully taking into account the mutual influence between devices and outputting a more accurate planned path.

[0016] In one possible implementation, the aforementioned determination of the driving path based on at least one first path and at least one second path includes: obtaining the intersection point between each of the at least one first path and at least one corresponding second path; determining the interaction information between the first device and at least one second device based on the intersection point, the interaction information including the interaction between the first device and at least one second device when the first device is driving along at least one first path; and determining the driving path based on the interaction information.

[0017] In this embodiment of the application, for the possible interactions between the first device and the second device, various possible interactions can be determined by simulating modal interactions, thereby planning a more accurate path for the first device based on the possible interactions.

[0018] In one possible implementation, the aforementioned determination of at least one first path for the first device based on the implicit field of the first heat flow map includes: sampling from the implicit field of the first heat flow map to obtain at least one first path. In this application embodiment, when determining the path of the first device, sampling can be performed from the first heat flow map, thereby utilizing the possible positional changes of the first device in space within a preset time period contained in the first heat flow map to determine the more likely position in space, thereby piecing together a possible travel path.

[0019] In one possible implementation, the aforementioned sampling from the implicit field of the first heat flow map to obtain at least one first path may include: sampling from the implicit field of the first heat flow map at preset time intervals to obtain at least one first path; or, obtaining at least one candidate path from the first heat flow map and determining at least one first path from the at least one candidate path based on a prior path; or, searching for local peak points from the first heat flow map at multiple times, selecting multiple peak points based on the distinguishability between the multiple local branch points obtained, and generating at least one first path based on the multiple peak points.

[0020] In this application, multiple possible path sampling methods are provided, so the appropriate sampling method can be selected according to the actual scenario, which has stronger generalization.

[0021] In one possible implementation, the aforementioned method further includes: acquiring a training dataset, the training dataset including perception data and motion paths of multiple third devices; and performing self-supervised training on an initial encoding network, an initial first decoder, and an initial second decoder using the training dataset to obtain a trained encoding network, a first decoder, and a second decoder.

[0022] In this embodiment of the application, the encoding network, the first decoder and the second decoder can also be pre-trained, so that the trained encoding network, the first decoder and the second decoder can output more accurate scene features and heat flow map implicit fields of each device.

[0023] Secondly, this application provides a path planning device, comprising:

[0024] An acquisition module is used to acquire sensing data, which includes data collected by at least one sensor in the first device.

[0025] The encoding module is used to input the perceived data into the encoding network and output scene features. The scene features include a first feature and a second feature. The first feature is the feature of the first device, and the second feature is the feature of at least one second device in the scene.

[0026] The first decoding module is used to input the first feature into the first decoder and output the first heat flow map implicit field, which is used to represent the probability of the first device appearing at each location within a preset time period.

[0027] The second decoding module is used to input the second feature and the first heatmap implicit field into the second decoder and output the second heatmap implicit field, which represents the probability that at least one second device appears at each location within a preset time period.

[0028] The planning module is used to determine the travel path of the first device based on the implicit fields of the first and second thermal flow maps.

[0029] The effects achieved by the second aspect or any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated here.

[0030] In one possible implementation, the aforementioned first decoding module is specifically used to: acquire a query representation, the query representation including at least one corresponding representation of location or time, the location being the location of the first device corresponding to the sensing data; input the first feature and the representation feature to the first decoder, and output the first heat flow map implicit field.

[0031] In one possible implementation, the aforementioned first decoder includes a first query decoding layer; the first query decoding layer is used to decode based on the first feature and the query representation to obtain the first heat flow graph implicit field.

[0032] In one possible implementation, the aforementioned second decoding module is specifically used to: input the first heat flow graph implicit field and the second feature to the second decoder, and output the second heat flow graph implicit field.

[0033] In one possible implementation, the aforementioned second decoder includes an interactive coding layer and a second query decoding layer;

[0034] The interactive coding layer is used to interact the second feature with the implicit field of the first heatmap to obtain the interactive feature;

[0035] The second query decoding layer is used to decode based on interaction features and query representations to obtain the implicit field of the second heat flow graph.

[0036] In one possible implementation, the aforementioned planning module is specifically used to: determine at least one first path for the first device based on the implicit field of the first heat flow map; determine at least one second path corresponding to each first path based on the implicit field of the second heat flow map; and determine a travel path based on the at least one first path and the at least one second path.

[0037] In one possible implementation, the aforementioned planning module is specifically used to: obtain the intersection point between each of the first paths in at least one first path and at least one second path corresponding to each first path; determine the interaction information between the first device and at least one second device based on the intersection point, the interaction information including the interaction between the first device and at least one second device when the first device travels along at least one first path; and determine the travel path based on the interaction information.

[0038] In one possible implementation, the aforementioned planning module is specifically used to sample from the implicit field of the first thermal flow graph to obtain at least one first path.

[0039] In one possible implementation, the aforementioned planning module is specifically used to: sample from the implicit field of the first heat flow map at preset time intervals to obtain at least one first path; or, obtain at least one candidate path from the first heat flow map and determine at least one first path from the at least one candidate path based on the prior path; or, search for local peak points from the first heat flow map at multiple times, select multiple peak points based on the distinguishability between the multiple local branch points obtained, and generate at least one first path based on the multiple peak points.

[0040] In one possible implementation, the aforementioned apparatus further includes a training module for acquiring a training dataset, the training dataset including perception data and motion paths of multiple third devices; and using the training dataset to perform self-supervised training on an initial encoding network, an initial first decoder, and an initial second decoder to obtain a trained encoding network, a first decoder, and a second decoder.

[0041] Thirdly, embodiments of this application provide a computing device including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0042] Fourthly, embodiments of this application provide a vehicle including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0043] Fifthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0044] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0045] In a seventh aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0046] Figure 1 is a structural schematic diagram of a vehicle provided in an embodiment of this application;

[0047] Figure 2 is a flowchart illustrating a path planning method provided in an embodiment of this application;

[0048] Figure 3 is a schematic diagram of the processing flow of an intelligent driving system for a vehicle provided in an embodiment of this application;

[0049] Figure 4 is a schematic diagram of a neural network architecture provided in an embodiment of this application;

[0050] Figure 5 is a schematic diagram of another neural network architecture provided in an embodiment of this application;

[0051] Figure 6 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0052] Figure 7 is a schematic diagram of a visual heat flow diagram provided in an embodiment of this application;

[0053] Figure 8 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0054] Figure 9 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0055] Figure 10 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0056] Figure 11 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0057] Figure 12 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0058] Figure 13 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0059] Figure 14 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0060] Figure 15 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0061] Figure 16 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0062] Figure 17 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0063] Figure 18 is a schematic diagram of a driving path provided in an embodiment of this application;

[0064] Figure 19 is a schematic diagram of a driving path provided in an embodiment of this application;

[0065] Figure 20 is a flowchart illustrating another path planning method provided in an embodiment of this application;

[0066] Figure 21 is a schematic diagram of a sampling path provided in an embodiment of this application;

[0067] Figure 22 is a schematic diagram of a path planning device provided in an embodiment of this application;

[0068] Figure 23 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0070] To facilitate understanding, some concepts or categories involved in the embodiments of this application will be explained below.

[0071] (1) transformer

[0072] A transformer architecture is a feature extraction network that includes both an encoder and a decoder. Of course, in some cases, a transformer architecture may not include an encoder but may include a decoder.

[0073] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0074] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0075] For example, the structure of a Transformer layer in an existing scheme may include a multi-head attention network and a feedforward network module. Taking natural language processing as an example, the multi-head attention network obtains corresponding weight values ​​by calculating the relevance between words, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0076] (2) Attention mechanism

[0077] Attention mechanisms can quickly extract important features from sparse data. They provide an effective modeling approach for capturing global contextual information through QKV (key-value pairs). Assuming the input is Q(query), and the context is stored as key-value pairs, the attention mechanism is essentially a mapping function from the query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query Q to a series of KV pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0078] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between Q and each K to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights to obtain a probability distribution where the sum of all weight coefficients is 1, and also highlight the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​V are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0079] Where d is the dimension of matrix Q,K.

[0080] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0081] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, has the advantage of global focus; it can obtain global spatial information about the feature map through simple lookups and assignments.

[0082] The query features or query conditions mentioned in the following embodiments of this application refer to data as Q input to the encoder or decoder.

[0083] (3) Loss Function

[0084] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, pre-configuring parameters for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. Common loss functions include mean squared error, cross-entropy, logarithmic, and exponential loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0085] (4) Self-supervised learning

[0086] This can also be called contrastive learning, which involves constructing pretext tasks. Self-supervised learning, through these pretext tasks, enables deep learning-based feature extractors (encoders) to encode deep features of point clouds even without label information. This makes the encoder's output effective for downstream tasks such as classification and segmentation.

[0087] Contrastive learning is a special type of self-supervised learning scheme. Unlike the encoder-decoder scheme of generative learning, it focuses on learning the common features among instances of the same class and distinguishing the differences between instances of different classes. The core of contrastive learning lies in differentiation; its encoder is better at learning overall, discriminative features.

[0088] (5) Implicit Field

[0089] An implicit field is a field characterized by fitting a continuous function using a neural network, where points in space are mapped to specific attributes. In the embodiments of this application, the implicit field can represent the probability that the first device is mapped to various locations in spacetime (i.e., time and space).

[0090] (6) Heatmap Flow (HMF)

[0091] A heatmap is a visual chart that uses color to represent the density or intensity of data. It shows the distribution and changes of data by mapping data to different colors.

[0092] (7) Conditional heatmap flow (CHMF)

[0093] A conditional heatmap is a heatmap based on specific conditions, which uses color to represent the distribution or intensity of data under different conditions.

[0094] In a conditional heatmap, data points are mapped to different colors based on whether they meet or not specific conditions, thus displaying the distribution pattern or intensity change of data under different conditions. In this embodiment, taking the planning of a vehicle's path as an example, the location of the vehicle can be used as a query condition to output the heatmap of other vehicles, i.e., the CHMF of other vehicles.

[0095] (8) Implicit field of heat flow diagram

[0096] In other words, the implicit field is represented by a heat flow diagram. In the embodiments of this application, the implicit field can represent the probability distribution of the vehicle or other vehicles at various locations in space and time. By combining the implicit field and the heat flow diagram, the probability changes of the vehicle or other vehicles at various locations in space and time over a period of time can be represented by the heat flow diagram.

[0097] (9) Implicit field of conditional thermoflow diagram

[0098] In other words, implicit fields are represented by conditional heatmaps. Implicit fields can represent the probability distribution of one's own vehicle or other vehicles at various locations in space and time. By combining implicit fields with conditional heatmaps, the probability changes of one's own vehicle or other vehicles at various locations in space and time over a period of time can be represented by conditional heatmaps.

[0099] The method provided in this application can be applied to various electronic devices, such as vehicles, humanoid robots, robotic vacuum cleaners, transport equipment in logistics systems, or other vehicle-mounted equipment. Based on data collected by sensors in the electronic device, it can identify the possible movement paths of the electronic device itself and the paths of other devices in the environment, and plan a driving path for the electronic device that avoids other devices.

[0100] In this application embodiment, the method provided in this application embodiment is exemplarily described using a vehicle as an example. The vehicle mentioned below can also be replaced with other electronic devices, and correspondingly, the architecture of the vehicle mentioned below can also be reduced or adjusted to obtain the architecture of other electronic devices.

[0101] For example, for intelligent vehicles (or intelligent driving vehicles, vehicles equipped with intelligent driving systems, etc.), the vehicle's perception of the environment and recognition of road structure are crucial. The intelligent driving functions of a vehicle can include modules such as perception, prediction, decision-making, planning, and control. The decision-making, planning, and control module can also be called the planning and control function. The method provided in this application embodiment can be deployed in the prediction module, or it can be deployed as a separate module between the perception module and the prediction module. Furthermore, if the aforementioned functional modules are implemented through an end-to-end model, the method provided in this application embodiment can also be deployed in the end-to-end module, depending on the actual application scenario. Using the method provided in this application embodiment, the driving path of the vehicle itself, as well as the possible driving paths of other vehicles along that path, can be predicted, thus enabling more accurate driving path planning for the vehicle.

[0102] Referring to Figure 1, which is a schematic diagram of a vehicle structure provided in an embodiment of this application, the vehicle 100 can be configured in an intelligent driving mode. For example, the vehicle 100 can control itself while in intelligent driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine whether there are obstacles in the surrounding environment, and control the vehicle 100 based on the obstacle information. When the vehicle 100 is in intelligent driving mode, it can also be set to operate without human interaction.

[0103] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to a full or partial intelligent driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an intelligent driving strategy based on the analysis of the surrounding environmental information to achieve full intelligent driving, or present the analysis results to the user to achieve partial intelligent driving.

[0104] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0105] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0106] Communication system 111 may include wireless communication system 111, which can communicate wirelessly with one or more devices directly or via a communication network. For example, wireless communication system 111 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. Wireless communication system 111 may communicate using WiFi and wireless local area network (WLAN). In some embodiments, wireless communication system 111 may communicate directly with devices using infrared links, Bluetooth, or ZigBee. Wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0107] In this embodiment of the application, communication between the vehicle and the cloud can be achieved through the communication system 111.

[0108] The entertainment system 112 may include a central control screen, a microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system 112; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate the system. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0109] The interactive interface provided in this application embodiment can be displayed in the infotainment system. Specifically, it can be displayed on devices in the vehicle, such as the central control screen, passenger screen, instrument panel screen, and head-up display, which can be used to display interactive interfaces to users. Users can perform interactive operations based on the interactive interface, which may include, but are not limited to, touch interaction, voice interaction, and gesture interaction, depending on the actual interaction methods supported by the vehicle.

[0110] The interactive interface mentioned below in this application may also include the interactive interface in a user terminal, where users can perform interactive operations and feedback content can be displayed to users through the interactive interface of the terminal.

[0111] Alternatively, in some scenarios, the vehicle can connect to the user terminal, as shown in Figure 2 below. For example, the user can interact with the vehicle's interface, and feedback content can be displayed on the user terminal's screen. Or, the user can interact with the terminal's interface, and feedback content can be displayed on the vehicle's interface, thus achieving linkage between the vehicle and the user terminal and improving the user experience.

[0112] The navigation system 113 may include map services to provide navigation for the vehicle 100, and the navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map may be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the vehicle's operation.

[0113] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (which may include a global navigation satellite system (GNSS), specifically GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU) 122, a lidar (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the vehicle 100. The perception dataset mentioned below may include data collected by various units in the perception system 120, such as point cloud data collected by lidar or environmental images collected by the camera device.

[0114] The Global Positioning System 121 can be used to determine the geographical location of vehicle 100.

[0115] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0116] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0117] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or direction of travel of objects.

[0118] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0119] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still image information or video stream information.

[0120] The decision control system 130 includes a computing system 131 that analyzes and makes decisions based on information acquired by the sensing system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, and a steering system 133, a throttle 134, and a braking system 135 for controlling the vehicle 100.

[0121] The computing system 131 can process and analyze various information acquired by the perception system 120 to identify targets, objects, and / or features in the environment surrounding the vehicle 100. The targets may include pedestrians or animals, and the objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 131 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other techniques. In some embodiments, the computing system 131 may be used to map the environment, track objects, estimate object speeds, etc. The computing system 131 can analyze the acquired information and derive a control strategy for the vehicle.

[0122] The vehicle controller 132 can be used to coordinate the control of the vehicle's power battery and engine 141 to improve the power performance of the vehicle 100.

[0123] The steering system 133 can be used to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0124] The throttle 134 is used to control the operating speed of the engine 141 and thus the speed of the vehicle 100.

[0125] Braking system 135 is used to control the deceleration of vehicle 100. Braking system 135 can use friction to slow down the rotational speed of wheel 144. In some embodiments, braking system 135 can convert the kinetic energy of wheel 144 into electric current. Braking system 135 may also take other forms to slow down the rotational speed of wheel 144 to control the speed of vehicle 100.

[0126] The drive system 140 includes components that provide powered motion to the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144. The engine 141 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 141 converts the energy source 142 into mechanical energy.

[0127] Examples of energy sources 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 142 may also provide energy to other systems of vehicle 100.

[0128] The drivetrain 143 transmits mechanical power from the engine 141 to the wheels 144. The drivetrain 143 may include a gearbox, a differential, and a drive shaft. In one embodiment, the drivetrain 143 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 144.

[0129] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151, which can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0130] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), a system-on-chip (SoC), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 can be located on a device remote from the vehicle and can communicate wirelessly with the vehicle.

[0131] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100. Memory 152 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 110, perception system 120, decision control system 130, and drive system 140.

[0132] In the method provided in this application embodiment, the steps performed by the vehicle can be executed by the computing platform 150. For example, the processor 151 can read the program stored in the memory 152, display the interactive interface on the display screen, and execute subsequent processes based on the user's interactive operation.

[0133] In addition to instruction 153, memory 152 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. It may also store higher-precision maps obtained by the methods provided in this application embodiment. This information can be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0134] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0135] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0136] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be construed as a limitation on the embodiments of this application.

[0137] The aforementioned vehicle 100 can be any vehicle or vehicle-mounted terminal capable of intelligent driving, such as a car, truck, motorcycle, bus, ship, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train. This application embodiment does not impose any special limitations on this type of vehicle.

[0138] Typically, intelligent driving systems in vehicles include multiple modules. For prediction networks, a multi-head, multi-task training model is often used. The tasks performed by the prediction network are usually divided into multiple sub-tasks, which are interconnected. Each head network in the prediction network focuses on a sub-task of planning and prediction. Although a multi-task training framework can support a single network learning multiple tasks, the consistency between tasks is poor. Inconsistencies between multiple tasks will lead to a fragmentation of the overall prediction and planning solution, making unified application impossible. For example, there is a consistency problem in planning the behavior of the vehicle itself and predicting the behavior of other vehicles. The behavior of both the vehicle and other vehicles is multimodal, so there may be interactions between the vehicle and other vehicles. Given the poor consistency of multi-task learning, it is usually difficult to learn a more accurate description of the interaction scenarios between the vehicle and other vehicles in multimodal scenarios.

[0139] Therefore, this application provides a path planning method and apparatus that can be used to interact with the features of a vehicle and other vehicles to plan the driving path of the vehicle. It can fully consider the possible interactions between the vehicle and other vehicles and improve the accuracy of the vehicle's path planning.

[0140] Based on the aforementioned vehicle architecture, the method flow provided in the embodiments of this application will be described below.

[0141] Referring to Figure 2, a flowchart of a path planning method provided in an embodiment of this application includes:

[0142] 201. Acquire sensory data.

[0143] This application embodiment can be deployed in a first device or other electronic devices connected to the first device. The sensing data includes data collected by at least one sensor in the first device, and the sensing data may include, but is not limited to, images, point clouds, or other types of data.

[0144] For example, in conjunction with the aforementioned vehicle architecture, the perception data may include data collected by various types of sensors in the aforementioned perception system 120 during vehicle operation.

[0145] 202. Input the perceived data into the encoding network and output the scene features.

[0146] The scene features may include a first feature and a second feature, wherein the first feature is a feature of a first device and the second feature is a feature of at least one second device in the scene.

[0147] This encoding network can be a pre-trained network used to encode input data and output scene features that can represent the characteristics of each target. Typically, different processing modules can be set in the encoding network for different types of input data. For example, corresponding feature extraction modules can be set for image or point cloud data, so that features can be extracted from different types of inputs, mapped to the same space, and encoded to obtain features that can be used to represent the overall scene, i.e., scene features.

[0148] Typically, scene features include features of multiple targets. The recognition module set up by the coding network can identify each target in the scene and mark each target in the scene, such as the vehicle itself and other vehicles, thereby distinguishing the features of the first device and the second device.

[0149] For example, in the intelligent driving function of a vehicle, the first device may include the vehicle itself, and the second device may include other vehicles or other obstacles in the environment.

[0150] For example, during the flight of a drone, the first device may specifically include the drone itself, and the second device may include other moving objects in space.

[0151] For example, during the movement of the intelligent robot, the first device specifically refers to the intelligent robot, while the second device refers to the environment in which the intelligent robot is located, such as pedestrians, vehicles, or other moving objects.

[0152] 203. Input the first feature into the first decoder and output the first heat flow map implicit field.

[0153] The implicit field of the first heat flow map is used to represent the probability of the first device appearing at various locations within a preset time period, or the probability of the first device appearing at various coordinates in space-time. It can also be understood as representing the probability change of the first device appearing at various spatial locations within various time periods in the form of a heat flow map.

[0154] In one possible implementation, a query representation can be obtained, which includes at least one corresponding representation of location or time, where the location is the position of the first device corresponding to the sensed data. For example, query coordinates can be generated, which include a location or time point in spatiotemporal space, and the query coordinates can be converted into a query representation that is easy to process using an embedding layer. Subsequently, the first feature and the representation feature are input to the first decoder, and the first heatmap implicit field is output. For example, combined with the aforementioned attention mechanism, the query representation and the first feature can be used as (Q, K) to obtain the corresponding output result, that is, the first heatmap implicit field can be a conditional heatmap implicit field.

[0155] Optionally, a first query decoding layer may be provided in the first decoder. The first query decoding layer is used to decode based on the first feature and the query representation to obtain the implicit field of the first heat flow graph.

[0156] 204. Input the second feature and the implicit field of the first heat flow graph into the second decoder, and output the implicit field of the second heat flow graph.

[0157] The implicit field of the second heat flow map represents the probability of at least one second device appearing at various locations within a preset time period, or the probability of each coordinate of the second device in spacetime.

[0158] This can be understood as follows: when decoding one or more second devices, the implicit field of the first heatmap and the second feature of the first device can both be input into the second decoder. Therefore, when decoding, the second decoder can refer to the positional changes of the first device in spacetime to decode the positional changes of the second device. For example, if the positions of the first and second devices in spacetime influence each other, the second decoder can output a more accurate positional change of the second device based on the positional changes of the first device in spacetime.

[0159] Optionally, the second decoder may include an interactive coding layer and a second query decoding layer. The interactive coding layer can be used to interact with the second feature and the implicit field of the first heatmap to obtain interactive features. The second query decoding layer is used to decode based on the interactive features and query representation to obtain the implicit field of the second heatmap. Therefore, when decoding the second device, an interactive coding layer can be set to interact with the positional changes of the first device and the features of the second device, thereby identifying the impact of the positional changes of the first device on the second device in space and time, and determining the positional changes of the second device at each query position in space and time based on the input query conditions. Especially for scenarios where the positional changes of the second device in space and time are affected by the first device, the positional changes of the second device in space and time can be predicted more accurately by exchanging the coding layers.

[0160] 205. Determine the travel path of the first device based on the implicit fields of the first and second thermal flow diagrams.

[0161] Then, a travel path can be planned for the first device based on the implicit fields of the first and second thermal flow maps.

[0162] In this embodiment, for scenarios involving one first device and one or more second devices, the spatial and temporal positional changes of the first device can be decoded first. Based on these changes and the characteristics of the second devices, the positional changes of the second devices can then be decoded, allowing for a more accurate prediction of the second devices' positional changes under the influence of the first device. Furthermore, based on these spatial and temporal positional changes of the second devices under the first device's influence, a more accurate driving path can be planned for the first device, thereby avoiding the presence of second devices in the environment and improving the first device's driving safety.

[0163] In one possible implementation, at least one first path for the first device can be determined based on the implicit field of a first heatmap; and at least one second path corresponding to each first path can be determined based on the implicit field of a second heatmap; subsequently, a travel path can be determined based on the at least one first path and the at least one second path. In this embodiment, one or more first paths for the first device can be determined first. Since the positional changes of the first device in space and time may affect the position of the second device, one or more corresponding second paths can be determined for each first path of the first device using the implicit field of the second heatmap. Thus, a travel path adapted to the scene can be planned for the first device based on the one or more second paths corresponding to each first path.

[0164] Optionally, when planning a first path for the first device, sampling can be performed from the implicit field of the first heatmap to obtain at least one first path. It is understood that the implicit field of the first heatmap can represent the probability of the first device's position change over a period of time. Based on this implicit field, locations with higher probabilities can be sampled to fit the possible travel paths of the first device, thereby obtaining one or more first paths with higher probabilities of occurrence.

[0165] Optionally, the method of sampling the first path from the first heat flow graph can include various methods, such as including but not limited to one or more of the following:

[0166] Samples are taken from the implicit field of the first thermal flow map at preset time intervals to obtain at least one first path. Usually, the at least one first path with the highest probability is selected.

[0167] Alternatively, based on at least one candidate path from the first heatmap, select at least one first path from the first heatmap;

[0168] Alternatively, local peak points can be searched from the first heat flow graph at multiple times, and multiple peak points can be selected based on the distinguishability between the multiple local branch points obtained, and at least one first path can be generated based on the multiple peak points.

[0169] In this embodiment, the first path can be obtained by sampling from the first heat map in multiple ways, so that the appropriate sampling method can be selected according to the actual application scenario, which can adapt to more scenarios.

[0170] Alternatively, before deploying the method provided in the embodiments of this application, the initial encoding network, the first decoder, and the second decoder may be trained, and the trained encoding network, the first decoder, and the second decoder may be deployed in the first device or other computing device.

[0171] For example, a training dataset can be obtained, which includes perception data and motion paths representing multiple third devices; the initial encoder network, the initial first decoder, and the initial second decoder can be self-supervised using the training dataset to obtain the trained encoder network, the first decoder, and the second decoder.

[0172] Therefore, in this embodiment, training data can be collected to perform self-supervised training on the initial encoding network, the first decoder, and the second decoder. That is, the motion path of the device included in the training data is used as the supervision value. No manual data labeling is required, which can achieve lower-cost neural network training. The neural network can also be updated in real time during device operation, making the output results of the finally deployed network more accurate.

[0173] The foregoing has described the method flow provided in the embodiments of this application. For ease of understanding, a more detailed description is provided below, in conjunction with specific application scenarios.

[0174] The following example demonstrates the deployment of the method provided in this application's embodiments into a vehicle's intelligent driving system. The processing flow of the vehicle's intelligent driving system is shown in Figure 3. Specifically, it can be divided into a perception module, a CHMF module, a prediction module, a decision-making module, a planning module, and a control module. The perception module senses the data collected by the vehicle's sensors and outputs environmental information. The CHMF module, prediction module, and decision-making module are the modules deployed using the method provided in this application's embodiments. The CHMF module can output implicit CHMF fields representing the paths of the vehicle and other vehicles. The prediction module can plan a driving path for the vehicle based on its own HMF and other vehicles' CHMF. The decision-making module can plan driving decisions for the vehicle. The planning module can plan specific behavioral decisions for the vehicle when driving according to the driving path. The control module can control the vehicle's driving based on the behavioral decisions output by the planning module.

[0175] Referring to Figure 4, which illustrates the architecture of a neural network according to an embodiment of this application, the aforementioned intelligent driving system structure is relevant. This neural network may include, but is not limited to, an encoding network 401, a first decoder 402, and a second decoder 403.

[0176] The encoding network 401 can be used to extract features from the input data and encode them, outputting encoded scene features, including features of the vehicle itself and features of other vehicles.

[0177] The first decoder 402 can be used to decode the input vehicle features and output the vehicle HMF implicit field.

[0178] The second decoder 403 can be used to combine the input vehicle's HMF implicit field to decode the features of other vehicles and output the other vehicle's CHMF implicit field.

[0179] Based on this neural network architecture, the deployment of the method provided in the embodiments of this application is described below. The overall method architecture provided in the embodiments of this application can be shown in Figure 5.

[0180] This mainly involves generating the vehicle's Hidden Mode Model (HMF) implicit field and the other vehicle's Hidden Mode Model (CHMF) implicit field using real-time road environment information. The top m predicted paths for the vehicle are generated using the vehicle's HMF implicit field, and the top n predicted paths for the other vehicle are generated by conditioning each predicted path using the other vehicle's CHMF implicit field. Based on the top m*n paths from both the vehicle and other vehicles, the joint probability calculation of their interaction modes is performed.

[0181] Referring to Figure 5, for ease of understanding, the method provided in this application embodiment can be divided into multiple stages, such as outputting scene features, outputting the vehicle's HMF implicit field, outputting the other vehicle's CHMF implicit field, predicting the paths of the vehicle and the other vehicle, and calculating the joint probability of the interaction modes. Each stage will be described below.

[0182] I. Output Scene Characteristics

[0183] The input to the encoding network can include various types of data collected by the sensor, or features extracted from the data collected by the sensor.

[0184] For example, images, point clouds, or maps collected by the vehicle can be used as input to an encoding network. The encoding network extracts features from the input data, identifies and encodes various targets in the scene based on these features, and outputs scene features representing the scene. This scene can include features of targets such as the vehicle itself and other vehicles. Of course, in addition to other vehicles, it can also include road structures or obstacles such as pedestrians. This embodiment of the application uses other vehicles in the scene as an example for illustrative purposes and does not limit the targets included in the environment.

[0185] II. Output of the vehicle's HMF implicit field

[0186] The scene features output from the aforementioned stage one are used as the input to the first decoder and output from the vehicle HMF implicit field, which is the aforementioned first heat flow graph implicit field.

[0187] For example, the process of generating the vehicle HMF implicit field can be shown in Figure 6. First, real-time road environment and other information are input and encoded into the vehicle scene latent layer features, i.e., the first feature or vehicle feature, through the encoding network Transformer Encoder. At the same time, the query point is input to the query decoding layer through position embedding to construct a queryable decoding network for generating the vehicle HMF implicit field.

[0188] Typically, the implicit HMF field of a vehicle is a continuous field, representing comprehensive information and allowing querying in an infinite space (x,y) and an infinite time t. A heatmap visualization corresponding to a specific time t is shown in Figure 7. The CHMF implicit field near the vehicle's ground truth position (0,0) at t=3s is visualized in Figure 7, which is the heatmap shown therein. Generally, in commonly used heatmaps, darker colors represent higher prediction probabilities for the corresponding points. In Figure 7, the solid line represents the vehicle's ground truth path for the next 5s. The darkest parts are concentrated near the vehicle's 3s ground truth position; the further away from the ground truth, the lighter the color and the lower the prediction probability.

[0189] Furthermore, a single CHMF can characterize the probability distribution of a vehicle's position at a future time t, i.e., P (x,y) Therefore, the probability distribution of a vehicle's position over a continuous time period t in the future, such as 5 seconds, can be represented as a Heatmap Flow, or HMF for short. When using CHMF later, the task of predicting the future of the vehicle can be decomposed into corresponding queries, which are iteratively input into the network to obtain the corresponding probability values. Then, through simple post-processing, the predicted path and behavioral modality of the vehicle can be obtained.

[0190] III. Outputting the CHMF implicit field of other vehicles

[0191] The input to the second decoder may include the vehicle's own HMF implicit field or the path collected from the vehicle's own HMF implicit field and other vehicle scene hidden layer features (i.e. the aforementioned second feature), and may also include query representations, and output the other vehicle's CHMF implicit field.

[0192] For example, the generation process of the implicit field of the other vehicle CHMF can be shown in Figure 8. In addition to a network structure similar to that of the self vehicle HMF generation, it adds conditional constraint input, namely, the predicted path of the self vehicle obtained by querying the implicit field of the self vehicle HMF. As shown in Figure 8, the self vehicle decision hidden layer features are output by the self vehicle decision encoding layer, and the self vehicle decision hidden layer features and the hidden layer features of the other vehicles are input into the interaction encoding layer. Accordingly, the obtained other vehicle CHMF is the other vehicle modal response under the input modal constraints of the self vehicle, which can be understood as a more effective representation of the modal interaction relationship between the self vehicle and the other vehicle.

[0193] Specifically, the generator network for the CHMF implicit field of other vehicles can be understood as a queryable implicit field decoding network containing conditional constraints. It generates the CHMF implicit field of other vehicles using the self-vehicle's predicted path (e.g., self-vehicle decision-making hidden layer features), real-time road environment information (e.g., including features of other vehicles), and the query volume (query points) decomposed from the self-vehicle query task (i.e., querying other vehicles' paths using the self-vehicle's path as a query condition). This field is then used for subsequent generation of other vehicle predicted paths and behavioral modality calculation.

[0194] The predicted path of the self-vehicle is input as a conditional constraint into the CHMF generation network of other vehicles, and encoded into the self-vehicle decision hidden layer feature by the self-vehicle decision encoding layer. This is then fused with the hidden layer features of other vehicles obtained through the encoding network to construct the self-vehicle-other-vehicle interaction encoding layer. The purpose of this interaction layer is to model the interaction between self-vehicle and other-vehicle modalities through the interaction between the high-dimensional hidden features of the self-vehicle and other vehicles; a typical structure is a multi-head multi-layer self-attention network. The query point is then fused with the interaction layer through positional encoding and input into the query decoding layer. A typical structure of the query decoding layer is a multilayer perceptron, which can decode the implicit field of the CHMF of other vehicles.

[0195] When using it, you simultaneously input the predicted path of the vehicle you want to predict and the query volume of the other vehicle prediction task. Iteratively query the CHMF implicit field of the other vehicle to obtain the corresponding probability value, and then obtain the predicted path and behavior mode of the other vehicle through simple post-processing.

[0196] IV. Predicting the paths of your own vehicle and other vehicles

[0197] After outputting the implicit HMF field of the vehicle and the implicit CHMF field of the other vehicle in stages two and three, the paths of the vehicle and the other vehicle can be predicted based on the implicit HMF field of the vehicle and the implicit CHMF field of the other vehicle.

[0198] In stages two and three, the process of outputting the implicit HMF field of the vehicle and the implicit CHMF field of other vehicles can be decomposed into iteratively querying the implicit HMF field of the vehicle, then constructing the predicted path of the vehicle (i.e., the first path mentioned above) based on the query results, and calculating the corresponding probability value. The processing method for predicting the path of other vehicles is similar, except that a certain predicted path of the vehicle is used as a condition constraint to query the implicit CHMF field of other vehicles, and then the predicted path of other vehicles (i.e., the second path mentioned above) is constructed based on the query results and the corresponding probability value is calculated.

[0199] If HMF is used as the output head network for both the vehicle and other vehicles, it is a continuous implicit field network that can infinitely query the probability P(x,y,t) of any position (x,y) at any time t. The planning and control representation information is comprehensive and the form is simple.

[0200] Specifically, it can generate the top m predicted paths for the vehicle itself, and use the CHMF implicit field of other vehicles to generate the top n predicted paths for other vehicles by taking each predicted path of the vehicle itself as a condition. That is, the paths of other vehicles contain m*n predicted paths.

[0201] Specifically, the process of sampling the first path from the implicit field of the vehicle's own HMF is similar to that of sampling the second path from the implicit field of the other vehicle's CHMF. The following description will take sampling the first path from the implicit field of the vehicle's own HMF as an example.

[0202] Specific sampling methods can include a variety of methods, as shown in Figure 9. For example, they can be divided into Goal point sampling, candidate path Anchor scoring and selection, or multi-mode peak path sampling.

[0203] For example, the specific sampling method for Goal point sampling may include: uniformly scattering points within the target object's spatial range at each time t to generate a sampling point query set Query(x,y,t), iteratively querying the HMF implicit field to obtain the probability Probability(x,y,t) of each sampling point, sorting them, selecting the highest probability point at each time t as the goal point, and connecting the goal points at each time t to obtain the prediction path, with the corresponding probability being the product of the probabilities of all goal points.

[0204] Besides uniform sampling, other methods can be customized, such as hierarchical focusing or other sampling methods. Taking hierarchical focusing sampling as an example, at each time t, sparser sampling points are first uniformly distributed to obtain the goal point. Then, denser sampling points are distributed in a smaller area near the goal point to obtain a new, more accurate goal point. This process of focusing layer by layer is used to obtain the final goal point. The number of focusing layers and the density of each layer can be customized, and usually need to be determined in conjunction with the inference latency.

[0205] For example, the sampling method for selecting candidate path anchors can include: first clustering N candidate paths (called anchors) using real driving routes, or clustering 2000 candidate paths using a clustering algorithm (such as K-Means or other clustering algorithms) as input anchors. Iteratively querying the HMF implicit field obtains the single-point probability of each path point on each candidate path; the probability of a single candidate path is the product of the probabilities of all path points on that path. traj After sorting, the path with the highest probability is selected as the final predicted path.

[0206] For example, the sampling method for multimode peak path sampling can include: searching for local probability peak points in the HMF implicit field at each time t, then obtaining multimode peak points with high discriminative power by clustering and merging similar peak points, and finally connecting the multimode peak points at each time t through multimode path tracking to generate a multimode path that conforms to kinematic laws. This discriminative power can be measured by the distance between probability peaks; for example, the greater the distance between probability peaks, the greater the discriminative power, and correspondingly, the smaller the distance between probability peaks, the smaller the discriminative power.

[0207] Therefore, the application implementation provides multiple sampling methods for path sampling, so that the appropriate sampling method can be selected according to actual needs to output the required predicted path and corresponding probability.

[0208] After obtaining the predicted paths and probabilities through iterative querying based on the HMF implicit field, the path sampling relationship between the vehicle and other vehicles can be shown in Figure 10. In the process of obtaining the predicted paths and probabilities from the HMF implicit field iterative query, firstly, the top m predicted paths and probabilities of the vehicle are output through the vehicle's HMF implicit field. Then, each predicted path of the vehicle is used as a query condition and input into the CHMF implicit field query decoding layer of other vehicles to obtain the corresponding top n predicted paths and probabilities. This is equivalent to opening up parallel spatiotemporal predictions of other vehicles under each vehicle mode change based on the vehicle's modal changes. The top m*n pairs of predicted paths and probabilities obtained in this way can be used as input for the next step of interactive modality calculation.

[0209] For example, taking the selection of vehicle paths based on candidate path anchor scores as an example, the top k predicted paths are sampled. As shown in Figure 11, firstly, ground truth paths are collected, that is, the vehicle driving paths collected in the real scene. The ground truth paths are then clustered; here, K-Means clustering is used as an example, but other clustering algorithms can also be used, outputting candidate path group anchors. Subsequently, the candidate path group anchors are input into the vehicle HMF decoder, outputting the probability score for each path. traj This can be represented as using the truth path as a priori to determine the probability of each possible path being applied in the current scenario. Then, the top k paths are scored. To further improve the diversity of the output paths, the Non-Maximum Suppression (NMS) algorithm can be introduced to select the top k paths. The score is updated by taking the negative score of the path that is modally similar to the top-1 path after each selection, setting it to a minimum to ensure it is not selected in the next iteration. In practical applications, modally similar to the top-1 path is defined as a path whose FDE difference with the top-1 path is less than a certain threshold. The NMS algorithm is repeatedly started to select the top k paths, outputting the top k paths and their top k scores.

[0210] The aforementioned scheme can balance path score and multi-modal coverage, outputting the top k multi-modal paths with the highest possible probability, providing more candidate modalities for subsequent interaction modality calculations, adapting to more vehicle condition scenarios, and exhibiting very strong generalization ability.

[0211] V. Calculation of Joint Probability of Interactive Modes

[0212] The calculation of the joint probability of the interaction mode between the self-vehicle and other-vehicles is specifically implemented by calculating the interaction game tags and probabilities between each pair of self-vehicle and other-vehicle paths, as well as combining and normalizing them. The calculation of the game tags and probabilities between each pair of paths is further divided into the calculation of vertical game tags and probabilities and the calculation of horizontal game tags and probabilities.

[0213] For example, there are three categories of vertical game labels, representing the corresponding behavioral modalities of the vehicle during interactions with other vehicles: GW (GrabWay), YD (Yield), and IG (Ignore). The key to the judgment is the order in which the vehicle and other vehicles pass through the interaction point. The specific judgment process can be shown in Figure 12.

[0214] First, obtain the top m paths of the vehicle and the probability of each path. Also, obtain the top n paths of other vehicles under each path of the vehicle and their corresponding probabilities. The total number of combinations of these other vehicle paths is the top m*n paths. Then, determine the intersection points between the m*n path pairs.

[0215] For each path pair, first calculate whether there is any interaction between them laterally. This can be done by calculating the closest point pairs in the paths between the vehicle and the other vehicle whose lateral distance is less than a certain threshold. As shown in Figure 13, when the lateral distance is too close, it indicates that there may be interaction between the two paths. Then, the path pair yielding relationship can be calculated according to the process shown in Figure 14.

[0216] If there is no interaction point between the two vehicles, it means that the other vehicle's path does not intrude into the owner's path, and there is no interactive game relationship between them. In this case, the owner ignores the other vehicle, and the game label is set to IG. If there is an interaction point between the two vehicles, it means that the other vehicle's path intrudes into the owner's path, and the yielding relationship between the two vehicles needs to be calculated. If the intrusion point is the end point of the other vehicle's path, it means that the other vehicle arrives at the interaction area at the last moment. In this case, the owner arrives at the interaction area first, so the owner preempts the other vehicle, and the label is GW. If the intrusion point is not the end point of the other vehicle's path, and the owner arrives at the conflict point before the other vehicle, then the owner preempts the other vehicle, and the game label is GW. Otherwise, the owner yields, and the game label is YD.

[0217] For each path pair, calculate the vertical game label and multiply the probabilities of the corresponding path pairs as the label probability of that path pair. Repeatedly calculate and normalize the vertical labels and probabilities of the top m*n path pairs (i.e., merge probabilities of the same label) to obtain the vertical three-class label probability of the vehicle relative to a specific other vehicle. Perform this calculation for each other vehicle to obtain the vertical three-class label probability of the vehicle relative to all other vehicles.

[0218] Horizontal game theory can be divided into LB (Left Bypass) and RB (Right Bypass). The key to judgment is the left or right position of the vehicle and the other vehicle at the interaction point. The specific judgment process can be shown in Figure 15.

[0219] First, obtain the top m paths of the vehicle and the probability of each path. Also, obtain the top n paths of other vehicles under each path of the vehicle and their corresponding probabilities. The total number of combinations of these other vehicle paths is the top m*n paths. Then, determine the intersection points between the m*n path pairs.

[0220] For each path pair, first calculate whether there is a longitudinal interaction between them. Specifically, this can include calculating time points at the same time t where the longitudinal distance between the paths of the self-vehicle and the other-vehicle is less than a certain threshold, as shown in Figure 16. Then, the left-right relationship of the path pair can be calculated according to the process in Figure 17. If such an interaction point pair exists, it indicates that the paths of the self-vehicle and the other-vehicle interact longitudinally. If the interaction point of the self-vehicle is to the left of the other-vehicle, the self-vehicle avoids to the left (labeled LB); otherwise, the self-vehicle avoids to the right (labeled RB). If no such interaction point pair exists, it indicates that the paths of the self-vehicle and the other-vehicle do not interact longitudinally. If the starting point of the self-vehicle is to the left of the starting point of the other-vehicle, the self-vehicle avoids to the left (labeled LB); otherwise, the self-vehicle avoids to the right (labeled RB).

[0221] For each path pair, calculate the lateral game label and multiply the probabilities of the corresponding path pairs as the label probability of that path pair. After iteratively calculating and normalizing the lateral labels and probabilities of the top m*n path pairs, the lateral binary label probability of the vehicle relative to a specific other vehicle can be obtained. Performing this calculation for each other vehicle yields the lateral binary label probability of the vehicle relative to all other vehicles.

[0222] Therefore, in this embodiment, HMF is used as the output head network for both the vehicle and other vehicles. As a continuous implicit field network, it can infinitely query the probability P(x,y,t) of any position (x,y) at any time t, resulting in comprehensive and concise planning and control representation information. Typically, HMF has a fast inference speed, making the design and control of output results flexible and not limited by speed constraints.

[0223] The conditional pattern-based interactive query network demonstrates stronger consistency in interactions between the vehicle and other vehicles. By inputting the vehicle's spatiotemporal decisions (such as path planning) as conditions into the prediction network of other vehicles, a more consistent prediction network for other vehicles is constructed based on the vehicle's results. Furthermore, by modeling the modal interactions between the hidden high-dimensional features of both the vehicle and other vehicles, the modal consistency between them is stronger, with other vehicle modalities containing information about the vehicle's planning modality, resulting in modal self-consistency in both lateral and longitudinal actions. In addition, the deployment of the queryable network reduces the inference latency for outputting predicted paths for other vehicles with conditions from the vehicle itself.

[0224] Furthermore, the end-to-end process from outputting the HMF implicit field to predicting the path and then to modal interaction exhibits very strong overall consistency. For example, both the predicted path and the interaction modality originate from the HMF implicit field; they are not independent products, sharing a common source and exhibiting continuity, resulting in strong consistency between path prediction and preemption relationship decisions. Moreover, the spatiotemporal continuity of the HMF implicit field allows for the output of multiple predicted paths, covering more modalities and providing more candidate modalities for subsequent interaction modality calculations. Additionally, when sampling paths from the HMF implicit field, the paths collected in the actual scenario have certain kinematic constraints, making the final interaction modality more consistent with kinematic constraints and resulting in more reasonable horizontal and vertical planning.

[0225] For example, the vehicle path determined by the vehicle's HMF implicit field can be shown in Figure 18, and the top n other vehicle paths corresponding to one of the vehicle paths determined by the other vehicle's CHMF implicit field can be shown in Figure 19. Clearly, the method provided in this application embodiment can provide more diverse predicted paths for both the vehicle and other vehicles. Correspondingly, the vehicle's predicted path and the other vehicle's predicted paths can be displayed simultaneously on the vehicle's display interface, allowing users to more intuitively understand changes in vehicle position in the environment, facilitating timely vehicle control.

[0226] Furthermore, the neural network provided in this application embodiment needs to be trained before deployment.

[0227] For example, the training process can be shown in Figure 20.

[0228] First, a training dataset is constructed. The data in this dataset can come from data collected during vehicle operation or data collected by users, etc., and is used to build self-supervised training data that requires no human intervention. The data can typically contain two main categories: one is time-series data such as the historical driving paths of the vehicle and other vehicles, and the other is topological data containing road structure information, such as lane lines and road obstacles.

[0229] Combining the aforementioned methodology, self-supervised data is input into an encoding network and encoded as scene features, which are then separated into self-vehicle features and other-vehicle features. To enhance the consistency of self-vehicle and other-vehicle interactions, the implicit field generation of the HMF for other vehicles incorporates the conditional constraints of the self-vehicle's path. The resulting CHMF for other vehicles is an HMF constrained by the self-vehicle's behavior, naturally possessing interaction consistency. Therefore, the features for other vehicles still require the input of the self-vehicle's ground truth path as a condition for feature updates.

[0230] Subsequently, the corresponding implicit HMF fields of the vehicle itself and the implicit CHMF fields of the other vehicle are generated using the vehicle's HMF Decoder and the other vehicle's HMF Decoder, respectively.

[0231] A continuous distribution about (x, y, t) is learned and generated using an implicit field (HMF). For each time t, the ground truth probability for each specific location (x, y) is constructed using a Gaussian distribution, with the center of the Gaussian distribution representing the two-dimensional ground truth coordinates of the target object (e.g., your own vehicle or another vehicle). The variance acts as a hyperparameter, controlling the width of the Gaussian distribution. This variance determines the level of detail in the heatmap learning; for example, samples within a 3σ interval of the ground truth location at each time are considered positive, while samples outside this interval are considered negative. The magnitude of the variance determines how far beyond the ground truth location is considered a positive sample. The smaller the variance, the more concentrated the dark areas of the learned heatmap, and the more difficult the learning task. The distributions at different times t are independent and unconstrained.

[0232] During training, a spatiotemporal supervision point set P(x,y,t) is selected for learning through hierarchical sampling, with the main principle being that the sampling resolution increases at different levels. Specifically, location points can be selected from ranges of different sizes, and the density of the selected locations may vary. The sampling gradually clusters towards the ground truth location, with higher density sampling closer to the ground truth. This is because supervision points near the ground truth are more correlated with the inference result and are more difficult to learn. For example, a hierarchical sampling example with the ground truth at (0,0) can be shown in Figure 21.

[0233] The optimizer used during training can employ various loss functions with different characteristics, such as L1 Loss, L2 Loss, cross-entropy, and Focal Loss. The optimization objective is to reduce the distance between the network predictions at the supervised points and the ground truth distribution. For example, L1 Loss can be used, which optimizes the L1 distance between the predicted probabilities of the supervised points and the ground truth probabilities generated by the Gaussian distribution. By iteratively optimizing the loss function until convergence, the converged implicit HMF fields of the self-vehicle and CHMF fields of the other-vehicle can be output.

[0234] The foregoing has described the method flow provided in the embodiments of this application. The following describes the structure of the apparatus for executing the foregoing method flow.

[0235] Referring to Figure 22, a schematic diagram of a path planning device provided in an embodiment of this application is shown. This path planning device can be used in the steps of the aforementioned method process.

[0236] The path planning device includes:

[0237] The acquisition module 2201 is used to acquire sensing data, which includes data collected by at least one sensor in the first device;

[0238] Encoding module 2202 is used to input sensing data into encoding network and output scene features. The scene features include a first feature and a second feature. The first feature is the feature of the first device, and the second feature is the feature of at least one second device in the scene.

[0239] The first decoding module 2203 is used to input the first feature into the first decoder and output the first heat flow map implicit field. The first heat flow map implicit field is used to represent the probability of the first device appearing at each location within a preset time period.

[0240] The second decoding module 2204 is used to input the second feature and the first heatmap implicit field to the second decoder and output the second heatmap implicit field, which represents the probability that at least one second device appears at each location within a preset time period.

[0241] Planning module 2205 is used to determine the travel path of the first device based on the implicit field of the first thermal flow diagram and the implicit field of the second thermal flow diagram.

[0242] In one possible implementation, the aforementioned first decoding module 2203 is specifically used to: acquire a query representation, the query representation including at least one corresponding representation representing location or time, the location being the location of the first device corresponding to the sensing data; input the first feature and the representation feature to the first decoder, and output the first heat flow map implicit field.

[0243] In one possible implementation, the aforementioned first decoder includes a first query decoding layer; the first query decoding layer is used to decode based on the first feature and the query representation to obtain the first heat flow graph implicit field.

[0244] In one possible implementation, the aforementioned second decoding module 2204 is specifically used to: input the first heat flow graph implicit field and the second feature to the second decoder, and output the second heat flow graph implicit field.

[0245] In one possible implementation, the aforementioned second decoder includes an interactive coding layer and a second query decoding layer;

[0246] The interactive coding layer is used to interact the second feature with the implicit field of the first heatmap to obtain the interactive feature;

[0247] The second query decoding layer is used to decode based on interaction features and query representations to obtain the implicit field of the second heat flow graph.

[0248] In one possible implementation, the aforementioned planning module 2205 is specifically used to: determine at least one first path for the first device based on the implicit field of the first heat flow map; determine at least one second path corresponding to each first path based on the implicit field of the second heat flow map; and determine a travel path based on the at least one first path and the at least one second path.

[0249] In one possible implementation, the aforementioned planning module 2205 is specifically used to: obtain the intersection point between each of the first paths in at least one first path and at least one second path corresponding to each first path; determine the interaction information between the first device and at least one second device based on the intersection point, the interaction information including the interaction between the first device and at least one second device when the first device travels along at least one first path; and determine the travel path based on the interaction information.

[0250] In one possible implementation, the aforementioned planning module 2205 is specifically used to sample from the implicit field of the first thermal flow graph to obtain at least one first path.

[0251] In one possible implementation, the aforementioned planning module 2205 is specifically used to: sample from the implicit field of the first heat flow map at preset time intervals to obtain at least one first path; or, obtain at least one candidate path from the first heat flow map and determine at least one first path from the at least one candidate path based on the prior path; or, search for local peak points from the first heat flow map at multiple times, select multiple peak points based on the distinguishability between the multiple local branch points obtained, and generate at least one first path based on the multiple peak points.

[0252] In one possible implementation, the aforementioned apparatus further includes a training module 2206 for acquiring a training dataset, the training dataset including perception data and motion paths of multiple third devices; and using the training dataset to perform self-supervised training on an initial encoding network, an initial first decoder, and an initial second decoder to obtain a trained encoding network, a first decoder, and a second decoder.

[0253] Figure 23 shows a schematic diagram of the hardware structure of a computing device 230 provided in an embodiment of this application. This computing device 230 can be used to implement the steps of the methods shown in Figures 2 to 21, and may specifically include, for example, a vehicle-mounted device or a cloud-based device.

[0254] The computing device 230 shown in Figure 23 may include a processor 2301, a memory 2302, a communication interface 2303, and a bus 2304. The processor 2301, the memory 2302, and the communication interface 2303 can be connected to each other via the bus 2304.

[0255] The processor 2301 is the control center of the computing device 230. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0256] As an example, processor 2301 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 23.

[0257] The memory 2302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0258] In one possible implementation, the memory 2302 may exist independently of the processor 2301. The memory 2302 can be connected to the processor 2301 via a bus 2304 and is used to store data, instructions, or program code. When the processor 2301 calls and executes the instructions or program code stored in the memory 2302, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 2 to 21.

[0259] In another possible implementation, the memory 2302 can also be integrated with the processor 2301.

[0260] The communication interface 2303 is used for connecting the computing device 230 to other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 2303 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0261] Bus 2304 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 23, but this does not indicate that there is only one bus or one type of bus.

[0262] It should be noted that the structure shown in FIG23 does not constitute a limitation on the computing device 230. In addition to the components shown in FIG23, the computing device 230 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0263] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a data quantization device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0264] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0265] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 2 to 21 above.

[0266] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0267] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0268] The apparatus provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 2 to 21. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the computing device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0269] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0270] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0271] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0272] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0273] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0274] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A path planning method, characterized in that, include: Acquire sensing data, the sensing data including data collected by at least one sensor in the first device; The perceived data is input into the encoding network, and scene features are output. The scene features include a first feature and a second feature. The first feature is the feature of the first device, and the second feature is the feature of at least one second device in the scene. The first feature is input into the first decoder, and the first heat flow map implicit field is output. The first heat flow map implicit field is used to represent the probability of the first device appearing at each location within a preset time period. The second feature and the first heatmap implicit field are input to the second decoder to output the second heatmap implicit field, which represents the probability of the at least one second device appearing at each location within a preset time period. The travel path of the first device is determined based on the implicit field of the first thermal flow map and the implicit field of the second thermal flow map.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a query representation, the query representation including at least one corresponding representation of location or time, wherein the location is the location of the first device corresponding to the sensing data; The step of inputting the first feature into the first decoder and outputting the first heat flow map implicit field includes: The first feature and the characterization feature are input into the first decoder to output the first thermal flow graph implicit field.

3. The method according to claim 2, characterized in that, The first decoder includes a first query decoding layer; The first query decoding layer is used to decode based on the first feature and the query representation to obtain the first heat flow graph implicit field.

4. The method according to claim 2 or 3, characterized in that, The step of inputting the second feature and the first heatmap implicit field into the second decoder and outputting the second heatmap implicit field includes: The first heatmap implicit field and the second feature are input into the second decoder to output the second heatmap implicit field.

5. The method according to claim 4, characterized in that, The second decoder includes an interactive coding layer and a second query decoding layer; The interactive coding layer is used to interact the second feature with the implicit field of the first heatmap to obtain interactive features; The second query decoding layer is used to decode based on the interaction features and the query representation to obtain the second heat flow graph implicit field.

6. The method according to any one of claims 1-5, characterized in that, Determining the travel path of the first device based on the implicit field of the first thermal flow map and the implicit field of the second thermal flow map includes: At least one first path for the first device is determined based on the implicit field of the first thermal flow map; Based on the implicit field of the second heat flow map, at least one second path is determined corresponding to each first path; The driving route is determined based on the at least one first path and the at least one second path.

7. The method according to claim 6, characterized in that, Determining the driving route based on the at least one first path and the at least one second path includes: Obtain the intersection point between each of the at least one first path and at least one corresponding second path; Based on the intersection point, the interaction information between the first device and the at least one second device is determined, and the interaction information includes the interaction between the first device and the at least one second device when the first device travels along the at least one first path; The driving route is determined based on the interactive information.

8. The method according to claim 7, characterized in that, Determining at least one first path for the first device based on the implicit field of the first thermal flow map includes: The first path is obtained by sampling from the implicit field of the first thermal flow graph.

9. The method according to claim 8, characterized in that, The step of sampling from the implicit field of the first thermal flow map to obtain the at least one first path includes: Sampling is performed from the implicit field of the first thermal flow map at preset time intervals to obtain the at least one first path; Alternatively, at least one candidate path can be obtained from the first heat flow graph; The at least one first path is determined from the at least one candidate path based on the prior path; Alternatively, local peak points can be searched from the first heat flow graph at multiple times, and multiple peak points can be selected based on the distinguishability between the multiple local branch points, and the at least one first path can be generated based on the multiple peak points.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Obtain a training dataset, which includes perception data and motion paths representing multiple third-party devices; The initial encoder network, the initial first decoder, and the initial second decoder are trained using the training dataset to obtain the trained encoder network, the first decoder, and the second decoder.

11. A path planning device, characterized in that, include: The acquisition module is used to acquire sensing data, which includes data collected by at least one sensor in the first device; An encoding module is used to input the perceived data into an encoding network and output scene features. The scene features include a first feature and a second feature. The first feature is a feature of the first device, and the second feature is a feature of at least one second device in the scene. The first decoding module is used to input the first feature into the first decoder and output the first heat flow map implicit field, which is used to represent the probability of the first device appearing at each location within a preset time period. The second decoding module is used to input the second feature and the first heatmap implicit field into the second decoder and output the second heatmap implicit field, wherein the second heatmap implicit field represents the probability of the at least one second device appearing at each location within a preset time period; The planning module is used to determine the travel path of the first device based on the implicit field of the first thermal flow map and the implicit field of the second thermal flow map.

12. The apparatus according to claim 11, characterized in that, The first decoding module is specifically used for: Obtain a query representation, the query representation including at least one corresponding representation of location or time, wherein the location is the location of the first device corresponding to the sensing data; The first feature and the characterization feature are input into the first decoder to output the first thermal flow graph implicit field.

13. The apparatus according to claim 12, characterized in that, The first decoder includes a first query decoding layer; The first query decoding layer is used to decode based on the first feature and the query representation to obtain the first heat flow graph implicit field.

14. The apparatus according to claim 12 or 13, characterized in that, The second decoding module is specifically used for: The first heatmap implicit field and the second feature are input into the second decoder to output the second heatmap implicit field.

15. The apparatus according to claim 14, characterized in that, The second decoder includes an interactive coding layer and a second query decoding layer; The interactive coding layer is used to interact the second feature with the implicit field of the first heatmap to obtain interactive features; The second query decoding layer is used to decode based on the interaction features and the query representation to obtain the second heat flow graph implicit field.

16. The apparatus according to any one of claims 11-15, characterized in that, The planning module is specifically used for: At least one first path for the first device is determined based on the implicit field of the first thermal flow map; Based on the implicit field of the second heat flow map, at least one second path is determined corresponding to each first path; The driving route is determined based on the at least one first path and the at least one second path.

17. The apparatus according to claim 16, characterized in that, The planning module is specifically used for: Obtain the intersection point between each of the at least one first path and at least one corresponding second path; Based on the intersection point, the interaction information between the first device and the at least one second device is determined, and the interaction information includes the interaction between the first device and the at least one second device when the first device travels along the at least one first path; The driving route is determined based on the interactive information.

18. The apparatus according to claim 17, characterized in that, The planning module is specifically used for: The first path is obtained by sampling from the implicit field of the first thermal flow graph.

19. The apparatus according to claim 18, characterized in that, The planning module is specifically used for: Sampling is performed from the implicit field of the first thermal flow map at preset time intervals to obtain the at least one first path; Alternatively, at least one candidate path can be obtained from the first heat flow graph; The at least one first path is determined from the at least one candidate path based on the prior path; Alternatively, local peak points can be searched from the first heat flow graph at multiple times, and multiple peak points can be selected based on the distinguishability between the multiple local branch points, and the at least one first path can be generated based on the multiple peak points.

20. The apparatus according to any one of claims 11-19, characterized in that, The device further includes a training module for: Obtain a training dataset, which includes perception data and motion paths representing multiple third-party devices; The initial encoder network, the initial first decoder, and the initial second decoder are trained using the training dataset to obtain the trained encoder network, the first decoder, and the second decoder.

21. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the management device to perform the operational steps of the method as described in any one of claims 1 to 10.

22. A vehicle, characterized in that, The vehicle includes: at least one sensor, a processor, and a memory; The processor is configured to execute instructions stored in the memory based on data acquired by the at least one sensor, so that the management device performs the operation steps of the method as described in any one of claims 1 to 10.

23. A computer program product containing instructions, characterized in that, When the instruction is executed by the device, the device performs the operational steps of the method as described in any one of claims 1 to 10.

24. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by the device, cause the device to perform the operational steps of the method as described in any one of claims 1 to 10.

Images