Method and apparatus for encoding targets input automatic driving model

Abstract

The invention relates to the field of automatic driving, in particular to a method and device for encoding perceptual information about a target input into an automatic driving model, and a computer readable storage medium and a computer program product containing computer programs / instructions for implementing the method. According to the method, the number of the data frames to be processed is dynamically set according to the first data frame, the data frames are processed according to the time sequence from near to far, and the coding efficiency and the target expression ability are improved under the constraint condition that the total number of tokens is fixed.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving, and more specifically to methods and apparatus for encoding perception information about a target input to an autonomous driving model, computer-readable storage media and computer program products comprising computer programs / instructions for implementing the above methods, and vehicles comprising the above apparatus. Background Technology

[0002] Currently, in end-to-end autonomous driving models, the perception module generates perception information for various targets (including static and dynamic targets) based on sensor data (such as images and point cloud data). This perception information describes the target's position and size in physical space. Before inputting this information into the model, it needs to be encoded to form a fixed number of tokens. Dynamic targets typically refer to targets whose position or size can change, such as pedestrians, non-motorized vehicles, and motorized vehicles. In many cases, the number of dynamic targets included in the sensor data collected at different times varies, which increases the complexity of target encoding. Summary of the Invention

[0003] One embodiment of this disclosure relates to a method for encoding perceived information about a target input to an autonomous driving model, wherein the target includes dynamic targets and static targets, and the input of the autonomous driving model has a fixed number of tokens, the method comprising: The number of multiple data frames is determined based on the set number of tokens used to encode dynamic targets and the number of dynamic targets to be encoded in the first data frame, wherein the multiple data frames include the first data frame and a second data frame preceding the first data frame; and The encoding process of the dynamic targets of the multiple data frames is performed sequentially in order of time from most recent to furthest.

[0004] Optionally, in the above method, the types of dynamic targets include motor vehicles, non-motor vehicles, and pedestrians, and the types of static targets include lane lines, road edges, guardrails, roadblocks, and static obstacles.

[0005] Optionally, the above method further includes: The set quantity is determined based on the fixed quantity and the number of static targets in the first data frame.

[0006] Optionally, in the above method, the first data frame and the second data frame are video frames acquired using an image sensor, or generated by fusing sensing data from multiple sensors, wherein the multiple sensors are any combination of the following sensors: image sensor, lidar, millimeter-wave radar, and ultrasonic radar.

[0007] Optionally, in the above method, the dynamic target includes moving targets and stationary targets, and the number of the plurality of data frames is determined as follows: If condition A is met: Then the number of the plurality of data frames is determined to be the maximum value; If condition B is met: , The number of the plurality of data frames is then determined as follows: ; If neither condition A nor condition B is met, then the number of the plurality of data frames is determined to be the minimum value. Wherein, static_count is the number of stationary targets to be encoded in the first data frame, dynamic_count is the number of moving targets to be encoded in the first data frame, n_state is the set number, h is the number of multiple data frames, and max_h and min_h are the maximum and minimum values ​​of the number of multiple data frames, respectively.

[0008] Optionally, in the above method, sequentially performing the encoding processing of the dynamic targets of the plurality of data frames includes processing the first data frame in the following manner: If condition A or B is met, then all dynamic targets that need to be encoded in the first data frame are encoded; otherwise, dynamic targets that need to be encoded that are closer to the vehicle are encoded first.

[0009] Optionally, in the above method, sequentially performing the encoding processing of the dynamic targets of the plurality of data frames includes processing the second data frame in the following manner: If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in the previously processed data frame and is a moving target, then the dynamic target is encoded. If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in a previously processed data frame and is a stationary target, then the dynamic target is not encoded; and If the dynamic target that needs to be encoded in the second data frame being processed has not appeared in the previous data frames, then the dynamic target that is closer to the vehicle is encoded first.

[0010] Optionally, in the above method, the dynamic target to be encoded is determined based on the dynamic target's direction of motion or its position relative to the vehicle.

[0011] Another embodiment of this disclosure relates to an apparatus for encoding perceived information about a target input to an autonomous driving model, wherein the target includes dynamic and static targets, the input of the autonomous driving model has a fixed number of tokens, and the computing device includes: At least one processor; At least one memory; and A computer program / instruction stored in the memory, when executed on the processor, causes the following operations: The number of multiple data frames is determined based on a set number of tokens that can be allocated to dynamic target encoding and the number of dynamic targets to be encoded in the first data frame, wherein the multiple data frames include the first data frame and a second data frame preceding the first data frame; and The encoding process of the dynamic targets of the multiple data frames is performed sequentially in order of time from most recent to furthest.

[0012] Optionally, in the above-described device, the types of dynamic targets include motor vehicles, non-motor vehicles, and pedestrians, and the types of static targets include lane lines, road edges, guardrails, roadblocks, and static obstacles.

[0013] Optionally, in the above-described apparatus, the execution of the computer program / instructions on the processor also results in the following operations: The set quantity is determined based on the fixed quantity and the number of static targets in the first data frame.

[0014] Optionally, in the above-described apparatus, the first data frame and the second data frame are video frames acquired using an image sensor, or generated by fusing sensing data from multiple sensors, wherein the multiple sensors are any combination of the following sensors: image sensor, lidar, millimeter-wave radar, and ultrasonic radar.

[0015] Optionally, in the above-described apparatus, the dynamic target includes moving targets and stationary targets, and the number of the plurality of data frames is determined as follows: If condition A is met: Then the number of the plurality of data frames is determined to be the maximum value; If condition B is met: , The number of the plurality of data frames is then determined as follows: ; If neither condition A nor condition B is met, then the number of the plurality of data frames is determined to be the minimum value. Wherein, static_count is the number of stationary targets to be encoded in the first data frame, dynamic_count is the number of moving targets to be encoded in the first data frame, n_state is the set number, h is the number of multiple data frames, and max_h and min_h are the maximum and minimum values ​​of the number of multiple data frames, respectively.

[0016] Optionally, in the above apparatus, sequentially performing the encoding process of the dynamic targets of the plurality of data frames includes processing the first data frame in the following manner: If condition A or B is met, then all dynamic targets that need to be encoded in the first data frame are encoded; otherwise, dynamic targets that need to be encoded that are closer to the vehicle are encoded first.

[0017] Optionally, in the above apparatus, sequentially performing the encoding process of the dynamic target of the plurality of data frames includes processing the second data frame in the following manner: If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in the previously processed data frame and is a moving target, then the dynamic target is encoded. If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in a previously processed data frame and is a stationary target, then the dynamic target is not encoded; and If the dynamic target that needs to be encoded in the second data frame being processed has not appeared in the previous data frames, then the dynamic target that is closer to the vehicle is encoded first.

[0018] Optionally, in the above-described apparatus, the dynamic target to be encoded is determined based on the dynamic target's direction of motion or its position relative to the vehicle.

[0019] Another embodiment of this disclosure relates to a non-transitory computer-readable storage medium having a computer program / instructions stored thereon that, when executed by a processor, implements the steps of the method described above.

[0020] Another embodiment of this disclosure relates to a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0021] Another embodiment of this disclosure relates to a vehicle that includes the device described above.

[0022] In one or more embodiments of this disclosure, by first determining a reasonable number of processing units based on the most recent data frame, then processing the data frames sequentially from nearest to farthest, and simultaneously prioritizing the encoding of high-importance targets within each data frame, both encoding efficiency and target representation capability can be improved under the constraint of a fixed number of tokens. Attached Figure Description

[0023] The above and / or other aspects and advantages of this disclosure will become clearer and more readily understood from the following description in conjunction with the accompanying drawings, in which the same or similar elements are denoted by the same reference numerals.

[0024] Figure 1 This is a diagram illustrating the sequence of tokens input to the autonomous driving model each time.

[0025] Figure 2 This is a schematic diagram of the token sequence for each input to the autonomous driving model, which is generated using methods from one or more embodiments of this disclosure.

[0026] Figure 3 This is a flowchart of a method for encoding perception information about a target into an input autonomous driving model, according to an embodiment of the present disclosure.

[0027] Figure 4 This is a schematic block diagram of a computing device. Detailed Implementation

[0028] The present disclosure will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments thereof. However, the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. The foregoing embodiments are intended to make the disclosure herein complete and thorough, so as to more fully convey the scope of protection of the present disclosure to those skilled in the art.

[0029] In this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples.

[0030] In this disclosure, a "token" refers to the smallest unit of information formed by encoding perceived information used in autonomous driving, and is the basic input unit of the autonomous driving model. For example, each token corresponds to a set of structured data in numerical / vector form, which is used to represent all or part of the core features of a target.

[0031] In this disclosure, "target" refers to various objects perceived by an autonomous vehicle in its driving environment that can influence the vehicle's decision-making, planning, or control. These include, for example, static targets and dynamic targets. Static targets are generally defined as targets whose attributes such as position, shape, and size do not change substantially over time, such as, but not limited to, road infrastructure and fixed objects such as lane lines, road edges, guardrails, roadblocks, and static obstacles. Dynamic targets are generally defined as targets whose position, attitude, size, and motion state can change over time, such as, but not limited to, mobile traffic participants such as motor vehicles, non-motor vehicles, and pedestrians. It should be noted that the above definitions are merely exemplary; in practical applications, static and dynamic targets can be flexibly defined based on different scenarios.

[0032] In this disclosure, "data frame" refers to a time-series data unit used to describe the state of the environment surrounding an autonomous vehicle. It can be a video frame acquired by an image sensor, or a sensing data unit generated by data fusion of multiple sensors, including image sensors, lidar, millimeter-wave radar, and ultrasonic radar.

[0033] In this disclosure, "autonomous vehicle" refers to a vehicle equipped with perception sensors and an autonomous driving model, which is the main body that performs functions such as environmental perception, autonomous driving decision-making, path planning, and vehicle control.

[0034] The following describes a typical workflow for encoding target perception information input to an autonomous driving model. The perception information to be encoded may include the target's position, size, and tracking identifier (track_id), and is encoded as a fixed number of tokens input to the autonomous driving model. In the described workflow, the target contained in a fixed number of data frames is encoded each time to generate the input to the autonomous driving model. For example, assume that h data frames are processed each time (denoted as F1, F2...F... in chronological order). h (The same applies below). The number of tokens input to the autonomous driving model is n_total, the number of tokens used to encode static targets is n_env, and the number of tokens used to encode dynamic targets is n_state. Therefore, the above parameters regarding tokens satisfy the following relationship: (1).

[0035] Figure 1 This is a diagram illustrating the sequence of tokens (of size n_total) input to the autonomous driving model each time. For example... Figure 1 As shown, the token sequence contains n_env tokens (represented by hollow rectangles) for encoding static targets and n_state tokens (represented by filled rectangles) for encoding dynamic targets. When starting to generate model input, it can be based on the data from the h-th data frame F. h (It is in data frames F1, F2...F) h The perceptual information generated from the latest data frame (in time) determines the number of static targets, thereby determining the number of tokens n_env and n_state used to encode static and dynamic targets. On the other hand, in a typical encoding process, the n_state tokens allocated to dynamic targets are evenly divided into h groups, and each group of tokens is used in each data frame F1, F2…F… h The encoding of dynamic targets within. This leads to the formation of... Figure 1 The token sequence shown is used as an input to the autonomous driving model.

[0036] The above methods have shortcomings in terms of encoding efficiency and target representation capabilities. Specifically, on the one hand, when there are many dynamic targets in a data frame, they may not be fully encoded due to insufficient allocated tokens; on the other hand, when there are few dynamic targets in a data frame, it may lead to a waste of token resources.

[0037] For those skilled in the art, the intuitive solution is twofold: firstly, to keep the number of data frames h to be processed fixed to simplify the processing logic; and secondly, to optimize the value of h based on the expected range of change of the dynamic target in a specific application scenario, so that the token resources allocated to each data frame can match the encoding requirements of the dynamic target. However, this solution has limited versatility and relies heavily on a deep understanding of the specific application scenario.

[0038] In some embodiments of this disclosure, the number of data frames h to be processed is no longer a fixed value. That is, although each input of the autonomous driving model has a fixed number of tokens, the number of data frames corresponding to it is a dynamically changing value. In one specific implementation, dynamic targets are classified into moving targets currently in motion and stationary targets currently in a stationary state, and based on the first processed data frame (which can be denoted as data frame F)... h However, unlike Figure 1 The scenario shown is that, at the start of processing the first data frame F hPreviously, the number h was to be determined. The target included in this determination is used to determine the number h of data frames used as input to generate the current autonomous driving model. The specific method for determining the number h is as follows: If condition A, as expressed in equation (2a), is satisfied, it indicates that the token resources for encoding dynamic targets are sufficient, and therefore the number of data frames h is determined to be the maximum value max_h: (2a) If condition B, as expressed in equation (2b), is satisfied, it indicates that there are sufficient token resources for encoding dynamic targets; therefore, the number of data frames h is determined to be... To balance coding efficiency and target representation capability: (2b) If neither of the above conditions A nor B is met, it indicates a shortage of token resources for encoding dynamic targets. Therefore, the number of data frames h is determined to be the minimum value min_h to encode as many dynamic targets as possible.

[0039] In equations (2a) and (2b) above, static_count is the first data frame F processed. h The number of stationary targets that need to be encoded in the data, where dynamic_count is the first data frame F processed. h The number of moving targets that need to be encoded, symbol " " indicates rounding down, n_state is the number of tokens for dynamic target encoding allocated to h data frames, which can be determined according to equation (1), and max_h and min_h are the maximum and minimum values ​​that the number of data frames h to be processed can take, respectively.

[0040] The maximum value max_h and minimum value min_h can be set according to the actual application scenario. For example, in a scenario where the expected number of moving targets is small, the maximum value max_h can be set larger or the minimum value min_h can be set smaller. Conversely, in a scenario where the expected number of moving targets is large, the maximum value max_h can be set smaller or the minimum value min_h can be set larger.

[0041] It should be noted that for stationary targets within dynamic targets, since their perceived information (e.g., position or size) is expected to remain largely unchanged over h data frames, only one consumption of token resources is considered in equations (2a) and (2b) (i.e., only one encoding is performed for stationary targets). On the other hand, for moving targets within dynamic targets, considering that their perceived information (e.g., position or size) is likely to change over h data frames, multiple consumptions of token resources will be considered in equations (2a) and (2b) (i.e., h encodings are expected for moving targets).

[0042] It should also be noted that the necessity of encoding can be determined based on actual application requirements. In one specific implementation, various criteria can be used to determine whether a dynamic target is one that needs to be encoded. For example, a dynamic target that is moving away from the vehicle or located in a specific type of area (e.g., outside the road edge) is generally considered a target with low tracking value and therefore may not need to be encoded (i.e., no input to the autonomous driving model is required). In another specific implementation, all dynamic targets in the data frame can be identified as targets that need to be encoded.

[0043] Furthermore, in some other embodiments of this disclosure, the tokens used for dynamic targets are no longer evenly distributed across each data frame, but are dynamically adjusted based on the number of dynamic targets contained in the data frame. Specifically, the first processed or the latest (most recent) data frame F is used as the basis for allocation. h For example, when condition A or B is met, all dynamic targets requiring encoding within the data frame can be encoded; conversely, when neither condition A nor B is met, only important targets among all dynamic targets requiring encoding within the data frame can be encoded. In some specific implementations, dynamic targets closer to the vehicle can be encoded first, based on their distance from the vehicle, until token resources (e.g., n_state tokens in the example above) are exhausted. Another data frame F... i For example, (i=1.…..h-1), for dynamic targets that need to be encoded that have appeared in previously processed data frames, if they are moving targets, then the dynamic targets are encoded; if they are stationary targets, then no encoding is performed. For dynamic targets that need to be encoded that appear newly in the data frame, the high-importance targets can be encoded (for example, priority is given to encoding dynamic targets that are closer to the vehicle).

[0044] As new dynamic targets that need to be encoded emerge, the token resources available for encoding these dynamic targets will be continuously consumed. In some other implementations, the parameter n_left_state is used to indicate the latest available encoding token resources, and this parameter can be dynamically updated after each data frame is processed.

[0045] For example, when the first data frame F h After processing is complete, n_left_state can be updated to: (3a) (When condition A or B is satisfied) (3b) (When neither condition A nor B is satisfied) Here, n_left_state(h) is the data frame F. h The remaining token resources that can be used for dynamic targets after the target encoding process is completed.

[0046] When other data frames F i After each of the (i=1.….h-1) operations is completed, the update method for n_left_state is as follows: (4) Here, n_left_state(i) is the currently processed data frame F. i The remaining token resources available for dynamic targets after target encoding processing is completed, n_left_state(i-1) represents the remaining token resources available for dynamic targets after target encoding processing of the previous processed data frame, and static_count(i) represents the current processed data frame F. i The number of stationary targets encoded in the data frame, where dynamic_count(i) is the number of data frames currently being processed. i The number of moving targets encoded in the data frame F, where h' is the number of data frames currently being processed. i The number of data frames to be processed next.

[0047] Furthermore, in some other embodiments of this disclosure, after determining the number h of data frames to be processed, they will be processed in chronological order from most recent to furthest (i.e., according to F...). h F h-1...F2, F1, etc., are processed sequentially for each data frame. The processing method for each data frame has been described above and will not be repeated here. It should be noted that although in the field of autonomous driving, the method of performing target encoding in chronological order from history to the present is more intuitive and easier to understand in terms of technical concept, and conforms to the conventional technical understanding of those skilled in the art, the inventors of this disclosure have found after research that, compared with this conventional method, processing data frames in order from recent to distant can significantly improve the smoothness of changes in the number of data frames h, and improve the feature learning effect of the autonomous driving model.

[0048] Figure 2 This is a schematic diagram of the token sequence for each input to the autonomous driving model, which is generated using methods from one or more embodiments of this disclosure. Figure 2 The token sequence shown also contains n_env tokens (represented by hollow rectangles) for encoding static targets and n_state tokens (represented by filled rectangles) for encoding dynamic targets, but it differs from... Figure 1 In the scenario shown, the number of data frames h is no longer a fixed value, but rather a value that depends on the h-th data frame F. h (It is in data frames F1, F2...F) h The variable that is processed latest in time and first. Additionally, n_state tokens are used to encode dynamic targets in data frames F1, F2...F... h The token resources are no longer evenly distributed, but are dynamically adjusted based on the previously processed data frames.

[0049] It should be noted that the combined synergistic effect of techniques such as determining the number of data frames to be processed based on the most recent data frame, processing data frames in chronological order from nearest to furthest, and prioritizing the encoding of high-importance targets is unattainable by any single technique. Specifically, using the most recent data frame as a benchmark to determine the number of data frames to be processed ensures that the determined number matches the actual number and distribution of targets at the current moment, avoiding imbalances in encoding resource allocation caused by too many or too few targets in previous data frames. This provides a reliable and stable temporal basis for subsequent processing of data frames from nearest to furthest and prioritizing the encoding of high-importance targets. Furthermore, the chronological processing order naturally aligns with the data frame number determination mechanism, allowing more dynamic targets nearing the prediction time to be input into the autonomous driving model, improving decision-making and planning capabilities. Prioritizing the encoding of high-importance targets tilts limited token resources toward the most valuable perception information, further optimizing token resource utilization efficiency and decision-making effectiveness.

[0050] Figure 3This is a flowchart of a method for encoding perceived information about a target into an input autonomous driving model, according to an embodiment of the present disclosure. For illustrative purposes, the following description is presented in the context of the above description.

[0051] Figure 3 The process shown begins at step 310. In this step, the onboard computing platform receives the first data frame to be processed (e.g., the aforementioned data frame F) from the onboard perception system. h The vehicle-mounted computing platform described herein typically includes a processor for performing perception information encoding and autonomous driving model inference, and computer program instructions for storing perception information, encoded token sequences, autonomous driving models, and implementing the encoding methods and model execution. The vehicle-mounted perception system described herein typically includes multiple heterogeneous sensors and perception processing units connected to the sensors, used to collect raw data of the environment surrounding the autonomous vehicle, and obtain target perception information through data processing and fusion, including the position, size, category, and tracking identifier (track_id) of dynamic and static targets.

[0052] In the subsequent step 320, the onboard computing platform is based on the first data frame F h The objective within this framework determines the number h of data frames used to generate the token sequence for this input autonomous driving model. The specific method for determining the number h has been described in detail above and will not be repeated here.

[0053] Next, proceed to step 330, where the onboard computing platform processes data frame F. h The targets within the data frame are encoded. The targets include both static and dynamic targets, with dynamic targets including both moving and stationary targets. For static targets that need to be encoded, since their perceptual information is expected to remain substantially unchanged over h data frames, they are only encoded within data frame F. h Encoding is performed on the static target (i.e., no further encoding is performed during the processing of the subsequent (h-1) data frames). For example, assuming that encoding each static target consumes one token, the number of tokens consumed by the static targets, n_env, is equal to the number of static targets to be encoded. Similarly, for data frame F... h The stationary target that needs to be encoded is only processed in data frame F. h Encode it at that time. For data frame F h The motion targets that need to be encoded, in addition to processing data frames F hIn addition to encoding it at the time of encoding, it will also be encoded if it appears in subsequent data frames. After encoding is completed, the vehicle computing platform will also update the parameter n_left_state, for example, as shown in equations (3a) and (3b) above.

[0054] After completing step 330 Figure 3 The process shown proceeds to step 340. In this step, the onboard computing platform receives perception information about the target from the onboard perception system for the next data frame to be processed. Here, "next" refers to the temporal order, for example, for data frame F. h The next data frame is F. h-1 For data frame F h-1 The next data frame is F. h-2 And so on.

[0055] Then, in step 350, the onboard computing platform processes the currently used data frame F. i The target within (i=1,2……h-1) is encoded, and the parameter n_left_state is updated after encoding is complete (e.g., as shown in equation (4) above). In this step, for data frame F i For stationary targets within a data frame, if they have already appeared in a previously processed data frame, they will not be encoded again; for data frame F i For moving targets that need to be encoded, if they have already appeared in previously processed data frames, they will still be encoded; based on this, if data frame F... i If a moving target that needs to be encoded has not appeared in the previously processed data frame, then if token resources allow, all newly appearing moving targets that need to be encoded will be encoded; if token resources do not allow, only the high-importance targets will be encoded.

[0056] Next, proceed to step 360, where the vehicle computing platform determines whether the currently processed data frame is the last data frame (e.g., data frame F1). If so, proceed to step 370; otherwise, return to step 340.

[0057] In step 370, the vehicle-mounted computing platform, based on h data frames F1, F2...F... h The encoding result generates a sequence of tokens that will be input into the autonomous driving model (e.g., Figure 2 (The token sequence shown).

[0058] It should be noted that by repeating the execution Figure 3 The process shown will generate multiple sets of tokens to enable continuous input of perceived information into the autonomous driving model.

[0059] Figure 4 This is a schematic block diagram of a computing device. Figure 4 The computing device shown can be used to implement the aforementioned vehicle-mounted computing platform. For example... Figure 4 The computing device 400 shown includes one or more memories 410 (such as non-volatile memory such as flash memory, ROM, hard disk drive, disk, optical disk, etc.), one or more processors 420, and computer programs / instructions 430.

[0060] Computer program / instructions 430 are stored in memory 410 and can be executed by one processor 420 or by multiple processors 420 working together to perform the above-mentioned tasks. Figure 3 The steps of the described method. The memory 410 may also store data associated with the execution of computer programs / instructions (e.g., perception information, encoded token sequences, number of data frames h, encoding parameters (e.g., n_total, n_env, n_state, n_left_state, etc.) and parameters and weights of the autonomous driving model, etc.).

[0061] Processor 420 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 420 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), special-purpose artificial intelligence (AI) computing chip, computing unit running machine learning model algorithms, digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc.

[0062] To illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above according to their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in varying ways for specific applications; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0063] Although only some specific embodiments of this disclosure have been described, those skilled in the art will understand that this disclosure may be implemented in many other forms without departing from its spirit and scope. Therefore, the examples and embodiments shown are to be considered illustrative rather than restrictive, and various modifications and substitutions may be made without departing from the spirit and scope of this disclosure as defined by the appended claims.

[0064] The embodiments and examples presented herein are provided to best illustrate embodiments according to the present technology and its particular applications, thereby enabling those skilled in the art to implement and use the disclosure. However, those skilled in the art will understand that the above description and examples are provided for ease of illustration and example only. The descriptions presented are not intended to cover all aspects of the disclosure or to limit the disclosure to its precise forms.

Claims

1. A method for encoding perceived information about a target from an input autonomous driving model, wherein, The objectives include dynamic and static objectives, the input of the autonomous driving model contains a fixed number of tokens, and the method includes: The number of multiple data frames is determined based on the set number of tokens used to encode dynamic targets and the number of dynamic targets to be encoded in the first data frame, wherein the multiple data frames include the first data frame and a second data frame preceding the first data frame; and The encoding process of the dynamic targets of the multiple data frames is performed sequentially in order of time from most recent to furthest.

2. The method as described in claim 1, wherein, The method further includes: The set quantity is determined based on the fixed quantity and the number of static targets in the first data frame.

3. The method as described in claim 1, wherein, The dynamic target includes moving targets and stationary targets, and the number of the multiple data frames is determined as follows: If condition A is met: Then the number of the plurality of data frames is determined to be the maximum value; If condition B is met: ， The number of the plurality of data frames is then determined as follows: ; If neither condition A nor condition B is met, then the number of the plurality of data frames is determined to be the minimum value. Wherein, static_count is the number of stationary targets to be encoded in the first data frame, dynamic_count is the number of moving targets to be encoded in the first data frame, n_state is the set number, h is the number of multiple data frames, and max_h and min_h are the maximum and minimum values ​​of the number of multiple data frames, respectively.

4. The method of claim 3, wherein, The sequential encoding processing of the dynamic targets of the plurality of data frames includes processing the first data frame in the following manner: If condition A or B is met, then all dynamic targets that need to be encoded in the first data frame are encoded; otherwise, dynamic targets that need to be encoded that are closer to the vehicle are encoded first.

5. The method of claim 4, wherein, The sequential execution of the dynamic target encoding processing of the plurality of data frames includes processing the second data frame in the following manner: If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in the previously processed data frame and is a moving target, then the dynamic target is encoded. If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in a previously processed data frame and is a stationary target, then the dynamic target will not be encoded. as well as If the dynamic target that needs to be encoded in the second data frame being processed has not appeared in the previous data frames, then the dynamic target that is closer to the vehicle is encoded first.

6. The method as described in any one of claims 1-5, wherein, The dynamic target that needs to be encoded is determined based on the dynamic target's direction of motion or its position relative to the vehicle.

7. An apparatus for encoding perceived information about a target from an input autonomous driving model, wherein, The objectives include dynamic and static objectives, the input of the autonomous driving model contains a fixed number of tokens, and the device includes: At least one processor; At least one memory; and A computer program / instruction stored in the memory, when executed on the processor, causes the following operations: The number of multiple data frames is determined based on a set number of tokens that can be allocated to dynamic target encoding and the number of dynamic targets to be encoded in the first data frame, wherein the multiple data frames include the first data frame and a second data frame preceding the first data frame; and The encoding process of the dynamic targets of the multiple data frames is performed sequentially in order of time from most recent to furthest.

8. The apparatus of claim 7, wherein, The execution of the computer program / instructions on the processor also results in the following operations: The set quantity is determined based on the fixed quantity and the number of static targets in the first data frame.

9. The apparatus of claim 7, wherein, The dynamic target includes moving targets and stationary targets, and the number of the multiple data frames is determined as follows: If condition A is met: Then the number of the plurality of data frames is determined to be the maximum value; If condition B is met: ， The number of the plurality of data frames is then determined as follows: ; If neither condition A nor condition B is met, then the number of the plurality of data frames is determined to be the minimum value. Wherein, static_count is the number of stationary targets to be encoded in the first data frame, dynamic_count is the number of moving targets to be encoded in the first data frame, n_state is the set number, h is the number of multiple data frames, and max_h and min_h are the maximum and minimum values ​​of the number of multiple data frames, respectively.

10. The apparatus of claim 9, wherein, The sequential encoding processing of the dynamic targets of the plurality of data frames includes processing the first data frame in the following manner: If condition A or B is met, then all dynamic targets that need to be encoded in the first data frame are encoded; otherwise, dynamic targets that need to be encoded that are closer to the vehicle are encoded first.

11. The apparatus of claim 10, wherein, The sequential execution of the dynamic target encoding processing of the plurality of data frames includes processing the second data frame in the following manner: If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in the previously processed data frame and is a moving target, then the dynamic target is encoded. If the dynamic target that needs to be encoded in the currently processed second data frame has already appeared in a previously processed data frame and is a stationary target, then the dynamic target will not be encoded. as well as If the dynamic target that needs to be encoded in the second data frame being processed has not appeared in the previous data frames, then the dynamic target that is closer to the vehicle is encoded first.

12. The apparatus according to any one of claims 7-11, wherein, The dynamic target that needs to be encoded is determined based on the dynamic target's direction of motion or its position relative to the vehicle.

13. A non-transitory computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-6.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-6.

15. A vehicle comprising the means as described in any one of claims 7-12.

Images