Automatic driving environment occupation prediction method, device and system
By using a combination of learnable query vectors and decoders, the occupancy information in the autonomous driving environment is represented by sparse representation, which solves the problem of wasted computing resources in the prior art and improves the efficiency of occupancy prediction in the autonomous driving environment.
Patent Information
- Application Number
- CN202411068420.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-03
AI Technical Summary
Existing methods for autonomous driving environment occupation prediction lead to waste of computing resources, especially in the understanding of fine-grained environments, lack of efficient solutions.
Use a learnable query vector to obtain target position features from the description features of the autonomous driving environment, and determine the second occupancy set through the first occupancy set output by the decoder, and use sparse representation to represent the occupancy information to avoid using dense space representations that are not occupied. to improve prediction efficiency.
The waste of computing resources is reduced through sparse representation, the efficiency of occupation prediction is improved, and the waste of computing resources is solved by the fine-grained environmental understanding method.
Smart Images

Figure CN119068455A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving, and in particular to an autonomous driving environment occupancy prediction method, device, and system. Background Art
[0002] In the field of autonomous driving, vehicle decision-making and control are highly dependent on accurate perception and prediction of the environment. Occupancy prediction is a key technology that first divides the entire space into three-dimensional grids of uniform size, and then predicts each grid by inputting sensor signals such as images and radar point clouds. Specifically, occupancy prediction requires predicting two key quantities: (1) whether the grid is occupied, and (2) if it is occupied, what is the category of the grid, such as Figure 1 As shown. With this method, the autonomous driving system can build a detailed three-dimensional environmental model to identify and classify surrounding vehicles, pedestrians, buildings and other obstacles. This fine-grained environmental understanding not only helps to improve the accuracy of path planning and obstacle avoidance, but also enhances the safety and reliability of vehicles in complex traffic environments. However, the above-mentioned fine-grained environmental understanding method will also bring about a large amount of computing resource waste. For this technical problem, the relevant technology has not yet proposed an effective solution. Summary of the invention
[0003] The embodiments of the present application provide an autonomous driving environment occupancy prediction method, device, and system to solve one or more of the above-mentioned technical problems.
[0004] In a first aspect, an embodiment of the present application provides an occupancy prediction method for an autonomous driving environment, comprising: using a learnable query vector to obtain target position features from autonomous driving environment description features; predicting occupancy information based on the target position features, and determining a second occupancy set in combination with a first occupancy set output by an upper-layer decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; training the second occupancy set to determine a target occupancy prediction result.
[0005] In a second aspect, an embodiment of the present application provides an autonomous driving environment occupancy prediction device, comprising: a first acquisition module, used to use a learnable query vector to obtain target position features from autonomous driving environment description features; a second acquisition module, used to predict occupancy information on the target position features, and determine a second occupancy set in combination with a first occupancy set output by an upper-layer decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; a prediction module, used to train the second occupancy set to determine a target occupancy prediction result.
[0006] On the third aspect, an embodiment of the present application provides an autonomous driving environment occupancy prediction system, including: an encoder, which encodes the autonomous driving environment description information to obtain the autonomous driving environment description features; a decoder, which uses the above method to predict the occupancy of the autonomous driving environment and determine the target occupancy prediction result.
[0007] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.
[0008] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.
[0009] Compared with the related art, this application has the following advantages:
[0010] In an embodiment of the present application, a learnable query vector is used to obtain a target position feature from an autonomous driving environment description feature; occupancy information is predicted on the target position feature, and a second occupancy set is determined in combination with the first occupancy set output by the previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; the second occupancy set is trained to determine the target occupancy prediction result. That is, the embodiment of the present application uses a set to represent occupancy. If some grids are not occupied, they will not appear in the set, and there is no need to use the dense space in the related art to represent occupancy (for example, the dense space needs to save an "unoccupied" state for some grids). Further, the present application trains the occupancy set to determine the target occupancy prediction result, which solves the problem of a large amount of computing resource waste caused by the fine-grained environment understanding method in the related art, and achieves the technical effect of further improving the efficiency of occupancy prediction.
[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.
[0013] Figure 1 A schematic diagram of a scene of an occupancy prediction solution provided in the related art is shown;
[0014] Figure 2 A schematic diagram showing a scenario of an autonomous driving environment occupancy prediction solution provided in an embodiment of the present application
[0015] Figure 3 A flowchart of an autonomous driving environment occupancy prediction method provided in an embodiment of the present application is shown;
[0016] Figure 4 A structural block diagram of an autonomous driving environment occupancy prediction device provided in an embodiment of the present application is shown;
[0017] Figure 5 A structural block diagram of an autonomous driving environment occupancy prediction system provided in an embodiment of the present application is shown;
[0018] Figure 6 A schematic diagram of another autonomous driving environment occupancy prediction system provided in an embodiment of the present application is shown;
[0019] Figure 7 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION
[0020] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.
[0021] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.
[0022] The main occupancy prediction methods currently include: inputting (A) a series of images, using a neural network to predict (B) very dense features, and finally predicting (C) the occupancy output based on the features at each position. The core of this method is that it requires a dense representation of the entire space, using a convolutional neural network to perform high-density feature extraction, and then predicting the results at each corresponding position. In other words, the entire task is considered to be a classification task at each position. However, from Figure 2 As can be seen from the figure, the actual occupied space is very sparse. In other words, most of the grids are actually not occupied, and the prediction of many points is meaningless. Using dense representation is not only unnecessary, but also leads to a large waste of computing resources. Therefore, it is particularly important to find a more effective way to deal with this problem.
[0023] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0024] The present application embodiment provides a method for predicting the occupancy of an autonomous driving environment. Figure 3 The flowchart of a resource management method according to an embodiment of the present application is shown, and the method may include:
[0025] Step S302: Using the learnable query vector to obtain target location features from the autonomous driving environment description features.
[0026] It should be noted that the above-mentioned learnable query vector can be a query vector (query) in a deep learning model Transformer (composed of an encoder and a decoder). In the transformer, as the input of each stage, each query is a one-dimensional vector. In the network, this query feature will be updated and optimized in the decoder, and finally the corresponding prediction result will be output. The above-mentioned learnable query vector can include multiple ones. The above-mentioned autonomous driving environment description includes but is not limited to: images, lidar point clouds, depth maps, etc. The above-mentioned features can be the information output after the encoder encodes the autonomous driving environment description. The encoder can encode the autonomous driving environment description based on the self-attention mechanism. Among them, in the self-attention mechanism, for each position in the input sequence, the model calculates the relationship between it and all other positions and obtains a weight distribution. Then, the input sequence is weighted and summed according to this weight distribution to obtain the output of each position. In addition, it should be noted that the above-mentioned target position can include multiple ones.
[0027] Step S304, predicting occupancy information based on the target position feature, and determining a second occupancy set in combination with the first occupancy set output by the previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids.
[0028] It should be noted that the above-mentioned first occupied set and the above-mentioned second occupied set can be marked as {P, C}, where P and C are two sets, each set contains N elements, and for any pair of elements {p, c}∈{P, C}, p represents the three-dimensional coordinates of the center point of an occupied grid, and c represents the category corresponding to the grid.
[0029] In addition, it should be noted that the first occupied set and the second occupied set represent the occupied grid information. If the grid is not occupied, it will not appear in the set. In other words, the embodiment of the present application uses a sparse method to express the occupation result.
[0030] Step S306: training the second occupancy set to determine a target occupancy prediction result.
[0031] Optionally, in the embodiment of the present application, the above-mentioned method of training the second occupancy set includes but is not limited to: a set-to-set training method. For example, the second occupancy set {P, C} is compared with the real N g The set of actually occupied grids {P g , C g}Perform supervised learning to determine the target occupancy prediction results.
[0032] Through the above steps S302 to S306, the target position feature is obtained from the autonomous driving environment description feature using a learnable query vector; the occupancy information is predicted on the target position feature, and the second occupancy set is determined in combination with the first occupancy set output by the previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; the second occupancy set is trained to determine the target occupancy prediction result. That is, the embodiment of the present application uses a set to represent occupancy. If some grids are not occupied, they will not appear in the set, and there is no need to use the dense space in the related technology to represent occupancy (for example, the dense space needs to save an "unoccupied" state for some grids). Further, the present application trains the occupancy set to determine the target occupancy prediction result, which solves the problem of a large amount of computing resource waste caused by the fine-grained environment understanding method in the related technology, and achieves the technical effect of further improving the efficiency of occupancy prediction.
[0033] In one possible implementation, the second occupancy set is trained to determine the target occupancy prediction result, including: S11, obtaining a third occupancy set, wherein the third occupancy set describes the actual occupancy information in the autonomous driving environment, and the third occupancy set includes multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid; S12, using the loss between the second occupancy set and the third occupancy set to train the second occupancy set to determine the target occupancy prediction result. That is, in an embodiment of the present application, the second occupancy set is trained by the loss function determined by the second occupancy set and the third occupancy set, for example, minimizing the loss function so that the second occupancy set and the third occupancy set have the same distribution. Through the above S11 to S12, the correspondence between the predicted second occupancy set and the real labeled third occupancy set can be efficiently determined.
[0034] The above-mentioned S12 may include: S121, determining a first loss between the three-dimensional coordinates of the center point of the occupied grid in the second occupation set and the three-dimensional coordinates of the center point of the occupied grid in the third occupation set; S122, determining a second loss between the category of the occupied grid in the second occupation set and the category of the occupied grid in the third occupation set; training the second occupation set by the first loss and the second loss to determine the target occupancy prediction result.
[0035] Optionally, the above S121 may include: S1211, determining the chamfer distance between the three-dimensional coordinates of the center point of the occupied grid in the second occupation set and the three-dimensional coordinates of the center point of the occupied grid in the third occupation set; S1212, setting the chamfer distance as the first loss. The above S122 may include: S1221, obtaining the target point in the third occupation set that is closest to the three-dimensional coordinates of the center point of the occupied grid in the second occupation set; S1222, obtaining the target category corresponding to the target point; S1223, determining the loss between the category of the occupied grid in the second occupation set and the target category; S1224, setting the loss to the second loss. By comparing the second occupation set with the third occupation set through the above S1211~S1212 and S1221~S1224, the network can be quickly trained.
[0036] In a possible implementation, the above S304 may include: S21, obtaining a first occupancy set output by the previous layer decoder; S22, updating the occupancy information predicted on the target position feature into the first occupancy set to obtain the second occupancy set. That is, the second occupancy set output by the current layer decoder is determined by updating the first occupancy set, so that the occupancy prediction result is more accurate.
[0037] Since the above-mentioned learnable query vector is updated in each query process, the embodiment of the present application also proposes the following method: S31, determining the target query vector corresponding to the learnable query vector through the self-attention mechanism; S32, outputting the target query vector. Through the above-mentioned S31-S32, the learnable query vector can obtain the increasingly important target position features in the description features of the autonomous driving environment.
[0038] The embodiments of the present application are described below with reference to specific examples.
[0039] In this example, for the annotation of the ground truth, the real N g The set of actually occupied grids is denoted as {P g , C g}(corresponding to the third occupied set), where P g and C g There are two sets, each containing N g elements, where for any pair of elements {p g , c g}∈{P g , C}, p g represents the three-dimensional coordinates of the center point of an occupied grid, c gRepresents the category corresponding to this grid. For prediction, a set is also used to represent N predictions {P, C} (corresponding to the second occupancy set). Because all results are represented in the form of a set, there is no need to use dense space to express occupancy. This is because if a grid is not occupied, it will not appear in the set. And the dense space needs to save an "unoccupied" state for this grid, which will lead to unnecessary waste of resources. By using set representation, occupancy information can be managed and processed more efficiently, avoiding a lot of meaningless data storage and calculation.
[0040] In the previous dense representation, each output is based on a fixed position, so a one-to-one correspondence can be established to supervise model training. However, due to the disorder of the set, this correspondence becomes difficult to find. This brings challenges to supervised model training because it is impossible to directly find the corresponding benchmark truth value of the prediction result for supervision. Therefore, this example proposes a new set-to-set training method.
[0041] For the matching algorithm, one possible approach is to use Hungarian matching. However, the Hungarian matching algorithm (an algorithm for finding a match between two sets) is too complex and cannot be applied in large-scale scenarios. Therefore, this example designs a new approach. The overall approach is divided into two steps:
[0042] (1) The first step is to let the predicted point P and the ground truth point P g This example uses the Chamfer distance loss, as shown in Formula 1.
[0043]
[0044] By minimizing the above loss function, the predicted points and the reference truth points can achieve similar distributions, and the overall algorithm can be calculated in parallel to support large-scale operations.
[0045] (2) The second step is to make the predicted category C also accurate. Similarly, we cannot directly compare C and C g Perform a one-to-one comparison, because there is no corresponding order between the two sets, and the set sizes are also different (N and N respectively). g ). Therefore, we use nearest neighbor matching to find a category set C n , as shown in Formula 2.
[0046]
[0047] That is, for each prediction p, find the point closest to it in the benchmark truth, and then save the category corresponding to this point, and finally form C n. You can see that C and C n The corresponding relationship is consistent, so the existing classification loss can be used to train L c (C,C n ), where L c Represents the classification loss function. There are many representation options, such as focal loss, cross-entropy loss, etc.
[0048] Through the above two steps, the output occupancy prediction {P, C} can be compared with the benchmark truth, so that the network can be trained quickly.
[0049] In summary, the input and output of the structure based on the Transformer in the embodiment of the present application are queries, which are some discrete features. This feature makes the Transformer very suitable for processing sparse data, because it can directly focus on and process important position information while ignoring those irrelevant parts. By utilizing the self-attention mechanism of the Transformer model, sparse occupied spaces can be efficiently predicted and classified, significantly reducing the waste of computing resources. Specifically, in the embodiment of the present application, the task of occupancy prediction is redefined for the first time from a classification problem at a specific location to a set prediction problem. In addition, the embodiment of the present application proposes a set-to-set supervised training method.
[0050] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides an autonomous driving environment occupancy prediction device. Figure 4 The figure shows a structural block diagram of an autonomous driving environment occupancy prediction device according to an embodiment of the present application. The device may include:
[0051] A first acquisition module 42, configured to acquire target position features from autonomous driving environment description features using a learnable query vector;
[0052] A second acquisition module 44 is used to predict occupancy information based on the target position feature, and determine a second occupancy set in combination with a first occupancy set output by a previous layer decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids;
[0053] The prediction module 46 is used to train the second occupancy set to determine a target occupancy prediction result.
[0054] pass Figure 4The device shown uses a learnable query vector to obtain the target position feature from the autonomous driving environment description feature; predicts the occupancy information on the target position feature, and determines the second occupancy set in combination with the first occupancy set output by the previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; the second occupancy set is trained to determine the target occupancy prediction result. That is, the embodiment of the present application uses a set to represent occupancy. If some grids are not occupied, they will not appear in the set, and there is no need to use the dense space in the related technology to represent occupancy (for example, the dense space needs to save an "unoccupied" state for some grids). Further, the present application trains the occupancy set to determine the target occupancy prediction result, which solves the problem of a large amount of computing resource waste caused by the fine-grained environment understanding method in the related technology, and achieves the technical effect of further improving the efficiency of occupancy prediction.
[0055] In one possible implementation, the prediction module 46 includes: a first acquisition unit, used to acquire a third occupancy set, wherein the third occupancy set describes the actual occupancy information in the autonomous driving environment, and the third occupancy set includes multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid; a determination unit, used to train the second occupancy set using the loss between the second occupancy set and the third occupancy set to determine the target occupancy prediction result.
[0056] The determination unit includes: a first determination subunit, used to determine a first loss between the three-dimensional coordinates of the center point of the occupied grid in the second occupation set and the three-dimensional coordinates of the center point of the occupied grid in the third occupation set; a second determination subunit, used to determine a second loss between the category of the occupied grid in the second occupation set and the category of the occupied grid in the third occupation set; and a third determination subunit, used to train the second occupation set through the first loss and the second loss to determine the target occupancy prediction result.
[0057] The first determination subunit is further used to determine a chamfer distance between the three-dimensional coordinates of the center point of the occupied grid in the second occupation set and the three-dimensional coordinates of the center point of the occupied grid in the third occupation set; and set the chamfer distance as the first loss.
[0058] The above-mentioned second determination subunit is also used to obtain, in the third occupation set, a target point that is closest to the three-dimensional coordinates of the center point of the occupied grid in the second occupation set; obtain a target category corresponding to the target point; determine the loss between the category of the occupied grid in the second occupation set and the target category; and set the loss as the second loss.
[0059] The second acquisition module 44 includes: a second acquisition unit, used to acquire the first occupancy set output by the previous layer decoder; and a third acquisition unit, used to update the occupancy information predicted on the target position feature into the first occupancy set to obtain the second occupancy set.
[0060] The above-mentioned device also includes: a determination module, which is used to determine the target query vector corresponding to the learnable query vector through a self-attention mechanism; and an output module, which is used to output the target query vector.
[0061] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.
[0062] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides an autonomous driving environment occupancy prediction system. Figure 5 The figure shows a block diagram of the autonomous driving environment occupancy prediction system according to an embodiment of the present application, including:
[0063] An encoder 52 encodes the autonomous driving environment description information to obtain autonomous driving environment description features;
[0064] The decoder 54 uses the above method to predict the occupancy of the autonomous driving environment and determines the target occupancy prediction result.
[0065] In one possible implementation, the above-mentioned decoders may include multiple ones, the input of the i-th decoder includes the description features of the autonomous driving environment, the learnable query vector, and the occupancy prediction result output by the i-1-th decoder, the output of the i-th decoder includes the target query vector, and the occupancy prediction result of the i-th decoder, where i is an integer not less than 1.
[0066] pass Figure 5The system shown uses a learnable query vector to obtain target position features from the description features of the autonomous driving environment; predicts occupancy information on the target position features, and determines a second occupancy set in combination with the first occupancy set output by the previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; the second occupancy set is trained to determine the target occupancy prediction result. That is, the embodiment of the present application uses a set to represent occupancy. If some grids are not occupied, they will not appear in the set, and there is no need to use the dense space in the related technology to represent occupancy (for example, the dense space needs to save an "unoccupied" state for some grids). Further, the present application trains the occupancy set to determine the target occupancy prediction result, which solves the problem of a large amount of computing resource waste caused by the fine-grained environment understanding method in the related technology, and achieves the technical effect of further improving the efficiency of occupancy prediction.
[0067] The embodiments of the present application are described below with reference to examples.
[0068] like Figure 6 As shown in the figure, this example is an encoder-decoder structure based on Transformer. First, use the image encoder to convert the image into the corresponding feature vector. Note that the input can be not only an image, but can be extended to any input, such as LiDAR point cloud, depth map, etc.
[0069] Next, a series of decoders are needed to decode the results. This example framework uses 6, but any number is fine. The input to the i-th decoder includes: (1) the features produced by the encoder; (2) multiple query points Q i-1 , the number and feature dimensions of these queries are the same everywhere. (3) The occupancy prediction value {P i-1 , C i-1}. Optionally, the first decoder input Q 0 , P 0 , C 0 Random initialization can be used. The output of the i-th layer decoder is Q after some updated features i , and we start from Q i Through several linear layers, the new occupancy prediction {P i , C i}. Figure 6Where Q is the number of queries, R is the number of points predicted by Q, and new occupancy results can be predicted from the query. Finally, during the training process, the occupancy predictions at each stage are supervised by the ground truth, and the final loss is shown in Formula 3 and Formula 4.
[0070]
[0071] in,
[0072] The entire framework described above is trained end-to-end using a sparse query approach. In the test, this example uses the occupancy prediction of the last layer of the decoder as the final result.
[0073] The functions of each module in the system of the embodiment of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.
[0074] Figure 7 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 7 As shown, the electronic device includes: a memory 701 and a processor 702. The memory 701 stores a computer program that can be run on the processor 702. When the processor 702 executes the computer program, the method in the above embodiment is implemented. The number of the memory 701 and the processor 702 can be one or more.
[0075] The electronic device also includes:
[0076] The communication interface 703 is used to communicate with external devices and perform data exchange transmission.
[0077] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the memory 701, the processor 702 and the communication interface 703 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0078] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.
[0079] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.
[0080] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.
[0081] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.
[0082] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.
[0083] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).
[0084] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0086] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0087] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.
[0088] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.
[0089] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.
[0090] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.
[0091] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for predicting an autonomous driving environment occupancy, comprising: Use a learnable query vector to obtain target location features from autonomous driving environment description features; Predicting occupancy information based on the target position feature, and determining a second occupancy set in combination with a first occupancy set output by a previous decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; The second occupancy set is trained to determine a target occupancy prediction result.
2. The method according to claim 1, wherein: Training the second occupancy set to determine a target occupancy prediction result includes: Acquire a third occupancy set, wherein the third occupancy set describes real occupancy information in the autonomous driving environment, and the third occupancy set includes multiple pairs of elements, and any pair of elements in the multiple pairs of elements includes the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid; The second occupancy set is trained using the loss between the second occupancy set and the third occupancy set to determine the target occupancy prediction result.
3. The method according to claim 2, wherein: Using the loss between the second occupancy set and the third occupancy set to train the second occupancy set, and determining the target occupancy prediction result includes: Determining a first loss between the three-dimensional coordinates of the center points of the occupied grids in the second occupation set and the three-dimensional coordinates of the center points of the occupied grids in the third occupation set; determining a second loss between the categories of occupied cells in the second occupation set and the categories of occupied cells in the third occupation set; The second occupancy set is trained by using the first loss and the second loss to determine the target occupancy prediction result.
4. The method according to claim 3, wherein: Determining a first loss between the three-dimensional coordinates of the center point of the occupied grid in the second occupation set and the three-dimensional coordinates of the center point of the occupied grid in the third occupation set includes: Determine a chamfer distance between the three-dimensional coordinates of the center point of the occupied grid in the second occupied set and the three-dimensional coordinates of the center point of the occupied grid in the third occupied set; The chamfer distance is set to the first loss.
5. The method according to claim 3, wherein: Determining a second loss between the category of the occupied grids in the second occupied set and the category of the occupied grids in the third occupied set includes: Acquire, in the third occupied set, a target point that is closest to the three-dimensional coordinates of the center point of the occupied grid in the second occupied set; Obtaining a target category corresponding to the target point; Determining a loss between a category of occupied grids in the second occupied set and the target category; The loss is set to be the second loss.
6. The method according to claim 1, wherein: Predicting occupancy information based on the target position feature and determining a second occupancy set in combination with a first occupancy set output by a previous decoder includes: Obtain a first occupied set output by the previous layer decoder; The occupancy information predicted on the target position feature is updated into the first occupancy set to obtain the second occupancy set.
7. The method according to claim 1, wherein: The method further comprises: Determining a target query vector corresponding to the learnable query vector through a self-attention mechanism; Output the target query vector.
8. An autonomous driving environment occupancy prediction device, comprising: A first acquisition module is used to acquire target position features from autonomous driving environment description features using a learnable query vector; a second acquisition module, configured to predict occupancy information based on the target position feature, and determine a second occupancy set in combination with a first occupancy set output by a previous layer decoder, wherein the first occupancy set and the second occupancy set include multiple pairs of elements, any pair of elements in the multiple pairs of elements include the three-dimensional coordinates of the center point of the occupied grid and the category of the occupied grid, and the autonomous driving environment is divided into multiple three-dimensional grids; The prediction module is used to train the second occupancy set to determine a target occupancy prediction result.
9. An autonomous driving environment occupancy prediction system, comprising: An encoder encodes the autonomous driving environment description information to obtain autonomous driving environment description features; A decoder, using the method described in any one of claims 1 to 7 to perform occupancy prediction on the autonomous driving environment and determine a target occupancy prediction result.
10. The system according to claim 9, wherein: The decoders include multiple decoders, the input of the i-th decoder includes the autonomous driving environment description features, the learnable query vector, and the occupancy prediction result output by the i-1-th decoder, the output of the i-th decoder includes the target query vector and the occupancy prediction result of the i-th decoder, where i is an integer not less than 1.
11. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
12. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
