Visual information-based four-dimensional occupation grid panoramic sensing method and device

Abstract

The invention relates to a four-dimensional occupied grid panoramic sensing method and device based on visual information. The method comprises the following steps: after acquiring a multi-view image, inputting the multi-view image into a pre-constructed four-dimensional panorama perception model based on visual information, and outputting a four-dimensional panorama occupation grid prediction result; wherein the visual information-based four-dimensional panoramic perception model comprises an encoder module, a decoder module and a query vector propagation module, the encoder module is used for extracting three-dimensional volume features, the query vector propagation module is used for updating four-dimensional query vectors, and the decoder module is used for realizing interaction between the three-dimensional volume features and the four-dimensional query vectors. Compared with the prior art, the method has the advantages that the final four-dimensional panoramic occupied grid prediction result is generated in an end-to-end and streaming mode, and a large amount of post-processing requirements can be eliminated.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a four-dimensional occupancy grid panoramic perception method and device based on visual information. Background Art

[0002] In dynamic environments, perception systems of robots and autonomous vehicles need to estimate the geometry, semantic information, and identity of the current scene in a spatially continuous and temporally consistent manner, so as to be able to interact with complex and changing three-dimensional (i.e., 3D) environments.

[0003] In the prior art, when the perception systems of robots and self-driving cars face the problem of four-dimensional (i.e., 4D) panoramic perception, the mainstream method is a panoramic segmentation method based on the time-consistent LiDAR. This method divides the entire LiDAR sequence into small data frame windows, superimposes the point clouds in the windows, and then performs three-dimensional panoramic segmentation. After that, between each window, objects are associated according to the degree of overlap. Finally, the point cloud with the result label is split into the original data frame to achieve panoramic perception. However, this method of achieving 4D panoramic perception will inevitably bring post-processing problems due to panoramic segmentation in 3D, limited semantic richness and high cost of use. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a four-dimensional occupancy grid panoramic perception method and device based on visual information, which can extract three-dimensional volume features from multi-view images, and use a query vector propagation module to provide a four-dimensional query vector to perform panoramic segmentation and target tracking. The decoder is used to interact with the three-dimensional volume feature and four-dimensional query vector module to generate the final four-dimensional panoramic occupancy grid prediction result in an end-to-end, streaming manner, which can eliminate the need for a large amount of post-processing.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a four-dimensional occupancy grid panoramic perception method based on visual information is provided, comprising the following steps: acquiring multi-view images; inputting the multi-view images into a pre-built four-dimensional panoramic perception model based on visual information, and outputting a four-dimensional panoramic occupancy grid prediction result; wherein the four-dimensional panoramic perception model based on visual information comprises an encoder module, a decoder module and a query vector propagation module, the encoder module is used to extract three-dimensional volume features, the query vector propagation module is used to update a four-dimensional query vector, and the decoder is used to realize the interaction between the three-dimensional volume features and the four-dimensional query vector.

[0007] As a preferred technical solution, the encoder module includes an image feature extractor and a two-dimensional-three-dimensional conversion submodule, the image feature extractor is used to capture multi-scale features from the multi-view image, and the two-dimensional-three-dimensional conversion submodule is used to convert the multi-scale features into three-dimensional volume features.

[0008] As a preferred technical solution, the four-dimensional query vector is a concatenation of a new query vector and a tracking query vector, the new query vector is used to determine a newly detected object, and the tracking query vector is used to determine a tracked object that has been detected.

[0009] As a preferred technical solution, the encoder outputs an updated four-dimensional query vector, and the updated four-dimensional query vector is returned to the decoder through the query vector propagation module for next frame sample processing, specifically including: during the training process of the four-dimensional panoramic perception model, the query vectors matching countable objects in the updated new query vectors are converted into tracking query vectors, and the query vectors that have not been matched for multiple consecutive frames in the updated tracking query vectors are removed.

[0010] As a preferred technical solution, during the testing process of the four-dimensional panoramic perception model, the predicted classification scores are used to determine newly detected objects and disappeared tracked objects, wherein, in the updated new query vector, the query vector whose classification score is higher than a first preset score threshold and matches countable objects is retained, and in the updated tracking query vector, the query vector whose score is lower than a second preset score threshold for multiple consecutive frames is removed.

[0011] As a preferred technical solution, the interaction between the three-dimensional volume feature and the four-dimensional query vector is achieved through a volume cross-attention mechanism, which specifically includes: each new query uses a multi-layer perceptron layer to predict the corresponding 3D reference point; each tracking query maintains the corresponding reference point in time; the three-dimensional volume feature is sampled around each reference point; the sampled features are weighted summed and the interaction result is output.

[0012] As a preferred technical solution, the volumetric cross-attention mechanism includes a volumetric cross-attention mechanism based on deformable attention.

[0013] As an optimal technical solution, during the training process, the four-dimensional panoramic perception model based on visual information adopts a total loss function for supervised training. The total loss function is the sum of a mask classification loss function and a positioning perception loss function. The mask classification loss is used to evaluate the quality of the predicted mask, and the positioning perception loss is used to evaluate the quality of the predicted query position.

[0014] As a preferred technical solution, the expression of the positioning perception loss function is:

[0015]

[0016] In the formula, represents the positioning perception loss, P ′ is the center point of each object in the prediction result, is the center point of the corresponding object in the true value, and L1 represents the Manhattan distance.

[0017] According to a second aspect of the present invention, there is provided a four-dimensional occupancy grid panoramic perception device based on visual information, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the method when executing the program.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. The present invention introduces a query vector propagation module to realize a query mechanism for four-dimensional panoramic occupancy grid perception. This mechanism can realize panoramic perception of three-dimensional grids with consistent temporal sequence. Compared with the panoramic perception based on lidar with consistent temporal sequence, it can use cheaper and richer image information, and realize a perception method that is more suitable for actual use scenarios in a streaming and end-to-end manner, which can eliminate the need for a large amount of post-processing, and reduce the cost of use while having rich semantic information.

[0020] 2. The present invention proposes a positioning perception loss function, which can guide the four-dimensional panoramic perception model based on visual information to perceive the position information of the tracked target more accurately, thereby improving the performance of the temporal consistency panoramic segmentation of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the framework flow of the method provided by the present invention;

[0022] in: Represents element-wise product. DETAILED DESCRIPTION

[0023] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0024] Embodiment 1:

[0025] Occupancy grid is a three-dimensional visual representation in the field of autonomous driving perception. This representation divides the entire environment into cubic grids, and each grid is assigned a semantic label. The method proposed in the present invention adds the time dimension to the occupancy grid to achieve end-to-end 4D occupancy grid panoramic perception.

[0026] In the 4D panoramic perception problem, the mainstream lidar-based time-consistent panoramic segmentation method uses a 3D panoramic perception model plus post-processing to achieve 4D panoramic perception, which is costly and has limited semantic richness. Currently, there is no corresponding method to achieve the end-to-end time-consistent panoramic perception problem in dynamic environments based on pure vision.

[0027] Specifically, in actual use, the machine perception system needs to be able to rely on images collected by cheap multi-cameras as input, and then assign time-consistent semantic labels to the surrounding dynamic scenes, and assign time-consistent identity labels to each individual in the countable object. The entire reasoning process needs to be end-to-end, rather than requiring post-processing, otherwise it cannot be deployed in actual scenarios. If this requirement is met, the machine perception system will be able to achieve a higher level of understanding of the surrounding dynamic objects and scenes, thereby achieving a higher level of self-relative position positioning, obstacle avoidance path planning, and reliable interaction with the dynamic environment and other downstream tasks.

[0028] Based on this, the present embodiment provides a four-dimensional occupancy grid panoramic perception method based on visual information (TrackOcc method), which is a tracker method based on end-to-end learning, and aims to realize 4D panoramic occupancy tracking based on visual information, and solve the problems of occupied panoramic segmentation and target tracking. This method adopts a mask classification method to realize panoramic occupancy prediction, and introduces a 4D query vector in the prediction framework. The main implementation method is: after acquiring multi-view images, the multi-view images are input into a pre-built four-dimensional panoramic perception model based on visual information, and a four-dimensional panoramic occupancy grid prediction result is output; wherein, the four-dimensional panoramic perception model based on visual information includes an encoder module, a decoder module and a query vector propagation module, the encoder module is used to extract three-dimensional volume features, the query vector propagation module is used to provide a four-dimensional query vector, and the decoder is used to realize the interaction between three-dimensional volume features and four-dimensional query vectors. Its framework process is as follows Figure 1 As shown, the specific process of the method is as follows:

[0029] Step S1, using a multi-camera to obtain a video stream, that is, a multi-view image, can be expressed as:

[0030]

[0031] Where M represents the number of surrounding cameras, K represents the sequence length, and t represents time.

[0032] Given a predetermined set of C semantic categories, encoded as C:={0,…,C-1}, the 4D panoramic occupancy tracking task requires the neural network to map each voxel i in the grid into a pair (c i ,z i )∈C×N, where c irepresents the semantic category of voxel i, z i Represents the object ID of voxel i. The semantic label set includes countable objects (referring to objects with clear boundaries and can be counted) and stuff (referring to areas in the environment without clear boundaries and uncountable). When a voxel is labeled as stuff, its corresponding object ID z i is irrelevant. Voxels in free space are assigned a special label of free. Object ID z i Voxels of the same category are grouped into different parts, which should remain unchanged throughout the sequence. The dimensions of the grid are X×Y×Z, which represent the height, width, and depth of the grid, respectively. A voxel here represents a small cube or unit volume data element of a regular grid in three-dimensional space.

[0033] Step S2, input the multi-view images into a pre-built four-dimensional panoramic perception model based on visual information, and output the four-dimensional panoramic occupancy grid prediction result. The prediction process specifically includes three parts:

[0034] In the first part, the multi-view images pass through the encoder module to obtain 3D volume features.

[0035] The encoder module includes an image feature extractor and a two-dimensional-three-dimensional (i.e., 2D-3D) conversion submodule. The image feature extractor extracts multi-scale features from multi-view images, and the 2D-3D conversion submodule converts the multi-scale features into downsampled three-dimensional volume features. The three-dimensional volume feature V is expressed as:

[0036]

[0037] In the formula, X, Y, and Z represent the height, width, and depth of the grid, respectively. represents the number of characteristic channels, and R represents a common real number set in the field of mathematics.

[0038] In this embodiment, the encoder of COTR is used as the 2D-3D conversion submodule. In some other embodiments, the 2D-3D conversion submodule can use other methods such as LSS, BevDepth, BEVFormer, FB-Occ, etc.

[0039] In the second part, the four-dimensional query vector is updated through the query vector propagation module.

[0040] The TrackOcc method proposed in this embodiment introduces a four-dimensional query vector to interact with the query points in the three-dimensional volume features through a volume cross-attention mechanism. Specifically, the newly detected object is bound to the new query vector Q em The object detected in the previous frame uses the previous query vector, which is named the tracking query vector Q trThe new query vector and the tracking query vector are concatenated into a four-dimensional query vector

[0041] The nascent query vector is used for stuff and detecting new things, while the tracking query vector is responsible for predicting all existing tracked objects. For simplicity, stuff is assigned to the nascent query vector since it does not need to be tracked. The set of tracking query vectors is dynamically updated and its size changes over time. In the first frame, there is no tracking query vector; only a fixed-length, learnable nascent query vector is input to the decoder. In subsequent frames, Q can be assigned to the tracking query vector. t These queries are then fed into the decoder where they are interacted with the 3D volume features to generate an updated 4D query vector Updated 4D query vector It is used to generate the final 4D panoramic occupancy grid prediction result and is also passed to the query vector propagation module to generate the tracking query for the next frame.

[0042] The query vector propagation module is introduced to handle objects that may appear or disappear in the intermediate frames. The updated four-dimensional query vector The query vector propagation module returns to the sample processing of the next frame. Specifically:

[0043] During the training process, the updated new query vector In the above example, only query vectors corresponding to countable objects are converted into tracking query vectors. Updated tracking query vector Medium, continuous T f Query vectors that are not matched in the frame are removed;

[0044] During test inference, the predicted classification scores are used to determine newly detected objects and missing tracked objects, and the updated new query vector In the above example, only query vectors with scores higher than the first preset score threshold τ1 and corresponding to countable objects will be retained. The updated tracking query vector Medium, continuous T f Query vectors whose frames are below the second score threshold τ2 will be removed.

[0045] In the third part, the 3D volume features interact with the 4D query vector in the decoder module.

[0046] In the decoder, the 3D volume features interact with the 4D query vector through the volumetric cross attention (VCA) mechanism. The detailed process is as follows:

[0047] First, query vectors interact with each other to enhance features through a self-attention layer. Then, they find and aggregate volume features through a volume cross-attention (VCA) layer. Due to the large input size of volume features, the computational cost of ordinary attention is high. Therefore, a deformable attention-based VCA layer is adopted, which is a resource-efficient attention mechanism in which each query only interacts with its region of interest in the volume.

[0048] Specifically, each new query q∈Q em Predict a 3D reference point using a multi-layer perceptron (MLP) layer And each tracking query q∈Q tr The corresponding reference points are kept in time, and the volume features V are sampled around these reference points. The sampled features are then weighted summed as the output.

[0049] The process of VCA can be expressed as:

[0050]

[0051] Where j is the sampling point index, is the weight, are learnable weights and bias terms.

[0052] The second part is combined with the third part, and the four-dimensional panoramic occupancy grid prediction result can be obtained by performing element-wise product of the updated four-dimensional query vector and the corresponding three-dimensional volume feature.

[0053] In addition, for countable objects in the prediction results, supervised training is performed using a loss function. In this embodiment, the total loss function is the sum of a commonly used mask classification loss function and a positioning perception loss function proposed by this method.

[0054] The optimization objectives of the TrackOcc method mainly focus on two aspects: mask prediction and query position prediction.

[0055] Mask classification loss

[0056] Different types of queries are matched to the true labels using different strategies. Tracking query vectors persist throughout the entire stream, which enables each query vector to be assigned a true label only once across all time steps. Once created by inheriting from a nascent query vector, a tracking query vector is bound to the corresponding true label and remains unchanged.

[0057] The new query vector is converted between different time frames and does not need to be explicitly assigned at each time step. According to Mask2Former, the correspondence between the true label (4D panoramic label) and the 4D panoramic prediction result is established through bipartite matching, which is solved by the Hungarian algorithm. Once the matching relationship is established, the mask classification loss associated with the query is calculated and recorded as These include binary mask loss and multi-class cross entropy loss

[0058] Positioning Awareness Loss

[0059] Accurately localizing and tracking each query position is both important and challenging. Previous work that relies on mask classification for 3D panorama prediction, such as SparseOcc, has limited localization capabilities. Therefore, this method proposes a localization-aware loss function. The localization-aware loss function uses the center point P of each object in the prediction result to obtain the localization information of the object. ’ The Manhattan distance (i.e., L1 distance) between the center point P of the corresponding object in the true value is used for supervised training.

[0060] The expression of the positioning perception loss function is:

[0061]

[0062] The location-aware loss function is used to improve the accuracy of querying specific 3D locations, since these locations are used as reference points for attention in the VCA module. It should be noted that the center point P of each object in the prediction result ′ It comes only from queries bound to the thing class, since the stuff region lacks suitable and consistent ground-truth labeled points; P can be obtained by simply computing the center of mass of each object.

[0063] Therefore, the final total loss function is:

[0064]

[0065] Next, the effectiveness of the TrackOcc method proposed in this embodiment is verified through experiments:

[0066] According to the above method flow, the model sets 200 new query vectors in each frame. The dimension of each new query vector is 256, τ1, τ2 and T f The values ​​of are 0.3, 0.25 and 3 respectively. 24 training runs were performed on the training set, and the learning rate was 2×10 -4, the optimizer uses AdamW. Then the test is performed on the test set as described above. The comparison results of the TrackOcc method with other methods are shown in the following table. It can be seen that compared with similar methods, this method can achieve an end-to-end mode and significantly surpass other solutions in terms of evaluation indicators.

[0067]

[0068]

[0069] Among them, MinVIS and CTVIS are inter-frame correlation methods based on query vectors; AB3DMOT is a method for tracking through target detection; 4D-LCA, similar to the processing method of related problems in lidar data modality, realizes four-dimensional segmentation by performing three-dimensional segmentation on time-series superimposed data.

[0070] The sources of the various comparison schemes are as follows:

[0071] [1] D.-A.Huang, Z.Yu, and A.Anandkumar, "Minvis: A minimal video instancesegmentation framework without video-based training," Advances in NeuralInformation Processing Systems, vol.35, pp.31 265–31 277, 2022.

[0072] [2] K.Ying, Q.Zhong, W.Mao, Z.Wang, H.Chen, LYWu, Y.Liu, C.Fan, Y.Zhuge, andC.Shen, "Ctvis: Consistent training for online video instance segmentation," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp.899–908.

[0073] [3] X. Weng, J. Wang, D. Held, and K. Kitani, "3DMulti-ObjectTracking: ABaseline and New Evaluation Metrics," IROS, 2020.

[0074] [4] M. Aygun, A. Osep, M. Weber, M. Maximov, C. Stachniss, J. Behley, and L. Leal-Taixe′, “4d panoptic lidar segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5527–5537.

[0075] Example 2:

[0076] The present embodiment provides a four-dimensional occupancy grid panoramic perception device based on visual information, including a memory, a processor, and a program stored in the memory, and the processor implements one or more steps of the method in Example 1 when executing the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit to a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as one or more steps of the method in Example 1. For example, in some embodiments, one or more steps of the method in embodiment 1 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the method in embodiment 1 described above may be executed. Alternatively, in other embodiments, the CPU may be configured to execute one or more steps of the method in embodiment 1 by any other appropriate means (e.g., by means of firmware). The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0077] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A four-dimensional occupancy grid panoramic perception method based on visual information, characterized in that: The following steps are involved: Acquire multi-view images; Inputting the multi-view images into a pre-built four-dimensional panoramic perception model based on visual information, and outputting a four-dimensional panoramic occupancy grid prediction result; Among them, the four-dimensional panoramic perception model based on visual information includes an encoder module, a decoder module and a query vector propagation module, the encoder module is used to extract three-dimensional volume features, the query vector propagation module is used to update the four-dimensional query vector, and the decoder is used to realize the interaction between the three-dimensional volume features and the four-dimensional query vector.

2. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 1 is characterized in that: The encoder module includes an image feature extractor and a two-dimensional-three-dimensional conversion submodule, wherein the image feature extractor is used to capture multi-scale features from the multi-view images, and the two-dimensional-three-dimensional conversion submodule is used to convert the multi-scale features into three-dimensional volume features.

3. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 1, characterized in that: The four-dimensional query vector is a concatenation of a new query vector and a tracking query vector. The new query vector is used to determine a newly detected object, and the tracking query vector is used to determine a tracked object that has been detected.

4. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 3 is characterized in that: The encoder outputs an updated four-dimensional query vector, and the updated four-dimensional query vector is returned to the decoder through the query vector propagation module for processing the next frame of samples, specifically including: During the training process of the four-dimensional panoramic perception model, query vectors matching countable objects in the updated new query vectors are converted into tracking query vectors, and query vectors that are not matched for multiple consecutive frames in the updated tracking query vectors are removed.

5. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 4, characterized in that: During the testing process of the four-dimensional panoramic perception model, the predicted classification scores are used to determine newly detected objects and disappeared tracked objects, wherein, in the updated new query vectors, query vectors whose classification scores are higher than a first preset score threshold and match countable objects are retained, and in the updated tracking query vectors, query vectors whose classification scores are lower than a second preset score threshold for multiple consecutive frames are removed.

6. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 1, characterized in that: The interaction between the three-dimensional volume feature and the four-dimensional query vector is achieved through a volume cross-attention mechanism, which specifically includes: Each new query uses a multi-layer perceptron layer to predict the corresponding 3D reference point; Each tracking query maintains a corresponding reference point in time; Sampling three-dimensional volume features around each reference point; Perform weighted summation on the sampled features and output the interaction results.

7. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 6, characterized in that: The volumetric criss-cross attention mechanism includes a volumetric criss-cross attention mechanism based on deformable attention.

8. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 1, characterized in that: During the training process, the four-dimensional panoramic perception model based on visual information adopts a total loss function for supervised training. The total loss function is the sum of a mask classification loss function and a positioning perception loss function. The mask classification loss is used to evaluate the quality of the predicted mask, and the positioning perception loss is used to evaluate the quality of the predicted query position.

9. The four-dimensional occupancy grid panoramic perception method based on visual information according to claim 8, characterized in that: The expression of the positioning perception loss function is: In the formula, represents the positioning perception loss, P ′ is the center point of each object in the prediction result, is the center point of the corresponding object in the true value, and L1 represents the Manhattan distance.

10. A four-dimensional occupancy grid panoramic perception device based on visual information, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images