A space occupation grid map generation method and device, a storage medium and an equipment

Abstract

The application discloses a space occupation grid map generation method and device, a storage medium and equipment, and the method comprises the steps of acquiring multi-view video data collected by a preset number of shooting devices, and acceleration data and angular velocity data collected by an inertial measurement unit; jointly analyzing the video data, the acceleration data and the angular velocity data to obtain the six-degree-of-freedom pose of each frame of image in the video data under the world coordinate system; inputting the each frame of image and the corresponding six-degree-of-freedom pose into a two-stage self-supervised occupation prediction model to output a space occupation grid map. The application realizes accurate occupation prediction of visible areas and occluded areas without a large amount of training data.

Description

Technical Field

[0001] This invention relates to the field of computer vision and environmental perception technology, and in particular to a method, apparatus, storage medium and device for generating spatial occupancy grid maps. Background Technology

[0002] With the increasing demand for drones in complex environments such as indoor space inspection, disaster relief, and dense jungle flight, achieving real-time and stable perception and map building in such environments has become a core issue for drone autonomous navigation and obstacle avoidance. These complex environments present numerous challenges, including compact and irregular structures, numerous obstacles, and variable lighting conditions.

[0003] During autonomous flight, unmanned aerial vehicles (UAVs) need real-time environmental awareness to support functions such as path planning and obstacle avoidance. Their perception target is typically a two-dimensional or three-dimensional occupancy grid map (OGM) of the generated scene. Occupancy grid maps are a widely used map representation method in robotics. The basic idea is to divide the environment into several grid cells, each containing only two states: occupied or vacant. Through occupancy grid maps, UAVs can intuitively understand the surrounding spatial structure, providing a foundation for subsequent navigation and decision-making.

[0004] Currently, there are two main approaches to constructing occupancy grid maps. One is a local mapping approach based on Simultaneous Localization and Mapping (SLAM). This approach can generate maps directly during runtime, but the occupancy results obtained by SLAM are limited to the visible surface areas of the environment and cannot reasonably predict occupancy areas. The other is an occupancy prediction approach based on neural networks. This approach trains a grid occupancy prediction model using a large amount of data. Then, during the model inference phase, the model inputs sensor data to directly predict the occupancy status of each voxel or grid in the environment. However, this approach requires the pre-construction of a large amount of environmental occupancy maps as training data, resulting in high annotation costs.

[0005] In summary, while existing camera-based SLAM local mapping methods do not require a large amount of training data, the results are sparse and cannot predict occluded areas; while neural network-based methods can predict occluded areas, they require a large amount of training label data. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, storage medium, and device for generating spatial occupancy grid maps, which can effectively predict the occupancy of visible and occupied areas without requiring a large amount of training data.

[0007] To achieve the above objectives, a first aspect of the present invention provides a method for generating a spatially occupied raster map, the method comprising: Acquire a preset number of multi-view video data collected by shooting devices, and acceleration and angular velocity data collected by inertial measurement units; By jointly analyzing the video data, the acceleration data, and the angular velocity data, the six-degree-of-freedom pose of each frame in the video data in the world coordinate system is obtained. Each frame of image and its corresponding six-degree-of-freedom pose are input into a two-stage self-supervised occupancy prediction model, which outputs a spatial occupancy grid map.

[0008] Furthermore, the six-degree-of-freedom pose of each frame in the video data in the world coordinate system is obtained through the following steps: Each frame of the video data is synchronized with the acceleration data and the angular velocity data in time to obtain a time-aligned image sequence, acceleration measurement value sequence, and angular velocity measurement value sequence. Feature tracking is performed on adjacent frames in the image sequence, and the visual relative pose between the adjacent frames is solved. The motion constraint terms between adjacent frame images are obtained by calculating the acceleration measurement sequence and the angular velocity measurement sequence. Using the visual relative pose and the motion constraint as constraints, the pose variables between adjacent frame images are jointly optimized to obtain the six-degree-of-freedom pose of each frame image in the image sequence in the world coordinate system.

[0009] Further, the step of performing feature tracking on adjacent frames in the image sequence and solving for the visual relative pose between adjacent frames includes: The optical flow tracing method is used to perform feature tracking on the adjacent frame images in the image sequence to obtain the pixel correspondence between the adjacent frame images, and the visual relative pose between the adjacent frame images is solved based on the pixel correspondence.

[0010] Further, the calculation of the acceleration measurement sequence and the angular velocity measurement sequence to obtain the motion constraint term between adjacent frame images includes: Pre-integration is performed on the acceleration measurement sequence and the angular velocity measurement sequence to obtain the inertial pre-integration between adjacent frame images, and the inertial pre-integration is the motion constraint term.

[0011] Furthermore, the spatially occupied raster map is output through the following steps: Image feature encoding is performed on the image sequence extracted from the video data to obtain the image encoding features of each frame image; Context features are extracted from the image coding features, and the discrete depth distribution corresponding to each pixel of each frame of the image is calculated based on the image coding features; The first-stage depth map of each frame of the image is obtained by weighted summation based on the discrete depth distribution and the input viewpoint. Based on the input viewpoint depth information in the first stage depth map, the discrete depth distribution and the context features are subjected to tensor outer product operation to construct the view frustum features corresponding to each frame image; Based on the six-degree-of-freedom pose, the view frustum features are sampled and transformed to obtain voxel features; The voxel features are input into a three-dimensional feature encoding network to extract three-dimensional structural information and output the density field of the scene; The density field is filtered using a set threshold to determine the occupancy status of each grid cell, and the spatial occupancy grid map is output.

[0012] Further, the step of extracting contextual features from the image coding features and calculating the discrete depth distribution corresponding to each pixel of each frame image based on the image coding features includes: The image encoding features are input into the feature extraction subnetwork to obtain the context features used to characterize the semantic and structural information of the scene. The image encoding features are input into the depth prediction subnetwork to obtain the discrete depth distribution corresponding to each pixel of each frame image; wherein, the discrete depth distribution represents the probability value of the pixel falling into each preset depth interval.

[0013] Further, the step of filtering the density field to determine the occupancy status of each grid cell and outputting the spatial occupancy grid map includes: Depth rendering is performed on the density field and the camera pose corresponding to the preset new viewpoint to obtain a second-stage depth map; wherein, the second-stage depth map contains information on the occlusion region of the input image; Based on the first-stage depth map, the second-stage depth map, and images of adjacent frames, self-supervised constraints are constructed, and the parameters of the two-stage self-supervised occupancy prediction model are updated according to the self-supervised constraints. The density field is filtered based on the updated two-stage self-supervised occupancy prediction model to obtain the occupancy determination results of each grid, and the occupancy determination results are converted into the spatial occupancy grid map. Output the space-occupying raster map.

[0014] To achieve the above objectives, a second aspect of the present invention also provides a spatial occupancy raster map generation apparatus for implementing the spatial occupancy raster map generation method described in any of the first aspects, the apparatus comprising: The data acquisition module is used to acquire multi-view video data collected by a preset number of shooting devices, and acceleration and angular velocity data collected by the inertial measurement unit; The data analysis module is used to perform joint analysis on the video data, the acceleration data, and the angular velocity data to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system. The map generation module is used to input each frame of image and its corresponding six-degree-of-freedom pose into a two-stage self-supervised occupancy prediction model and output a spatial occupancy grid map.

[0015] A third aspect of the present invention also provides a computer-readable storage medium comprising a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform a spatial occupancy grid map generation method as described in any of the first aspects.

[0016] A fourth aspect of the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a spatial occupancy grid map generation method as described in any of the first aspects above. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of a spatial occupancy grid map generation method provided in the first aspect of the present invention; Figure 2 This is a flowchart of a preferred embodiment of a spatial occupancy raster map generation method provided in the first aspect of the present invention, showing a self-supervised occupancy prediction algorithm. Figure 3 This is a schematic diagram of a three-camera fan-shaped arrangement device structure, which is a preferred embodiment of a spatial occupancy grid map generation method provided in the first aspect of the present invention. Figure 4 This is a structural block diagram of a preferred embodiment of a spatial occupancy grid map generation device provided in the second aspect of the present invention; Figure 5 This is a structural block diagram of a preferred embodiment of a terminal device provided in the fourth aspect of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that the data involved in this invention (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] In this embodiment of the invention, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplarily" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] In this invention description, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In this invention description, unless otherwise stated, "a plurality of" means two or more. In this invention description, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] The first aspect of this invention provides a method for generating a spatial occupancy grid map, see [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of a preferred embodiment of a spatial occupancy raster map generation method provided by the first aspect of the present invention. The method includes steps S1 to S3, as follows: Step S1: Acquire a preset number of multi-view video data collected by the shooting device, and acceleration and angular velocity data collected by the inertial measurement unit; Step S2: Perform joint analysis on the video data, the acceleration data, and the angular velocity data to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system; Step S3: Input each frame image and its corresponding six-degree-of-freedom pose into a two-stage self-supervised occupancy prediction model, and output a spatial occupancy grid map.

[0025] In one example, the capturing device can be multiple cameras, and the inertial measurement unit (IMU) can be an inertial measurement unit. To improve the accuracy of the joint analysis of multi-view video data and inertial measurement data, the predetermined number of capturing devices are calibrated to obtain the intrinsic parameters of each capturing device, and extrinsic parameters are calibrated to establish the relative pose relationship between the capturing devices. Simultaneously, the pose relationship between the capturing devices and the inertial measurement unit is calibrated. Further, the timestamps of the capturing devices and the inertial measurement unit are synchronized or aligned to obtain a temporally aligned image sequence and corresponding acceleration and angular velocity data.

[0026] In a specific example, applied to environmental perception and mapping scenarios of mobile platforms such as drones, in complex environments such as indoors, woodlands, or narrow city streets, the mobile platform collects video data from multiple perspectives by the preset number of shooting devices during its movement, while the inertial measurement unit outputs acceleration and angular velocity data in real time, thereby forming multi-view video data and inertial measurement data that can be used for joint analysis.

[0027] Furthermore, the pose estimation obtained through the fusion of visual and inertial information is as follows: First, feature tracking is performed on adjacent frames based on image information in the video data to obtain the pixel correspondence between adjacent frames, and the visual relative pose between adjacent frames is thus solved. Simultaneously, motion constraint information between adjacent frames is calculated based on the acceleration data and the angular velocity data. Subsequently, the visual relative pose and the motion constraint information are jointly optimized under a joint optimization framework to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system. The data output in this step is used as training data for the model.

[0028] Furthermore, each frame of the image and its corresponding six-DOF pose are used as input data for the two-stage self-supervised occupancy prediction model, outputting a spatial occupancy grid map. During the training phase, self-supervised constraints are constructed using temporal images and multi-view geometric consistency information to train the two-stage self-supervised occupancy prediction model, thus eliminating the need for manually labeled ground-value spatial occupancy data. During the inference phase, the two-stage self-supervised occupancy prediction model generates a spatial occupancy grid map based on each input frame of the image and its corresponding pose, which is used for subsequent path planning and obstacle avoidance by the aircraft.

[0029] In another preferred embodiment, the six-degree-of-freedom pose of each frame in the video data in the world coordinate system is obtained through the following steps: Each frame of the video data is synchronized with the acceleration data and the angular velocity data in time to obtain a time-aligned image sequence, acceleration measurement value sequence, and angular velocity measurement value sequence. Feature tracking is performed on adjacent frames in the image sequence, and the visual relative pose between the adjacent frames is solved. The motion constraint terms between adjacent frame images are obtained by calculating the acceleration measurement sequence and the angular velocity measurement sequence. Using the visual relative pose and the motion constraint as constraints, the pose variables between adjacent frame images are jointly optimized to obtain the six-degree-of-freedom pose of each frame image in the image sequence in the world coordinate system.

[0030] In another preferred embodiment, the step of performing feature tracking on adjacent frame images in the image sequence and solving the visual relative pose between the adjacent frame images includes: The optical flow tracing method is used to perform feature tracking on the adjacent frame images in the image sequence to obtain the pixel correspondence between the adjacent frame images, and the visual relative pose between the adjacent frame images is solved based on the pixel correspondence.

[0031] In yet another preferred embodiment, the step of calculating the motion constraint term between adjacent frame images by analyzing the acceleration measurement sequence and the angular velocity measurement sequence includes: Pre-integration is performed on the acceleration measurement sequence and the angular velocity measurement sequence to obtain the inertial pre-integration between adjacent frame images, and the inertial pre-integration is the motion constraint term.

[0032] In yet another preferred embodiment, the spatial occupancy grid map is output through the following steps: Image feature encoding is performed on the image sequence extracted from the video data to obtain the image encoding features of each frame image; Context features are extracted from the image coding features, and the discrete depth distribution corresponding to each pixel of each frame of the image is calculated based on the image coding features; The first-stage depth map of each frame of the image is obtained by weighted summation based on the discrete depth distribution and the input viewpoint. Based on the input viewpoint depth information in the first stage depth map, the discrete depth distribution and the context features are subjected to tensor outer product operation to construct the view frustum features corresponding to each frame image; Based on the six-degree-of-freedom pose, the view frustum features are sampled and transformed to obtain voxel features; The voxel features are input into a three-dimensional feature encoding network to extract three-dimensional structural information and output the density field of the scene; The density field is filtered using a set threshold to determine the occupancy status of each grid cell, and the spatial occupancy grid map is output.

[0033] In another preferred embodiment, the step of extracting context features from the image coding features and calculating the discrete depth distribution corresponding to each pixel of each frame of the image based on the image coding features includes: The image encoding features are input into the feature extraction subnetwork to obtain the context features used to characterize the semantic and structural information of the scene. The image encoding features are input into the depth prediction subnetwork to obtain the discrete depth distribution corresponding to each pixel of each frame image; wherein, the discrete depth distribution represents the probability value of the pixel falling into each preset depth interval.

[0034] In yet another preferred embodiment, filtering the density field to determine the occupancy status of each grid cell and outputting the spatial occupancy grid map includes: Depth rendering is performed on the density field and the camera pose corresponding to the preset new viewpoint to obtain a second-stage depth map; wherein, the second-stage depth map contains information on the occlusion region of the input image; Based on the first-stage depth map, the second-stage depth map, and images of adjacent frames, self-supervised constraints are constructed, and the parameters of the two-stage self-supervised occupancy prediction model are updated according to the self-supervised constraints. The density field is filtered based on the updated two-stage self-supervised occupancy prediction model to obtain the occupancy determination results of each grid, and the occupancy determination results are converted into the spatial occupancy grid map. Output the space-occupying raster map.

[0035] In one example, see Figure 2 This is a flowchart of a two-stage self-supervised occupancy prediction algorithm, a preferred embodiment of a spatial occupancy grid map generation method provided in the first aspect of the present invention. The two-stage self-supervised occupancy prediction model processes each input frame image and its corresponding six-degree-of-freedom pose in the following manner to output a spatial occupancy grid map.

[0036] First, each frame of the multi-view video data is input into the image feature encoding module to encode the original image into image coded features. These image coded features can be understood as a high-level representation of the original pixel information after feature extraction, used for subsequent context feature extraction and depth distribution prediction.

[0037] Subsequently, the image encoding features are input into the feature extraction subnetwork and the depth prediction subnetwork, respectively. The feature extraction subnetwork is used to extract context features from the image encoding features. The context features are used to characterize the semantic and structural information of the scene. The depth prediction subnetwork is used to predict a discrete depth distribution for each pixel on the two-dimensional image plane. The discrete depth distribution represents the probability value of the pixel falling into each preset depth interval.

[0038] In one specific implementation, the discrete depth distribution can be weighted and summed to obtain a first-stage depth map for each frame of the image under the input viewpoint. This first-stage depth map characterizes the depth estimation results under the input viewpoint and provides the foundation for the subsequent construction of the 3D spatial representation based on the input viewpoint depth information.

[0039] Furthermore, after obtaining the context features and the discrete depth distribution, based on the input viewpoint depth information represented by the first-stage depth map, a tensor outer product operation is performed between the discrete depth distribution and the context features to construct a three-dimensional feature volume. Since this three-dimensional feature volume unfolds along the camera's line of sight in different depth intervals and corresponds to a view frustum structure in physical space, it is defined as a view frustum feature. Each image acquired by the imaging device can generate a corresponding view frustum feature.

[0040] Next, based on the six-degree-of-freedom pose corresponding to each frame of the image, the view frustum features are sampled and transformed to map the view frustum features from different viewpoints to a unified world coordinate system or a unified Cartesian coordinate system, thereby obtaining voxel features that fuse multi-view information.

[0041] In one specific implementation, the sampling and coordinate transformation can employ trilinear interpolation to map the view frustum features onto a regular voxel mesh, facilitating subsequent 3D network processing. It should be noted that this invention does not limit the specific interpolation method or voxel partitioning method.

[0042] Through the above steps, the spatial representations corresponding to multiple imaging devices are aggregated into a unified coordinate system, thereby forming a voxel feature representation for three-dimensional structural reasoning.

[0043] Furthermore, the voxel features are input into a three-dimensional feature encoding network to extract the three-dimensional structural information of the scene and output the density field of the scene. It should be noted that the three-dimensional feature encoding network can be a three-dimensional convolutional neural network or other network structures capable of encoding three-dimensional voxel features; this invention does not limit this.

[0044] During the inference phase, the density field is filtered using a set threshold to determine the occupancy state of each grid cell, and the occupancy state is discretized into grid cells to output a spatial occupancy grid map. Threshold filtering transforms the density field from a continuous representation into discrete occupancy determination results; for example, voxels / grids with values ​​greater than the threshold are determined to be occupied, while voxels / grids with values ​​less than the threshold are determined to be unoccupied or idle.

[0045] During the training phase, to achieve inference capabilities regarding occluded regions, in addition to the aforementioned inference process, depth rendering can be further performed on the density field to obtain a second-stage depth map from a new perspective. The camera pose corresponding to this new perspective changes relative to the input perspective used to generate the first-stage depth map; for example, the camera pose at the trajectory position in front of the current input perspective can be used. This ensures that the second-stage depth map includes information about occluded regions within the input perspective. It should be noted that the depth rendering method can employ ray-based rendering (e.g., Ray-casting) or point / Gaussian-based rendering (e.g., Gaussian Splatting), and this invention does not limit the specific method used.

[0046] In one specific implementation, self-supervised constraints are constructed based on the first-stage depth map, the second-stage depth map, and adjacent frame images. The parameters of the two-stage self-supervised occupancy prediction model are then updated according to these self-supervised constraints. Specifically, for an input image It at any time t and its first-stage depth map Dt, the pose transformation relationship from time t to time t-1 can be obtained by combining the six-degree-of-freedom poses corresponding to time t and the adjacent time t-1. If the camera intrinsics are known or can be obtained through calibration, Dt is back-projected onto the camera coordinate system to obtain a three-dimensional point set. Based on the pose transformation relationship and the camera intrinsics, the three-dimensional point set is reprojected onto the adjacent frame image It-1 to sample the corresponding pixel values ​​in It-1, thereby synthesizing a reprojected image. The difference between the synthesized reprojected image and the real image It is then used as a self-supervised error term, for example, using brightness error, structural similarity error, or a combination thereof to construct self-supervised constraints.

[0047] Similarly, consistency constraints can be constructed for the second-stage depth map of the new perspective obtained by the density field rendering, so that the model can simultaneously satisfy the depth consistency between the input view and the new view during the training process, thereby driving the density field to learn the three-dimensional structure of the occluded region.

[0048] The trained occupancy prediction model is deployed on the drone's computing platform, for example, by converting the model into TensorRT format and deploying it on NVIDIA's OriginX computing platform. During flight, multiple cameras acquire environmental images in real time, and the model generates a dense occupancy grid map through online inference. The generated occupancy information is transmitted to the drone's path planning and control module for obstacle avoidance and autonomous navigation.

[0049] Through the self-supervised training described above, the two-stage self-supervised occupancy prediction model can learn to infer the three-dimensional spatial structure of a scene from multi-view images and their poses without the need for manual annotation of ground truth spatial occupancy data, and output a spatial occupancy raster map by filtering the density field threshold during the inference stage.

[0050] In yet another preferred embodiment, see Figure 3This is a schematic diagram of a preferred embodiment of a spatial occupancy grid map generation method provided in the first aspect of the present invention, showing a three-camera fan-shaped arrangement device. The device includes camera 1, camera 2, and camera 3, arranged in a fan shape on the UAV body to form a wide-angle field of view covering the forward direction of flight. Camera 1 and camera 3 face the left and right front respectively, while camera 2 faces forward, thus ensuring overlapping and complementary coverage of the multi-view fields of view in the forward region. By calibrating the extrinsic parameters of cameras 1 to 3, the relative pose relationship between the three cameras can be established, and the pose relationship between the shooting device and the inertial measurement unit (IMU) can be further calibrated to improve the accuracy of joint analysis of multi-view video data and inertial measurement data.

[0051] Specifically, the multi-view video data acquired by cameras 1 to 3, as well as the acceleration and angular velocity data acquired by the IMU, are all input into the computing unit, which then executes the spatial occupancy grid map generation method described in claim 1. On one hand, the multi-view video data, acceleration data, and angular velocity data are jointly analyzed to obtain the six-degree-of-freedom pose of each frame in the world coordinate system. On the other hand, each frame and its corresponding six-degree-of-freedom pose are input into a two-stage self-supervised occupancy prediction model, outputting a spatial occupancy grid map. Thus, the UAV platform can generate dense and more complete local spatial occupancy grid maps in complex scenarios such as indoor spaces and dense jungles, providing reliable environmental representation support for subsequent path planning and obstacle avoidance.

[0052] A second aspect of the present invention provides a spatial occupancy raster map generation apparatus for implementing the spatial occupancy raster map generation method described in any of the first aspects above. See also... Figure 4 The diagram shown is a structural block diagram of a preferred embodiment of a spatial occupancy grid map generation apparatus provided in the second aspect of the present invention. The apparatus includes: The data acquisition module 11 is used to acquire a preset number of multi-view video data collected by the shooting device, and acceleration data and angular velocity data collected by the inertial measurement unit; Data analysis module 12 is used to perform joint analysis on the video data, the acceleration data and the angular velocity data to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system; The map generation module 13 is used to input each frame image and its corresponding six-degree-of-freedom pose into a two-stage self-supervised occupancy prediction model and output a spatial occupancy grid map.

[0053] It should be noted that the apparatus provided in the second aspect of the present invention can realize all the processes of the spatial occupancy grid map generation method described in the first aspect above. The functions and technical effects of each module and unit in the apparatus are the same as those of the spatial occupancy grid map generation method described in the first aspect above, and will not be repeated here.

[0054] A third aspect of the present invention also provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a spatial occupancy grid map generation method as described in any of the first aspects above.

[0055] The fourth aspect of the present invention also provides a terminal device, see [link to documentation]. Figure 5 The diagram shown is a structural block diagram of a preferred embodiment of a terminal device provided in the fourth aspect of the present invention. The terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10. When the processor 10 executes the computer program, it implements a spatial occupancy grid map generation method as described in any of the above embodiments.

[0056] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0057] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor 10 may be any conventional processor. The processor 10 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0058] The memory 20 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory 20 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a smart media card (SMC), a secure digital card (SD), and a flash card, or other volatile solid-state storage devices.

[0059] It should be noted that the aforementioned terminal device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the above content is merely an example describing the structure of the terminal device and does not constitute a limitation on the structure of the aforementioned terminal device. The aforementioned terminal device may include more or fewer components than those described above, or combine certain components, or different components.

[0060] In summary, the spatial occupancy grid map generation method, apparatus, and storage medium provided by the embodiments of the present invention have at least the following beneficial effects: A two-stage self-supervised occupancy prediction model is adopted, utilizing multi-view and temporal consistency to construct self-supervised constraints. Model training can be completed without manually labeling ground-value spatial occupancy data, significantly reducing data collection and labeling costs. Furthermore, in the first stage, frustum features are constructed from the depth map and aggregated into voxel features. Then, a density field is output by a 3D feature encoding network. Combined with the self-supervised training mechanism of novel viewpoint rendering, the model has the ability to infer occluded regions, thereby generating a more complete local spatial occupancy raster map.

[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary hardware platforms, and of course, it can also be implemented entirely by hardware. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0062] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating a spatially occupied raster map, characterized in that, include: Acquire a preset number of multi-view video data collected by shooting devices, and acceleration and angular velocity data collected by inertial measurement units; By jointly analyzing the video data, the acceleration data, and the angular velocity data, the six-degree-of-freedom pose of each frame in the video data in the world coordinate system is obtained. Each frame of image and its corresponding six-degree-of-freedom pose are input into a two-stage self-supervised occupancy prediction model, which outputs a spatial occupancy grid map.

2. The spatial occupancy raster map generation method as described in claim 1, characterized in that, The following steps are used to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system: Each frame of the video data is synchronized with the acceleration data and the angular velocity data in time to obtain a time-aligned image sequence, acceleration measurement value sequence, and angular velocity measurement value sequence. Feature tracking is performed on adjacent frames in the image sequence, and the visual relative pose between the adjacent frames is solved. The motion constraint terms between adjacent frame images are obtained by calculating the acceleration measurement sequence and the angular velocity measurement sequence. Using the visual relative pose and the motion constraint as constraints, the pose variables between adjacent frame images are jointly optimized to obtain the six-degree-of-freedom pose of each frame image in the image sequence in the world coordinate system.

3. The spatial occupancy raster map generation method as described in claim 2, characterized in that, The step of performing feature tracking on adjacent frames in the image sequence and solving for the visual relative pose between adjacent frames includes: The optical flow tracing method is used to perform feature tracking on the adjacent frame images in the image sequence to obtain the pixel correspondence between the adjacent frame images, and the visual relative pose between the adjacent frame images is solved based on the pixel correspondence.

4. The spatial occupancy raster map generation method as described in claim 2, characterized in that, The calculation of the acceleration measurement sequence and the angular velocity measurement sequence to obtain the motion constraint terms between adjacent frame images includes: Pre-integration is performed on the acceleration measurement sequence and the angular velocity measurement sequence to obtain the inertial pre-integration between adjacent frame images, and the inertial pre-integration is the motion constraint term.

5. The spatial occupancy raster map generation method as described in claim 1, characterized in that, Output a spatially occupied raster map using the following steps: Image feature encoding is performed on the image sequence extracted from the video data to obtain the image encoding features of each frame image; Context features are extracted from the image coding features, and the discrete depth distribution corresponding to each pixel of each frame of the image is calculated based on the image coding features; The first-stage depth map of each frame of the image is obtained by weighted summation based on the discrete depth distribution and the input viewpoint. Based on the input viewpoint depth information in the first stage depth map, the discrete depth distribution and the context features are subjected to tensor outer product operation to construct the view frustum features corresponding to each frame image; Based on the six-degree-of-freedom pose, the view frustum features are sampled and transformed to obtain voxel features; The voxel features are input into a three-dimensional feature encoding network to extract three-dimensional structural information and output the density field of the scene; The density field is filtered using a set threshold to determine the occupancy status of each grid cell, and the spatial occupancy grid map is output.

6. The spatial occupancy raster map generation method as described in claim 5, characterized in that, The step of extracting context features from the image coding features and calculating the discrete depth distribution corresponding to each pixel of each frame of the image based on the image coding features includes: The image encoding features are input into the feature extraction subnetwork to obtain the context features used to characterize the semantic and structural information of the scene. The image encoding features are input into the depth prediction subnetwork to obtain the discrete depth distribution corresponding to each pixel of each frame image; wherein, the discrete depth distribution represents the probability value of the pixel falling into each preset depth interval.

7. The spatial occupancy raster map generation method as described in claim 5, characterized in that, The step of filtering the density field to determine the occupancy status of each grid cell and outputting the spatial occupancy grid map includes: Depth rendering is performed on the density field and the camera pose corresponding to the preset new viewpoint to obtain a second-stage depth map; wherein, the second-stage depth map contains information on the occlusion region of the input image; Based on the first-stage depth map, the second-stage depth map, and images of adjacent frames, self-supervised constraints are constructed, and the parameters of the two-stage self-supervised occupancy prediction model are updated according to the self-supervised constraints. The density field is filtered based on the updated two-stage self-supervised occupancy prediction model to obtain the occupancy determination results of each grid, and the occupancy determination results are converted into the spatial occupancy grid map. Output the space-occupying raster map.

8. A spatial occupancy raster map generation apparatus, used to implement the spatial occupancy raster map generation method as described in any one of claims 1 to 7, the apparatus comprising: The data acquisition module is used to acquire multi-view video data collected by a preset number of shooting devices, and acceleration and angular velocity data collected by the inertial measurement unit; The data analysis module is used to perform joint analysis on the video data, the acceleration data, and the angular velocity data to obtain the six-degree-of-freedom pose of each frame in the video data in the world coordinate system. The map generation module is used to input each frame of image and its corresponding six-degree-of-freedom pose into a two-stage self-supervised occupancy prediction model and output a spatial occupancy grid map.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform a spatial occupancy grid map generation method as described in any one of claims 1 to 7.

10. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a spatial occupancy raster map generation method as described in any one of claims 1 to 7.

Images