Model training method, indoor scene occupancy prediction method, device and medium

Abstract

The application provides a model training method, an indoor scene occupancy prediction method, equipment and a medium. The model training method comprises the following steps: acquiring an indoor scene data set; performing occupancy prediction on each image frame according to a spatial semantic set to obtain actual spatial occupancy information of each image frame; performing prediction on an initial occupancy world model according to the actual spatial occupancy information of a current image frame, the actual spatial occupancy information of a historical image frame, and a camera pose corresponding to a next image frame of the current image frame to obtain predicted spatial occupancy information of the next image frame; and training the initial occupancy world model according to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame to obtain a target occupancy world model. With the model, the next spatial occupancy information can be predicted according to the current spatial occupancy information, the historical spatial occupancy information, and the next camera pose, and the robot can perceive the occupancy of the indoor scene in the next state.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a model training method, an indoor scene occupancy prediction method, equipment and a medium. BACKGROUND

[0002] With the rapid development of robots, robots are enabled to perceive in unknown environments and make corresponding decisions in advance, which has attracted more and more attention.

[0003] In the related art, an occupancy world model is usually used to describe the occupancy in the visual field space of a robot, wherein the occupancy world model is a model used to simulate and predict the evolution of the surrounding scene, which divides a 3D space into individual voxels and assigns a label to each voxel to represent whether it is occupied and by what object.

[0004] However, the current occupancy world model mainly focuses on outdoor scenes and lacks prediction of the occupancy of robots in indoor scenes. SUMMARY

[0005] Therefore, the embodiments of the present application provide a model training method, an indoor scene occupancy prediction method, equipment and a medium to use a target occupancy world model to predict the next spatial occupancy information of a robot according to current spatial occupancy information, historical spatial occupancy information and the next camera pose, so as to facilitate the robot to perceive the occupancy of the indoor scene in the next state.

[0006] In a first aspect, the embodiments of the present application provide a model training method, comprising:

[0007] Obtain an indoor scene dataset, the indoor scene dataset comprising: sample data of at least one indoor scene, wherein the sample data of each indoor scene comprises: a sample image sequence of the each indoor scene and a spatial semantic set, each image frame in the sample image sequence has a corresponding camera pose, and the spatial semantic set comprises: semantic categories of object objects in the each indoor scene;

[0008] According to the spatial semantic set, occupancy prediction is performed on the image frames to obtain actual spatial occupancy information of the image frames, wherein the actual spatial occupancy information of the image frames comprises actual occupancy information of each voxel grid in a corresponding three-dimensional visual field space;

[0009] According to the actual spatial occupancy information of the current image frame of the each indoor scene, the actual spatial occupancy information of the historical image frames before the current image frame and the camera pose corresponding to the next image frame of the current image frame, an initial occupancy world model is used for prediction to obtain the predicted spatial occupancy information of the next image frame.

[0010] According to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame, the initial occupancy world model is trained to obtain a target occupancy world model.

[0011] In an optional implementation, the occupancy prediction of the image frames according to the spatial semantic set comprises:

[0012] According to the spatial semantic set, camera calibration parameters corresponding to the sample image sequence, and a target voxel resolution, a preset monocular occupancy prediction model is used to perform occupancy prediction on the image frames to obtain the actual spatial occupancy information of the image frames.

[0013] In an optional implementation, the initial occupancy world model comprises an initial encoder, an initial intermediate layer, and an initial decoder; and the prediction of the next image frame according to the actual spatial occupancy information of the current image frame of each indoor scene, the actual spatial occupancy information of historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame comprises:

[0014] The actual spatial occupancy information of the image frames is encoded by using the initial encoder to obtain initial scene occupancy features of the image frames.

[0015] The initial scene occupancy features of the image frames are aggregated under a plurality of preset space-time windows by using the initial intermediate layer to obtain scene occupancy aggregation features under the plurality of preset space-time windows.

[0016] According to the camera pose corresponding to the next image frame and the scene occupancy aggregation features under the plurality of preset space-time windows, scene occupancy features of the next image frame are obtained.

[0017] The scene occupancy features of the next image frame are decoded by using the initial decoder to obtain the predicted spatial occupancy information of the next image frame.

[0018] In an optional implementation, the initial intermediate layer comprises a space-time aggregation module, and the aggregation of the initial scene occupancy features of the image frames under a plurality of preset space-time windows by using the initial intermediate layer comprises:

[0019] According to the plurality of preset space-time windows, to-be-aggregated features of the plurality of preset space-time windows are determined from the initial scene occupancy features of the image frames.

[0020] The spatio-temporal aggregation module is used to aggregate the scene occupancy aggregated features in the plurality of preset spatio-temporal windows to obtain target occupancy aggregated features.

[0021] In an optional implementation, the initial intermediate layer includes an autoregressive converter configured to obtain the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the scene occupancy aggregated features in the plurality of preset spatio-temporal windows, and the obtaining the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the scene occupancy aggregated features in the plurality of preset spatio-temporal windows includes:

[0022] The spatio-temporal aggregation module is used to aggregate the scene occupancy aggregated features in the plurality of preset spatio-temporal windows to obtain target occupancy aggregated features.

[0023] The autoregressive converter is used to obtain the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the target occupancy aggregated features.

[0024] In an optional implementation, the autoregressive converter includes an attention branch and a feedforward branch, and the autoregressive converter is used to obtain the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the target occupancy aggregated features, and the obtaining the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the target occupancy aggregated features includes:

[0025] The camera pose is encoded to obtain a pose feature.

[0026] The attention branch is used to obtain an attention occupancy aggregated feature corresponding to the target occupancy aggregated feature based on the pose feature and the target occupancy aggregated feature.

[0027] The feedforward branch is used to obtain the scene occupancy features of the next image frame based on the pose feature, the attention occupancy aggregated feature.

[0028] In an optional implementation, the sample data further includes actual scene occupancy features of the image frames, and the obtaining the target occupancy world model by training the initial occupancy world model based on the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame includes:

[0029] The initial encoder is trained based on the initial scene occupancy features of the image frames and the actual scene occupancy features of the image frames to obtain a target encoder.

[0030] The initial intermediate layer and the initial decoder are trained based on the predicted spatial occupancy information and the actual spatial occupancy information to obtain a target intermediate layer and a target decoder, and the target occupancy world model includes the target encoder, the target intermediate layer, and the target decoder.

[0031] In a second aspect, the embodiments of the present application further provide an indoor scene occupancy prediction method, comprising:

[0032] obtaining a current indoor image frame, a historical indoor image frame and a next camera pose collected by a camera of a target robot in a target indoor scene;

[0033] performing occupancy prediction on the current indoor image frame and the historical indoor image frame respectively according to a spatial semantic set of the target indoor scene, to obtain current spatial occupancy information and historical spatial occupancy information, wherein the current spatial occupancy information comprises occupancy information of each voxel grid in a current field of view space of the target robot, and the historical spatial occupancy information comprises occupancy information of each voxel grid in a historical field of view space of the target robot;

[0034] adopting a target occupancy world model to predict next spatial occupancy information according to the current spatial occupancy information, the historical spatial occupancy information and the next camera pose, wherein the next spatial occupancy information comprises occupancy information of each voxel grid in a next field of view space of the target robot, and the target occupancy world model is a model trained by the method of any one of the first aspect.

[0035] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine readable instructions to execute the method of any one of the first aspect.

[0036] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to execute the method of any one of the first aspect.

[0037] The application provides a model training method, an indoor scene occupancy prediction method, equipment and a medium. The model training method comprises the following steps: acquiring an indoor scene data set; performing occupancy prediction on each image frame according to a spatial semantic set to obtain actual spatial occupancy information of each image frame; performing prediction on an initial occupancy world model according to the actual spatial occupancy information of a current image frame, the actual spatial occupancy information of a historical image frame and the camera pose corresponding to a next image frame of the current image frame to obtain predicted spatial occupancy information of the next image frame; and training the initial occupancy world model according to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame to obtain a target occupancy world model. The model is used to predict next spatial occupancy information according to current spatial occupancy information, historical spatial occupancy information and a next camera pose, which facilitates a robot to perceive the occupancy of an indoor scene in a next state. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0039] Figure 1 Flowchart of the occupancy world model training method provided by the embodiment of the application Figure 1

[0040] Figure 2 Flowchart of the occupancy world model training method provided by the embodiment of the application Figure 2

[0041] Figure 3 Flowchart of the occupancy world model training method provided by the embodiment of the application Figure 3

[0042] Figure 4 Flowchart of the occupancy world model training method provided by the embodiment of the application Figure 4

[0043] Figure 5 Flowchart of the occupancy world model training method provided by the embodiment of the application Figure 5

[0044] Figure 6 Flowchart of the indoor scene occupancy prediction method provided by the embodiment of the application

[0045] Figure 7 Structural schematic diagram of the occupancy world model training device provided by the embodiment of the application ​​​​​

[0046] Figure 8 A structural schematic diagram of an indoor scene occupancy prediction device provided by an embodiment of the present application is shown in the figure.

[0047] Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0049] In view of the great increase in the diversity of camera positions and directions of robots in indoor scenes, there is a lack of spatiotemporal regularity between different observed image frames, resulting in a lack of prediction of the occupancy of robots in indoor scenes. Based on this, the actual spatial occupancy information of the image frames observed by vision and the camera pose corresponding to the image frames are used to train a target occupancy world model, so that the target occupancy world model can understand the spatiotemporal connection. Then, the target occupancy world model is used to predict the next spatial occupancy information of the robot by using the current spatial occupancy information of the current indoor image frame observed by vision, the historical spatial occupancy information of the historical indoor image frame and the potential next camera pose, thereby facilitating the robot to perceive the occupancy of the indoor scene in the next state.

[0050] Figure 1 A flowchart of an occupancy world model training method provided by an embodiment of the present application is shown in the figure. Figure 1 The execution subject of the present embodiment can be an electronic device, such as a notebook computer, a desktop computer, etc.

[0051] As shown in the figure, the method can include: Figure 1

[0052] S101, acquiring an indoor scene data set.

[0053] ​The indoor scene dataset comprises sample data of at least one indoor scene, and the sample data of each indoor scene comprises a sample image sequence of each indoor scene and a spatial semantic set, each image frame in the sample image sequence has a corresponding camera pose, and the spatial semantic set comprises a semantic category of each object in each indoor scene.

[0054] The indoor scene may be, for example, a scene comprising a ceiling, a floor, a wall, a table, a sofa and the like, and different indoor scenes comprise different object.

[0055] The sample image sequence of each indoor scene is a continuous image sequence obtained by capturing each indoor scene by a camera, and the sample image sequence comprises a plurality of continuous image frames. Each image frame in the sample image sequence has a corresponding camera pose, which is the pose of the camera when capturing each image frame. The camera pose may comprise, for example, a camera position and a camera direction.

[0056] The semantic category of each object in the indoor scene refers to the category of each object obtained by classifying objects according to semantic information (such as function, use, form and the like) of each object in the indoor scene. The semantic category may comprise, for example, a ceiling, a floor, a wall, a table, a sofa and the like.

[0057] In S102, actual spatial occupancy information of each image frame is obtained by performing occupancy prediction on each image frame according to the spatial semantic set.

[0058] The actual spatial occupancy information of each image frame comprises actual occupancy information of each voxel grid in a corresponding three-dimensional field of view space.

[0059] The actual spatial occupancy information of each voxel grid in the corresponding three-dimensional field of view space of each image frame is obtained by performing occupancy prediction on each image frame according to the spatial semantic set. The corresponding three-dimensional field of view space of each image frame is a field of view space captured by the camera for each image frame, that is, the camera captures the corresponding three-dimensional field of view space in the indoor scene to obtain each image frame.

[0060] The corresponding three-dimensional field of view space is divided into a plurality of voxel grids, and the actual occupancy information of each voxel grid comprises actual occupancy indication information for indicating whether each voxel grid is occupied, and an actual semantic category of an occupied object when each voxel grid is occupied. The occupied object is an object occupying each voxel grid, and the actual semantic category of the occupied object belongs to a subset of the spatial semantic set.

[0061] S103, predicting according to the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frame before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, to obtain the predicted space occupancy information of the next image frame by using the initial occupancy world model.

[0062] wherein the current image frame of each indoor scene is a plurality of image frames randomly selected from a sample image sequence of each indoor scene, the historical image frame before the current image frame is at least one image frame before the current image frame in the sample image sequence, the at least one image frame includes the last image frame of the current image frame, the next image frame of the current image frame is a next image frame of the current image frame in the sample image sequence, and the camera pose corresponding to the next image frame is the pose of the camera when the next image frame is captured.

[0063] initializing to obtain the initial occupancy world model, taking the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frame, and the camera pose corresponding to the next image frame as inputs of the initial occupancy world model, predicting by using the initial occupancy world model to obtain the predicted space occupancy information of the next image frame, wherein the predicted space occupancy information includes: predicted occupancy information of each voxel grid in the three-dimensional field of view space corresponding to the next image frame, the predicted occupancy information includes: predicted occupancy indication information for indicating whether each voxel grid in the three-dimensional field of view space corresponding to the next image frame is occupied, and a predicted semantic category of an occupancy object when each voxel grid is occupied. Wherein the predicted occupancy indication information is predicted occupancy indication information, and the predicted semantic category is a predicted semantic category, which may include, for example, a ceiling, a floor, a wall, a table, a sofa, etc.

[0064] S104, training the initial occupancy world model according to the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model.

[0065] According to the predicted space occupancy information of the next image frame and the actual space occupancy information of the next image frame, the model loss is calculated, and the initial occupancy world model is trained according to the model loss until the model loss does not exceed the preset loss threshold. The occupancy world model obtained when the model loss does not exceed the preset loss threshold is taken as the target occupancy world model.

[0066] Wherein the specific selection of the preset loss threshold can be selected according to actual selection, which is not particularly limited in this embodiment.

[0067] In some embodiments, the occupancy loss is calculated according to the actual occupancy information and the predicted occupancy information of the next image frame, the category loss is calculated according to the actual semantic category and the predicted semantic category of the next image frame, and then the model loss is calculated according to the occupancy loss and the category loss, wherein the occupancy loss and the category loss can be weighted to obtain the model loss.

[0068] In the occupancy world model training method provided in the embodiment, the target occupancy world model is trained by using the actual spatial occupancy information of the image frame observed by vision and the camera pose corresponding to the image frame, so that the target occupancy world model can understand the space-time relationship.

[0069] In an optional implementation, the step S102 of predicting the occupancy of each image frame according to the spatial semantic set to obtain the actual spatial occupancy information of each image frame can include:

[0070] The occupancy of each image frame is predicted by using a preset monocular occupancy prediction model according to the spatial semantic set, the camera calibration parameters corresponding to the sample image sequence, and the target voxel resolution to obtain the actual spatial occupancy information of each image frame.

[0071] The preset monocular occupancy prediction model can be, for example, an EmbodiedOcc model or a RoboOCC model.

[0072] The camera calibration parameters corresponding to the sample image sequence are calibration parameters of the camera when the sample image sequence is collected, including camera extrinsic parameters and camera intrinsic parameters, and the camera extrinsic parameters and the camera intrinsic parameters can be represented by a camera extrinsic parameter matrix and a camera intrinsic parameter matrix.

[0073] The target voxel resolution is the resolution of each voxel grid in the three-dimensional field of view space corresponding to each image frame, which is used to determine the size of each voxel grid and is represented as {X, Y, Z}.

[0074] The spatial semantic set, the camera calibration parameters, and the target voxel resolution are used as the input of the monocular occupancy prediction model, the occupancy of each image frame is predicted by using the monocular occupancy prediction model, and the actual spatial occupancy information of each image frame is obtained.

[0075] Figure 2 Flowchart of the occupancy world model training method provided in the embodiment Figure 2 For example, Figure 3As shown, in an optional embodiment, the initial occupancy world model comprises an initial encoder, an initial intermediate layer and an initial decoder, and step S103 comprises: using the initial occupancy world model to predict the predicted spatial occupancy information of the next image frame according to the actual spatial occupancy information of the current image frame of each indoor scene, the actual spatial occupancy information of the historical image frames before the current image frame and the camera pose corresponding to the next image frame of the current image frame, which can comprise:

[0076] S201, using the initial encoder to encode the actual spatial occupancy information of each image frame to obtain the initial scene occupancy features of each image frame.

[0077] The initial encoder can be a VQ-VAE encoder, and the actual spatial occupancy information of each image frame is encoded using the initial encoder to obtain the initial scene occupancy features of each image frame.

[0078] S202, using the initial intermediate layer to aggregate the initial scene occupancy features of each image frame under a plurality of preset spatio-temporal windows to obtain scene occupancy aggregation features under the plurality of preset spatio-temporal windows.

[0079] The preset spatio-temporal window refers to a window that combines time and space information for data processing when performing spatio-temporal data analysis. The preset spatio-temporal window includes a window defined in two dimensions of time and space.

[0080] The scales defined by the plurality of preset spatio-temporal windows in the time dimension and the space dimension can be different, and the plurality of preset spatio-temporal windows can include a long-term spatio-temporal window (L) and a short-term spatio-temporal window (S), wherein the time scale corresponding to the long-term spatio-temporal window is longer, the time scale corresponding to the short-term spatio-temporal window is shorter, and the space scales corresponding to the long-term spatio-temporal window and the short-term spatio-temporal window can be the same or different, which can be set according to actual needs, and this embodiment does not make special limitations.

[0081] It should be noted that the long-term spatio-temporal window (L) is used to capture the global consistency of the time series, and the short-term spatio-temporal window (S) is used to capture fine-grained local spatio-temporal information. The number of long-term spatio-temporal windows (L) includes but is not limited to one, the time scales corresponding to different long-term spatio-temporal windows (L) are different, and the space scales corresponding to different long-term spatio-temporal windows (L) can be the same or different, the number of short-term spatio-temporal windows (S) includes but is not limited to one, for example, 2 short-term spatio-temporal windows, the time scales corresponding to different short-term spatio-temporal windows (S) are different, and the space scales corresponding to different short-term spatio-temporal windows (S) can be the same or different, which can be set according to actual needs, and this embodiment does not make special limitations.

[0082] The time scale is used to indicate the number of image frames, the larger the time scale, the more the number of image frames, the smaller the time scale, the less the number of image frames, and the initial scene occupancy feature of the image frame is visualized as a bird's eye view. The spatial scale is used to indicate a spatial region in the bird's eye view of the image frame. The feature of the spatial region is determined from the initial scene occupancy feature as the to-be-aggregated feature of the image frame.

[0083] The initial intermediate layer is adopted to determine, under each preset space-time window, a to-be-processed image frame corresponding to a time scale of each preset space-time window from each image frame, and determine a feature of a spatial scale corresponding to each preset space-time window from the initial scene occupancy feature of the to-be-processed image frame as a to-be-aggregated feature of each preset space-time window, wherein the to-be-aggregated feature is a feature of a spatial region indicated by the spatial scale corresponding to each preset space-time window in the bird's eye view of the to-be-processed image frame.

[0084] Then, the to-be-aggregated features of each preset space-time window are aggregated to obtain scene occupancy aggregated features under each preset space-time window.

[0085] It should be noted that by defining a plurality of preset space-time windows, a multi-scale space-time window is adopted to obtain an aggregated space-time receptive field to capture dynamic changes in the scene, so that the evolution of the scene occupancy can be better understood and predicted.

[0086] S203, according to the camera pose corresponding to the next image frame and the scene occupancy aggregated features under the plurality of preset space-time windows, obtaining the scene occupancy feature of the next image frame.

[0087] The initial intermediate layer further comprises: an autoregressive converter, which adopts the space-time aggregation module to aggregate the scene occupancy aggregated features under the plurality of preset space-time windows to obtain target occupancy aggregated features, and then adopts the autoregressive converter to obtain the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregated features.

[0088] The camera pose corresponding to the next image frame and the target occupancy aggregated features are used as inputs of the autoregressive converter, and the autoregressive converter outputs the scene occupancy feature of the next image frame.

[0089] It should be noted that due to the large increase in camera position diversity in indoor scenes, the inherent spatiotemporal regularity of history is destroyed. Based on this, a conditional causal attention (CCSA) mechanism is adopted, and the camera pose corresponding to the next image frame is used as a condition to guide the autoregressive converter to make reasonable occupancy prediction, thereby improving the prediction accuracy.

[0090] In some embodiments, the camera pose corresponding to the next image frame, the target occupancy aggregated feature, and the historical occupancy aggregated feature are taken as inputs of the autoregressive transformer, and the output is the scene occupancy feature of the next image frame. The historical occupancy aggregated feature is obtained by using the initial intermediate layer, aggregating the initial scene occupancy features of the image frames under a plurality of historical spatio-temporal windows, and then aggregating the scene occupancy aggregated features under the plurality of historical spatio-temporal windows. For the implementation process of the historical occupancy aggregated feature, refer to the implementation process of the target occupancy aggregated feature described above, which will not be repeated here.

[0091] S204, using the initial decoder to decode the scene occupancy feature of the next image frame to obtain the predicted spatial occupancy information of the next image frame.

[0092] The initial decoder can be a VQ-VAE decoder. The initial decoder is used to decode the scene occupancy feature of the next image frame to obtain the predicted spatial occupancy information of the next image frame.

[0093] Figure 3 Flowchart of the occupancy world model training method provided by the embodiments of the present application Figure 4 As shown in Figure 4 In an optional embodiment, the initial intermediate layer includes a spatio-temporal aggregation module, which can be a hybrid spatio-temporal aggregation (HSTA) module.

[0094] The step S202 of aggregating the initial scene occupancy features of the image frames under a plurality of preset spatio-temporal windows to obtain the scene occupancy aggregated features under the plurality of preset spatio-temporal windows can include:

[0095] S301, determining the to-be-aggregated features of each preset spatio-temporal window from the initial scene occupancy features of the image frames according to each preset spatio-temporal window.

[0096] According to the time scale corresponding to each preset spatio-temporal window, the to-be-processed image frames of each preset spatio-temporal window under the corresponding time scale are determined from the image frames, the to-be-processed image frames are visualized as bird's eye view images, and then according to the spatial scale corresponding to each preset spatio-temporal window, the features of the corresponding spatial scale are determined from the bird's eye view images of the to-be-processed image frames as the to-be-aggregated features of each preset spatio-temporal window.

[0097] S302, using the spatio-temporal aggregation module to aggregate the to-be-aggregated features of each preset spatio-temporal window to obtain the scene occupancy aggregated features under each preset spatio-temporal window.

[0098] The space-time aggregation module is used to aggregate the to-be-aggregated features of each preset space-time window to obtain scene occupancy aggregated features under each preset space-time window.

[0099] In some embodiments, the space-time aggregation model of the long-term space-time window (L) includes 2 layers of causal convolution, normalizes the to-be-aggregated features extracted from the bird's eye view, stabilizes the feature distribution, accelerates the training, and then is processed by the first layer of causal 3D convolution (Causal3DConv1) to ensure that the calculation of the current time step only uses information at past or current time, avoids future information leakage, is suitable for time series prediction, and then the features obtained after processing are subjected to ReLU activation for nonlinear transformation to enhance the model expression capability. The second layer of causal convolution (Causal3DConv2) is used to perform causal convolution on the features output by the ReLU activation in the time dimension, further refine the long-term features, then add the to-be-aggregated features and the output of the second layer of causal convolution to retain the initial information (to-be-aggregated features) and alleviate the gradient vanishing problem of the deep network, and then perform nonlinear transformation on the addition result through ReLU activation.

[0100] The specific implementation process of the space-time aggregation model of the short-term space-time window (S) is similar to that of the space-time aggregation model of the long-term space-time window (L) described above, and will not be repeated here.

[0101] In this embodiment, by defining a plurality of preset space-time windows, an aggregated space-time receptive field is obtained based on multi-scale space-time windows to capture dynamic changes in the scene, and by using a space-time aggregation module to aggregate features, scene occupancy aggregated features under each preset space-time window are obtained, so that the evolution of scene occupancy can be better understood and predicted.

[0102] Figure 4 Flowchart of the occupancy world model training method provided by the embodiment of the present application γ As shown in OI In an optional implementation, the autoregressive transformer includes: a feedforward branch, which uses the autoregressive transformer to obtain scene occupancy features of a next image frame according to a camera pose corresponding to the next image frame and the target occupancy aggregated features, and can include:

[0103] S401, encode the camera pose to obtain a pose feature.

[0104] The camera pose is encoded by a linear encoding layer to obtain the pose feature.

[0105] S402, using an attention branch, obtaining attention occupancy aggregated features corresponding to the target occupancy aggregated features according to the pose feature and the target occupancy aggregated features.

[0106] The attention branch is based on a causal state attention (CSA) mechanism, which is an inter-frame cross-attention operation with causal attention masks, ensuring that in a time series, the features of the current moment can only pay attention to the historical information of the past or current moment, avoiding future information leakage, and conforming to the causal logic of time series prediction.

[0107] In some embodiments, an attention branch is adopted to obtain attention occupancy aggregated features according to the pose features, target occupancy aggregated features and historical occupancy aggregated features.

[0108] A multi-layer perceptron (MLP) is adopted to generate adjustment parameters under the attention branch according to the pose features, including a first residual scaling factor α 1. First layer normalization scaling factor β 1. First layer normalization shift factor OI 1.

[0109] Then, the target occupancy aggregated features are taken as current input features (i.e. query), the historical occupancy aggregated features are taken as historical marker features (as key and value respectively), and the cross-attention mechanism of the attention branch is used to update the current input features by using the historical marker features, so as to obtain the attention occupancy aggregated features, which are represented as follows:

[0110] OH 1=( α 1×CSA( OH , OI , OI ))+ OI

[0111] Wherein, OH 1 is the attention occupancy aggregated features, OI is the target occupancy aggregated features, OI is the historical occupancy aggregated features.

[0112] In some embodiments, the attention occupancy aggregated features can also be subjected to layer normalization and parameter adjustment, which are represented as follows:

[0113] γ 12=Norm( OI 1)×(1+ β 1)+ OI 1

[0114] Wherein, OI 12 is the adjusted attention occupancy aggregated features, Norm( γ 1) represents layer normalization processing on γ 1 to stabilize the feature distribution, and (1+β 1) and OI 1 respectively scale and shift the normalized features, so that the model can adaptively adjust the feature representation according to the current camera view.

[0115] S403, adopt the feed-forward branch to aggregate the features according to the pose features and the attention occupancy, to obtain the scene occupancy of the next image frame.

[0116] The feed-forward branch includes a feed-forward network (FFN) for performing nonlinear transformation on the attention occupancy aggregated features output by the attention branch, to capture more complex features and enhance the expression ability of the model on fine-grained spatiotemporal clues. The FFN can be composed of 2 linear layers.

[0117] A multi-layer perceptron (MLP) is adopted to generate adjustment parameters under the attention branch according to the pose features, including the second residual scaling factor α 2, the second layer normalization scaling factor β 2, the second layer normalization shift factor OI 2.

[0118] According to the pose features and the attention occupancy aggregated features, the feed-forward branch is adopted to obtain the scene occupancy of the next image frame, which is represented as follows:

[0119] OI 2=FFN( α 2)•(1+ OI 2)+ OI 1

[0120] Wherein, OI 2 is the scene occupancy of the next image frame, OI 1 is the attention occupancy aggregated features, and FFN( OI 1) represents nonlinear transformation on OI 1.

[0121] In some embodiments, the scene occupancy of the next image frame can also be subjected to layer normalization and parameter adjustment, which is represented as follows:

[0122] γ 22=Norm( OI 2)•(1+ β 2)+ OI 2

[0123] Wherein, OI 22 is the adjusted scene occupancy of the next image frame, and Norm( γ 2) represents normalization on Figure 52. Perform layer normalization to stabilize feature distribution, (1 β 2) represents scaling the normalized features to adapt to feature scale changes under different camera perspectives, Figure 5 2 represents shifting the features to adjust the distribution center and enhance the robustness of the model to perspective shifts.

[0124] In this embodiment, the attention branch updates the current features based on historical information to solve the time sequence dependence problem, and the feedforward branch enhances the feature expression through nonlinear transformation, further improving the model training effect.

[0125] Figure 5 Flowchart of the occupancy world model training method provided by the embodiments of the present application Figure 6 As shown in the figure, in an optional implementation, the sample data further includes actual scene occupancy features of each image frame, and the step S104 of training the initial occupancy world model according to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame includes: Figure 6 S501, according to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame, training the initial encoder to obtain the target encoder.

[0126] According to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame, a first stage loss is calculated to train the initial encoder according to the first stage loss to obtain the target encoder. The first stage loss is used to measure the difference between the initial scene occupancy features and the actual scene occupancy features.

[0127] In some embodiments, the first stage loss can include at least one of a cross-entropy loss, a semantic segmentation loss, and an embedding loss, wherein the cross-entropy loss is used to measure the difference between the occupancy object class indicated by the initial scene occupancy features and the actual class indicated by the actual scene occupancy features, the occupancy object class corresponding to the initial scene occupancy features is the semantic class of the occupied object in the voxel grid indicated by the initial scene occupancy features, and the actual class corresponding to the actual scene occupancy features is the semantic class of the occupied object in the voxel grid indicated by the actual scene occupancy features.

[0128] The semantic segmentation loss is used to measure the difference between the boundary of the occupied object indicated by the initial scene occupancy features and the boundary of the occupied object indicated by the actual scene occupancy features, and the embedding loss is used to measure the feature distance between the initial scene occupancy features and the actual scene occupancy features.

[0129]

[0130] ​It should be noted that the cross-entropy loss is used to optimize the classification accuracy of the occupied object, the semantic segmentation loss is used to optimize the spatial structure integrity of the occupied object, and the embedding loss is used to optimize the feature similarity, so that the initial encoder is optimized from three dimensions to obtain the target encoder.

[0131] S502, according to the predicted space occupation information and the actual space occupation information, the initial intermediate layer and the initial decoder are trained to obtain the target intermediate layer and the target decoder.

[0132] According to the preset space occupation information and the actual space occupation information, the second stage loss is calculated, and the initial intermediate layer and the initial decoder are trained according to the second stage loss to obtain the target intermediate layer and the target decoder. The target occupancy world model includes: the target encoder, the target intermediate layer and the target decoder.

[0133] The second stage loss can include a cross-entropy loss for measuring the difference between the preset space occupation information and the actual space occupation information to constrain the accuracy of spatio-temporal prediction.

[0134] Figure 7 The flowchart of the indoor scene occupancy prediction method provided by the embodiments of the present application is shown in the figure. The execution subject of the embodiments can be an electronic device, such as a target robot.

[0135] As shown in the figure, the method can include: Figure 7

[0136] S601, obtaining a current indoor image frame, a historical indoor image frame and a next camera pose of a camera of a target robot in a target indoor scene.

[0137] The target robot can be a humanoid robot, for example, and the target robot is in a target indoor scene, which can be any of the at least one indoor scene described above.

[0138] The current indoor image frame can be an image frame obtained by the camera of the target robot capturing the target indoor scene at the current time, and the historical indoor image frame can be an image frame obtained by the camera of the target robot capturing the target indoor scene at a historical time before the current time.

[0139] The camera of the target robot can be a monocular camera, and the current indoor image frame and the historical indoor image frame can both be monocular RGB images.

[0140] During the movement of the target robot, the movement algorithm of the target robot can be used to obtain the next camera pose of the target robot, which refers to the camera pose of the target robot at the next time after the current time.

[0141] ​S602, respectively, according to the spatial semantic set of the target indoor scene, the current indoor image frame and the historical indoor image frame are occupied prediction, get current space occupation information and historical space occupation information.

[0142] Among them, the spatial semantic set of the target indoor scene includes the semantic category of each object object in the target indoor scene.

[0143] The camera calibration parameters of the target robot are obtained, and according to the spatial semantic set of the target indoor scene, the camera calibration parameters of the target robot and the target voxel resolution, a preset monocular occupancy prediction model is used to respectively predict the current indoor image frame and the historical indoor image frame, and the current space occupation information and the historical space occupation information are obtained.

[0144] Among them, the current space occupation information includes the occupation information of each voxel grid in the current field of view space of the target robot, and the historical space occupation information includes the occupation information of each voxel grid in the historical field of view space of the target robot.

[0145] The current field of view space refers to the field of view space captured by the camera of the target robot for the current indoor image frame, and the historical field of view space refers to the field of view space captured by the camera of the target robot for the historical indoor image frame.

[0146] For the current indoor image frame and the historical indoor image frame, the occupation information of each voxel grid in the three-dimensional field of view space includes actual occupancy indication information for indicating whether each voxel grid is occupied, and actual semantic category of the occupied object object when each voxel grid is occupied, wherein the occupied object object is the object object occupying each voxel grid, and the actual semantic category of the occupied object object belongs to a subset of the spatial semantic set of the target indoor scene. For details, please refer to the related description in the above embodiment.

[0147] S603, according to the current space occupation information, the historical space occupation information and the next camera pose, a target occupancy world model is used to predict the next space occupation information corresponding to the next indoor image frame.

[0148] The current space occupation information, the historical space occupation information and the next camera pose are used as the input of the target occupancy world model, and the target occupancy world model is used for prediction to obtain the next space occupation information. Among them, the next space occupation information includes the occupation information of each voxel grid in the next field of view space of the target robot, and the next field of view space is the field of view space that can be collected by the camera of the target robot at the next time based on the next camera pose. Among them, the target occupancy world model is a model trained by the above method.

[0149] It should be noted that the current field of view space, the historical field of view space and the next field of view space are all three-dimensional field of view spaces.

[0150] The next spatial occupancy information includes: the occupancy information of each voxel grid in the next field of view includes predicted occupancy indication information for indicating whether each voxel grid is occupied, and the predicted semantic category of the occupying object when each voxel grid is occupied, wherein the occupying object is the object that occupies each voxel grid, and the predicted semantic category of the occupying object belongs to a subset of the spatial semantic set of the target indoor scene.

[0151] For details on the implementation process of the indoor scene occupancy prediction method, please refer to the relevant process in the above-mentioned occupancy world model training method, which will not be repeated here.

[0152] In this embodiment, a target occupancy world model is used to predict the robot's next space occupancy information based on the current space occupancy information, historical space occupancy information, and the next camera pose, so that the robot can perceive the indoor scene occupancy status in the next state.

[0153] Figure 8 This is a schematic diagram of the structure of the occupancy world model training device provided in the embodiments of this application. The device can be integrated into electronic devices, such as laptops and desktop computers.

[0154] like Figure 8 As shown, the device may include:

[0155] The acquisition module 701 is used to acquire an indoor scene dataset. The indoor scene dataset includes: sample data of at least one indoor scene, wherein the sample data of each indoor scene includes: a sample image sequence and a spatial semantic set for each indoor scene. Each image frame in the sample image sequence has a corresponding camera pose, and the spatial semantic set includes: the semantic category of each object in each indoor scene.

[0156] Processing module 702 is used to perform occupancy prediction on each image frame according to the spatial semantic set, and obtain the actual space occupancy information of each image frame. The actual space occupancy information of each image frame includes the actual occupancy information of each voxel grid in the corresponding three-dimensional field space.

[0157] The processing module 702 is also used to predict the predicted space occupancy information of the next image frame by using the initial occupancy world model based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame.

[0158] The processing module 702 is also used to train the initial occupancy world model based on the predicted space occupancy information and the actual space occupancy information of the next image frame, so as to obtain the target occupancy world model.

[0159] In an optional implementation, the processing module 702 is specifically configured to:

[0160] According to the spatial semantic set, the camera calibration parameter corresponding to the sample image sequence, and the target voxel resolution, a preset monocular occupancy prediction model is used to perform occupancy prediction on each image frame to obtain actual spatial occupancy information of each image frame.

[0161] In an optional implementation, the processing module 702 is specifically configured to:

[0162] An initial encoder is used to encode the actual spatial occupancy information of each image frame to obtain initial scene occupancy features of each image frame.

[0163] An initial intermediate layer is used to aggregate the initial scene occupancy features of each image frame under a plurality of preset space-time windows to obtain scene occupancy aggregation features under the plurality of preset space-time windows.

[0164] According to the camera pose corresponding to the next image frame and the scene occupancy aggregation features under the plurality of preset space-time windows, scene occupancy features of the next image frame are obtained.

[0165] An initial decoder is used to decode the scene occupancy features of the next image frame to obtain predicted spatial occupancy information of the next image frame.

[0166] In an optional implementation, the processing module 702 is specifically configured to:

[0167] According to each preset space-time window, the to-be-aggregated features of each preset space-time window are determined from the initial scene occupancy features of each image frame.

[0168] A space-time aggregation module is used to aggregate the to-be-aggregated features of each preset space-time window to obtain scene occupancy aggregation features under each preset space-time window.

[0169] In an optional implementation, the processing module 702 is specifically configured to:

[0170] A space-time aggregation module is used to aggregate the scene occupancy aggregation features under the plurality of preset space-time windows to obtain target occupancy aggregation features.

[0171] An autoregressive converter is used to obtain scene occupancy features of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregation features.

[0172] In an optional implementation, the processing module 702 is specifically configured to:

[0173] The camera pose is encoded to obtain a pose feature.

[0174] The attention branch is adopted to obtain attention occupancy aggregated features corresponding to the target occupancy aggregated features according to the pose features and the target occupancy aggregated features.

[0175] The feedforward branch is adopted to obtain scene occupancy features of the next image frame according to the pose features and the attention occupancy aggregated features.

[0176] In an optional implementation, the processing module 702 is specifically configured to:

[0177] The initial encoder is trained according to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame, to obtain a target encoder.

[0178] The initial intermediate layer and the initial decoder are trained according to the predicted spatial occupancy information and the actual spatial occupancy information, to obtain a target intermediate layer and a target decoder, wherein the target occupancy world model comprises the target encoder, the target intermediate layer and the target decoder.

[0179] The description of the processing procedure of each module in the device and the interaction procedure between the modules can refer to the related description in the above method embodiments, and will not be described in detail here.

[0180] Figure 9 The indoor scene occupancy prediction device provided by the embodiments of the present application is shown in a structural schematic diagram. The device can be integrated in an electronic device, such as a target robot.

[0181] As shown in Figure 9 , the device can comprise:

[0182] The obtaining module 801 is configured to obtain a current indoor image frame, a historical indoor image frame and a next camera pose of a camera of a target robot in a target indoor scene collected by the camera.

[0183] The processing module 802 is configured to perform occupancy prediction on the current indoor image frame and the historical indoor image frame respectively according to a spatial semantic set of the target indoor scene, to obtain current spatial occupancy information and historical spatial occupancy information. The current spatial occupancy information comprises occupancy information of each voxel grid in a current field of view space of the target robot, and the historical spatial occupancy information comprises occupancy information of each voxel grid in a historical field of view space of the target robot.

[0184] The processing module 802 is further configured to predict a next spatial occupancy information by using a target occupancy world model according to the current spatial occupancy information, the historical spatial occupancy information and the next camera pose, wherein the next spatial occupancy information comprises occupancy information of each voxel grid in a next field of view space of the target robot, and the target occupancy world model is a model trained by using the above method.

[0185] The description of the processing flow of each module in the device and the interaction flow between the modules can refer to the related description in the method embodiments, and will not be described in detail here.

[0186] ​ A structural schematic diagram of an electronic device provided in the embodiments of the present application is shown in FIG. 9, which can include a processor 901, a memory 902, and a bus 903. The memory 902 stores machine readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 through the bus 903. The processor 901 executes the machine readable instructions to perform the above method. ​

[0187] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is run by a processor to execute the above method.

[0188] In the embodiments of the present application, the computer program is run by the processor to further execute other machine readable instructions to perform the methods described in other embodiments. For the specific method steps and principles, refer to the description of the embodiments, which will not be described in detail here.

[0189] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, and can be electrical, mechanical or other forms.

[0190] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiments of the present application.

[0191] In addition, each functional unit in the embodiments provided in the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0192] ​If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0193] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0194] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. These modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An occupancy world model training method, comprising: The method comprises the following steps: acquiring an indoor scene data set, the indoor scene data set comprising: sample data of at least one indoor scene, wherein the sample data of each indoor scene comprises: a sample image sequence of the indoor scene and a spatial semantic set, each image frame in the sample image sequence having a corresponding camera pose, and the spatial semantic set comprising: a semantic category of each object in the indoor scene; performing occupancy prediction on the image frames according to the spatial semantic set to obtain actual spatial occupancy information of the image frames, the actual spatial occupancy information of the image frames comprising actual occupancy information of each voxel grid in a corresponding three-dimensional field of view space; performing prediction on the next image frame of the current image frame of each indoor scene by using an initial occupancy world model according to actual spatial occupancy information of the current image frame, actual spatial occupancy information of historical image frames before the current image frame, and a camera pose corresponding to the next image frame to obtain predicted spatial occupancy information of the next image frame; training the initial occupancy world model according to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame to obtain a target occupancy world model.

2. The method of claim 1, wherein, The method comprises the following steps: performing occupancy prediction on the image frames according to the spatial semantic set, the corresponding camera calibration parameters of the sample image sequence, and a target voxel resolution by using a preset monocular occupancy prediction model to obtain actual spatial occupancy information of the image frames.

3. The method of claim 1, wherein, The initial occupancy world model comprises: an initial encoder, an initial intermediate layer, and an initial decoder; the method comprises the following steps: encoding the actual spatial occupancy information of the image frames by using the initial encoder to obtain initial scene occupancy features of the image frames; aggregating the initial scene occupancy features of the image frames under a plurality of preset space-time windows by using the initial intermediate layer to obtain scene occupancy aggregation features under the plurality of preset space-time windows; obtaining scene occupancy features of the next image frame according to the camera pose corresponding to the next image frame and the scene occupancy aggregation features under the plurality of preset space-time windows; decoding the scene occupancy features of the next image frame by using the initial decoder to obtain predicted spatial occupancy information of the next image frame.

4. The method of claim 3, wherein, The initial intermediate layer comprises: a space-time aggregation module; the method comprises the following steps: According to the initial scene occupancy features of the image frames, the to-be-aggregated features of the preset space-time windows are determined; The to-be-aggregated features of the preset space-time windows are aggregated by using the space-time aggregation module, so that the scene occupancy aggregated features under the preset space-time windows are obtained.

5. The method of claim 4, wherein, The initial intermediate layer includes: an autoregressive converter, and the scene occupancy features of the next image frame are obtained according to the camera pose corresponding to the next image frame and the scene occupancy aggregated features under the plurality of preset space-time windows, including: The scene occupancy aggregated features under the plurality of preset space-time windows are aggregated by using the space-time aggregation module, so that target occupancy aggregated features are obtained; The scene occupancy features of the next image frame are obtained by using the autoregressive converter according to the camera pose corresponding to the next image frame and the target occupancy aggregated features.

6. The method of claim 5, wherein, The autoregressive converter includes: an attention branch and a feedforward branch, and the scene occupancy features of the next image frame are obtained by using the autoregressive converter according to the camera pose corresponding to the next image frame and the target occupancy aggregated features, including: The camera pose is encoded to obtain a pose feature; The attention occupancy aggregated features corresponding to the target occupancy aggregated features are obtained by using the attention branch according to the pose feature and the target occupancy aggregated features; The scene occupancy features of the next image frame are obtained by using the feedforward branch according to the pose feature, the attention occupancy aggregated features.

7. The method of claim 3, wherein, The sample data further includes: actual scene occupancy features of the image frames, and the initial occupancy world model is trained to obtain a target occupancy world model according to the predicted spatial occupancy information and the actual spatial occupancy information of the next image frame, including: The initial encoder is trained to obtain a target encoder according to the initial scene occupancy features of the image frames and the actual scene occupancy features of the image frames; The initial intermediate layer and the initial decoder are trained to obtain a target intermediate layer and a target decoder according to the predicted spatial occupancy information and the actual spatial occupancy information, wherein the target occupancy world model includes: the target encoder, the target intermediate layer and the target decoder.

8. A method of indoor scene occupancy prediction, characterized in that, including: A current indoor image frame, a historical indoor image frame and a next camera pose of a target robot in a target indoor scene are obtained, the next camera pose being a camera pose of the target robot at a next time point of a current time point; According to a spatial semantic set of the target indoor scene, occupancy prediction is performed on the current indoor image frame and the historical indoor image frame respectively, so that current spatial occupancy information and historical spatial occupancy information are obtained, the current spatial occupancy information including occupancy information of each voxel grid in a current field of view space of the target robot, and the historical spatial occupancy information including occupancy information of each voxel grid in a historical field of view space of the target robot; According to the current spatial occupancy information, the historical spatial occupancy information, and the next camera pose, a target occupancy world model is used to predict next spatial occupancy information, the next spatial occupancy information including occupancy information of each voxel grid in a next field of view space of the target robot, wherein the target occupancy world model is a model trained by using the method in any one of claims 1-7.

9. An electronic device, comprising: Comprising: A processor, a memory, and a bus, the memory storing machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicating through the bus, the processor executing the machine-readable instructions to perform the method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is executed by the processor to perform the method in any one of claims 1-8.

Images