Model training method, indoor scene occupation prediction method, equipment and medium

Abstract

The invention provides a model training method, an indoor scene occupancy prediction method, equipment and a medium, and the method comprises the steps: obtaining an indoor scene data set, carrying out the occupancy prediction of each image frame according to a spatial semantic set, obtaining the actual space occupancy information of each image frame, according to the actual space occupation information of the current image frame of each indoor scene, the actual space occupation information of the historical image frame and the camera pose corresponding to the next image frame of the current image frame, an initial occupation world model is adopted for prediction, and predicted space occupation information of the next image frame is obtained; and according to the predicted space occupation information and the actual space occupation information of the next image frame, training the initial occupation world model to obtain a target occupation world model. And by adopting the model, the next space occupation information is predicted according to the current space occupation information, the historical space occupation information and the next camera pose, so that the robot can conveniently sense the indoor scene occupation condition of the next state.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, an indoor scene occupancy prediction method, a device, and a medium. Background Art

[0002] With the rapid development of robots, enabling robots to perceive in unknown environments and make corresponding decisions in advance has received more and more attention.

[0003] In related technologies, the occupancy world model is usually used to describe the occupancy status of the robot's field of view. The occupancy world model is a model used to simulate and predict the evolution of the surrounding scene. It uses occupancy as the core representation method, divides the 3D space into voxels, and assigns a label to each voxel to indicate whether it is occupied and what kind of object occupies it.

[0004] However, current occupancy world models mainly focus on outdoor scenes and lack the prediction of robot occupancy in indoor scenes. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a model training method, an indoor scene occupancy prediction method, a device and a medium, which adopt a target occupancy world model to predict the robot's next space occupancy information based on the current space occupancy information, historical space occupancy information and the next camera pose, so as to facilitate the robot to perceive the indoor scene occupancy in the next state.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising: Acquire an indoor scene dataset, the indoor scene dataset comprising: sample data of at least one indoor scene, wherein the sample data of each indoor scene comprises: a sample image sequence and a spatial semantic set of each indoor scene, each image frame in the sample image sequence having a corresponding camera pose, and the spatial semantic set comprising: a semantic category of each object in each indoor scene; performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame, wherein the actual spatial occupancy information of each image frame includes actual occupancy information of each voxel grid in a corresponding three-dimensional field of view; Based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, an initial occupancy world model is used to perform prediction to obtain predicted space occupancy information of the next image frame; The initial occupancy world model is trained according to the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model.

[0007] In an optional implementation, performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame includes: According to the spatial semantic set, the camera calibration parameters corresponding to the sample image sequence, and the target voxel resolution, a preset monocular occupancy prediction model is used to perform occupancy prediction on each image frame to obtain actual spatial occupancy information of each image frame.

[0008] In an optional embodiment, the initial occupancy world model includes: an initial encoder, an initial intermediate layer, and an initial decoder; the initial occupancy world model is used to perform prediction based on the actual spatial occupancy information of the current image frame of each indoor scene, the actual spatial occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame to obtain the predicted spatial occupancy information of the next image frame, including: Using the initial encoder, encoding the actual space occupancy information of each image frame to obtain the initial scene occupancy features of each image frame; Using the initial intermediate layer, aggregating the initial scene occupancy features of each image frame in a plurality of preset spatiotemporal windows, to obtain the scene occupancy aggregate features in the plurality of preset spatiotemporal windows; Obtaining a scene occupancy feature of the next image frame according to a camera pose corresponding to the next image frame and scene occupancy aggregation features under the multiple preset spatiotemporal windows; The initial decoder is used to decode the scene occupancy feature of the next image frame to obtain predicted space occupancy information of the next image frame.

[0009] In an optional embodiment, the initial intermediate layer includes: a spatiotemporal aggregation module, wherein the initial intermediate layer is used to aggregate the initial scene occupancy features of each image frame in multiple preset spatiotemporal windows to obtain the scene occupancy aggregate features in the multiple preset spatiotemporal windows, including: Determining, according to the preset spatiotemporal windows, features to be aggregated for the preset spatiotemporal windows from the initial scene occupancy features of the image frames; The spatiotemporal aggregation module is used to aggregate the features to be aggregated in each of the preset spatiotemporal windows to obtain the scene occupancy aggregation features in each of the preset spatiotemporal windows.

[0010] In an optional embodiment, the initial intermediate layer includes an autoregressive transformer, and obtaining the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the scene occupancy aggregate features under the multiple preset spatiotemporal windows includes: Using the spatiotemporal aggregation module, aggregating the scene occupancy aggregation features under the multiple preset spatiotemporal windows to obtain the target occupancy aggregation features; The autoregressive transformer is used to obtain a scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregate feature.

[0011] In an optional embodiment, the autoregressive transformer includes an attention branch and a feedforward branch, and the autoregressive transformer is used to obtain the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregate feature, including: Encoding the camera pose to obtain pose features; Using the attention branch, according to the posture feature and the target occupancy aggregate feature, obtain an attention occupancy aggregate feature corresponding to the target occupancy aggregate feature; The feedforward branch is used to obtain the scene occupancy feature of the next image frame according to the posture feature and the attention occupancy aggregation feature.

[0012] In an optional embodiment, the sample data further includes: actual scene occupancy features of each image frame; and the training of the initial occupancy world model based on the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model includes: training the initial encoder according to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame to obtain a target encoder; The initial intermediate layer and the initial decoder are trained according to the predicted space occupancy information and the actual space occupancy information to obtain a target intermediate layer and a target decoder, wherein the target occupancy world model includes: the target encoder, the target intermediate layer and the target decoder.

[0013] In a second aspect, an embodiment of the present application further provides an indoor scene occupancy prediction method, comprising: Obtain the current indoor image frame, historical indoor image frames, and next camera pose captured by the camera of the target robot in the target indoor scene; performing occupancy prediction on the current indoor image frame and the historical indoor image frame according to the spatial semantic set of the target indoor scene, respectively, to obtain current space occupancy information and historical space occupancy information, wherein the current space occupancy information includes occupancy information of each voxel grid in the current field of view of the target robot, and the historical space occupancy information includes occupancy information of each voxel grid in the historical field of view of the target robot; According to the current space occupancy information, the historical space occupancy information and the next camera pose, the target occupancy world model is used to predict the next space occupancy information, wherein the next space occupancy information includes the occupancy information of each voxel grid in the next field of view of the target robot, wherein the target occupancy world model is a model trained using the method described in any one of the first aspects.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute any one of the methods described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the first aspects is executed.

[0016] The present application provides a model training method, an indoor scene occupancy prediction method, a device, and a medium. The model training method includes: obtaining an indoor scene dataset, performing occupancy prediction on each image frame based on a spatial semantic set to obtain actual spatial occupancy information for each image frame, using an initial occupancy world model to perform prediction based on the actual spatial occupancy information of the current image frame of each indoor scene, the actual spatial occupancy information of historical image frames, and the camera pose corresponding to the next image frame of the current image frame to obtain predicted spatial occupancy information for the next image frame, and training the initial occupancy world model based on the predicted spatial occupancy information and actual spatial occupancy information of the next image frame to obtain a target occupancy world model. By using this model, the next spatial occupancy information is predicted based on the current spatial occupancy information, historical spatial occupancy information, and the next camera pose, thereby facilitating the robot's perception of the indoor scene occupancy in the next state. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 1 ; Figure 2 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 2 ; Figure 3 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 3 ; Figure 4 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 4 ; Figure 5 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 5 ; Figure 6 A flowchart of an indoor scene occupancy prediction method provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an occupancy world model training device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an indoor scene occupancy prediction device provided in an embodiment of the present application; Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0020] The diversity of camera positions and directions of robots in indoor scenes has increased significantly, resulting in a lack of spatiotemporal regularity between different observed image frames, leading to the current lack of prediction of robot occupancy in indoor scenes. Based on this, the present application uses the actual spatial occupancy information of visually observed image frames and the camera pose corresponding to the image frames to train a target occupancy world model, so that the target occupancy world model can understand the spatiotemporal connection, and then uses the target occupancy world model to use the current spatial occupancy information of the current indoor image frame observed visually, the historical spatial occupancy information of the historical indoor image frames, and the potential next camera pose to predict the next spatial occupancy information of the robot, thereby facilitating the robot to perceive the occupancy of the indoor scene in the next state.

[0021] Figure 1 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 1 The execution subject of this embodiment can be an electronic device, such as a laptop computer, a desktop computer, etc.

[0022] like Figure 1 As shown, the method may include: S101: Acquire an indoor scene dataset.

[0023] The indoor scene dataset includes: sample data of at least one indoor scene, and the sample data of each indoor scene includes: a sample image sequence and a spatial semantic set of each indoor scene. Each image frame in the sample image sequence has a corresponding camera posture, and the spatial semantic set includes: the semantic category of each object in each indoor scene.

[0024] The indoor scene may be, for example, a scene including objects such as a ceiling, a floor, a wall, a table, and a sofa. Different indoor scenes may include different objects.

[0025] The sample image sequence for each indoor scene is a continuous sequence of images captured by the camera for each indoor scene. The sample image sequence includes multiple consecutive image frames. Each image frame in the sample image sequence has a corresponding camera pose. The camera pose is the pose of the camera when each image frame was captured. The camera pose may include, for example, camera position and camera orientation.

[0026] The semantic categories of each object in the indoor scene refer to the categories of each object obtained by classifying the objects according to the semantic information (such as function, purpose, form, etc.) of each object in the indoor scene. Among them, the semantic categories may include ceiling, floor, wall, table, sofa, etc.

[0027] S102 : performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame.

[0028] The actual space occupancy information of each image frame includes: actual occupancy information of each voxel grid in the corresponding three-dimensional field of view space.

[0029] Based on the spatial semantic set, occupancy prediction is performed on each image frame to obtain the actual occupancy information of each voxel grid in the 3D field of view corresponding to each image frame. The 3D field of view corresponding to each image frame is the field of view captured by the camera for each image frame. In other words, the camera captures the corresponding 3D field of view in the indoor scene to obtain each image frame.

[0030] The corresponding three-dimensional field of view space is divided into multiple voxel grids, and the actual occupancy information of each voxel grid includes: actual occupancy indication information for indicating whether each voxel grid is occupied, and the actual semantic category of the occupied object object when each voxel grid is occupied, wherein the occupied object object is the object object occupying each voxel grid, and the actual semantic category of the occupied object object belongs to a subset of the spatial semantic set.

[0031] S103. Based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, an initial occupancy world model is used to perform prediction to obtain the predicted space occupancy information of the next image frame.

[0032] Among them, the current image frame of each indoor scene is a plurality of image frames arbitrarily selected from the sample image sequence of each indoor scene, the historical image frame before the current image frame is at least one image frame before the current image frame in the sample image sequence, the at least one image frame includes the previous image frame of the current image frame, the next image frame of the current image frame is the next image frame of the current image frame in the sample image sequence, and the camera pose corresponding to the next image frame is the pose of the camera when the next image frame is captured.

[0033] Initialize and obtain an initial occupancy world model. Take the actual spatial occupancy information of the current image frame of each indoor scene, the actual spatial occupancy information of the historical image frames, and the camera pose corresponding to the next image frame as inputs to the initial occupancy world model. Use the initial occupancy world model to perform predictions to obtain predicted spatial occupancy information for the next image frame. The predicted spatial occupancy information includes: predicted occupancy information for each voxel grid in the three-dimensional field of view corresponding to the next image frame; the predicted occupancy information includes: predicted occupancy indication information indicating whether each voxel grid in the three-dimensional field of view corresponding to the next image frame is occupied; and predicted semantic categories of occupied objects when each voxel grid is occupied. The predicted occupancy indication information is the predicted occupancy indication information, and the predicted semantic categories are the predicted semantic categories, which may include, for example, ceiling, floor, wall, table, sofa, etc.

[0034] S104 : Training the initial occupancy world model based on the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model.

[0035] The model loss is calculated based on the predicted spatial occupancy information of the next image frame and the actual spatial occupancy information of the next image frame. The initial occupied world model is trained based on the model loss until the model loss does not exceed a preset loss threshold. The occupied world model obtained when the model loss does not exceed the preset loss threshold is used as the target occupied world model.

[0036] The specific selection of the preset loss threshold may be based on actual conditions and is not particularly limited in this embodiment.

[0037] In some embodiments, the occupancy loss is calculated based on the actual occupancy information and the predicted occupancy information of the next image frame, the category loss is calculated based on the actual semantic category and the predicted semantic category of the next image frame, and then the model loss is calculated based on the occupancy loss and the category loss, wherein the occupancy loss and the category loss can be weighted to obtain the model loss.

[0038] In the occupancy world model training method provided in this embodiment, the target occupancy world model is obtained by training the actual spatial occupancy information of the image frame observed visually and the camera pose corresponding to the image frame, so that the target occupancy world model can understand the spatiotemporal connection.

[0039] In an optional embodiment, the above step S102, performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame, may include: According to the spatial semantic set, the camera calibration parameters corresponding to the sample image sequence, and the target voxel resolution, a preset monocular occupancy prediction model is used to predict the occupancy of each image frame to obtain the actual spatial occupancy information of each image frame.

[0040] The preset monocular occupancy prediction model may be, for example, an EmbodiedOcc model or a RoboOCC model.

[0041] The camera calibration parameters corresponding to the sample image sequence are calibration parameters of the camera when the sample image sequence is collected, including camera extrinsic parameters and camera intrinsic parameters. The camera extrinsic parameters and camera intrinsic parameters can be represented by a camera extrinsic parameter matrix and a camera intrinsic parameter matrix.

[0042] The target voxel resolution is the resolution of each voxel grid in the three-dimensional field of view corresponding to each image frame. It is used to determine the size of each voxel grid and is expressed as {X, Y, Z}.

[0043] The spatial semantic set, camera calibration parameters and target voxel resolution are used as inputs of the monocular occupancy prediction model. The monocular occupancy prediction model is used to predict the occupancy of each image frame and obtain the actual spatial occupancy information of each image frame.

[0044] Figure 2 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, in an optional embodiment, the initial occupancy world model includes: an initial encoder, an initial intermediate layer, and an initial decoder. Step S103, based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, uses the initial occupancy world model to predict, to obtain the predicted space occupancy information of the next image frame, which may include: S201 : Using an initial encoder, encode actual space occupancy information of each image frame to obtain initial scene occupancy features of each image frame.

[0045] The initial encoder may be a VQ-VAE encoder, which is used to encode the actual spatial occupancy information of each image frame to obtain the initial scene occupancy features of each image frame.

[0046] S202 : using the initial intermediate layer, respectively aggregating the initial scene occupancy features of each image frame in a plurality of preset spatiotemporal windows to obtain the scene occupancy aggregate features in the plurality of preset spatiotemporal windows.

[0047] A preset spatiotemporal window refers to a window that combines temporal and spatial information for data processing during spatiotemporal data analysis. The preset spatiotemporal window includes a window defined in both time and space dimensions.

[0048] The scales defined by multiple preset space-time windows in the time dimension and the space dimension may be different. Multiple preset space-time windows may include a long-term space-time window (L) and a short-term space-time window (S), wherein the time scale corresponding to the long-term space-time window is longer, and the time scale corresponding to the short-term space-time window is shorter. The spatial scales corresponding to the long-term space-time window and the short-term space-time window may be the same or different, and may be set according to actual needs. This embodiment does not specifically limit this.

[0049] It should be noted that the long-term spatiotemporal window (L) is used to capture the global consistency of the time series, and the short-term spatiotemporal window (S) is used to capture fine-grained local spatiotemporal information. The number of long-term spatiotemporal windows (L) includes, but is not limited to, one. Different long-term spatiotemporal windows (L) correspond to different time scales. The spatial scales corresponding to different long-term spatiotemporal windows (L) can be the same or different. The number of short-term spatiotemporal windows (S) includes, but is not limited to, one. For example, there can be two short-term spatiotemporal windows. Different short-term spatiotemporal windows (S) correspond to different time scales. The spatial scales corresponding to different short-term spatiotemporal windows (S) can be the same or different. The specific setting can be based on actual needs and is not particularly limited in this embodiment.

[0050] Among them, the time scale is used to indicate the number of image frames. The larger the time scale, the more image frames, and the smaller the time scale, the fewer image frames. The initial scene occupancy features of the image frame are visualized as a bird's-eye view. The spatial scale is used to indicate the spatial area in the bird's-eye view of the image frame. The features of the spatial area are determined from the initial scene occupancy features as the features to be aggregated of the image frame.

[0051] Using the initial intermediate layer, under each preset spatiotemporal window, the image frames to be processed at the time scale corresponding to each preset spatiotemporal window are determined from each image frame, and the features of the spatial scale corresponding to each preset spatiotemporal window are determined from the initial scene occupancy features of the image frames to be processed as the features to be aggregated of each preset spatiotemporal window, wherein the features to be aggregated are the features indicating the spatial area of ​​the spatial scale corresponding to each preset spatiotemporal window in the bird's-eye view of the image frame to be processed.

[0052] Then, the features to be aggregated in each preset spatiotemporal window are aggregated to obtain the scene occupancy aggregate features in each preset spatiotemporal window.

[0053] It should be noted that by defining multiple preset spatiotemporal windows, an aggregated spatiotemporal receptive field is obtained based on multi-scale spatiotemporal windows to capture the dynamic changes in the scene, thereby better understanding and predicting the evolution of scene occupancy.

[0054] S203 : Obtaining a scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the scene occupancy aggregation features under a plurality of preset spatiotemporal windows.

[0055] The initial intermediate layer also includes: an autoregressive transformer, which uses a spatiotemporal aggregation module to aggregate the scene occupancy aggregation features under multiple preset spatiotemporal windows to obtain the target occupancy aggregation features, and then uses an autoregressive transformer to obtain the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the target occupancy aggregation features.

[0056] The camera pose corresponding to the next image frame and the target occupancy aggregation feature are used as the input of the autoregressive converter, and the output of the autoregressive converter is the scene occupancy feature of the next image frame.

[0057] It should be noted that due to the significant increase in the diversity of camera positions in indoor scenes, the inherent spatiotemporal regularity of history is destroyed. Based on this, the Conditional Causal State Attention (CCSA) mechanism is adopted. The camera pose corresponding to the next image frame is used as a condition to guide the autoregressive transformer to make reasonable occupancy predictions, thereby improving the prediction accuracy.

[0058] In some embodiments, the camera pose corresponding to the next image frame, the target occupancy aggregate feature, and the historical occupancy aggregate feature are used as inputs to the autoregressive transformer, and the output is the scene occupancy feature of the next image frame. Among them, the historical occupancy aggregate feature is obtained by using an initial intermediate layer to aggregate the initial scene occupancy features of each image frame under multiple historical spatiotemporal windows to obtain the scene occupancy aggregate features under multiple historical spatiotemporal windows, and then aggregating the scene occupancy aggregate features under multiple historical spatiotemporal windows. Regarding the implementation process of the historical occupancy aggregate feature, please refer to the implementation process of the target occupancy aggregate feature above, which will not be repeated here.

[0059] S204 : Using the initial decoder, decode the scene occupancy features of the next image frame to obtain predicted space occupancy information of the next image frame.

[0060] The initial decoder may be a VQ-VAE decoder, which is used to decode the scene occupancy features of the next image frame to obtain predicted space occupancy information of the next image frame.

[0061] Figure 3 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 3 ,like Figure 3 As shown, in an optional embodiment, the initial intermediate layer includes: a spatio-temporal aggregation module, which may be, for example, a hybrid spatio-temporal aggregation (HSTA) module.

[0062] In step S202, the initial intermediate layer is used to aggregate the initial scene occupancy features of each image frame in multiple preset spatiotemporal windows to obtain the scene occupancy aggregate features in the multiple preset spatiotemporal windows, which may include: S301 : According to each preset spatiotemporal window, determine the features to be aggregated of each preset spatiotemporal window from the initial scene occupancy features of each image frame.

[0063] According to the time scale corresponding to each preset spatiotemporal window, the image frames to be processed at the corresponding time scale of each preset spatiotemporal window are determined from each image frame, and the image frames to be processed are visualized as a bird's-eye view. Then, according to the spatial scale corresponding to each preset spatiotemporal window, the features of the corresponding spatial scale are determined from the bird's-eye view of the image frames to be processed as the features to be aggregated of each preset spatiotemporal window.

[0064] S302: Using a spatiotemporal aggregation module, aggregate the features to be aggregated in each preset spatiotemporal window to obtain the scene occupancy aggregation features in each preset spatiotemporal window.

[0065] The spatiotemporal aggregation module is used to aggregate the features to be aggregated in each preset spatiotemporal window to obtain the scene occupancy aggregation features under each preset spatiotemporal window.

[0066] In some embodiments, the spatiotemporal aggregation model for a long-term spatiotemporal window (L) includes two layers of causal convolution. These layers normalize the features to be aggregated, extracting them from the bird's-eye view, stabilize their distribution, and accelerate training. These features are then processed by a first layer of causal 3D convolution (Causal3DConv1) to ensure that the calculation of the current time step uses only information from the past or current moment, preventing future information leakage. This makes them suitable for time series prediction. The processed features are then nonlinearly transformed using ReLU activation to enhance the model's expressiveness. A second layer of causal convolution (Causal3DConv2) performs causal convolution in the temporal dimension on the features output by the ReLU activation to further refine long-term features. The features to be aggregated are then summed with the output of the second layer of causal convolution to preserve the initial information (features to be aggregated), alleviating the vanishing gradient problem in deep networks. The summed result is then nonlinearly transformed using ReLU activation.

[0067] The specific implementation process of the spatiotemporal aggregation model for the short-term spatiotemporal window (S) is similar to that of the long-term spatiotemporal window (L) mentioned above, and will not be repeated here.

[0068] In this embodiment, by defining multiple preset spatiotemporal windows, an aggregated spatiotemporal receptive field is obtained based on multi-scale spatiotemporal windows to capture dynamic changes in the scene, and by adopting the spatiotemporal aggregation module to perform feature aggregation, the scene occupancy aggregation features under each preset spatiotemporal window are obtained, so as to better understand and predict the evolution of scene occupancy.

[0069] Figure 4 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 4 ,like Figure 4As shown, in an optional embodiment, the autoregressive transformer includes: a feedforward branch, using the autoregressive transformer, obtaining the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregate feature, which may include: S401: Encode the camera pose to obtain pose features.

[0070] A linear encoding layer is used to encode the camera pose to obtain pose features.

[0071] S402: Using the attention branch, obtain the attention occupancy aggregation feature corresponding to the target occupancy aggregation feature according to the posture feature and the target occupancy aggregation feature.

[0072] The attention branch is based on the Causal State Attention (CSA) mechanism. CSA is an inter-frame cross-attention operation with a causal attention mask. It ensures that in a time series, the features of the current moment can only focus on historical information of the past or current moment, avoiding future information leakage and conforming to the causal logic of time series prediction.

[0073] In some embodiments, an attention branch is used to obtain attention occupancy aggregate features based on posture features, target occupancy aggregate features, and historical occupancy aggregate features.

[0074] A multi-layer perceptron (MLP) is used to generate adjustment parameters under the attention branch based on the posture features, including the first residual scaling factor. α 1. First layer normalization scaling factor β 1. First layer normalized shift factor γ 1.

[0075] Then, the target occupancy aggregate feature is used as the current input feature (i.e., the query), and the historical occupancy aggregate feature is used as the historical tag feature (as the key and value respectively). Through the cross-attention mechanism of the attention branch, the historical tag feature is used to update the current input feature to obtain the attention occupancy aggregate feature, which is expressed as follows: OI 1=( α 1×CSA( OI , OH , OH ))+ OI in, OI 1 is the attention occupation aggregation feature, OI is the target occupancy aggregation feature, OH is the historical occupancy aggregation feature.

[0076] In some embodiments, layer normalization and parameter adjustment may be performed on the attention occupation aggregate feature, as shown below: OI 12=Norm( OI 1)×(1+ β 1)+ γ 1 in, OI 12 is the adjusted attention occupation aggregation feature, Norm( OI 1) Express OI 1 layer normalization to stabilize the feature distribution, (1+ β 1) and γ 1. Scale and shift the normalized features respectively so that the model can adaptively adjust the feature representation according to the current camera perspective.

[0077] S403: Using the feedforward branch, obtain the scene occupancy features of the next image frame based on the posture features and the attention occupancy aggregation features.

[0078] The feed-forward branch includes a feed-forward network (FFN), which is used to perform nonlinear transformations on the aggregated attention occupancy features output by the attention branch, capturing more complex features and enhancing the model's ability to express fine-grained spatiotemporal cues. The FFN can be composed of two linear layers.

[0079] A multi-layer perceptron (MLP) is used to generate adjustment parameters under the attention branch according to the posture features, including the second residual scaling factor. α 2. Second layer normalization scaling factor β 2. Second layer normalized shift factor γ 2.

[0080] According to the posture features and attention occupancy aggregation features, the feedforward branch is used to obtain the scene occupancy features of the next image frame. The scene occupancy features of the next image frame are expressed as follows: OI 2=( α 2•FFN( OI 1))+ OI 1 in, OI 2 is the scene occupancy feature of the next image frame, OI 1 is the attention occupancy aggregation feature, FFN( OI 1) Express OI 1 Perform nonlinear transformation.

[0081] In some embodiments, layer normalization and parameter adjustment may also be performed on the scene occupancy features of the next image frame, as shown below: OI 22=Norm( OI 2)•(1+ β 2)+ γ 2 in, OI 22 is the scene occupancy feature of the next image frame after adjustment, Norm( OI 2) Express OI 2 layer normalization to stabilize the feature distribution, (1+ β 2) Indicates scaling of the normalized features to adapt to the feature scale changes under different camera perspectives. γ 2 means shifting the features, adjusting the distribution center, and enhancing the model's robustness to perspective shifts.

[0082] In this embodiment, the attention branch updates the current features through historical information to solve the problem of temporal dependency, and the feedforward branch enhances the feature expression through nonlinear transformation, further improving the model training effect.

[0083] Figure 5 Schematic diagram of the process of the occupancy world model training method provided in the embodiment of the present application Figure 5 ,like Figure 5 As shown, in an optional embodiment, the sample data further includes: actual scene occupancy features of each image frame. In step S104, the initial occupancy world model is trained based on the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model, including: S501 : Train an initial encoder based on the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame to obtain a target encoder.

[0084] Based on the initial scene occupancy features and the actual scene occupancy features of each image frame, a first-stage loss is calculated to train the initial encoder based on the first-stage loss to obtain the target encoder. The first-stage loss is used to measure the difference between the initial scene occupancy features and the actual scene occupancy features.

[0085] In some embodiments, the first-stage loss may include at least one of a cross-entropy loss, a semantic segmentation loss, and an embedding loss, wherein the cross-entropy loss is used to measure the difference between the occupied object category indicated by the initial scene occupancy feature and the actual category indicated by the actual scene occupancy feature, the occupied object category corresponding to the initial scene occupancy feature is the semantic category of the occupied object object of the voxel grid indicated by the initial scene occupancy feature, and the actual category corresponding to the actual scene occupancy feature is the semantic category of the occupied object object of the voxel grid indicated by the actual scene occupancy feature.

[0086] The semantic segmentation loss is used to measure the difference between the boundaries of the occupied object objects indicated by the initial scene occupancy features and the boundaries of the occupied object objects indicated by the actual scene occupancy features, and the embedding loss is used to measure the feature distance between the initial scene occupancy features and the actual scene occupancy features.

[0087] It should be noted that the cross entropy loss is used to optimize the classification accuracy of occupied objects, the semantic segmentation loss is used to optimize the spatial structure integrity of occupied objects, and the embedding loss is used to optimize feature similarity, thereby optimizing the initial encoder from three dimensions to obtain the target encoder.

[0088] S502: Train the initial intermediate layer and the initial decoder according to the predicted space occupancy information and the actual space occupancy information to obtain a target intermediate layer and a target decoder.

[0089] Based on the preset space occupancy information and the actual space occupancy information, a second-stage loss is calculated. The initial intermediate layer and initial decoder are trained based on the second-stage loss to obtain a target intermediate layer and target decoder. The target occupancy world model includes a target encoder, a target intermediate layer, and a target decoder.

[0090] Among them, the second stage loss can include cross entropy loss, which is used to measure the difference between the preset space occupancy information and the actual space occupancy information to constrain the accuracy of spatiotemporal prediction.

[0091] Figure 6 This is a flow chart of the indoor scene occupancy prediction method provided in an embodiment of the present application. The execution subject of this embodiment may be an electronic device, such as a target robot.

[0092] like Figure 6 As shown, the method may include: S601: Acquire a current indoor image frame, historical indoor image frames, and a next camera pose captured by a camera of a target robot in a target indoor scene.

[0093] The target robot may be, for example, an embodied robot, and the target robot is located in a target indoor scene, which is any one of the at least one indoor scene mentioned above.

[0094] The current indoor image frame may be an image frame acquired by the target robot's camera of the target indoor scene at the current moment, and the historical indoor image frame may be an image frame acquired by the target robot's camera of the target indoor scene at a historical moment before the current moment.

[0095] The camera of the target robot can be a monocular camera, and the current indoor image frame and the historical indoor image frame can both be monocular RGB images.

[0096] During the movement of the target robot, the movement algorithm of the target robot can be used to obtain the next camera pose of the target robot. The next camera pose refers to the camera pose of the target robot at the next moment after the current moment.

[0097] S602: Perform occupancy prediction on the current indoor image frame and the historical indoor image frames according to the spatial semantic set of the target indoor scene to obtain current space occupancy information and historical space occupancy information.

[0098] The spatial semantic set of the target indoor scene includes: the semantic category of each object in the target indoor scene.

[0099] The camera calibration parameters of the target robot are obtained, and based on the spatial semantic set of the target indoor scene, the camera calibration parameters of the target robot, and the target voxel resolution, a preset monocular occupancy prediction model is used to perform occupancy prediction on the current indoor image frame and the historical indoor image frames to obtain the current space occupancy information and the historical space occupancy information.

[0100] The current space occupancy information includes occupancy information of each voxel grid in the current field of view of the target robot, and the historical space occupancy information includes occupancy information of each voxel grid in the historical field of view of the target robot.

[0101] The current field of view space refers to the field of view space captured by the target robot's camera for the current indoor image frame, and the historical field of view space refers to the field of view space captured by the target robot's camera for the historical indoor image frames.

[0102] For the current indoor image frame and the historical indoor image frames, the occupancy information for each voxel grid within the corresponding three-dimensional field of view includes actual occupancy indication information indicating whether each voxel grid is occupied, and the actual semantic category of the occupied object when each voxel grid is occupied. The occupied object is the object occupying each voxel grid, and the actual semantic category of the occupied object belongs to a subset of the spatial semantic set of the target indoor scene. For details, please refer to the relevant description in the above embodiments.

[0103] S603: According to the current space occupancy information, the historical space occupancy information and the next camera pose, the target occupancy world model is used to predict the next space occupancy information corresponding to the next indoor image frame.

[0104] The target occupancy world model uses the current space occupancy information, historical space occupancy information, and the next camera pose as inputs. The target occupancy world model is used to predict the next space occupancy information. This next space occupancy information includes the occupancy information of each voxel grid within the target robot's next field of view. The next field of view is the field of view that the target robot's camera can capture at the next moment based on the next camera pose. The target occupancy world model is the model trained using the above method.

[0105] It should be noted that the current field of view, the historical field of view, and the next field of view are all three-dimensional field of view spaces.

[0106] The next space occupancy information includes: the occupancy information of each voxel grid in the next field of view space includes predicted occupancy indication information for indicating whether each voxel grid is occupied, and the predicted semantic category of the occupied object object when each voxel grid is occupied, wherein the occupied object object is the object object occupying each voxel grid, and the predicted semantic category of the occupied object object belongs to a subset of the spatial semantic set of the target indoor scene.

[0107] For the specific implementation process of the indoor scene occupancy prediction method, please refer to the relevant process in the above-mentioned occupancy world model training method, which will not be repeated here.

[0108] In this embodiment, the target occupancy world model is used to predict the robot's next space occupancy information based on the current space occupancy information, historical space occupancy information and the next camera pose, so that the robot can perceive the indoor scene occupancy in the next state.

[0109] Figure 7 This is a structural diagram of the occupancy world model training device provided in an embodiment of the present application. The device can be integrated into electronic devices such as laptops, desktop computers, etc.

[0110] like Figure 7 As shown, the device may include: An acquisition module 701 is configured to acquire an indoor scene dataset, the indoor scene dataset comprising: sample data of at least one indoor scene, wherein the sample data of each indoor scene comprises: a sample image sequence and a spatial semantic set of each indoor scene, wherein each image frame in the sample image sequence has a corresponding camera pose, and the spatial semantic set comprises: a semantic category of each object in each indoor scene; Processing module 702, configured to perform occupancy prediction on each image frame based on the spatial semantic set to obtain actual spatial occupancy information for each image frame, where the actual spatial occupancy information for each image frame includes actual occupancy information for each voxel grid within the corresponding three-dimensional field of view; The processing module 702 is further configured to use the initial occupancy world model to perform prediction based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, to obtain predicted space occupancy information of the next image frame; The processing module 702 is further configured to train the initial occupancy world model according to the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model.

[0111] In an optional implementation, the processing module 702 is specifically configured to: According to the spatial semantic set, the camera calibration parameters corresponding to the sample image sequence, and the target voxel resolution, a preset monocular occupancy prediction model is used to predict the occupancy of each image frame to obtain the actual spatial occupancy information of each image frame.

[0112] In an optional implementation, the processing module 702 is specifically configured to: Using an initial encoder, the actual space occupancy information of each image frame is encoded to obtain the initial scene occupancy features of each image frame; Using the initial intermediate layer, the initial scene occupancy features of each image frame are aggregated under multiple preset spatiotemporal windows to obtain the scene occupancy aggregate features under multiple preset spatiotemporal windows; Obtain the scene occupancy feature of the next image frame based on the camera pose corresponding to the next image frame and the scene occupancy aggregation features under multiple preset spatiotemporal windows; The initial decoder is used to decode the scene occupancy features of the next image frame to obtain the predicted space occupancy information of the next image frame.

[0113] In an optional implementation, the processing module 702 is specifically configured to: According to each preset spatiotemporal window, determining features to be aggregated for each preset spatiotemporal window from initial scene occupancy features of each image frame; The spatiotemporal aggregation module is used to aggregate the features to be aggregated in each preset spatiotemporal window to obtain the scene occupancy aggregation features under each preset spatiotemporal window.

[0114] In an optional implementation, the processing module 702 is specifically configured to: The spatiotemporal aggregation module is used to aggregate the scene occupancy aggregation features under multiple preset spatiotemporal windows to obtain the target occupancy aggregation features; An autoregressive transformer is used to obtain the scene occupancy features of the next image frame based on the camera pose corresponding to the next image frame and the object occupancy aggregate features.

[0115] In an optional implementation, the processing module 702 is specifically configured to: Encode the camera pose to obtain pose features; Using the attention branch, according to the posture features and the target occupancy aggregate features, the attention occupancy aggregate features corresponding to the target occupancy aggregate features are obtained; The feedforward branch is used to obtain the scene occupancy features of the next image frame based on the posture features and attention occupancy aggregation features.

[0116] In an optional implementation, the processing module 702 is specifically configured to: The initial encoder is trained according to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame to obtain a target encoder; The initial intermediate layer and the initial decoder are trained according to the predicted space occupancy information and the actual space occupancy information to obtain a target intermediate layer and a target decoder, wherein the target occupancy world model includes: a target encoder, a target intermediate layer, and a target decoder.

[0117] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0118] Figure 8 This is a structural diagram of an indoor scene occupancy prediction device provided in an embodiment of the present application. The device can be integrated into an electronic device, such as a target robot.

[0119] like Figure 8 As shown, the device may include: An acquisition module 801 is configured to acquire a current indoor image frame, a historical indoor image frame, and a next camera pose captured by a camera of a target robot in a target indoor scene; Processing module 802 is configured to perform occupancy prediction on the current indoor image frame and the historical indoor image frames based on the spatial semantic set of the target indoor scene, thereby obtaining current space occupancy information and historical space occupancy information. The current space occupancy information includes occupancy information of each voxel grid within the current field of view of the target robot, and the historical space occupancy information includes occupancy information of each voxel grid within the historical field of view of the target robot. The processing module 802 is further used to use the target occupancy world model to predict the next space occupancy information based on the current space occupancy information, the historical space occupancy information and the next camera pose, where the next space occupancy information includes the occupancy information of each voxel grid in the next field of view of the target robot, wherein the target occupancy world model is a model trained using the above method.

[0120] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0121] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 9 As shown, the device may include: a processor 901, a memory 902 and a bus 903. The memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 through the bus 903, and the processor 901 executes the machine-readable instructions to perform the above method. An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is executed.

[0122] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0123] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0126] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0127] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0128] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for training an occupancy world model, characterized in that: include: Acquire an indoor scene dataset, the indoor scene dataset comprising: sample data of at least one indoor scene, wherein the sample data of each indoor scene comprises: a sample image sequence and a spatial semantic set of each indoor scene, each image frame in the sample image sequence having a corresponding camera pose, and the spatial semantic set comprising: a semantic category of each object in each indoor scene; performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame, wherein the actual spatial occupancy information of each image frame includes actual occupancy information of each voxel grid in a corresponding three-dimensional field of view; Based on the actual space occupancy information of the current image frame of each indoor scene, the actual space occupancy information of the historical image frames before the current image frame, and the camera pose corresponding to the next image frame of the current image frame, an initial occupancy world model is used to perform prediction to obtain predicted space occupancy information of the next image frame; The initial occupancy world model is trained according to the predicted space occupancy information and the actual space occupancy information of the next image frame to obtain a target occupancy world model.

2. The method according to claim 1, characterized in that The performing occupancy prediction on each image frame according to the spatial semantic set to obtain actual spatial occupancy information of each image frame includes: According to the spatial semantic set, the camera calibration parameters corresponding to the sample image sequence, and the target voxel resolution, a preset monocular occupancy prediction model is used to perform occupancy prediction on each image frame to obtain actual spatial occupancy information of each image frame.

3. The method according to claim 1, characterized in that The initial occupancy world model includes: an initial encoder, an initial intermediate layer, and an initial decoder; the initial occupancy world model is used to perform prediction based on actual space occupancy information of the current image frame of each indoor scene, actual space occupancy information of historical image frames before the current image frame, and a camera pose corresponding to the next image frame of the current image frame to obtain predicted space occupancy information of the next image frame, including: Using the initial encoder, encoding the actual space occupancy information of each image frame to obtain the initial scene occupancy features of each image frame; Using the initial intermediate layer, aggregating the initial scene occupancy features of each image frame in a plurality of preset spatiotemporal windows, to obtain the scene occupancy aggregate features in the plurality of preset spatiotemporal windows; Obtaining a scene occupancy feature of the next image frame according to a camera pose corresponding to the next image frame and scene occupancy aggregation features under the multiple preset spatiotemporal windows; The initial decoder is used to decode the scene occupancy feature of the next image frame to obtain predicted space occupancy information of the next image frame.

4. The method according to claim 3, characterized in that The initial intermediate layer includes a spatiotemporal aggregation module, which uses the initial intermediate layer to aggregate the initial scene occupancy features of each image frame in multiple preset spatiotemporal windows to obtain the scene occupancy aggregation features in the multiple preset spatiotemporal windows, including: Determining, according to the preset spatiotemporal windows, features to be aggregated for the preset spatiotemporal windows from the initial scene occupancy features of the image frames; The spatiotemporal aggregation module is used to aggregate the features to be aggregated in each of the preset spatiotemporal windows to obtain the scene occupancy aggregation features in each of the preset spatiotemporal windows.

5. The method according to claim 4, characterized in that The initial intermediate layer includes an autoregressive transformer, which obtains the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the scene occupancy aggregation features under the multiple preset spatiotemporal windows, including: Using the spatiotemporal aggregation module, aggregating the scene occupancy aggregation features under the multiple preset spatiotemporal windows to obtain the target occupancy aggregation features; The autoregressive transformer is used to obtain a scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregate feature.

6. The method according to claim 5, characterized in that The autoregressive transformer includes an attention branch and a feedforward branch. The autoregressive transformer is used to obtain the scene occupancy feature of the next image frame according to the camera pose corresponding to the next image frame and the target occupancy aggregate feature, including: Encoding the camera pose to obtain pose features; Using the attention branch, according to the posture feature and the target occupancy aggregate feature, obtain an attention occupancy aggregate feature corresponding to the target occupancy aggregate feature; The feedforward branch is used to obtain the scene occupancy feature of the next image frame according to the posture feature and the attention occupancy aggregation feature.

7. The method according to claim 3, characterized in that The sample data further includes: actual scene occupancy features of each image frame; training the initial occupancy world model based on the predicted space occupancy information and actual space occupancy information of the next image frame to obtain a target occupancy world model, including: training the initial encoder according to the initial scene occupancy features of each image frame and the actual scene occupancy features of each image frame to obtain a target encoder; The initial intermediate layer and the initial decoder are trained according to the predicted space occupancy information and the actual space occupancy information to obtain a target intermediate layer and a target decoder, wherein the target occupancy world model includes: the target encoder, the target intermediate layer and the target decoder.

8. A method for predicting indoor scene occupancy, characterized in that: include: Obtain the current indoor image frame, historical indoor image frames, and next camera pose captured by the camera of the target robot in the target indoor scene; performing occupancy prediction on the current indoor image frame and the historical indoor image frame according to the spatial semantic set of the target indoor scene, respectively, to obtain current space occupancy information and historical space occupancy information, wherein the current space occupancy information includes occupancy information of each voxel grid in the current field of view of the target robot, and the historical space occupancy information includes occupancy information of each voxel grid in the historical field of view of the target robot; Based on the current space occupancy information, the historical space occupancy information, and the next camera pose, a target occupancy world model is used to predict the next space occupancy information, where the next space occupancy information includes the occupancy information of each voxel grid in the next field of view of the target robot. The target occupancy world model is a model trained using the method described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images