Gaussian coding model training method, indoor occupancy prediction method, equipment and medium

Abstract

The invention provides a Gaussian coding model training method, an indoor occupancy prediction method, equipment and a medium, and the model training method comprises the steps: carrying out the feature extraction of a sample indoor scene image, obtaining a sample image feature map corresponding to the sample indoor scene image, and carrying out the prediction of the indoor occupancy according to the sample image feature map. Using a preset initial Gaussian coding model to update the randomly generated initial Gaussian feature to obtain a target Gaussian feature corresponding to the sample indoor scene image, obtaining predicted space occupation information corresponding to the sample indoor scene image according to the target Gaussian feature, and determining the indoor scene image according to the sample space occupation information and the predicted space occupation information. And performing parameter adjustment training on the preset initial Gaussian coding model to obtain a target Gaussian coding model. The target Gaussian coding model is adopted to predict the current Gaussian features of the current indoor scene image collected by the robot, then the predicted space occupation information is obtained, the accuracy is high, and the robot can conveniently and accurately sense the indoor scene occupation condition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a Gaussian coding model training method, an indoor occupancy prediction method, a device, and a medium. Background Art

[0002] Occupancy prediction is a key task for robots to achieve embodied perception. It can help robots obtain spatial geometry and semantic information of the environment and support downstream tasks such as exploration and decision-making.

[0003] In related technologies, occupancy prediction models are built based on scene images captured by robots to predict the occupancy of the robot's surrounding scene. These models indicate whether the robot's surrounding scene is occupied and what objects occupy it. However, the accuracy of existing occupancy prediction models is insufficient, hindering the execution of downstream tasks. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a Gaussian coding model training method, indoor occupancy prediction method, device and medium to solve the problem that the existing occupancy prediction model is used to predict occupancy, which is insufficient in accuracy and affects the execution of downstream tasks.

[0005] In a first aspect, an embodiment of the present application provides a Gaussian coding model training method, comprising: Acquire an indoor scene dataset, the indoor scene dataset comprising: sample indoor scene images, wherein the sample indoor scene images are annotated with sample space occupancy information, the sample space occupancy information comprising: actual semantic categories of occupied objects in the sample indoor scene images and actual occupied voxel information; Performing feature extraction on the sample indoor scene image to obtain a sample image feature map corresponding to the sample indoor scene image; According to the sample image feature map, a preset initial Gaussian coding model is used to update the randomly generated initial Gaussian features to obtain target Gaussian features corresponding to the sample indoor scene image; Acquiring predicted space occupancy information corresponding to the sample indoor scene image according to the target Gaussian feature, the predicted space occupancy information including: predicted semantic categories of occupied objects in the sample indoor scene image and predicted occupied voxel information; According to the sample space occupancy information and the predicted space occupancy information, the preset initial Gaussian coding model is trained to obtain a target Gaussian coding model.

[0006] In an optional embodiment, the preset initial Gaussian coding model includes: an opacity-guided autoencoder and a geometry-aware cross encoder; The method of updating the randomly generated initial Gaussian features using a preset initial Gaussian coding model according to the sample image feature map to obtain target Gaussian features corresponding to the sample indoor scene image includes: Using the opacity-guided autoencoder, the initial Gaussian feature is autoencoded according to the opacity information in the initial Gaussian feature to obtain an encoded Gaussian feature; The geometrically perceptual cross encoder is used to perform geometric cross perception on the sample image feature map according to the encoded Gaussian features to obtain the target Gaussian features.

[0007] In an optional embodiment, the preset initial Gaussian coding model further includes: an optimization module; The method of using the geometrically perceptual cross encoder to perform geometric cross perception on the sample image feature map according to the encoded Gaussian feature to obtain the target Gaussian feature includes: Using the geometrically perceptual cross encoder, geometrically cross-perceive the sample image feature map according to the encoded Gaussian features to obtain cross-perceived Gaussian features; The optimization module is used to obtain the target Gaussian feature according to the cross-perception Gaussian feature.

[0008] In an optional embodiment, the opacity-guided autoencoder includes: a voxelization module, and a plurality of sparse convolutional layers; The method of using the opacity-guided autoencoder to autoencode the initial Gaussian feature according to the opacity information in the initial Gaussian feature to obtain the encoded Gaussian feature includes: Using the voxelization module, voxelize the initial Gaussian features to obtain Gaussian features corresponding to a plurality of voxels; The plurality of sparse convolutional layers are sequentially used to add the spatial opaque information in the initial Gaussian features to the Gaussian features corresponding to the plurality of voxels layer by layer to generate the encoded Gaussian features.

[0009] In an optional embodiment, the geometry-aware cross encoder includes: a sampling module, a semantic mixing module, and a geometric mixing module; The method of using the geometrically-aware cross encoder to perform geometric cross-awareness on the sample image feature map according to the encoded Gaussian features to obtain cross-awareness Gaussian features includes: Using the sampling module, sampling the sample image feature map according to the coded Gaussian feature, to obtain image features of the sampled Gaussian points corresponding to the coded Gaussian feature; Using the semantic mixing module, semantic fusion is performed based on the image features of the sampled Gaussian points and the semantic information in the encoded Gaussian features to obtain the semantic fusion features of the sampled Gaussian points; The geometric mixing module is used to perform geometric fusion based on the semantic fusion features of the sampled Gaussian points and the geometric information in the encoded Gaussian features, so as to obtain the geometric fusion features of the sampled Gaussian points as the cross-perception Gaussian features.

[0010] In an optional embodiment, the sampling module is used to sample the sample image feature map according to the coded Gaussian feature to obtain image features of the sampled Gaussian points corresponding to the coded Gaussian feature, including: The sampling module is used to sample the encoded Gaussian features to obtain the sampled Gaussian points, and the sampled Gaussian points are projected into the sample image feature map to obtain the image features of the sampled Gaussian points.

[0011] In an optional implementation, obtaining predicted space occupancy information corresponding to the sample indoor scene image according to the target Gaussian feature includes: A preset Gaussian voxel mapping module is used to perform voxel mapping on the target Gaussian feature to obtain the predicted space occupancy information.

[0012] In a second aspect, an embodiment of the present application further provides an indoor occupancy prediction method, comprising: Obtaining the current indoor scene image captured by the target robot in the target indoor scene; Performing feature extraction on the current indoor scene image to obtain a current image feature map corresponding to the current indoor scene image; According to the current image feature map, the randomly generated initial Gaussian features are updated using a target Gaussian coding model to obtain current Gaussian features corresponding to the current indoor scene image, where the target Gaussian coding model is a model trained using the method described in any one of the first aspects; According to the current Gaussian feature, current space occupancy information corresponding to the current indoor scene image is obtained, where the current space occupancy information includes: semantic categories and occupied voxel information of occupied objects in the current indoor scene image.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute any one of the methods described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the first aspects is executed.

[0015] The present application provides a Gaussian coding model training method, indoor occupancy prediction method, device, and medium, wherein the model training method includes: extracting features from a sample indoor scene image to obtain a sample image feature map corresponding to the sample indoor scene image; updating the randomly generated initial Gaussian features using a preset initial Gaussian coding model based on the sample image feature map to obtain a target Gaussian feature corresponding to the sample indoor scene image; obtaining predicted spatial occupancy information corresponding to the sample indoor scene image based on the target Gaussian feature; and adjusting parameters of the preset initial Gaussian coding model based on the sample spatial occupancy information and the predicted spatial occupancy information to obtain a target Gaussian coding model. The target Gaussian coding model is used to predict the current Gaussian features of the current indoor scene image collected by the robot, thereby obtaining predicted spatial occupancy information with high accuracy, which facilitates the robot to accurately perceive the occupancy of the indoor scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 1 ; Figure 2 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 2 ; Figure 3 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 3 ; Figure 4 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 4 ; Figure 5 A flow chart of an indoor occupancy prediction method provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a Gaussian coding model training device provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an indoor occupancy prediction device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0019] In order to address the problem that the existing occupancy prediction model is used to predict occupancy status, which is not accurate enough and affects the execution of downstream tasks, this application obtains a target Gaussian coding model through training, and uses the target Gaussian coding model to predict the current Gaussian features of the current indoor scene image collected by the robot, that is, the Gaussian features are used to describe the spatial scene, and then the predicted space occupancy information is obtained with high accuracy, which facilitates the robot to accurately perceive the occupancy status of the indoor scene and is beneficial for the robot to execute downstream tasks.

[0020] Figure 1 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 1 The execution subject of this embodiment can be an electronic device, such as a laptop computer, a desktop computer, etc.

[0021] like Figure 1 As shown, the method may include: S101: Acquire an indoor scene dataset.

[0022] The indoor scene dataset includes: sample indoor scene images, the number of sample indoor scene images is multiple, wherein the sample indoor scene images are annotated with sample space occupancy information, and the sample space occupancy information includes: the actual semantic category of the occupied objects in the sample indoor scene images and the actual occupied voxel information.

[0023] The indoor scene may be, for example, a scene including objects such as a ceiling, a floor, a wall, a table, and a sofa. Different indoor scenes may include different objects.

[0024] The occupied objects in the sample indoor scene image are objects within the occupied voxel grid in the three-dimensional field of view space corresponding to the sample indoor scene image, wherein the three-dimensional field of view space is the field of view space captured by the camera for the sample indoor scene image, that is, the sample indoor scene image is obtained by performing image acquisition on the three-dimensional field of view space of the indoor scene, and the three-dimensional field of view space is divided into multiple voxel grids. The occupied objects are objects within the occupied voxel grids in the multiple voxel grids.

[0025] Among them, the actual semantic category of the occupied objects in the sample indoor scene image is used to indicate the category of the occupied objects in the sample indoor scene image, such as ceiling, floor, wall, table, sofa, etc., and the actual occupied voxel information is used for the voxel grid where the occupied objects in the sample indoor scene image are located in the three-dimensional field of view space corresponding to the sample indoor scene image.

[0026] S102: Extract features from the sample indoor scene image to obtain a sample image feature map corresponding to the sample indoor scene image.

[0027] A preset image encoder is used to perform feature extraction on the sample indoor scene image to obtain image features of the sample indoor scene image, and a sample image feature map corresponding to the sample indoor scene image is generated.

[0028] In some embodiments, multi-scale feature extraction is performed on the sample indoor scene image to obtain image features of the sample indoor scene image at different scales (different resolutions), and a multi-scale sample image feature map is generated based on the multi-scale image features, wherein the multi-scale features can capture features at different scales in the image and can provide a more comprehensive image representation.

[0029] It should be noted that the preset image encoder may be a pre-trained image encoder.

[0030] S103 . Based on the sample image feature map, a preset initial Gaussian coding model is used to update the randomly generated initial Gaussian features to obtain target Gaussian features corresponding to the sample indoor scene image.

[0031] Initial Gaussian features are randomly generated and used to initially describe objects in indoor scenes from dimensions such as geometric information, opacity information, and semantic information (semantic category).

[0032] The sample image feature map and initial Gaussian features are used as input. A preset initial Gaussian encoding model is used to update the initial Gaussian features to obtain target Gaussian features. The target Gaussian features are used to describe objects in the indoor scene corresponding to the sample indoor scene image from dimensions such as geometry, opacity, and semantic category. The initial Gaussian features and target Gaussian features can be 3D Gaussian features.

[0033] S104: Obtain predicted space occupancy information corresponding to the sample indoor scene image according to the target Gaussian feature.

[0034] The predicted space occupancy information includes: predicted semantic categories of occupied objects in the sample indoor scene image and predicted occupied voxel information.

[0035] Among them, the predicted semantic category of the occupied object in the sample indoor scene image is used to indicate the category of the occupied object in the predicted sample indoor scene image, and the predicted occupied voxel information is used to indicate the voxel grid where the occupied object in the predicted sample indoor scene image is located in the three-dimensional field of view space corresponding to the sample indoor scene image.

[0036] In an optional embodiment, a preset Gaussian voxel mapping module is used to perform voxel mapping on the target Gaussian features to obtain predicted space occupancy information.

[0037] The preset Gaussian voxel mapping module may be a pre-trained Gaussian voxel mapping module for converting a sparse 3D Gaussian representation into a dense voxel grid. In this embodiment, the Gaussian voxel mapping module is used to map the target Gaussian feature to the voxel grid to obtain preset space occupancy information.

[0038] S105. According to the sample space occupancy information and the predicted space occupancy information, the preset initial Gaussian coding model is trained to obtain a target Gaussian coding model.

[0039] According to the sample space occupancy information and the predicted space occupancy information, the model loss is calculated, and the preset initial Gaussian coding model is trained according to the model loss until the model loss does not exceed the preset loss threshold. The Gaussian coding model obtained when the model loss does not exceed the preset loss threshold is used as the target Gaussian coding model.

[0040] The specific selection of the preset loss threshold may be based on actual conditions and is not particularly limited in this embodiment.

[0041] In some embodiments, semantic loss (semantic affinity loss) is calculated based on actual semantic categories and predicted semantic categories, occupancy loss (geometric affinity loss) is calculated based on actual occupied voxel information and predicted occupied voxel information, and model loss is calculated based on semantic loss and occupancy loss, where the model loss can be a weighted sum of semantic loss and occupancy loss.

[0042] It should be noted that, in some embodiments, the sample space occupancy information and the predicted space occupancy information can also be used to adjust the parameters of the preset image encoder, the preset Gaussian voxel mapping module and the preset initial Gaussian coding model to obtain the target image encoder, the target Gaussian voxel mapping module and the target Gaussian coding model.

[0043] In the Gaussian coding model training method provided in this embodiment, a target Gaussian coding model is obtained through training, and the target Gaussian coding model is used to predict the current Gaussian features of the current indoor scene image collected by the robot, that is, the Gaussian features are used to describe the spatial scene, and then the predicted space occupancy information is obtained with high accuracy, which facilitates the robot to accurately perceive the occupancy of the indoor scene.

[0044] Figure 2 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, in an optional embodiment, the preset initial Gaussian coding model includes: an opacity-guided autoencoder and a geometry-aware cross encoder; the above step S103, based on the sample image feature map, uses the preset initial Gaussian coding model to update the randomly generated initial Gaussian features to obtain the target Gaussian features corresponding to the sample indoor scene image, which may include: S201. Using an opacity-guided autoencoder, the initial Gaussian feature is autoencoded according to the opacity information in the initial Gaussian feature to obtain an encoded Gaussian feature.

[0045] The number of initial Gaussian features is multiple, that is, multiple initial Gaussian features are used to initially describe the indoor scene. The initial Gaussian features include opacity information, which is used to characterize the visibility of objects within the indoor scene described by the initial Gaussian features. Objects with higher opacity have greater visibility, and their semantic information should be retained more. Objects with lower opacity have lower visibility. For example, a foreground object (such as a chair) has higher opacity than a background object (such as a wall), and when overlapping, the semantic information of the chair should be retained first.

[0046] An opacity-guided autoencoder is used to self-encode the initial Gaussian features according to the opacity information, so as to enhance the initial Gaussian features with high opacity (non-empty Gaussian) among multiple initial Gaussian features, and weaken the initial Gaussian features with low opacity (empty Gaussian) among multiple initial Gaussian features, thereby obtaining encoded Gaussian features.

[0047] S202: Use a geometrically-aware cross encoder to perform geometric cross-awareness on the sample image feature map according to the encoded Gaussian features to obtain target Gaussian features.

[0048] The encoded Gaussian features are input into the geometry-aware cross encoder, and the image features of the sampled Gaussian points in the encoded Gaussian features are determined from the sample image feature map. According to the semantic information and geometric information in the encoded Gaussian features, the image features of the sampled Gaussian points are geometrically fused and semantically fused to obtain the target Gaussian features.

[0049] In an optional embodiment, the preset initial Gaussian coding model further includes: an optimization module; The above step S202, using a geometrically-aware cross encoder to perform geometric cross-aware analysis on the sample image feature map according to the encoded Gaussian features to obtain target Gaussian features, may include: A geometrically perceptual cross encoder is used to perform geometric cross perception on the sample image feature map according to the encoded Gaussian features to obtain cross perception Gaussian features. The optimization module is used to obtain the target Gaussian features based on the cross-sensing Gaussian features.

[0050] The encoded Gaussian features are input into the geometric-aware cross encoder, and the image features of the sampled Gaussian points in the encoded Gaussian features are determined from the sample image feature map. According to the semantic information and geometric information in the encoded Gaussian features, the image features of the sampled Gaussian points are geometrically fused and semantically fused to obtain the cross-aware Gaussian features. Then, the optimization module is used to optimize the cross-aware Gaussian features to obtain the target Gaussian features.

[0051] It should be noted that the optimization module can optimize the geometric information, opacity information and semantic information in the cross-perception Gaussian features so that they can more accurately match the real structure and semantic information of the indoor scene corresponding to the sample indoor scene image.

[0052] In the Gaussian coding model training method provided in this embodiment, an opacity-guided autoencoder is used to enhance non-empty Gaussians and weaken empty Gaussians, and a geometric-aware cross encoder is used to perform semantic fusion and geometric fusion for model training, thereby improving the accuracy of model training.

[0053] Figure 3 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 3 ,like Figure 3 As shown, in an optional embodiment, in an optional embodiment, the opacity-guided autoencoder includes: a voxelization module, and multiple sparse convolutional layers. The above step S201, using the opacity-guided autoencoder, self-encodes the initial Gaussian feature according to the opacity information in the initial Gaussian feature to obtain the encoded Gaussian feature, which may include: S301 , using a voxelization module to perform voxelization processing on the initial Gaussian features to obtain Gaussian features corresponding to multiple voxels.

[0054] Among them, the number of initial Gaussian features is multiple, the Gaussian mean in the geometric information of an initial Gaussian feature corresponds to a Gaussian point in the three-dimensional space, and all Gaussian points are aggregated to form a point cloud in the three-dimensional space, which is used to indicate the spatial distribution of objects in the three-dimensional space.

[0055] The voxelization module is used to discretize the point cloud into multiple voxels (sparse voxel grid) according to the preset voxel size, where the multiple voxels are voxels containing Gaussian points, and empty voxels are not retained. Then, the corresponding initial Gaussian features are assigned to the corresponding voxels to obtain the Gaussian features corresponding to the corresponding voxels.

[0056] It's important to note that the Gaussian mean serves as the voxel's coordinate point, while the semantic and opacity information in the initial Gaussian features serve as the voxel's attribute value. These two are combined to generate a voxel-specific sparse tensor, which contains both geometric, semantic, and opacity information. This sparse tensor leverages the spatial sparsity of indoor scenes (most areas are empty), reducing storage and computational overhead compared to dense tensors.

[0057] S302 , sequentially using multiple sparse convolutional layers to add the spatially opaque information in the initial Gaussian features to the Gaussian features corresponding to multiple voxels layer by layer to generate encoded Gaussian features.

[0058] The number of the multiple sparse convolutional layers can be 2, 3, etc., and this embodiment does not specifically limit this. The input of the first sparse convolutional layer in the multiple sparse convolutional layers is the Gaussian features corresponding to the multiple voxels, the input of the second sparse convolutional layer is the output of the first sparse convolutional layer, and so on. The output of the last sparse convolutional layer is the coded Gaussian features. The coded Gaussian features here include the coded Gaussian features corresponding to the multiple voxels.

[0059] In some embodiments, for a sparse convolution layer, a sparse convolution layer is used to add opacity information to the Gaussian features corresponding to multiple voxels to perform convolution operations on non-empty areas, reducing the amount of computation and memory usage. In addition, the convolution operation can also enhance the Gaussian features of voxels with high opacity (non-empty Gaussian) and weaken the Gaussian features of voxels with low opacity (empty Gaussian).

[0060] Among them, the processing of the sparse convolutional layer is expressed as follows: OGSPConv(Q,o)=SPConv3×3(Q)⊙(Sigmoid(o)+Sigmoid(SPConv1×1(Q))) Among them, Q is the Gaussian feature corresponding to the voxel (the feature matrix of the sparse tensor), o is the opacity information, SPConvk×k represents the sparse convolution of the (k×k) kernel, ⊙ represents the dot multiplication operation, Sigmoid(o) represents the application of Sigmoid to the opacity o and maps it to (0,1). The higher the opacity, the closer the weight is to 1, indicating that the corresponding Gaussian is more likely to be an object area and its features need to be strengthened. Among them, Sigmoid is the activation function.

[0061] In some embodiments, the opacity-guided autoencoder includes a voxelization module, a first sparse convolutional layer, and a second sparse convolutional layer.

[0062] Among them, a multi-scale module is also included between the first sparse convolutional layer and the second sparse convolutional layer. The multi-scale module is used to adjust the spatial receptive field of the first sparse convolutional layer to capture information of different scales in the scene and describe the scene more comprehensively. For example, the receptive field of a single layer of 3×3 convolution is 3×3 pixels, while after stacking multiple layers of convolution using the multi-scale module, the spatial receptive field is 7×7 pixels.

[0063] In the Gaussian coding model training method provided in this embodiment, when multiple objects overlap in space, they will jointly affect the semantic prediction of the same voxel. For example, the voxel may be misclassified as the semantic category of any overlapping Gaussian (e.g., a voxel containing both "chair" and "wall" Gaussians may be misclassified as a wall). Based on this, opacity is used to guide the autoencoder. Through the opacity information, the semantic contribution of the foreground object is enhanced and background interference is suppressed, thereby achieving accurate classification of the voxels.

[0064] Figure 4 Schematic diagram of the process of the Gaussian coding model training method provided in the embodiment of the present application Figure 4 ,like Figure 4 As shown, in an optional embodiment, the geometry-aware cross encoder includes: a sampling module, a semantic mixing module and a geometric mixing module; The above step S202, using a geometrically-aware cross encoder to perform geometric cross-awareness on the sample image feature map according to the encoded Gaussian features to obtain cross-aware Gaussian features, may include: S401 , using a sampling module to sample a sample image feature map according to the coded Gaussian feature, and obtaining image features of the sampled Gaussian points corresponding to the coded Gaussian feature.

[0065] A sampling module is used to sample the encoded Gaussian features to obtain sampling Gaussian points, and the sampling Gaussian points are projected into the sample image feature map to obtain the image features of the sampling Gaussian points.

[0066] The encoded Gaussian feature is visualized as an ellipsoid, and the offset Δm0 is initialized. According to the encoded Gaussian feature and the scaling information s and rotation information r in the geometric information of the encoded Gaussian feature, a matrix multiplication operation (MatMul) is used to obtain the Gaussian geometric offset Δm. It is expressed as follows: Δm=MatMul(r,Δm0s) Then, the sampling Gaussian points are determined based on the Gaussian geometric offset Δm and the Gaussian mean in the geometric information encoding the Gaussian feature. It is expressed as follows: P=m+Δm Where m is the mean coordinate of the Gaussian mean, corresponding to the center of the ellipsoid, and P is the position coordinate of the sampled Gaussian points. The sampled Gaussian points are used to describe the geometric contours of the corresponding object. The sampled Gaussian points are 3D points. For example, for an encoded Gaussian feature representing a table, P can be the edge point of the table.

[0067] The camera intrinsic parameter matrix and camera extrinsic parameter matrix when collecting the sample indoor scene image are obtained, and the sampled Gaussian points are projected onto the sample image feature map according to the camera intrinsic parameter matrix and camera extrinsic parameter matrix to obtain the image features of the sampled Gaussian points.

[0068] It is expressed as follows: Qp=Sampling(Q,π(P,E,K),F) Where Qp is the image feature of the sampled Gaussian point, π(P, E, K) is the projection function used to convert the sampled Gaussian point into 2D image coordinates, E is the camera extrinsic parameter matrix, K is the camera intrinsic parameter matrix, and Sampling(Q,…, F) represents the sampling function, which is used to extract the image features corresponding to the 2D image coordinates from the sample image feature map as the image features of the sampled Gaussian point based on the 2D image coordinates.

[0069] It should be noted that the process of Gaussian projection to the image is essentially to establish a cross-modal mapping between the 3D scene representation and the 2D visual perception to achieve geometric alignment. Through the constraints of image features, during the model training process, it gradually converges to a representation that matches the object geometry and semantics, thereby achieving high-precision indoor scene occupancy prediction.

[0070] S402 , using a semantic mixing module to perform semantic fusion based on the image features of the sampled Gaussian points and the semantic information in the encoded Gaussian features, to obtain semantic fusion features of the sampled Gaussian points.

[0071] Among them, the semantic information in the encoded Gaussian features is used to enhance the key semantic information of the image features of the sampled Gaussian points.

[0072] A semantic mixing module is used to obtain semantic weights based on the semantic information in the encoded Gaussian features, where the semantic weights are used to indicate the importance of the semantic information in the Gaussian encoded features. Then, semantic fusion is performed based on the semantic weights and the image features of the sampled Gaussian points to obtain the semantic fusion features of the sampled Gaussian points, where the semantic fusion features are the image features of the sampled Gaussian points fused with the semantic weights.

[0073] It is expressed as follows: Qs=ReLU(LayerNorm(MatMul(Qp,Ws))) Among them, Ws is the semantic weight, MatMul represents matrix multiplication operation, LayerNorm represents layer normalization processing, ReLU represents activation function, and Qs is the semantic fusion feature.

[0074] S403 , using a geometric mixing module to perform geometric fusion based on the semantic fusion features of the sampled Gaussian points and the geometric information in the encoded Gaussian features, to obtain the geometric fusion features of the sampled Gaussian points as cross-perception Gaussian features.

[0075] Among them, the geometric information in the encoded Gaussian features is used to enhance the geometric information of the semantic fusion features of the sampled Gaussian points.

[0076] A geometric mixing module is used to obtain geometric weights based on the geometric information in the encoded Gaussian features, where the geometric weights are used to indicate the importance of the geometric information in the Gaussian encoded features. Then, geometric fusion is performed based on the geometric weights and the semantic fusion features of the sampled Gaussian points to obtain the geometric fusion features of the sampled Gaussian points. The geometric fusion features are cross-perception Gaussian features, where the geometric fusion features are semantic fusion features fused with the geometric weights.

[0077] It is expressed as follows: Qg = ReLU (LayerNorm(MatM ul(Wg , Qs))) Among them, Qg is the geometric fusion feature and Wg is the geometric weight.

[0078] The Gaussian coding model training method provided in this embodiment uses semantic and geometric weights to obtain semantically and geometrically aware image features of the sampled Gaussian points, highlighting the role of semantic and geometric information in enhancing image features. This enables the model to more accurately grasp the semantic and geometric features of objects in the scene, thereby achieving more precise indoor scene occupancy prediction.

[0079] Figure 5 This is a flow chart of the indoor occupancy prediction method provided in an embodiment of the present application. The execution subject of this embodiment may be an electronic device, such as a target robot.

[0080] like Figure 6 As shown, the method may include: S501: Acquire a current indoor scene image captured by a target robot in a target indoor scene.

[0081] The target robot may be, for example, an embodied robot, and the target robot is located in a target indoor scene, which is any one of the aforementioned indoor scenes.

[0082] The current indoor image frame may be an image frame acquired by a camera of the target robot from a target indoor scene. The camera of the target robot may be a monocular camera, and the current indoor image frame may be a monocular RGB image.

[0083] S502: Perform feature extraction on the current indoor scene image to obtain a current image feature map corresponding to the current indoor scene image.

[0084] A preset image encoder is used to perform feature extraction on the current indoor scene image to obtain image features of the current indoor scene image, and a current image feature map corresponding to the current indoor scene image is generated.

[0085] In some embodiments, the target image encoder obtained through the above training is used to perform feature extraction on the current indoor scene image to obtain a current image feature map.

[0086] S503 : According to the current image feature map, the randomly generated initial Gaussian features are updated using a target Gaussian coding model to obtain current Gaussian features corresponding to the current indoor scene image.

[0087] The opacity-guided autoencoder in the target Gaussian coding model is used to autoencode the opacity information in the randomly generated initial Gaussian features to obtain encoded Gaussian features. Then, the geometry-aware cross encoder in the target Gaussian coding model is used to perform geometric cross-aware analysis on the current image feature map based on the encoded Gaussian features to obtain the current Gaussian features corresponding to the current indoor scene image. The target Gaussian coding model is the model trained using the above method.

[0088] The implementation process of the current Gaussian feature can be found in the implementation process of the target Gaussian feature in the above-mentioned occupancy world model training method, which will not be repeated here.

[0089] S504: Obtain current space occupancy information corresponding to the current indoor scene image according to the current Gaussian feature.

[0090] The current space occupancy information includes: semantic categories of occupied objects in the current indoor scene image and occupied voxel information.

[0091] Among them, the semantic category of the occupied object in the current indoor scene image is used to indicate the category of the occupied object in the current indoor scene image, and the occupied voxel information is used to indicate the voxel grid where the occupied object is located in the sample indoor scene image in the three-dimensional field of view space corresponding to the current indoor scene image.

[0092] The preset Gaussian voxel mapping module is used to perform voxel mapping on the current Gaussian feature to obtain the current space occupancy information corresponding to the current indoor scene image.

[0093] In some embodiments, the target Gaussian voxel mapping module obtained through the above training is used to perform voxel mapping on the current Gaussian feature to obtain current space occupancy information.

[0094] In this embodiment, a target Gaussian coding model is used to predict the current Gaussian features of the current indoor scene image collected by the robot, thereby obtaining predicted space occupancy information with high accuracy, which facilitates the robot to accurately perceive the occupancy of the indoor scene.

[0095] Figure 6 A schematic diagram of the structure of the Gaussian coding model training device provided in the embodiment of the present application Figure 1 , the device can be integrated into electronic devices such as laptops, desktop computers, etc. Figure 6 As shown, the device may include: An acquisition module 601 is configured to acquire an indoor scene dataset, the indoor scene dataset comprising: sample indoor scene images, wherein the sample indoor scene images are annotated with sample space occupancy information, the sample space occupancy information comprising: actual semantic categories of occupied objects in the sample indoor scene images and actual occupied voxel information; The processing module 602 is used to extract features from the sample indoor scene image to obtain a sample image feature map corresponding to the sample indoor scene image; The processing module 602 is further configured to update the randomly generated initial Gaussian features using a preset initial Gaussian coding model according to the sample image feature map to obtain target Gaussian features corresponding to the sample indoor scene image; The acquisition module 601 is further configured to acquire predicted space occupancy information corresponding to the sample indoor scene image based on the target Gaussian feature, wherein the predicted space occupancy information includes: predicted semantic categories of occupied objects in the sample indoor scene image and predicted occupied voxel information; The processing module 602 is further used to adjust the parameters of the preset initial Gaussian coding model according to the sample space occupancy information and the predicted space occupancy information to obtain the target Gaussian coding model.

[0096] In an optional embodiment, the preset initial Gaussian coding model includes: an opacity-guided autoencoder, a geometry-aware cross encoder; The processing module 602 is specifically configured to: An opacity-guided autoencoder is used to autoencode the initial Gaussian features according to the opacity information in the initial Gaussian features to obtain encoded Gaussian features. A geometrically aware cross encoder is used to perform geometric cross perception on the sample image feature map according to the encoded Gaussian features to obtain the target Gaussian features.

[0097] In an optional embodiment, the preset initial Gaussian coding model further includes: an optimization module; The processing module 602 is specifically configured to: A geometrically perceptual cross encoder is used to perform geometric cross perception on the sample image feature map according to the encoded Gaussian features to obtain cross perception Gaussian features. The optimization module is used to obtain the target Gaussian features based on the cross-sensing Gaussian features.

[0098] In an optional embodiment, the opacity-guided autoencoder includes: a voxelization module, and a plurality of sparse convolutional layers; The processing module 602 is specifically configured to: The voxelization module is used to voxelize the initial Gaussian features to obtain Gaussian features corresponding to multiple voxels; Multiple sparse convolutional layers are used in sequence to add the spatial opaque information in the initial Gaussian features to the Gaussian features corresponding to multiple voxels layer by layer to generate encoded Gaussian features.

[0099] In an optional embodiment, the geometry-aware cross encoder includes: a sampling module, a semantic mixing module, and a geometric mixing module; The processing module 602 is specifically configured to: The sampling module is used to sample the sample image feature map according to the coded Gaussian feature to obtain the image features of the sampled Gaussian points corresponding to the coded Gaussian feature; A semantic mixing module is used to perform semantic fusion based on the image features of the sampled Gaussian points and the semantic information in the encoded Gaussian features to obtain the semantic fusion features of the sampled Gaussian points. A geometric mixing module is used to perform geometric fusion based on the semantic fusion features of the sampled Gaussian points and the geometric information in the encoded Gaussian features, and the geometric fusion features of the sampled Gaussian points are obtained as cross-perception Gaussian features.

[0100] In an optional implementation, the processing module 602 is specifically configured to: A sampling module is used to sample the encoded Gaussian features to obtain sampling Gaussian points, and the sampling Gaussian points are projected into the sample image feature map to obtain the image features of the sampling Gaussian points.

[0101] In an optional implementation, the acquisition module 601 is specifically configured to: The preset Gaussian voxel mapping module is used to perform voxel mapping on the target Gaussian features to obtain the predicted space occupancy information.

[0102] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0103] Figure 7 A schematic diagram of the structure of the Gaussian coding model training device provided in the embodiment of the present application Figure 2 , the device can be integrated into electronic devices such as target robots.

[0104] An acquisition module 701 is used to acquire a current indoor scene image captured by a target robot in a target indoor scene; The processing module 702 is used to extract features from the current indoor scene image to obtain a current image feature map corresponding to the current indoor scene image; The processing module 702 is further configured to update the randomly generated initial Gaussian features using a target Gaussian coding model based on the current image feature map to obtain current Gaussian features corresponding to the current indoor scene image, where the target Gaussian coding model is the model trained using the above method. The acquisition module 701 is further configured to acquire current space occupancy information corresponding to the current indoor scene image based on the current Gaussian feature. The current space occupancy information includes semantic categories and occupied voxel information of occupied objects in the current indoor scene image.

[0105] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0106] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the device may include: a processor 801, a memory 802 and a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device is running, the processor 801 communicates with the memory 802 through the bus 803, and the processor 801 executes the machine-readable instructions to perform the above method.

[0107] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is executed.

[0108] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0109] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0111] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0113] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0114] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A Gaussian coding model training method, characterized in that: include: Acquire an indoor scene dataset, the indoor scene dataset comprising: sample indoor scene images, wherein the sample indoor scene images are annotated with sample space occupancy information, the sample space occupancy information comprising: actual semantic categories of occupied objects in the sample indoor scene images and actual occupied voxel information; Performing feature extraction on the sample indoor scene image to obtain a sample image feature map corresponding to the sample indoor scene image; According to the sample image feature map, a preset initial Gaussian coding model is used to update the randomly generated initial Gaussian features to obtain target Gaussian features corresponding to the sample indoor scene image; Acquiring predicted space occupancy information corresponding to the sample indoor scene image according to the target Gaussian feature, the predicted space occupancy information including: predicted semantic categories of occupied objects in the sample indoor scene image and predicted occupied voxel information; According to the sample space occupancy information and the predicted space occupancy information, the preset initial Gaussian coding model is trained to obtain a target Gaussian coding model.

2. The method according to claim 1, characterized in that The preset initial Gaussian coding model includes: an opacity-guided autoencoder and a geometry-aware cross encoder; The method of updating the randomly generated initial Gaussian features using a preset initial Gaussian coding model according to the sample image feature map to obtain target Gaussian features corresponding to the sample indoor scene image includes: Using the opacity-guided autoencoder, the initial Gaussian feature is autoencoded according to the opacity information in the initial Gaussian feature to obtain an encoded Gaussian feature; The geometrically perceptual cross encoder is used to perform geometric cross perception on the sample image feature map according to the encoded Gaussian features to obtain the target Gaussian features.

3. The method according to claim 2, characterized in that The preset initial Gaussian coding model further includes: an optimization module; The method of using the geometrically perceptual cross encoder to perform geometric cross perception on the sample image feature map according to the encoded Gaussian feature to obtain the target Gaussian feature includes: Using the geometrically perceptual cross encoder, geometrically cross-perceive the sample image feature map according to the encoded Gaussian features to obtain cross-perceived Gaussian features; The optimization module is used to obtain the target Gaussian feature according to the cross-perception Gaussian feature.

4. The method according to claim 2, characterized in that The opacity-guided autoencoder includes: a voxelization module and a plurality of sparse convolutional layers; The method of using the opacity-guided autoencoder to autoencode the initial Gaussian feature according to the opacity information in the initial Gaussian feature to obtain the encoded Gaussian feature includes: Using the voxelization module, voxelize the initial Gaussian features to obtain Gaussian features corresponding to a plurality of voxels; The plurality of sparse convolutional layers are sequentially used to add the spatial opaque information in the initial Gaussian features to the Gaussian features corresponding to the plurality of voxels layer by layer to generate the encoded Gaussian features.

5. The method according to claim 3, characterized in that The geometric perception cross encoder includes: a sampling module, a semantic mixing module and a geometric mixing module; The method of using the geometrically-aware cross encoder to perform geometric cross-awareness on the sample image feature map according to the encoded Gaussian features to obtain cross-awareness Gaussian features includes: Using the sampling module, sampling the sample image feature map according to the coded Gaussian feature, to obtain image features of the sampled Gaussian points corresponding to the coded Gaussian feature; Using the semantic mixing module, semantic fusion is performed based on the image features of the sampled Gaussian points and the semantic information in the encoded Gaussian features to obtain the semantic fusion features of the sampled Gaussian points; The geometric mixing module is used to perform geometric fusion based on the semantic fusion features of the sampled Gaussian points and the geometric information in the encoded Gaussian features, so as to obtain the geometric fusion features of the sampled Gaussian points as the cross-perception Gaussian features.

6. The method according to claim 5, characterized in that The sampling module is used to sample the sample image feature map according to the coded Gaussian feature to obtain image features of sampling Gaussian points corresponding to the coded Gaussian feature, including: The sampling module is used to sample the encoded Gaussian features to obtain the sampled Gaussian points, and the sampled Gaussian points are projected into the sample image feature map to obtain the image features of the sampled Gaussian points.

7. The method according to claim 1, characterized in that The obtaining, according to the target Gaussian feature, predicted space occupancy information corresponding to the sample indoor scene image includes: A preset Gaussian voxel mapping module is used to perform voxel mapping on the target Gaussian feature to obtain the predicted space occupancy information.

8. A method for indoor occupancy prediction, characterized in that: include: Obtaining the current indoor scene image captured by the target robot in the target indoor scene; Performing feature extraction on the current indoor scene image to obtain a current image feature map corresponding to the current indoor scene image; According to the current image feature map, the randomly generated initial Gaussian features are updated using a target Gaussian coding model to obtain current Gaussian features corresponding to the current indoor scene image, wherein the target Gaussian coding model is a model trained using the method according to any one of claims 1 to 7; According to the current Gaussian feature, current space occupancy information corresponding to the current indoor scene image is obtained, where the current space occupancy information includes: semantic categories and occupied voxel information of occupied objects in the current indoor scene image.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images