Training methods, equipment, robot control methods, and robots for scene perception models

Abstract

This application provides a training method, device, robot control method, and robot for a scene perception model, relating to the field of machine learning technology. The method includes: acquiring Gaussian geometric features and first semantic features of multiple 3D points using a pre-set Gaussian decoder based on their spatial location information and initial features; performing 3D reconstruction of the 3D Gaussian point cloud map from multiple perspectives to obtain reconstructed scene images and first semantic feature maps from multiple perspectives; performing multi-layer semantic encoding on the reconstructed scene images from multiple perspectives using a pre-set hierarchical semantic encoding to obtain second semantic feature maps from multiple perspectives; and training a pre-set initial perception model based on the first and second semantic feature maps to obtain a target perception model. This application utilizes semantic features in the training process of the target perception model, improving the target perception model's ability to understand scene semantics and the accuracy of its decision-making.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and more specifically, to a training method, device, robot control method, and robot for a scene perception model. Background Technology

[0002] In the field of visual reinforcement learning (RL), the quality of the latent representation of a scene directly determines the training effect and task execution capability of the RL agent. How to accurately and efficiently construct scene representations has become the core challenge to improve the agent's environmental perception and decision-making capabilities.

[0003] Currently, integrating 3D-perceptive Neural Radiation Field (NeRF) technology into scene representation learning has become one of the mainstream solutions. NeRF, through volume rendering, can reconstruct 3D-perceptive scene models from multi-view images, providing visual information for intelligent agents. However, in practical applications, this technology has revealed a deficiency in the fineness of semantic information. When processing scenes, a homogenized processing logic is adopted for free space and occupied space, failing to fully exploit the geometric prior information such as depth and texture contained in the observation data. This results in the inability to effectively extract and utilize key information such as the semantic features of objects and spatial relationships in the scene, leading to insufficient semantic richness and accuracy of the scene representation. This limits the intelligent agent's ability to understand scene semantics and the accuracy of its decision-making. Summary of the Invention

[0004] This application addresses the shortcomings of the prior art by providing a training method, device, robot control method, and robot for a scene perception model, in order to solve the problems existing in the prior art.

[0005] The technical solution adopted in the embodiments of this application is as follows:

[0006] In a first aspect, embodiments of this application provide a method for training a scene-aware model, comprising:

[0007] A preset initial perception model is used to perceive real scene images from multiple perspectives to obtain initial features; the real scene images from multiple perspectives are images collected from the working scene of the target robot from the multiple perspectives;

[0008] Based on the real scene images from the multiple perspectives, obtain the spatial position information of multiple three-dimensional points of the target robot;

[0009] Based on the spatial location information of the plurality of three-dimensional points and the initial features, a preset Gaussian decoder is used to obtain the Gaussian geometric features and the first semantic features of the plurality of three-dimensional points.

[0010] Based on the Gaussian geometric features and first semantic features of the multiple three-dimensional points, Gaussian prediction is performed on the multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map.

[0011] From the multiple perspectives, the three-dimensional Gaussian point cloud map is reconstructed in three dimensions to obtain the reconstructed scene images from the multiple perspectives and the first semantic feature map.

[0012] A preset hierarchical semantic encoding method is used to perform multi-layer semantic encoding on the reconstructed scene images from the multiple perspectives to obtain the second semantic feature map of the multiple perspectives;

[0013] The preset initial perception model is trained based on the first semantic feature map and the second semantic feature map to obtain the target perception model.

[0014] In one embodiment, obtaining the spatial position information of multiple three-dimensional points within the three-dimensional space where the target robot is located based on the real scene images from the multiple perspectives includes:

[0015] Based on the real scene images from the multiple perspectives, a 3D projection is performed to obtain the image projection information of the multiple 3D points of the target robot;

[0016] A preset position encoder is used to encode the image projection information of the multiple three-dimensional points to obtain the spatial position information of the multiple three-dimensional points.

[0017] In one embodiment, the step of obtaining the Gaussian geometric features and first semantic features of the plurality of three-dimensional points using a preset Gaussian decoder based on the spatial location information of the plurality of three-dimensional points and the initial features includes:

[0018] The spatial position information of the plurality of three-dimensional points and the initial features are fused to obtain the fused features of the plurality of three-dimensional points;

[0019] Based on the fusion features of the multiple 3D points, the Gaussian geometric features and first semantic features of the multiple 3D points are obtained using the preset Gaussian decoder.

[0020] In one embodiment, the method further includes:

[0021] Based on the image projection information of the plurality of three-dimensional points, the color information and coordinate information of the plurality of three-dimensional points are obtained;

[0022] The step of performing Gaussian prediction on the plurality of three-dimensional points based on the Gaussian geometric features and the first semantic features to obtain a three-dimensional Gaussian point cloud map includes:

[0023] Based on the Gaussian geometric features, first semantic features, color information, and coordinate information of the multiple three-dimensional points, Gaussian prediction is performed on the multiple three-dimensional points to obtain the three-dimensional Gaussian point cloud map.

[0024] In one embodiment, the step of performing Gaussian prediction on the plurality of three-dimensional points based on their Gaussian geometric features, first semantic features, color information, and coordinate information to obtain the three-dimensional Gaussian point cloud map includes:

[0025] Based on the Gaussian geometric features, first semantic features, color information and coordinate information of the multiple three-dimensional points, Gaussian point query is performed on the multiple three-dimensional points to obtain the Gaussian points corresponding to the multiple three-dimensional points.

[0026] The three-dimensional Gaussian point cloud map is obtained by rasterizing the Gaussian points corresponding to the multiple three-dimensional points.

[0027] In one embodiment, the step of performing 3D reconstruction on the 3D Gaussian point cloud map from the multiple viewpoints to obtain the reconstructed scene images and the first semantic feature map from the multiple viewpoints includes:

[0028] From the multiple perspectives, a three-dimensional Gaussian sputtering reconstruction technique is used to reconstruct the three-dimensional Gaussian point cloud map in three dimensions, thereby obtaining the reconstructed scene images from the multiple perspectives and the first semantic feature map.

[0029] In one embodiment, the preset hierarchical semantic encoding includes: a semantic segmentation model, a feature extractor, and an autoencoder;

[0030] The method employs a preset hierarchical semantic encoding to perform multi-layer semantic encoding on the reconstructed scene images from multiple perspectives, obtaining a second semantic feature map for the multiple perspectives, including:

[0031] Using the aforementioned semantic segmentation model, semantic segmentation is performed on the reconstructed scene image from each viewpoint to obtain a multi-layer semantic segmentation image;

[0032] Using the aforementioned feature extractor, semantic extraction is performed on the multi-layer semantic segmentation image to obtain multi-layer semantic features;

[0033] The autoencoder is used to fuse and encode the multi-layer semantic features to obtain a second semantic feature map for each perspective.

[0034] In one embodiment, the method further includes:

[0035] The first loss parameter is calculated based on the reconstructed scene images from the multiple viewpoints and the real scene images from the multiple viewpoints.

[0036] The step of training the preset initial perception model based on the first semantic feature map and the second semantic feature map to obtain the target perception model further includes:

[0037] Calculate the second loss parameter based on the first semantic feature map and the second semantic feature map;

[0038] The preset initial perception model is trained based on the first loss parameter and the second loss parameter to obtain the target perception model.

[0039] In one embodiment, the semantic segmentation model is used to perform semantic segmentation on the reconstructed scene image from each viewpoint to obtain a multi-layer semantic segmentation image, including:

[0040] Based on the part levels of the target robot, the semantic segmentation model is used to perform semantic segmentation on the reconstructed scene image from each viewpoint to obtain the multi-layer semantic segmentation image, which corresponds to multiple part levels of the target robot.

[0041] Secondly, embodiments of this application also provide a training apparatus for a scene-aware model, comprising:

[0042] The first perception module is used to perceive real scene images from multiple perspectives using a preset initial perception model to obtain initial features; the real scene images from multiple perspectives are images collected from the working scene of the target robot from the multiple perspectives.

[0043] The first acquisition module is used to acquire the spatial position information of multiple three-dimensional points of the target robot based on the real scene images from the multiple perspectives.

[0044] The decoding module is used to obtain the Gaussian geometric features and first semantic features of the multiple three-dimensional points using a preset Gaussian decoder based on the spatial position information of the multiple three-dimensional points and the initial features.

[0045] The Gaussian prediction module is used to perform Gaussian prediction on the multiple three-dimensional points based on the Gaussian geometric features and the first semantic features of the multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map.

[0046] The 3D reconstruction module is used to perform 3D reconstruction on the 3D Gaussian point cloud map from the multiple viewpoints to obtain the reconstructed scene images from the multiple viewpoints and the first semantic feature map.

[0047] The semantic encoding module is used to perform multi-layer semantic encoding on the reconstructed scene images from the multiple perspectives using a preset hierarchical semantic encoding method to obtain the second semantic feature map of the multiple perspectives.

[0048] The training module is used to train the preset initial perception model based on the first semantic feature map and the second semantic feature map to obtain the target perception model.

[0049] Thirdly, embodiments of this application also provide a computer device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the training method of any of the above-described scene perception models.

[0050] Fourthly, embodiments of this application also provide a robot control method, including:

[0051] Acquire the current scene image of the target robot's working environment;

[0052] Based on the current scene image, a pre-trained target perception model is used to perform scene perception and obtain the current scene features. The target perception model is obtained according to the training method of any of the above-mentioned scene perception models.

[0053] Based on the current scene characteristics, the target robot's motion control parameters are generated using a preset reinforcement learning model.

[0054] The target robot is controlled according to the motion control parameters.

[0055] Fifthly, embodiments of this application also provide a robot, comprising at least: a robot body and a controller disposed within the robot body, the controller being used to execute the robot control method described above.

[0056] Sixthly, embodiments of this application also provide a robot control device, including:

[0057] The second acquisition module is used to acquire the current scene image of the working scene where the target robot is located;

[0058] The second perception module is used to perform scene perception based on the current scene image using a pre-trained target perception model to obtain the current scene features, wherein the target perception model is obtained according to the training method of the above-mentioned scene perception model.

[0059] The generation module is used to generate motion control parameters for the target robot based on the current scene features and using a preset reinforcement learning model of the target robot.

[0060] The control module is used to control the target robot according to the motion control parameters.

[0061] The beneficial effects of this application are as follows: This application provides a training method for a scene perception model, including using a preset initial perception model to perceive real scene images from multiple perspectives to obtain initial features; the real scene images from multiple perspectives are images collected from the working scene of a target robot from the multiple perspectives; based on the real scene images from multiple perspectives, spatial position information of multiple three-dimensional points of the target robot is obtained; based on the spatial position information of the multiple three-dimensional points and the initial features, a preset Gaussian decoder is used to obtain Gaussian geometric features and first semantic features of the multiple three-dimensional points; based on the Gaussian geometric features and first semantic features of the multiple three-dimensional points, Gaussian prediction is performed on the multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map; under the multiple perspectives, the three-dimensional Gaussian point cloud map is reconstructed in three dimensions to obtain reconstructed scene images and first semantic feature maps from the multiple perspectives; a preset hierarchical semantic encoding is used to perform multi-layer semantic encoding on the reconstructed scene images from the multiple perspectives to obtain second semantic feature maps from the multiple perspectives; based on the first semantic feature map and the second semantic feature map, the preset initial perception model is trained to obtain a target perception model.

[0062] In this process, Gaussian prediction is performed on multiple three-dimensional points based on their Gaussian geometric features and first semantic features to obtain a three-dimensional Gaussian point cloud map. This is further used to obtain a reconstructed scene image, a first semantic feature map, and a second semantic feature map, which are then used to train a target perception model. In the process of training the target perception model, this application uses semantic features to improve the target perception model's ability to understand scene semantics and the accuracy of its decision-making. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the overall architecture of an embodiment of this application;

[0065] Figure 2 This is one of the flowcharts illustrating the training method for the scene-aware model provided in the embodiments of this application;

[0066] Figure 3 A second schematic flowchart illustrating the training method for the scene-aware model provided in this application embodiment;

[0067] Figure 4The third flowchart illustrates the training method for the scene-aware model provided in this application embodiment;

[0068] Figure 5 The fourth flowchart illustrates the training method for the scene-aware model provided in this application embodiment.

[0069] Figure 6 Fifth flowchart illustrating the training method of the scene perception model provided in the embodiments of this application;

[0070] Figure 7 A schematic flowchart of the training method for the scene-aware model provided in the embodiments of this application is shown in Figure 6.

[0071] Figure 8 The seventh flowchart illustrates the training method for the scene-aware model provided in this application embodiment;

[0072] Figure 9 Eighth flowchart illustrating the training method of the scene perception model provided in the embodiments of this application;

[0073] Figure 10 A schematic diagram of the structure of a training device for a scene perception model provided in an embodiment of this application;

[0074] Figure 11 This is a schematic diagram of the structure of the robot control device provided in the embodiments of this application;

[0075] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0077] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0078] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0080] Vision-based reinforcement learning (RL) is gaining increasing attention. For example, NeRF is used to learn 3D consistent implicit representations of robots and their environment. However, this method fails to effectively extract and utilize key information such as semantic features of objects and spatial relationships in the scene, resulting in insufficient semantic richness and accuracy of the scene representation. This limits the agent's ability to understand scene semantics and the accuracy of its decision-making.

[0081] Figure 1 This is a schematic diagram of the overall architecture of an embodiment of this application. The following is in conjunction with the appendix. Figure 1 The training method of the scene perception model provided in this application is illustrated with several examples.

[0082] Figure 2 This is one of the flowcharts illustrating the training method for the scene-aware model provided in the embodiments of this application, such as... Figure 2 As shown, the method includes:

[0083] S101. Using a preset initial perception model, perceive real scene images from multiple perspectives to obtain initial features.

[0084] The real-world scene images from multiple perspectives are images captured from the target robot's working environment at various angles. Two to four cameras are used to acquire multi-view images of the robot's working environment, which are then input into a pre-defined initial perception model (Encoder) to extract initial visual features.

[0085] S102. Based on real scene images from multiple perspectives, obtain the spatial position information of multiple three-dimensional points of the target robot.

[0086] By using the camera intrinsic parameter matrix K and extrinsic parameter matrix P, real scene images from multiple perspectives are projected into 3D space to obtain the three-dimensional point coordinates (x, y, z), which is the spatial position information.

[0087] S103. Based on the spatial location information and initial features of multiple three-dimensional points, a preset Gaussian decoder is used to obtain the Gaussian geometric features and first semantic features of multiple three-dimensional points.

[0088] The spatial location information of multiple 3D points is concatenated with the initial features and input into a Gaussian decoder to obtain the Gaussian geometric features (Hgeo) and the first semantic features (Hfeat) of multiple 3D points.

[0089] S104. Based on the Gaussian geometric features and first semantic features of multiple three-dimensional points, Gaussian prediction is performed on multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map.

[0090] By combining Gaussian geometric features and first semantic features, a 3D Gaussian point cloud map is generated through rasterization rendering.

[0091] S105. Perform 3D reconstruction on the 3D Gaussian point cloud map from multiple perspectives to obtain reconstructed scene images from multiple perspectives and the first semantic feature map.

[0092] Specifically, the 3D Gaussian sputtering reconstruction technique (3DGS technique) can be used to reconstruct the 3D Gaussian point cloud map from multiple perspectives, thereby obtaining reconstructed scene images and first semantic feature maps from multiple perspectives.

[0093] S106. Using a preset hierarchical semantic encoding, multi-layer semantic encoding is performed on the reconstructed scene images from multiple perspectives to obtain the second semantic feature maps from multiple perspectives.

[0094] The SAM model is used to generate hierarchical semantic codes at the robot part level. The CLIP model is used to extract part-level semantic features, and the second semantic feature map is obtained by an autoencoder.

[0095] S107. Based on the first semantic feature map and the second semantic feature map, train the preset initial perception model to obtain the target perception model.

[0096] Calculate the loss function L1 between the first semantic feature map and the second semantic feature map, optimize the preset initial perception model, and obtain the target perception model.

[0097] In summary, this embodiment provides a training method for a scene perception model. Based on the Gaussian geometric features and first semantic features of multiple 3D points, Gaussian prediction is performed on multiple 3D points to obtain a 3D Gaussian point cloud map. Further, a reconstructed scene image, a first semantic feature map, and a second semantic feature map are obtained to train a target perception model. This embodiment uses semantic features in the process of training the target perception model, which improves the target perception model's ability to understand scene semantics and the accuracy of decision-making.

[0098] In one embodiment, Figure 3 This is the second flowchart illustrating the training method for the scene-aware model provided in this application embodiment. Figure 3 As shown, S102 obtains the spatial position information of multiple 3D points in the 3D space where the target robot is located based on real scene images from multiple perspectives, including:

[0099] S201. Perform 3D projection based on real scene images from multiple perspectives to obtain image projection information of multiple 3D points of the target robot.

[0100] A 3D projection algorithm is used to project multiple 3D points on the target robot onto the image plane, converting them into 2D coordinates on the image plane, thereby obtaining the image projection information of multiple 3D points on the target robot.

[0101] S202. Using a preset position encoder, the image projection information of multiple three-dimensional points is position encoded to obtain the spatial position information of multiple three-dimensional points.

[0102] A pre-trained preset position encoder is used. The image projection information of multiple three-dimensional points obtained in step S201 is used as the input of the preset position encoder. The preset position encoder performs feature extraction and encoding processing on the input image projection information and outputs the spatial position information of multiple three-dimensional points.

[0103] In one embodiment, Figure 4 This is the third flowchart illustrating the training method for the scene-aware model provided in this application embodiment, as shown below. Figure 4 As shown, S103 uses a preset Gaussian decoder to obtain the Gaussian geometric features and first semantic features of multiple three-dimensional points based on their spatial location information and initial features, including:

[0104] S301. The spatial position information and initial features of multiple three-dimensional points are fused to obtain the fused features of multiple three-dimensional points.

[0105] The spatial location information of multiple 3D points and the initial features are fused by feature concatenation. Specifically, the spatial location coordinates of each 3D point are concatenated with the corresponding initial feature data to form a new feature vector, which is the fused feature.

[0106] S302. Based on the fusion features of multiple 3D points, a preset Gaussian decoder is used to obtain the Gaussian geometric features and first semantic features of multiple 3D points.

[0107] The Preset Gaussian Decoder is a deep learning-based network model whose structure is designed based on the principles of the Gaussian process. It aims to extract statistically significant geometric features and semantic information from fused features. The decoder consists of multiple network layers, including fully connected layers and activation function layers.

[0108] The fused features of multiple 3D points obtained in step S301 are input into a preset Gaussian decoder. Inside the decoder, the fused features are first linearly transformed through a fully connected layer to adjust the feature dimensions and weight distribution. Then, an activation function is used to nonlinearly activate the linearly transformed features, introducing nonlinear factors to enhance the model's expressive power. After processing by multiple layers of the network, the preset Gaussian decoder outputs two parts: one part is the Gaussian geometric features of multiple 3D points, which represent the geometric attributes of the 3D points in space in the form of Gaussian distributed parameters, such as the opacity of the position, occupancy rate, and angle; the other part is the first semantic feature, which encodes the semantic information of the 3D points, such as their category and function, reflecting the semantic attributes of different parts of the target robot, such as distinguishing different parts like the robot's arm and joints, or distinguishing between the whole, local, and sub-parts.

[0109] In one embodiment, Figure 5 This is the fourth flowchart illustrating the training method for the scene-aware model provided in this application embodiment. Figure 5 As shown, the method of this application further includes:

[0110] S401. Based on the image projection information of multiple three-dimensional points, obtain the color information and coordinate information of multiple three-dimensional points.

[0111] When acquiring image projection information of multiple 3D points of the target robot, the color information corresponding to each 3D point is simultaneously extracted from the image data acquired by the image acquisition device. Color information can be represented using common color models (such as the RGB model). For example, the color of a 3D point in the image can be recorded as (R:255, G:0, B:0), i.e., red. Simultaneously, the coordinate information of each 3D point in the image plane, such as two-dimensional coordinates (X, Y), is parsed from the previously acquired image projection information of multiple 3D points. This coordinate is used to locate the specific position of the 3D point in the image, providing a spatial positional reference for subsequent information fusion and processing.

[0112] Then S104 performs Gaussian prediction on multiple three-dimensional points based on the Gaussian geometric features and the first semantic features of multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map, including: S402, performing Gaussian prediction on multiple three-dimensional points based on the Gaussian geometric features, the first semantic features, color information and coordinate information of multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map.

[0113] Specifically, it includes:

[0114] S501. Based on the Gaussian geometric features, first semantic features, color information, and coordinate information of multiple three-dimensional points, perform Gaussian point query on the multiple three-dimensional points to obtain the Gaussian points corresponding to the multiple three-dimensional points.

[0115] The color and coordinate information of multiple 3D points obtained in step S401 are integrated with the Gaussian geometric features and the first semantic features obtained in step S302. The Gaussian geometric features reflect the spatial geometric attributes of the 3D points in the form of Gaussian distribution parameters, and the first semantic features encode the semantic information of the 3D points. The integrated data contains multi-dimensional information such as the spatial location, geometric distribution, semantic category, and visual color of the 3D points.

[0116] Based on the integrated multidimensional information, a pre-defined Gaussian point lookup algorithm is used to process multiple 3D points. This algorithm, based on the multidimensional information of the 3D points, calculates the similarity and spatial relationships between points to determine a corresponding Gaussian point for each 3D point in Gaussian space. For example, 3D points with similar geometric and semantic features, and similar colors and coordinates, will be mapped to nearby positions in Gaussian space during the Gaussian point lookup process, forming corresponding Gaussian points, thus establishing a correspondence between 3D points and Gaussian points.

[0117] S502. Perform rasterization rendering based on the Gaussian points corresponding to multiple 3D points to obtain a 3D Gaussian point cloud map.

[0118] After obtaining the Gaussian points corresponding to multiple 3D points, these Gaussian points are rendered using rasterization techniques in computer graphics. During rasterization, the Gaussian points are transformed from 3D space to 2D screen space, and drawn on screen pixels according to their attribute information (such as position and color). Following certain rendering rules, each Gaussian point is displayed on the screen with an appropriate shape and color, and these numerous Gaussian points together form a 3D Gaussian point cloud. This cloud visually displays the distribution, geometric characteristics, and semantic information of the target robot's 3D points, providing a visualized data foundation for subsequent robot simulation, environmental perception, and human-computer interaction applications.

[0119] Through the steps S401-S502 described above, the generation of a 3D Gaussian point cloud map of the target robot based on multi-information fusion was realized, providing important data support and visualization tools for further research and application in robot-related fields.

[0120] In one embodiment, the preset hierarchical semantic encoding includes a semantic segmentation model, a feature extractor, and an autoencoder. The semantic segmentation model is used to divide the image into layers and separate different objects in the image; the feature extractor can extract semantic information from the image; and the autoencoder is responsible for fusing and encoding the extracted semantic features. The three work together to achieve multi-layer semantic encoding of the image.

[0121] S106 describes the use of preset hierarchical semantic coding to perform multi-layer semantic coding on reconstructed scene images from multiple perspectives, resulting in second semantic feature maps from multiple perspectives, including:

[0122] S601. Using a semantic segmentation model, semantic segmentation is performed on the reconstructed scene image from each viewpoint to obtain a multi-layer semantic segmentation image.

[0123] Based on the part hierarchy of the target robot, a semantic segmentation model is used to process the reconstructed scene images from each viewpoint. The semantic segmentation model is built based on deep learning algorithms, and through learning from a large amount of labeled data, it can identify the regions corresponding to different part hierarchies in the image.

[0124] During processing, the semantic segmentation model performs pixel-level classification on the reconstructed scene image, grouping pixels belonging to the same part level into one category, thus obtaining multi-layer semantic segmentation images. For example, for a target robot, it can be divided into different levels such as whole, local, and sub-parts. After processing by the semantic segmentation model, multi-layer semantic segmentation images corresponding to these part levels will be obtained, with each layer focusing on a specific part level region of the robot.

[0125] S602. Using a feature extractor, semantic extraction is performed on the multi-layer semantic segmentation image to obtain multi-layer semantic features.

[0126] After acquiring multi-layer semantic segmentation images, a feature extractor is used to perform semantic extraction operations on each layer of the image. The feature extractor adopts a convolutional neural network (CNN) structure, and extracts representative semantic features from each layer of semantic segmentation image through multi-layer convolution and pooling operations.

[0127] During the extraction process, the feature extractor focuses on information such as shape, texture, and color in the image, transforming this information into feature vectors. For each semantic segmentation image, the feature extractor outputs corresponding multi-layer semantic features, which contain key semantic information of the corresponding part-level image.

[0128] S603. An autoencoder is used to fuse and encode the multi-layer semantic features to obtain the second semantic feature map for each perspective.

[0129] The multi-layer semantic features obtained by the feature extractor are input into the autoencoder. The autoencoder consists of two parts: an encoder and a decoder. The encoder compresses and encodes the multi-layer semantic features to extract more representative core semantic information; the decoder attempts to reconstruct the original features from the compressed features, thereby achieving the fusion of multi-layer semantic features.

[0130] During the fusion encoding process, the autoencoder optimizes the loss function to ensure that the fused features retain the information of the original multi-layer semantic features to the greatest extent, ultimately outputting a second semantic feature map for each viewpoint. This feature map integrates semantic information from multiple part levels of the target robot, providing rich semantic data support for subsequent Gaussian point cloud map generation and robot-related applications.

[0131] Through the steps S601-S603 above, multi-layer semantic encoding of reconstructed scene images from multiple perspectives was achieved using preset hierarchical semantic encoding, successfully obtaining second semantic feature maps from multiple perspectives. This lays the semantic information foundation for subsequent operations such as generating 3D Gaussian point cloud maps of the target robot based on multi-information fusion.

[0132] Figure 8 The seventh flowchart illustrates the training method for the scene-aware model provided in this application embodiment. Figure 8 As shown, the method of this application further includes:

[0133] S701. Calculate the first loss parameter based on the reconstructed scene images from multiple perspectives and the real scene images from multiple perspectives.

[0134] The system acquires reconstructed scene images from multiple viewpoints, along with corresponding real-world scene images from multiple viewpoints. A first loss parameter is used to measure the difference between the reconstructed scene images and the real-world scene images; its calculation is based on pixel-level or feature-level information of the images.

[0135] The mean squared error (MSE) algorithm can be used to calculate the average of the sum of squared differences between corresponding pixels in the reconstructed scene image and the real scene image, thus obtaining a pixel-level first loss parameter. Alternatively, feature-level information can be used, employing a convolutional neural network to extract features from the reconstructed and real scene images, and then calculating the cosine similarity or Euclidean distance between the features to construct the first loss parameter, reflecting the differences between the two at the semantic feature level. This parameter measures the degree to which the reconstructed scene image reproduces the real scene, providing image-level error feedback for subsequent perceptual model training.

[0136] S107, which describes training a preset initial perception model based on a first semantic feature map and a second semantic feature map to obtain a target perception model, further includes:

[0137] S702. Calculate the second loss parameter based on the first semantic feature map and the second semantic feature map.

[0138] Both the first and second semantic feature maps contain semantic information about the image. After obtaining the first and second semantic feature maps, a second loss parameter is used to evaluate the differences between these two semantic feature maps. The second loss parameter can reflect the differences between feature maps under different semantic encoding methods, providing error guidance at the semantic feature level for the optimization of the perceptual model.

[0139] S703. Based on the first loss parameter and the second loss parameter, train the preset initial perception model to obtain the target perception model.

[0140] The calculated first and second loss parameters are combined as the optimization objective for training the preset initial perceptual model. For example, optimization algorithms such as stochastic gradient descent (SGD) and adaptive moment estimation (Adam) are used to calculate the gradient of the model parameters based on the first and second loss parameters, adjusting the weights and biases of the preset initial perceptual model. During training, through continuous iteration, the sum of the first and second loss parameters is minimized, making the semantic feature map output by the model closer to reality, while improving the similarity between the reconstructed scene image and the real scene image.

[0141] Training stops when the loss function converges to a certain extent, or when the preset number of training rounds or performance metrics are reached. The resulting model is the target perception model. This model can more accurately perceive the semantic information of the target robot, providing more reliable model support for subsequent semantic information-based robot applications, such as path planning, object recognition, and grasping.

[0142] Through the steps S701-S703 described above, based on the loss parameters calculated from the reconstructed scene image and the real scene image, the first semantic feature map and the second semantic feature map, the preset initial perception model is effectively trained, resulting in a target perception model that can accurately perceive the semantic information of the target robot, thereby improving the robot's perception performance and application capabilities in complex scenarios.

[0143] Figure 9 The eighth flowchart illustrates the training method for the scene-aware model provided in this application embodiment. Figure 9 As shown, this application also provides a robot control method, including:

[0144] S801. Obtain the current scene image of the working scene where the target robot is located.

[0145] Taking a handling robot in a logistics warehouse as an example, in such an environment, the robot needs to perform handling tasks in an environment with numerous shelves, a large quantity of goods, and frequent personnel movement. To obtain images of the current work scene, a high-definition camera is installed on the handling robot. The camera's field of view can cover a certain area in front of and around the robot, ensuring complete information about the work scene. When the handling robot begins to perform its task, the camera continuously captures images at a frequency of 5 frames per second to obtain images of the current work scene.

[0146] S802. Based on the current scene image, use a pre-trained target perception model to perform scene perception and obtain the current scene features.

[0147] The target perception model is obtained according to the training method of the scene perception model described in the above embodiments. It can understand and make decisions on scene semantics. After the current scene image is acquired, the image is input into the pre-trained target perception model, and the model can generate a feature vector containing semantic information as the current scene feature.

[0148] S803. Based on the characteristics of the current scene, the target robot's motion control parameters are generated using a preset reinforcement learning model.

[0149] The default reinforcement learning model uses policy learning, which is based on directly optimizing action policies through the policy gradient method, rather than indirectly learning the value function.

[0150] After inputting the current scene features into the preset reinforcement learning model, the model evaluates various possible actions (such as moving forward, backward, turning left, turning right, accelerating, decelerating, etc.) based on the current scene features and outputs the optimal action control parameters.

[0151] S804. Control the target robot according to the motion control parameters.

[0152] After receiving the motion control parameters, the robot's data processing module converts them into specific control commands and transmits them to the robot's drive system. The drive system then controls the robot's motors, servos, and other actuators according to these commands, enabling the robot to perform tasks at specified actions and speeds. During task execution, the robot continuously repeats steps S801-S804, constantly sensing changes in the scene and adjusting the motion control parameters based on new scene characteristics to ensure the robot can safely and efficiently complete the handling task.

[0153] This application also provides a robot, which includes at least: a robot body and a controller disposed within the robot body. The controller is used to execute the robot control method provided in the above embodiments to control the robot to perform corresponding actions.

[0154] The following will continue to explain the apparatus, device and storage medium for implementing the training method of the scene perception model provided in any of the above embodiments of this application. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, the parts not mentioned in the following embodiments can be referred to the corresponding content in the method embodiments.

[0155] Figure 10 This is a schematic diagram of the structure of the training device for the scene perception model provided in the embodiments of this application, as shown below. Figure 10 As shown, the device includes:

[0156] The first perception module 1 is used to perceive real scene images from multiple perspectives using a preset initial perception model to obtain initial features; the real scene images from multiple perspectives are images collected from the working scene of the target robot from the multiple perspectives.

[0157] The first acquisition module 2 is used to acquire the spatial position information of multiple three-dimensional points of the target robot based on the real scene images from the multiple perspectives.

[0158] Decoding module 3 is used to obtain the Gaussian geometric features and first semantic features of the multiple three-dimensional points by using a preset Gaussian decoder based on the spatial position information of the multiple three-dimensional points and the initial features.

[0159] The Gaussian prediction module 4 is used to perform Gaussian prediction on the multiple three-dimensional points based on the Gaussian geometric features and the first semantic features of the multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map.

[0160] The 3D reconstruction module 5 is used to perform 3D reconstruction on the 3D Gaussian point cloud map from the multiple viewpoints to obtain the reconstructed scene images from the multiple viewpoints and the first semantic feature map.

[0161] The semantic encoding module 6 is used to perform multi-layer semantic encoding on the reconstructed scene images from the multiple perspectives using a preset hierarchical semantic encoding method, so as to obtain the second semantic feature map of the multiple perspectives.

[0162] Training module 7 is used to train the preset initial perception model based on the first semantic feature map and the second semantic feature map to obtain the target perception model.

[0163] Figure 11 This is a schematic diagram of the structure of the robot control device provided in the embodiments of this application, such as... Figure 11 As shown, the device includes:

[0164] The second acquisition module 8 is used to acquire the current scene image of the working scene where the target robot is located.

[0165] The second perception module 9 is used to perform scene perception based on the current scene image using a pre-trained target perception model to obtain the current scene features, wherein the target perception model is obtained according to the training method of the scene perception model provided in any of the above embodiments.

[0166] The generation module 10 is used to generate motion control parameters for the target robot based on the current scene features and using a preset reinforcement learning model of the target robot.

[0167] The control module 11 is used to control the target robot according to the motion control parameters.

[0168] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0169] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0170] Figure 12 The schematic diagram of the computer device provided in the embodiments of this application includes: a processor 100, a storage medium 200 and a bus 300. The storage medium stores program instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to implement the training method of the scene perception model described in any of the above embodiments.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0174] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a scene-aware model, characterized in that, The method comprises the following steps: adopting a preset initial perception model to perform perception on real scene images of multiple perspectives to obtain initial features; the real scene images of the multiple perspectives are images obtained by collecting a working scene of a target robot from the multiple perspectives; acquiring spatial position information of multiple three-dimensional points in a three-dimensional space where the target robot is located according to the real scene images of the multiple perspectives; adopting a preset Gaussian decoder to acquire Gaussian geometric features and first semantic features of the multiple three-dimensional points according to the spatial position information of the multiple three-dimensional points and the initial features; performing Gaussian prediction on the multiple three-dimensional points according to the Gaussian geometric features and the first semantic features of the multiple three-dimensional points to obtain a three-dimensional Gaussian point cloud map; performing three-dimensional reconstruction on the three-dimensional Gaussian point cloud map under the multiple perspectives to obtain reconstructed scene images of the multiple perspectives and a first semantic feature map; adopting a preset hierarchical semantic coding to perform multi-layer semantic coding on the reconstructed scene images of the multiple perspectives to obtain second semantic feature maps of the multiple perspectives; training the preset initial perception model according to the first semantic feature map and the second semantic feature map to obtain a target perception model.

2. The method of claim 1, wherein, The acquiring of the spatial position information of the multiple three-dimensional points in the three-dimensional space where the target robot is located according to the real scene images of the multiple perspectives comprises: performing three-dimensional projection according to the real scene images of the multiple perspectives to obtain image projection information of the multiple three-dimensional points of the target robot; adopting a preset position encoder to perform position coding on the image projection information of the multiple three-dimensional points to obtain the spatial position information of the multiple three-dimensional points.

3. The method of claim 1, wherein, The acquiring of the Gaussian geometric features and the first semantic features of the multiple three-dimensional points according to the spatial position information of the multiple three-dimensional points and the initial features by adopting the preset Gaussian decoder comprises: fusing the spatial position information of the multiple three-dimensional points and the initial features to obtain fused features of the multiple three-dimensional points; adopting the preset Gaussian decoder to acquire the Gaussian geometric features and the first semantic features of the multiple three-dimensional points according to the fused features of the multiple three-dimensional points.

4. The method of claim 2, wherein, The method further comprises: acquiring color information and coordinate information of the multiple three-dimensional points according to the image projection information of the multiple three-dimensional points; The performing of the Gaussian prediction on the multiple three-dimensional points according to the Gaussian geometric features and the first semantic features of the multiple three-dimensional points to obtain the three-dimensional Gaussian point cloud map comprises: performing Gaussian prediction on the multiple three-dimensional points according to the Gaussian geometric features, the first semantic features, the color information and the coordinate information of the multiple three-dimensional points to obtain the three-dimensional Gaussian point cloud map.

5. The method of claim 4, wherein, The performing of the Gaussian prediction on the multiple three-dimensional points according to the Gaussian geometric features, the first semantic features, the color information and the coordinate information of the multiple three-dimensional points to obtain the three-dimensional Gaussian point cloud map comprises: performing Gaussian point query on the multiple three-dimensional points according to the Gaussian geometric features, the first semantic features, the color information and the coordinate information of the multiple three-dimensional points to obtain Gaussian points corresponding to the multiple three-dimensional points; According to the Gaussian points corresponding to the plurality of three-dimensional points, rasterization rendering is performed to obtain the three-dimensional Gaussian point cloud image.

6. The method of claim 1, wherein, The preset hierarchical semantic coding includes a semantic segmentation model, a feature extractor, and an autoencoder. The preset hierarchical semantic coding is used to perform multi-layer semantic coding on the reconstructed scene images of the plurality of views to obtain second semantic feature maps of the plurality of views, including: The semantic segmentation model is used to perform semantic segmentation on the reconstructed scene images of each view to obtain multi-layer semantic segmentation images. The feature extractor is used to perform semantic extraction on the multi-layer semantic segmentation images respectively to obtain multi-layer semantic features. The autoencoder is used to perform fusion coding on the multi-layer semantic features to obtain the second semantic feature map of each view.

7. The method of claim 1, wherein, The method further includes: According to the reconstructed scene images of the plurality of views and the real scene images of the plurality of views, a first loss parameter is calculated. The preset initial perception model is trained according to the first semantic feature map and the second semantic feature map to obtain a target perception model, and the method further includes: According to the first semantic feature map and the second semantic feature map, a second loss parameter is calculated. The preset initial perception model is trained according to the first loss parameter and the second loss parameter to obtain the target perception model.

8. A computer device, comprising: It includes: A processor, a storage medium, and a bus, the storage medium storing program instructions executable by the processor, when the computer device is running, the processor and the storage medium communicate through the bus, the processor executes the program instructions to implement the scene perception model training method of any one of claims 1 to 7.

9. A robot control method characterized by, It includes: Obtaining a current scene image of a working scene where a target robot is located; According to the current scene image, a pre-trained target perception model is used for scene perception to obtain a current scene feature, wherein the target perception model is obtained according to the scene perception model training method of any one of claims 1 to 7; According to the current scene feature, a preset reinforcement learning model of the target robot is used to generate action control parameters of the target robot; According to the action control parameters, the target robot is controlled.

10. A robot, characterized in that At least including: A robot body, a controller arranged in the robot body, the controller being configured to execute the robot control method of claim 9.

Images