Hidden space world model construction method for robot grabbing operation scene and related equipment thereof

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a hidden space world model construction method for a robot grabbing operation scene and related equipment thereof. The method comprises the steps of performing semantic segmentation on a scene image through an image semantic segmentation network to generate a semantic segmentation mask of each object; processing the semantic segmentation mask through a multi-task scene understanding network to generate implicit representation; based on the implicit representation of each object at the current moment and the action information of the mechanical arm, predicting implicit representation at the next moment through a state transition network; and according to the predicted implicit representation at the next moment, generating a scene segmentation image reconstruction result, an object existence judgment result and a contact relation judgment result through a decoder in the multi-task scene understanding network architecture, and constructing an implicit space world model of the scene. Based on the method, efficient characterization and dynamic prediction of the robot grabbing operation scene are achieved, and the generalization ability and the multi-task cooperation efficiency of the model in a complex environment are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for constructing a latent space world model for robot grasping operation scenarios and related equipment. Background Art

[0002] Robotic grasping refers to the process by which a robot uses its end effector to capture, secure, and control a target object. This requires the robot to perceive the object through sensors, plan a motion path, and adjust the end effector's posture and force to accurately complete the grasping task. Robotic grasping scenarios typically rely on explicit geometric modeling or single visual feature extraction methods, using fixed rules or shallow neural networks to describe the scene.

[0003] However, such methods face significant limitations in complex operation scenarios: on the one hand, a single feature cannot capture both the geometric and semantic information of an object simultaneously, resulting in limited representation capabilities; on the other hand, existing models are not adaptable enough to unseen grasping / pushing actions or scene changes, and their generalization is poor, making it difficult to cope with dynamic environments such as occlusion, deformation, or multi-object interaction. In addition, tasks such as object reconstruction, existence judgment, and contact relationship recognition in traditional solutions usually require independent models to handle them separately, resulting in computational redundancy and information inconsistency, and inefficient multi-task collaboration. Therefore, traditional methods are limited in complex scenarios by the insufficient representation capabilities of a single feature, poor model generalization, and inefficient coordination of multi-task processing, making it difficult to cope with problems such as occlusion, deformation, and multi-object interaction in dynamic environments. Summary of the Invention

[0004] The embodiments of the present invention provide a method for constructing a latent space world model for robot grasping operation scenarios and related equipment, which at least solve the problems in related technologies that are limited by the insufficient representation ability of a single feature in complex scenes, poor model generalization, and inefficient coordination of multi-task processing, making it difficult to cope with occlusion, deformation, and multi-object interaction in dynamic environments.

[0005] According to a first aspect of an embodiment of the present invention, a method for constructing a latent space world model for a robot grasping operation scenario is provided, comprising:

[0006] The image semantic segmentation network is used to perform semantic segmentation on the scene images in the robot grasping operation task, and the semantic segmentation mask of each object in the working scene is generated;

[0007] The semantic segmentation mask is processed by a multi-task scene understanding network to generate an implicit representation, wherein the implicit representation includes a position and shape feature identifier of the object, an existence identifier for indicating whether the object exists, and a contact identifier for characterizing the contact relationship between objects. The multi-task scene understanding network is trained by jointly optimizing the semantic segmentation mask image reconstruction loss, the object existence classification loss, and the contact relationship classification loss;

[0008] Based on the implicit representation of each object at the current moment and the robot's manipulator motion information, predicting the implicit representation at the next moment through a state transition network, the motion information including motion type, action position and direction parameters;

[0009] Based on the predicted implicit representation of the next moment, the decoder in the multi-task scene understanding network architecture generates scene segmentation image reconstruction results, object existence judgment results and contact relationship judgment results to construct a latent space world model of the scene.

[0010] According to a second aspect of an embodiment of the present invention, there is provided a device for constructing a latent space world model for a robot grasping operation scene, comprising:

[0011] The segmentation module is used to perform semantic segmentation on the scene image in the robot grasping operation task through the image semantic segmentation network, and generate semantic segmentation masks for each object in the work scene;

[0012] a processing module, configured to process the semantic segmentation mask through a multi-task scene understanding network to generate an implicit representation, wherein the implicit representation includes a position and shape feature identifier of the object, an existence identifier for indicating whether the object exists, and a contact identifier for characterizing contact relationships between objects, wherein the multi-task scene understanding network is trained by jointly optimizing a semantic segmentation mask image reconstruction loss, an object existence classification loss, and a contact relationship classification loss;

[0013] An updating module, configured to predict the implicit representation at the next moment through a state transition network based on the implicit representation of each object at the current moment and the robot's manipulator motion information, wherein the motion information includes motion type, action position, and direction parameters;

[0014] A construction module is used to generate scene segmentation image reconstruction results, object existence judgment results and contact relationship judgment results based on the predicted implicit representation of the next moment through a decoder in a multi-task scene understanding network architecture, and construct a latent space world model of the scene.

[0015] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the method described in the first aspect.

[0016] According to a fourth aspect of an embodiment of the present invention, a non-transitory machine-readable medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0017] Beneficial effects of the embodiments of the present invention:

[0018] The embodiment of the present invention provides a method for constructing a latent space world model for a robot grasping operation scene. The method performs semantic segmentation on the scene image through an image semantic segmentation network to generate an object mask, and then uses a multi-task scene understanding network to process the mask to generate an implicit representation containing the object's position and shape feature identifiers, existence identifiers and contact identifiers. The implicit representation realizes a unified representation of the object's position and shape information and semantic information through the joint optimization of the semantic segmentation mask image reconstruction loss, the existence classification loss and the contact relationship classification loss, thereby constructing a latent space representation that can simultaneously reflect the object's position, shape, existence state and contact relationship; on this basis, by introducing the robot arm action information (action type, action position and direction parameters) and the implicit representation of each object at the current moment The multi-task scene understanding network unifies the semantic segmentation mask image reconstruction, existence judgment and contact relationship recognition tasks under the same network framework by sharing the generation process of implicit representation, reduces the redundant calculation between tasks by jointly optimizing the objective function, and improves the efficiency and consistency of multi-task collaboration. Finally, the decoder in the multi-task scene understanding network performs multimodal decoding on the predicted implicit representation to generate scene segmentation image reconstruction results and symbol judgment results, thereby constructing a latent space world model that can dynamically reflect the changes in the robot's operating environment.

[0019] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be derived from these drawings without inventive effort.

[0021] Figure 1A flowchart of a method for constructing a latent space world model for a robot grasping operation scenario provided by an embodiment of the present invention.

[0022] Figure 2 A schematic diagram of the improved DeepLabV3+ image semantic segmentation network structure provided by an embodiment of the present invention.

[0023] Figure 3 A schematic diagram of a channel attention network architecture provided by an embodiment of the present invention.

[0024] Figure 4 A schematic diagram of a spatial attention network architecture provided by an embodiment of the present invention.

[0025] Figure 5 A schematic diagram of a multi-task scene understanding network architecture provided by an embodiment of the present invention.

[0026] Figure 6 A schematic diagram of the structure of a shared feature encoder provided in an embodiment of the present invention.

[0027] Figure 7 A schematic diagram of the structure of a semantic segmentation image decoder provided by an embodiment of the present invention.

[0028] Figure 8 A schematic diagram of the structure of a unary symbol decoder provided by an embodiment of the present invention.

[0029] Figure 9 A schematic diagram of the structure of a binary relation decoder provided by an embodiment of the present invention.

[0030] Figure 10 A schematic diagram of a state transition model prediction process provided by an embodiment of the present invention.

[0031] Figure 11 A schematic diagram of the state transition network model structure provided by an embodiment of the present invention.

[0032] Figure 12 A schematic diagram of the semantic segmentation results of an improved DeepLabV3+ provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0034] Robotic grasping is a precise manipulation process performed by an end-effector, relying on sensors, motion planning, and end-effector adjustment. Traditional approaches are limited in complex scenarios by the inadequate representation of single features, poor model generalization, and inefficient multi-task coordination. They struggle to cope with issues such as occlusion, deformation, and multi-object interaction in dynamic environments.

[0035] In order to solve the above problems, an embodiment of the present invention provides a method for constructing a latent space world model for a robot grasping operation scenario. Figure 1 This is a flow chart of a method for constructing a latent space world model for a robot operation grasping scene provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps.

[0036] Step S101: semantically segment the scene image in the robot grasping operation task through an image semantic segmentation network to generate a semantic segmentation mask for each object in the work scene.

[0037] In step S102, the semantic segmentation mask is processed by a multi-task scene understanding network to generate an implicit representation. The implicit representation includes a position and shape feature identifier of the object, an existence identifier for indicating whether the object exists, and a contact identifier for characterizing the contact relationship between objects. The multi-task scene understanding network is trained by jointly optimizing the semantic segmentation mask image reconstruction loss, the object existence classification loss, and the contact relationship classification loss.

[0038] In step S103, based on the implicit representation of each object at the current moment and the robot's manipulator motion information, the implicit representation at the next moment is predicted through a state transition network. The motion information includes the motion type, action position, and direction parameters.

[0039] In step S104, based on the predicted implicit representation of the next moment, the decoder in the multi-task scene understanding network architecture generates scene segmentation image reconstruction results, object existence judgment results, and contact relationship judgment results to construct a latent space world model of the scene.

[0040] First, the scene images in the robot grasping operation task are semantically segmented through the image semantic segmentation network to generate semantic segmentation masks for each object in the work scene.

[0041] The purpose of the image semantic segmentation network is to perform semantic segmentation on the scene images in the robot grasping operation task and generate semantic segmentation masks for each object in the work scene. In this embodiment, the boundary and category information of each object can be extracted to provide basic data for the subsequent implicit representation generation. The semantic segmentation network can adopt a deep learning model, wherein the deep learning model can be DeepLabV3+, which realizes high-precision semantic segmentation through an encoder-decoder structure, and can capture the detailed information of objects in the scene, including edge-sensitive information and global semantic information. Through this segmentation method, objects in complex scenes can be distinguished, providing accurate data support for subsequent feature extraction and modeling.

[0042] Next, the semantic segmentation mask is processed by a multi-task scene understanding network to generate an implicit representation. In this embodiment, the implicit representation includes the position and shape feature identifiers of the object, the presence identifier indicating the existence of the object, and the contact identifier used to characterize the contact relationship between objects. The multi-task scene understanding network is trained by jointly optimizing the semantic segmentation mask image reconstruction loss, the object presence classification loss, and the contact relationship classification loss.

[0043] In this embodiment, multiple tasks (such as semantic segmentation mask image reconstruction, existence judgment and contact relationship recognition) are unified into a shared implicit representation through a multi-task scene understanding network, thereby improving the representation ability and efficiency of the model. The multi-task scene understanding network can be composed of a shared feature encoder, a semantic segmentation image decoder, a unary symbol decoder and a binary relationship decoder, wherein the shared feature encoder encodes the semantic segmentation mask to generate an implicit representation containing the object's position, shape and symbol information, and the semantic segmentation image decoder performs transposed convolution and convolution operations on the implicit representation to extract the object's position and shape information and reconstruct the object's semantic segmentation mask image, while the unary symbol decoder and the binary relationship decoder are used to judge the object's existence and contact relationship, respectively. By jointly optimizing the semantic segmentation mask image reconstruction loss, the object existence classification loss and the contact relationship classification loss, it is ensured that the implicit representation has good performance on multiple tasks.

[0044] Then, based on the implicit representation of each object at the current moment and the robot's manipulator motion information, a state transition network is used to predict the implicit representation at the next moment. The motion information includes the motion type, position, and direction parameters.

[0045] In this embodiment, the motion information of the robot arm is fused with the implicit representation through a state transition network to learn the mapping relationship between motion and state changes. The state transition network can include an action encoding module, a cross-modal fusion module, and a state update module. The action encoding module is responsible for encoding the action type, action position, and direction parameters. The cross-modal fusion module fuses the motion information with the implicit representation, and the state update module generates the implicit representation at the next moment by mapping the fused features. In this way, the model can predict the changes in the scene after the robot performs the action, thereby providing a basis for subsequent decision-making and control.

[0046] Finally, based on the predicted implicit representation of the next moment, the decoder in the multi-task scene understanding network architecture generates scene segmentation image reconstruction results, object existence judgment results, and contact relationship judgment results to construct a latent space world model of the scene.

[0047] In this embodiment, the decoder in the multi-task scene understanding network architecture may include a semantic segmentation image decoder, a unary symbol decoder, and a binary relationship decoder. The semantic segmentation image decoder is responsible for reconstructing the implicit representation into a scene segmentation image, the unary symbol decoder is used to determine whether an object exists, and the binary relationship decoder is used to identify the contact relationship between objects. Through the collaborative work of these decoders, high-quality semantic segmentation image reconstruction results and accurate symbol judgment results can be generated, thereby constructing a latent space world model that can reflect the dynamic changes of the scene. This process not only verifies the predictive ability of the model, but also provides reliable data support for the robot's autonomous decision-making in complex environments.

[0048] In an embodiment of the present invention, in order to construct a latent space world model, a grasping operation data acquisition system can be built through a simulation platform to overcome the limitations of real experiments, improve data acquisition efficiency, and provide reliable data support for vision-based grasping operation model training.

[0049] Specifically, while conducting experiments on a real-world experimental platform can verify the model's performance in real-world scenarios, it has certain limitations due to real-world conditions. For example, experiments in a real environment are expensive, and errors during debugging can damage the robot. In contrast, conducting experiments in a simulated environment allows for the acquisition of a large number of grasping operation samples, significantly reducing both time and cost.

[0050] Therefore, in this embodiment, a robot grasping operation scene can be built in the V-rep simulation platform to train the Visual Pushing for Grasping (VPG) model. The VPG model uses deep reinforcement learning (DRL) technology to enable the robot to gradually learn how to perform pushing and grasping operations through trial and error and feedback mechanisms. During the training process, the landing position of the object is randomly generated, and each training contains 5 objects. When the robot successfully grasps all objects, the system will automatically update the grasping environment and enter the next round of training. To construct a data set, the VPG model can save the RGB-D images captured by the camera and generate 3D point cloud data. Subsequently, by performing orthogonal reverse projection on the point cloud data in the direction of gravity, an RGB-D image from the perspective directly above is obtained. The data set used in this embodiment is prepared based on manually annotated color images.

[0051] In an optional embodiment, the image semantic segmentation network includes a DeepLabV3+ encoder and a DeepLabV3+ decoder with an added attention mechanism.

[0052] Among them, when performing semantic segmentation on the scene image in the robot grasping operation task through the image semantic segmentation network and generating the semantic segmentation mask of each object in the working scene, the following steps can be performed:

[0053] First, the Atrous Spatial Pyramid Pooling (ASPP) module in the DeepLabV3+ encoder can be used to extract multi-scale features from the scene image. Then, after attention-weighted feature fusion and 1×1 convolution, the deep feature map is output. Finally, the DeepLabV3+ decoder upsamples the deep feature map and cross-layers it is fused with the shallow features of the DeepLabV3+ encoder. After convolution and upsampling, the semantic segmentation mask is output.

[0054] Robotic grasping scenarios often involve multiple objects of different types, each with different semantic information. By performing semantic image segmentation on objects in scene images, we can not only obtain information about the object's position and shape, but also its semantic information, thereby enabling an understanding of the robot's operating environment. To better accomplish this task, this example introduces an attention mechanism to improve the classic DeepLabV3+ image semantic segmentation network, enhancing its image semantic segmentation performance in complex grasping scenarios.

[0055] After multi-scale feature extraction, the ASPP module introduces an attention mechanism for weighted feature fusion, and then uses 1×1 convolution to obtain deep features and input them into the decoder. The core idea of ​​the attention mechanism is to imitate the human visual system and make the model pay more attention to key areas by weighting the input information. The decoding part first receives shallow features and reduces the number of channels through 1×1 convolution to maintain consistency. It is then stacked with the upsampled deep features and then subjected to 3×3 convolution for feature extraction. After four upsamplings, a high-precision semantic segmentation mask is finally generated. The improved DeepLabV3+ image semantic segmentation network structure in this embodiment is as follows: Figure 2 shown.

[0056] In this embodiment, the attention mechanism introduced for the above-mentioned image semantic segmentation network combines the channel attention and spatial attention mechanisms.

[0057] Specifically, if Figure 3 As shown in Figure 2, the channel attention mechanism extracts channel-level features through global average pooling, global maximum pooling, and normalized pooling, and then calculates the channel attention weight through a two-layer MLP structure. The specific formula is as follows:

[0058]

[0059] Among them, M c is the channel attention weight; F is the input feature; is the channel feature vector obtained by performing global average pooling on the input features, is the channel feature vector obtained by performing global maximum pooling on the input features, is the channel feature vector obtained by normalizing and pooling the input features, σ is the Sigmoid function; W0 and W1 are the weights of MLP.

[0060] like Figure 4 As shown in Figure 2, the spatial attention mechanism focuses on the importance of different spatial locations in the input features. By performing average pooling, maximum pooling, and normalized pooling along the channel axis, the three feature maps are concatenated and fed into a 7×7 convolutional layer to generate the spatial attention weights. The formula is as follows:

[0061]

[0062] Among them, M s is the generated spatial attention weight; F is the input feature; It is the spatial feature map obtained by average pooling along the channel axis; It is the spatial feature map obtained by performing maximum pooling along the channel axis; is the spatial feature map obtained by normalizing the pooling along the channel axis; f 7×7 is a 7×7 convolution operation; σ is a Sigmoid function.

[0063] This example combines channel attention and spatial attention mechanisms and embeds them into the DeepLabV3+ image semantic segmentation network to form an improved DeepLabV3+ image semantic segmentation network. This improves the model's performance on image semantic segmentation tasks, especially in complex capture scenarios, achieving results superior to the original image semantic segmentation network and other attention-enhanced models.

[0064] In an optional embodiment, the multi-task scene understanding network includes a shared feature encoder, a semantic segmentation image decoder, a unary symbol decoder, and a binary relation decoder.

[0065] In this embodiment, processing the semantic segmentation mask by a multi-task scene understanding network to generate an implicit representation includes the following steps:

[0066] The semantic segmentation mask is encoded through a shared feature encoder to generate an implicit representation containing the object's position, shape, and symbol information; the semantic segmentation image decoder performs transposed convolution and convolution operations on the implicit representation to extract the object's position and shape information and reconstruct the object's semantic segmentation mask image; the unary symbol decoder determines whether the object exists in the current scene based on the implicit representation and outputs a binary existence identifier; based on the implicit representation, the binary relationship decoder calculates the spatial interaction features between the two objects based on the ACmix self-attention and convolution hybrid module to generate a contact relationship identifier.

[0067] When planning robotic manipulations based on observational data, a key challenge is learning an appropriate representation of the manipulation scene. Scene understanding tasks can learn implicit representations that can be transferred to unseen objects. However, implicit representations containing only abstract concepts are insufficient for manipulation planning; more detailed information, such as the location and shape of objects, is required.

[0068] To this end, this embodiment provides the above-mentioned multi-task scene understanding network, which is composed of a shared feature encoder, a semantic segmentation image decoder, a unary symbol decoder, and a binary relationship decoder. Through the multi-task scene understanding network, the semantic segmentation information and the symbol information are uniformly encoded into a space, and the semantic information and symbol information of each object are encoded into an implicit representation in the form of a coding matrix. The architecture diagram of the multi-task scene understanding network is shown as follows Figure 5 shown.

[0069] In this embodiment, the shared feature encoder can extract features from the input semantic segmentation mask to generate an implicit representation. Furthermore, it can extract the object's position, shape, and symbol information, providing basic features for subsequent semantic segmentation image reconstruction, existence judgment, and contact relationship reasoning.

[0070] Specifically, the shared feature encoder can take the single object segmentation image containing semantic information output by the improved DeepLabV3+ semantic segmentation network in the above embodiment as input, and use the output of the shared feature encoder as the learned implicit representation. The implicit representation needs to be as simple as possible, so the input needs to be compressed and reduced in dimension. The structure of the shared feature encoder is as follows: Figure 6 As shown in Figure 2, a multi-layer network structure offers enhanced feature extraction capabilities, avoiding the limitations of traditional machine learning algorithms that require manual feature selection. Therefore, the shared feature encoder employs a four-layer convolutional pooling structure. The shared feature encoder compresses and reduces the dimensionality of the input image. The learned implicit representation is then fed into the semantic segmentation image decoder, the unary symbol decoder, and the binary relation decoder, respectively, to decode different types of information.

[0071] In this embodiment, the semantic segmentation image decoder can extract the position and shape information of the object and reconstruct the semantic segmentation mask image of the object by performing transposed convolution and convolution operations on the implicit representation.

[0072] Specifically, the semantic segmentation image decoder takes the implicit representation extracted by the shared feature encoder as input and outputs the restored object segmentation image, such as Figure 7 As shown in Figure 3, the semantic segmentation image decoder's network architecture consists of four layers of transposed convolutions and one convolutional layer. The semantic segmentation image decoder is responsible for converting the features extracted by the shared feature encoder back to the original image space. This process typically involves gradual upsampling or transposed convolution to restore the image size. Therefore, the semantic segmentation image decoder uses four layers of transposed convolutions to gradually increase the dimensionality and reconstruct the input image. Furthermore, a convolutional layer is added to adjust the channels, and backpropagation is used to optimize the network structure parameters, enabling the semantic segmentation image decoder to accurately restore the input image.

[0073] In this embodiment, the unary symbol decoder can determine whether an object exists in the current scene based on implicit representation. It can output a binary existence identifier (such as "existence" or "non-existence") through classification processing to assist the robot in determining the state of objects in the current scene.

[0074] Specifically, the unary symbol decoder is used to determine whether a specific object exists in the image. It takes the learned implicit representation as input and outputs a binary result of whether the object exists in the image. The network structure of the unary symbol decoder is as follows: Figure 8As shown in the figure, it includes two Dropout layers and two fully connected layers. The unary symbol decoder first performs dimensionality conversion on the learned implicit representation and then sends it to the Dropout layer. During the training process, the Dropout layer randomly sets the output of some neurons to zero, thereby preventing the network from over-relying on specific neurons and enhancing the generalization ability of the model. The processed feature data is then sent to the fully connected layer. After two repeated processes, the final output is obtained, thereby realizing the judgment of the existence of objects in the image.

[0075] In this embodiment, the binary relation decoder can calculate the spatial interaction features between two objects based on implicit representations. Furthermore, it can use the attention mechanism to capture the relative positions and overlapping areas between the objects, and output contact markers to determine whether there is a contact relationship between the objects (such as collision, overlap, etc.).

[0076] Specifically, the binary relation decoder is used to determine whether there is contact between objects. Its input is the two implicit representations of the semantic segmentation images of the two objects output by the shared feature encoder. The fusion method adopts image weighted average fusion. This method is simple to implement and suitable for fast image processing requirements. In addition, pixel-level operations require less computing resources and time, so it is more practical in environments with limited computing resources.

[0077] The output of the binary relation decoder is used to indicate whether objects are in contact with each other. Figure 9 As shown in Figure 3, the network structure of the binary relation decoder consists of an ACmix self-attention and convolution hybrid module, two repeated Dropout layers, and a fully connected layer, where the ACmix module combines the self-attention mechanism with the convolution operation.

[0078] For the self-attention path, the intermediate features are divided into N groups, each containing 3 features. Each feature comes from a 1×1 convolution, and the corresponding three feature maps are used as query, key, and value respectively. Following the traditional multi-head self-attention module, the features are aggregated by calculating the attention weight and the value matrix. The specific formulas are shown in (3) and (4):

[0079]

[0080] Among them, f ij is the tensor corresponding to pixel (i, j); The projection matrices of query, key and value respectively; represents the concatenation of the outputs of N attention heads; N k (i, j) represents the region centered at (i, j) with a spatial range of k; N k Attention weight of the feature within (i,j); gij is the output corresponding to pixel (i, j).

[0081] For the convolution path, the input features are linearly projected and then subjected to translation aggregation operations. The specific formulas are shown in (5), (6), and (7):

[0082]

[0083] Among them, K p,q is the kernel weight; f ij is the tensor corresponding to pixel (i, j); is the linear projection of the input feature; Shift is the translation operation; k is the convolution kernel size; g ij is the convolution output after translation aggregation.

[0084] The output of the self-attention path is set to Fatt, and the output of the convolution path is set to Fconv. Finally, the two are weightedly added together to obtain the final output, as shown in formula (8):

[0085] F out =αF att +βF conv (8)

[0086] Among them, α and β are two learnable path weights.

[0087] In an optional embodiment, the multi-task scene understanding network can be trained based on the following method.

[0088] A multi-task training dataset is constructed, where each set of training samples contains the semantic segmentation mask images, object existence labels, and contact relationship labels corresponding to the objects in the grasping operation scene; the semantic segmentation mask images are feature extracted through a shared feature encoder to generate an implicit representation; the implicit representation is transposed and convolved through a semantic segmentation image decoder to output a reconstructed image, which is compared with the semantic segmentation mask image to calculate the semantic segmentation mask image reconstruction loss; the implicit representation is classified through a unary symbol decoder to predict the existence state of the object, and the cross-entropy loss is calculated as the object existence classification loss by comparing it with the object existence label; the spatial relationship between objects is classified through a binary relation decoder, the contact relationship between objects is predicted, and the cross-entropy loss is calculated as the contact relationship classification loss by comparing it with the contact relationship label; a joint loss function is constructed based on the semantic segmentation mask image reconstruction loss, object existence classification loss, and contact relationship classification loss, and the parameters of the multi-task scene understanding network are back-propagated and updated until the model converges.

[0089] In this embodiment, the multi-task model differs significantly from the single-task model. Multi-task learning enables the coordinated optimization of multiple tasks by sharing knowledge, while the backpropagation algorithm is used to optimize the neural network corresponding to each task. Therefore, in this embodiment, multi-task learning can improve the model's generalization ability through parameter sharing, thereby achieving better performance on multiple tasks.

[0090] For the semantic segmentation mask image reconstruction loss, this embodiment uses a binary cross entropy loss function, as shown in formula (9):

[0091] BCE(y,p)=-ylog(p)-(1-y)log(1-p) (9)

[0092] Among them, y represents the true label of the sample; p represents the predicted probability output by the model.

[0093] This loss function has obvious advantages in binary classification problems and can be directly applied to models that output probabilities without the need for additional conversion or post-processing steps.

[0094] For the object existence classification loss and contact relationship classification loss, this embodiment adopts the cross entropy loss function, as shown in formula (10):

[0095]

[0096] Among them, y i represents the true category label, p i Represents the predicted class probability.

[0097] This loss function can optimize the performance of the recognition model by setting category weights, and can work well with the optimizer Adam to achieve efficient model training.

[0098] The final total loss L is obtained by adding the output losses of the three decoders, as shown in formula (11):

[0099] L=BCE1+Loss1+Loss2 (11)

[0100] Among them, BCE1, Loss1, and Loss2 correspond to semantic segmentation mask image reconstruction loss, object existence classification loss, and contact relationship classification loss, respectively.

[0101] The back-propagation algorithm is used to calculate the gradient of the total loss function with respect to the model parameters, and the Adam optimization algorithm is used to update the model parameters according to the gradient value to gradually reduce the loss and improve the model performance.

[0102] In an optional embodiment, the state transition network includes an action encoding module, a cross-modal fusion module, and a state update module. In this embodiment, the state transition network can predict the implicit representation at the next moment based on the implicit representation of each object at the current moment and the robot's manipulator motion information.

[0103] Specifically, the action type is embedded and encoded through the action encoding module, and the action position is encoded through the action encoding module, and the angle feature of the direction parameter is encoded to generate an action semantic vector, and a spatial action feature map is generated by repeated filling. The implicit representation of each object at the current moment is spliced ​​and fused with the spatial action feature map through the cross-modal fusion module, and the atrous spatial pyramid pooling (ASPP) is introduced to process the fused information to achieve multi-scale feature extraction. The coordinate attention (CA) mechanism is then introduced to enhance the feature response of important positions and channels, and generate an action-aware feature tensor. Finally, the state update module maps the action-aware feature tensor through a fully connected network and outputs the implicit representation of the next moment.

[0104] In this embodiment, the main function of the motion encoding module is to encode the motion information of the robot arm (including motion type, action position and direction parameters) and convert it into a numerical form suitable for model input. Through encoding, the original information is converted into a unified feature vector, which is convenient for subsequent fusion with other data (such as implicit representation). The specific implementation method is as follows:

[0105] Action type encoding: Encode “grab” and “push” as binary values ​​(e.g., “grab” = 1, “push” = 0) to distinguish different action categories;

[0106] Direction coding: Use integers from 0 to 15 to encode 16 possible direction information;

[0107] Position encoding: The pixel coordinate information is directly encoded as a numerical value and input as a two-dimensional coordinate.

[0108] The action encoding module can improve the model's processing efficiency of action information, generate spatial action feature maps through repeated filling, and realize the alignment of action information with the implicitly represented multimodal feature space.

[0109] The function of the cross-modal fusion module is to perform multimodal fusion on the spatial motion feature map output by the motion encoding module and the implicit representation of each object at the current moment to generate a motion-aware feature tensor.

[0110] The cross-modal fusion module combines the implicit representations of each object at the current moment with the spatial action feature map. Atrous Spatial Pyramid Pooling (ASPP) is then introduced to process the fused information, enabling multi-scale feature extraction. The Coordinate Attention (CA) mechanism is then introduced to enhance the feature responses of important locations and channels, generating an action-aware feature tensor. This module achieves joint modeling of action information and scene state information, enhancing the model's understanding of the action-state interaction during robot operation and improving the accuracy and robustness of state transition prediction.

[0111] The function of the state update module is to map the implicit representation of the current moment based on the action perception feature tensor, thereby predicting the implicit representation of the next moment. The specific implementation methods include:

[0112] The action-perception feature tensor is fed into a fully connected network to predict the implicit representation for the next moment. This module enables the model to predict state changes after action execution in complex scenarios, supporting real-time state prediction and decision-making for robots in dynamic environments.

[0113] In robotic grasping operations, the state transition network is a key module for achieving dynamic environmental perception and autonomous decision-making. This model needs to predict the implicit representation of the next moment based on the implicit representation of the current scene and the motion information of the robotic arm. However, traditional methods lack effective modeling of the relationship between motion information and scene changes during training, resulting in inaccurate prediction results. In this embodiment, the state transition network can be trained based on the following method:

[0114] A state transition training dataset is constructed, where each set of training samples contains the implicit representation data of the current moment, the robot arm motion information data, and the implicit representation label data of the next moment; the motion information data is encoded through the action encoding module to generate a motion semantic vector and a spatial motion feature map; the implicit representation data of the current moment is spliced ​​and fused with the spatial motion feature map through the cross-modal fusion module, and the atrous spatial pyramid pooling (ASPP) is introduced to process the fused information to achieve multi-scale feature extraction, and the coordinate attention (CA) mechanism is introduced to enhance the feature response of important positions and channels to generate a motion-aware feature tensor; the state update module maps the motion-aware feature tensor through a fully connected network and outputs the predicted implicit representation of the next moment; the MSE loss value between the predicted implicit representation and the implicit representation label data of the next moment is calculated, and the parameters of the state transition network are back-propagated and updated until the model converges.

[0115] The state transition network provided by the embodiment of the present invention is described in further detail below.

[0116] To improve the ability to predict scene state changes in robotic grasping scenarios, this embodiment provides a state transition network model based on implicit representations. By training a multi-task scene understanding encoder-decoder network, the learned implicit representation not only includes object position and shape information, but also includes symbolic information used to determine the presence of objects and whether objects are in contact.

[0117] On this basis, this embodiment uses a deep neural network architecture to construct a state transition model. This model takes the implicit representation of the current moment and the action information of the robot arm as input, and outputs the implicit representation of the next moment after the robot arm performs the corresponding action. The state transition model is trained by using a data set of grasping operation scenes, so that the model can accurately predict the implicit representation of the next moment after the robot arm performs the corresponding action, and generates the corresponding state output (including the position, shape and symbol information of the objects in the scene at the next moment) through the trained decoder.

[0118] To ensure that the state transition model can effectively predict state changes in actual grasping operations, this example uses a large amount of pushing and grasping data collected by the VPG model to construct a training dataset. The VPG model is trained over 5,000 steps, recording image data before and after each robotic arm movement, and saving the action category (push or grasp), the pixel location, and the direction of the action.

[0119] Furthermore, this embodiment uses a trained multi-task scene understanding encoder-decoder network to convert semantically segmented image information of each object in the grasping operation scene into an implicit representation. This implicit representation, as the corresponding state information, is fused with the robot's motion category, direction, and position information. Encoding vectors are used to convert the robot's motion information into numerical form for easier computer processing. The encoding vectors have a fixed dimension and structure, which helps improve the efficiency of model training and inference.

[0120] Specifically, the robot's grasping and pushing actions are encoded as 1 and 0, respectively; there are 16 types of direction information, encoded from 0 to 15; and position coordinate information is encoded as pixel coordinate values. A spatial action feature map is then generated through repeated filling, mapping the robot's motion information and state information into the same feature space, achieving effective fusion of multimodal data.

[0121] Figure 10 A schematic diagram of a state transition model prediction process provided by an embodiment of the present invention. Figure 10 The state transition model prediction process based on implicit representation in this embodiment is as follows:

[0122] First, the semantically segmented image is fed into a trained shared feature encoder to obtain an implicit representation of each object. This implicit representation is then fused with the robot's action category, position, and orientation information in a latent space and fed into a trained state transition model. Based on this input, the model predicts the implicit representation at the next moment. Finally, three trained decoders are used to generate a predicted semantic segmentation mask image, determine the presence of an object, and identify contact relationships between objects.

[0123] The state transition model network constructed in this embodiment is as follows Figure 11 As shown in Figure 2, its input is the implicit representation of each object at the current moment and the robot's motion information. The output is the implicit representation of the next moment after the robot performs the action. By training the state transition model, it can accurately predict the implicit representation at the next moment, thereby modeling the scene state changes.

[0124] To improve model performance, this embodiment introduces atrous spatial pyramid pooling (ASPP) in the state transition model to process the fused information and achieve multi-scale feature extraction. The coordinate attention (CA) mechanism is then introduced to enhance the feature response of important positions and channels. The Coordinate Attention (CA) module alleviates the gradient vanishing problem through a residual processing mechanism and jump connections, accelerating model convergence. In addition, to address the problem that traditional average pooling cannot distinguish spatial directions, the CA module introduces one-dimensional horizontal average pooling and one-dimensional vertical average pooling to retain spatial information in the horizontal and vertical directions, respectively.

[0125] The horizontal average pooling formula is as follows:

[0126]

[0127] The vertical average pooling formula is as follows:

[0128]

[0129] The input feature information is c is the number of channels; H is the height; W is the width; The output has c channels and h height; The output is c with a channel width of w.

[0130] After completing the horizontal and vertical pooling, the feature information is spliced ​​and processed through the shared convolution operation to obtain the intermediate feature representation f, which is as follows:

[0131] f=δ(C([z h ,z w ])) (14)

[0132] Among them, δ is a nonlinear activation function; C is a shared convolution operation; z h and z w They are the pooling outputs in the horizontal and vertical directions respectively.

[0133] Then, the feature representation f is decomposed into a height-wise tensor f along the spatial dimension h With the width tensor f w , and adjust the number of channels through two convolutional layers respectively, and then generate the weights in the height direction and width direction through the nonlinear activation function:

[0134]

[0135] Among them, g h 、g w is the attention weight in the height and width directions; σ is the Sigmoid function; C h 、C w is the convolution operation.

[0136] The final output is:

[0137]

[0138] Among them, y c (i, j) is the output of the CA module; x c (i, j) is the output of the ASPP module; are the attention weights in the height and width directions respectively.

[0139] Finally, the processed feature information is input into the fully connected network to predict the implicit representation of the next moment. The model uses mean square error (MSE) as the loss function and updates the network parameters through backpropagation.

[0140] In an alternative embodiment, the decoder comprises Figure 10 The semantic segmentation image decoder, unary symbol decoder, and binary relation decoder shown.

[0141] In this embodiment, the implicit representation of the next moment is decoded by a semantic segmentation image decoder to generate a pixel-level semantic segmentation image reconstruction result; the existence identifiers in the implicit representation of the next moment are classified and processed by a unary symbol decoder to output a binary judgment result of whether each object exists; the contact identifiers in the implicit representation of the next moment are analyzed by a binary relationship decoder to generate a contact relationship judgment result between each object.

[0142] According to embodiments of the present invention, to accurately predict scene changes, it is necessary to extract geometric information (such as object position and shape), presence information (determining whether an object exists in the current scene), and contact relationship information (identifying whether objects are touching or overlapping) from the implicit representation. A traditional single decoder struggles to simultaneously meet these requirements, thus requiring the multi-task decoder architecture provided by this embodiment that can process multiple types of information.

[0143] The decoder architecture consists of three modules: a semantic segmentation image decoder, a unary symbol decoder, and a binary relationship decoder. The semantic segmentation image decoder takes the implicit representation as input and outputs pixel-level semantic segmentation image reconstruction results, which are used to predict the visual changes in the scene after the robot moves and support subsequent decision-making and control. The unary symbol decoder takes the implicit representation as input and outputs a binary judgment result on the existence of each object. Classification is performed through a fully connected network to achieve accurate judgment of the object's existence status. The binary relationship decoder takes the implicit representation as input and outputs the contact relationship judgment result between each object. Based on the ACmix self-attention and convolution hybrid module, it improves the accuracy and robustness of contact relationship recognition.

[0144] The decoder shares the implicit representation from the state transition network as input and outputs three types of information: semantic segmentation image reconstruction results, existence, and contact relationship. This avoids the information inconsistency problem caused by using independent models for different tasks in traditional methods. The semantic segmentation image decoder gradually restores the spatial resolution of the image through multiple layers of transposed convolution, so that the decoded image can retain rich detail information, thereby more accurately reflecting the changes in the scene. The unary symbol decoder classifies the implicit representation through a fully connected network and outputs a binary result, which is used to judge the existence status of objects in the scene and support the robot's perception and decision-making of the environment. The binary relationship decoder introduces the ACmix self-attention and convolution hybrid module to model the spatial relationship between pairs of objects and improve the accuracy of contact relationship recognition.

[0145] The following describes in detail a method for constructing a latent space world model for a robot grasping operation scenario provided by an embodiment of the present invention in conjunction with specific embodiments.

[0146] This embodiment is carried out in the Ubuntu 20.04 operating system environment using Python 3.7. The encoder and decoder parts are built based on the deep learning framework PyTorch, and are used with the corresponding versions of CUDA and the acceleration library cuDNN to improve computing efficiency. During the training process, the Adam optimizer is used to optimize the model. In order to speed up the training speed of the multi-task scene understanding encoder-decoder network model, the improved DeepLabV3+ semantic segmentation network and the state transition model, this embodiment uses GPU (graphics processing unit) acceleration. The GPU has high parallelism and can process different tasks at the same time, significantly improving the data processing speed. In terms of multi-task scene understanding encoder-decoder model training, the model uses the Adam gradient optimization algorithm, the number of training rounds is 2000, the momentum is set to 0.9, the weight decay is set to 1e-4, the initial learning rate is 1e-5, and the batch size is set to 5. The momentum parameter and weight decay rate of the state transition model are set to 0.9 and 1e-3 respectively, the number of training rounds is 1000, the initial learning rate is 1e-4, and the training batch size is set to 5.

[0147] In the semantic segmentation stage, during the training of VPG, the depth camera in the simulation environment will save the image information of the current scene. The VPG model projects the saved RGB-D image onto the 3D point cloud, and after orthogonal back-projection in the direction of gravity, generates an RGB-D image from the perspective directly above. VPG was trained for 5000 steps on the Ubuntu operating system, and LabelMe was used to select the pictures saved in the VPG training for annotation processing to prepare a semantic segmentation dataset. The dataset consists of three parts: training set, validation set, and test set. First, the improved DeepLabV3+ model with the added attention mechanism was trained, and the results showed that good results were achieved in the training. The mIoU value reached 90.8% in the training set. In order to verify whether the improved DeepLabV3+ can perform segmentation in complex scenes, a captured scene picture was selected for verification, and the segmentation results are shown as follows. Figure 12 shown.

[0148] To verify the generalization ability of the improved DeepLabV3+ model, the optimal weight parameters of the network were saved during the training process. After training, images that had never been trained were selected from the test set, and the segmentation performance of the model was evaluated using the mIoU value, mPA value, and mPrecision value. The experimental results demonstrated the accuracy of the model in classification. Among them, the experimental results showed mIoU (90.94%), mPA (93.16%), and mPrecision (97.42%). The comprehensive experimental results indicate that the model has strong generalization ability and can perform good segmentation of various objects in the image.

[0149] At the same time, to verify whether the DeepLabV3+ network with the improved attention mechanism can achieve better results, a control experiment was conducted. The improved DeepLabV3+ network was compared with the original DeepLabV3+ network, the DeepLabV3+ network model with the SEAttention module for the channel attention mechanism, the DeepLabV3+ network model with the CoordAttention module for the spatial attention mechanism, and the U-Net and PSP-Net network models. The experimental results are shown in Table 1.

[0150] Table 1 Experimental results of different semantic segmentation networks

[0151]

[0152] Experimental results show that the improved DeepLabV3+ model achieves superior results compared to other networks in terms of mIoU, mPA, and mPrecision on the test set, demonstrating its superior semantic segmentation capabilities. Furthermore, during training, it was found that while additional computation was introduced to the network, the additional parameters and computational effort were relatively small, significantly improving model performance with little impact on model size or inference speed.

[0153] After semantic segmentation is completed, the output of semantic segmentation is input into the multi-task scene understanding encoder-decoder network model to start predicting and analyzing the position and shape of objects in the scene, whether objects exist, and whether objects are in contact with each other, and the network is trained through backpropagation.

[0154] After 2000 rounds of training, when the total loss stabilized, the unary symbol decoder and binary relation decoder achieved an accuracy of 100% and 97.5% on the training set, and an accuracy of 100% and 97.2% on the validation set. The semantic segmentation image decoder can also effectively restore the position and shape of objects.

[0155] To verify the model's accuracy in various grasping scenarios, five different grasping scenarios were selected during the VPG training process and the current scene was judged. The objects in these five scenarios had different contact states. After semantic segmentation was performed on these five scenes to obtain individual object images, the model performed predictions and analysis on the position and shape of the five different objects in the scene, their presence, and whether they were in contact with each other. Statistics were also collected for each of the five different objects in each scene. The experimental results are shown in Table 2.

[0156] Table 2 Experimental results of multi-task scene understanding encoder-decoder in different scenarios

[0157] Scenario Unary symbol decoding success rate Success rate of binary relationship decoding Test scenario a 100％ 97.4％ Test scenario b 100％ 96.5％ Test scenario c 100％ 94.8％ Test scenario d 100％ 96.5％ Test scenario 100％ 98.2％

[0158] The prediction results show that in five different grasping scenarios, the trained model can accurately judge whether objects exist in the scene and whether objects are in contact with each other. At the same time, experiments found that the semantic segmentation image decoder can effectively restore different objects in different scenarios, proving the effectiveness of the designed multi-task scene understanding encoder-decoder.

[0159] Multi-task deep neural networks offer significant advantages in learning task dependencies and enhancing generalization capabilities, particularly when processing multiple related tasks. However, if the dependencies between tasks are unclear or conflicting, multi-task networks can be affected by inter-task interference, leading to performance degradation. Furthermore, in order to achieve balanced performance across multiple tasks, multi-task networks may not be able to achieve the same level of performance on a single task as single-task networks.

[0160] In order to fully verify the feasibility of the multi-task scene understanding encoder-decoder designed in this embodiment, a set of experimental controls were carried out, in which the encoder-semantic segmentation decoder combination was called model A, the encoder-unary symbol decoder combination was called model B, and the encoder-binary relationship decoder combination was called model C. A total of six models were set up for comparison with the multi-task model, including three single-task models and three dual-task models. For the single-task model, the A, B and C models were trained and evaluated independently to verify their performance in semantic segmentation image decoding, unary symbol decoding and binary relationship decoding. For the dual-task model, the encoder and the three decoders were trained in pairs to verify the impact of dual-task learning on the model training effect.

[0161] Finally, the multi-task model combines all tasks of the encoder and the three decoders to verify whether the network's ability to maintain its original accuracy when learning multiple tasks simultaneously in one model is verified. The experimental results are shown in Table 3.

[0162] Table 3 Multi-task scene understanding encoder-decoder comparison experiment

[0163]

[0164] The results show that the multi-task scene understanding encoder-decoder model maintains considerable accuracy on both the training and validation sets compared to single-task and dual-task encoder-decoder networks, demonstrating that the multi-task model provided by this embodiment can maintain good performance when handling multiple tasks. By training the multi-task scene understanding encoder-decoder network model, the learned implicit representation can be used to reconstruct semantically segmented images of objects in the scene to obtain object position and shape information, and to determine whether objects exist and whether objects are in contact. This can serve as an effective representation method for learning scene representations for robotic grasping operations.

[0165] In the state transition model experiments, the state transition dataset adds information about the robot's action category (push or grasp), pixel location, and direction to the image dataset. The state transition model takes the implicit representation of the semantically segmented image and the robot's action information as input and outputs the implicit representation at the next moment. The dataset label consists of the implicit representation obtained by passing the semantically segmented image at the next moment after the robot performs the action through a multi-task encoder-decoder. To improve the model's generalization and robustness, not only samples of successful grasps and effective pushes are selected, but also samples of failed grasps or ineffective pushes are used as controls.

[0166] The semantic segmentation model is used to semantically segment objects in the grasping operation scene, obtaining a semantic segmentation image of the object. This semantic segmentation image is then input into a trained multi-task encoder-decoder to obtain an implicit representation. Simultaneously, the robot's motion information is encoded to obtain a one-dimensional encoding vector, which is then multiplied by repeated padding to generate an action feature map. The action feature map is concatenated and fused with the implicit representation and then fed into a state transition network model to generate a predicted implicit representation for the next moment. The predicted implicit representation is compared with the labeled data to construct a loss function, and the state transition network model is trained. The trained model can generate an implicit representation for the next moment after the robot performs an action. The implicit representation can then be decoded using a semantic segmentation image decoder, a unary symbol decoder, and a binary relation decoder to obtain object information in the scene at the next moment.

[0167] The model was trained for 3000 rounds. After completion, four different scene images were selected for prediction, corresponding to two grasping and two pushing maneuvers. After the state transition model predicted the implicit representation for the next moment, the resulting implicit representation was fed into the semantic segmentation image decoder of the trained multi-task encoder-decoder for decoding. The results showed that the decoded image based on the predicted implicit representation for the next moment accurately reflected the changes in the object after the maneuver was performed. The quality of the generated implicit representation and its performance in the multi-task encoder-decoder were then evaluated.

[0168] The DICE coefficient is a metric used to evaluate the quality of binary images generated by generative models. Its core concept is to measure the performance of the generative model by comparing the overlap between the generated binary image and the ground-truth binary image in the target area. The results show an average DICE coefficient evaluation of 97.8%, demonstrating that the state transition model designed in this embodiment is capable of generating high-quality implicit representations and predicting object changes through the decoder.

[0169] After evaluating the quality of image generation, we evaluate whether the generated implicit representation can correctly reflect the existence of objects and the contact relationships between objects. The generated implicit representation is input into the trained unary symbol decoder and binary relation decoder to determine whether they can correctly identify the existence of objects and the contact relationships between objects.

[0170] Experimental results show that the implicit representation generated by the state transfer model successfully determines the presence of objects and the contact between them within a multi-task encoder-decoder framework. The model achieved 100% accuracy in the object presence task, meaning it correctly predicted the presence of an object in a scene image. In the object contact task, the model achieved 91% accuracy, demonstrating high accuracy in correctly predicting contact between objects. Experimental results demonstrate that, based on the implicit representation generated by the state transfer model, the decoder can effectively predict the presence and contact of objects.

[0171] In order to verify whether the trained model still has good prediction results in different grasping operation scenarios, five different grasping operation scenarios are selected to predict the objects in the scenes, and the prediction results of different objects in the five scenarios are statistically analyzed. The experimental results are shown in Table 4.

[0172] Table 4 Experimental results of the state transition model in different scenarios

[0173] Scenario DICE Unary symbol decoding success rate Success rate of binary relationship decoding Test scenario a 96.69％ 100％ 92.8％ Test scenario b 97.15％ 100％ 91.1％ Test scenario c 96.33％ 100％ 89.3％ Test scenario d 97.45％ 100％ 90.2％ Test scenario 97.85％ 100％ 93.7％

[0174] As shown in Table 4, the model is able to determine the existence of objects and the contact between them in different scenarios, and predict the segmented image at the next moment using the semantic segmentation image decoder. The predicted images generated by semantic segmentation image decoding achieved an average DICE evaluation score of 97.1%, demonstrating that the state transition model provided by this embodiment can achieve excellent results in different scenarios. Testing found that predicting the scene after grasping was more effective than predicting the scene after pushing. Different push positions may lead to deviations in the predicted image.

[0175] Based on the above-mentioned method for constructing a latent space world model for a robot grasping operation scene provided by an embodiment of the present invention, an embodiment of the present invention further provides a device for constructing a latent space world model for a robot grasping operation scene, the device comprising:

[0176] The segmentation module is used to perform semantic segmentation on the scene image in the robot grasping operation task through the image semantic segmentation network, and generate semantic segmentation masks for each object in the work scene;

[0177] A processing module is used to process the semantic segmentation mask through a multi-task scene understanding network to generate an implicit representation. The implicit representation includes the position and shape feature identifiers of the object, the existence identifier for indicating whether the object exists, and the contact identifier for representing the contact relationship between objects. The multi-task scene understanding network is trained by jointly optimizing the semantic segmentation mask image reconstruction loss, the object existence classification loss, and the contact relationship classification loss;

[0178] The update module is used to predict the implicit representation of each object at the next moment through the state transition network based on the implicit representation of each object at the current moment and the robot's manipulator motion information. The motion information includes the motion type, action position and direction parameters;

[0179] The construction module is used to generate scene segmentation image reconstruction results, object existence judgment results, and contact relationship judgment results based on the predicted implicit representation of the next moment through the decoder in the multi-task scene understanding network architecture, and build a latent space world model of the scene.

[0180] An embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the method of an embodiment of the present invention.

[0181] An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method of the embodiment of the present invention.

[0182] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to perform the method of the embodiment of the present invention.

[0183] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open-ended, i.e., "including but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; and the term "some embodiments" means "at least some embodiments." The modifications of "one" and "a plurality of" mentioned in the embodiments of the present invention are illustrative and non-restrictive. Those skilled in the art should understand that, unless the context clearly indicates otherwise, they should be understood as "one or more."

[0184] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0185] The various steps described in the method implementation scheme provided in the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation scheme may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0186] The term "embodiment" in this specification refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments are referenced to each other. In particular, for the embodiments of the device, equipment, and system, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the partial description of the method embodiment.

[0187] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. A method for constructing a latent space world model for a robot grasping operation scene, characterized in that: include: The image semantic segmentation network is used to perform semantic segmentation on the scene images in the robot grasping operation task, and the semantic segmentation mask of each object in the working scene is generated; The semantic segmentation mask is processed by a multi-task scene understanding network to generate an implicit representation, wherein the implicit representation includes a position and shape feature identifier of the object, an existence identifier for indicating whether the object exists, and a contact identifier for characterizing the contact relationship between objects. The multi-task scene understanding network is trained by jointly optimizing the semantic segmentation mask image reconstruction loss, the object existence classification loss, and the contact relationship classification loss; Based on the implicit representation of each object at the current moment and the robot's manipulator motion information, predicting the implicit representation at the next moment through a state transition network, the motion information including motion type, action position and direction parameters; Based on the predicted implicit representation of the next moment, the decoder in the multi-task scene understanding network architecture generates scene segmentation image reconstruction results, object existence judgment results and contact relationship judgment results to construct a latent space world model of the scene.

2. The method according to claim 1, characterized in that The image semantic segmentation network includes a DeepLabV3+ encoder and a DeepLabV3+ decoder with an added attention mechanism; The method of performing semantic segmentation on the scene image in the robot grasping operation task by using the image semantic segmentation network to generate the semantic segmentation mask of each object in the working scene includes: Perform multi-scale feature extraction on the scene image through the DeepLabV3+ encoder, and then perform attention-weighted feature fusion and 1×1 convolution to output a deep feature map; The deep feature map is upsampled by the DeepLabV3+ decoder and then cross-layer fused with the shallow features of the DeepLabV3+ encoder, and then convolved and upsampled to output the semantic segmentation mask.

3. The method according to claim 1, characterized in that The multi-task scene understanding network includes a shared feature encoder, a semantic segmentation image decoder, a unary symbol decoder, and a binary relation decoder; The processing of the semantic segmentation mask by a multi-task scene understanding network to generate an implicit representation includes: encoding the semantic segmentation mask through the shared feature encoder to extract the implicit representation containing object position, shape, and sign information; Performing transposed convolution and convolution operations on the implicit representation through the semantic segmentation image decoder to extract position and shape information corresponding to the object and reconstruct the object semantic segmentation mask image; Determining whether an object exists in the current scene based on the implicit representation by the unary symbol decoder, and outputting the binary existence flag; Based on the implicit representation, the spatial interaction features between the two objects are calculated by the binary relationship decoder based on the ACmix self-attention and convolution hybrid module to generate the contact relationship identifier.

4. The method according to claim 3, characterized in that The training method of the multi-task scene understanding network includes the following steps: Construct a multi-task training dataset. Each set of training samples contains semantic segmentation mask images corresponding to objects in the grasping operation scene, object existence labels, and contact relationship labels between objects. performing feature extraction on the semantic segmentation mask image by the shared feature encoder to generate the implicit representation; Performing transposed convolution and convolution operations on the implicit representation through the semantic segmentation image decoder, outputting a reconstructed image, and comparing the image with the semantic segmentation mask image to calculate the semantic segmentation mask image reconstruction loss; Classify the implicit representation using the unary symbol decoder to predict the existence state of the object, and calculate the cross entropy loss by comparing it with the object existence label as the object existence classification loss; Classifying the spatial relationship between objects through the binary relationship decoder, predicting the contact relationship between the objects, and calculating the cross entropy loss as the contact relationship classification loss by comparing with the contact relationship label; A joint loss function is constructed based on the semantic segmentation mask image reconstruction loss, the object existence classification loss, and the contact relationship classification loss, and the parameters of the multi-task scene understanding network are back-propagated and updated until the model converges.

5. The method according to claim 1, wherein The state transfer network includes an action encoding module, a cross-modal fusion module and a state update module; The predicting of the implicit representation of each object at the current moment and the robot arm motion information by a state transition network at the next moment includes: The action type is embedded and encoded by the action encoding module, the action position is encoded by the action encoding module, and the direction parameter is angle-encoded to generate an action semantic vector, and then a spatial action feature map is generated by repeated filling. The implicit representation of each object at the current moment is concatenated and fused with the spatial action feature map through the cross-modal fusion module, and the fused information is processed by introducing the dilated spatial pyramid pooling to achieve multi-scale feature extraction. The coordinate attention mechanism is then introduced to enhance the feature response of important positions and channels, thereby generating an action-aware feature tensor. The state update module maps the action perception feature tensor through a fully connected network and outputs the implicit representation of the next moment.

6. The method according to claim 5, characterized in that The training method of the state transfer network comprises the following steps: Construct a state transition training dataset. Each set of training samples contains the implicit representation data of the current moment, the robot arm motion information data, and the implicit representation label data of the next moment. Encoding the action information data by the action encoding module to generate an action semantic vector and a spatial action feature map; The implicit representation data at the current moment is concatenated and fused with the spatial action feature map through the cross-modal fusion module, and the fused information is processed by introducing the dilated spatial pyramid pooling to achieve multi-scale feature extraction. The coordinate attention mechanism is then introduced to enhance the feature response of important positions and channels, thereby generating an action-aware feature tensor. The state update module maps the action perception feature tensor through a fully connected network and outputs a predicted implicit representation of the next moment; The MSE loss value between the predicted implicit representation and the implicit representation label data at the next moment is calculated, and the parameters of the state transition network are updated by backpropagation until the model converges.

7. The method according to claim 1, characterized in that The decoder in the multi-task scene understanding network architecture includes a semantic segmentation image decoder, a unary symbol decoder, and a binary relation decoder; The method of generating a scene segmentation image reconstruction result, an object existence judgment result, and a contact relationship judgment result by a decoder based on the predicted next moment implicit representation includes: Decoding the next-moment implicit representation by the semantic segmentation image decoder to generate a pixel-level semantic segmentation image reconstruction result; Classifying the existence identifiers in the implicit representation at the next moment through a unary symbol decoder, and outputting a binary judgment result of whether each object exists; The contact identifiers in the implicit representation at the next moment are analyzed by a binary relationship decoder to generate a contact relationship judgment result between the objects.

8. A device for constructing a latent space world model for a robot grasping operation scene, characterized in that: include: The segmentation module is used to perform semantic segmentation on the scene image in the robot grasping operation task through the image semantic segmentation network, and generate semantic segmentation masks for each object in the work scene; a processing module, configured to process the semantic segmentation mask through a multi-task scene understanding network to generate an implicit representation, wherein the implicit representation includes a position and shape feature identifier of the object, an existence identifier for indicating whether the object exists, and a contact identifier for characterizing contact relationships between objects, wherein the multi-task scene understanding network is trained by jointly optimizing a semantic segmentation mask image reconstruction loss, an object existence classification loss, and a contact relationship classification loss; An updating module, configured to predict the implicit representation at the next moment through a state transition network based on the implicit representation of each object at the current moment and the robot's manipulator motion information, wherein the motion information includes motion type, action position, and direction parameters; A construction module is used to generate scene segmentation image reconstruction results, object existence judgment results and contact relationship judgment results based on the predicted implicit representation of the next moment through a decoder in a multi-task scene understanding network architecture, and construct a latent space world model of the scene.

9. An electronic device comprising: A processor and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images