A method, system, and storage medium for robotic handling in low gravity environments

Abstract

The application discloses a robot carrying method and system in a low-gravity environment and a storage medium. A simple sketch drawn by an operator and a real scene image in a current cabin are collected, the simple sketch and the real scene image are input into a double-branch visual encoder model which has been trained, a basic action sequence instruction for controlling a robot to complete a material carrying operation is generated, and then a target rotation angular velocity is monitored in real time during target grabbing by executing the basic action sequence instruction. If the target rotation angular velocity is greater than a set threshold, the robot gripper is driven to rotate in a reverse direction for fine adjustment. Thus, pure visual instruction interaction is realized to adapt to the situation that voice interaction is completely unavailable in a low-gravity scene, and the problem that an object is easily subjected to irregular rotation and is difficult to be grabbed under low gravity is solved.

Description

A robotic handling method, system, and storage medium in a low-gravity environment Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot material handling method, a robot, and a storage medium. Background Technology

[0002] In industrial robot material handling scenarios, automated material handling in low-gravity environments, such as under a spacecraft, faces numerous challenges. On one hand, traditional handling control methods rely on voice commands, but voice interaction is completely unavailable in low-gravity environments like space. On the other hand, relying on pure text commands is cumbersome for describing complex objects or workflows, requiring tedious input and operator training. Furthermore, low gravity causes objects to float without fixed support, making stable object grasping difficult. Therefore, these problems directly restrict the application of robots in low-gravity environments and their handling efficiency. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by providing a robot handling method for low-gravity environments, comprising:

[0004] Collect simple sketches drawn by the operator and images of the actual scene inside the cabin. The simple sketches include a first pattern for depicting the shape of the work target, a second pattern for constraining the execution actions, and a third pattern for depicting the shape of the target's destination.

[0005] The simplified drawing and the real scene image are input into the trained dual-branch visual encoder model to generate basic motion sequence instructions for controlling the robot to complete material handling operations.

[0006] During the target grasping process, the target's rotational angular velocity is monitored in real time. If it exceeds a set threshold, the robot gripper is driven to rotate in the opposite direction for fine-tuning until the target's rotational angular velocity is lower than the set threshold.

[0007] Preferably, the acquisition of simple sketches drawn by the operator and images of the actual scene inside the cabin also includes:

[0008] Acquire continuous frame images of the current cabin and perform anti-glare and motion blur preprocessing. Extract target motion vectors based on the shape features of simple line drawings. Perform dynamic modeling and multi-step prediction on the target state space. Map the predicted trajectory to the image plane to generate a dynamic mask and fuse it with the target mask of the current frame.

[0009] Preferably, the dual-branch visual encoder model includes a line drawing encoder, a real image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. The line drawing encoder extracts features from the input line drawing to obtain a first feature vector containing target shape features. The real image encoder extracts features from the input real scene image to obtain a second feature vector containing texture and color information. The first and second feature vectors have the same dimension. The feature embedding layer normalizes the first and second feature vectors to eliminate feature vector length differences and maps them to a cross-modal semantic space through nonlinear transformation. The cross-modal comparison module performs comparative learning on the embedded features, forcing line drawing features of similar objects to be close to real image features in the embedding space, while dissimilar object features are kept far apart, generating cross-modal aligned fused features. The action generation network module decodes the fused features through multiple layers to generate basic action sequence instructions.

[0010] Preferably, training the dual-branch visual encoder model includes:

[0011] The real material image and the line drawing are encoded by a dual-channel input network. The real image channel is processed by a neural network and connected to a dilated convolutional layer to extract multi-scale geometric features, while the line drawing channel is processed by a simple convolutional neural network to extract shape features.

[0012] Cross-modal fusion is performed on real material images and line drawing images. Multi-scale geometric features and shape features are aligned through contrastive learning and an association matrix is ​​constructed to assign the most similar line drawing category to each image.

[0013] Construct a graph structure, defining image and sketch pair as object nodes, action template as action nodes, and object nodes and action nodes as connected by executing actions. The edge weights are determined by the reinforcement learning reward value.

[0014] Automatic generation of motion templates is performed. For each material type in a simple line drawing, an initial motion template is generated through a rule library. Objects of the same shape can share the same motion template parameters by adjusting only the variable. The kinematic constraints of the industrial robot are integrated through the motion generator.

[0015] Real images are input into a graph neural network to automatically match the material type in a simple line drawing, and corresponding optimized action templates are extracted from graph action nodes to generate triplet annotations consisting of images, line drawings, and actions, which are then stored in a cross-modal dataset for model training.

[0016] Preferably, the object node includes object feature information of the task target or object feature information of the transport destination, and the object feature information includes, but is not limited to, shape, size and position; the action node includes gripper type and gripper movement path speed; the additional node includes scene constraint information and priority information.

[0017] Preferably, the second pattern includes a priority marker, wherein the edge weight between two nodes is determined by the spatial distance between the two nodes, the dynamic conflict probability, and the priority marker.

[0018] Preferably, the robot handling method in low gravity environment further includes: monitoring the position of the target in real time during the robot's movement using a two-dimensional camera; after detecting visual feature fluctuations caused by light reflection, using an image segmentation model to generate a mask based on the target's outline in a simple drawing to retain only the pixels of the top or bottom area of ​​the non-reflective target.

[0019] This invention also discloses a robot handling method in a low-gravity environment, comprising:

[0020] Collect simple sketches drawn by the operator and images of the actual scene inside the cabin. The simple sketches include a first pattern for depicting the shape of the work target, a second pattern for constraining the execution actions, and a third pattern for depicting the shape of the target's destination.

[0021] Feature extraction is performed on the simplified drawing to obtain a simplified drawing feature vector containing the target shape features; feature extraction is performed on the input real scene image to obtain a real image feature vector containing texture and color information; the simplified drawing feature vector and the real image feature vector have the same dimension;

[0022] The input line drawing feature vector and real image feature vector are normalized and then nonlinearly transformed to map them into a cross-modal semantic space. After comparative learning, the embedded features are forced to be close to the real image features of similar objects and far away from the features of dissimilar objects in the embedding space, generating cross-modal aligned fused features. The fused features are then input into the action generation network and decoded through multiple layers to generate basic action sequence instructions for controlling the robot to complete material handling operations.

[0023] This invention also discloses a robot handling system in a low-gravity environment, comprising: an image acquisition module for acquiring a simple sketch drawn by an operator and an image of the actual scene inside the current compartment, wherein the simple sketch includes a first pattern for depicting the shape of the target, a second pattern for constraining the execution of actions, and a third pattern for depicting the shape of the target's destination; a basic instruction generation module for inputting the simple sketch and the image of the actual scene into a trained dual-branch visual encoder model to generate basic action sequence instructions for controlling the robot to complete material handling operations; and an angular velocity fine-tuning module for real-time monitoring of the target's rotational angular velocity during target grasping while executing the basic action sequence instructions, and if the angular velocity exceeds a set threshold, driving the robot gripper to rotate in the opposite direction for fine-tuning until the target's rotational angular velocity falls below the set threshold.

[0024] The present invention also discloses a computer-readable storage medium, characterized in that, when an executable computer program in the storage medium is executed by a processor, it is capable of implementing the method described in any of the preceding claims.

[0025] This invention discloses a robot handling method, system, and storage medium for low-gravity environments. By acquiring simplified drawings created by the operator and images of the actual scene inside the robot's compartment, and inputting these images into a trained dual-branch visual encoder model, a basic sequence of motion instructions is generated to control the robot to perform material handling operations. During target grasping, the target's rotational angular velocity is monitored in real time. If it exceeds a set threshold, the robot's gripper is driven to rotate in the opposite direction for fine-tuning. This achieves pure visual command interaction to adapt to situations where voice interaction is completely unavailable in low-gravity environments, while also solving the problem of irregular rotation and difficulty in grasping objects under low gravity.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0028] Figure 1 is a flowchart illustrating a robot handling method in a low-gravity environment according to an embodiment of the present invention.

[0029] Figure 2 is a schematic diagram of the structure of a dual-branch visual encoder model disclosed in an embodiment of the present invention.

[0030] Figure 3 is a schematic diagram of the training steps of a dual-branch visual encoder model disclosed in an embodiment of the present invention.

[0031] Figure 4 is a schematic diagram of the structure of a visual action graph neural network disclosed in an embodiment of the present invention.

[0032] Figure 5 is a simplified line drawing diagram of an embodiment of the present invention.

[0033] Figure 6 is a schematic diagram of the specific process of a robot handling method in a low gravity environment disclosed in another embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0035] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a limitation of quantity, but rather indicate the presence of at least one.

[0036] This embodiment discloses a robot handling method in a low-gravity environment. Targeting the low-gravity space scenario, it enhances the generalization ability and environmental adaptability of industrial robots by abstracting object shapes and inputting visual commands. As shown in Figure 1, it can specifically include the following:

[0037] Step S1: Collect the simple sketch drawn by the operator and the real scene image inside the cabin. The simple sketch includes a first pattern for depicting the shape of the work target, a second pattern for constraining the execution action, and a third pattern for depicting the shape of the target transport destination.

[0038] The robot can capture images of simple sketches drawn manually by the operator on physical or electronic media using a camera. For example, the operator can use paper and pen to draw a simple sketch of the object to be moved or the material to be placed, and then show it to the camera mounted on the industrial robot. The industrial robot can then recognize the sketch and input it into its processing. Alternatively, the operator can also manually draw a simple sketch directly on the control screen connected to the robot via wired or wireless means, or on the robot's built-in touchscreen. This touchscreen is connected to the industrial robot, allowing the sketch to be input into its processing.

[0039] The second pattern includes a first type of identifier for identifying constraint information of the grasping scenario of the task target, and a second type of identifier for identifying constraint information of the operation scenario of the task target, wherein the robot is configured to perform an operation action after completing the grasping action of the task target. Additionally, the second type of identifier may also include constraint information for identifying the handling, transfer, or storage location of the task target.

[0040] For example, when drawing simple sketches to indicate handling tasks, the sketches of materials, such as cylindrical cans or square cardboard boxes in a space station, serve as the first image depicting the shape of the target. The sketches of objects where the materials are placed, such as shelves, serve as the third image depicting the shape of the target, i.e., the destination. Simultaneously, special symbols, such as arrows (→), are designed to indicate both. The starting point of the arrow is the object to be handled, and the ending point is the object to be placed. These special symbols serve as the second image used to constrain the execution of actions.

[0041] In this embodiment, a second pattern identifier library can also be pre-configured as needed to configure and store the specific information of each second pattern identifier for subsequent simplified diagram recognition. These second pattern identifiers may include the following:

[0042] For situations where there are different types of materials to be grabbed or moved, you can label each type of material with a simple line drawing, such as "①", "②", or "③", to indicate the order of handling.

[0043] For situations where there are multiple similar materials to be moved, Arabic numerals such as "1", "2", and "3" can be marked next to the simple line drawing of the materials to indicate the number of materials to be moved; and special marking symbols can be designed to indicate that all materials to be moved need to be moved, such as "@".

[0044] In situations where there are multiple similar materials to be moved or objects to be placed, to identify the actual materials to be moved or objects to be placed, all similar materials to be moved or objects to be placed can be drawn, and the identified materials to be moved or objects to be placed can be specially marked, such as by marking them with a special symbol. ".

[0045] For scenarios where multiple materials to be moved need to be stacked sequentially, a stacking task symbol can be defined, such as the symbol " ".

[0046] For scenarios requiring the sequential laying of multiple materials to be moved, a laying task symbol can be defined, such as the symbol "". ".

[0047] In this embodiment, multiple cameras mounted on the robot can acquire two-dimensional images of the task scene in real time. Key points of objects are detected in the two-dimensional images, and the three-dimensional poses of each object are calculated by combining camera intrinsic parameters and prior dimensions. The three-dimensional poses are then matched with target shape features to obtain target location information and target object type. Alternatively, a depth camera or LiDAR mounted on the robot can acquire point cloud data of the task scene in real time. The three-dimensional poses of each object are identified and obtained from the point cloud data, and the three-dimensional poses are matched with target shape features to obtain target location information and target object type.

[0048] Step S2: Input the simplified drawing and the real scene image into the trained dual-branch visual encoder model to generate basic action sequence instructions for controlling the robot to complete material handling operations.

[0049] In this embodiment, as shown in Figure 2, the dual-branch visual encoder model includes a line drawing encoder, a real image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. The line drawing encoder extracts features from the input line drawing to obtain a first feature vector containing target shape features. The real image encoder extracts features from the input real scene image to obtain a second feature vector containing texture and color information. The first and second feature vectors have the same dimension. The feature embedding layer normalizes the first and second feature vectors to eliminate feature vector length differences and maps them to the cross-modal semantic space through nonlinear transformation. The cross-modal comparison module performs comparative learning on the embedded features, forcing line drawing features of similar objects to be close to real image features in the embedding space and distancing features of dissimilar objects to be far apart, generating cross-modal aligned fused features. The action generation network module decodes the fused features through multiple layers to generate basic action sequence instructions.

[0050] Specifically, the real image encoder can use image coding models such as ViT-B / 16, SigLIP, or DINO, taking a real image as input and outputting a feature vector I containing texture, color, etc. The line drawing encoder can use the same image encoder as the real image encoder, such as ViT-B / 16, SigLIP, or DINO, taking a line drawing image as input and outputting a shape feature vector S, which focuses on extracting abstract features such as contours and geometric centers. Since the same encoder is used, S and I have the same dimensions, facilitating subsequent cross-modal comparative learning.

[0051] Feature Embedding Layer: The feature embedding layer consists of a normalization layer and a nonlinear neural network mapping layer. First, L2 normalization is performed on the real image feature vector I and the sketch feature vector S output by the dual-branch encoder to eliminate the influence of feature vector length differences. The normalization calculation method is as follows:

[0052]

[0053] Subsequently, a shared fully connected layer, such as a single-layer MLP, performs a nonlinear transformation, mapping the data to a cross-modal semantic space while maintaining the same output dimension as the input. Here, the feature embedding layer unifies the feature space by normalizing and performing nonlinear transformations, ensuring that features from different modalities, including textured real-image features and shape-based line drawings, reside in the same metric space, facilitating subsequent comparison learning and similarity calculation. Furthermore, it enhances semantic alignment, highlighting cross-modal "shape-instance" associations, such as the contour semantics of a rectangular line drawing and a cardboard box image, while suppressing background noise, such as complex textures in real-image data. The input to the feature embedding layer is the feature vectors I∈R^768 (real image) and S∈R^768 (line drawing) output from the dual-branch encoder; the output is the embedded feature vectors I'∈R^768 and S'∈R^768, used by the subsequent cross-modal comparison module to calculate the contrast loss. .

[0054] Cross-modal contrast module: Employs a contrastive learning method to learn the target, forcing the simplified line drawing features of similar objects to be close to the features of real images in the embedding space, while keeping the features of dissimilar objects away. The loss function is calculated as follows:

[0055]

[0056] in, This is a temperature parameter used to control the exponential scaling of feature vector similarity; its value typically ranges from (0, 1). The features of a single negative sample are the true image features.

[0057] Action generation network module: Input fused features The action sequence is generated by decoding through multiple layers of Transformer, such as "move to (x,y,z) → adjust gripper angle θ → grab", and the output is an action sequence.

[0058] In this embodiment, semantic alignment can be achieved by constructing a marker-object association graph GNN through cross-modal graph structure modeling, as detailed below:

[0059] Define nodes: Object nodes: Examples include the aspect ratio of a rectangular object and the number of shelves; action nodes: Such as the gripper type and movement path speed for the "grab" action; additional nodes: Examples include constraints such as "handle with care" and priority markers such as "①". Object nodes may include object feature information of the target task or the destination task, including but not limited to shape, size, and location; action nodes may include gripper type and gripper movement path speed; additional nodes may include scene constraint information and priority information.

[0060] Edge relationship construction:

[0061] Object-Action Edge: Connections are established based on shape matching degree (e.g., the cosine similarity between a cylindrical object and the "ring gripper" action is ≥0.8), with the weight being the action suitability score;

[0062] Object-Attached Edges: Numerical priority is directly mapped to node attributes, and color constraints are associated with physical parameters through table lookup (e.g., red → gripping force ≤ 5N).

[0063] Run the structured instruction generation algorithm again:

[0064] Based on the GNN output, operation instructions are generated through a Transformer encoder-decoder architecture: First, graph feature encoding is performed, encoding node attributes and edge weights into feature vectors. A multi-head attention mechanism is used to capture cross-node dependencies (such as the spatial association between "second shelf" and "move to z=150cm"). Then, instruction template matching is performed. A predefined instruction template library (such as "move [object] to [location] [action parameters] [constraints]") is used to select templates and fill in parameters through a pointer network.

[0065] Example: Move rectangular object #1 to the second shelf, priority 1, handle with care.

[0066] {

[0067] "object": "rectangle①",

[0068] "target": "Second shelf level (z=150cm)",

[0069] "action": ["parallel gripper grasp", "movement (speed=0.1m / s)"],

[0070] "constraint": ["force≤5N", "priority=1"]

[0071] }

[0072] Perform physical feasibility verification: Verify whether the joint angles and paths in the instructions are within the robot's workspace using an inverse kinematics model. If they exceed the limits, trigger template adjustments such as changing the gripper type or splitting the task.

[0073] In this embodiment, due to the low gravity, the object is prone to floating without fixed support, and strong glare may occur due to reflections from equipment inside the cabin, such as metal pipes. A high dynamic range (HDR) camera can be used to avoid motion blur caused by the floating object. At the same time, a floating object trajectory prediction module can be added in the image preprocessing stage to analyze the object's motion vector based on continuous frame difference, which helps to stabilize and identify the target contour.

[0074] Specifically, the method may also include acquiring continuous frame images of the current cabin and performing anti-glare and motion blur preprocessing, extracting target motion vectors based on the shape features of simple line drawings, dynamically modeling and multi-step predicting the target state space, mapping the predicted trajectory to the image plane to generate a dynamic mask and fusing it with the target mask of the current frame.

[0075] In a specific embodiment, continuous frame images are acquired using an HDR camera and subjected to anti-glare preprocessing and motion blur correction. An improved PWC-Net model is used to calculate dense optical flow, and the target motion vector is extracted using a cross-modal attention mechanism, incorporating the shape features of a simple line drawing. Furthermore, an adaptive Kalman filter is employed to dynamically model and predict the target state space in multiple steps. Based on the prediction confidence, the predicted trajectory is mapped onto the image plane to generate a dynamic mask, which is then fused with the current frame mask. Simultaneously, the shape features extracted by the line drawing encoder are used as constraints to guide the optical flow calculation. Visual features output from the real image encoder are used in the Kalman filter observation model, and the alignment of the line drawing and real image features is optimized using a cross-modal contrastive loss function combined with the trajectory prediction confidence. Finally, the predicted trajectory features are input into the action to generate a Transformer, enabling collaborative work with the dual-branch visual encoder.

[0076] Step S3: During the target grasping process, the target rotation angular velocity is monitored in real time. If it is greater than the set threshold, the robot gripper is driven to rotate in the opposite direction for fine adjustment until the target rotation angular velocity is lower than the set threshold.

[0077] Specifically, due to the tendency of objects to rotate randomly under low gravity (e.g., even a slight force during grasping can cause a can to spin), a posture damping control step can be added to the motion sequence. After grasping the target, the object's rotational angular velocity is monitored in real time using an end effector force sensor. If the velocity is ≥5° / s, the gripper is triggered to make minor adjustments, such as ±2° every 100ms. Simultaneously, the movement speed is limited to half or even less than that of the ground scene to avoid collisions caused by inertia. The specific control measures can include the following:

[0078] Perform reverse fine-tuning: Based on the object's rotation direction monitored by the gyroscope, the grippers perform fine-tuning in the opposite direction. For example, if the object rotates clockwise around the X-axis, the grippers rotate counterclockwise to counteract the rotational torque. Set the adjustment range: The single fine-tuning angle is ≤ ±2°, and the adjustment frequency is once every 100ms (10Hz) to avoid over-adjustment causing new rotations.

[0079] Here, the trigger threshold can be adjusted. For example, fine-tuning can be triggered when the object's angular velocity is ≥5° / s. The goal of fine-tuning is to reduce the angular velocity to a safe range, such as less than 1° / s. Therefore, the angular velocity of the robotic arm's fine-tuning must match the object's rotational angular velocity to ensure that rotation is suppressed within a reasonable time.

[0080] In addition, after adjustment, the torque change of the force sensor is monitored. If the torque is not reduced by more than 50%, the rotation is not effectively suppressed. Then the adjustment range is increased to ±3°, so that the fine-tuning angular velocity needs to be linked to the torque attenuation effect.

[0081] In this embodiment, the dual-branch visual encoder model described above can be trained in the following way, as shown in Figure 3, which may include the following content.

[0082] Step S101: The real material image and the line drawing are encoded by a dual-channel input network. The real image channel is processed by a neural network and connected to a dilated convolutional layer to extract multi-scale geometric features, while the line drawing channel is processed by a simple convolutional neural network to extract shape features.

[0083] Step S102: Perform cross-modal fusion on real material images and line drawing images, align multi-scale geometric features and shape features through contrastive learning and construct an association matrix, and assign the most similar line drawing category to each image.

[0084] Step S103: Construct a graph structure, defining the image and sketch pair as object nodes and the action template as action nodes. Object nodes and action nodes are connected by executing actions, and the edge weights are determined by the reinforcement learning reward value. The edge weights between two nodes can be determined by the spatial distance between the two nodes, the dynamic conflict probability, and the priority label.

[0085] Step S104: Automatic generation of motion templates. An initial motion template is generated for each material type of the sketch through the rule library. Objects of the same shape share the same motion template parameters by adjusting only the variable. The kinematic constraints of the industrial robot are integrated through the motion generator.

[0086] Step S105: Input the real image into the graph neural network to automatically match the material type of the line drawing, extract the corresponding optimized action template from the graph action node, generate a triplet annotation consisting of image, line drawing and action, and store it in the cross-modal dataset for model training.

[0087] Specifically, each of the above training processes includes the following:

[0088] Step S201: Collect and preprocess the training data.

[0089] For real image data, images of transported materials and images of objects on which materials are placed can be captured.

[0090] Specifically, this can be achieved by using a camera to capture images of materials in industrial scenes, such as cardboard boxes, gears, and cylindrical workpieces, and labeling them with categories such as "rectangular" and "cylindrical"; and images of objects where materials are placed, such as workpiece platforms, assembly lines, and shelves, and labeling them with categories such as "platform" and "shelf".

[0091] There are two ways to obtain images of materials and objects placed in industrial scenarios: a. For unknown material handling scenarios, directly use material images from open-source industrial robot material handling datasets, such as the Open X-Embodiment dataset, RoboMIND dataset, and AgiBot World dataset; b. For known material handling scenarios, take photos with a camera to simultaneously obtain images of the materials and objects placed in the dataset.

[0092] For a simple line drawing dataset, it can be collected in the following two ways:

[0093] The purpose of this simplified drawing dataset is to abstract material images (such as "cardboard box" as a cube, "bearing" as a cylinder) and material placement objects into simplified scene drawing elements (such as "shelf" as a multi-layered rectangle, "conveyor belt" as a long rectangle with arrows). It can be collected in the following two ways:

[0094] By manually drawing, real material images and images of material placement objects are abstracted into two-dimensional line drawings, and corresponding object category labels are added. Then, the CLIP model can be used to filter out low-similarity samples as negative samples.

[0095] The algorithm model is used to automatically generate images based on the Stable Diffusion model. The input is a real image of the material and an image of the object where the material is placed. The model can automatically generate corresponding line drawings. Then, the CLIP model can be used to filter out low-similarity samples as negative samples.

[0096] After collecting a small dataset of simplified line drawings through the above steps, the real image data can be generated in reverse using this dataset, as detailed below:

[0097] Step S2011: Constructing the parametric geometric primitive library, defining basic shape modules, variable attributes, parametric representations, and combination rules, and establishing a set of basic geometric primitives for industrial materials. Specifically, this includes:

[0098] Define several basic shape templates, including rectangles for cardboard boxes, books, etc.; cylinders for bottles, jars, tubes, etc.; spheres for ball bearings, glass marbles, etc.; and cones for ice cream cones, etc.

[0099] Define the variable properties of a rigid shape, including dimensions (length / width / height / radius), color (RGB value), texture label (smooth / rough / striped), etc.

[0100] Define a parameterized representation, where each primitive is represented by a parameter vector, such as a rectangle as (width, height, color, texture) and a cylinder as (radius, height, color, surface).

[0101] Multiple combination rules are defined, and primitives are combined through spatial transformations (translation, rotation, stacking) to simulate complex materials (such as a "rectangle + cylinder" combination representing a water cup with a handle). Through these combination rules, complex, unconventional, or irregularly shaped materials can be expressed using simple rigid shapes.

[0102] Step S2012: Generate realistic image data based on specific scene constraints. According to the industrial robot's usage scenario, the user inputs existing line drawing image data, then the line drawing is parsed to correspond to the primitive type, and scene constraints are parsed, such as the material placement object shelf having 3 layers. The primitive parameter generator adopts a Transformer architecture, taking as input a condition vector (primitive type + scene constraints) and outputting primitive parameters (such as basic shape, size, color, texture, etc.).

[0103] Use a scene renderer, such as the Blender API, to build a physics engine and generate multi-view images based on the primitive parameters output from the previous steps. This includes lighting simulation, supporting common lighting in industrial scenes (such as top light, side light, and backlight), as well as diverse viewing angles, automatically generating multi-angle images such as 0°-90° top and side views to simulate different installation positions of industrial robot cameras.

[0104] Data augmentation and automatic annotation can be performed during the generation of realistic image data. By synchronously outputting a mask during image generation, category labeling is automatically completed without manual intervention. By adding industrial scene noise such as lens distortion, dust occlusion, and motion blur to the rendered images, the diversity of realistic images is enhanced, thereby improving the robustness of the model during subsequent model training.

[0105] By generating the aforementioned reverse-image data, we achieve two key benefits: First, improved data acquisition efficiency, with a single primitive combination capable of generating thousands of variant images, saving over 80% of labor costs compared to traditional photography. Second, by adding industrial scene noise such as lens distortion, dust occlusion, and motion blur to the rendered images, we enhance the diversity of realistic images, thereby improving the robustness of the model during subsequent training. Furthermore, we increase the flexibility of scene transfer; simply adjusting scene constraint parameters allows for the rapid generation of new scene data to adapt to production line changes. Through the combination of abstract primitives, parameter control, and industrial scene rendering, we overcome the limitations of conventional generative models, precisely adapting to the dataset construction needs of industrial material handling.

[0106] Step S202: Perform cross-modal annotation on the collected data: Through the aforementioned steps, each real image has a corresponding line drawing. Then, for materials of the same shape, the following specific method can be used for annotation. By utilizing the Visual Action Graph Neural Network (VGNN) to construct a self-supervised annotation link of "image-line drawing-action", the traditional manual annotation is replaced, realizing the automated generation and standardization of action templates for materials of the same shape.

[0107] The Visual Action Graph Neural Network (VGNN), as shown in Figure 4, is used to encode real material images and line drawings. First, a dual-channel input network is constructed, with the two channels corresponding to the real image channel and the line drawing channel, respectively. For the real image channel, the traditional ResNet-50 architecture neural network is optimized by adding dilated convolutional layers with a dilation rate of [n, 2n, 3n] after the residual block, where n can take integer values ​​such as 1, 2, 3, 4, and 5, thus enabling better extraction of multi-scale geometric features. For the line drawing image channel, since it is generally simple, a simple convolutional neural network (CNN) can be used to extract shape features.

[0108] Then, cross-modal fusion is performed on the real material images and the line drawing images. Here, by comparative learning such as CLIP loss, the image and line drawing features are aligned to construct the association matrix: M=cosine(V,S), and the most similar line drawing category is automatically assigned to each image, such as rectangle matching cardboard box images.

[0109] Next, the graph structure is constructed. For the "nodes" in the graph structure, the image-sketch pair is defined as the "object node" and the action template is defined as the "action node". For the "edges" in the graph structure, the object node and the action node are connected by "execute an action", and the weight is determined by the reinforcement learning reward value.

[0110] You can define that a simple sketch of the material to be moved is drawn on the left side of the drawing or panel, and the object to which the material is placed is also drawn on the left side of the drawing or panel; or you can design special marking symbols, such as the arrow symbol →, to indicate the two, with the starting end of the arrow symbol indicating the object to be moved and the ending end indicating the object to which the material is placed.

[0111] Step S203: Automatically generate action templates.

[0112] First, initial motion generation is performed, starting with zero-sample initialization. For each sketch category, an initial motion template is generated based on rules. For example, for rectangles: the gripper closes parallel (opening = object width × 0.8), and the gripping center coordinates are (object's geometric center); for cylinders: the gripper closes in a ring (radius = object radius + 1cm), and the gripping height is (at 1 / 2 of the object's height). For objects of the same shape (such as rectangles of different sizes), the motion template parameters are shared through VGNN, and only variable parameters such as gripper opening and gripping height need to be adjusted.

[0113] The motion template generator can be a multi-layer transformer-based neural network with embedded physical constraints. Industrial robot kinematic constraints (such as the maximum gripper opening) can be integrated into the motion generator, and the Sigmoid function can be used at the end of the network.

[0114]

[0115] Here, x is the input value, and the function output value ranges from (0, 1). This function can map the probability of illegality of industrial robot actions to the interval (0, 1), thereby achieving soft constraints on physical constraints (such as setting the probability of actions exceeding the joint angle range to a value close to 0). This filters illegal actions and ensures that the generated action template is physically feasible.

[0116] Specifically, the method for integrating industrial robot kinematic constraints into the motion generator is as follows:

[0117] Step S2031, constraint parameter encoding, encoding robot kinematic constraints (such as joint angle range) Maximum opening of the gripper Load limit Encode as constraint vector Each element corresponds to a one-dimensional constraint parameter (such as a normalized upper bound for joint angles). This is achieved through a learnable neural network embedding layer. Mapped to constrained feature vectors This is concatenated with the action features input from the aforementioned Transformer.

[0118] Step S2032: Introduce a conditional attention mechanism in the Transformer decoder, that is, when calculating self-attention, constrained features are included. With current action characteristics Perform dot product interaction: This mechanism forces the alignment of motion features with constraint features, ensuring that the generated motion parameters (such as joint angles and gripper openings) implicitly contain constraint information.

[0119] Step S2033: Then, add a constraint projection layer after the Transformer output layer to map the motion parameters to the physical feasible space through linear transformation. ;in, The function is based on the constraint vector Upper and lower limit truncation output, such as gripper opening. ,make sure .

[0120] By incorporating kinematic constraints into the entire process of Transformer input encoding, attention calculation, and output mapping, end-to-end constraint awareness for action generation is achieved.

[0121] The entire self-supervised annotation process described above involves inputting a real image into a VGNN, automatically matching it to a simple line drawing category such as a rectangle, then extracting the optimized action template corresponding to that category from the GNN action nodes, and finally generating "image-line drawing-action" triplet annotations, which are then stored in a cross-modal dataset. Through these steps, the association between image, shape, and action can be automatically completed, improving annotation efficiency by over 90% compared to manual annotation. Furthermore, the GNN learns the generalization relationship between shape and action through graph edge weights; for example, "large rectangles" and "small rectangles" share the same grasping logic, but the gripper opening automatically adapts to size differences.

[0122] The self-supervised annotation scheme described above achieves automation and standardization of cross-modal annotation by combining graph neural networks and reinforcement learning, breaking through the efficiency bottleneck of traditional manual annotation. It is especially suitable for handling tasks with a variety of materials and obvious shape regularity in industrial scenarios.

[0123] During the model pre-training phase, the dual-branch encoder is first pre-trained using image-text pair datasets such as Conceptual Captions to establish a basic "shape-semantic" association.

[0124] Then, model fine-tuning is performed, preferably using both supervised learning and reinforcement learning simultaneously. In supervised learning, the input is a real image-sketch pair, and the output is a standardized action label, optimized using cross-entropy loss.

[0125] Where C is the number of action categories, Tag it as real action. To predict probabilities.

[0126] In reinforcement learning, the Proximal Policy Optimization (PPO) algorithm is used to optimize action sequences, and the reward function can be designed as follows:

[0127] Additionally, a constraint violation penalty term can be added to the reward function.

[0128] During the reinforcement learning phase, a constraint violation penalty term can also be added to the reward function:

[0129]

[0130] in The penalty coefficient is... To constrain violations of the indicator function, the model is forced to prioritize generating action sequences that conform to physical constraints.

[0131] After the trained model is deployed to the industrial robot, visual commands are input first when the robot begins its task. The industrial robot operator can input simple drawings into the robot in two ways:

[0132] Draw using a touchscreen; draw simple sketches of objects to be moved or materials to be placed. This touchscreen is connected to the industrial robot, so the data can be input into the industrial robot for processing.

[0133] Use paper and pen to draw; draw a simple sketch of the object to be moved or the object to be placed, and then show it to the camera mounted on the industrial robot. In this way, the industrial robot can recognize the sketch and input it into the industrial robot for processing.

[0134] Then, the industrial robot can perform real-time environmental perception during operation: Preferably, the industrial robot is equipped with two 2D cameras, which acquire image data in real time and detect key points of objects (such as corners of shelves, midpoints of material boxes) in the 2D images. Using the PnP algorithm, combined with camera intrinsic parameters and prior dimensions, the 3D pose of the object is calculated. For example, given the pixel coordinates of the four corners of a shelf, combined with the actual dimensions of the shelf (e.g., width 150cm, height 200cm), the pose matrix of the shelf in the camera coordinate system is solved, and the material type is confirmed by matching with simple line drawing features.

[0135] When an industrial robot detects an obstacle, such as a temporarily stacked workpiece, during the execution of a task, it can use the A* algorithm, commonly used in path planning, combined with the radius of a simple line drawing shape, to dynamically replan the obstacle avoidance path.

[0136] The following specific example illustrates the handling of canned food in the low gravity environment of space.

[0137] Inside the space capsule, operators need to direct robots to move food cans of different sizes to designated storage shelves. Due to the weightlessness of the space environment and the inability to use voice interaction, they rely on purely visual commands to interact with the robots. The food cans are cylindrical in shape and can be abstracted as a simple line drawing of a cylinder. The storage shelves have a multi-layered grid structure and can be abstracted as a multi-layered line drawing of rectangles. The background contains complex interference such as equipment pipelines and reflective surfaces.

[0138] Step S301: User input is performed. As shown in Figure 5, the operator draws the following content using a tablet:

[0139] Target object: The operator draws a simple cylindrical sketch to represent the cans, and marks them with the number "@" to indicate that all the cans have been moved.

[0140] Target location: The operator draws a simple two-tiered rectangular sketch to represent the storage shelf, and marks the second tier with an arrow "↔". The instruction means: "Place all cylindrical food cans flat on the second tier of the storage shelf."

[0141] Step S302: Perform model processing, cross-modal analysis, and motion planning.

[0142] Step S3021, Dual-branch feature extraction:

[0143] Simple line drawing encoder: Extracting shape features of a cylindrical line drawing using ViT-B / 16 Includes the outline (circular) and size ratio (height:diameter = 2:1, based on common space can specifications).

[0144] Scene image encoder: A 2D camera captures scenes inside the spacecraft, and visual features are extracted using ViT-B / 16. This includes the reflective surface of the cans, the storage rack grid, and the background piping.

[0145] Step S3022, Cross-modal contrastive learning and background decoupling:

[0146] Using contrast loss Forced and Alignment and filtering of background interference (such as the rectangular features of pipelines).

[0147]

[0148] in, Characterized by canned goods. Background pipeline features, Lowering the characteristic distribution temperature improves the distinguishability.

[0149] Step S3023, Spatial Relationship Reasoning and Action Generation, VGNN Scene Graph Construction:

[0150] Nodes: Can (cylinder, position (10, 5, 200)), Storage shelf (three-layer rectangle, second layer coordinates) );

[0151] Edge: Spatial relationship edge of "canned food → second shelf of storage", the weight is calculated by distance and path reachability.

[0152] The final Transformer module of the model outputs the predicted sequence of action instructions:

[0153] [

[0154] "move_to(pos=(10, 5, 200), speed=0.1m / s)", # Low-speed approach to avoid collision.

[0155] "adjust_gripper(radius=5cm)", # Circular gripper adapts to can radius

[0156] "grasp(force=5N)", # Low gripping force to prevent squeezing

[0157] "move_to(pos=(8, 5, 150), speed=0.05m / s)",

[0158] "release()"

[0159] ]

[0160] In this low-gravity environment of space, the speed is set to 1 / 5 of the normal value according to the preset configuration.

[0161] Step S303, Robot execution and visual feedback.

[0162] Dynamic execution and collision avoidance are implemented by using a 2D camera to monitor the target's position in real time during robot movement. Upon detecting visual feature fluctuations due to light reflection, an image segmentation model generates a mask based on the target's outline in a simplified sketch. This mask retains only pixels from the top or bottom areas of the target that are not reflective, eliminating the influence of sidewall reflections on visual feature comparison. Specifically, the robot monitors the can's position in real time using a 2D camera. When visual feature fluctuations due to light reflection are detected, dynamic mask generation is triggered: the U-Net model generates a mask based on the simplified cylindrical outline, retaining only the circular area pixels and eliminating background reflection interference; the can's coordinates are updated based on the mask, correcting the path to (8.2, 5.1, 150).

[0163] Grasp verification and retry mechanism: After the first grasp, if the overlap area between the can and the gripper in the image detected by the camera is less than a preset area threshold, which can be less than 80%, it is determined that the can is not stable, and reinforcement learning is triggered to retry. The gripper angle is adjusted to an adjusted value to increase the contact area, for example, the gripper angle can be adjusted to 45°. If the second grasp is successful, the changes in the image mask before and after are compared. If the image mask does not change, the position of the can is fixed, and the task is confirmed to be completed.

[0164] In this embodiment, simplified line drawing hierarchical markers replace absolute coordinates, adapting to the zero-gravity coordinate system environment within the spacecraft. Contrastive learning and masking techniques effectively filter background interference such as equipment pipelines and reflections, achieving an accuracy rate of over 99% (compared to only 70% for traditional methods), demonstrating strong anti-interference capabilities. Furthermore, low-speed, low-force control based on physical constraints avoids collisions with sensitive equipment, meeting the high reliability requirements of space missions and ensuring high operational safety. When adding new can types, only the simplified line drawing library needs updating, such as adjusting the cylinder height ratio; retraining the model is unnecessary, demonstrating strong generalization ability. This method, through simplified line drawing abstraction, cross-modal learning, and dynamic feedback, solves the core problems of space transport, such as the inability to use voice interaction, difficulties in visual recognition, limited interaction, and high requirements for motion precision.

[0165] This embodiment discloses a robot handling method in a low-gravity environment. It collects simplified drawings created by the operator and images of the actual scene inside the robot's compartment. These are then input into a trained dual-branch visual encoder model to generate a basic sequence of motion commands for controlling the robot to perform material handling operations. During target grasping, the target's rotational angular velocity is monitored in real time. If the velocity exceeds a set threshold, the robot's gripper is driven to rotate in the opposite direction for fine-tuning. This achieves pure visual command interaction to adapt to situations where voice interaction is completely unavailable in low-gravity environments, while also solving the problem of irregular rotation and difficulty in grasping objects under low gravity.

[0166] Furthermore, this dual-branch visual encoder model employs an architecture comprising a line drawing encoder, a real-world image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. By acquiring line drawings and real-world scene images and generating action sequence commands using the dual-branch visual encoder model, it achieves pure visual command interaction to adapt to situations where voice interaction is completely unavailable in low-gravity environments. Simultaneously, it leverages cross-modal feature alignment to enhance generalization ability for objects of different shapes. This solves the technical problems of existing technologies where voice interaction is unavailable in low-gravity environments and multi-object generalization ability is insufficient.

[0167] In another embodiment, as shown in Figure 6, a robot handling method in a low-gravity environment is also disclosed, which may specifically include the following.

[0168] Step S401: Collect the simple sketch drawn by the operator and the real scene image inside the cabin. The simple sketch includes a first pattern for depicting the shape of the work target, a second pattern for constraining the execution action, and a third pattern for depicting the shape of the target transport destination.

[0169] Step S402: Perform feature extraction on the sketch to obtain a sketch feature vector containing the target shape features; perform feature extraction on the input real scene image to obtain a real image feature vector containing texture and color information; the sketch feature vector and the real image feature vector have the same dimension.

[0170] This step also includes: using the target shape features of the line drawing as a prior constraint, performing cross-modal encoding on the line drawing and the real scene image to extract shape features and visual features; using a contrastive learning mechanism to force the visual features of the real scene image to align with the shape features of the line drawing; and using a network model to generate a target region mask based on the aligned features, while retaining the regions in the image that overlap with the shape of the line drawing.

[0171] Step S403: Normalize the input line drawing feature vector and the real image feature vector, and after nonlinear transformation, map the line drawing feature vector and the real image feature vector to the cross-modal semantic space; after comparative learning of the embedded features, force the line drawing features of similar objects to be close to the real image features in the embedding space, and the features of dissimilar objects to be far away, generating cross-modal aligned fused features; input the fused features into the action generation network, and after multi-layer decoding, generate basic action sequence instructions for controlling the robot to complete material handling operations.

[0172] The steps of the robot handling method in the low gravity environment described above are similar to the processes performed by the structures in the dual-branch visual encoder model in the previous embodiment, so they will not be described in detail here. For details, please refer to the steps and technical effects in the previous embodiment.

[0173] In another embodiment, a robot handling system for low-gravity environments is also disclosed. An image acquisition module is used to acquire both a simplified drawing created by an operator and an image of the actual scene inside the current compartment. The simplified drawing includes a first pattern depicting the shape of the target, a second pattern constraining the execution of actions, and a third pattern depicting the shape of the target's destination. A basic instruction generation module is used to input the simplified drawing and the actual scene image into a trained dual-branch visual encoder model to generate a basic action sequence instruction for controlling the robot to complete material handling operations. An angular velocity fine-tuning module is used to monitor the target's rotational angular velocity in real time during target grasping while executing the basic action sequence instruction. If the angular velocity exceeds a set threshold, the robot gripper is driven to rotate in the opposite direction for fine-tuning until the target's rotational angular velocity falls below the set threshold.

[0174] In this embodiment, the basic instruction generation module is used to input the simplified drawing and the real scene image into a trained dual-branch visual encoder model to generate basic action sequence instructions for controlling the robot to complete material handling operations. The dual-branch visual encoder model includes a simplified drawing encoder, a real image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. The simplified drawing encoder extracts features from the input simplified drawing to obtain a first feature vector containing target shape features. The real image encoder extracts features from the input real scene image to obtain a second feature vector containing texture and color information. The first and second feature vectors have the same dimension. The feature embedding layer normalizes the first and second feature vectors to eliminate feature vector length differences and maps them to a cross-modal semantic space through nonlinear transformation. The cross-modal comparison module performs comparative learning on the embedded features, forcing the simplified drawing features of similar objects to be close to the real image features in the embedding space, and distancing the features of dissimilar objects, generating cross-modal aligned fused features. The action generation network module decodes the fused features through multiple layers to generate basic action sequence instructions.

[0175] The functions of the robot handling system in the low-gravity environment described above basically correspond to the steps of the robot handling method in the low-gravity environment disclosed in the previous embodiments. Therefore, they will not be described in detail here. For details, please refer to the embodiments of the robot handling method in the low-gravity environment disclosed above. It should be noted that the embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0176] In other embodiments, a controller is also provided, including a processor and a memory for storing a computer program executable by the processor, wherein the processor is configured to execute the computer program in the memory to implement the various steps of the robot handling method in a low gravity environment as described in the above embodiments.

[0177] If the robot handling method in a low-gravity environment is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above embodiments of the robot handling method in a low-gravity environment. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0179] In summary, the above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be covered by the present invention.

Claims

1. A robot handling method for low-gravity environments, characterized in that, include: The system collects simplified drawings created by the operator and real-world images of the current cabin environment. The simplified drawings include a first pattern depicting the shape of the target, a second pattern constraining the execution of actions, and a third pattern depicting the shape of the target's destination. These simplified drawings and real-world images are then input into a trained dual-branch visual encoder model to generate basic action sequence instructions for controlling the robot to perform material handling operations. The dual-branch visual encoder model includes a simplified drawing encoder, a real-world image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. The simplified drawing encoder extracts features from the input simplified drawings to obtain information including the target shape. The first feature vector is used to extract features from the input real scene image and obtain a second feature vector containing texture and color information. The first and second feature vectors have the same dimension. The feature embedding layer is used to normalize the input first and second feature vectors to eliminate the difference in feature vector lengths and to map the first and second feature vectors to the cross-modal semantic space through nonlinear transformation. The cross-modal comparison module is used to compare and learn the embedded features and force the line drawing features of similar objects to be close to the real image features and the features of dissimilar objects to be far away in the embedding space, generating cross-modal aligned fused features. The motion generation network module is used to generate basic motion sequence instructions after multi-layer decoding of the fused features; during the target grasping process of executing the basic motion sequence instructions, the target rotation angular velocity is monitored in real time. If it is greater than a set threshold, the robot gripper is driven to rotate in the opposite direction for fine adjustment until the target rotation angular velocity is lower than the set threshold.

2. The robot handling method in a low-gravity environment according to claim 1, characterized in that, The process includes acquiring simple sketches drawn by the operator and real scene images inside the cabin, as well as: acquiring continuous frame images inside the cabin and performing anti-glare and motion blur preprocessing, extracting target motion vectors based on the shape features of the simple sketches, dynamically modeling and multi-step predicting the target state space, mapping the predicted trajectory to the image plane to generate a dynamic mask and fusing it with the target mask of the current frame.

3. The robot handling method in a low-gravity environment according to claim 2, characterized in that, Training the dual-branch visual encoder model includes: encoding real material images and line drawings through a dual-channel input network, where the real image channel extracts multi-scale geometric features through a neural network connected to a dilated convolutional layer, and the line drawing channel extracts shape features using a simple convolutional neural network; performing cross-modal fusion of the real material images and line drawing images, aligning multi-scale geometric features and shape features through contrastive learning and constructing an association matrix, assigning the most similar line drawing category to each image; constructing a graph structure, defining image-line drawing pairs as object nodes, and action templates as action nodes, connecting object nodes and action nodes through action execution, with edge weights determined by reinforcement learning reward values; automatically generating action templates, generating initial action templates for each line drawing material type through a rule base, sharing the same action template parameters for objects of the same shape by adjusting only variable ones, and integrating industrial robot kinematic constraints through an action generator; inputting real images into the graph neural network to automatically match the line drawing material type, extracting corresponding optimized action templates from the graph action nodes, generating triplet annotations consisting of images, line drawings, and actions, and storing them in a cross-modal dataset for model training.

4. The robot handling method in a low-gravity environment according to claim 3, characterized in that: The object node includes object feature information of the task target or object feature information of the transport destination, and the object feature information includes shape, size and position; the action node includes gripper type and gripper movement path speed; the additional node includes scene constraint information and priority information.

5. The robot handling method in a low-gravity environment according to claim 4, characterized in that: The second pattern includes a priority marker, wherein the edge weight between two nodes is determined by the spatial distance between the two nodes, the dynamic conflict probability, and the priority marker.

6. The robot handling method in a low-gravity environment according to claim 5, characterized in that, Also includes: The robot monitors the target position in real time during its movement using a 2D camera. After detecting visual feature fluctuations caused by light reflection, an image segmentation model is used to generate a mask based on the target's outline in a simple sketch, which is used to retain only the pixels in the top or bottom areas of the non-reflective target.

7. A robotic handling system for low-gravity environments, characterized in that, include: The image acquisition module is used to acquire the simple sketches drawn by the operator and the real scene images inside the cabin. The simple sketches include a first pattern for depicting the shape of the work target, a second pattern for constraining the execution actions, and a third pattern for depicting the shape of the target transport destination. The basic instruction generation module is used to input the simplified drawing and real scene image into the trained dual-branch visual encoder model to generate basic action sequence instructions for controlling the robot to complete material handling operations. The dual-branch visual encoder model includes a simplified drawing encoder, a real image encoder, a feature embedding layer, a cross-modal comparison module, and an action generation network module. The simplified drawing encoder is used to extract features from the input simplified drawing to obtain a first feature vector containing target shape features. The real image encoder is used to extract features from the input real scene image to obtain a second feature vector containing texture and color information. The first and second feature vectors have the same dimension. The feature embedding layer is used to normalize the input first and second feature vectors to eliminate the difference in feature vector lengths and to map the first and second feature vectors to the cross-modal semantic space through nonlinear transformation. The cross-modal comparison module is used to perform comparative learning on the embedded features to force the simplified drawing features of similar objects to be close to the real image features in the embedding space, and to force the features of dissimilar objects to be far away, generating cross-modal aligned fused features. The motion generation network module is used to generate basic motion sequence instructions after multi-layer decoding of the fused features; the angular velocity fine-tuning module is used to monitor the target rotation angular velocity in real time during the target grasping process of executing the basic motion sequence instructions. If it is greater than a set threshold, the robot gripper is driven to rotate in the opposite direction for fine-tuning until the target rotation angular velocity is lower than the set threshold.

8. A computer-readable storage medium, characterized in that, When the executable computer program in the storage medium is executed by a processor, it can implement the method as described in any one of claims 1-6.

Images