Robot control method, device, equipment, medium and program product

Abstract

The invention provides a robot control method and device, equipment, a medium and a program product. Relates to the technical field of intelligent robots. The method comprises the following steps: acquiring multi-view video data and mechanical arm joint state data; semantic features and spatial features of the multi-view video data and time sequence features of the joint state data of the mechanical arm are extracted; performing bidirectional interactive calculation on the semantic features, the spatial features and the time sequence features by adopting a preset cross-modal cross attention fusion model to generate multi-modal feature representation; and generating a robot control scheme based on the multi-modal feature representation. According to the method, the multi-view video data and the joint state data are deeply associated, and the dependency relationship on the time sequence is captured, so that the robot control precision is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to a robot control method, device, equipment, medium and program product. Background Technology

[0002] Driven by fields such as smart homes, the demand for autonomous operation of intelligent devices such as industrial robots and service robots in complex and dynamic environments is becoming increasingly urgent. These devices need to simultaneously process multimodal inputs such as RGB video, joint angles, and voice commands, establish a dynamic correlation between environmental perception and the device's own state, and complete continuous tasks such as high-precision assembly, sorting, and home services. This places stringent requirements on multimodal information fusion, temporal dependency modeling, and motion control accuracy.

[0003] Current robot control is mostly based on the vision-language-action (VLA) model. It collects visual data and body state time-series data through RGB cameras, built-in sensors, etc., and combines them with natural language commands as input. Various features are extracted by pre-trained visual encoders and text encoders, and then multimodal features are integrated through feature splicing and shallow attention mechanisms. Finally, based on language models or decision networks, action commands are output to drive the actuator to complete operations such as grasping, assembly, and sorting, supporting applications in industrial production, home services and other scenarios.

[0004] However, existing methods suffer from insufficient multimodal feature fusion depth, failing to accurately model the complex relationship between scene changes and joint movements, and exhibiting poor temporal consistency in action sequences. This results in low robot task execution success rates and limited control precision. Summary of the Invention

[0005] The robot control methods, devices, equipment, media, and program products provided in this application are used to achieve highly precise robot control processes.

[0006] In a first aspect, embodiments of this application provide a robot control method, the method comprising:

[0007] Acquire multi-view video data and robotic arm joint status data;

[0008] Extract the semantic and spatial features of the multi-view video data, as well as the temporal features of the robotic arm joint state data;

[0009] A pre-defined cross-modal attention fusion model is used to perform bidirectional interactive calculations on the semantic features, spatial features, and temporal features to generate multimodal feature representations;

[0010] A robot control scheme is generated based on the multimodal feature representation.

[0011] In one possible implementation, the extraction of temporal features from the robotic arm joint state data includes:

[0012] A preset time series model is used to analyze and process the joint state data of the robotic arm to obtain the periodic change data and dependency data in the joint state data;

[0013] The joint state data and the multi-view video data are time-aligned.

[0014] The joint state data, after time alignment and analysis processing, is encoded into a feature vector that matches the semantic and spatial feature dimensions through a fully connected layer.

[0015] Rotational position encoding is used to perform position-aware transformation on the joint state data in order to extract temporal features containing relative position information.

[0016] In one possible implementation, the preset cross-modal attention fusion model includes multiple independent attention heads. The step of using the preset cross-modal attention fusion model to perform bidirectional interactive computation on the semantic features, spatial features, and temporal features to generate a multimodal feature representation includes:

[0017] Using the semantic and spatial features as queries and the temporal features as keys and values, a first round of attention calculation is performed to generate visual features enhanced with joint information.

[0018] Using the temporal features as the query and the semantic and spatial features as the keys and values, a second round of attention calculation is performed to generate temporal features with enhanced visual information.

[0019] The visual features and temporal features are captured and correlated in multiple dimensions using multiple attention heads;

[0020] The output features of multiple attention heads are concatenated to obtain a fused feature matrix;

[0021] The fused feature matrix is ​​normalized and its features are optimized by a linear projection layer to generate a multimodal feature representation.

[0022] In one possible implementation, generating a robot control scheme based on the multimodal feature representation includes:

[0023] The multimodal feature representation and the text sequence of natural language instructions are input into a preset language model for joint encoding;

[0024] The joint encoded features are regressed using a multilayer perceptron motion head to generate a robot control scheme that includes continuous control commands for the robotic arm.

[0025] In one possible implementation, the training steps of the preset cross-modal cross-attention fusion model include:

[0026] Obtain a training dataset, which includes multiple sets of training samples labeled with real motion data. Each set of training samples contains multi-view training video data, training joint state data of the corresponding robotic arm, and natural language training instructions.

[0027] The training samples are preprocessed, including performing brightness adjustment and random occlusion data augmentation operations on the multi-view training video data, aligning the training joint state data with the multi-view training video data in time, and unifying the feature dimensions through a fully connected layer.

[0028] Using the preprocessed multi-view training video data, training joint state data, and natural language training instructions as inputs, and the real motion data as labels, an initial cross-modal cross-attention fusion model is trained.

[0029] If the initial cross-modal cross-attention fusion model satisfies the preset loss function, the training is considered complete, and the preset cross-modal cross-attention fusion model is obtained.

[0030] In one possible implementation, the preset time series model includes a weight adjustment mechanism, and further includes:

[0031] Extract task features from natural language instructions, the task features including task type, operation objective and scenario requirement information;

[0032] The learnable attention parameters of the preset temporal model are interactively calculated with the task features to obtain the dynamic weight values ​​for each time step.

[0033] The joint state data of the robotic arm are analyzed and processed based on the dynamic weight values.

[0034] Secondly, embodiments of this application provide a robot control device, including:

[0035] The acquisition module is used to acquire multi-view video data and robotic arm joint status data;

[0036] An extraction module is used to extract the semantic and spatial features of the multi-view video data, as well as the temporal features of the robotic arm joint state data;

[0037] The computation module is used to perform bidirectional interactive computation on the semantic features, spatial features and temporal features using a preset cross-modal cross-attention fusion model to generate multimodal feature representations;

[0038] The generation module is used to generate a robot control scheme based on the multimodal feature representation.

[0039] Thirdly, embodiments of this application provide a robot control device, including: a memory and a processor;

[0040] The memory stores computer-executed instructions;

[0041] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0043] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0044] The robot control methods, devices, equipment, media, and program products provided in this application can capture information about the robot's surrounding environment from different angles using multi-view video data, reducing information loss and misjudgment caused by limited perspective. The joint state data of the robotic arm accurately reflects the motion state of each joint, providing fundamental data for accurate control of the robotic arm. Extracting semantic features from multi-view video data enables the robot to understand the categories, actions, and semantic meanings of objects in the scene. The high abstractness and versatility of semantic features help the robot quickly identify key information in different environments, improving its adaptability to complex environments. Extracting spatial features allows the robot to accurately determine the spatial position and posture of objects, perform spatial reasoning, plan reasonable motion paths and operation strategies, and ensure accurate and safe operation. Extracting temporal features provides a deeper understanding of the robotic arm's motion patterns and dynamic characteristics, providing a basis for precise control, predicting its future motion state, adjusting strategies in advance, achieving smooth and stable control, and improving operational efficiency and response speed. The cross-modal attention fusion model effectively integrates semantic, spatial, and temporal information to obtain more comprehensive and richer feature representations, overcoming the limitations of a single modality. Bidirectional interactive computing enhances the interaction between different modal features, uncovers potential correlations, and generates more discriminative and representative multimodal feature representations. These multimodal feature representations provide the robot with comprehensive and accurate environmental and self-state information, enabling the generation of more precise control schemes that meet practical needs. Comprehensive utilization of multimodal information leverages the strengths of each modality, compensates for weaknesses, and improves the overall performance of the robot's control system, allowing it to complete tasks more efficiently and reliably. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 An application scenario diagram of a robot control method provided in this application;

[0047] Figure 2 A flowchart illustrating a robot control method provided in an embodiment of this application;

[0048] Figure 3 A flowchart illustrating a robot control method provided in another embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a robot control device provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a robot control device provided in an embodiment of this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0054] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0055] In the rapid development of smart homes, the demand for autonomous operation of intelligent devices such as industrial robots and service robots in complex and dynamic scenarios is becoming increasingly strong. These devices need to simultaneously receive multimodal information such as RGB video, joint angle data, and language commands to build a dynamic correlation between environmental perception and their own state, thereby completing continuous tasks such as high-precision assembly, material sorting, and home services. This places extremely high demands on the efficiency of multimodal information fusion, the ability to model temporal dependencies, and the accuracy of motion control. Current mainstream robot control solutions are based on the Vision-Language-Motion (VLA) model. This model collects visual data and temporal data of the robot's own state through RGB cameras and built-in sensors, and combines this with natural language commands to form a comprehensive input. After the data is processed by pre-trained visual encoders and text encoders to extract various features, multimodal feature integration is achieved through feature concatenation or shallow attention mechanisms. Then, using language models or dedicated decision networks for reasoning, motion commands are output to drive the actuators, enabling operations such as grasping, assembly, and sorting, providing technical support for industrial production and home services. However, existing solutions lack deep interaction in multimodal feature fusion, making it difficult to accurately depict the complex relationship between scene changes and joint movements. Furthermore, the generated action sequences lack consistency in the time dimension, ultimately resulting in a low success rate for robot task execution and control precision that fails to meet the needs of practical applications.

[0056] Therefore, when facing technical problems in existing technologies, to ensure the integrity and synergy of input data, multi-view video data is acquired to fully cover the operation scene and avoid blind spots. Simultaneously, robotic arm joint state data is collected to accurately capture the robot's motion state. The collaborative input of these two types of data enables the robot to more accurately understand the relationship between the environmental state and its own motion. To improve the targeting and effectiveness of feature extraction, considering that multi-view video data needs to clearly define both target attributes and spatial location, semantic and spatial features are extracted. The core value of robotic arm joint state data lies in the motion patterns in the time dimension; therefore, its temporal features are extracted to support the temporal consistency of the action sequence, improving the temporal consistency and periodic adaptability of the action sequence. To strengthen the tightness of multimodal data association, a pre-set cross-modal cross-attention fusion model is adopted, allowing visual features and joint temporal features to act as queries, keys, and values. Through bidirectional interactive computation, the deep associations between modalities are fully explored, thereby more accurately capturing the complex dependencies between visual scene changes and joint movements. This enables the robot to more accurately understand task requirements in dynamic environments and generate more adaptive action sequences, thus improving the accuracy and robustness of task execution.

[0057] Figure 1 This is an application scenario diagram illustrating how the robot control method provided in this application can be implemented. Figure 1As shown, the scene diagram corresponding to the robot control method provided in this application includes: robot 101 and server 102. The robot control device can be integrated into server 102.

[0058] Specifically, when robot 101 begins performing a task, server 102 acquires multi-view video data and robotic arm joint state data of robot 101 in real time; it extracts semantic and spatial features from the multi-view video data, as well as temporal features from the robotic arm joint state data. Then, server 102 uses a preset cross-modal cross-attention fusion model to perform bidirectional interactive calculations on the semantic, spatial, and temporal features to generate a multi-modal feature representation; based on the multi-modal feature representation, it generates a robot control scheme so that robot 101 can perform the task based on the control scheme.

[0059] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0060] Figure 2 This is a flowchart illustrating a robot control method provided in an embodiment of this application, as shown below. Figure 2 As shown, the execution entity in this embodiment is a robot control device. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc, or through a physical device integrating or installing the relevant computer program, such as a chip or robot control equipment. The robot control device can be a server, server cluster, etc. The robot control method provided in this embodiment includes the following steps:

[0061] S201. Acquire multi-view video data and robotic arm joint status data.

[0062] Multi-view video data refers to video stream data that is simultaneously collected by multiple cameras in different locations and completely covers the working area of ​​the robotic arm.

[0063] Among them, the joint status data of the robotic arm refers to the core data that reflects the movement status of the robotic arm, which is collected in real time by the built-in sensors of the robotic arm, such as the rotation angle of each joint of the robotic arm and the opening and closing status of the gripper.

[0064] Optionally, multiple cameras can be configured on the robot for data acquisition, such as a main-view camera and two wrist cameras. The main-view camera is mounted at a fixed high position on the robot's head or in the operating scene to capture the overall operating scene; the two wrist cameras are mounted on both sides of the end effector of the robotic arm to capture close-up details of the interaction between the end effector and the target object, such as the contact state between the gripper and the workpiece, and the local features of the workpiece. All cameras record video at a high frame rate. During the acquisition process, brightness adjustment is actively introduced to simulate different lighting environments such as strong light, weak light, and backlight, as well as random occlusion and situations such as clutter obstruction and local field of view in the scene. This data augmentation strategy improves the adaptability of the subsequent model to complex environments.

[0065] Optionally, for a six-axis robotic arm, real-time rotation angle data of each joint is collected, along with the opening and closing status data of the gripper. High-precision position sensors built into the robotic arm, such as encoders, are used to collect joint angle and gripper status data in real time, with the acquisition frequency consistent with the frame rate of the multi-view video data.

[0066] Specifically, the robot control device communicates with the robot to acquire multi-view video data and robotic arm joint status data collected from the robot.

[0067] S202. Extract semantic and spatial features from multi-view video data, as well as temporal features from robotic arm joint state data.

[0068] Among them, semantic features refer to high-level abstract information extracted from multi-view video data to represent what something is. They can reflect the category of the target object in the video, such as parts or workstations; scene attributes, such as assembly tables or sorting areas; and task-related semantics, such as objects to be grabbed or their placement locations.

[0069] Spatial features refer to detailed information extracted from multi-view video data to characterize where or what form the object is, including the shape and structure of the target object, such as cylindrical or square; and its spatial location, such as 30cm to the left of the robotic arm.

[0070] Among them, temporal features refer to the features extracted from the joint state data of the robotic arm that reflect the changing patterns in the time dimension. These features include the periodic patterns of joint movement, such as the joint angle change cycle of repeated grasping actions; dynamic dependencies, such as the linkage change of joint 2 after joint 1 rotates; and action temporal logic, such as the position adjustment sequence of joints before the gripper closes.

[0071] Optionally, the acquired multi-view video stream is split into consecutive image frames according to timestamps, and blurry or abnormal frame data is removed; all image frames are uniformly adjusted to a fixed size and the image frames are standardized.

[0072] Specifically, a dual-encoder collaborative approach is employed to extract two types of features. A pre-trained semantic-aware encoder, such as the SigLIP model, encodes image patches for each frame, focusing on identifying core semantic information such as target object categories and scene attributes, outputting a high-dimensional semantic feature vector. A pre-trained spatial-aware encoder, such as the DinoV2 model, focuses on capturing the edge contours, texture details, spatial coordinate relationships, and relative positional relationships of objects in the image, outputting a spatial feature vector. The semantic and spatial feature vectors corresponding to the same image frame are concatenated along the feature channel dimension. The joint features of all image frames are then sorted chronologically to preserve the temporal dimension information of the multi-view video data.

[0073] Optionally, the collected joint angle and gripper status data are cleaned to remove outliers caused by sensor noise, smoothing techniques are used to correct data fluctuations, and the data is sorted according to timestamp order to form a complete joint status time sequence.

[0074] Furthermore, using frame timestamps from multi-view video data as a benchmark, the joint state temporal sequence is time-aligned, and the aligned joint data is standardized. A specialized temporal modeling model is employed to analyze the standardized joint state sequence, using a dynamic weighting mechanism to capture periodic motion patterns and long-range dependencies in the sequence, such as the influence of earlier joint positions on subsequent movements. Feature transformation maps the original joint data into high-dimensional temporal feature vectors.

[0075] S203. A pre-defined cross-modal attention fusion model is used to perform bidirectional interactive computation on semantic features, spatial features, and temporal features to generate multimodal feature representations.

[0076] Among them, the pre-trained cross-modal attention fusion model refers to a pre-trained model used to integrate different types of features.

[0077] Among them, bidirectional interactive computing refers to the bidirectional information enhancement process in which visual features and temporal features act as both the queryer and the queryee.

[0078] Among them, multimodal feature representation refers to the unified feature vector formed after cross-modal fusion.

[0079] Optionally, the three types of features extracted in the early stage are uniformly standardized. The visual joint features formed by splicing semantic features and spatial features and the separate temporal features are subjected to dimensional standardization. Timestamps are added to the two types of features to preserve their correspondence in the time dimension.

[0080] Specifically, using joint visual features as queries and temporal features as keys and values, an attention mechanism is used to filter out temporal information strongly correlated with the current visual scene, generating visually enhanced temporal features. Conversely, using temporal features as queries and joint visual features as keys and values, visual information matching the current joint motion state is filtered out. For example, when the temporal features indicate that the gripper is closing, the focus is on the contact details between the gripper and the part, the changes in the part's posture, etc., in the visual features, strengthening the feature representation of this associated visual information, and generating visual features enhanced with joint information.

[0081] Furthermore, all the enhanced visual features and enhanced temporal features from the output are summarized and concatenated to form a fusion feature matrix containing multi-dimensional correlation information. This matrix is ​​then optimized and its dimensions are regularized using a linear projection layer to remove redundant information, strengthen core correlation features, and transform the matrix into a high-dimensional vector with fixed dimensions; the final vector obtained is the multimodal feature representation.

[0082] S204. Robot control scheme based on multimodal feature representation.

[0083] Among them, the robot control scheme refers to the set of specific instructions that can drive the robot to accurately execute the target task, including the motion parameters of each joint of the robotic arm, the opening and closing control parameters of the gripper, etc.

[0084] Specifically, user-issued task instructions, such as placing part A into workstation B, are standardized and converted into text feature vectors recognizable by the model. The multimodal feature representation and the text feature vectors are then input into a pre-defined language model backbone network. Through the network's multi-layered encoding structure, a deep binding between scene information, action patterns, and task objectives is achieved. A multilayer perceptron action head is used as the decision output module, taking the jointly encoded comprehensive features as input. Through multiple rounds of feature transformation and inference, continuous action control parameters are directly regressed and output. The temporal features in the multimodal feature representation are combined to verify the coherence and rationality of the action sequence, eliminating abnormal parameters that may lead to equipment malfunction or task failure. The verified action parameters and temporal logic are standardized and encapsulated according to the robot actuator's protocol requirements, forming a structured robot control scheme.

[0085] The robot control method provided in this application can capture information about the robot's surrounding environment from different angles using multi-view video data, reducing information loss and misjudgment caused by limited perspective. The joint state data of the robotic arm accurately reflects the motion state of each joint, providing fundamental data for accurate control of the robotic arm. Extracting semantic features from multi-view video data enables the robot to understand the categories, actions, and semantic meanings of objects in the scene. The high abstraction and versatility of semantic features help the robot quickly identify key information in different environments, improving its adaptability to complex environments. Extracting spatial features allows the robot to accurately determine the spatial position and posture of objects, perform spatial reasoning, plan reasonable motion paths and operation strategies, and ensure accurate and safe operation. Extracting temporal features provides a deeper understanding of the robotic arm's motion patterns and dynamic characteristics, providing a basis for precise control and predicting its future motion state, allowing for advance strategy adjustments, achieving smooth and stable control, and improving operational efficiency and response speed. The cross-modal cross-attention fusion model effectively integrates semantic, spatial, and temporal information to obtain more comprehensive and richer feature representations, overcoming the limitations of a single modality. Bidirectional interactive computing enhances the interaction between different modal features, uncovers potential correlations, and generates more discriminative and representative multimodal feature representations. These multimodal feature representations provide the robot with comprehensive and accurate environmental and self-state information, enabling the generation of more precise control schemes that meet practical needs. Comprehensive utilization of multimodal information leverages the strengths of each modality, compensates for weaknesses, and improves the overall performance of the robot's control system, allowing it to complete tasks more efficiently and reliably.

[0086] As an optional implementation, based on the above embodiments, the temporal features of the robotic arm joint state data are extracted, including:

[0087] A preset time series model is used to analyze and process the joint state data of the robotic arm to obtain the periodic change data and dependency data in the joint state data;

[0088] Time alignment processing is performed on joint state data and multi-view video data;

[0089] The joint state data, after time alignment and analysis processing, is encoded into a feature vector that matches the semantic and spatial feature dimensions through a fully connected layer.

[0090] Rotational position coding is used to perform position-aware transformation on joint state data in order to extract temporal features containing relative position information.

[0091] Among them, the preset time series model refers to the model used to mine the patterns of time series data. It has the ability to capture the periodic changes and long-term dependencies of data and can extract valuable motion features from continuous joint state data.

[0092] Among them, periodic change data refers to information in joint state data that presents a cyclical pattern, such as the cyclical change pattern of joint angles from the initial value to the target value and back to the initial value when a robotic arm repeatedly performs a grasping action.

[0093] Among them, dependency data refers to the logical relationship information reflected in the joint state data, including the state relationship of the same joint at different time steps, the linkage relationship between different joints, etc.

[0094] Specifically, joint state data is organized into a continuous time series according to timestamps. This series is then input into a preset time series model, such as the Moriai model. Through a dynamic weighting mechanism, the system automatically focuses on key change nodes in joint movement, capturing periodic change data, such as identifying the cycle period of joint angles and the regular frequency of gripper opening and closing during repetitive operations of the robotic arm. Dependency data is also mined, for example, analyzing the linkage logic between joints, such as joint 4 only starting to adjust after joint 3 rotates to a specific angle. The final output contains preliminary time series features containing periodicity and dependencies.

[0095] Furthermore, frame timestamps from the multi-view video data and acquisition timestamps from the joint state data are extracted. Using the frame rate of the video data as a benchmark, the joint state data is time-calibrated. For acquisition points in the joint state data that exceed the number of video frames, interpolation is used to preserve key motion information; for missing acquisition points, smooth fitting of data from adjacent time points is used to supplement them.

[0096] Furthermore, the joint state data, after time-series model analysis and time alignment, is input into a pre-defined fully connected layer. This preserves core information about periodic changes and dependencies, mapping the original low-dimensional data such as joint angles and gripper states into high-dimensional feature vectors. Simultaneously, the output dimension of the fully connected layer is adjusted based on the dimensional parameters of the semantic and spatial features to ensure that the encoded joint state feature vectors match the dimensions of the visual features.

[0097] Furthermore, rotational position encoding is applied to the feature vector encoded by the fully connected layer for positional awareness enhancement. By simulating spatial rotation transformations, the relative positional information of the joint state data in the time series is injected into the feature vector. For adjacent feature points in the time series, rotational transformations are used to reflect their sequential dependence; for feature points spaced a certain number of time steps apart, the difference in rotation angles is used to characterize their relative positional associations, thereby extracting temporal features containing relative positional information.

[0098] The robot control method provided in this application employs a preset temporal model to analyze and process joint state data. This enables precise acquisition of periodic change data, providing a basis for subsequent motion control. Simultaneously, the acquired dependency data reveals the interrelationships between the movements of various joints in the robotic arm, preventing motion errors caused by neglecting the mutual influence between joints. Time alignment processing is performed on the joint state data and multi-view video data to ensure consistency in the temporal dimension between the two different data types. A fully connected layer encodes the processed joint state data into feature vectors matching semantic and spatial feature dimensions, solving the problem of difficulty in fusion due to inconsistent dimensions of different modal data. Rotational position encoding is used to perform position-aware transformation on the joint state data, greatly enhancing the expressive power of temporal features and enabling a more accurate description of the robotic arm's motion process and operational details, providing strong support for generating more precise control schemes.

[0099] As an optional implementation, based on the above embodiments, a preset cross-modal attention fusion model includes multiple independent attention heads. This preset cross-modal attention fusion model is used to perform bidirectional interactive computation on semantic features, spatial features, and temporal features to generate multimodal feature representations, including:

[0100] Using semantic and spatial features as queries and temporal features as keys and values, the first round of attention calculation is performed to generate visual features enhanced with joint information.

[0101] Using temporal features as queries and semantic and spatial features as keys and values, a second round of attention calculation is performed to generate temporal features with enhanced visual information.

[0102] Multiple attention heads are used to capture visual features and temporal features in a multi-dimensional correlation.

[0103] The output features of multiple attention heads are concatenated to obtain a fused feature matrix;

[0104] The fused feature matrix is ​​normalized and its features are optimized by a linear projection layer to generate a multimodal feature representation.

[0105] In this context, an attention head refers to an independent feature association calculation unit in a pre-defined cross-modal attention fusion model, with each attention head focusing on a specific intermodal association dimension.

[0106] Specifically, the semantic and spatial features extracted earlier are concatenated to form a unified visual joint feature, which serves as the query. The extracted temporal features of the robotic arm joints are used as both keys and values ​​for calculation. By calculating the similarity between the visual joint feature and the temporal feature (key), joint motion information strongly related to the current scene is selected. The selected joint motion information (value) is then weighted and fused with the original visual joint feature to generate visual features enhanced with joint information.

[0107] Furthermore, the temporal features of the robotic arm joints are used as the query, while the visual joint features formed by concatenating semantic and spatial features are used as both the key and value. By calculating the similarity between the temporal features (query) and the visual joint features (key), scene information matching the current joint motion state is filtered out. The filtered scene information (value) is then weighted and fused with the original temporal features to strengthen the expression of motion patterns related to scene adaptation in the temporal features, generating visually enhanced temporal features.

[0108] Furthermore, multiple independent attention heads configured in the model are activated. Each attention head performs parallel computation on the visual features enhanced by joint information generated in the first round and the temporal features enhanced by visual information generated in the second round, targeting different association dimensions. After each attention head independently completes association capture, it outputs the feature interaction results for the corresponding dimension. The one-dimensional feature vectors output by each attention head are arranged horizontally to form a two-dimensional fusion feature matrix. The fusion feature matrix is ​​input into a linear projection layer. Through dimension normalization, the high-dimensional fusion feature matrix is ​​compressed into a fixed-dimensional feature vector. Simultaneously, through weight optimization of the projection layer, redundant information in the feature matrix is ​​removed, and core association features are strengthened, ultimately outputting a compact and comprehensive multimodal feature representation.

[0109] The robot control method provided in this application generates visual features enhanced with joint information, enriching the information about the dynamic movement of the robotic arm joints within the visual features; it also generates temporal features enhanced with visual information, integrating the temporal features with the semantic and spatial layout information of the visual scene, overcoming the limitations of the one-sidedness of single-modal information. Multiple independent attention heads can correlate and capture visual and temporal features from different dimensions, further enriching the information contained in the features, enabling the generated multimodal feature representation to more comprehensively reflect the complex state of the robotic arm and its operating environment. The fusion of feature matrices avoids information omissions or biases that may exist with a single attention head. Through linear projection layers, the fusion feature matrix is ​​dimensionally regularized and feature optimized, resulting in a more refined and effective multimodal feature representation that better serves subsequent tasks such as robot control.

[0110] As an optional implementation, based on the above embodiments, a robot control scheme is generated based on multimodal feature representation, including:

[0111] The multimodal feature representation and the text sequence of natural language instructions are input into a pre-defined language model for joint encoding;

[0112] By using a multilayer perceptron motion head to perform regression calculations on the jointly encoded features, a robot control scheme including continuous control commands for the robotic arm is generated.

[0113] Among them, the text sequence of natural language instructions refers to the continuous text data formed after the user's task objective description has been standardized.

[0114] Among them, the pre-defined language model refers to the backbone network that has the ability to jointly encode multimodal features and text features.

[0115] Specifically, the natural language instructions are standardized, redundant expressions are removed, and core task information is extracted. The processed text is then converted into text feature vectors that the model can recognize. The adapted text feature vectors, along with multimodal feature representations, are input into a pre-defined language model, such as the Llama 2 model. The model uses a self-attention mechanism with a multi-layer Transformer structure to achieve deep association of the three types of information. Finally, the output is a joint encoded feature that integrates the task objective, scene information, and motion patterns.

[0116] Furthermore, the jointly encoded comprehensive features are input into the multilayer perceptron motion head. First, core information directly related to motion decisions is filtered out, such as the joint angle range corresponding to the part's position. This information is then mapped into intermediate feature vectors related to the motion parameters. Based on these intermediate feature vectors, continuous motion control parameters are directly output through regression calculations. For each joint of the robotic arm, precise target rotation angle, motion speed, and acceleration threshold are output; for the gripper, clear opening / closing degree values ​​and motion triggering timing are output. Finally, all continuous control commands are sorted according to the motion execution sequence, clarifying the cooperative relationship between each joint and the gripper, and supplementing emergency adjustment strategies. If no part is detected, the motion is paused and a task anomaly is reported, forming a complete robot control scheme.

[0117] The robot control method provided in this application integrates multimodal feature representations with the text sequence of natural language commands into a preset language model for joint encoding. This achieves deep fusion of multi-dimensional information such as vision, space, and temporal sequence with language command information, enabling a more accurate understanding of the specific operational intent corresponding to the natural language commands and avoiding erroneous operations caused by information misunderstanding. The preset language model possesses powerful semantic understanding capabilities, capturing the semantic relationships between multimodal features and natural language commands during the joint encoding process. By performing regression calculations on the jointly encoded features using a multilayer perceptron motion head, a robot control scheme including continuous control commands for the robotic arm can be generated. These continuous control commands can adjust the gripping force and movement speed of the robotic arm in real time based on factors such as the object's position, shape, and weight, ensuring the stability and accuracy of the gripping process.

[0118] As an optional implementation, based on the above embodiments, a training step for the cross-modal cross-attention fusion model is preset, including:

[0119] Obtain the training dataset, which includes multiple sets of training samples labeled with real motion data. Each set of training samples contains multi-view training video data, corresponding robotic arm training joint state data, and natural language training instructions.

[0120] The training samples are preprocessed, including performing brightness adjustment and random occlusion data augmentation on the multi-view training video data, aligning the training joint state data with the multi-view training video data in time, and unifying the feature dimensions through a fully connected layer.

[0121] The initial cross-modal cross-attention fusion model was trained using preprocessed multi-view training video data, training joint state data, and natural language training instructions as inputs, and real motion data as labels.

[0122] If the initial cross-modal cross-attention fusion model satisfies the preset loss function, the training is considered complete, and the preset cross-modal cross-attention fusion model is obtained.

[0123] Among them, real motion data refers to the actual motion parameters of the robot when performing tasks, which are labeled in the training samples, including the actual rotation angles of each joint of the robotic arm, the actual opening and closing states of the gripper, and the timing logic.

[0124] The preset loss function is a quantitative indicator used to measure the difference between the model's predicted actions and the actual action data. When the indicator reaches the preset standard, it indicates that the model training effect has met the standard.

[0125] Understandably, the training dataset revolves around the robot's target operational tasks, such as parts grasping and opening the electrical box door, and is collected using multi-view cameras and sensors built into the robotic arm under different environmental conditions, such as different lighting and different target positions. Each training sample needs to simultaneously collect three parts of input data: multi-view training video data, training joint state data, and natural language training commands. Finally, the motion process of each sample is accurately labeled, recording the actual rotation angle of each joint of the robotic arm at different times, the actual opening and closing degree of the gripper, and the timing coordination logic, forming complete training samples corresponding to inputs and labels, which are finally summarized into the training dataset.

[0126] Specifically, an initial model is constructed, comprising a multimodal feature extraction module, a cross-modal attention fusion module, and a feature encoding module. The cross-attention module is configured with multiple independent attention heads to uncover deep intermodal relationships. Next, input and labels are defined: preprocessed multi-view training video features, training joint state features, and natural language training command features are used as model input, while labeled real action data is used as training labels. Iterative training is performed: the model calculates predicted action data through forward propagation; the difference between the predicted action data and the real action data is compared, and the model parameters are adjusted using the backpropagation algorithm; this process is repeated to continuously reduce the prediction error until the model performance stabilizes.

[0127] Furthermore, during training, the value of a preset loss function, such as the smooth L1 loss function, is calculated in real time. This value directly reflects the magnitude of the error between the predicted action and the actual action. When the loss function value remains stable for several consecutive rounds and is below a preset threshold (e.g., the error value is less than 0.01), it indicates that the model has fully learned the correlation between multimodal inputs and action outputs, and the training effect has met the target. At this point, training is stopped, the current model parameters are fixed, and the final usable preset cross-modal cross-attention fusion model is obtained.

[0128] The robot control method provided in this application acquires a training dataset containing multi-view training video data, corresponding robotic arm training joint state data, and natural language training instructions, all labeled with real motion data, providing comprehensive and rich information for model training. During training, the model attempts to generate predicted motion data based on the input multi-view video data, joint state data, and natural language instructions, and calculates the loss function value by comparing it with the real motion data. The model continuously adjusts its parameters based on the feedback from the loss function, gradually learning the mapping relationship from input data to correct actions, thereby effectively guiding the model to learn and optimize in the correct direction.

[0129] As an optional implementation, based on the above embodiments, the preset time series model includes a weight adjustment mechanism, and further includes:

[0130] Extract task features from natural language instructions. Task features include task type, operation objective, and scenario requirements.

[0131] The learnable attention parameters of the pre-set time series model are interactively calculated with the task features to obtain the dynamic weight values ​​for each time step.

[0132] The joint state data of the robotic arm is analyzed and processed based on dynamic weight values.

[0133] The weight adjustment mechanism refers to the core mechanism in the preset time series model used to dynamically allocate computing resources. By adjusting the importance weights of joint state data at different time steps in real time in combination with task requirements, the model can focus on key action data of the task, weaken redundant information, and improve the pertinence of time series feature extraction.

[0134] Among them, learnable attention parameters refer to parameters in the preset time series model that can be optimized through training iterations, and have the ability to capture the relationship between task features and joint data.

[0135] Among them, the dynamic weight value refers to the personalized weight calculated for each time step of the joint state data.

[0136] Specifically, the input natural language instructions are semantically parsed to extract task information, including the task type (identifying the core actions the robot needs to perform), the operational objective (determining the object to which the actions affect), and the scenario requirements (clarifying the constraints for task execution). This information is integrated into structured task features. The extracted task features are input into a pre-defined temporal model, which uses learnable attention parameters to uncover the correlation between task features and time steps in joint state data. For example, if the task feature is high-precision part grasping, time steps related to the gripper approaching the part or gripper closing will be assigned a high correlation. Based on this correlation calculation, a dynamic weight value is generated for each time step of the joint state data, with high weight values ​​for time steps strongly related to the core task actions and low weight values ​​for time steps irrelevant to the task or redundant.

[0137] Furthermore, the calculated dynamic weight values ​​are matched with the time-series data of the robotic arm joint states, and model computational resources are allocated according to the weight values. For key time-step data with high weight values, the model performs refined analysis, focusing on capturing motion details, joint linkages, and subtle changes; for redundant time-step data with low weight values, the model simplifies the calculation process, retaining only core trend information and downplaying detail processing. Through analysis, the model extracts periodic change data and dynamic dependencies adapted to the current task, generating more targeted time-series features.

[0138] The robot control method provided in this application extracts task features from natural language instructions, covering task type, operation objectives, and scenario requirements. It can accurately extract key content closely related to the robotic arm operation from complex natural language descriptions. By combining the extracted task features with subsequent time-series data processing, the model can establish a direct correlation between the task and the robotic arm joint state data. A dynamic weighting mechanism allows the model to flexibly adjust its focus on the robotic arm joint state data according to the specific circumstances of different tasks and time points. Through dynamic weight adjustment, the model can increase its focus on key time step data, more accurately capture this key information, and thus better understand the robotic arm's motion process and task execution state, providing strong support for generating reasonable control instructions.

[0139] Figure 3 A flowchart illustrating a robot control method provided in another embodiment of this application is shown below. Figure 3 As shown, the robot control method provided in this embodiment includes the following steps:

[0140] S301. Acquire multi-view video data and robotic arm joint status data.

[0141] S302. Extract semantic and spatial features from multi-view video data.

[0142] S303. Use a preset time series model to analyze and process the joint state data of the robotic arm to obtain the periodic change data and dependency data in the joint state data.

[0143] S304. Perform time alignment processing on the joint state data and multi-view video data.

[0144] S305. The joint state data after time alignment and analysis processing is encoded into a feature vector that matches the semantic and spatial feature dimensions through a fully connected layer.

[0145] S306. Rotational position coding is used to perform position-aware transformation on joint state data in order to extract temporal features containing relative position information.

[0146] S307. Using semantic and spatial features as queries and temporal features as keys and values, perform the first round of attention calculation to generate visual features enhanced with joint information.

[0147] S308. Using temporal features as the query and semantic and spatial features as keys and values, perform a second round of attention calculation to generate temporal features with enhanced visual information.

[0148] S309. Visual features and temporal features are captured in a multi-dimensional way by using multiple attention heads.

[0149] S310. Concatenate the output features of multiple attention heads to obtain a fused feature matrix.

[0150] S311. The fusion feature matrix is ​​normalized and its features are optimized through a linear projection layer to generate a multimodal feature representation.

[0151] S312. Input the multimodal feature representation and the text sequence of natural language instructions into the preset language model for joint encoding.

[0152] S313. Regression calculation is performed on the jointly encoded features through a multilayer perceptron motion head to generate a robot control scheme that includes continuous control commands for the robotic arm.

[0153] In this embodiment, the implementation method and technical effect of S301-S313 are similar to those of the corresponding solutions in the above embodiments, and will not be repeated here.

[0154] Figure 4 A schematic diagram of the robot control device provided in this application is shown below. Figure 4 As shown, the robot control device 40 provided in this embodiment includes: an acquisition module 41, an extraction module 42, a calculation module 43, and a generation module 44.

[0155] The acquisition module 41 is used to acquire multi-view video data and robotic arm joint state data; the extraction module 42 is used to extract semantic features and spatial features of multi-view video data, as well as temporal features of robotic arm joint state data; the calculation module 43 is used to perform bidirectional interactive calculation on semantic features, spatial features and temporal features using a preset cross-modal cross-attention fusion model to generate multi-modal feature representation; and the generation module 44 is used to generate robot control scheme based on multi-modal feature representation.

[0156] The robot control device provided in this embodiment can perform... Figure 2 The implementation principles and technical effects of the methods shown are similar, and will not be repeated here.

[0157] Optionally, the extraction module 42, when extracting the temporal features of the robotic arm joint state data, specifically performs the following: analyzes and processes the robotic arm joint state data using a preset temporal model to obtain periodic change data and dependency data in the joint state data; performs time alignment processing on the joint state data and multi-view video data; encodes the time-aligned and analyzed joint state data into a feature vector matching the semantic and spatial feature dimensions through a fully connected layer; and performs position-aware transformation on the joint state data using rotational position encoding to extract temporal features containing relative position information.

[0158] Optionally, the preset cross-modal attention fusion model includes multiple independent attention heads. The computation module 43, when using the preset cross-modal attention fusion model to perform bidirectional interactive computation on semantic features, spatial features, and temporal features to generate a multimodal feature representation, specifically performs the following: First-round attention computation using semantic and spatial features as queries and temporal features as keys and values ​​to generate visual features enhanced with joint information; second-round attention computation using temporal features as queries and semantic and spatial features as keys and values ​​to generate temporal features enhanced with visual information; multi-dimensional association capture of visual and temporal features is performed through multiple attention heads; the output features of multiple attention heads are concatenated to obtain a fusion feature matrix; and the fusion feature matrix is ​​dimensionally regularized and feature optimized through a linear projection layer to generate a multimodal feature representation.

[0159] Optionally, the generation module 44, when generating a robot control scheme based on multimodal feature representation, is specifically used to: input the multimodal feature representation and the text sequence of natural language instructions into a preset language model for joint encoding; and perform regression calculation on the jointly encoded features through a multilayer perceptron motion head to generate a robot control scheme including continuous control instructions for the robotic arm.

[0160] Optionally, the robot control device provided in this embodiment further includes a preprocessing module, a training module, and a determination module.

[0161] Correspondingly, the acquisition module 41 is also used to acquire the training dataset, which includes multiple sets of training samples labeled with real motion data. Each set of training samples contains multi-view training video data, corresponding robotic arm training joint state data, and natural language training instructions. The preprocessing module is used to preprocess the training samples. Preprocessing includes performing data augmentation operations such as brightness adjustment and random occlusion on the multi-view training video data, temporally aligning the training joint state data with the multi-view training video data, and unifying the feature dimensions through a fully connected layer. The training module is used to train an initial cross-modal cross-attention fusion model using the preprocessed multi-view training video data, training joint state data, and natural language training instructions as input and real motion data as labels. The determination module is used to determine the end of training in response to the initial cross-modal cross-attention fusion model satisfying the preset loss function, so as to obtain the preset cross-modal cross-attention fusion model.

[0162] Optionally, the robot control device provided in this embodiment also includes an analysis module.

[0163] Correspondingly, the extraction module 42 is also used to extract task features from natural language instructions, which include task type, operation target and scenario requirements information; the calculation module 43 is also used to perform interactive calculations with the task features through the learnable attention parameters of the preset time series model to obtain the dynamic weight values ​​of each time step; the analysis module is used to analyze and process the joint state data of the robotic arm based on the dynamic weight values.

[0164] Figure 5 A schematic diagram of the robot control device provided in this application. Figure 5 As shown, the robot control device 50 provided in this embodiment includes a processor 51 and a memory 52. ​​The processor 51 and the memory 52 are connected via a bus and communicate with each other.

[0165] In the specific implementation process, the processor 51 executes the computer execution instructions stored in the memory 52, causing the processor 51 to perform the above-described method.

[0166] The specific implementation process of processor 51 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0167] In the above embodiments, it should be understood that the processor 51 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0168] The memory 52 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0169] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0170] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0171] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0172] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0173] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0174] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0177] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0179] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A robot control method, characterized in that, The method includes: Acquire multi-view video data and robotic arm joint status data; Extract the semantic and spatial features of the multi-view video data, as well as the temporal features of the robotic arm joint state data; A pre-defined cross-modal attention fusion model is used to perform bidirectional interactive calculations on the semantic features, spatial features, and temporal features to generate multimodal feature representations; A robot control scheme is generated based on the multimodal feature representation.

2. The method according to claim 1, characterized in that, The extraction of temporal features from the robotic arm joint state data includes: A preset time series model is used to analyze and process the joint state data of the robotic arm to obtain the periodic change data and dependency data in the joint state data; The joint state data and the multi-view video data are time-aligned. The joint state data, after time alignment and analysis processing, is encoded into a feature vector that matches the semantic and spatial feature dimensions through a fully connected layer. Rotational position encoding is used to perform position-aware transformation on the joint state data in order to extract temporal features containing relative position information.

3. The method according to claim 1, characterized in that, The preset cross-modal attention fusion model includes multiple independent attention heads. The preset cross-modal attention fusion model is used to perform bidirectional interactive computation on the semantic features, spatial features, and temporal features to generate a multimodal feature representation, including: Using the semantic and spatial features as queries and the temporal features as keys and values, a first round of attention calculation is performed to generate visual features enhanced with joint information. Using the temporal features as the query and the semantic and spatial features as the keys and values, a second round of attention calculation is performed to generate temporal features with enhanced visual information. The visual features and temporal features are captured and correlated in multiple dimensions using multiple attention heads; The output features of multiple attention heads are concatenated to obtain a fused feature matrix; The fused feature matrix is ​​normalized and its features are optimized by a linear projection layer to generate a multimodal feature representation.

4. The method according to claim 1, characterized in that, The robot control scheme generated based on the multimodal feature representation includes: The multimodal feature representation and the text sequence of natural language instructions are input into a preset language model for joint encoding; The joint encoded features are regressed using a multilayer perceptron motion head to generate a robot control scheme that includes continuous control commands for the robotic arm.

5. The method according to claim 1, characterized in that, The training steps of the preset cross-modal cross-attention fusion model include: Obtain a training dataset, which includes multiple sets of training samples labeled with real motion data. Each set of training samples contains multi-view training video data, training joint state data of the corresponding robotic arm, and natural language training instructions. The training samples are preprocessed, including performing brightness adjustment and random occlusion data augmentation operations on the multi-view training video data, aligning the training joint state data with the multi-view training video data in time, and unifying the feature dimensions through a fully connected layer. Using the preprocessed multi-view training video data, training joint state data, and natural language training instructions as inputs, and the real motion data as labels, an initial cross-modal cross-attention fusion model is trained. If the initial cross-modal cross-attention fusion model satisfies the preset loss function, the training is considered complete, and the preset cross-modal cross-attention fusion model is obtained.

6. The method according to claim 2, characterized in that, The preset time series model includes a weight adjustment mechanism, and also includes: Extract task features from natural language instructions, the task features including task type, operation objective and scenario requirement information; The learnable attention parameters of the preset temporal model are interactively calculated with the task features to obtain the dynamic weight values ​​for each time step. The joint state data of the robotic arm are analyzed and processed based on the dynamic weight values.

7. A robot control device, characterized in that, include: The acquisition module is used to acquire multi-view video data and robotic arm joint status data; An extraction module is used to extract the semantic and spatial features of the multi-view video data, as well as the temporal features of the robotic arm joint state data; The computation module is used to perform bidirectional interactive computation on the semantic features, spatial features and temporal features using a preset cross-modal cross-attention fusion model to generate multimodal feature representations; The generation module is used to generate a robot control scheme based on the multimodal feature representation.

8. A robot control device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images