Large model and small model collaborative robot operation action real-time control method and system

Abstract

The invention relates to a large model and small model collaborative robot operation action real-time control method and a large model and small model collaborative robot operation action real-time control system. The method comprises the following steps: firstly, collecting and preprocessing multi-modal data; carrying out data coding on the multi-modal data to obtain a feature vector; aligning the multi-modal data by using a cross-modal token alignment technology to obtain a feature representation; then training a neural network model by using the feature vector and the feature representation to obtain a trained large model; pruning, distilling and quantifying the trained large model to generate a small model; the small model is deployed to an edge computing device, the edge computing device obtains current instruction text data in real time and conducts reasoning on the current instruction text data, and an action plan of the robot is generated; and finally, the robot is controlled to operate and act in real time by means of action planning of the robot and feedback signals of the sensors. Compared with the prior art, the method has the advantages of low delay, high efficiency, capability of improving tactile perception, self-adaptive grabbing and real-time action planning and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a real-time control method and system for robot operation actions coordinated by a large model and a small model. Background Art

[0002] With the rapid development of artificial intelligence technology, robotics has been widely used in manufacturing, service, medical, logistics and other fields. Especially in robot operation and grasping tasks in complex environments, as the diversity and complexity of tasks increase, how robots can complete tasks quickly and accurately has become an important research direction.

[0003] At present, the success of robot manipulation and grasping tasks often depends on its perception ability and control strategy. Traditional robot grasping methods mainly rely on single sensor data, such as visual sensors or tactile sensors, which are difficult to effectively handle complex environmental changes and multiple interference factors, so they have certain limitations in practical applications.

[0004] In order to improve the robot's grasping accuracy and efficiency in complex environments, researchers have proposed a method that integrates multiple sensory information. Visual sensors can provide information such as the position, shape, and color of an object, but they are highly sensitive to changes in ambient lighting and surface texture of the object. Tactile sensors can sense force feedback in real time during the grasping process, but in complex operations, relying solely on force perception cannot effectively respond to changes in object shape, stiffness, and other characteristics.

[0005] In addition, with the advancement of deep learning technology, large language models have achieved remarkable results in image recognition, speech processing, natural language processing and other fields. Large models can acquire powerful understanding capabilities for complex tasks by learning from massive amounts of data. However, in practical applications, large models have disadvantages such as high consumption of computing resources and slow reasoning speed, making it difficult to meet real-time and edge computing requirements.

[0006] At present, robot operating systems based on edge computing are gradually being used. By deploying models to edge devices, fast real-time reasoning and response can be achieved, but they still face challenges in data processing, computing resources, real-time performance, etc. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a real-time control method and system for robot operation actions in which a large model and a small model are coordinated.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to one aspect of the present invention, a real-time control method for robot operation actions in which a large model and a small model are coordinated is provided, and the method comprises the following steps:

[0010] S1. Collect environmental data using various sensors, combine them with instruction text data to form multimodal data, and preprocess the multimodal data to obtain preprocessed multimodal data;

[0011] S2, encoding the preprocessed multimodal data to obtain a feature vector;

[0012] S3, using cross-modal token alignment technology to align the pre-processed multimodal data to obtain feature representation;

[0013] S4, using the feature vector and feature representation to train the neural network model to obtain a trained large model;

[0014] S5. Prune, distill and quantize the trained large model to generate a small model.

[0015] S6. Deploy the small model to the edge computing device. The edge computing device obtains the current command text data in real time and infers it to generate the robot's action plan.

[0016] S7. Use the robot's motion planning and the feedback signals from each sensor to control the robot's operating actions in real time.

[0017] As a preferred technical solution, S1's environmental data includes image data, positioning data, tactile feedback data, torque data and posture data; the process of collecting multimodal data is specifically as follows: using visual sensors combined with deep convolutional networks to collect image data of the surrounding environment and positioning data of objects to be grasped, using tactile sensors to collect tactile feedback data of the manipulator, using torque sensors to collect torque data of each joint of the manipulator, and using torque sensors combined with inertial measurement units to collect posture information of the manipulator.

[0018] As a preferred technical solution, the preprocessing in S1 includes data denoising, data normalization, data completion and data enhancement technology.

[0019] As a preferred technical solution, the data encoding in S2 uses the CLIP model for encoding, and its encoding formula is:

[0020] V enc =Encoder(D pro )

[0021] Where V enc is the encoded feature vector, D pro The pre-processed data.

[0022] As a preferred technical solution, the cross-modal token alignment technology in S3 uses deep learning architectures such as Transformer for joint training to fuse multimodal data. Its alignment formula is:

[0023] V ali =Align(V text ,V image ,V force ,V pose )

[0024] Where V enc is the encoded feature vector, D pro is the preprocessed data, V text is the text feature vector, V image is the image feature vector, V force is the force sensing data feature vector, V pose It is the robot arm joint pose data.

[0025] As a preferred technical solution, when using feature vectors and feature representations to train the neural network model in S4, data parallelism and model parallelism strategies are adopted, that is, the training data is divided into multiple batches, and each batch is calculated on a different GPU to obtain each update parameter, and each update parameter is synchronized through communication.

[0026] As a preferred technical solution, the pruning operation in S5 is to remove low-importance parameters; the distillation operation is to transfer the knowledge of the large model to the small model; and the quantization operation is to convert the model from a high-precision floating-point representation to a low-precision representation.

[0027] As a preferred technical solution, the edge computing layer in S6 uses a processor with hardware acceleration function.

[0028] As a preferred technical solution, the process of real-time control of the robot's operating actions in S7 is specifically as follows: the robot's motion planning controls the real-time actions of the robotic arm and manipulator, and the electrical signals fed back by the tactile sensor adjust the grasping force in real time; the electrical signals fed back by the torque sensor and inertial measurement unit are combined with the motion control algorithm to optimize the grasping path and operating actions in real time.

[0029] According to another aspect of the present invention, a real-time control system for robot operation actions in which a large model and a small model are coordinated is provided. The system works by applying a real-time control method for robot operation actions in which a large model and a small model are coordinated as described above, and includes a data acquisition and preprocessing module, a data encoding and alignment module, a model training and optimization module, and a model deployment and control module.

[0030] Among them, the data acquisition and preprocessing module is used to collect environmental data using various sensors, form multimodal data with instruction text data, preprocess the multimodal data, and obtain preprocessed multimodal data;

[0031] The data encoding and alignment module is used to encode the preprocessed multimodal data to obtain feature vectors, and align the preprocessed multimodal data using cross-modal token alignment technology to obtain feature representation;

[0032] The model training and optimization module is used to train the neural network model using feature vectors and feature representations to obtain a large trained model, and to perform pruning, distillation, and quantization operations on the trained large model to generate a small model.

[0033] The model deployment and control module is used to deploy the small model to the edge computing device. The edge computing device obtains the current instruction text data in real time and infers it, generates the robot's motion plan, and uses the robot's motion plan and various sensor feedback signals to control the robot's operating actions in real time.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. In the present invention, firstly, multimodal data is collected and preprocessed; then, data encoding is performed on the preprocessed multimodal data to obtain feature vectors; at the same time, cross-modal token alignment technology is used to align the preprocessed multimodal data to obtain feature representation; then, the neural network model is trained using the feature vector and feature representation to obtain a trained large model; the trained large model is pruned, distilled and quantized to generate a small model; then, the small model is deployed to the edge computing device, and the edge computing device obtains the current instruction text data in real time and infers it to generate the robot's motion plan; finally, the robot's motion plan and the feedback signals from each sensor are used to control the robot's operating actions in real time. Through multimodal data fusion and miniaturized models, low-latency, high-efficiency real-time motion planning and execution are achieved on the edge computing device.

[0036] 2. The present invention utilizes visual sensors combined with deep convolutional networks to collect image data of the surrounding environment and positioning data of the object to be grasped. Through visual sensors and deep learning technology, the features of the object can be accurately identified, and its spatial position and motion trajectory can be obtained, effectively improving the grasping accuracy.

[0037] 3. In the present invention, the robot's motion planning controls the real-time motion of the manipulator arm and manipulator, and the electrical signal fed back by the tactile sensor adjusts the grasping force in real time; by using the tactile sensor to monitor and adjust the grasping force in real time, it is ensured that the object is grasped stably to avoid damage or slipping.

[0038] 4. The motion planning of the robot in the present invention controls the real-time motion of the robotic arm and manipulator, and optimizes the grasping path and operating action in real time in combination with the motion control algorithm. The motion path of the robotic arm is optimized by combining the torque sensor and IMU to improve the stability and accuracy of the grasping action.

[0039] 5. In the present invention, the collected multimodal data are preprocessed, and the preprocessing process includes data denoising, data normalization, data completion and data enhancement technology to improve data quality and ensure data stability and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the method steps of the present invention;

[0041] Figure 2 It is the overall flow chart of the present invention;

[0042] Figure 3 It is a schematic diagram of the cooperation principle between the large model and the small model of the present invention;

[0043] Figure 4 It is a schematic diagram of the model implementation process of the present invention;

[0044] Figure 5 It is a schematic diagram of the model quantification implementation process of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] With the rapid development of artificial intelligence technology, robotics has been widely used in manufacturing, service, medical, logistics and other fields. Especially for robot operation and grasping tasks in complex environments, as the diversity and complexity of tasks increase, how robots can complete tasks quickly and accurately has become an important research direction. At present, the success of robot operation and grasping tasks often depends on its perception ability and control strategy. Traditional robot grasping methods mainly rely on single sensor data, such as visual sensors or tactile sensors, which are difficult to effectively handle complex environmental changes and multiple interference factors, so there are certain limitations in practical applications.

[0047] In order to improve the robot's grasping accuracy and efficiency in complex environments, researchers have proposed a method to integrate multiple sensory information. Visual sensors can provide information such as the position, shape, and color of an object, but they are highly sensitive to changes in ambient lighting and surface texture of the object. Tactile sensors can sense force feedback during the grasping process in real time, but in complex operations, relying solely on force perception cannot effectively respond to changes in object shape, stiffness and other characteristics. Therefore, how to deeply integrate visual and tactile perception data to optimize the robot's operation path and grasping strategy has become the key to solving this problem.

[0048] In addition, with the advancement of deep learning technology, large language models have achieved remarkable results in image recognition, speech processing, natural language processing and other fields. Large models can acquire powerful understanding capabilities for complex tasks by learning from massive amounts of data. However, in practical applications, large models have disadvantages such as high computing resource consumption and slow reasoning speed, making it difficult to meet real-time and edge computing requirements. Therefore, how to compress large models into small models for specific tasks, retain key features and improve reasoning efficiency has become a major challenge in current technology.

[0049] At present, robot operating systems based on edge computing are gradually being used. By deploying models to edge devices, fast real-time reasoning and response can be achieved, but they still face challenges in data processing, computing resources, real-time performance, etc. Therefore, how to improve the accuracy of robot operation while ensuring computing efficiency, especially in the application of grasping tasks, is still a technical problem that needs to be solved urgently.

[0050] The present invention proposes a real-time control method for robot operation actions based on the collaboration of a large model and a dedicated small model. Through the deep fusion of vision and touch, combined with the collaborative work of the large model and the small model, it breaks through the limitations of traditional robot operation methods and provides an innovative solution for robot operation tasks in complex environments.

[0051] Example 1

[0052] In this embodiment, a real-time control method for robot operation based on the collaboration of a large model and a dedicated small model is applied. The method performs visual sensing, tactile feedback, torque monitoring, posture perception, multi-modal large model, and fast reasoning based on edge computing. The method steps are shown as follows: Figure 1 As shown, specifically including:

[0053] S1. Collect environmental data using various sensors, combine them with instruction text data to form multimodal data, and preprocess the multimodal data to obtain preprocessed multimodal data;

[0054] S2, encoding the preprocessed multimodal data to obtain a feature vector;

[0055] S3, using cross-modal token alignment technology to align the pre-processed multimodal data to obtain feature representation;

[0056] S4, using the feature vector and feature representation to train the neural network model to obtain a trained large model;

[0057] S5. Prune, distill and quantize the trained large model to generate a small model.

[0058] S6. Deploy the small model to the edge computing device. The edge computing device obtains the current command text data in real time and infers it to generate the robot's action plan.

[0059] S7. Use the robot's motion planning and the feedback signals from each sensor to control the robot's operating actions in real time.

[0060] In this embodiment, if Figure 2 As shown, test the real-time motion planning of the robot arm and the real-time grasping ability of the robot hand. The specific steps are as follows:

[0061] In this embodiment, a visual sensor (such as a depth camera, an RGB camera, or a laser radar, etc.) is used to collect image information of the surrounding environment. The image is processed through the semantic segmentation technology of deep learning to accurately identify and locate the pixel-level features of the object to be captured.

[0062] Image processing formula:

[0063] I seg =Segmentation·I raw

[0064] Among them, I seg is the processed image, I raw is the original image, and Segmentation represents the semantic segmentation operation.

[0065] In this embodiment, the geometric boundary B, surface feature S, and spatial position P (x, y, z) of the object are calculated using features such as texture, color, and shape in the image:

[0066] P object =f(I seg )

[0067] Where f represents the feature extraction model based on deep learning.

[0068] In this embodiment, a depth camera, RGB camera or laser radar sensor is used to obtain the three-dimensional position and volume of the object to be grasped. The position formula is as follows:

[0069] P3d =(x,y,z)

[0070] In this embodiment, the tactile sensor monitors the gripping force and displacement applied by the manipulator in real time. The gripping force is adjusted by current detection technology, and the gripping force is adjusted in real time according to the feedback signal from the tactile sensor to ensure stability. The torque sensor and inertial measurement unit (IMU) are used to monitor the angular torque and posture changes of each joint of the manipulator in real time.

[0071] τ=Torque(θ,Δθ)=∫T(t)dt

[0072] Among them, τ is the joint torque, θ and Δθ are the joint angles and their changes, and T(t) is the output of the torque sensor.

[0073] R=R IMU (θ)

[0074] Among them, R is the pose matrix of the robot arm, R IMU Indicates the real-time pose data provided by the IMU.

[0075] In this embodiment, multimodal data from vision, touch, torque sensor and posture information are preprocessed, and the steps include denoising, normalization, feature selection and data enhancement, etc.:

[0076] D nor =(D raw -μ) / σ

[0077] Among them, D raw is the original data, μ is the mean, σ is the standard deviation, D oor is the normalized data.

[0078] In this embodiment, the multimodal data is encoded, and the cross-modal token alignment technology is used to fuse the instruction text, image, torque, posture and other data to generate an efficient feature vector. The CLIP model is selected for encoding, and the encoding formula is:

[0079] V enc =Encoder(D pro )

[0080] Among them, V enc is the encoded feature vector, D pro For preprocessed data, the alignment formula is:

[0081] V ali =Align(V text ,V image ,V force ,V pose )

[0082] Among them, V enc is the encoded feature vector, D pro is the preprocessed data, V text is the text feature vector, V image is the image feature vector, V force is the force sensing data feature vector, V pose It is the robot arm joint pose data.

[0083] In this embodiment, if Figure 3 As shown in Figure 2, in the pre-training phase, LLaMA uses a strategy similar to autoregressive language modeling to learn contextual information. Its core goal is to predict the next word based on previous words. Given an input text sequence x1, x2…x T , the model needs to predict for each word x t The conditional probability of:

[0084] P(x t |x1,x2,…,x t-1 )=Softmax(W·h t +b)

[0085] Where: h t is the hidden state of the Transformer model at time step t. W is the weight matrix used to output word probabilities. b is the bias term. Softmax is used to transform the model output into a probability distribution.

[0086] The goal of the model is to maximize the likelihood function and minimize the loss function (Cross-Entropy Loss):

[0087]

[0088] By minimizing this loss function, the model gradually learns how to predict the next word given the context.

[0089] In this embodiment, one of the key components of the Transformer architecture used by LLaMA is the self-attention mechanism. For each position in the input sequence, the model calculates the correlation between that position and other positions. The calculation process can be described by the following formula:

[0090]

[0091] Among them, Q, K, and V are query, key, and value matrices respectively. kis the dimension of the key vector. The self-attention mechanism calculates the similarity between the query at each position and the key at other positions and adds the weighted value matrix V to obtain the weighted sum at each position. This mechanism allows the model to consider all other words in the context when generating each word, thereby obtaining richer semantic information.

[0092] In this example, the pre-trained model can be fine-tuned to adapt to specific tasks, such as text classification, sentiment analysis, etc. During the fine-tuning process, the model continues to optimize its parameters by adjusting the task-specific loss function:

[0093]

[0094] Among them, y i is the label, x i is the input data (text). P(y i |x i ) is the conditional probability of the predicted label given the input text.

[0095] In this embodiment, when training large-scale models, data parallelism and model parallelism strategies are usually adopted. In data parallelism, the training data is divided into multiple batches, each batch is calculated on a different GPU, and the updated parameters are synchronized through communication:

[0096]

[0097] in, is the model parameter on the i-th GPU, γ is the learning rate, is the gradient computed on the i-th GPU.

[0098] In this embodiment, if Figure 4 As shown in the figure, in order to reduce the storage and computational complexity of large models, the model usually needs to be quantized. The quantization operation converts high-precision floating-point representation into low-precision representation (such as INT8) to reduce computational overhead and storage requirements:

[0099] W quantized =Quantize(W)

[0100] Among them, Quantize represents a quantization operation.

[0101] In this embodiment, the compressed small model is deployed to edge computing devices (such as FPGA, GPU, etc.), and low-latency and high-efficiency data reasoning is achieved through hardware acceleration. The process of real-time reasoning can be described by the following formula:

[0102]

[0103] in, M is the result of the operation reasoning, which represents the model’s prediction of the input X (such as instruction text, sensor data, etc.). smail It is the optimized model after compression.

[0104] The inference process typically involves extracting features from the input data and computing predictions via a forward pass:

[0105] h t = Transformer(X t ,θ)

[0106] In this embodiment, the inference result will eventually be associated with the corresponding subroutine, thereby controlling the robot arm and the robot hand to generate corresponding operation actions.

[0107] In summary, this method can generate the planning actions of the robotic arm and the grasping actions of the manipulator in real time, realizing efficient and precise robot operation. This method improves the robot's grasping efficiency and operation accuracy through the deep integration of vision and touch, and is suitable for intelligent robots to grasp, operate and execute tasks in complex environments.

[0108] Example 2

[0109] In this embodiment, a real-time control system for robot operation actions in which a large model and a small model are coordinated is applied, and the system includes a data acquisition and preprocessing module, a data encoding and alignment module, a model training and optimization module, and a model deployment and control module;

[0110] Among them, the data acquisition and preprocessing module is used to collect environmental data using various sensors, form multimodal data with instruction text data, preprocess the multimodal data, and obtain preprocessed multimodal data; the data encoding and alignment module is used to encode the preprocessed multimodal data to obtain feature vectors, and align the preprocessed multimodal data using cross-modal token alignment technology to obtain feature representation; the model training and optimization module is used to train the neural network model using feature vectors and feature representations to obtain a trained large model, and prune, distill and quantize the trained large model to generate a small model; the model deployment and control module is used to deploy the small model to the edge computing device, the edge computing device obtains the current instruction text data in real time and infers it, generates the robot's motion plan, and uses the robot's motion plan and the feedback signals from various sensors to control the robot's operating actions in real time.

[0111] In this embodiment, first, the robot collects environmental information through visual sensors and processes images through semantic segmentation technology to obtain the precise features of the target object; then, the three-dimensional position information and relative motion trajectory of the object are obtained in combination with the depth camera data; the tactile sensor provides real-time feedback on the force and displacement information during the grasping process, and ensures the stability of the object grasping process by adjusting the torque and grasping force. At the same time, the torque sensor and IMU sense the motion state of the robotic arm and optimize the grasping path. In this process, all multimodal data are preprocessed, encoded, and cross-modally aligned, and then input into a large-scale pre-trained neural network. The optimized robot operation strategy is generated through cloud training, and converted into a dedicated small model for edge computing to achieve real-time action planning and execution.

[0112] In this embodiment, the system is used to collect and process visual information: a visual sensor (such as a depth camera, RGB camera, or laser radar) is used to collect image information of the surrounding environment, and the image is processed using the semantic segmentation technology of deep learning to accurately identify and locate the pixel-level features of the object to be captured. The category and physical properties of the object are determined through information such as texture, color, and shape in the image, and the geometric boundaries, surface features, and spatial positions of the object are generated.

[0113] In this embodiment, the system is used for three-dimensional positioning and motion trajectory acquisition: combining sensors such as depth cameras, RGB cameras or laser radars to obtain the three-dimensional position, volume and relative motion trajectory of the object to be grasped. This information provides accurate object positioning in a high-dynamic environment and provides support for subsequent grasping tasks.

[0114] In this embodiment, the system is used for tactile perception and grasping force adjustment. The tactile sensor monitors the force and displacement applied by the manipulator during the grasping process in real time. The grasping force is accurately adjusted through current detection technology, and the grasping force is adjusted in real time according to the feedback signal of the tactile sensor to ensure the stability of the grasped object and avoid damage or slipping of the object. The system is also used for torque and posture monitoring. The torque sensor is used to monitor the angular torque changes of each joint of the manipulator in real time, and the inertial measurement unit (IMU) is combined to perceive the posture information of the manipulator in real time. Through these data, the posture changes of the manipulator during the movement are calculated, and the posture of the manipulator is optimized through the motion control algorithm to accurately adjust the operation path and action.

[0115] In this embodiment, a visual sensor is used to collect image information of the surrounding environment, and the image is processed through a deep learning semantic segmentation technology to accurately identify and locate the pixel-level features of the object to be grasped, and generate the geometric boundary, surface features and spatial position of the object;

[0116] Furthermore, the category and physical properties of the object can be determined through the texture, color, shape and other information in the image;

[0117] In this embodiment, on the basis of the visual sensor, a depth camera, an RGB camera or a lidar and other sensors are combined to obtain the three-dimensional position, volume and relative motion trajectory of the object to be grasped, thereby providing accurate object positioning in a high-dynamic environment;

[0118] In this embodiment, the tactile sensor monitors the force and displacement applied by the manipulator during the grasping process in real time, accurately adjusts the grasping force through current detection technology, and adjusts the grasping force in real time according to the signal fed back by the force sensor to ensure the stability of the grasped object and avoid damage or slipping of the object;

[0119] In this embodiment, a torque sensor is used to monitor the angular torque changes of each joint of the robot arm in real time, and an inertial measurement unit (IMU) is used to sense the posture information of the robot arm in real time, calculate the posture changes of the robot arm during movement, and realize high-precision posture control of the robot arm through a motion control algorithm to optimize the grasping path and operation action;

[0120] In this embodiment, the visual sensor includes at least one depth camera, RGB camera, lidar or stereo vision camera, which is used to simultaneously obtain a two-dimensional image, depth information and position information of the target object in three-dimensional space.

[0121] In this embodiment, the semantic segmentation technology uses a deep convolutional neural network (CNN) to perform pixel-level target segmentation and classification on the input image through a multi-layer convolutional network, generate edge contours, geometric information and feature points of the object, and use a deep learning model to perform semantic understanding of the scene, further enhancing the image feature recognition capability.

[0122] In this embodiment, the tactile sensor is a magnetic sensor and a visual tactile sensor integrated in the robot finger, which is used to sense the real-time force changes applied during the grasping process, and adjust the grasping force in real time through the current detection feedback signal to ensure that the robot can flexibly adapt to the shape, stiffness and surface characteristics of the object.

[0123] In this embodiment, the torque sensor monitors the angular torque changes of each joint of the robotic arm in real time, obtains the real-time position and posture of the robotic arm through the inertial measurement unit (IMU), and optimizes the movement of the robotic arm in combination with the dynamic model, thereby ensuring that the operation of the robot is accurate and reliable.

[0124] In this embodiment, the system is used for multimodal data preprocessing and fusion: multimodal data from vision, touch, torque sensors and posture information are preprocessed, including denoising, data normalization, feature selection and data enhancement to improve data quality and stability. These preprocessing operations are used to remove environmental noise, reduce data fluctuations, improve data processing accuracy and stability, ensure consistency of data from different sources, and facilitate subsequent fusion and analysis.

[0125] In this embodiment, the system is applied to perform data encoding and cross-modal alignment: the processed multi-modal data is encoded, and the multi-source data such as instruction text, image, torque, and posture are fused to generate an efficient feature vector. The cross-modal token alignment technology is used to effectively connect the text instructions and sensor data to generate a unified feature representation for unified task modeling and reasoning. Ensure effective interaction and collaborative processing between different modal data.

[0126] In this embodiment, the system is applied to train a large model and generate a small model: the fused data is input into a large-scale multimodal pre-trained neural network model in the cloud, and a multimodal task execution model that meets the task requirements is generated through steps such as joint learning, pre-training, and fine-tuning. The model can generate optimized robot operation strategies based on the scene environment, grasping tasks, and object characteristics. When training large-scale models, data parallelism and model parallelism strategies are usually adopted. In data parallelism, the training data is divided into multiple batches, each batch is calculated on a different GPU, and the updated parameters are synchronized through communication. After completing the training of the large model, the storage space and computational complexity of the model are reduced through pruning, distillation, and quantization operations to generate a small dedicated model to ensure efficient task execution and reduce computational overhead. Distillation and quantization operations involve transferring the knowledge of the large model to the small model, such as Figure 5 As shown in the figure, quantization technology is used to reduce computational overhead and ensure the performance and accuracy of the compressed model.

[0127] In this embodiment, the system is applied for edge computing and real-time reasoning: the small model is deployed to the edge computing device, and low-latency and high-efficiency data reasoning is achieved through hardware acceleration. Combined with the instruction text data obtained in real time, reasoning calculations are performed to generate real-time motion planning of the robot arm and real-time grasping actions of the manipulator. The edge computing layer uses a processor with hardware acceleration function (such as FPGA, GPU, etc.) to achieve low-latency and high-efficiency real-time reasoning calculations, further accelerate the robot arm motion generation process, and work in collaboration with the cloud system through an efficient communication protocol.

[0128] In summary, this system overcomes the shortcomings of existing robot manipulators in the grasping process, such as the lack of high-precision tactile perception, adaptive grasping ability and real-time motion planning ability. This system can efficiently process multimodal data from multiple sensors and generate the optimal operation strategy based on the grasping task and object characteristics, thereby achieving precise grasping and operation in complex environments.

[0129] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A real-time control method for robot operation actions in which a large model and a small model are coordinated, characterized in that: The method comprises the following steps: S1. Collect environmental data using various sensors, form multimodal data with instruction text data, and preprocess the multimodal data to obtain preprocessed multimodal data; S2, encoding the preprocessed multimodal data to obtain a feature vector; S3, using cross-modal token alignment technology to align the pre-processed multimodal data to obtain feature representation; S4, using the feature vector and feature representation to train the neural network model to obtain a trained large model; S5. Prune, distill and quantize the trained large model to generate a small model. S6. Deploy the small model to the edge computing device. The edge computing device obtains the current command text data in real time and infers it to generate the robot's action plan. S7. Use the robot's motion planning and the feedback signals from each sensor to control the robot's operating actions in real time.

2. According to claim 1, a real-time control method for robot operation actions in which a large model and a small model cooperate is characterized in that: The environmental data of S1 includes image data, positioning data, tactile feedback data, torque data and posture data; the process of collecting multimodal data is specifically as follows: using a visual sensor in combination with a deep convolutional network to collect image data of the surrounding environment and positioning data of the object to be grasped, using a tactile sensor to collect tactile feedback data of the manipulator, using a torque sensor to collect torque data of each joint of the manipulator, and using a torque sensor in combination with an inertial measurement unit to collect posture information of the manipulator.

3. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: The preprocessing in S1 includes data denoising, data normalization, data completion and data enhancement techniques.

4. The method for real-time control of robot operation actions in which a large model and a small model cooperate according to claim 1 is characterized in that: The data encoding in S2 is encoded using the CLIP model, and its encoding formula is: V enc =Encoder(D pro ) Where V enc is the encoded feature vector, D pro The pre-processed data.

5. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: The cross-modal token alignment technology in S3 uses deep learning architectures such as Transformer for joint training to fuse multimodal data. Its alignment formula is: V ali =Align(V text ,V image ,V force ,V pose ) Where V enc is the encoded feature vector, D pro is the preprocessed data, V text is the text feature vector, V image is the image feature vector, V force is the force sensing data feature vector, V pose It is the robot arm joint pose data.

6. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: When the neural network model is trained using feature vectors and feature representations in S4, data parallelism and model parallelism strategies are adopted, that is, the training data is divided into multiple batches, each batch is calculated on a different GPU to obtain each update parameter, and each update parameter is synchronized through communication.

7. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: The pruning operation in S5 is to remove low-importance parameters; the distillation operation is to transfer the knowledge of the large model to the small model; and the quantization operation is to convert the model from a high-precision floating-point representation to a low-precision representation.

8. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: The edge computing layer in S6 uses a processor with hardware acceleration function.

9. The method for real-time control of robot operation actions in which a large model and a small model are coordinated according to claim 1, characterized in that: The process of real-time control of the robot's operating actions in S7 is specifically as follows: the real-time actions of the robot arm and manipulator are controlled by the robot's motion planning, and the grasping force is adjusted in real time by the electrical signal fed back by the tactile sensor; the electrical signal fed back by the torque sensor and the inertial measurement unit is combined with the motion control algorithm to optimize the grasping path and operating actions in real time.

10. A real-time control system for robot operation actions in which a large model and a small model cooperate, characterized in that: The system applies a real-time control method for robot operation actions in which a large model and a small model cooperate as described in any one of claims 1 to 9, and the system includes a data acquisition and preprocessing module, a data encoding and alignment module, a model training and optimization module, and a model deployment and control module; The data acquisition and preprocessing module is used to collect environmental data using various sensors, form multimodal data with instruction text data, and preprocess the multimodal data to obtain preprocessed multimodal data; The data encoding and alignment module is used to encode the preprocessed multimodal data to obtain a feature vector, and align the preprocessed multimodal data using a cross-modal token alignment technology to obtain a feature representation; The model training and optimization module is used to train the neural network model using feature vectors and feature representations to obtain a trained large model, and to perform pruning, distillation and quantization operations on the trained large model to generate a small model; The model deployment and control module is used to deploy the small model to the edge computing device. The edge computing device obtains the current instruction text data in real time and infers it, generates the robot's motion plan, and uses the robot's motion plan and various sensor feedback signals to control the robot's operating actions in real time.

Images