Mechanical arm control method, system and equipment for realizing multi-mode general operation task

Abstract

The invention discloses a mechanical arm control method, system and equipment for realizing a multi-modal universal operation task. The method comprises the following steps: acquiring multi-modal data of text, voice and vision; performing modal alignment on the multi-modal input data, encoding the multi-modal input data into a uniform input representation vector, and splicing the uniform input representation vector into a multi-modal instruction paragraph; retrieving a user information voice database based on voiceprint features of the voice instruction data to obtain task associated voice data; instruction information is input into the strategy large model to obtain an action plan; a post-processing task action planning input controller controls a mechanical arm, and a multi-mode input instruction is updated in real time to perceive the latest task state; and performing closed-loop control based on the initial target instruction information and the environment perception information until the task is completed. The task planning and error correction capacity of the mechanical arm is improved through closed-loop control and voice data retrieval based on the large strategy model, and the dynamic adaptive capacity, the autonomous error correction capacity and the multi-mode interaction capacity of the mechanical arm are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot arm control technology, and in particular to a robot arm control method, system and device for realizing multi-modal general operation tasks. Background Art

[0002] Traditional robotic arm control systems are mostly used in industrial production lines, such as automobile assembly, food processing, and electronic component assembly. The tasks they perform are characterized by single purpose, clear process, and stable environment. With the advancement of sensor technology and artificial intelligence algorithms, robotic arm control has gradually acquired the ability to perceive and understand the environment and human dynamic instructions, thereby demonstrating certain processing capabilities in complex tasks and unstructured environments. Its application scope has also expanded to more diverse fields such as medical assistants, catering services, and home assistants.

[0003] Most of the existing robot arm control methods are based on single-modal input, such as speech recognition, image recognition or text recognition, among which speech recognition is usually parsed by converting speech commands into text, and finally constitutes the robot's control commands together with additional text commands. However, in practical applications, single-modal input has many limitations and cannot adapt to complex and changing control requirements. For example, it is difficult for text recognition systems to fully describe task information, image recognition systems are sensitive to changes in lighting and viewing angles, and speech recognition systems are susceptible to interference in noisy environments. In addition, in complex environments, traditional robot arm control methods often find it difficult to achieve efficient interactive operations. For example, as a home assistant robot with highly customized characteristics, a robot arm based on a single modality or only on text or visual modalities is difficult to naturally accept user feedback guidance during the execution of user tasks, and it is also impossible to obtain personalized information about the user based on the user's voice characteristics to achieve better task planning, such as providing more humane services based on the user's emotions and habits. In existing multimodal robotic arm control schemes, a multi-level multimodal signal processing scheme is usually adopted, that is, modal information such as speech will be converted into unified text modal information for subsequent processing. However, affected by environmental interference and model capabilities, this hierarchical conversion scheme is likely to introduce cumulative errors, and it is difficult to grasp the richer emotional intentions, voiceprint characteristics and other information in the original speech signal, thereby limiting the upper limit of the model's capabilities.

[0004] In summary, the existing robot arm control methods still have the following limitations: (1) Most current robotic arm control strategies are based on single-modal input (such as speech recognition, image recognition, or text recognition). In a non-standardized, dynamically changing operating environment, robots usually find it difficult to fully understand environmental information and complex human needs, and are easily misled by specific modal information, which affects the success rate of task execution and operational accuracy. (2) Traditional robotic arm interaction methods are limited to simple text, picture command control or preset program selection. They lack the ability to understand complex commands with rich modalities, rich semantics, and contextual associations, and lack the ability to accept rich modal information in the environment in real time to interact and adjust during task execution. (3) Traditional multimodal robotic arm control solutions lack the end-to-end processing capabilities for modal signals such as speech. They rely on the hierarchical processing, understanding, and planning of multimodal signal conversion, which may lead to the accumulation of errors layer by layer and the loss of important information. (4) Traditional robotic arm control methods are still not flexible, efficient, and accurate enough in application scenarios such as home assistants and catering services that require dynamic adaptability, autonomous error correction, and multimodal interaction capabilities. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and device for controlling a robot arm that realizes multi-modal general operation tasks in view of the shortcomings of the prior art. By combining multiple input modal information such as voice, vision and text, a more flexible and efficient robot arm control method is provided, thereby solving the problem that a single modal input in the prior art is insufficient to meet complex operation requirements and lacks the ability to adapt to dynamic environments.

[0006] The objective of the present invention is achieved through the following technical solutions. In a first aspect, the present invention provides a method for controlling a robot arm for realizing a multi-modal general operation task, the method comprising the following steps: Step 1: Obtain the multimodal input data required for the task, including voice command data, text command data, visual perception data in the current task environment, and robotic arm posture perception data; Step 2: Perform modal alignment on the input data of multiple modalities, encode them into input representation vectors with a unified form, and concatenate them into multimodal input instructions in the form of paragraphs; Step 3: Based on the voiceprint features of the voice command data, retrieve and extract the task-related voice data of the corresponding user from the collected user information voice database as supplementary information of the command, and splice them in the same format; Step 4: Input the multimodal input command into the strategy large model backbone for command understanding, and generate a set of robot end effector poses for several future time steps, wherein the end effector poses include the action commands of the robot at each time step; Step 5: Control the motion of the robot arm based on the position of the end effector of the robot arm, and then obtain new multimodal input data to update the multimodal input instructions to perceive the latest task status; Step 6: The strategy model performs closed-loop control by comparing and understanding the initial target command information and the current environmental perception information, and continuously makes new action plans until the robotic arm completes the initially set mission objectives.

[0007] Furthermore, the implementation method of step one includes: Step 101: The operator inputs text instruction data, and uses system prompt words, model dialogue templates, and supplementary prompt words to package them into a complete text string instruction; Step 102: The operator's voice command is converted into a Mel-spectrogram of several frequency bands using short-time Fourier transform (STFT), and then padded to a total frame number with a fixed length. If the original frame number is less than the length, zeros are padded at the end of the time axis, otherwise it is truncated to the length. The number of frequency bands and the total number of frames are determined by the parameters used in pre-training of the speech feature encoder used.

[0008] Step 103: Obtain visual perception data and robot arm posture perception data through the sensor component. The visual perception data includes a side-view RGB image taken by a side-view monocular camera fixed on the base of the robot arm and an RGB image taken by a monocular camera fixed on the gripper of the robot arm. The images are resized, multi-image stitched, border pixel filled, and data format converted to form an image format that can be processed by the model. The robot arm posture perception data is obtained through the robot arm body sensor, including the coordinates of the robot arm, the angles of each joint, and the opening and closing information of the gripper. After reading, it is spliced ​​with the text instruction data in the form of a character string.

[0009] Furthermore, the implementation method of step 2 includes: Step 201: reserve a special mark symbol representing an image or voice in the text instruction data to indicate the insertion position of multiple modal information in the multimodal instruction; Step 202: For different modal information, use independent feature encoders to convert the information into a multimodal vector space with consistent dimensions, and use the input dimensions required by the strategy large model backbone to perform dimensional alignment; after the alignment is completed, splice and insert different modal information according to the reserved special marking symbols to form a multimodal instruction segment.

[0010] Furthermore, the implementation method of step three includes: Step 301: Use a pre-trained or open-source speech recognition model to extract the user voiceprint information carried by the collected voice commands, and use it as reference information for retrieving the user information voice database; the collection process of the user information voice database includes the following steps: pre-collecting user personal information, pre-coding voice information, storing voice data, and updating voice information.

[0011] Step 302: Use voiceprint matching to locate the personal voice information of the user who currently issues the instruction, extract the voice information and then splice it into the special mark marked as user personal information in the multimodal instruction paragraph to indicate the supplementary user information.

[0012] Furthermore, the strategy big model in step 4 adopts the vision-language-action big model obtained by fine-tuning the vision-language big model through full supervision, and the implementation method includes: a data set construction part, a model training part and a deployment reasoning part; The data set construction part includes two parts: speech dialogue data set construction and robot arm operation data set construction. The speech dialogue data set is obtained by performing speech synthesis on the community open source visual text dialogue data set and performing speech recognition on the visual speech dialogue data set, which is used to increase the model's support for end-to-end understanding of speech modal information. The robot arm operation data set is realized by collecting the operation process data of professional robot arm operators using the robot arm to perform different tasks. The original data includes task instructions, environmental perception and robot arm posture perception data for each task and each moment. The robot arm posture perception data needs to be further constructed with an action word list before model training can be performed, that is, the floating point posture perception data is normalized and discretized according to the vocabulary usage frequency in the tokenizer word list from low to high, and mapped one by one to the vocabulary sequence number in the tokenizer word list. The model training part includes two parts: speech modality training and robotic arm operation training. The speech dialogue data set and the robotic arm operation data set are used for training respectively, wherein the speech modality training is used as the first stage of training, and the robotic arm operation training is used as the second stage of training. The two-stage training process uses the cross entropy loss function to calculate the loss value, and the real label uses the one-hot distribution method to calculate the cross entropy with the probability distribution predicted by the model; The deployment reasoning part reads the RGB images taken by the side-view monocular camera and the monocular camera at the gripper of the robot arm, and the robot arm posture perception data in real time, and processes them together with the text instruction data of the current task, voice instruction data, and supplementary information extracted from the user information voice database into a multimodal instruction paragraph, and inputs the trained strategy large model backbone to obtain a set of robot arm end effector postures for several future time steps, wherein the strategy large model runs on the server computer, and the client computer collects instruction data, packages and sends instruction data, and obtains server response data; the server and the client communicate through a wireless network, and the client and the robot arm transmit information and control the robot arm through ROS topic nodes.

[0013] Furthermore, the implementation method of step five is to perform inverse kinematics solution on the robot arm end effector posture for several future time steps output by the strategy large model to obtain the rotation angle of each axis of the robot arm, and then control the motor through a PID controller; after completing several time step actions planned by the large model, re-read the multiple modal data in the current environment, use the newly input visual perception data, robot arm posture perception data, text command data and voice command data to update the multimodal instructions, and wait for the strategy large model to perform a new round of motion planning.

[0014] Furthermore, the implementation method of step six is ​​to construct a closed-loop control loop with the strategy large model as the controller based on steps one to five, and through the model, repeatedly compare and understand the initial multimodal command information and the current environmental perception information, continuously output new action plans in a dynamically changing environment until it is determined that the current task has been completed, stop the robot arm movement, and wait for the next task command to restart.

[0015] In a second aspect, the present invention provides a robot arm control system for realizing multi-modal general operation tasks, the system comprising a multi-modal data acquisition module, a task planning module and a motion control and feedback module; The multimodal data acquisition module is used to collect perception data and instruction data of various modes in the current working area, perform modal alignment and encode and splice them into multimodal instruction paragraphs, including text instruction data, voice instruction data describing tasks or supplementary information, and visual perception data and robot arm posture perception data in the current task environment. In addition, it also includes a user information voice database that is additionally collected and established before the task is executed; The task planning module consists of two parts: a server and a client. The model deployment and reasoning are completed on the server using a GPU processor, and a set of robot arm end effector postures for several future time steps are generated based on multimodal instruction data. The end effector posture contains the action instructions of the robot arm at each time step. A computer equipped with only a CPU processor and an SSD solid-state drive is used on the client to collect, forward and process instruction information, and the local robot arm is controlled based on the robot arm end effector posture in the form of a network service.

[0016] The motion control and feedback module is used to receive the planning information processed by the client in the task planning module, that is, the position of the end effector of the robotic arm at several time steps, obtain the rotation angle required to control each axis of the robotic arm through inverse kinematics solution, and use the PID controller to complete the motion control of the robotic arm; after completing the planned action, the current working area perception data and instruction data are re-read through the multimodal data acquisition module, and fed back to the task planning module for new action planning.

[0017] In a third aspect, the present invention provides a robot arm control device for realizing multi-modal general operation tasks, the device comprising: a visual sensor, an audio sensor, a text input interface, a client computer, a server computer and a robot arm kit; The visual sensor is a set of monocular cameras, which are composed of a side-view monocular camera fixed to the base of the robotic arm and a monocular camera fixed above the gripper of the robotic arm. The fixed camera at the base is used to provide a stable side-view panoramic view for detecting the spatial position of the environment and the object, and the camera at the gripper can change with the movement of the robotic arm to provide a close-up image of the object in real time. The audio sensor uses a microphone array, and after the audio signal is processed by noise reduction, key features are extracted, including user voice and ambient sound.

[0018] The text input interface uses a mobile phone or computer keyboard device to collect the user's text instruction data by sending it to the client computer through a wireless network or by typing the text instruction to be executed on the keyboard; The client computer is used to store and execute part of the program code, is equipped with a CPU processor and an SSD solid state drive, is directly connected to the robotic arm kit, the visual sensor, the audio sensor, and the text input interface to send control signals and collect sensor data, and communicates with the server computer via the network; The server computer is used to deploy the trained strategy model, use the GPU processor to complete the deployment reasoning of the model, generate a set of robot arm end effector positions for several future time steps, obtain the rotation angle required to control each axis of the robot arm through inverse kinematics solution, and use the PID controller to complete the motion control of the robot arm; after completing the planned action, perform new action planning; The robotic arm kit includes a robotic arm with an actuator and a controller, which is responsible for executing the motion control signal processed by the client to realize specific physical operations.

[0019] The beneficial effects of the present invention are: (1) The present invention implements a manipulator control strategy and an end-to-end control model training method that supports rich modal information, so that in a non-standardized and dynamically changing operating environment, the robot can effectively understand environmental information and human needs through multi-modal information fusion and correctly perform the current task; (2) The present invention achieves a deeper and more comprehensive understanding of multimodal data and construction of instructions. Through end-to-end speech signal understanding (rather than conversion into text) and the construction of multi-modal interleaved instruction paragraphs, it can capture more comprehensive and rich semantic information, avoid the cumulative errors and information loss caused by hierarchical processing, and thus improve the probability of successful task completion; (3) The present invention realizes a more efficient human-computer interaction and instruction construction method. Through closed-loop control planning with a large strategy model as the controller and information retrieval and extraction from the user information voice database, the task information is supplemented and updated, so that the robot arm has the ability to dynamically adjust the task objectives and make timely corrections from erroneous execution processes.

[0020] (4) The present invention implements a resource-efficient deployment and reasoning method, deploying the reasoning model as a cloud service, sending the reasoning requirements of one or more local devices through network requests and processing them in parallel, and making full use of the expensive server-side computing resources; (5) The system proposed in the present invention improves the dynamic adaptability, autonomous error correction capability and multimodal interaction capability of the robotic arm, effectively expands the applicable scenarios of the robotic arm, increases the upper limit of its ability to complete tasks, and enables the robotic arm system to be applied in more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of the robot arm control method of the present invention.

[0023] Figure 2 A flow chart is generated for the multimodal data processing and planning of the present invention.

[0024] Figure 3 This is a block diagram of the multimodal signal fusion and feedback control loop of the present invention.

[0025] Figure 4 It is a block diagram of the robot arm system of the present invention.

[0026] Figure 5 A schematic diagram of a robotic arm system device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0028] Embodiment 1: like Figure 1 As shown, the present invention provides a method for controlling a robotic arm to realize a multi-modal general operation task, the method comprising the following steps: Step 1: Acquire multimodal command data required for the task, including the user's voice command data and text command data input via keyboard or sent via wireless network, and acquire visual perception data and robotic arm posture perception data in the current task environment through the sensor component, wherein the visual perception data includes RGB images taken by cameras with different viewing angles, and the robotic arm posture perception data includes information such as the coordinates of the robotic arm, the angles of each joint, and the opening and closing of the gripper; Step 2: Perform modal alignment on the input data of multiple modalities, encode them into a unified input representation vector and concatenate them into a multimodal input instruction in the form of a paragraph; Step 3: Based on the voiceprint features of the voice command data, retrieve and extract the task-related voice data of the corresponding user from the collected user information voice database as supplementary information of the command, and splice them in the same format; Step 4: Input the command information into the strategy model backbone for command understanding, and generate a set of robot end effector poses for several future time steps. The end effector poses contain the action instructions of the robot at each time step, such as angle, coordinate, gripper opening and closing degree, etc. Step 5: The position and posture of the end effector of the robot arm at the above-mentioned several time steps are processed and sent to the controller to control the motion of the robot arm, and then the new sensor component data and user input data are read to update the multimodal input instructions to perceive the latest task status; Step 6: The strategy model performs closed-loop control by comparing and understanding the initial target command information and the current environmental perception information, and continuously makes new action plans until the robotic arm completes the initially set mission objectives.

[0029] like Figure 2 As shown, the implementation method of step one includes: Step 101: The operator inputs text instruction data through keyboard, network transmission, etc., and uses system prompt words (system message), model dialogue template (prompt template), and reasonable supplementary prompt words to package and form a complete text instruction string.

[0030] The system prompts, i.e., systematic guidance for the policy big model, are used to indicate the basic rules that the model should follow, the basic expectations of the user for the model, and the role that the model needs to play. They need to be consistent in both the training process and the inference service process. The model dialogue template is a formatted template for the interaction between the user and the model. Due to limited training resources, fine-tuning is usually used to train the model and transfer capabilities in actual task scenarios. Therefore, it is necessary to keep the dialogue template consistent with the model used in the pre-training process. For example, a dialogue template in ChatML format can be used to divide the system prompt words, user instruction data and model answers into three roles of "system", "user" and "assistant", and distinguish them with special characters to form a dialogue text in the form of "<|im_start|>system\n(system prompt words)<|im_end|>\n<|im_start|>user\n(user instruction)<|im_end|>\n<|im_start|>assistant\n(model answer)<|im_end|>", where the model answer part is generated by the model; The supplementary prompt words are used to help the user input additional supplementary requirements or information to the model and are spliced ​​into the above-mentioned user instruction data. They are usually used to provide additional supplementary information that the model needs to know in the current difficult task, such as informing whether there are additional action restrictions in the current environment, time limits for tasks, or explaining unclear user instructions.

[0031] Step 102: The operator's voice command is converted into a Mel spectrogram of several frequency bands using a short-time Fourier transform (STFT), and then padded to a total frame number of a fixed length. If the original frame number is less than the length, zero padding is performed at the end of the time axis, otherwise it is truncated to the length. The number of frequency bands and the total number of frames are determined by the parameters used in the pre-training of the speech feature encoder used. The calculation formula for the short-time Fourier transform and Mel spectrogram generation process is:

[0032]

[0033]

[0034]

[0035] in, is the complex spectrum calculated by STFT, is the original time domain speech signal, is the window function, is the frame index, is the frame shift, is the frequency index, is the number of FFT (Fast Fourier Transform) points. is the power spectrum calculated from the complex spectrum. is the linear frequency used in the STFT process, is the Mel scale converted from the Mel frequency scale, Indicates The weights of the Mel filters cover several frequency bands. The Mel spectrum is obtained by weighted summing of the power spectrum using a Mel filter bank.

[0036] Step 103: For the visual perception data and the robotic arm posture perception data obtained through the sensor assembly, the former includes the side-view RGB image taken by the side-view monocular camera fixed on the base of the robotic arm and the RGB image taken by the monocular camera fixed on the gripper of the robotic arm. The image is resized, the border pixels are filled, multiple images are spliced, and the data format is converted to form an image format that can be processed by the subsequent model. The latter is obtained through the robotic arm body sensor, including the coordinates of the robotic arm, the angles of each joint, and the opening and closing information of the gripper. After reading, it is spliced ​​with the text instruction data in the form of a string.

[0037] like Figure 2 As shown, the implementation method of step 2 includes: Step 201: reserve special marking symbols, such as those representing images or voices, in the formed text instructions to facilitate the insertion of various modal information during the construction of multimodal instructions.

[0038] The special marking symbol is used to identify the position where other modal information needs to be replaced and inserted in the subsequent instruction splicing stage. The commonly used special marking symbols include " <audio>" ” (indicates voice data), " ” (indicating image data), etc. For multiple consecutive pictures or multiple voice messages, spaces or line breaks can be used to separate them.

[0039] Step 202: For the pre-processed different modal information, use independent feature encoders to convert the information into their respective multimodal vector spaces, and then use linear layers or MLP (multi-layer perceptron) to perform dimension conversion and alignment. By default, the input dimensions required by the strategy large model backbone are used. After the alignment is completed, the different modal information is spliced ​​and inserted according to the reserved special marking symbols to form a multimodal instruction paragraph.

[0040] The feature encoders include: speech feature encoder, text feature encoder, and image feature encoder. All three encoders use the Transformer architecture. In order to reduce R&D costs and migrate the capabilities of existing powerful models as much as possible, open source, large-scale pre-trained models from academia or industry can be used, or a relatively low-priced model API can be used.

[0041] like Figure 2 As shown, the implementation method of step three includes: Step 301: Use a pre-trained or open-source speech recognition model to extract the user voiceprint information carried by the collected voice command, and use it as reference information for retrieving the user information voice database.

[0042] Step 302: Use voiceprint matching to locate the personal information voice data of the user who is currently issuing the instruction, extract the voice data and then splice it into the special mark marked as user personal information in the multimodal instruction paragraph to indicate the supplementary user information. If it is empty, delete the special mark directly.

[0043] The acquisition process of the user information voice database includes the following steps: (1) Pre-collection of user personal information: Through audio recording, all users in the current environment can store important personal information related to the execution of tasks, such as personal operation preferences, scene item ownership, potential usage motivations, security tendencies, etc., to supplement text and voice instructions that may be unstructured and have insufficient information. (2) Voice information precoding: The voice feature encoder is used to precode all voice information and process it into a voice feature vector that can be processed by the subsequent strategy large model backbone, which is convenient for the splicing of multimodal instructions. In addition, a voice recognition model is used to extract voiceprint information from different user voice information to improve the efficiency of the voiceprint matching process.

[0044] (3) Voice data storage: The user information voice database is stored in a manner where the user number is used as the key used when indexing, and the voiceprint information and user voice information are used as the value, and is stored in the form of a JSON (a lightweight data exchange format) file.

[0045] (4) Voice information update: According to the failure cases that occur when the robot arm performs tasks and the dynamic changes of information in the current environment, the user information voice database needs to dynamically adjust the information. For newly entered user information, the user information can be supplemented by matching the existing key-value pairs according to the user's voiceprint features. If the returned matching result does not exist, a new key-value pair is created. For information that needs to be deleted or adjusted, the specific user information is matched according to the voiceprint features, and the user can independently select the part that needs to be deleted or adjusted in the existing voice information storage file.

[0046] The strategy big model in the step 4 adopts the speech-vision-language-action big model obtained by fully supervised fine-tuning based on VLM (visual-language big model), and the implementation method includes: data set construction part, model training part and deployment reasoning part.

[0047] The data set construction part includes two parts: speech dialogue data set construction and robotic arm operation data set construction.

[0048] The speech dialogue dataset is constructed. Since the basic model based on the training is an open source, large-scale trained and fine-tuned visual-language model, in order to increase the support for end-to-end understanding of speech modality without speech recognition, it must be trained and adjusted in the form of voice questions and text replies. The visual speech dialogue dataset is obtained by performing speech synthesis through the community open source visual text dialogue dataset and performing speech recognition through the visual speech dialogue dataset. , which is divided into a training set and validation set .

[0049] The robot arm operation dataset is constructed by collecting the operation process data of professional robot arm operators using the robot arm to perform different tasks. When the operator performs a task, at each moment of each task, a side-view monocular camera fixed to the robot arm base and a monocular camera fixed above the robot arm gripper are used to take pictures to obtain the perception dataset of the strategy large model. Record the text-based task instructions executed by the operator, use the posture information read by the robot body sensor at the current moment to form the robot arm posture perception data in text form, use the posture information read by the robot body sensor after the operator's operation at several subsequent moments to form the training truth data in text form, and use the dialogue template to splice the above text to obtain the text instruction data set of the strategy large model In the case of using voice to supplement task information or describe task instructions, collect and store the original voice clips to obtain the voice instruction data set of the strategy model. In addition, the user information voice database collection process described in step 3 is used to collect additional user information data and construct a vector database. , used to assist in the construction of multi-modal instruction information. The acquired data sets are combined into a robotic arm operation data set according to the tasks performed and the execution time. Each data takes the current text task instruction, the current voice task instruction, the current camera perception data and the current robot arm posture perception data as input data, and the robot arm posture perception data of several subsequent moments as the training truth value. Among them, the voice task instruction and the camera perception data are both represented in the form of file storage addresses. The constructed robot arm operation data set is divided into a training set and validation set .

[0050] The robot arm posture perception data needs to be further constructed into an action word list before model training can be performed. The process includes: converting the robot arm posture perception data back into floating point numbers, normalizing the data, and discretizing it into specific integers within a certain length interval. According to the frequency of vocabulary usage in the tokenizer word list from low to high, the floating point posture perception data is normalized, discretized, and mapped one by one to the word sequence number in the tokenizer word list. Through the construction of the action word list, the floating point posture data that is difficult to traverse can be unified into a limited number of numbers within a certain length interval.

[0051] The model training part includes two parts: speech modality training and robot arm operation training. The speech modality training is the first stage of training, and the robot arm operation training is the second stage of training.

[0052] The speech modality training, input training set To the model, load the pre-trained visual-language large model weights and speech feature encoder weights, use the multi-modal data processing process in step 1, the modal alignment and multi-modal instruction splicing process in step 2 to obtain the multi-modal instruction paragraph, input the strategy large model trunk to obtain the text response, and only use the text response part to calculate the loss function value to update the model weights during training, so that it has the ability to respond to the input visual and speech questions normally. After the training is completed, the training weight file is output and tested on the validation set. Verify the effect of the obtained weight file and select the weight file with the best performance as the optimal weight file .

[0053] The robot arm operation training, input training set To the model, load the weights trained with speech modality , the multi-modal data processing process in step 1, the modal alignment and multi-modal instruction splicing process in step 2 are also used to obtain the multi-modal instruction segment, and the strategy large model trunk is input to obtain the model response. For the non-model response part, a mask is used to avoid participating in the loss function calculation. Only the loss function value between the model response part and the training true value is calculated to update the model weight. After the training is completed, the training weight file is output and verified through the validation set Verify the effect of the obtained weight file and select the weight file with the best performance as the optimal weight file .

[0054] The two training processes both use the cross entropy loss function to calculate the loss value. The real label uses the one-hot distribution method to calculate the cross entropy with the probability distribution predicted by the model. The calculation formula is expressed as:

[0055] in, is the total loss value that needs to be optimized to minimize, represents the length of the input sequence, Indicates the first The predicted token at each position, Indicates The context before the position, Represents the probability distribution predicted by the model, indicating that given Under this condition, the model predicts that the next token is probability.

[0056] The deployment reasoning part reads the RGB images taken by the side-view monocular camera and the monocular camera at the gripper of the robot arm in real time, and the robot arm posture perception data, and the text instruction data of the current task, voice instruction data, and supplementary information extracted from the user information voice database are processed into a multimodal instruction paragraph, and the trained strategy model trunk is input to obtain a set of robot arm end effector postures for several future time steps, where the strategy model runs on the server computer, and the client computer collects instruction data, packages and sends instruction data, and obtains server response data. The server and the client communicate through WIFI (wireless network), and the client and the robot arm transmit information and control the robot arm through ROS topic nodes.

[0057] The autoregressive generation process of the strategy model under a given multimodal instruction paragraph can be modeled as:

[0058] in, Indicates the prefix part of the prompt, corresponding to the multimodal instruction part provided by the user. It means the subsequent text content predicted based on the given context corresponds to the position of the end effector of the robot arm in the next several time steps predicted by the large strategy model. Represents the current context, including the prompt and all predicted tokens. Indicates the next token to be predicted based on the current context. Represents the total probability of the model predicting the subsequent text content, It means predicting the probability of each subsequent token based on the current context.

[0059] The implementation method of step 5 is to perform inverse kinematics on the robot arm end effector posture of the future several time steps output by the strategy large model to obtain the rotation angle of each axis of the robot arm, and then control the motor through the PID controller. After completing several time-step actions planned by the large model, re-read the multiple modal data in the current environment, use the newly input visual perception data, robot arm posture perception data, text instruction data and voice instruction data to update the multi-modal instructions, and wait for the strategy large model to perform a new round of motion planning.

[0060] like Figure 3 As shown, the implementation method of step six is ​​to construct a closed-loop control loop with the strategy large model as the controller based on steps one to five, and through the model, repeatedly compare and understand the initial multimodal command information and the current environmental perception information, continuously output new action plans in a dynamically changing environment until it is determined that the current task has been completed, stop the robot arm action, and wait for the next task command to restart.

[0061] Embodiment 2: like Figure 4 As shown, the present invention provides a robotic arm control system for realizing multi-modal general operation tasks, the system comprising: a multi-modal data acquisition module, a task planning module, and a motion control and feedback module.

[0062] The multimodal data acquisition module uses sensors to collect various modal perception data and command data of the current working area, including multi-view RGB images taken by a side-view monocular camera fixed to the base of the robotic arm and a monocular camera fixed above the gripper of the robotic arm, text command data acquired through keyboard and network transmission, and voice command data describing tasks or supplementary information acquired through a microphone array. In addition, it also includes user information voice data that is additionally collected and stored before the task is executed.

[0063] The task planning module consists of two parts, a server and a client. It uses a high-cost GPU processor on the server to complete the deployment and reasoning of large models with huge computing volume, and uses a low-performance computer equipped with only a CPU processor and an SSD solid-state drive on the client to collect, forward and process instruction information, thereby decoupling the contradiction between the huge computing resources required for multi-modal large model reasoning and the limited local processor performance. The local robotic arm is controlled based on the end effector posture of the robotic arm in the form of a network service, which can keep the control system cost low even with a high number of requests.

[0064] Furthermore, the client in the task planning module is responsible for information transmission and parsing of local control instructions. When the task is executed, the collected text, image, voice and other modal data are first sent to the server through the network for processing. For the robotic arm end effector posture fed back by the server, effective information extraction, error checking and format conversion are performed, and then transmitted to the motion control and feedback module for execution.

[0065] Furthermore, the server in the task planning module is responsible for model deployment, reasoning and processing of multimodal information. When the task is executed, the received multimodal information is processed by the model deployed on the server to form a multimodal instruction segment, and the strategy large model backbone is input to obtain the position of the end effector of the robot arm for multiple time steps in the future, and then the planning information is sent to the client through the network for processing.

[0066] The motion control and feedback module receives the planning information processed by the client in the task planning module, that is, the position of the end effector of the manipulator at several time steps, obtains the rotation angle required to control each axis of the manipulator through inverse kinematics, and uses a PID controller to complete the motion control of the manipulator. After completing the planned action, the multimodal data acquisition module re-reads the current working area perception data and instruction data, and feeds back to the task planning module for new action planning.

[0067] Embodiment 3: like Figure 5 As shown, the present invention provides a robotic arm control device for realizing multi-modal general operation tasks, and the device includes: a visual sensor, an audio sensor, a text input interface, a client computer 2, a server computer 4, and a robotic arm kit.

[0068] The visual sensor is a group of monocular cameras 7, which are composed of a side-view monocular camera fixed to the base of the robotic arm and a monocular camera fixed above the gripper of the robotic arm. The fixed camera at the base is used to provide a stable side-view panoramic view for detecting the spatial position of the environment and objects, while the camera at the gripper can change with the movement of the robotic arm 5 and provide a close-up image of the object 8 to be grasped in real time.

[0069] The audio sensor uses a microphone array 6, and after the audio signal is subjected to noise reduction processing, key features such as user voice, ambient sound, etc. are extracted.

[0070] The text input interface uses devices such as a mobile phone 3 and a computer keyboard 1 to collect the user's text instruction data by sending the text instruction to be executed to the client computer 2 via the network or by typing the text instruction to be executed via the keyboard 1.

[0071] The client computer 2 is used to store and execute part of the program code, is equipped with a CPU processor and an SSD solid state drive, is directly connected to the robotic arm 5, a visual sensor, an audio sensor, and a text input interface to send control signals and collect sensor data, and communicates with the server computer 4 via the network. The client computer 2 and the robotic arm kit transmit information and control the robotic arm 5 through the ROS topic node.

[0072] The server computer 4 is used to deploy the trained strategy model, use a high-performance GPU processor to complete the deployment reasoning of the model, generate a set of robot arm end effector postures for several future time steps, obtain the rotation angles required to control each axis of the robot arm through inverse kinematics solution, and use a PID controller to complete the motion control of the robot arm; after completing the planned action, perform new action planning; provide efficient and fast result response and high-concurrency robot arm control services.

[0073] The robotic arm kit includes a robotic arm 5 (including an actuator), a controller, etc., which is responsible for executing the motion control signal processed by the client computer 2 to realize specific physical operations.

[0074] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.< / audio>

Claims

1. A method for controlling a robotic arm to realize a multi-modal general operation task, characterized in that: The method comprises the following steps: Step 1: Obtain the multimodal input data required for the task, including voice command data, text command data, visual perception data in the current task environment, and robotic arm posture perception data; Step 2: Perform modal alignment on the input data of multiple modalities, encode them into input representation vectors of a unified form, and concatenate them into multimodal input instructions in the form of paragraphs; Step 3: Based on the voiceprint features of the voice command data, retrieve and extract the task-related voice data of the corresponding user from the collected user information voice database as supplementary information of the command, and splice them in the same format; Step 4: Input the multimodal input command into the strategy large model backbone for command understanding, and generate a set of robot end effector poses for several future time steps, wherein the end effector poses include the action commands of the robot at each time step; Step 5: Control the motion of the robot arm based on the position of the end effector of the robot arm, and then obtain new multimodal input data to update the multimodal input instructions to perceive the latest task status; Step 6: The strategy model performs closed-loop control by comparing and understanding the initial target command information and the current environmental perception information, and continuously makes new action plans until the robotic arm completes the initially set mission objectives.

2. A method for controlling a robot arm to realize a multi-modal general operation task according to claim 1, characterized in that: The implementation method of step 1 includes: Step 101: The operator inputs text instruction data, and uses system prompt words, model dialogue templates, and supplementary prompt words to package them into a complete text string instruction; Step 102: The operator's voice command is converted into Mel spectrograms of several frequency bands using short-time Fourier transform (STFT), and then padded to a total frame number of a fixed length. If the original frame number is less than the length, zero padding is performed at the end of the time axis, otherwise it is truncated to the length; wherein the number of frequency bands and the total number of frames are determined by the parameters used in pre-training of the speech feature encoder used; Step 103: Obtain visual perception data and robot arm posture perception data through the sensor component. The visual perception data includes a side-view RGB image taken by a side-view monocular camera fixed on the base of the robot arm and an RGB image taken by a monocular camera fixed on the gripper of the robot arm. The images are resized, multi-image stitched, border pixel filled, and data format converted to form an image format that can be processed by the model. The robot arm posture perception data is obtained through the robot arm body sensor, including the coordinates of the robot arm, the angles of each joint, and the opening and closing information of the gripper. After reading, it is spliced ​​with the text instruction data in the form of a character string.

3. A method for controlling a robotic arm to realize a multi-modal general operation task according to claim 1, characterized in that: The implementation method of step 2 includes: Step 201: reserve a special mark symbol representing an image or voice in the text instruction data to indicate the insertion position of multiple modal information in the multimodal instruction; Step 202: For different modal information, use independent feature encoders to convert the information into a multimodal vector space with consistent dimensions, and use the input dimensions required by the strategy large model backbone to perform dimensional alignment; after the alignment is completed, splice and insert different modal information according to the reserved special marking symbols to form a multimodal instruction segment.

4. A method for controlling a robot arm to realize a multi-modal general operation task according to claim 3, characterized in that: The implementation method of step three includes: Step 301: Use a pre-trained or open-source speech recognition model to extract the user voiceprint information carried by the collected voice command, and use it as reference information for retrieving the user information voice database; the collection process of the user information voice database includes the following steps: pre-collecting user personal information, pre-coding voice information, storing voice data, and updating voice information; Step 302: Use voiceprint matching to locate the personal voice information of the user who currently issues the instruction, extract the voice information and then splice it into the special mark marked as user personal information in the multimodal instruction paragraph to indicate the supplementary user information.

5. The method for controlling a robot arm to realize a multi-modal general operation task according to claim 1, characterized in that: The strategy big model in step 4 adopts the vision-language-action big model obtained by fine-tuning the vision-language big model with full supervision, and the implementation method includes: a data set construction part, a model training part and a deployment reasoning part; The data set construction part includes two parts: speech dialogue data set construction and robot arm operation data set construction. The speech dialogue data set is obtained by performing speech synthesis on the community open source visual text dialogue data set and performing speech recognition on the visual speech dialogue data set, which is used to increase the model's support for end-to-end understanding of speech modal information. The robot arm operation data set is realized by collecting the operation process data of professional robot arm operators using the robot arm to perform different tasks. The original data includes task instructions, environmental perception and robot arm posture perception data for each task and each moment. The robot arm posture perception data needs to be further constructed with an action word list before model training can be performed, that is, the floating point posture perception data is normalized and discretized according to the vocabulary usage frequency in the tokenizer word list from low to high, and mapped one by one to the vocabulary sequence number in the tokenizer word list. The model training part includes two parts: speech modality training and robotic arm operation training. The speech dialogue data set and the robotic arm operation data set are used for training respectively, wherein the speech modality training is used as the first stage of training, and the robotic arm operation training is used as the second stage of training. The two-stage training process uses the cross entropy loss function to calculate the loss value, and the real label uses the one-hot distribution method to calculate the cross entropy with the probability distribution predicted by the model; The deployment reasoning part reads the RGB images taken by the side-view monocular camera and the monocular camera at the gripper of the robot arm, and the robot arm posture perception data in real time, and processes them together with the text instruction data of the current task, voice instruction data, and supplementary information extracted from the user information voice database into a multimodal instruction paragraph, and inputs the trained strategy large model backbone to obtain a set of robot arm end effector postures for several future time steps, wherein the strategy large model runs on the server computer, and the client computer collects instruction data, packages and sends instruction data, and obtains server response data; the server and the client communicate through a wireless network, and the client and the robot arm transmit information and control the robot arm through ROS topic nodes.

6. A method for controlling a robot arm to realize a multi-modal general operation task according to claim 1, characterized in that: The implementation method of step five is to perform inverse kinematics solution on the robot arm end effector posture for several future time steps output by the strategy large model to obtain the rotation angle of each axis of the robot arm, and then control the motor through a PID controller; after completing several time step actions planned by the large model, re-read the multiple modal data in the current environment, use the newly input visual perception data, robot arm posture perception data, text command data and voice command data to update the multimodal instructions, and wait for the strategy large model to perform a new round of motion planning.

7. A method for controlling a robotic arm to realize a multi-modal general operation task according to claim 1, characterized in that: The implementation method of step six is ​​to construct a closed-loop control circuit with the strategy large model as the controller based on steps one to five, and through the model, repeatedly compare and understand the initial multimodal command information and the current environmental perception information, continuously output new action plans in a dynamically changing environment, until it is determined that the current task has been completed, stop the robot arm action, and wait for the next task command to restart.

8. A robotic arm control system for realizing multi-modal general operation tasks, characterized in that: The system includes a multimodal data acquisition module, a task planning module, and a motion control and feedback module; The multimodal data acquisition module is used to collect perception data and instruction data of various modes in the current working area, perform modal alignment and encode and splice them into multimodal instruction paragraphs, including text instruction data, voice instruction data describing tasks or supplementary information, and visual perception data and robot arm posture perception data in the current task environment. In addition, it also includes a user information voice database that is additionally collected and established before the task is executed; The task planning module consists of a server and a client. The model deployment and reasoning are completed on the server using a GPU processor, and a set of robot arm end effector postures for several future time steps are generated based on multimodal instruction data. The end effector postures include the action instructions of the robot arm at each time step. The client uses a computer equipped with only a CPU processor and an SSD solid-state hard disk to collect, forward and process instruction information, and controls the local robot arm based on the robot arm end effector posture in the form of a network service. The motion control and feedback module is used to receive the planning information processed by the client in the task planning module, that is, the position of the end effector of the robotic arm at several time steps, obtain the rotation angle required to control each axis of the robotic arm through inverse kinematics solution, and use the PID controller to complete the motion control of the robotic arm; after completing the planned action, the current working area perception data and instruction data are re-read through the multimodal data acquisition module, and fed back to the task planning module for new action planning.

9. A robot arm control device for realizing multi-modal general operation tasks, characterized in that: The device includes: visual sensors, audio sensors, text input interface, client computer, server computer and robotic arm kit; The visual sensor is a set of monocular cameras, which are composed of a side-view monocular camera fixed to the base of the robotic arm and a monocular camera fixed above the gripper of the robotic arm. The fixed camera at the base is used to provide a stable side-view panoramic view for detecting the spatial position of the environment and the object, and the camera at the gripper can change with the movement of the robotic arm to provide a close-up image of the object in real time. The audio sensor uses a microphone array, and after the audio signal is processed by noise reduction, key features are extracted, including user voice and ambient sound; The text input interface uses a mobile phone or computer keyboard device to collect the user's text instruction data by sending it to the client computer through a wireless network or by typing the text instructions to be executed on the keyboard; The client computer is used to store and execute part of the program code, is equipped with a CPU processor and an SSD solid state drive, is directly connected to the robotic arm kit, the visual sensor, the audio sensor, and the text input interface to send control signals and collect sensor data, and communicates with the server computer via the network; The server computer is used to deploy the trained strategy model, use the GPU processor to complete the deployment reasoning of the model, generate a set of robot arm end effector positions for several future time steps, obtain the rotation angle required to control each axis of the robot arm through inverse kinematics solution, and use the PID controller to complete the motion control of the robot arm; after completing the planned action, perform new action planning; The robotic arm kit includes a robotic arm with an actuator and a controller, which is responsible for executing the motion control signal processed by the client to realize specific physical operations.

Images