A method and apparatus for training data extraction based on robot-based multi-modal monitoring data

Abstract

The embodiment of the application relates to a kind of method and device for training data extraction based on robot multi-modal monitoring data, the method comprises: loading voice activity detection model, automatic speech recognition model, instruction recognition language model, mechanical arm joint detection model, object detection model, ROI point cloud generation model, touch point planning model;Task recognition set of robot is identified using voice activity detection model, automatic speech recognition model, instruction recognition language model and mechanical arm joint detection model;Instruction recognition, target detection, touch point planning correctness verification are carried out based on the task recognition set identified by model and the multi-modal data generated by robot, and training sample is generated when verifying error.The application can find abnormality in time, and can improve the generation efficiency and annotation precision of training sample.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a method and apparatus for extracting training data based on multimodal monitoring data of a robot. Background Technology

[0002] Intelligent robots possess the processing ability to understand instructions and generate actions. Single-arm mobile manipulation robots (i.e., single-arm robots) are a common type. These robots have a fixed base, and their actuators typically consist of a multi-axis articulated robotic arm (six- or seven-axis) and a dexterous hand (e.g., the Shadow Hand, a five-fingered dexterous hand). They are used to retrieve specified items from the work environment and move them to a designated location based on voice commands. To achieve the object retrieval and placement task, the robot is equipped with multiple perception and planning modules. For example, a voice command recognition module converts environmental speech into structured task instructions (including the object to be retrieved and the object to be placed); a target detection module identifies the positions of the retrieved and placed objects from the environmental image; and a touch point planning module plans the initial positions of the five fingers of the dexterous hand when grasping the target object. The object to be retrieved, as mentioned above, is the "designated item," specifically a target object in the environment. The object to be placed, as mentioned above, is the "designated location," specifically a target loading object or target placement area in the environment used to place the retrieved object. For example, if the text content of the ambient voice is "put the apple in the fruit basket," then the retrieved object is "apple," and the placed object is "fruit basket." This type of robot is widely used in service, assembly, and logistics scenarios, requiring it to have reliable perception, planning, and execution capabilities.

[0003] Observations revealed that factors such as changes in ambient lighting, the diversity of object postures, differences in voice command accents, and background noise interference can cause perception or planning deviations in the robot's perception and planning modules, leading to task errors. For example, errors in voice command recognition can result in picking up the wrong item or placing it in the wrong location; deviations in target detection can cause target position shifts during object retrieval or placement; and deviations in contact point planning can lead to incorrect or unstable object grasping, resulting in grasping failure or the object slipping during movement. Once these problems occur, it is necessary to promptly collect corresponding training data to optimize and train the relevant modules.

[0004] Currently, the conventional method for collecting training data involves first recording the robot's multimodal process data (including command data, environmental images, target detection results, touch point planning results, etc.) during operation, and then manually or semi-manually labeling and organizing the training data. Due to limitations in manual operation efficiency and labeling experience, this conventional method often suffers from poor timeliness, low labeling efficiency, and low labeling accuracy. To meet the labeling quality requirements for optimized training, one or more rounds of manual or semi-manual verification of the training data are usually necessary. In other words, it is difficult to simultaneously meet the requirements of accuracy and timeliness of training data using conventional methods.

[0005] To address this, we propose a novel training data extraction scheme: incorporating a speech activity detection model, an automatic speech recognition model, a command recognition language model, a robotic arm joint detection model, an object handling detection model, an ROI point cloud generation model, and a touchpoint planning model. This scheme automates the verification of multimodal process data during robot operation and automatically generates corresponding training samples when errors are detected, achieving an automated closed loop from robot task execution to high-quality training data generation. The execution of this scheme requires no manual intervention, enabling timely anomaly detection and improving the efficiency and accuracy of training sample generation and annotation. This scheme simultaneously meets the requirements for both accuracy and timeliness of training data. The key technical problem this invention aims to solve is how to implement this scheme. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, apparatus, electronic device, and computer-readable storage medium for training data extraction based on multimodal monitoring data of a robot. This invention introduces seven types of models (voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object handling detection model, ROI point cloud generation model, and touch point planning model) to automatically verify the multimodal process data during the operation of a single-arm robot. Microphones and cameras are installed in the robot's operating environment for observation. During the monitoring period, the observation microphones / cameras are used to receive, record, and capture real-time audio / video of the robot's operating environment and process, generating corresponding audio / video recording files. After the monitoring period ends, the multimodal dataset generated during the monitoring period is obtained from the single-arm robot. The invention utilizes the Voice Activity Detection (VAD) model and the Automatic Speech (ASM) model. This invention utilizes an Audio Recognition (ASR) model, a command recognition language model, and a robotic arm joint detection model. Based on audio and video recordings, it identifies the command audio, command information, and task time periods for a single robot task within a monitoring period to obtain a task recognition set. The multimodal dataset is then segmented into corresponding task data subsets based on task time periods. Command recognition correctness is verified based on each command information and its corresponding command data; training samples are generated and added to the audio recognition dataset when command recognition errors occur. Similarly, object detection correctness is verified based on a retrieval and placement detection model, command data for each task, environmental image data, retrieval target recognition data, and placement target recognition data; training samples are generated and added to the object detection dataset when object detection errors occur. Finally, touchpoint planning correctness is verified based on a Region of Interest (ROI) point cloud generation model, a touchpoint planning model, command data for each task, ROI image data, and touchpoint planning data; training samples are generated and added to the touchpoint planning dataset when touchpoint planning errors occur. This invention can promptly detect anomalies and improve the efficiency and accuracy of training sample generation and annotation.

[0007] To achieve the above objectives, a first aspect of the present invention provides a method for extracting training data based on multimodal monitoring data of a robot, the method comprising: The monitoring device is loaded with trained voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object picking and placing detection model, ROI point cloud generation model, and touch point planning model; The device connects to the monitored single-arm robot and is also connected to an observation microphone and an observation camera installed in the robot's operating environment. It locally stores a robot storage area, an observation storage area, and a dataset storage area. The single-arm robot retrieves a specified object from the operating environment based on received voice commands and moves it to the specified location. The single-arm robot also has a locally configured task data area for storing multimodal process data generated during its operation. This multimodal process data includes multiple command data, multiple environmental image data, multiple object retrieval target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple touchpoint planning data. The audio reception range of the observation microphone and the shooting angle range of the observation camera both cover the robot's operating space, and the observation camera can capture unobstructed images of the robot's object retrieval process. The observation camera is a depth camera. The dataset storage area stores audio recognition datasets, target detection datasets, and touchpoint planning datasets. During the preset monitoring period, the single-arm robot is activated to perform object retrieval tasks according to environmental voice commands in its working environment. The observation microphone is used to receive and record real-time audio of the robot's working environment, and the observation camera is used to capture and record the robot's operation process. Corresponding audio and video recording files are generated in the observation storage area, and the recording data from the microphone and camera is continuously stored. At the end of the monitoring period, the audio and video recording from the microphone and camera is stopped, and the multimodal dataset generated during the monitoring period is obtained from the task data area of ​​the single-arm robot and stored in the robot storage area. Using the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model, a task identification set is obtained by identifying the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files; the task identification set includes multiple task information items; the task information includes the command audio, the command information, and the task time period; The multimodal dataset is segmented based on each of the task time periods to obtain corresponding task data subsets; The correctness of instruction recognition is verified based on the instruction data of each instruction information and its corresponding task data subset, and training samples are generated and added to the audio recognition dataset when the instruction recognition is incorrect. Based on the object detection model and the instruction data, environmental image data, object detection target recognition data, and object placement target recognition data of each task data subset, the object detection correctness is verified, and training samples are generated and added to the object detection dataset when the object detection is incorrect. The correctness of touch point planning is verified based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset. If the touch point planning is incorrect, training samples are generated and added to the touch point planning dataset.

[0008] Preferably, the speech activity detection model includes at least the Silero VAD model and the FunASR VAD model; the speech activity detection model is used to detect speech segments on the input audio and output the corresponding speech segment detection set; when the speech segment detection set is not empty, it consists of one or more speech segment detection data; the speech segment detection data includes the segment start time and the segment end time; The automatic speech recognition model includes at least the Qwen-ASR series model and the Whisper series model; the automatic speech recognition model is used to perform text recognition on the input audio and output the corresponding audio text. The instruction recognition language model is implemented based on a class of generative large language models, including at least the Qwen series models, the SmolVLA model, and the DeepSeek series models. The instruction recognition language model is used to identify whether the source text carried in the recognition task instruction input to the model is a retrieval instruction text that requires first obtaining a specified item and then placing it in a specified location. Based on the text recognition result, the model identifies the obtained item and the placement location in the source text and outputs the corresponding text recognition result, the obtained item, and the placement location. The identification task instructions are generated based on a preset identification task instruction template; The recognition task instruction template is a formatted text template, and the configurable parameters of the template include the source text. The recognition task instruction template is used to identify whether the source text is a retrieval instruction text that requires first obtaining a specified item and then placing it in a specified location, and to provide a specific yes or no text recognition result. If the text recognition result is yes, then target recognition is performed on the obtained item and the placement location in the source text, and it is stated that the placement location is another target item or target area that can be used to place / load the obtained item. If the text recognition result is no, then the corresponding obtained item and placement location are set to empty. The obtained text recognition result, the obtained item, and the placement location are output as the model processing result. The robotic arm joint detection model is based on a type of visual target detection model, including at least the YOLO series models. The model is used to detect robotic arm joints on the input image and output a corresponding set of joint detection boxes. The set of joint detection boxes includes multiple joint detection boxes. Each joint detection box includes the target joint type, the coordinates of the detection box center point, and the detection box size. The target joint types include six or seven robotic arm joints, and the total number of types matches the total number of robotic arm joints in the single-arm robot. The object detection model is based on a class of open vocabulary target detection models, including at least the GroundingDINO Tiny model, the OV-DINO model, the YOLOE-v8s-seg model, and the YOLO-World v2 series models. The model extracts the corresponding object to be picked up and the object to be placed from the input text, and detects the objects on the input image that correspond to the object to be picked up and the object to be placed, respectively, and outputs corresponding object detection boxes and object detection boxes. The object detection box includes a first target type, the coordinates of the center point of the first detection box, and the size of the first detection box; the first target type matches the item type of the object to be picked up. The object detection box includes a second target type, the coordinates of the center point of the second detection box, and the size of the second detection box; the second target type matches the item type or region type of the object to be placed. The ROI point cloud generation model consists of a semantic segmentation model and a point cloud conversion module. The semantic segmentation model is based on the SAM series models. The model input of the ROI point cloud generation model includes the ROI type and an RGB-D image, and the model output is the corresponding ROI point cloud. The ROI type is a type of object, and the RGB-D image is an image with depth information captured by a depth camera. The semantic segmentation model is used to perform semantic segmentation on the RGB-D image input to the model to obtain the corresponding semantic segmentation image. The point cloud conversion module is used to extract the pixels in the semantic segmentation image whose semantic type matches the ROI type input to the model to form the corresponding target mask image, and to perform color point cloud conversion on the target mask image to obtain the corresponding ROI point cloud and output it. The touch point planning model consists of a DexVLG model and a touch point calculation module. The model inputs include the ROI point cloud and prompt text, and the model output is the five-finger touch point positions. The ROI point cloud is a 3D color point cloud of the object to be grasped. The prompt text indicates that the corresponding object in the ROI point cloud should be grasped. The five-finger touch point positions include the thumb, index finger, middle finger, ring finger, and little finger touch points. The DexVLG model plans the grasping posture of the robot's dexterous hand grasping the current object based on the prompt text and the ROI point cloud, and outputs the corresponding wrist joint coordinates, wrist orientation, and multiple finger joint angles. The touch point calculation module calculates the corresponding five-finger touch point positions based on the wrist joint coordinates, wrist orientation, and all finger joint angles using the forward kinematics of the Shadow Hand, and outputs them. The instruction data includes task instructions and instruction timestamps; the task instructions include the object to be retrieved and the object to be placed; the object to be retrieved is a target object to be retrieved in the environment, and the object to be placed is a target loading object or target placement area in the environment used to place the object to be retrieved. The environmental image data includes images of the work environment and image timestamps; The object retrieval target identification data includes an object retrieval target identification box, object retrieval target location, and object retrieval identification timestamp; the object retrieval target identification box includes object retrieval target type, object retrieval box center point coordinates, and object retrieval box size; the object retrieval target type matches the item type of the corresponding object being retrieved. The object recognition data includes an object recognition frame, object location, and object recognition timestamp; the object recognition frame includes object type, frame center coordinates, and frame size; the object type matches the item type or region type of the corresponding object. The ROI image data includes the object ROI image and the image timestamp; The touch point planning data includes the initial positions of the five fingers and the planning timestamp; the initial positions of the five fingers include the initial positions of the thumb, index finger, middle finger, ring finger, and little finger, and some or all of the initial positions are in contact with the surface of the object to be picked up; The multimodal dataset includes command sequences, environmental image sequences, object retrieval target sequences, object placement target sequences, ROI image sequences, and touch point planning sequences; wherein, the command sequences, environmental image sequences, object retrieval target sequences, object placement target sequences, ROI image sequences, or touch point planning sequences are each composed of multiple sets of command data, environmental image data, object retrieval target identification data, object placement target identification data, ROI image data, or touch point planning data that occur within the monitoring period and correspond to the current sequence; The instruction information includes the object to be retrieved and the object to be placed; The task time period includes the task start time and the task end time; The task data subset is a subset of the multimodal dataset in the corresponding task time period, consisting of a set of corresponding instruction data, environmental image data, object retrieval target recognition data, object placement target recognition data, ROI image data, and touch point planning data; The audio recognition dataset includes multiple first training data sets; the first training data sets include training audio and instruction labels. The target detection dataset includes multiple second training data sets; the second training data sets include training environment images, object capture box labels, and object placement box labels. The touch point planning dataset includes multiple third training data sets; the third training data sets include training ROI images and touch point labels.

[0009] Preferably, the step of acquiring the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and storing it in the robot's storage area specifically includes: In the task data area, the instruction timestamps within the monitoring period are extracted to form the corresponding instruction sequence; the image timestamps within the monitoring period are extracted to form the corresponding environmental image sequence; the object retrieval recognition timestamps within the monitoring period are extracted to form the corresponding object retrieval target sequence; the object placement recognition timestamps within the monitoring period are extracted to form the corresponding object placement target sequence; the image timestamps within the monitoring period are extracted to form the corresponding ROI image sequence; and the planning timestamps within the monitoring period are extracted to form the corresponding touchpoint planning sequence. The resulting multimodal dataset, composed of the instruction sequence, environmental image sequence, object retrieval target sequence, object placement target sequence, ROI image sequence, and touchpoint planning sequence, is then stored in the robot storage area.

[0010] Preferably, the step of using the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model to identify the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files specifically includes: The audio recording file is used as the model input audio to the speech activity detection model to detect speech segments and obtain the corresponding speech segment detection set; and based on the speech segment detection data of each speech segment in the speech segment detection set, the corresponding speech segments are extracted from the audio recording file; and each extracted speech segment is used as the model input audio to the automatic speech recognition model to perform text recognition and obtain the corresponding audio text; and each audio text is used as the corresponding source text to the recognition task instruction template to generate the corresponding recognition task instruction; and each recognition task instruction is input to the instruction recognition language model for processing to obtain the corresponding text recognition result, the obtained item, and the placement location; The text recognition results of each specific item are taken as the current result; the item to be retrieved and the placement location corresponding to the current result are taken as a set of corresponding items to be retrieved and placed to form the corresponding instruction information; and the corresponding audio segment of the current result and the corresponding segment start time are taken as the corresponding instruction audio and the task start time. The start time of each task is taken as the current time t. s And set the corresponding time t based on the preset maximum task duration L1. e =t s +L1; and transfer the time segment [t] from the video recording file. s ,t eThe video clip is extracted as the current video; the current video is then processed into frames; all frames are traversed in chronological order; during this traversal, the currently traversed image is used as the current image; it is determined whether the current image is the first frame; if so, the previous frame is set to empty; otherwise, the previous frame is set as the previous frame. The current image is then input into the robotic arm joint detection model to perform robotic arm joint target detection and obtain the corresponding set of joint detection boxes; based on the camera intrinsic and extrinsic parameters of the observation camera, the coordinates of the center points of each detection box in the set of joint detection boxes are transformed into three-dimensional world coordinates to obtain the coordinates of the corresponding joint center points; the three-dimensional coordinate difference between every two joint center point coordinates is calculated to obtain the corresponding first coordinate difference; and the three-dimensional coordinate difference between every two initial positions of the robotic arm joints in the preset initial positions of the robot joints is calculated. The first and second coordinate differences are calculated to obtain the corresponding second coordinate difference; and the first and second coordinate differences with the same joint relative relationship are used to form corresponding difference pairs; and the first and second coordinate differences of each difference pair are calculated again to obtain the corresponding third coordinate difference; and it is identified whether each of the third coordinate differences meets the preset three-dimensional coordinate difference range. If it does, the corresponding matching result is set as a match; if it does not, the matching result is set as a mismatch; and it is identified whether all matching results corresponding to the current image are matches. If they are, the current image is marked as a repositioned image; otherwise, the current image is marked as a non-repositioned image; and when the current image is a repositioned image and the previous frame image is a non-repositioned image, the current round of traversal is terminated and the image time of the current image is taken as the corresponding task end time; if the corresponding task end time is not obtained until the last frame of the current round of traversal, then the current time t is used as the task end time. e Set the corresponding task end time; wherein, the initial position of the robot joint includes the initial positions of multiple robotic arm or hand joints, each initial position being a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; the range of the three-dimensional coordinate difference consists of the difference ranges of the X, Y, and Z dimensions. If the difference of each dimension of the third coordinate difference satisfies the corresponding difference range, then the current third coordinate difference satisfies the range of the three-dimensional coordinate difference; if at least one dimension of the third coordinate difference does not satisfy the corresponding difference range, then the current third coordinate difference does not satisfy the range of the three-dimensional coordinate difference. The task time period is composed of the start time of each task and its corresponding end time; the task information is composed of the task time period and its corresponding instruction audio and instruction information; and the task identification set is composed of all the obtained task information.

[0011] Preferably, the step of segmenting the multimodal dataset based on each of the task time periods to obtain corresponding task data subsets specifically includes: Each task time period in the task identification set is taken as the current task time period; and the latest instruction timestamp among all instruction data in the instruction sequence of the multimodal dataset that satisfies the current task time period is taken as the current instruction data; the latest image timestamp among all environmental image data in the environmental image sequence that satisfies the current task time period is taken as the current environmental image data; the latest object recognition timestamp among all object recognition data in the object retrieval target sequence that satisfies the current task time period is taken as the current object recognition data; and the latest object recognition timestamp among all object recognition data in the object placement target sequence that satisfies the current task time period is taken as the current object recognition data. The latest time among all the object placement target recognition data is used as the current object placement target recognition data. The image timestamp in the ROI image sequence that satisfies the latest time among all the ROI image data of the current task time period is used as the current ROI image data. The planning timestamp in the touch point planning sequence that satisfies the latest time among all the touch point planning data of the current task time period is used as the current touch point planning data. The obtained current instruction data, current environment image data, current object retrieval target recognition data, current object placement target recognition data, current ROI image data, and current touch point planning data form the corresponding task data subset.

[0012] Preferably, the step of verifying the correctness of instruction recognition based on the instruction data of each instruction information and its corresponding task data subset, and generating training samples to be added to the audio recognition dataset when instruction recognition is incorrect, specifically includes: Each instruction information in the task recognition set and its corresponding task data subset are taken as the corresponding current instruction 1 and current subset; the instruction data of the current subset is taken as current instruction 2; and the object to be picked up and placed in the current instructions 1 and 2 are identified. If they are, the corresponding current verification result is set as instruction recognition correct; otherwise, the corresponding current verification result is set as instruction recognition error. When the current verification result is instruction recognition error, the instruction audio corresponding to the current instruction 1 is taken as the corresponding training audio, and the current instruction 1 is taken as the corresponding instruction tag. The obtained training audio and the instruction tag are combined to form the corresponding first training data and added to the audio recognition dataset.

[0013] Preferably, the step of verifying the correctness of target detection based on the object retrieval detection model and the instruction data, environmental image data, object retrieval target recognition data, and object placement target recognition data of each of the task data subsets, and generating training samples to be added to the target detection dataset when the target detection is incorrect, specifically includes: Each of the task data subsets is taken as the current subset; and the corresponding instruction data, environmental image data, object retrieval target recognition data, and object placement target recognition data are extracted from the current subset; The object to be picked and the object to be placed in the instruction data are extracted to form the corresponding current model input text; the working environment image in the environmental image data is used as the current model input image; and the current model input text and the current model input image are input into the object picking and placing detection model for processing to obtain the corresponding object picking detection box and object placing detection box. The object detection frame and the object target recognition frame of the object detection data are combined to form a first frame group, and the object placement detection frame and the object placement target recognition frame of the object placement target recognition data are combined to form a second frame group. The first or second frame group is used as the current frame group. The target types of the two frames in the current frame group are checked for matching. If not, the corresponding frame group verification result is set to failure. If yes, the straight-line distance between the coordinates of the two center points of the current frame group and the height-width difference between the two frames are calculated to obtain the corresponding current center distance, current height difference, and current width difference. The current center distance is set to a preset value. When the center distance is within the preset range, and the current height difference meets the preset height difference range, and the current width difference meets the preset width difference range, the corresponding frame group verification result is set to successful. When the current center distance does not meet the preset center distance range, or the current height difference does not meet the preset height difference range, or the current width difference does not meet the preset width difference range, the corresponding frame group verification result is set to failed. The system also identifies whether both frame group verification results are successful. If so, the corresponding current verification result is set to "target detection correct"; otherwise, the corresponding current verification result is set to "target detection incorrect". When the current verification result is an object detection error, the current working environment image, the object detection box, and the object placement detection box are used as a set of corresponding training environment images, object detection box labels, and object placement box labels to form a corresponding second training data, and the current second training data is added to the object detection dataset.

[0014] Preferably, the step of verifying the correctness of touch point planning based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset, and generating training samples to be added to the touch point planning dataset when touch point planning errors occur, specifically includes: Each of the task data subsets is taken as the current subset; and the corresponding instruction data, ROI image data, and touch point planning data are extracted from the current subset; Based on the object being retrieved according to the instruction data, the corresponding ROI type is set, and the ROI image of the retrieved object from the ROI image data is used as the corresponding RGB-D image. The current ROI type and the RGB-D image are then input into the ROI point cloud generation model for processing to obtain the corresponding ROI point cloud. According to the preset DexVLG model prompt text setting rules, the corresponding prompt text is set based on the object being retrieved. The current ROI point cloud and the prompt text are then input into the touch point planning model for processing to obtain the corresponding five-finger touch point position. The coordinates of all five finger contact points and the initial positions of the five fingers in the contact point planning data are unified to the same three-dimensional coordinate system, which is either the world coordinate system or the robot's XYZ three-dimensional coordinate system. The five contact points of the five fingers are then sequentially recorded as their corresponding positions. The five touch points at the initial positions of the five fingers are sequentially recorded as the corresponding positions. 1 ≤ index i ≤ 5; and calculate the geometric position deviation e between the five-finger contact point position and the initial position of the five fingers. ; It identifies whether the geometric position deviation e is less than a preset geometric position deviation threshold; if yes, the corresponding current verification result is set as correct contact planning; if no, the corresponding current verification result is set as incorrect contact planning. When the current verification result indicates a touch point planning error, the current ROI image data and the five-finger touch point positions are used as a set of corresponding training ROI images and touch point labels to form a corresponding third training data set; and the current third training data is added to the touch point planning dataset.

[0015] Preferably, the base of the single-arm robot is fixed; The actuator of the single-arm robot includes an articulated robotic arm and a five-fingered dexterous hand; the articulated robotic arm is a six-axis or seven-axis robotic arm, corresponding to six or seven robotic arm joints, and its specific types include at least the KUKA robotic arm and the UR robotic arm; the base joint of the articulated robotic arm is connected to a fixed body base, and the end joint is connected to the five-fingered dexterous hand; The five-finger dexterous hand is a Shadow Hand dexterous hand, which is connected to the end joint of the jointed robotic arm via the wrist joint; the palm of the five-finger dexterous hand is equipped with a palm camera and a tactile sensor, and each fingertip is equipped with the tactile sensor; The tactile sensor is used to sense multidimensional forces and the coefficient of friction of the palm / fingertip contact surface with the object; the multidimensional forces include multiple sensed force data, and the sensed force data includes force type and three-dimensional force components; the force type includes at least contact force, contact surface pressure, normal force, and shear force; the three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system; the coefficient of friction is estimated based on the normal force and shear force; The sensors of the single-arm robot also include an audio receiver and a rotating camera; the audio receiver's audio reception range and the rotating camera's rotation angle range can fully cover the robot's working space; the rotating camera is a depth camera. When the single-arm robot is not performing a task based on voice commands, it controls all joint positions of the articulated robotic arm and the five-fingered dexterous hand to remain consistent with the preset initial positions of the robot joints; the initial positions of the robot joints include multiple initial positions of the robotic arm or hand joints, and each initial position is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; The single-arm robot is specifically used when, according to the received voice command, it retrieves the object specified by the command from the work environment and moves it to the specified location for placement: The system performs instruction conversion on the real-time environmental audio received by the audio receiver and identifies whether the conversion instruction is a valid task instruction. After identifying a valid task instruction, the corresponding object to be retrieved and the object to be placed are extracted from the current task instruction; and the instruction data composed of the current task instruction and its corresponding instruction timestamp is stored in the task data area. The system uses the rotating camera to capture a panoramic view of the robot's working environment, obtaining an image of the working environment. It then identifies the objects to be picked up or placed, identifies the corresponding target positions, and plans the palm-shaped target positions near the object's location. The system stores the environmental image data, composed of the working environment image and its corresponding timestamp, in the task data area. Finally, it stores the corresponding object recognition data, composed of the object recognition frame, object location, and object recognition timestamp for each object, in the task data area. With the constraint that the robotic arm and dexterous hand do not collide with any object during the movement, real-time motion planning and drive control are performed on all joints of the robotic arm and dexterous hand, so that the palm of the five-fingered dexterous hand moves to the target position of the palm. Upon reaching the target location in the palm, the palm camera is invoked to capture a local environment of interest image centered on the object to be retrieved, thereby obtaining the ROI image; and the ROI image data, composed of the ROI image and its corresponding image timestamp, is stored in the task data area. Based on the object retrieval ROI image, the initial positions and initial torques of the five fingers are planned; and based on the initial positions of the five fingers, the motion planning and drive control of all joints of the dexterous hand are performed, so that the center positions of the fingertips of the dexterous hand reach their respective initial positions and make pre-contact with the object being retrieved; and the contact point planning data composed of the initial positions of the five fingers and the corresponding final planning timestamps is stored in the task data area; wherein, the initial torque of the five fingers is composed of multiple finger joint torques, and the finger joint torque is a six-dimensional torque in the XYZ three-dimensional coordinate system of the robot body, which is composed of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes; After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers as the initial torque value. During the operation, the tactile perception data fed back by all the tactile sensors in real time is used to confirm whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all the real-time feedback tactile perception data, with the constraint of not causing damage to the object. The five-finger dexterity hand is then driven to continue grasping the object using the current optimized five-finger torque. The optimized five-finger torque is composed of the torques of multiple corresponding finger joints. After confirming that the object being retrieved has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target position and the shape and material characteristics of the object being retrieved. With the constraint that the robotic arm and the dexterous hand do not collide with any object during the movement and that the object held by the dexterous hand will not loosen and fall, the motion planning and drive control of all joints of the robotic arm and the dexterous hand are performed in real time, so that the center point of the object being retrieved moves to the release center position. After reaching the release center position, the object is released by driving the hand joints of the five-fingered dexterous hand; and after successfully releasing the object, the jointed robotic arm and all joints of the five-fingered dexterous hand are driven back to their original positions based on the initial positions of the robot joints.

[0016] A second aspect of the present invention provides an apparatus for implementing the method for extracting training data based on multimodal monitoring data of a robot as described in the first aspect above. The apparatus includes: a model loading module, a monitoring preparation module, a monitoring acquisition module, a task segment identification module, a monitoring data segmentation module, an instruction recognition training data extraction module, a target detection training data extraction module, and a touch point planning training data extraction module. The model loading module is used to load the trained speech activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object retrieval and placement detection model, ROI point cloud generation model, and touch point planning model. The monitoring preparation module connects to the monitored single-arm robot and to an observation microphone and camera installed in the robot's operating environment. It also locally stores a robot storage area, an observation storage area, and a dataset storage area. The single-arm robot retrieves a specified object from the operating environment based on received voice commands and moves it to the designated location. The robot also has a locally configured task data area for storing multimodal process data generated during its operation. This multimodal process data includes multiple command data, multiple environmental image data, multiple object retrieval target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple touchpoint planning data. The audio reception range of the observation microphone and the shooting angle range of the observation camera both cover the robot's operating space, and the observation camera can capture unobstructed images of the robot's object retrieval process. The observation camera is a depth camera. The dataset storage area stores audio recognition datasets, target detection datasets, and touchpoint planning datasets. The monitoring and acquisition module is used to start the single-arm robot to perform object retrieval tasks according to environmental voice commands in its working environment within a preset monitoring period, and to call the observation microphone to receive and record real-time audio of the robot's working environment, and to call the observation camera to shoot and record the robot's working process, and to generate corresponding audio and video recording files in the observation storage area to continuously store the recording data of the microphone and camera; and at the end of the monitoring period, to stop the audio and video recording of the microphone and camera, and to obtain the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and store it in the robot storage area; The task segment recognition module is used to utilize the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model to identify the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files to obtain a task recognition set; the task recognition set includes multiple task information; the task information includes the command audio, the command information, and the task time period; The monitoring data segmentation module divides the multimodal dataset into corresponding task data subsets based on each task time period; The instruction recognition training data extraction module performs instruction recognition correctness verification based on the instruction data of each instruction information and the corresponding task data subset, and generates training samples to be added to the audio recognition dataset when instruction recognition is incorrect. The target detection training data extraction module performs target detection correctness verification based on the object retrieval and placement detection model and the instruction data, environmental image data, object retrieval target recognition data and object placement target recognition data of each task data subset, and generates training samples to be added to the target detection dataset when the target detection is incorrect; The touch point planning training data extraction module performs touch point planning correctness verification based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset. When touch point planning is incorrect, training samples are generated and added to the touch point planning dataset.

[0017] Preferably, the base of the single-arm robot is fixed; The actuator of the single-arm robot includes an articulated robotic arm and a five-fingered dexterous hand; the articulated robotic arm is a six-axis or seven-axis robotic arm, corresponding to six or seven robotic arm joints, and its specific types include at least the KUKA robotic arm and the UR robotic arm; the base joint of the articulated robotic arm is connected to a fixed body base, and the end joint is connected to the five-fingered dexterous hand; The five-finger dexterous hand is a Shadow Hand dexterous hand, which is connected to the end joint of the jointed robotic arm via the wrist joint; the palm of the five-finger dexterous hand is equipped with a palm camera and a tactile sensor, and each fingertip is equipped with the tactile sensor; The tactile sensor is used to sense multidimensional forces and the coefficient of friction of the palm / fingertip contact surface with the object; the multidimensional forces include multiple sensed force data, and the sensed force data includes force type and three-dimensional force components; the force type includes at least contact force, contact surface pressure, normal force, and shear force; the three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system; the coefficient of friction is estimated based on the normal force and shear force; The sensors of the single-arm robot also include an audio receiver and a rotating camera; the audio receiver's audio reception range and the rotating camera's rotation angle range can fully cover the robot's working space; the rotating camera is a depth camera. When the single-arm robot is not performing a task based on voice commands, it controls all joint positions of the articulated robotic arm and the five-fingered dexterous hand to remain consistent with the preset initial positions of the robot joints; the initial positions of the robot joints include multiple initial positions of the robotic arm or hand joints, and each initial position is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; The single-arm robot is specifically used when, according to the received voice command, it retrieves the object specified by the command from the work environment and moves it to the specified location for placement: The system performs instruction conversion on the real-time environmental audio received by the audio receiver and identifies whether the conversion instruction is a valid task instruction. After identifying a valid task instruction, the corresponding object to be retrieved and the object to be placed are extracted from the current task instruction; and the instruction data composed of the current task instruction and its corresponding instruction timestamp is stored in the task data area. The system uses the rotating camera to capture a panoramic view of the robot's working environment, obtaining an image of the working environment. It then identifies the objects to be picked up or placed, identifies the corresponding target positions, and plans the palm-shaped target positions near the object's location. The system stores the environmental image data, composed of the working environment image and its corresponding timestamp, in the task data area. Finally, it stores the corresponding object recognition data, composed of the object recognition frame, object location, and object recognition timestamp for each object, in the task data area. With the constraint that the robotic arm and dexterous hand do not collide with any object during the movement, real-time motion planning and drive control are performed on all joints of the robotic arm and dexterous hand, so that the palm of the five-fingered dexterous hand moves to the target position of the palm. Upon reaching the target location in the palm, the palm camera is invoked to capture a local environment of interest image centered on the object to be retrieved, thereby obtaining the ROI image; and the ROI image data, composed of the ROI image and its corresponding image timestamp, is stored in the task data area. Based on the object retrieval ROI image, the initial positions and initial torques of the five fingers are planned; and based on the initial positions of the five fingers, the motion planning and drive control of all joints of the dexterous hand are performed, so that the center positions of the fingertips of the dexterous hand reach their respective initial positions and make pre-contact with the object being retrieved; and the contact point planning data composed of the initial positions of the five fingers and the corresponding final planning timestamps is stored in the task data area; wherein, the initial torque of the five fingers is composed of multiple finger joint torques, and the finger joint torque is a six-dimensional torque in the XYZ three-dimensional coordinate system of the robot body, which is composed of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes; After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers as the initial torque value. During the operation, the tactile perception data fed back by all the tactile sensors in real time is used to confirm whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all the real-time feedback tactile perception data, with the constraint of not causing damage to the object. The five-finger dexterity hand is then driven to continue grasping the object using the current optimized five-finger torque. The optimized five-finger torque is composed of the torque of the corresponding multiple finger joints. After confirming that the object being retrieved has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target position and the shape and material characteristics of the object being retrieved. With the constraint that the robotic arm and the dexterous hand do not collide with any object during the movement and that the object held by the dexterous hand will not loosen and fall, the motion planning and drive control of all joints of the robotic arm and the dexterous hand are performed in real time, so that the center point of the object being retrieved moves to the release center position. After reaching the release center position, the object is released by driving the hand joints of the five-fingered dexterous hand; and after successfully releasing the object, the jointed robotic arm and all joints of the five-fingered dexterous hand are driven back to their original positions based on the initial positions of the robot joints.

[0018] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver; The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0020] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for training data extraction based on multimodal monitoring data of a robot. As described above, this invention introduces seven types of models (voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object retrieval and placement detection model, ROI point cloud generation model, and touch point planning model) to automatically verify the multimodal process data during the operation of a single-arm robot. Microphones and cameras are installed in the robot's operating environment for observation. During the monitoring period, the observation microphones / cameras are used to receive, record, and capture real-time audio / video recordings of the robot's operating environment and process, generating corresponding audio / video recording files. After the monitoring period ends, the multimodal dataset generated during the monitoring period is obtained from the single-arm robot. The voice activity detection model, automatic speech recognition model, command recognition language model, and robotic arm joint detection model are used to analyze the training data based on the audio and video recording files. The invention employs several methods to verify the correctness of robot task recognition. First, it identifies the audio and information of robot commands for a single task within a given time period, along with the task time period itself. Then, it segments the multimodal dataset into corresponding task data subsets based on the task time period. Second, it verifies the correctness of command recognition by matching command information with corresponding command data, generating training samples and adding them to the audio recognition dataset when command recognition errors occur. Third, it verifies the correctness of target detection by using a retrieval and placement detection model, along with command data, environmental image data, retrieval target recognition data, and placement target recognition data for each task, generating training samples and adding them to the target detection dataset when target detection errors occur. Finally, it verifies the correctness of touchpoint planning by using a ROI point cloud generation model, a touchpoint planning model, command data, ROI image data, and touchpoint planning data for each task, generating training samples and adding them to the touchpoint planning dataset when touchpoint planning errors occur. These methods improve both the timeliness of anomaly detection and the efficiency and accuracy of training sample generation and annotation. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a method for extracting training data based on multimodal monitoring data of a robot, provided in Embodiment 1 of the present invention; Figure 2 This is a module structure diagram of a device for extracting training data based on multimodal monitoring data of a robot, provided in Embodiment 2 of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] Embodiment 1 of the present invention provides a method for extracting training data based on multimodal monitoring data of a robot, such as... Figure 1 The schematic diagram shows a method for extracting training data based on multimodal monitoring data of a robot, as provided in Embodiment 1 of the present invention. The method mainly includes the following steps: Step 1: The monitoring device loads the trained voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object retrieval and placement detection model, ROI point cloud generation model, and touch point planning model.

[0024] Here, the monitoring device in this embodiment of the invention can be a computer, an embedded AI computing platform, an edge computing device, a server, a high-performance industrial control computer, a workstation, a system, or a platform.

[0025] The speech activity detection model in this embodiment of the invention includes at least the Silero VAD model and the FunASR VAD model. The speech activity detection model is used to detect speech segments from the input audio and output corresponding speech segment detection sets. When the speech segment detection set is not empty, it consists of one or more speech segment detection data; the speech segment detection data includes the segment start time and the segment end time.

[0026] It should be noted that the voice activity detection model has been pre-trained using its publicly available general-purpose voice dataset and fine-tuned using real voice datasets from a single-arm robot's working environment. It can effectively filter out various noises in the working environment and accurately detect robot control voices in the working environment.

[0027] For example, the operating environment of a single-arm robot is a closed, automated laboratory. The noise generated in this laboratory includes the operator's local or remote control voice, environmental noise, and equipment noise from other devices. The single-arm robot is used to pick up designated experimental materials from a shelf and move them to a designated material container based on the operator's control voice. During a certain monitoring period T, operator A... 1s ,t 1e ]、[t 2s ,t 2e Given two control voice segments v1 and v2, the voice activity detection model can identify two voice segment detection data d1 and d2 from the monitored audio during the monitoring period T. The start / end times of segment d1 are the corresponding times t. 1s / t 1e The start and end times of segment d2 are the corresponding times t. 2s / t 2e Time t 1s / t 1e / t 2s / t 2e ∈T.

[0028] The automatic speech recognition model in this invention includes at least the Qwen-ASR series model and the Whisper series model. The automatic speech recognition model is used to perform text recognition on the input audio and output the corresponding audio text.

[0029] It should be noted that the automatic speech recognition model has been pre-trained using its publicly available general speech dataset and fine-tuned using real speech datasets from the single-arm robot's operating environment, enabling it to accurately recognize control speech in the operating environment.

[0030] For example, if operator A's two control voice messages v1 and v2 are "Put the apple material into the fruit container" and "Put the banana material into the fruit container" respectively, then the two audio texts txt1 and txt2 obtained after inputting v1 and v2 into the automatic speech recognition model are "Put the apple material into the fruit container" and "Put the banana material into the fruit container" respectively.

[0031] The instruction recognition language model in this invention is based on a class of generative large language models, including at least the Qwen series models, SmolVLA models, and DeepSeek series models. The instruction recognition language model is used to identify whether the source text carried in the recognition task instruction input to the model is a retrieval instruction text requiring the first retrieval of a specified item and its placement in a specified location. Based on the text recognition result, it identifies the retrieved item and its placement location in the source text and outputs the corresponding text recognition result, the retrieved item, and the placement location. Here, the text recognition result is either yes or no.

[0032] It should be noted that the instruction recognition language model has been pre-trained using its publicly available general text dataset and fine-tuned using a real control speech text dataset from a single-arm robot's working environment, enabling it to accurately recognize control speech text in the working environment.

[0033] For example, after inputting the audio text txt1 into the command recognition language model, the text recognition result is "yes", the item to be obtained is an apple, and the placement location is a fruit container; after inputting the audio text txt1 into the command recognition language model, the text recognition result is "yes", the item to be obtained is a banana, and the placement location is a fruit container.

[0034] It should be noted that the recognition task instructions in this embodiment of the invention are generated based on a preset recognition task instruction template.

[0035] It should be noted that the recognition task instruction template in this embodiment of the invention is a formatted text template, and the configurable parameters of the template include the source text. The recognition task instruction template is used to identify whether the source text is a retrieval instruction text that requires first obtaining a specified item and then placing it in a specified location, and to provide a specific text recognition result of yes or no. If the text recognition result is yes, then target recognition is performed on the item to be obtained and the placement location in the source text, and it is stated that the placement location is another target item or target area that can be used to place / load the obtained item. If the text recognition result is no, then the corresponding item to be obtained and the placement location are set to empty. The obtained text recognition result, item to be obtained, and placement location are output as the model processing result.

[0036] The robotic arm joint detection model in this invention is based on a class of visual target detection models, including at least the YOLO series models. The robotic arm joint detection model is used to detect robotic arm joints on the input image and output a corresponding set of joint detection boxes. The set of joint detection boxes includes multiple joint detection boxes. Each joint detection box includes the target joint type, the coordinates of the detection box center point, and the detection box size. The target joint types include six or seven robotic arm joints, with the total number of types matching the total number of robotic arm joints in a single-arm robot.

[0037] It should be noted that the robotic arm joint detection model has been pre-trained using its publicly available general image dataset and fine-tuned using a real image dataset of the single-arm robot's working environment. It can accurately detect all robotic arm joints (six or seven categories) of the single-arm robot in the working environment image.

[0038] The object detection model in this invention is based on a class of Open-Vocabulary Object Detection (OVOD) models, including at least the Grounding DINO Tiny model, the OV-DINO model, the YOLOE-v8s-seg model, and the YOLO-World v2 series models. The object detection model extracts the corresponding objects to be picked up and placed from the input text, and detects the objects corresponding to these objects in the input image, outputting the corresponding object detection boxes.

[0039] Here, in this embodiment of the invention, the object to be retrieved is a target object in the environment, and the object to be placed is a target loading object or target placement area in the environment used to place the object to be retrieved.

[0040] The object detection frame of this invention includes a first target type, coordinates of the center point of the first detection frame, and a first detection frame size; the first target type matches the item type of the object to be retrieved, and the first detection frame size includes height and width.

[0041] The object detection frame of this invention includes a second target type, coordinates of the center point of the second detection frame, and dimensions of the second detection frame; the second target type matches the item type or area type of the object to be placed, and the dimensions of the second detection frame include height and width.

[0042] It should be noted that the object retrieval and placement detection model has been pre-trained on its publicly available general text-image bimodal dataset and fine-tuned on a real text-image bimodal dataset in a single-arm robot's working environment. It can accurately identify objects to be retrieved / placed in the input text and can use the identified objects to cues for accurate object detection in the input image.

[0043] For example, if the model input text is "object to be picked up is an apple, object to be placed is a fruit container", and the model input image is a work environment image containing apples and fruit containers, then the model outputs the object picking / placement detection boxes as the target detection boxes corresponding to the apple / fruit container in the work environment image, respectively. The first target type of the object picking detection box is apple, and the center point coordinates and size of the first detection box are the center point coordinates and detection box size corresponding to the apple detection box in the work environment image, respectively. The second target type of the object placement detection box is fruit container, and the center point coordinates and size of the second detection box are the center point coordinates and detection box size corresponding to the fruit container detection box in the work environment image, respectively.

[0044] The Region of Interest (ROI) point cloud generation model in this embodiment of the invention consists of a semantic segmentation model and a point cloud transformation module. The semantic segmentation model is based on the SAM series of models.

[0045] The ROI point cloud generation model takes ROI type and RGB-D image as input and outputs the corresponding ROI point cloud. The ROI type represents an object type; the RGB-D image is an image with depth information captured by a depth camera.

[0046] The working mechanism of the ROI point cloud generation model is as follows: the semantic segmentation model is used to perform semantic segmentation on the RGB-D image input by the model to obtain the corresponding semantic segmentation image; the point cloud conversion module is used to extract the pixels whose semantic type matches the ROI type input by the model in the semantic segmentation image to form the corresponding target mask image, and perform color point cloud conversion on the target mask image to obtain the corresponding ROI point cloud and output it.

[0047] It should be noted that the semantic segmentation model of the ROI point cloud generation model has been pre-trained using its publicly available RGB-D image segmentation dataset and has been fine-tuned using a real RGB-D image segmentation dataset from a single-arm robot's operating environment, enabling accurate segmentation of the RGB-D image input to the model.

[0048] It should be noted that the point cloud conversion module of the ROI point cloud generation model is a fixed processing module that does not require specific training. The pixels in the target mask generated by the point cloud conversion module have two semantic categories: foreground pixels and background pixels. Foreground pixels are those that match the ROI type, and their pixel features are consistent with the corresponding pixel features in the RGB-D image. When performing color point cloud conversion on the target mask, the point cloud conversion module processes it using the conventional depth map-to-color point cloud conversion method: based on known camera intrinsics and extrinsic parameters, it performs a world coordinate system transformation on the coordinates of the foreground pixels in the target mask to obtain corresponding spatial points; it then sets the RGB features of the corresponding spatial points based on the RGB features of each foreground pixel; finally, it combines all the converted spatial points to form a color point cloud, i.e., the ROI point cloud, and outputs it. The camera intrinsics and extrinsic parameters mentioned here refer to the camera intrinsics and extrinsic parameters of the observation camera mentioned below.

[0049] The touch planning model in this embodiment of the invention consists of a DexVLG model and a touch calculation module.

[0050] The input to the touch point planning model includes the ROI point cloud and prompt text, and the output is the five-finger touch point positions. The ROI point cloud is a 3D colored point cloud of the item to be grasped; the prompt text indicates the corresponding item in the ROI point cloud to be grasped; the five-finger touch point positions include the thumb, index finger, middle finger, ring finger, and little finger touch points.

[0051] The working mechanism of the touch point planning model is as follows: The DexVLG model is used to plan the grasping posture of the robot's dexterous hand that grasps the current object based on the prompt text and ROI point cloud, and output the corresponding wrist joint coordinates, wrist orientation, and multiple finger joint angles; The touch point calculation module is used to calculate the corresponding five-finger touch point positions based on the wrist joint coordinates, wrist orientation, and all finger joint angles of the dexterous hand according to the forward kinematics (FK) calculation method of Shadow Hand, and output them.

[0052] It should be noted that the DexVLG model for touch point planning has been pre-trained using its publicly available grasping posture dataset (DexGraspNet 3.0) and fine-tuned using a grasping posture dataset for large-scale training of single-arm robots. It can accurately plan the robot's dexterous hand's grasping posture (wrist joint coordinates, wrist orientation, and multiple finger joint angles) based on the input prompt text and ROI point cloud. Here, the grasping posture dataset for large-scale training of single-arm robots uses the real-world operating environment of the single-arm robot as the environmental benchmark, and is generated by collecting human operation actions and / or simulating robot operation actions.

[0053] It should be noted that the touch point calculation module of the touch point planning model is a fixed processing module that does not require specific training. The forward kinematics calculation method of Shadow Hand used by the touch point calculation module is a publicly available mathematical calculation process, which is essentially a chain product calculation process based on homogeneous transformations of DH parameters (Denavit-Hartenberg parameters). Simply put, after knowing the wrist joint coordinates, wrist orientation, and the angles of all finger joints, starting from the wrist joint coordinates, the DH transformation matrix is ​​applied sequentially to each finger joint on the wrist-to-finger joint path from the wrist joint to the fingertip, and the estimated fingertip position is obtained by chain multiplication. Then, based on the wrist orientation, each estimated fingertip position is rotated and translated to obtain the final fingertip position.

[0054] Step 2: Connect to the monitored single-arm robot and connect to the observation microphone and observation camera installed in the robot's operating environment; and set up a robot storage area, an observation storage area, and a dataset storage area locally on the device.

[0055] Here, the monitoring device of this embodiment of the invention is connected to the single-arm robot, the observation microphone, and the observation camera respectively via wired or wireless communication.

[0056] The single-arm robot of this embodiment has a fixed base and its actuator includes an articulated robotic arm and a five-fingered dexterous hand.

[0057] The articulated robotic arm in this embodiment of the invention is a six-axis or seven-axis robotic arm, corresponding to six or seven robotic arm joints. Specific types include at least the KUKA robotic arm and the UR robotic arm. The base joint of the articulated robotic arm is connected to a fixed base, and the end effector joint is connected to a five-fingered dexterous hand.

[0058] The five-finger dexterous hand in this embodiment of the invention is a Shadow Hand dexterous hand, which is connected to the end joint of an articulated robotic arm via a wrist joint. A palm camera and a tactile sensor are installed in the palm of the five-finger dexterous hand, and a tactile sensor is installed at each fingertip.

[0059] The tactile sensor of this invention is used to sense multidimensional forces and coefficients of friction on the palm / fingertip contact surface of an object. The multidimensional forces of this invention include multiple sensed force data; the sensed force data includes force type and three-dimensional force components.

[0060] The force types include at least contact force, contact surface pressure, normal force, and shear force; the three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system. The friction coefficient µ is estimated based on the normal force and shear force, i.e.: .

[0061] The sensors in a single-arm robot also include an audio receiver and a rotating camera. The audio receiver's audio reception range and the rotating camera's rotational field of view can both provide full coverage of the robot's workspace; the rotating camera is a depth camera.

[0062] When the single-arm robot is not performing a task based on voice commands, the positions of all joints of the articulated robotic arm and the five-fingered dexterous hand remain consistent with the preset initial positions of the robot joints. Here, the initial positions of the robot joints in this embodiment of the invention include multiple initial positions of the robotic arm or hand joints, each of which is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body.

[0063] A single-arm robot is used to retrieve a specified object from the work environment based on received voice commands and move it to the specified location. Specifically: Step A1: Perform instruction conversion on the real-time environmental audio received by the audio receiver, and identify whether the conversion instruction is a valid task instruction.

[0064] Step A2: After identifying a valid task instruction, extract the corresponding object to be retrieved and the object to be placed from the current task instruction; and store the corresponding instruction data, consisting of the current task instruction and its corresponding instruction timestamp, into the task data area.

[0065] Step A3 involves using a rotating camera to capture a panoramic view of the robot's working environment, obtaining an image of the working environment, identifying the corresponding objects to be picked up and placed, recognizing the target positions corresponding to the objects to be picked up and placed, and planning the palm target positions near the object to be picked up. The environmental image data, consisting of the working environment image and its corresponding image timestamp, is stored in the task data area. The corresponding object to be picked up / placed target recognition data, consisting of the object to be picked up / placed target recognition box, object to be picked up / placed target position, and object to be picked up / placed recognition timestamp, is also stored in the task data area.

[0066] Step A4, with the constraint that the robotic arm and dexterous hand do not collide with any object during the movement, performs real-time motion planning and drive control on all joints of the robotic arm and dexterous hand, so that the palm of the five-fingered dexterous hand moves to the target position of the palm.

[0067] Step A5: After reaching the target position in the palm, the palm camera is used to capture a local environment of interest image centered on the object to be picked up, thus obtaining the ROI image. The ROI image data, consisting of the ROI image and its corresponding image timestamp, is stored in the task data area.

[0068] Step A6 involves planning the initial positions and initial torques of the five fingers based on the object ROI image; and performing motion planning and drive control on all joints of the dexterous hand according to the initial positions of the five fingers, so that the center positions of the fingertips of the dexterous hand reach their respective initial positions and make pre-contact with the object; and storing the contact planning data composed of the initial positions of the five fingers and the corresponding final planning timestamps into the task data area.

[0069] Here, the initial torque of the five fingers in this embodiment of the invention is composed of multiple finger joint torques. The finger joint torque is a six-dimensional torque in the three-dimensional coordinate system of the robot body XYZ, which is composed of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes.

[0070] Step A7: After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers as the initial torque value. During the operation, the tactile perception data fed back by all tactile sensors in real time is used to confirm whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all real-time feedback tactile perception data with the constraint of not causing damage to the object, and the five-finger dexterity hand is driven to continue grasping the object with the current optimized five-finger torque.

[0071] Here, the optimized torque of the five fingers in this embodiment of the invention is composed of the torques of multiple corresponding finger joints.

[0072] Step A8: After confirming that the object has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target position and the shape and material characteristics of the object. With the constraint that the robotic arm and the dexterous hand do not collide with any object during the movement and that the object held by the dexterous hand will not loosen and fall, the motion planning and drive control of all joints of the robotic arm and the dexterous hand are performed in real time, so that the center point of the object is moved to the release center position.

[0073] Step A9, and after reaching the release center position, perform the object release operation by driving the hand joints of the five-finger dexterous hand; and after successfully releasing the object, drive all joints of the jointed robotic arm and the five-finger dexterous hand to return to their original positions based on the initial positions of the robot joints.

[0074] The single-arm robot has a locally configured task data area to store multimodal process data generated during its operation. This multimodal process data includes multiple instruction data, multiple environmental image data, multiple object pickup target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple contact point planning data.

[0075] The instruction data in this embodiment of the invention includes task instructions and instruction timestamps; wherein, the task instructions include objects to be retrieved and objects to be placed.

[0076] The environmental image data in this embodiment of the invention includes an image of the work environment and an image timestamp.

[0077] The object retrieval target recognition data in this embodiment of the invention includes an object retrieval target recognition box, an object retrieval target location, and an object retrieval recognition timestamp. The object retrieval target recognition box includes the object retrieval target type, the coordinates of the center point of the retrieval box, and the dimensions of the retrieval box; the object retrieval target type matches the item type of the corresponding object being retrieved.

[0078] The object recognition data in this embodiment of the invention includes an object recognition frame, an object location, and an object recognition timestamp. The object recognition frame includes the object type, coordinates of the frame's center point, and frame dimensions; the object type matches the item type or region type of the corresponding object.

[0079] The ROI image data in this embodiment of the invention includes the object-collected ROI image and the image timestamp.

[0080] The touch point planning data in this embodiment of the invention includes the initial positions of the five fingers and a planning timestamp. The initial positions of the five fingers include the initial position of the thumb, index finger, middle finger, ring finger, and little finger, and some or all of these initial positions are in contact with the surface of the object to be picked up.

[0081] In this embodiment of the invention, both the audio reception range of the observation microphone and the shooting angle range of the observation camera cover the robot's working space, enabling the observation camera to capture unobstructed images of the robot's object-retrieving process. The camera type of the observation camera in this embodiment of the invention is a depth camera.

[0082] In this embodiment of the invention, the local dataset storage area of ​​the monitoring device is used to store three types of training datasets: an audio recognition dataset, an object detection dataset, and a touch point planning dataset. Specifically: the audio recognition dataset includes multiple first training data sets; these first training data sets include training audio and instruction labels. The object detection dataset includes multiple second training data sets; these second training data sets include training environment images, object retrieval box labels, and object placement box labels. The touch point planning dataset includes multiple third training data sets; these third training data sets include training ROI images and touch point labels.

[0083] Step 3: During the preset monitoring period, start the single-arm robot to perform the task of picking up objects according to the environmental voice command in its working environment, call the observation microphone to receive and record the real-time audio of the robot's working environment, and call the observation camera to shoot and record the robot's working process, and generate corresponding audio and video recording files in the observation storage area to continuously store the recording data of the microphone and camera; at the end of the monitoring period, stop the audio and video recording of the microphone and camera, and retrieve the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and store it in the robot storage area.

[0084] Here, the monitoring period in this embodiment of the invention is a pre-set time period. This time period can be a long period, such as one week or one month; or it can be one or more consecutive periods within a day. The specific setting method can be set based on actual application needs.

[0085] The multimodal dataset in this embodiment of the invention includes a command sequence, an environmental image sequence, a target retrieval sequence, a target placement sequence, a region of interest (ROI) image sequence, and a touch point planning sequence. Each of these sequences comprises multiple sets of command data, environmental image data, target retrieval identification data, target placement identification data, ROI image data, or touch point planning data that occur within the monitoring period and correspond to the current sequence.

[0086] In step 3, the multimodal dataset generated during the monitoring period is obtained from the task data area of ​​the single-arm robot and stored in the robot's storage area. Specifically, the instruction data with the instruction timestamp during the monitoring period is extracted from the task data area to form the corresponding instruction sequence; the environmental image data with the image timestamp during the monitoring period is extracted to form the corresponding environmental image sequence; the object retrieval recognition data with the object retrieval recognition timestamp during the monitoring period is extracted to form the corresponding object retrieval sequence; the object placement recognition data with the object placement recognition timestamp during the monitoring period is extracted to form the corresponding object placement sequence; the ROI image data with the image timestamp during the monitoring period is extracted to form the corresponding ROI image sequence; and the touch point planning data with the planning timestamp during the monitoring period is extracted to form the corresponding touch point planning sequence. The obtained instruction sequence, environmental image sequence, object retrieval target sequence, object placement target sequence, ROI image sequence, and touch point planning sequence are combined to form the corresponding multimodal dataset and stored in the robot's storage area.

[0087] Step 4: Using the voice activity detection model, automatic speech recognition model, command recognition language model, and robotic arm joint detection model, the command audio, command information, and task time period of the robot's single task during the monitoring period are identified based on the audio and video recording files to obtain the task identification set.

[0088] The task identification set includes multiple task information items; the task information includes instruction audio, instruction information, and task time period. Instruction information includes the object to be retrieved and the object to be placed. The task time period includes the task start time and the task end time.

[0089] Specifically, it includes: Step 41: The audio recording file is used as the model input. The audio input speech activity detection model is used to detect speech segments to obtain the corresponding speech segment detection set. Based on the speech segment detection data of each speech segment in the speech segment detection set, the corresponding speech segments are extracted from the audio recording file. The extracted speech segments are used as the model input. The audio input automatic speech recognition model is used to perform text recognition to obtain the corresponding audio text. The audio text is used as the corresponding source text input. The recognition task instruction template is used to generate the corresponding recognition task instruction. The recognition task instruction is input into the instruction recognition language model for processing to obtain the corresponding text recognition results, item acquisition, and placement location.

[0090] Step 42, and take each specific text recognition result as the current result; take the item to be retrieved and the placement location corresponding to the current result as a set of corresponding retrieval objects and placement objects to form the corresponding instruction information, and take the corresponding audio segment of the current result and the corresponding segment start time as the corresponding instruction audio and task start time.

[0091] Step 43, and set the start time of each task as the current time t.s And set the corresponding time t based on the preset maximum task duration L1. e =t s +L1; and will add the time segment [t] to the video recording file. s ,t e The video clip is extracted as the current video; the current video is then processed into frames; all frames are traversed in chronological order; during this traversal, the currently traversed image is used as the current image; it is determined whether the current image is the first frame; if so, the previous frame is set to empty; otherwise, the previous frame is used as the previous frame. The current image is then input into the robotic arm joint detection model to detect robotic arm joint targets and obtain the corresponding set of joint detection boxes; based on the camera's intrinsic and extrinsic parameters, the coordinates of the center points of each detection box in the set of joint detection boxes are transformed into 3D world coordinates to obtain the corresponding joint center point coordinates; the 3D coordinate difference between every two joint center point coordinates is calculated to obtain the corresponding first coordinate difference; and the initial positions of every two robotic arm joints in the preset initial robot joint positions are calculated. The second coordinate difference is calculated using the 3D coordinate difference; and the first and second coordinate differences with the same joint relative relationship are used to form corresponding difference pairs; the first and second coordinate differences of each difference pair are calculated again to obtain the corresponding third coordinate difference; it is then identified whether each third coordinate difference meets the preset 3D coordinate difference range. If it does, the corresponding matching result is set as a match; otherwise, the matching result is set as a mismatch; it is then identified whether all matching results corresponding to the current image are matches. If they are, the current image is marked as a repositioned image; otherwise, the current image is marked as a non-repositioned image; when the current image is a repositioned image and the previous frame image is a non-repositioned image, the current round of traversal is terminated, and the image time of the current image is taken as the corresponding task end time; if the corresponding task end time is not obtained until the last frame of the current round of traversal, then the current time t is used as the end time. e Set the corresponding task end time.

[0092] The initial position of the robot joint includes the initial positions of multiple robotic arm or hand joints, each of which is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body.

[0093] In this embodiment of the invention, the maximum task duration L1 is a pre-set time length parameter. The three-dimensional coordinate difference range consists of the difference ranges in the X, Y, and Z dimensions. If the difference in each dimension of the third coordinate difference satisfies the corresponding difference range, then the current third coordinate difference satisfies the three-dimensional coordinate difference range. If at least one dimension of the third coordinate difference does not satisfy the corresponding difference range, then the current third coordinate difference does not satisfy the three-dimensional coordinate difference range.

[0094] Step 44, and the corresponding task time period is composed of the start time of each task and its corresponding end time; the corresponding task information is composed of the corresponding task time period and its corresponding instruction audio and instruction information; and the corresponding task identification set is composed of all the obtained task information.

[0095] Step 5: Based on each task time period, the multimodal dataset is segmented to obtain the corresponding task data subset.

[0096] Specifically, this includes: taking each task time period of the task recognition set as the current task time period; and taking the latest time among all instruction data in the instruction sequence of the multimodal dataset that satisfies the current task time period as the current instruction data, taking the latest time among all environmental image data in the environmental image sequence that satisfies the current task time period as the current environmental image data, taking the latest time among all object retrieval target recognition data in the object retrieval target sequence that satisfies the current ...

[0097] Here, the task data subset in this embodiment of the invention is a subset of the multimodal dataset in the corresponding task time period, which consists of a set of corresponding instruction data, environmental image data, object retrieval target recognition data, object placement target recognition data, ROI image data and touch point planning data.

[0098] Step 6: Verify the correctness of instruction recognition based on the instruction data of each instruction information and its corresponding task data subset, and generate training samples to be added to the audio recognition dataset when instruction recognition is incorrect.

[0099] Specifically, this includes: taking each instruction information in the task recognition set and its corresponding task data subset as the current instruction 1 and the current subset; taking the instruction data of the current subset as the current instruction 2; identifying whether the objects to be picked up and placed in the current instructions 1 and 2 match, and if so, setting the corresponding current verification result as instruction recognition correct, otherwise setting the corresponding current verification result as instruction recognition error; and when the current verification result is instruction recognition error, taking the instruction audio corresponding to the current instruction 1 as the corresponding training audio, and taking the current instruction 1 as the corresponding instruction label, and adding the obtained training audio and instruction label to the corresponding first training data to the audio recognition dataset.

[0100] Step 7: Verify the correctness of target detection based on the object retrieval and placement detection model and the instruction data, environmental image data, object retrieval target recognition data and object placement target recognition data of each task data subset, and generate training samples to be added to the target detection dataset when the target detection is incorrect.

[0101] Specifically, it includes: Step 71: Take each task data subset as the current subset; and extract the corresponding instruction data, environmental image data, object retrieval target recognition data, and object placement target recognition data from the current subset.

[0102] Step 72: Extract the object to be picked and the object to be placed from the instruction data to form the corresponding current model input text; use the working environment image from the environmental image data as the current model input image; and input the current model input text and the current model input image into the object picking and placing detection model for processing to obtain the corresponding object picking detection box and object placing detection box.

[0103] Step 73: Combine the object detection box and the object target recognition box from the object detection data into a first box group, and combine the object placement detection box and the object placement recognition box from the object target recognition data into a second box group; set either the first or second box group as the current box group; and identify whether the target types of the two boxes in the current box group match; if not, set the corresponding box group verification result as failed; if yes, calculate the straight-line distance between the coordinates of the two center points of the current box group and the height-width difference between the two boxes to obtain the corresponding current center distance, current height difference, and current width difference, and then set the current center distance. When the spacing meets the preset center distance range, the current height difference meets the preset height difference range, and the current width difference meets the preset width difference range, the corresponding frame group verification result is set to successful. When the current center spacing does not meet the center distance range, or the current height difference does not meet the height difference range, or the current width difference does not meet the width difference range, the corresponding frame group verification result is set to failed. The system also identifies whether both frame group verification results are successful. If so, the corresponding current verification result is set to target detection correct; otherwise, the corresponding current verification result is set to target detection error.

[0104] Here, the center distance range, height difference range, and width difference range in this embodiment of the invention are three preset numerical ranges.

[0105] Step 74: When the current verification result is an object detection error, the current working environment image, object detection box, and object placement detection box are used as a set of corresponding training environment images, object box labels, and object placement box labels to form a corresponding second training data, and the current second training data is added to the object detection dataset.

[0106] Step 8: Verify the correctness of touch point planning based on the ROI point cloud generation model, touch point planning model, instruction data, ROI image data, and touch point planning data of each task data subset, and generate training samples to be added to the touch point planning dataset when touch point planning is incorrect.

[0107] Specifically, it includes: Step 81: Take each task data subset as the current subset; and extract the corresponding instruction data, ROI image data and touch point planning data from the current subset.

[0108] Step 82: Based on the object being retrieved according to the instruction data, set the corresponding ROI type and use the retrieved ROI image data as the corresponding RGB-D image; input the current ROI type and RGB-D image into the ROI point cloud generation model for processing to obtain the corresponding ROI point cloud; set the corresponding prompt text based on the object being retrieved according to the preset DexVLG model prompt text setting rules; and input the current ROI point cloud and prompt text into the touch point planning model for processing to obtain the corresponding five-finger touch point position.

[0109] Here, according to the DexVLG model paper "DexVLG: Dexterous Vision-Language-Grasp Model at Scale," the paper provides a simple format specification for language instructions, i.e., prompt text (i.e., DexVLG model prompt text setting rules): it is recommended that the text, in addition to the grasping target, should ideally include two types of key information: target component and grasping style. The target component refers to the specific grasping location of the target; for example, if the target is a cup, then the cup handle is its target component. The grasping style refers to the grasping posture, such as picking, pinching, holding, or lifting. Both of these key pieces of information (target component and grasping style) can be empty simultaneously or partially. Even if neither is set and the prompt text only contains the grasping target, the DexVLG model can still output the corresponding planning results.

[0110] Step 83: Unify all position coordinates of the five-finger contact point positions and the initial positions of the five fingers in the contact point planning data to the same three-dimensional coordinate system, which can be the world coordinate system or the robot's XYZ three-dimensional coordinate system; and record the five contact point positions of the five fingers as the corresponding positions. The five initial contact points of the five fingers are sequentially recorded as the corresponding positions. 1≤indexi≤5; and calculate the geometric position deviation e between the five-finger contact point position and the initial position of the five fingers.

[0111] in, .

[0112] Step 84, and identify whether the geometric position deviation e is less than the preset geometric position deviation threshold; if yes, set the corresponding current verification result as correct contact planning; if no, set the corresponding current verification result as incorrect contact planning.

[0113] Here, the geometric position deviation threshold in this embodiment of the invention is a pre-set threshold parameter.

[0114] Step 85: If the current verification result is a touch point planning error, the current ROI image data and the five finger touch point positions are used as a set of corresponding training ROI images and touch point labels to form a corresponding third training data; and the current third training data is added to the touch point planning dataset.

[0115] Figure 2 This is a module structure diagram of a device for extracting training data based on multimodal monitoring data of a robot, provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiment, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a model loading module 201, a monitoring preparation module 202, a monitoring acquisition module 203, a task segment identification module 204, a monitoring data segmentation module 205, an instruction recognition training data extraction module 206, a target detection training data extraction module 207, and a touch point planning training data extraction module 208.

[0116] The model loading module 201 is used to load the trained speech activity detection model, automatic speech recognition model, instruction recognition language model, robotic arm joint detection model, object picking and placing detection model, ROI point cloud generation model, and touch point planning model.

[0117] The monitoring preparation module 202 is used to connect to the monitored single-arm robot and to the observation microphone and observation camera installed in the robot's working environment. It also locally sets up a robot storage area, an observation storage area, and a dataset storage area. The single-arm robot is used to retrieve a specified object from the working environment based on received voice commands and move it to the specified location. The single-arm robot also sets up a task data area locally to store multimodal process data generated during its operation. This multimodal process data includes multiple command data, multiple environmental image data, multiple object retrieval target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple touchpoint planning data. The audio reception range of the observation microphone and the shooting angle range of the observation camera both cover the robot's working space, and the observation camera can capture unobstructed images of the robot's object retrieval process. The observation camera is a depth camera. The dataset storage area stores the audio recognition dataset, the target detection dataset, and the touchpoint planning dataset.

[0118] The monitoring and acquisition module 203 is used to start the single-arm robot to perform the task of picking up objects according to the environmental voice command in its working environment during the preset monitoring period, and call the observation microphone to receive and record the real-time audio of the robot's working environment, and call the observation camera to shoot and record the robot's working process, and generate corresponding audio and video recording files in the observation storage area to continuously store the recording data of the microphone and camera; and at the end of the monitoring period, stop the audio and video recording of the microphone and camera, and obtain the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and store it in the robot storage area.

[0119] The task segment recognition module 204 is used to identify the command audio, command information and task time period of a single robot task within the monitoring period by using a voice activity detection model, an automatic speech recognition model, a command recognition language model and a robotic arm joint detection model, based on audio recording files and video recording files. The task recognition set includes multiple task information items. The task information includes command audio, command information and task time period.

[0120] The monitoring data segmentation module 205 segments the multimodal dataset based on each task time period to obtain the corresponding task data subset.

[0121] The instruction recognition training data extraction module 206 verifies the correctness of instruction recognition based on the instruction data of each instruction information and its corresponding task data subset, and generates training samples to be added to the audio recognition dataset when the instruction recognition is incorrect.

[0122] The target detection training data extraction module 207 verifies the correctness of target detection based on the object retrieval and placement detection model and the instruction data, environmental image data, object retrieval target recognition data and object placement target recognition data of each task data subset, and generates training samples to be added to the target detection dataset when the target detection is incorrect.

[0123] The touch point planning training data extraction module 208 verifies the correctness of touch point planning based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset. When the touch point planning is incorrect, it generates training samples and adds them to the touch point planning dataset.

[0124] The single-arm robot has a fixed base. Its actuators include an articulated robotic arm and a five-finger dexterous hand. The articulated robotic arm is a six- or seven-axis robotic arm, corresponding to six or seven joints; specific types include at least the KUKA and UR robotic arms. The base joint of the articulated robotic arm is connected to the fixed base, and its end effector joints are connected to the five-finger dexterous hand. The five-finger dexterous hand is a Shadow Hand type, connected to the end effector joint of the articulated robotic arm via its wrist joint. A palm camera and tactile sensor are installed in the palm of the five-finger dexterous hand, and a tactile sensor is installed at each fingertip. The tactile sensor is used to sense multidimensional forces and the coefficient of friction on the palm / fingertip contact surface of the object. The multidimensional forces include multiple force data points, which include force types and three-dimensional force components. Force types include at least contact force, contact surface pressure, normal force, and shear force. The three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system. The coefficient of friction is estimated based on the normal force and shear force. The single-arm robot's sensors also include an audio receiver and a rotating camera. The audio receiver's audio reception range and the rotating camera's rotational viewing angle range can fully cover the robot's workspace. The rotating camera is a depth camera.

[0125] When the single-arm robot is not performing a task based on voice commands, the positions of all joints of the control joint robotic arm and the five-fingered dexterous hand are consistent with the preset initial positions of the robot joints. The initial positions of the robot joints include the initial positions of multiple robotic arm or hand joints, and each initial position is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body.

[0126] A single-arm robot is specifically used to retrieve an object specified by a command from the work environment based on received voice instructions and move it to the specified location for placement. The system converts real-time ambient audio received by the audio receiver into commands and identifies whether the converted commands are valid task commands. After identifying valid task commands, the system extracts the corresponding object to be picked up and the object to be placed from the current task command. The system then stores the corresponding command data, consisting of the current task command and its corresponding command timestamp, into the task data area. The system uses a rotating camera to capture a panoramic view of the robot's working environment, obtaining images of the working environment. It then identifies the objects to be picked up and placed, determines their locations, and plans the palm positions near the object's location. The environmental image data, consisting of the working environment image and its corresponding timestamp, is stored in the task data area. Similarly, the object recognition data, consisting of the object recognition bounding box, object location, and object recognition timestamp, is also stored in the task data area. The system constrains the robotic arm and dexterous hand to avoid collisions with any objects during movement. It performs real-time motion planning and drive control on all joints of the robotic arm and dexterous hand, thereby moving the palm of the five-fingered dexterous hand to the target position. After reaching the target position, it uses the palm camera to capture a local environment of interest image centered on the object to be picked up, thus obtaining a ROI image. The ROI image data, consisting of the ROI image and its corresponding image timestamp, is stored in the task data area. The system plans the initial positions and initial torques of the five fingers based on the object ROI image. It then plans and controls the motion of all joints of the dexterous hand according to the initial positions of the five fingers, ensuring that the center positions of the fingertips reach their respective initial positions and make pre-contact with the object. The contact planning data, composed of the initial positions of the five fingers and their corresponding final planning timestamps, is stored in the task data area. The initial torque of the five fingers is composed of multiple finger joint torques, which are six-dimensional torques in the robot's XYZ three-dimensional coordinate system, consisting of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes. After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers. During the operation, the tactile perception data fed back by all tactile sensors in real time confirms whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all real-time feedback tactile perception data, with the constraint of not causing damage to the object, and the five-finger dexterity hand is driven to continue grasping the object using the current optimized five-finger torque. The optimized five-finger torque is composed of the torque of the corresponding multi-finger joints. After confirming that the object has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target location and the shape and material characteristics of the object. Constrained by the principle that the robotic arm and dexterous hand will not collide with any object during movement and that the object held by the dexterous hand will not loosen or fall, real-time motion planning and drive control are performed on all joints of the robotic arm and dexterous hand to move the center point of the object to the release center position. Upon reaching the release center position, the object is released by driving the joints of the five-fingered dexterous hand. After successfully releasing the object, all joints of the robotic arm and the five-fingered dexterous hand are driven back to their original positions based on the initial positions of the robot joints.

[0127] The present invention provides an apparatus for extracting training data based on multimodal monitoring data of a robot, which can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0128] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the monitoring and acquisition module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0129] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0130] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0131] Figure 3 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 3 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0132] exist Figure 3The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.

[0133] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0134] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0135] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for training data extraction based on multimodal monitoring data of a robot. As described above, this invention introduces seven types of models (voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object retrieval and placement detection model, ROI point cloud generation model, and touch point planning model) to automatically verify the multimodal process data during the operation of a single-arm robot. Microphones and cameras are installed in the robot's operating environment for observation. During the monitoring period, the observation microphones / cameras are used to receive, record, and capture real-time audio / video recordings of the robot's operating environment and process, generating corresponding audio / video recording files. After the monitoring period ends, the multimodal dataset generated during the monitoring period is obtained from the single-arm robot. The voice activity detection model, automatic speech recognition model, command recognition language model, and robotic arm joint detection model are used to analyze the training data based on the audio and video recording files. The invention employs several methods to verify the correctness of robot task recognition. First, it identifies the audio and information of robot commands for a single task within a given time period, along with the task time period itself. Then, it segments the multimodal dataset into corresponding task data subsets based on the task time period. Second, it verifies the correctness of command recognition by matching command information with corresponding command data, generating training samples and adding them to the audio recognition dataset when command recognition errors occur. Third, it verifies the correctness of target detection by using a retrieval and placement detection model, along with command data, environmental image data, retrieval target recognition data, and placement target recognition data for each task, generating training samples and adding them to the target detection dataset when target detection errors occur. Finally, it verifies the correctness of touchpoint planning by using a ROI point cloud generation model, a touchpoint planning model, command data, ROI image data, and touchpoint planning data for each task, generating training samples and adding them to the touchpoint planning dataset when touchpoint planning errors occur. These methods improve both the timeliness of anomaly detection and the efficiency and accuracy of training sample generation and annotation.

[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0137] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting training data based on multimodal monitoring data of a robot, characterized in that, The method includes: The monitoring device is loaded with trained voice activity detection model, automatic speech recognition model, command recognition language model, robotic arm joint detection model, object picking and placing detection model, ROI point cloud generation model, and touch point planning model; The device connects to the monitored single-arm robot and is also connected to an observation microphone and an observation camera installed in the robot's operating environment. It locally stores a robot storage area, an observation storage area, and a dataset storage area. The single-arm robot retrieves a specified object from the operating environment based on received voice commands and moves it to the specified location. The single-arm robot also has a locally configured task data area for storing multimodal process data generated during its operation. This multimodal process data includes multiple command data, multiple environmental image data, multiple object retrieval target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple touchpoint planning data. The audio reception range of the observation microphone and the shooting angle range of the observation camera both cover the robot's operating space, and the observation camera can capture unobstructed images of the robot's object retrieval process. The observation camera is a depth camera. The dataset storage area stores audio recognition datasets, target detection datasets, and touchpoint planning datasets. During the preset monitoring period, the single-arm robot is activated to perform object retrieval tasks according to environmental voice commands in its working environment. The observation microphone is used to receive and record real-time audio of the robot's working environment, and the observation camera is used to capture and record the robot's operation process. Corresponding audio and video recording files are generated in the observation storage area, and the recording data from the microphone and camera is continuously stored. At the end of the monitoring period, the audio and video recording from the microphone and camera is stopped, and the multimodal dataset generated during the monitoring period is obtained from the task data area of ​​the single-arm robot and stored in the robot storage area. Using the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model, a task identification set is obtained by identifying the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files; the task identification set includes multiple task information items; the task information includes the command audio, the command information, and the task time period; The multimodal dataset is segmented based on each of the task time periods to obtain corresponding task data subsets; The correctness of instruction recognition is verified based on the instruction data of each instruction information and its corresponding task data subset, and training samples are generated and added to the audio recognition dataset when the instruction recognition is incorrect. Based on the object detection model and the instruction data, environmental image data, object detection target recognition data, and object placement target recognition data of each task data subset, the object detection correctness is verified, and training samples are generated and added to the object detection dataset when the object detection is incorrect. The correctness of touch point planning is verified based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset. If the touch point planning is incorrect, training samples are generated and added to the touch point planning dataset.

2. The method for extracting training data based on multimodal monitoring data of a robot according to claim 1, characterized in that, The speech activity detection model includes at least the Silero VAD model and the FunASR VAD model; the speech activity detection model is used to detect speech segments on the input audio and output the corresponding speech segment detection set; when the speech segment detection set is not empty, it consists of one or more speech segment detection data; the speech segment detection data includes the segment start time and the segment end time. The automatic speech recognition model includes at least the Qwen-ASR series model and the Whisper series model; the automatic speech recognition model is used to perform text recognition on the input audio and output the corresponding audio text. The instruction recognition language model is implemented based on a class of generative large language models, including at least the Qwen series models, the SmolVLA model, and the DeepSeek series models. The instruction recognition language model is used to identify whether the source text carried in the recognition task instruction input to the model is a retrieval instruction text that requires first obtaining a specified item and then placing it in a specified location. Based on the text recognition result, the model identifies the obtained item and the placement location in the source text and outputs the corresponding text recognition result, the obtained item, and the placement location. The identification task instructions are generated based on a preset identification task instruction template; The recognition task instruction template is a formatted text template, and the configurable parameters of the template include the source text. The recognition task instruction template is used to identify whether the source text is a retrieval instruction text that requires first obtaining a specified item and then placing it in a specified location, and to provide a specific yes or no text recognition result. If the text recognition result is yes, then target recognition is performed on the obtained item and the placement location in the source text, and it is stated that the placement location is another target item or target area that can be used to place / load the obtained item. If the text recognition result is no, then the corresponding obtained item and placement location are set to empty. The obtained text recognition result, the obtained item, and the placement location are output as the model processing result. The robotic arm joint detection model is based on a type of visual target detection model, including at least the YOLO series models. The model is used to detect robotic arm joints on the input image and output a corresponding set of joint detection boxes. The set of joint detection boxes includes multiple joint detection boxes. Each joint detection box includes the target joint type, the coordinates of the detection box center point, and the detection box size. The target joint types include six or seven robotic arm joints, and the total number of types matches the total number of robotic arm joints in the single-arm robot. The object detection model is based on a class of open vocabulary target detection models, including at least the Grounding DINOTiny model, the OV-DINO model, the YOLOE-v8s-seg model, and the YOLO-World v2 series models. The model extracts the corresponding object to be picked up and the object to be placed from the input text, and detects the objects on the input image that correspond to the object to be picked up and the object to be placed, respectively, and outputs corresponding object detection boxes and object detection boxes. The object detection box includes a first target type, the coordinates of the center point of the first detection box, and the size of the first detection box; the first target type matches the item type of the object to be picked up. The object detection box includes a second target type, the coordinates of the center point of the second detection box, and the size of the second detection box; the second target type matches the item type or region type of the object to be placed. The ROI point cloud generation model consists of a semantic segmentation model and a point cloud conversion module. The semantic segmentation model is based on the SAM series models. The model input of the ROI point cloud generation model includes the ROI type and an RGB-D image, and the model output is the corresponding ROI point cloud. The ROI type is a type of object, and the RGB-D image is an image with depth information captured by a depth camera. The semantic segmentation model is used to perform semantic segmentation on the RGB-D image input to the model to obtain the corresponding semantic segmentation image. The point cloud conversion module is used to extract the pixels in the semantic segmentation image whose semantic type matches the ROI type input to the model to form the corresponding target mask image, and to perform color point cloud conversion on the target mask image to obtain the corresponding ROI point cloud and output it. The touch point planning model consists of a DexVLG model and a touch point calculation module. The model inputs include the ROI point cloud and prompt text, and the model output is the five-finger touch point positions. The ROI point cloud is a 3D color point cloud of the object to be grasped. The prompt text indicates that the corresponding object in the ROI point cloud should be grasped. The five-finger touch point positions include the thumb, index finger, middle finger, ring finger, and little finger touch points. The DexVLG model plans the grasping posture of the robot's dexterous hand grasping the current object based on the prompt text and the ROI point cloud, and outputs the corresponding wrist joint coordinates, wrist orientation, and multiple finger joint angles. The touch point calculation module calculates the corresponding five-finger touch point positions based on the wrist joint coordinates, wrist orientation, and all finger joint angles using the forward kinematics of the Shadow Hand, and outputs them. The instruction data includes task instructions and instruction timestamps; the task instructions include the object to be retrieved and the object to be placed; the object to be retrieved is a target object to be retrieved in the environment, and the object to be placed is a target loading object or target placement area in the environment used to place the object to be retrieved. The environmental image data includes images of the work environment and image timestamps; The object retrieval target identification data includes an object retrieval target identification box, object retrieval target location, and object retrieval identification timestamp; the object retrieval target identification box includes object retrieval target type, object retrieval box center point coordinates, and object retrieval box size; the object retrieval target type matches the item type of the corresponding object being retrieved. The object recognition data includes an object recognition frame, object location, and object recognition timestamp; the object recognition frame includes object type, frame center coordinates, and frame size; the object type matches the item type or region type of the corresponding object. The ROI image data includes the object ROI image and the image timestamp; The touch point planning data includes the initial positions of the five fingers and the planning timestamp; the initial positions of the five fingers include the initial positions of the thumb, index finger, middle finger, ring finger, and little finger, and some or all of the initial positions are in contact with the surface of the object to be picked up; The multimodal dataset includes command sequences, environmental image sequences, object retrieval target sequences, object placement target sequences, ROI image sequences, and touch point planning sequences; wherein, the command sequences, environmental image sequences, object retrieval target sequences, object placement target sequences, ROI image sequences, or touch point planning sequences are each composed of multiple sets of command data, environmental image data, object retrieval target identification data, object placement target identification data, ROI image data, or touch point planning data that occur within the monitoring period and correspond to the current sequence; The instruction information includes the object to be retrieved and the object to be placed; The task time period includes the task start time and the task end time; The task data subset is a subset of the multimodal dataset in the corresponding task time period, consisting of a set of corresponding instruction data, environmental image data, object retrieval target recognition data, object placement target recognition data, ROI image data, and touch point planning data; The audio recognition dataset includes multiple first training data sets; the first training data sets include training audio and instruction labels. The target detection dataset includes multiple second training data sets; the second training data sets include training environment images, object capture box labels, and object placement box labels. The touch point planning dataset includes multiple third training data sets; the third training data sets include training ROI images and touch point labels.

3. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The step of acquiring the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and storing it in the robot's storage area specifically includes: In the task data area, the instruction timestamps within the monitoring period are extracted to form the corresponding instruction sequence; the image timestamps within the monitoring period are extracted to form the corresponding environmental image sequence; the object retrieval recognition timestamps within the monitoring period are extracted to form the corresponding object retrieval target sequence; the object placement recognition timestamps within the monitoring period are extracted to form the corresponding object placement target sequence; the image timestamps within the monitoring period are extracted to form the corresponding ROI image sequence; and the planning timestamps within the monitoring period are extracted to form the corresponding touchpoint planning sequence. The resulting multimodal dataset, composed of the instruction sequence, environmental image sequence, object retrieval target sequence, object placement target sequence, ROI image sequence, and touchpoint planning sequence, is then stored in the robot storage area.

4. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The process of using the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model to identify the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files specifically includes: The audio recording file is used as the model input audio to the speech activity detection model to detect speech segments and obtain the corresponding speech segment detection set; and based on the speech segment detection data of each speech segment in the speech segment detection set, the corresponding speech segments are extracted from the audio recording file; and each extracted speech segment is used as the model input audio to the automatic speech recognition model to perform text recognition and obtain the corresponding audio text; and each audio text is used as the corresponding source text to the recognition task instruction template to generate the corresponding recognition task instruction; and each recognition task instruction is input to the instruction recognition language model for processing to obtain the corresponding text recognition result, the obtained item, and the placement location; The text recognition results of each specific item are taken as the current result; the item to be retrieved and the placement location corresponding to the current result are taken as a set of corresponding items to be retrieved and placed to form the corresponding instruction information; and the corresponding audio segment of the current result and the corresponding segment start time are taken as the corresponding instruction audio and the task start time. The start time of each task is taken as the current time t. s And set the corresponding time t based on the preset maximum task duration L1. e =t s +L1; and transfer the time segment [t] from the video recording file. s ,t e The video clip is extracted as the current video; the current video is then processed into frames; all frames are traversed in chronological order; during this traversal, the currently traversed image is used as the current image; it is determined whether the current image is the first frame; if so, the previous frame is set to empty; otherwise, the previous frame is set as the previous frame. The current image is then input into the robotic arm joint detection model to perform robotic arm joint target detection and obtain the corresponding set of joint detection boxes; based on the camera intrinsic and extrinsic parameters of the observation camera, the coordinates of the center points of each detection box in the set of joint detection boxes are transformed into three-dimensional world coordinates to obtain the coordinates of the corresponding joint center points; the three-dimensional coordinate difference between every two joint center point coordinates is calculated to obtain the corresponding first coordinate difference; and the three-dimensional coordinate difference between every two initial positions of the robotic arm joints in the preset initial positions of the robot joints is calculated. The first and second coordinate differences are calculated to obtain the corresponding second coordinate difference; and the first and second coordinate differences with the same joint relative relationship are used to form corresponding difference pairs; and the first and second coordinate differences of each difference pair are calculated again to obtain the corresponding third coordinate difference; and it is identified whether each of the third coordinate differences meets the preset three-dimensional coordinate difference range. If it does, the corresponding matching result is set as a match; if it does not, the matching result is set as a mismatch; and it is identified whether all matching results corresponding to the current image are matches. If they are, the current image is marked as a repositioned image; otherwise, the current image is marked as a non-repositioned image; and when the current image is a repositioned image and the previous frame image is a non-repositioned image, the current round of traversal is terminated and the image time of the current image is taken as the corresponding task end time; if the corresponding task end time is not obtained until the last frame of the current round of traversal, then the current time t is used as the task end time. e Set the corresponding task end time; wherein, the initial position of the robot joint includes the initial positions of multiple robotic arm or hand joints, each initial position being a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; the range of the three-dimensional coordinate difference consists of the difference ranges of the X, Y, and Z dimensions. If the difference of each dimension of the third coordinate difference satisfies the corresponding difference range, then the current third coordinate difference satisfies the range of the three-dimensional coordinate difference; if at least one dimension of the third coordinate difference does not satisfy the corresponding difference range, then the current third coordinate difference does not satisfy the range of the three-dimensional coordinate difference. The task time period is composed of the start time of each task and its corresponding end time; the task information is composed of the task time period and its corresponding instruction audio and instruction information; and the task identification set is composed of all the obtained task information.

5. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The step of segmenting the multimodal dataset based on each task time period to obtain corresponding task data subsets specifically includes: Each task time period in the task identification set is taken as the current task time period; and the latest instruction timestamp among all instruction data in the instruction sequence of the multimodal dataset that satisfies the current task time period is taken as the current instruction data; the latest image timestamp among all environmental image data in the environmental image sequence that satisfies the current task time period is taken as the current environmental image data; the latest object recognition timestamp among all object recognition data in the object retrieval target sequence that satisfies the current task time period is taken as the current object recognition data; and the latest object recognition timestamp among all object recognition data in the object placement target sequence that satisfies the current task time period is taken as the current object recognition data. The latest time among all the object placement target recognition data is used as the current object placement target recognition data. The image timestamp in the ROI image sequence that satisfies the latest time among all the ROI image data of the current task time period is used as the current ROI image data. The planning timestamp in the touch point planning sequence that satisfies the latest time among all the touch point planning data of the current task time period is used as the current touch point planning data. The obtained current instruction data, current environment image data, current object retrieval target recognition data, current object placement target recognition data, current ROI image data, and current touch point planning data form the corresponding task data subset.

6. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The step of verifying the correctness of instruction recognition based on the instruction data of each instruction information and its corresponding task data subset, and generating training samples to be added to the audio recognition dataset when instruction recognition errors occur, specifically includes: Each instruction information in the task recognition set and its corresponding task data subset are taken as the corresponding current instruction 1 and current subset; the instruction data of the current subset is taken as current instruction 2; and the object to be picked up and placed in the current instructions 1 and 2 are identified. If they are, the corresponding current verification result is set as instruction recognition correct; otherwise, the corresponding current verification result is set as instruction recognition error. When the current verification result is instruction recognition error, the instruction audio corresponding to the current instruction 1 is taken as the corresponding training audio, and the current instruction 1 is taken as the corresponding instruction tag. The obtained training audio and the instruction tag are combined to form the corresponding first training data and added to the audio recognition dataset.

7. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The process of verifying the correctness of target detection based on the object retrieval detection model and the instruction data, environmental image data, object retrieval target recognition data, and object placement target recognition data of each task data subset, and generating training samples to be added to the target detection dataset when target detection errors occur, specifically includes: Each of the task data subsets is taken as the current subset; and the corresponding instruction data, environmental image data, object retrieval target recognition data, and object placement target recognition data are extracted from the current subset; The object to be picked and the object to be placed in the instruction data are extracted to form the corresponding current model input text; the working environment image in the environmental image data is used as the current model input image; and the current model input text and the current model input image are input into the object picking and placing detection model for processing to obtain the corresponding object picking detection box and object placing detection box. The object detection frame and the object target recognition frame of the object detection data are combined to form a first frame group, and the object placement detection frame and the object placement target recognition frame of the object placement target recognition data are combined to form a second frame group. The first or second frame group is used as the current frame group. The target types of the two frames in the current frame group are checked for matching. If not, the corresponding frame group verification result is set to failure. If yes, the straight-line distance between the coordinates of the two center points of the current frame group and the height-width difference between the two frames are calculated to obtain the corresponding current center distance, current height difference, and current width difference. The current center distance is set to a preset value. When the center distance is within the preset range, and the current height difference meets the preset height difference range, and the current width difference meets the preset width difference range, the corresponding frame group verification result is set to successful. When the current center distance does not meet the preset center distance range, or the current height difference does not meet the preset height difference range, or the current width difference does not meet the preset width difference range, the corresponding frame group verification result is set to failed. The system also identifies whether both frame group verification results are successful. If so, the corresponding current verification result is set to "target detection correct"; otherwise, the corresponding current verification result is set to "target detection incorrect". When the current verification result is an object detection error, the current working environment image, the object detection box, and the object placement detection box are used as a set of corresponding training environment images, object detection box labels, and object placement box labels to form a corresponding second training data, and the current second training data is added to the object detection dataset.

8. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The process of verifying the correctness of touch point planning based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset, and generating training samples to be added to the touch point planning dataset when touch point planning errors occur, specifically includes: Each of the task data subsets is taken as the current subset; and the corresponding instruction data, ROI image data, and touch point planning data are extracted from the current subset; Based on the object being retrieved according to the instruction data, the corresponding ROI type is set, and the ROI image of the retrieved object from the ROI image data is used as the corresponding RGB-D image. The current ROI type and the RGB-D image are then input into the ROI point cloud generation model for processing to obtain the corresponding ROI point cloud. According to the preset DexVLG model prompt text setting rules, the corresponding prompt text is set based on the object being retrieved. The current ROI point cloud and the prompt text are then input into the touch point planning model for processing to obtain the corresponding five-finger touch point position. The coordinates of all five finger contact points and the initial positions of the five fingers in the contact point planning data are unified to the same three-dimensional coordinate system, which is either the world coordinate system or the robot's XYZ three-dimensional coordinate system. The five contact points of the five fingers are then sequentially recorded as their corresponding positions. The five touch points at the initial positions of the five fingers are sequentially recorded as the corresponding positions. 1 ≤ index i ≤ 5; and calculate the geometric position deviation e between the five-finger contact point position and the initial position of the five fingers. ; It identifies whether the geometric position deviation e is less than a preset geometric position deviation threshold; if yes, the corresponding current verification result is set as correct contact planning; if no, the corresponding current verification result is set as incorrect contact planning. When the current verification result indicates a touch point planning error, the current ROI image data and the five-finger touch point positions are used as a set of corresponding training ROI images and touch point labels to form a corresponding third training data set; and the current third training data is added to the touch point planning dataset.

9. The method for extracting training data based on multimodal monitoring data of a robot according to claim 2, characterized in that, The single-arm robot's base is fixed; The actuator of the single-arm robot includes an articulated robotic arm and a five-fingered dexterous hand; the articulated robotic arm is a six-axis or seven-axis robotic arm, corresponding to six or seven robotic arm joints, and its specific types include at least the KUKA robotic arm and the UR robotic arm; the base joint of the articulated robotic arm is connected to a fixed body base, and the end joint is connected to the five-fingered dexterous hand; The five-finger dexterous hand is a Shadow Hand dexterous hand, which is connected to the end joint of the jointed robotic arm via the wrist joint; the palm of the five-finger dexterous hand is equipped with a palm camera and a tactile sensor, and each fingertip is equipped with the tactile sensor; The tactile sensor is used to sense the multidimensional force and coefficient of friction of the palm / fingertip contact surface of the object; The multidimensional force includes multiple sensed force data, which includes force type and three-dimensional force components; the force type includes at least contact force, contact surface pressure, normal force, and shear force; the three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system; the friction coefficient is estimated based on the normal force and shear force. The sensors of the single-arm robot also include an audio receiver and a rotating camera; the audio receiver's audio reception range and the rotating camera's rotation angle range can fully cover the robot's working space; the rotating camera is a depth camera. When the single-arm robot is not performing a task based on voice commands, it controls all joint positions of the articulated robotic arm and the five-fingered dexterous hand to remain consistent with the preset initial positions of the robot joints; the initial positions of the robot joints include multiple initial positions of the robotic arm or hand joints, and each initial position is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; The single-arm robot is specifically used when, according to the received voice command, it retrieves the object specified by the command from the work environment and moves it to the specified location for placement: The system performs instruction conversion on the real-time environmental audio received by the audio receiver and identifies whether the conversion instruction is a valid task instruction. After identifying a valid task instruction, the corresponding object to be retrieved and the object to be placed are extracted from the current task instruction; and the instruction data composed of the current task instruction and its corresponding instruction timestamp is stored in the task data area. The system uses the rotating camera to capture a panoramic view of the robot's working environment, obtaining an image of the working environment. It then identifies the objects to be picked up or placed, identifies the corresponding target positions, and plans the palm-shaped target positions near the object's location. The system stores the environmental image data, composed of the working environment image and its corresponding timestamp, in the task data area. Finally, it stores the corresponding object recognition data, composed of the object recognition frame, object location, and object recognition timestamp for each object, in the task data area. With the constraint that the robotic arm and dexterous hand do not collide with any object during the movement, real-time motion planning and drive control are performed on all joints of the robotic arm and dexterous hand, so that the palm of the five-fingered dexterous hand moves to the target position of the palm. Upon reaching the target location in the palm, the palm camera is invoked to capture a local environment of interest image centered on the object to be retrieved, thereby obtaining the ROI image; and the ROI image data, composed of the ROI image and its corresponding image timestamp, is stored in the task data area. And based on the object ROI image, the initial position and initial torque of the five fingers are planned; Based on the initial position of the five fingers, the movement planning and drive control of all joints of the dexterous hand are performed, so that the center position of the fingertips of the dexterous hand reaches the corresponding initial position and makes pre-contact with the object to be picked up; The initial position of the five fingers and the corresponding final planning timestamp are combined to form the touch point planning data and stored in the task data area; wherein, the initial torque of the five fingers is composed of multiple finger joint torques, and the finger joint torque is a six-dimensional torque in the XYZ three-dimensional coordinate system of the robot body, which is composed of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes; After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers as the initial torque value. During the operation, the tactile perception data fed back by all the tactile sensors in real time is used to confirm whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all the real-time feedback tactile perception data, with the constraint of not causing damage to the object. The five-finger dexterity hand is then driven to continue grasping the object using the current optimized five-finger torque. The optimized five-finger torque is composed of the torques of multiple corresponding finger joints. After confirming that the object being retrieved has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target position and the shape and material characteristics of the object being retrieved. With the constraint that the robotic arm and the dexterous hand do not collide with any object during the movement and that the object held by the dexterous hand will not loosen and fall, the motion planning and drive control of all joints of the robotic arm and the dexterous hand are performed in real time, so that the center point of the object being retrieved moves to the release center position. After reaching the release center position, the object is released by driving the hand joints of the five-fingered dexterous hand; and after successfully releasing the object, the jointed robotic arm and all joints of the five-fingered dexterous hand are driven back to their original positions based on the initial positions of the robot joints.

10. An apparatus for performing the method for extracting training data based on multimodal monitoring data of a robot as described in any one of claims 1-9, characterized in that, The device includes: a model loading module, a monitoring preparation module, a monitoring acquisition module, a task segment identification module, a monitoring data segmentation module, an instruction recognition training data extraction module, a target detection training data extraction module, and a touch point planning training data extraction module. The model loading module is used to load the trained speech activity detection model, automatic speech recognition model, instruction recognition language model, robotic arm joint detection model, object retrieval and placement detection model, ROI point cloud generation model, and touch point planning model. The monitoring preparation module connects to the monitored single-arm robot and to an observation microphone and camera installed in the robot's operating environment. It also locally stores a robot storage area, an observation storage area, and a dataset storage area. The single-arm robot retrieves a specified object from the operating environment based on received voice commands and moves it to the designated location. The robot also has a locally configured task data area for storing multimodal process data generated during its operation. This multimodal process data includes multiple command data, multiple environmental image data, multiple object retrieval target recognition data, multiple object placement target recognition data, multiple ROI image data, and multiple touchpoint planning data. The audio reception range of the observation microphone and the shooting angle range of the observation camera both cover the robot's operating space, and the observation camera can capture unobstructed images of the robot's object retrieval process. The observation camera is a depth camera. The dataset storage area stores audio recognition datasets, target detection datasets, and touchpoint planning datasets. The monitoring and acquisition module is used to start the single-arm robot to perform object retrieval tasks according to environmental voice commands in its working environment within a preset monitoring period, and to call the observation microphone to receive and record real-time audio of the robot's working environment, and to call the observation camera to shoot and record the robot's working process, and to generate corresponding audio and video recording files in the observation storage area to continuously store the recording data of the microphone and camera; and at the end of the monitoring period, to stop the audio and video recording of the microphone and camera, and to obtain the multimodal dataset generated during the monitoring period from the task data area of ​​the single-arm robot and store it in the robot storage area; The task segment recognition module is used to utilize the voice activity detection model, the automatic speech recognition model, the command recognition language model, and the robotic arm joint detection model to identify the command audio, command information, and task time period of a single robot task within the monitoring period based on audio and video recording files to obtain a task recognition set; the task recognition set includes multiple task information; the task information includes the command audio, the command information, and the task time period; The monitoring data segmentation module divides the multimodal dataset into corresponding task data subsets based on each task time period; The instruction recognition training data extraction module performs instruction recognition correctness verification based on the instruction data of each instruction information and the corresponding task data subset, and generates training samples to be added to the audio recognition dataset when instruction recognition is incorrect. The target detection training data extraction module performs target detection correctness verification based on the object retrieval and placement detection model and the instruction data, environmental image data, object retrieval target recognition data and object placement target recognition data of each task data subset, and generates training samples to be added to the target detection dataset when the target detection is incorrect; The touch point planning training data extraction module performs touch point planning correctness verification based on the ROI point cloud generation model, the touch point planning model, and the instruction data, ROI image data, and touch point planning data of each task data subset. When touch point planning is incorrect, training samples are generated and added to the touch point planning dataset.

11. The apparatus according to claim 10, characterized in that, The single-arm robot's base is fixed; The actuator of the single-arm robot includes an articulated robotic arm and a five-fingered dexterous hand; the articulated robotic arm is a six-axis or seven-axis robotic arm, corresponding to six or seven robotic arm joints, and its specific types include at least the KUKA robotic arm and the UR robotic arm; the base joint of the articulated robotic arm is connected to a fixed body base, and the end joint is connected to the five-fingered dexterous hand; The five-finger dexterous hand is a Shadow Hand dexterous hand, which is connected to the end joint of the jointed robotic arm via the wrist joint; the palm of the five-finger dexterous hand is equipped with a palm camera and a tactile sensor, and each fingertip is equipped with the tactile sensor; The tactile sensor is used to sense the multidimensional force and coefficient of friction of the palm / fingertip contact surface of the object; The multidimensional force includes multiple sensed force data, which includes force type and three-dimensional force components; the force type includes at least contact force, contact surface pressure, normal force, and shear force; the three-dimensional force components consist of three force components along the X, Y, and Z axes of the robot's XYZ three-dimensional coordinate system; the friction coefficient is estimated based on the normal force and shear force. The sensors of the single-arm robot also include an audio receiver and a rotating camera; the audio receiver's audio reception range and the rotating camera's rotation angle range can fully cover the robot's working space; the rotating camera is a depth camera. When the single-arm robot is not performing a task based on voice commands, it controls all joint positions of the articulated robotic arm and the five-fingered dexterous hand to remain consistent with the preset initial positions of the robot joints; the initial positions of the robot joints include multiple initial positions of the robotic arm or hand joints, and each initial position is a three-dimensional coordinate in the XYZ three-dimensional coordinate system of the robot body; The single-arm robot is specifically used when, according to the received voice command, it retrieves the object specified by the command from the work environment and moves it to the specified location for placement: The system performs instruction conversion on the real-time environmental audio received by the audio receiver and identifies whether the conversion instruction is a valid task instruction. After identifying a valid task instruction, the corresponding object to be retrieved and the object to be placed are extracted from the current task instruction; and the instruction data composed of the current task instruction and its corresponding instruction timestamp is stored in the task data area. The system uses the rotating camera to capture a panoramic view of the robot's working environment, obtaining an image of the working environment. It then identifies the objects to be picked up or placed, identifies the corresponding target positions, and plans the palm-shaped target positions near the object's location. The system stores the environmental image data, composed of the working environment image and its corresponding timestamp, in the task data area. Finally, it stores the corresponding object recognition data, composed of the object recognition frame, object location, and object recognition timestamp for each object, in the task data area. With the constraint that the robotic arm and dexterous hand do not collide with any object during the movement, real-time motion planning and drive control are performed on all joints of the robotic arm and dexterous hand, so that the palm of the five-fingered dexterous hand moves to the target position of the palm. Upon reaching the target location in the palm, the palm camera is invoked to capture a local environment of interest image centered on the object to be retrieved, thereby obtaining the ROI image; and the ROI image data, composed of the ROI image and its corresponding image timestamp, is stored in the task data area. And based on the object ROI image, the initial position and initial torque of the five fingers are planned; Based on the initial position of the five fingers, the movement planning and drive control of all joints of the dexterous hand are performed, so that the center position of the fingertips of the dexterous hand reaches the corresponding initial position and makes pre-contact with the object to be picked up; The initial position of the five fingers and the corresponding final planning timestamp are combined to form the touch point planning data and stored in the task data area; wherein, the initial torque of the five fingers is composed of multiple finger joint torques, and the finger joint torque is a six-dimensional torque in the XYZ three-dimensional coordinate system of the robot body, which is composed of three torque components along the X, Y, and Z axes and three torque components rotating along the X, Y, and Z axes; After successful pre-contact, the five-finger dexterity hand is driven to grasp the object using the initial torque of the five fingers as the initial torque value. During the operation, the tactile perception data fed back by all the tactile sensors in real time is used to confirm whether the object has been firmly grasped by the dexterity hand. If it is confirmed that the object has not been firmly grasped, the five-finger torque is optimized based on all the real-time feedback tactile perception data, with the constraint of not causing damage to the object. The five-finger dexterity hand is then driven to continue grasping the object using the current optimized five-finger torque. The optimized five-finger torque is composed of the torque of the corresponding multiple finger joints. After confirming that the object being retrieved has been firmly grasped by the dexterous hand, the release center position within the space covered by the object is planned according to the target position and the shape and material characteristics of the object being retrieved. With the constraint that the robotic arm and the dexterous hand do not collide with any object during the movement and that the object held by the dexterous hand will not loosen and fall, the motion planning and drive control of all joints of the robotic arm and the dexterous hand are performed in real time, so that the center point of the object being retrieved moves to the release center position. After reaching the release center position, the object is released by driving the hand joints of the five-fingered dexterous hand; and after successfully releasing the object, the jointed robotic arm and all joints of the five-fingered dexterous hand are driven back to their original positions based on the initial positions of the robot joints.

12. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-9; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-9.

Images