A robot multi-modal data synchronous fusion acquisition method and system

Abstract

This invention belongs to the field of data acquisition technology and discloses a method and system for synchronous fusion acquisition of robot multimodal data. The method includes: generating an authorization index based on authorization verification; synchronously acquiring initial multimodal data according to the authorization index and binding it with a unified global timestamp to obtain target multimodal data; performing temporal alignment, mapping, and sensitive information processing on the target multimodal data to obtain fused multimodal data; performing hierarchical task segmentation on the fused multimodal data to obtain hierarchical task results; performing integrity verification and differentiation processing on the fused multimodal data based on a preset packaging period, and generating periodic data packets by combining the hierarchical task results and the authorization index; and performing structured storage of the periodic data packets according to the hierarchical task results to obtain multimodal data acquisition results. This invention can improve the spatiotemporal correspondence accuracy of multimodal data under different sampling frequencies and different coordinate systems.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a method and system for synchronous fusion acquisition of multimodal data from a robot. Background Technology

[0002] As AI robots are used in long-term continuous tasks such as flexible grasping, precision assembly, and human-computer interaction, they not only need to have high-precision perception capabilities, but also need to achieve the fusion and understanding of multimodal information to support the execution and decision-making of complex tasks. This is to improve the robot's work performance by building high-quality, multimodal training datasets with temporal consistency.

[0003] During the data acquisition process, different sensing devices are usually used to collect data separately. The sampling frequency and internal clock of each acquisition device are different, which can easily cause clock drift and data misalignment, making it difficult to ensure that the data of different modes correspond accurately at the same time.

[0004] In addition, existing technologies for segmenting long-term robotic tasks usually rely on manual labeling, which is easily affected by misoperation, occlusion and recognition errors, resulting in inaccurate data segment boundaries and chaotic task hierarchy, affecting the training quality of the dataset. Different modal data also have the problem of large differences in data volume, which can easily lead to a large consumption of storage space and transmission bandwidth.

[0005] Therefore, how to provide a method and system for synchronous fusion and acquisition of multimodal data of robots is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method and system for synchronous fusion acquisition of multimodal data of a robot, in order to solve the problems mentioned above in the prior art.

[0007] According to a first aspect of the present invention, a method for synchronous fusion acquisition of multimodal data of a robot is provided.

[0008] In one embodiment, the robot multimodal data synchronous fusion acquisition method includes: An authorization index is generated based on authorization verification. Initial multimodal data is collected synchronously according to the authorization index and bound to a unified global timestamp to obtain target multimodal data. The initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data. Based on a unified global timestamp, the target multimodal data is time-series aligned, and the time-series aligned target multimodal data is mapped through a preset coordinate transformation relationship; and based on the authorized index, sensitive information processing is performed on the coordinate-mapped target multimodal data to obtain fused multimodal data. Using audio data, the fused multimodal data is segmented into hierarchical tasks to determine the first boundary of the first task and the second boundary of the second task; a hierarchical task structure is constructed based on the first and second boundaries, and the execution status of the second task is determined based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; Based on a preset packaging cycle, integrity verification and differentiation processing are performed on the fused multimodal data, and periodic data packets are generated by combining the results of hierarchical tasks and authorized indexes. Based on the results of the hierarchical tasks and the preset storage mode, the periodic data packets are stored in a structured manner, and based on the first task or the second task, the corresponding multimodal data is read from the structured periodic data packets to obtain the multimodal data acquisition results.

[0009] According to a second aspect of the present invention, a robot multimodal data synchronous fusion acquisition system is provided.

[0010] In one embodiment, the robot multimodal data synchronous fusion acquisition system includes: The synchronous acquisition module is used to generate an authorization index based on authorization verification, synchronously acquire initial multimodal data according to the authorization index, and bind a unified global timestamp to obtain target multimodal data; wherein, the initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data; The multimodal fusion processing module is used to perform time-series alignment of target multimodal data based on a unified global timestamp, map the time-series aligned target multimodal data through a preset coordinate transformation relationship, and perform sensitive information processing on the coordinate-mapped target multimodal data according to the authorized index to obtain fused multimodal data. The hierarchical task processing module is used to perform hierarchical task segmentation on the fused multimodal data using audio data, determine the first boundary of the first task and the second boundary of the second task; construct a hierarchical task structure based on the first and second boundaries, and determine the execution status of the second task based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; The periodic data packet generation module is used to perform integrity verification and differentiation processing on the fused multimodal data based on a preset packaging period, and generate periodic data packets by combining the results of hierarchical tasks and authorized indexes. The structured storage module is used to perform structured storage of periodic data packets according to the results of hierarchical tasks and preset storage modes, and to read the corresponding multimodal data from the structured periodic data packets based on the first task or the second task to obtain the multimodal data acquisition results.

[0011] According to a third aspect of the present invention, a computer device is provided.

[0012] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: 1. This invention links the collected data with the authorization information, enabling pre-blocking of invalid authorizations and tracing of the collection process, improving the compliance of data collection, and enhancing the spatiotemporal correspondence accuracy of multimodal data under different sampling frequencies and coordinate systems.

[0014] 2. This invention enables hierarchical segmentation and state labeling of multimodal data, reducing the workload of subsequent data processing and manual annotation, making the data structure clear and easy to retrieve, split and reuse, while reducing the risk of overall data loss due to power outages or network outages.

[0015] 3. This invention performs task segmentation and status marking on multimodal data based on the results of hierarchical tasks, and provides hierarchical directory storage mode and fixed-period unified storage mode. It can reduce the overhead of the file system in massive task scenarios by eliminating the need to frequently create and read / write multi-level task directories. It also enables the rapid extraction of any task fragment by relying on timestamp index, balancing storage read / write efficiency with the flexibility of on-demand data retrieval.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 This is a flowchart illustrating a method for synchronous fusion and acquisition of multimodal data of a robot according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the principle of a robot multimodal data synchronous fusion acquisition system according to an exemplary embodiment; Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0019] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0020] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0021] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0022] Figure 1 An embodiment of a robot multimodal data synchronous fusion acquisition method of the present invention is shown.

[0023] In this optional embodiment, the robot multimodal data synchronous fusion acquisition method includes: S101. Generate an authorization index based on authorization verification, synchronously collect initial multimodal data according to the authorization index, and bind a unified global timestamp to obtain target multimodal data; wherein, the initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data; In this optional embodiment, the step of generating an authorization index based on authorization verification, synchronously collecting initial multimodal data according to the authorization index, and binding a unified global timestamp to obtain target multimodal data includes: Obtain notification logs, identity verification information, liveness verification information, confirmation audio information, and electronic signature data; The identity verification information, liveness verification information, and confirmation audio information are verified to obtain the authorization verification result; based on the authorization verification result, the notification letter viewing log and electronic signature data are stored as evidence to generate an authorization index; The authorization index is validated to obtain the permission validation result. The permission validation result is then used to drive the acquisition channel to collect data and obtain initial multimodal data. Based on the acquisition time corresponding to the acquisition channel, a unified global timestamp is bound to the visual data, audio data, joint force data, tactile data and motion trajectory data in the initial multimodal data to obtain the target multimodal data.

[0024] In this optional embodiment, the step of validating the authorization index to obtain the permission verification result, and using the permission verification result to drive the acquisition channel to acquire data and obtain initial multimodal data includes the following steps: When the permission verification result indicates that the authorized index is valid, the preset pulse signal is unlocked; when the permission verification result indicates that the authorized index is invalid, the preset pulse signal is cut off and the current multimodal data acquisition process is terminated. Using the unlocked pulse signal, data acquisition is driven through the visual acquisition channel, audio acquisition channel, joint force acquisition channel, tactile acquisition channel and motion trajectory acquisition channel to obtain visual data, audio data, joint force data, tactile data and motion trajectory data respectively; Visual data, audio data, joint force data, tactile data, and motion trajectory data are aggregated and processed to obtain initial multimodal data.

[0025] Specifically, in this embodiment, authorization verification is completed before the five acquisition channels are activated. The electronic signature authorization signing unit acquires the notification log, identity verification information, liveness verification information, confirmation audio information, and electronic signature data. Among them, the identity verification information, liveness verification information, and confirmation audio information correspond to ID card verification, liveness face verification, and confirmation audio verification, respectively. The electronic signature data includes the electronic handwritten signature process and its screen recording data.

[0026] The comparison objects, single-item pass conditions, long-term authorization reuse rules, and comprehensive authorization judgment rules for each of the three verification methods are as follows: ID Card Verification (Identity Verification Information): Comparison objects: Information stored in the physical ID card chip (name, ID number, image) read on-site and information filed in the public security identity database; Pass conditions: The document is authentic and valid, the chip reading is unaltered, the text and image information match perfectly, and the document's validity period has not expired; document forgery, inconsistent information, and expiration will all result in single-item failure. Live Face Verification (Live Face Verification Information): Comparison objects: Real-time capture of the user's dynamic facial video frames on-site and the facial photo embedded in the ID card; Pass conditions: Liveness detection passes (completes verification of random actions such as head shaking, blinking, and mouth opening, preventing photo / video re-engraving attacks); the facial feature similarity is greater than a preset threshold (preferably 92% in this embodiment); facial feature mismatch or liveness verification failure will result in single-item failure. Confirmation Audio Verification (Confirmation Audio Information). This solution differentiates between two business scenarios, adapting to scenarios without existing voiceprint templates: Scenario 1 (Existing User Voiceprint Templates): Comparison Object: The audio waveform and voiceprint features of the user reading the preset authorization confirmation script aloud, compared with the user's historically stored voiceprint templates; Passing Conditions: The ASR-transcribed text completely matches the standard authorization confirmation script; the real-time voiceprint matches the template with a matching degree of ≥88%. Both conditions must be met for a single item to pass. Scenario 2 (No Pre-stored Voiceprint Templates, First Authorization Collection): The comparison object is only the audio text content, without voiceprint comparison; Passing Conditions: The audio ASR recognition result is completely consistent with the preset standard authorization confirmation script, with no missing words, misreadings, or omissions; passing the text verification alone satisfies this verification requirement; blurry audio, recognition failure, or script mismatch all result in single item failure.

[0027] Comprehensive authorization verification judgment rules: The triple verification adopts the logic of full pass: Only when all three individual verifications of ID card verification, live face verification and confirmation audio verification pass, and the system detects that the notification log has been fully viewed, the electronic handwritten signing process has not been interrupted, and the signing screen recording has been fully retained, the overall authorization verification result is passed, and an authorization index with a one-week validity period is generated; if any verification fails, the notification is not fully viewed, the electronic signature is missing, or the process is exited midway, the overall authorization verification is judged to fail, no valid authorization index is generated, the collection pulse signal is directly cut off, and multimodal data collection is prohibited from starting.

[0028] The single-person, continuous, multi-task, long-term verification and reuse mechanism addresses scenarios where the same operator continuously performs long-duration, multi-segment robot task recordings. It sets up a time-limited reuse logic for the authorization index: After the first complete triple verification and generation of the authorization index, the system persistently stores the operator's identity, facial features, voiceprint information, and authorization start and end times. The authorization index has a preset validity period of one week; device power failures and system restarts will not clear the persistent authorization cache. Within this one-week validity period, when the operator starts the next new data collection task, there is no need to repeat the complete triple verification of ID card, face, and audio; a simple face capture comparison (with the comparison threshold lowered to 85%) quickly confirms that the operator has not changed, allowing the reuse of the existing authorization index and direct unlocking of the collection pulse to start a new round of data collection. Furthermore, the boundary conditions triggering a re-complete verification are: the authorization index generation time exceeds the preset one-week validity period; the simple face comparison does not match, indicating a change in operator; or the staff manually clears the current authorization cache. In any of these cases, the complete triple verification process must be repeated to generate a new authorization index.

[0029] When the authorization verification result indicates that the verification is successful, the notification log and electronic signature data are stored using the national cryptographic SM3 hash, and an authorization index corresponding to this authorization is generated. The authorization index is used to associate authorization information, subsequent periodic data packets, and audit records. When the authorization index is valid, the five independent GPIO hardware pulses (i.e., preset pulse signals) output by the global clock synchronization module are unlocked; when the authorization index is invalid, the five independent GPIO hardware pulses are cut off, thereby blocking the acquisition of data without valid authorization before acquisition.

[0030] It should be noted that the global clock synchronization module can be implemented by a controller with hardware timing function and multi-channel GPIO output function, which is used to generate five independent synchronization pulses based on a unified clock source.

[0031] After the acquisition permission is unlocked, the GPIO hardware pulses drive the visual acquisition channel, audio acquisition channel, joint force acquisition channel, tactile acquisition channel and motion trajectory acquisition channel to work synchronously. In this embodiment, the GPIO pulse delay is less than 10μs, and the overall timing alignment error of the five acquisition channels is less than 1ms.

[0032] It should be noted that, in this embodiment, the sampling frame rate of the visual acquisition channel can be selected as 30Hz, the sampling frequency of the audio acquisition channel can be selected as 48kHz, the sampling frequency of the joint force acquisition channel can be selected as 1kHz, the sampling frequency of the tactile acquisition channel can be selected as 2kHz, and the sampling frequency of the motion trajectory acquisition channel can be selected as 200Hz.

[0033] Simultaneously, the visual acquisition channel preferably uses an RGB camera, or a binocular stereo vision device, RGB-D camera, dToF device, or 3D LiDAR capable of acquiring depth-dimensional visual data; the audio acquisition channel acquires audio waveforms and obtains ASR-transcribed text; the joint force acquisition channel acquires six-dimensional torque timing data of each joint of the robot; the tactile acquisition channel acquires the distributed two-dimensional pressure matrix of the manipulator; and the motion trajectory acquisition channel reads the robot encoder and acquires joint angle, angular velocity, and displacement trajectory data. When acquiring corresponding modal data, each acquisition channel writes the acquisition time provided by the global clock into the corresponding data record as a unified global timestamp; subsequently, the five types of data are aggregated to obtain the target multimodal data.

[0034] When each acquisition channel outputs a single frame or a single piece of raw data, a fixed metadata field block is reserved in the header of the data structure, and a global timestamp is uniformly written there. The writing location specifications for the five types of modal data are as follows: Visual data (RGB, RGB-D, LiDAR single frame image): frame_meta.timestamp in the header of the image frame structure; Audio data (audio block waveform): audio_block.meta_ts header of audio block structure; Joint force six-dimensional torque data: force_ts header field of single torque time sequence structure; Tactile pressure matrix data: tactile_frame.ts header of single frame tactile array; Motion trajectory encoder data: traj_point.time_stamp single set of trajectory sampling structure.

[0035] All modal timestamps are uniformly stored as 64-bit unsigned integers to ensure that time series can be continuously compared across days and tasks.

[0036] S102. Based on a unified global timestamp, perform time-series alignment on the target multimodal data, map the time-series aligned target multimodal data through a preset coordinate transformation relationship, and perform sensitive information processing on the coordinate-mapped target multimodal data according to the authorized index to obtain fused multimodal data. In this optional embodiment, the step of performing time-series alignment of the target multimodal data based on a unified global timestamp, and mapping the time-series aligned target multimodal data through a preset coordinate transformation relationship, includes: Read the unified global timestamps corresponding to visual data, audio data, joint force data, tactile data and motion trajectory data in the target multimodal data respectively. Based on the unified global timestamps, establish the temporal correspondence between visual data, audio data, joint force data, tactile data and motion trajectory data to obtain the temporally aligned target multimodal data. Based on the preset coordinate transformation relationship, the visual target position corresponding to the time-aligned visual data, the joint force application point corresponding to the joint force data, and the tactile contact point corresponding to the tactile data are mapped to the robot base coordinate system respectively, so as to obtain the coordinate-mapped visual data, joint force data, and tactile data. Based on the time correspondence, the coordinate-mapped visual data, joint force data, and tactile data are associated with the time-aligned audio data and motion trajectory data to obtain the coordinate-mapped target multimodal data.

[0037] Specifically, the acquisition time of the five types of data is read according to a unified global timestamp, and the temporal correspondence between visual data, audio data, joint force data, tactile data and motion trajectory data within the same time range is established to obtain the target multimodal data after temporal alignment; each acquisition channel still retains the corresponding data record according to its own fixed sampling frequency.

[0038] It should be noted that the preset coordinate transformation relationships include the coordinate transformation relationships of the vision camera coordinate system, the joint force local coordinate system, and the haptic array local coordinate system relative to the robot base coordinate system. According to the corresponding coordinate transformation relationships, the visual target position in the visual data, the joint force application point in the joint force data, and the haptic contact point in the haptic data are mapped to the robot base coordinate system, respectively, to obtain the coordinate-mapped visual data, joint force data, and haptic data.

[0039] It should be noted that this solution pre-defines a homogeneous transformation matrix from the sensor's local coordinate system to the robot's base coordinate system. This matrix is ​​calibrated and fixed offline; it is only read and called during runtime and is not calculated in real time. Furthermore, the uniform unit for length is mm, the unit for angle is rad, and all coordinate systems follow the right-hand Cartesian coordinate system rule.

[0040] The robot's base coordinate system (Global Spatial Reference) has the following origin: the center of the robot's base; the X-axis points directly in front of the robot, the Y-axis points to the left side of the robot, and the Z-axis is vertically upward, perpendicular to the base.

[0041] The camera coordinate system Cam has its origin at the optical center of the camera lens; the X-axis extends horizontally to the right along the image plane, the Y-axis extends vertically downwards along the image plane, and the Z-axis is the lens optical axis, pointing towards the working space.

[0042] Force coordinate system, origin: geometric center of torque sensor mounting flange; three axes are rigidly bound to the joint linkage and rotate synchronously with the joint. Tact coordinate system, origin: center of tactile pressure array plane; X-axis along the long side of the array's transverse direction, Z-axis perpendicular to the tactile contact surface, with the outward direction being the positive force direction.

[0043] The expression for the transformation matrix is: ; Where R=[ , , , , , , , ] is the rotation matrix, representing the three-axis rotational attitude of the local coordinate system relative to the robot's base coordinate system; P=[ ; ; The matrix [] represents the translation vector, indicating the 3D coordinates of the local coordinate system origin in the base coordinate system, in mm. The last row of the matrix is ​​fixed at [0, 0, 0, 1], representing the homogeneous coordinate general constraint row. The system pre-stores three sets of fixed transformation matrices: camera transformation matrix. Joint force conversion matrix and tactile transformation matrix .

[0044] It should be noted that all the above matrices are solved offline through hand-eye calibration only during the equipment installation and debugging phase: the robot is controlled to switch between multiple postures to collect calibration data, the least squares method is used to solve the matrix parameters, and the results are stored in the controller configuration file. Recalibration is only required when the camera, torque sensor, or tactile gripper is disassembled, moved, or the hardware is replaced; the matrices are not recalculated during daily multi-task recording and operation.

[0045] The coordinate mapping calculation method during the runtime phase is as follows: ; In the formula, The sensor's local four-dimensional homogeneous coordinates include the coordinates of the visual target point, the joint force application point, and the tactile contact point in its own coordinate system. For the corresponding sensor, a dedicated offline calibration homogeneous transformation matrix is ​​required. To unify the converted points to the four-dimensional homogeneous space of the robot's base coordinate system.

[0046] By substituting the visual target position, joint force application point, and tactile contact point into this formula, spatial conversion is completed, unifying all multimodal perception data into the robot's base coordinate system and achieving spatial benchmark alignment of multi-source data.

[0047] It should be noted that the sensor local coordinates include the four-dimensional homogeneous coordinates of the visual target point, the joint force application point, and the tactile contact point in its own sensing coordinate system. The base coordinate system coordinates include the three-dimensional spatial points that have been unified to the robot's base coordinate system after matrix transformation.

[0048] The collected visual, joint force, and tactile spatial points are all converted through corresponding dedicated matrices to achieve a unified spatial benchmark for multimodal perception data.

[0049] Based on the above time correspondence, the coordinate-mapped visual data, joint force data, and tactile data are associated with the time-aligned audio data and motion trajectory data, so that data with different sampling frequencies have a unified time reference and a spatial reference under the robot's base coordinate system, thus obtaining the target multimodal data after coordinate mapping.

[0050] In this optional embodiment, the step of performing sensitive information processing on the target multimodal data after coordinate mapping according to the authorized index to obtain fused multimodal data includes: Based on the authorized index, determine the information processing methods corresponding to the visual and audio data in the target multimodal data after coordinate mapping; When the information processing method is raw data retention, visual data and audio data are identified as the primary target data; When the information processing method is desensitization, visual data and audio data are desensitized to obtain the second target data. Based on the time correspondence, the first target data or the second target data is associated with the coordinate-mapped joint force data, tactile data, and time-aligned motion trajectory data to obtain fused multimodal data.

[0051] Specifically, the authorization index records the information processing method corresponding to this authorization. When the authorization index is valid and the information processing method is original retention, visual and audio data are retained; when the authorization index is valid and the information processing method is desensitization, facial information in the visual data and voice information in the audio data are desensitized. When the authorization index is invalid, the acquisition pulse is cut off according to the pulse signal acquisition blocking logic, so that the visual acquisition channel, audio acquisition channel, joint force acquisition channel, tactile acquisition channel, and motion trajectory acquisition channel are not activated, no initial multimodal data is generated, and no subsequent sensitive information processing flow is entered.

[0052] Based on the time correspondence, the original retained or desensitized visual and audio data are associated with the coordinate-mapped joint force data, tactile data, and time-aligned motion trajectory data to obtain fused multimodal data.

[0053] S103. Using audio data, perform hierarchical task segmentation on the fused multimodal data to determine the first boundary of the first task and the second boundary of the second task; construct a hierarchical task structure based on the first and second boundaries, and determine the execution status of the second task based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; In this optional embodiment, the step of using audio data to perform hierarchical task segmentation on the fused multimodal data and determining the first boundary of the first task and the second boundary of the second task includes: Keyword recognition is performed on the audio data to obtain the first recognition result and the second recognition result; Based on the first recognition result, the start and end positions of the first task are determined, the first boundary is obtained, and the data range corresponding to the first task is determined from the fused multimodal data according to the first boundary. Within the data range corresponding to the first task, the start and end positions of the second task are determined based on the second recognition result, and the second boundary corresponding to each second task is obtained. Based on the second boundary, the data range corresponding to the second task is determined from the data range corresponding to the first task.

[0054] Specifically, the audio acquisition channel captures audio waveforms and performs ASR transcription and keyword recognition on the audio data. Simultaneously, it identifies task-layered keywords based on the audio ASR transcription results, thereby defining the boundaries between the overall task and sub-tasks. The keyword matching rules, confidence thresholds, and keyword-corresponding timestamp extraction methods are as follows: Keyword matching method: The system has two built-in keyword libraries: an overall task keyword library and a sub-task keyword library. A combination of precise text matching and fuzzy matching is used. Exact Match: Matches standard, fixed instruction text such as task start, task end, instruction start, and instruction end; a match is achieved if each word corresponds exactly to the previous one. Fuzzy Match: For scenarios with varying speech speeds or slight pronunciation difficulties, it uses word segmentation similarity matching; a successful match is determined when the word segmentation overlap reaches a preset standard.

[0055] Distinguish matching priorities: Prioritize exact matching, and only enable fuzzy matching if no exact match is found.

[0056] Confidence determination requirements: The ASR engine outputs a confidence score in the range of 0-100 for each recognized text, with two threshold levels set: Overall task boundary keywords (task start or end): Valid identification is only determined if the confidence score is ≥90. Subtask boundary keywords (instruction start or end): Valid identification is only determined if the confidence score is ≥80; identification results below the corresponding threshold are discarded and do not participate in task boundary segmentation.

[0057] How to determine the global timestamp corresponding to a keyword: During audio acquisition, audio segments are output in fixed blocks, and each audio block is bound to a unified global timestamp. After ASR recognition is completed, the global timestamp bound to the audio block containing the identified keyword is used as the time stamp corresponding to the keyword. The rule is as follows: if the keyword appears completely in an audio block, the global timestamp of that audio block is taken as the keyword timestamp; if the keyword spans two audio blocks, the timestamp of the audio block containing the first valid character of the keyword is taken as the keyword timestamp. Finally, this timestamp is the start and end position of the boundary between the total task or sub-task, which is used to divide the data range of the hierarchical task.

[0058] For example, the recognition results of the start and end of the task in the audio data are used as the first recognition result, and the recognition results of the start and end of the instruction are used as the second recognition result.

[0059] The first task is a complete set of long-running robotic operations defined by the task start and task end, and is taken as the overall task. The unified global timestamp at the start of the task is determined as the starting position of the first task, and the unified global timestamp at the end of the task is determined as the ending position of the first task, thus forming the first boundary. The data range corresponding to the first task is determined from the fused multimodal data.

[0060] The second task is an independent sub-action or sub-instruction defined by a set of start and end instructions within the first task. Within the data range of the first task, the unified global timestamps corresponding to the start and end of the instructions are determined as the start and end positions of the second task, forming a second boundary, and the data range of each second task is determined accordingly.

[0061] For example, in the complete operation of a home robot fetching and pouring water, the task start and task end define the first task, and the instruction start and instruction end can sequentially define the two second tasks: the water fetching action and the water pouring action.

[0062] In this optional embodiment, the step of constructing a hierarchical task structure based on the first and second boundaries, determining the execution state of the second task based on the fused multimodal data, and obtaining the hierarchical task result includes: Based on the first boundary, establish the main task directory corresponding to the first task. Combined with the second boundary corresponding to the second task, establish the sub-task directory corresponding to the second task within the main task directory. Associate the data range corresponding to the second task with the corresponding sub-task directory to obtain a hierarchical task structure. Based on the data range corresponding to the second task, extract the state data corresponding to the second task from the fused multimodal data; State recognition is performed on the visual data, tactile data, joint force data, and audio data in the state data to obtain visual information, tactile information, joint information, and audio information; By associating visual information, tactile information, joint information, and audio information, the state determination information corresponding to the second task is obtained; Based on the status determination information, determine the success or failure status of the second task, and associate the success or failure status of the second task with the corresponding sub-task directory in the hierarchical task structure to obtain the hierarchical task result.

[0063] Specifically, a master task directory corresponding to the first task is established based on the first boundary, and a corresponding sub-task directory is established in the master task directory based on the second boundary corresponding to each second task. The data range of each second task is associated with its sub-task directory to obtain a hierarchical task structure.

[0064] For each second task, visual data, tactile data, joint force data, and audio data are extracted from the fused multimodal data within the corresponding data range, and the visual positioning status, tactile steady state status, joint load status, and audio confirmation status are identified respectively to obtain visual information, tactile information, joint information, and audio information.

[0065] It should be noted that this solution integrates four types of information—visual, tactile, joint force, and audio—to determine the success or failure of sub-tasks. It supports both real-time online analysis and offline post-event judgment modes. Audio confirmation is an optional verification item. The judgment standards, thresholds, and fusion rules for various features are unified as follows: Single-class feature judgment criteria: visual positioning status; using the target standard reference three-dimensional coordinates as a reference, calculate the spatial Euclidean error between the actual coordinates of the detected object and the reference; allowable error threshold is 2mm, when the error is ≤2mm, the positioning is judged to be normal, and when it is greater than 2mm, the positioning is abnormal.

[0066] Tactile steady state status; synchronously read two indicators: pressure value of tactile array and effective contact area: if the pressure fluctuation of multiple consecutive frames is <5kPa and the effective contact area is ≥70% of the standard contact area, the tactile steady state is judged to be normal; if the pressure fluctuates drastically or the contact area is insufficient, the tactile sensation is abnormal.

[0067] Joint load status; collect six-dimensional torque time-series data of each joint of the robot and set a fixed overload threshold; if the torque of all joints is lower than the overload threshold and there are no continuous sudden peaks, the joint load is judged to be normal; if the torque continuously exceeds the limit, the load is abnormal.

[0068] Audio confirmation status (optional verification item); verify whether the ASR transcribed text contains preset confirmation keywords and the recognition confidence level is ≥80; if the standard confirmation statement is matched and the confidence level meets the standard, the audio confirmation is deemed valid; if no confirmation statement is recognized or the confidence level is insufficient, it only means that the audio item fails and does not directly determine that the task has failed.

[0069] The system can enable or disable the audio verification branch according to the task configuration, and directly ignore the judgment result of this dimension in scenarios without audio interaction.

[0070] Two judgment working modes: Mode 1: Real-time online analysis (synchronous judgment during data acquisition); Data acquisition and feature recognition run synchronously. After each periodic data packet is generated, multi-feature verification is immediately performed: visual, tactile, and joint force features of the current time period are extracted in real time to complete the basic judgment; if audio verification is enabled, the audio recognition result is read synchronously; if all three basic features (visual, tactile, and joint load) are normal, the current execution status of the subtask is marked as successful in real time; if any basic feature is abnormal, it is marked as a failure immediately, and the abnormality type is recorded and written into the data packet metadata; if audio verification is enabled, the audio confirmation result is stored as an additional tag in the metadata only if all three basic features are qualified, and it is not used as a condition for the success or failure of the task; feedback signals can be output synchronously to prompt the operator whether the current grasping or assembly action is in place, so as to facilitate real-time adjustment on site.

[0071] Mode 2: Offline Post-Event Judgment (Unified Analysis After All Data Collection is Completed); After all complete subtask data is recorded and stored, the full-time fusion multimodal data of the subtask is read for batch analysis: all periodic data packets within the complete time range of the subtask are traversed, and the full-time judgment results of the three basic features of vision, touch, and joint load are summarized; a unified judgment is made based on the frequency and duration of abnormalities of the basic features within the entire time series. If any segment of continuous abnormality exists, the subtask is marked as failed; if there are no abnormalities throughout the process, the task is judged as successful; if there is audio data during the recording process, the audio confirmation result is only stored as additional remarks in the task metadata and does not participate in the success or failure judgment; if there is no audio data, the audio dimension analysis is skipped directly; suitable for batch recording and subsequent unified dataset annotation scenarios, without consuming the computing power of edge devices during collection.

[0072] General rules for comprehensive determination of multi-feature association: Visual positioning, tactile steady state, and joint load are mandatory judgment dimensions, using AND logic: if all three are judged to be normal, the basic state of the subtask is qualified; if any one of the basic features is judged to be abnormal, the subtask is directly marked as failing.

[0073] Audio recognition is an optional additional dimension, used only to supplement the recording of interaction information. It does not participate in the core determination of task success or failure, and does not affect the final success or failure marking result. All abnormal features, audio recognition results, and corresponding start and end times are stored in the data packet metadata for future reference.

[0074] Visual, tactile, joint, and audio information are correlated to form status determination information. Based on this status determination information, the second task is marked as either successful or failed, and this execution status is associated with the corresponding subtask directory to obtain the hierarchical task results. The status determination information is written into the metadata along with the corresponding periodic data packets for subsequent retrieval and filtering.

[0075] S104. Based on the preset packaging cycle, perform integrity verification and differentiation processing on the fused multimodal data, and generate a periodic data package by combining the results of the hierarchical task and the authorized index. In this optional embodiment, the process of performing integrity verification and differentiation processing on the fused multimodal data based on a preset packaging period, and generating a periodic data packet by combining the hierarchical task results and authorization index, includes the following steps: Within a preset packaging period, the fused multimodal data is divided into periods using a preset unified time base to obtain candidate period datasets, and then integrity verification and filtering are performed to obtain the target period dataset. The visual and audio data in the target period dataset are compressed, while the joint force data, tactile data, and motion trajectory data in the target period dataset are preserved with their original precision to obtain differentiated data. Based on the correspondence between the periodic time range corresponding to the target periodic dataset and the second boundary in the hierarchical task results, the second task corresponding to the target periodic dataset is determined, and the execution status of the second task is obtained to obtain the periodic task information. The differentiated data, periodic task information, and authorization index are encapsulated to obtain periodic data packets.

[0076] Specifically, in this embodiment, the unified time base is preferably 0.1s, and 1s is set as the packaging period. The fused multimodal data is divided into continuous 1s candidate period datasets according to the unified global timestamp; the 0.1s unified time base can completely divide the sampling period of each acquisition channel, and when the visual acquisition frame rate is 30Hz, each candidate period dataset should correspond to 30 frames of complete visual data.

[0077] When performing integrity verification on the candidate period dataset, it is verified whether the visual data within a 1-second period contains 30 complete images, and whether the audio data, joint force data, tactile data, and motion trajectory data cover the period and do not have any truncated records; the candidate period dataset that passes the verification is determined as the target period dataset.

[0078] This solution performs integrity checks on the candidate period dataset during the packaging cycle. In addition to checking a fixed number of frames for visual data, audio, joint force, tactile, and motion trajectory modalities are each equipped with independent verification logic. Examples of the verification methods and complete verification processes for each modality are as follows: Visual data verification: Fixed frame count verification. In this embodiment, the visual channel is sampled at 30Hz, and 30 frames of images must be completely contained within a 1-second packetization period; missing frames, image truncation, or loss of grayscale or depth channels are all considered as data incompleteness.

[0079] Audio data verification: waveform continuity verification and sampling duration verification. The audio channel samples at 48kHz, with a fixed theoretical number of audio sampling points per second period; the program traverses the audio blocks, verifying two points: no truncation or gaps at the beginning and end of the audio, and no blank silence segments; the difference between the actual total number of sampling points and the theoretical number of sampling points is within the allowable error range.

[0080] For example, the theoretical number of sampling points in a 1-second cycle is 48,000. If the total number of points read in the audio block is less than 47,900 or there is a long period of zero-value silence in the middle of the audio, the audio data is determined to be incomplete and the dataset of that cycle is removed.

[0081] Joint force data verification: timing continuity verification and numerical validity verification. Joint forces are sampled at 1kHz, and there should be 1000 six-dimensional torque timing records in 1s cycle; two points are verified: the timestamps are continuous without jumps and there are no empty value records; the torque values ​​do not exceed the sensor hardware range and there are no illegal maximum or minimum outliers.

[0082] For example, if there are only 920 torque records in a cycle, or if the torque value of a certain data exceeds the sensor's ±100 N·m range, the joint force data is determined to be incomplete, and the data for the current cycle is discarded.

[0083] Tactile data verification: array integrity verification and time frame count verification. Tactile sampling at 2kHz, with a 1s cycle and a total of 200 frames of pressure matrix; two points are verified: no missing pressure pixels or empty pixels with reading failure in each frame of the tactile array; and no gaps in the timestamps of consecutive frames.

[0084] For example, if a frame of the tactile array has more than 10 empty pixel readings, or only 165 frames of tactile data within a period, the tactile data is determined to be incomplete.

[0085] Motion trajectory data verification: encoder timing continuity verification and motion parameter integrity verification. The trajectory channel is sampled at 200Hz, with a 1s cycle containing 200 sets of trajectory data; two points are verified: each set of data simultaneously contains three types of parameters: joint angle, angular velocity, and displacement, with no missing parameters; the encoder timing has no jumps and no duplicate timestamps.

[0086] For example, if some trajectory records are missing the angular velocity field, or if duplicate global timestamps appear within the same period, the trajectory data verification will fail.

[0087] Taking a 1-second packaging period candidate dataset as an example: The system extracts all five modal data within a 1-second interval based on the global timestamp; it then sequentially performs visual frame count verification, audio waveform point count and continuity verification, joint force timing and range verification, tactile array pixel integrity verification, and trajectory motion parameter integrity verification; if all five modalities pass the verification, the dataset is marked as the target period dataset and enters the differentiated compression and packaging process; if any modality fails the verification, the data for that period is discarded, and the missing modality, anomaly type, and anomaly time interval are recorded in the system log.

[0088] The minimum granularity of the system's underlying layer is fixed at 0.1s, and the default standard packaging cycle is 1s. Differentiated segmentation, verification, and storage strategies are adopted for short tasks with a duration of less than 1s and tasks with a duration of more than 1s but not an integer number of seconds (e.g., 7.2s).

[0089] Preconditions: The minimum duration of a valid data packet is 0.1s; if the duration of a task residual fragment is less than 0.1s, no data packet is generated, it is directly discarded and recorded in the system log.

[0090] I. Short-duration subtask processing with a total duration of 0.1s-1s: After identifying the end boundary of the subtask, there is no need to wait for the full 1s period to be reached. All multimodal data in the memory cache are directly extracted as short-duration candidate datasets.

[0091] Short-term data verification rules: The theoretical sampling amount is calculated using 0.1s as the basic granularity. The expected number of frames and sampling points for each modality are converted according to the actual duration of the task. Matching the standard sampling amount of 1s is not mandatory. Other verification judgment conditions are consistent with those of the standard data packet.

[0092] After successful verification, differential compression is performed to generate short-time data packets. The internal field structure of the data packets is completely consistent with that of standard 1-second data packets. The metadata block_meta is marked with is_short_task=1, and the start and end global timestamps and actual duration of the data packets are recorded synchronously. The file name is prefixed with Short_Block_, and it is stored in the corresponding subtask directory along with regular data packets. The global index is synchronously marked with the short-time task identifier, and data from different subtasks are not merged.

[0093] II. The total task duration is 7.2s (not an integer second long task). The processing logic is as follows: the task is divided into segments based on a 1s base period, with a minimum granularity of 0.1s at the bottom layer. The first 7 complete 1s data segments generate 7 standard periodic data packets. The remaining 0.2s segments at the end are larger than the minimum 0.1s granularity threshold and are extracted independently as short segments.

[0094] Segmented data verification: The first 7 complete 1-second segments are verified using the standard 1-second sampling rate; the last 0.2-second short segment is verified using the theoretical sampling rate calculated at a 0.1-second granularity, following short-time data verification rules. After successful verification, a short-time data packet is generated. Data packet markings distinguish the segments; the 7 standard data packets have no short-time identifier; the last 0.2-second data packet has metadata marked is_short_task=1 and its filename carries the Short_Block identifier; all data packets are bound to the same subtask number, and the metadata records the segmentation order.

[0095] Unified storage logic: Seven standard data packets and one short-time data packet are stored in the same subtask directory, sorted chronologically; a global index associates all segment files with subtask IDs, allowing for the reconstruction of the complete 7.2s task timeline during reading. General fallback rules: The standard and short-time data packets maintain consistent data structures, both containing compressed visual data, compressed audio data, raw joint force data, raw tactile data, raw motion trajectory data, and metadata; data merging across subtasks is strictly prohibited to ensure independent data boundaries for each subtask, facilitating subsequent dataset annotation and trimming; if a segment fails verification, only the current segment is discarded, while other valid segments are saved normally, and the system log records the abnormal time period and failure mode; if the remaining segment duration is less than the 0.1s minimum segment threshold after the subtask ends, the residual data is discarded without triggering the verification process.

[0096] The RGB two-dimensional visual data or the depth visual data obtained by binocular, RGB-D, dToF, or 3D LiDAR in the target periodic dataset are compressed and encoded, and the audio is also compressed and encoded. The joint force data, tactile data, and motion trajectory data are not compressed and are preserved according to their original accuracy, resulting in differentiated data.

[0097] This solution compresses visual and audio data, while retaining the original precision of joint force, tactile, and motion trajectory data without compression. It also distinguishes between ordinary RGB images, various depth visual and audio data, and configures dedicated compression schemes for each, including encoding format, compression parameters, and lossy or lossless attributes.

[0098] Specifically: Ordinary RGB 2D visual data compression, encoding format: JPEG, compression attribute: lossy compression, compression parameters: fixed image quality coefficient 75; chroma subsampling 4:2:0; no alpha channel preserved. Applicable scenarios: ordinary monocular RGB camera capturing color images, balancing storage volume and image recognition accuracy. Depth visual data (binocular / RGB-D / dToF / 3D LiDAR) is processed separately into two categories: Depth grayscale image (depth value mapped to 8-bit grayscale image), encoding format: PNG, compression attribute: lossless compression, parameters: compression level 6, retaining complete grayscale depth mapping information without precision loss. Original depth floating-point 16-bit depth image, encoding format: TIFF lossless compression, compression attribute: lossless compression, parameters: using DEFLATE lossless algorithm, completely retaining millimeter-level original depth values ​​without losing spatial ranging accuracy. Audio data: encoding format: AAC, compression attribute: lossy compression, compression parameters: sampling rate 48kHz, dual-channel, bitrate 128kbps; audio segments are encapsulated in blocks, retaining key features of human voice and environmental interaction audio.

[0099] It should be noted that the timing data of joint force, tactile sensation, and motion trajectory sensing are not compressed in any way, and the original sampled values ​​are stored intact to avoid distortion of key values ​​such as torque, pressure, and motion trajectory. All compression parameters are configurable, and the example provided is the default standard configuration. If higher image or audio fidelity is required, the RGB quality coefficient can be increased and the audio bitrate can be improved. When the dataset storage space is limited, the compression quality can be appropriately reduced.

[0100] Based on the correspondence between the start and end times of the target periodic dataset and each second boundary, the corresponding second task is determined, and the execution status of that second task is read. Differentiated data, second task identifiers, execution status, period start and end times, and authorization indexes are encapsulated into a periodic data packet. This periodic data packet includes six data fields: visual_compressed (compressed visual data), audio_compressed (compressed audio data), joint_force (raw joint force data), tactile (raw tactile data), trajectory (raw motion trajectory data), and block_meta.

[0101] It should be noted that block_meta is used to record metadata such as cycle start and end times, subtask numbers, task success / failure flags, and authorization indexes.

[0102] S105. Based on the results of the hierarchical task and the preset storage mode, the periodic data packets are stored in a structured manner, and based on the first task or the second task, the corresponding multimodal data is read from the structured periodic data packets to obtain the multimodal data acquisition results.

[0103] In this optional embodiment, the step of structuring and storing periodic data packets according to the hierarchical task results and a preset storage mode, and reading the corresponding multimodal data from the structured periodic data packets based on the first task or the second task to obtain the multimodal data acquisition results includes the following steps: From the hierarchical task results, obtain the task identifiers and start and end global times corresponding to the first and second tasks; When the preset storage mode is hierarchical directory storage mode, the main task directory corresponding to the first task and the sub-task directory corresponding to the second task are established according to the task identifier, and the periodic data packets are written to the corresponding sub-task directories; when the preset storage mode is fixed period unified storage mode, the periodic data packets are stored continuously according to the preset packaging period. Based on the start and end global times corresponding to the first and second tasks, a global index is established between the first and second tasks and the periodic data packets, and the periodic data packets and the global index are stored locally and in the cloud. Based on the first or second task, the corresponding start and end global times are obtained from the global index. Periodic data packets whose time ranges overlap with the start and end global times are filtered, and the filtered periodic data packets are subjected to time-series interception and sorting processing to obtain the corresponding multimodal data acquisition results.

[0104] Specifically, in the hierarchical directory storage mode, HDF5 hierarchical storage is adopted, and periodic data packets are stored according to a hierarchical structure of global index, main task folder, sub-task independent directory, and periodic data packets. For example, the global index can be meta / all_task_index.json.

[0105] When a task is detected to start, a main task folder corresponding to the first task is created, such as / task_001. Within the scope of the first task, a subtask directory corresponding to the second task is created for each detected start and end of a set of instructions, such as SubTask_001 or SubTask_002. According to the start and end time of the periodic data packet and the data range of the second task, the periodic data packet is written to the corresponding subtask directory in the form of Block_xxxx.

[0106] Furthermore, each Block_xxxx stores visual_compressed, audio_compressed, joint_force, tactile, trajectory, and block_meta, and each subtask directory uses the same data fields and storage structure.

[0107] In the hierarchical directory storage mode, a master task directory corresponding to the first task is established based on the hierarchical task results, and sub-task directories corresponding to each second task are established within the master task directory. Based on the correspondence between the start and end times of the periodic data packet and the start and end global times of the corresponding second task, the second task to which the periodic data packet belongs is determined, and the periodic data packet is written to the corresponding sub-task directory, so that the periodic data packets corresponding to different second tasks are independent of each other.

[0108] In the fixed-period unified storage mode, data packets for each period are stored continuously using a preset packaging period as the basic storage unit, without creating separate directories for the first or second task. In this embodiment, the preset packaging period is preferably 1 second. The global index records the task identifier, start and end global time, and correspondence between the first and second tasks and the data packets for each period. In the hierarchical directory storage mode, periodic data packets do not cross the boundary of the second task; in the fixed-period unified storage mode, periodic data packets can cross the boundary of adjacent second tasks, but the metadata of each periodic data packet records the corresponding start and end global time of the period.

[0109] When it is necessary to read the multimodal data corresponding to the first or second task, the global index is queried according to the corresponding task identifier to obtain the start and end global time corresponding to the first or second task; the periodic data packets stored continuously under the fixed periodic unified storage mode are traversed, and periodic data packets whose periodic time range overlaps with the corresponding start and end global time are filtered; according to the start and end global time, the filtered periodic data packets are truncated in time sequence, and multimodal data samples located outside the corresponding time range are removed; then the truncated multimodal data is sorted according to the unified global timestamp to obtain the multimodal data acquisition results corresponding to the first or second task.

[0110] The hierarchical directory storage mode facilitates direct location of periodic data packets according to task hierarchy, while the fixed-period unified storage mode eliminates the need for frequent creation and retrieval of multi-level task directories, enabling rapid retrieval of continuous task segments based on the global index and start and end global times. Both storage modes store periodic data packets and global indexes locally and in the cloud.

[0111] The main task folder, which contains the directories of each subtask and the periodic data package, is stored locally and in the cloud. The data is then stored in an authorized encrypted partition or an anonymized research partition based on the authorized index, while the full-link audit log is retained. After the task is completed, the global index is updated to obtain the multimodal data collection results.

[0112] Using the above methods, for example in the scenario of a home robot fetching and pouring water, after the operator completes the authorization verification, the system unlocks five data acquisition channels and simultaneously collects visual data, audio data, joint force data, tactile data, and motion trajectory data.

[0113] When a task is detected to start, a master task directory corresponding to the first task is created; when the first set of instructions is detected to start and end, the second task corresponding to the water-taking action is determined, and a corresponding sub-task directory is created; when the second set of instructions is detected to start and end, the second task corresponding to the water-pouring and placing action is determined, and a corresponding sub-task directory is created.

[0114] During data acquisition, periodic data packets are generated at 1-second packaging intervals. Based on the start and end times of each periodic data packet, they are written to the sub-task directory corresponding to the second task. When the task is detected to be finished, the current first task is terminated, and the overall task directory containing the sub-task directories and periodic data packets is stored, yielding the multimodal data acquisition results for this first task. For subsequent identical or different robot tasks, the above acquisition process is repeated, and corresponding overall task directories and sub-task directories are established respectively to form multiple sets of multimodal data acquisition results.

[0115] Figure 2 An embodiment of a robot multimodal data synchronous fusion acquisition system of the present invention is shown.

[0116] In this optional embodiment, the robot multimodal data synchronous fusion acquisition system includes: The synchronous acquisition module 201 is used to generate an authorization index based on authorization verification, synchronously acquire initial multimodal data according to the authorization index, and bind a unified global timestamp to obtain target multimodal data; wherein, the initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data; The multimodal fusion processing module 202 is used to perform time-series alignment of target multimodal data based on a unified global timestamp, map the time-series aligned target multimodal data through a preset coordinate transformation relationship, and perform sensitive information processing on the coordinate-mapped target multimodal data according to the authorized index to obtain fused multimodal data. The hierarchical task processing module 203 is used to perform hierarchical task segmentation on the fused multimodal data using audio data, determine the first boundary of the first task and the second boundary of the second task; construct a hierarchical task structure based on the first boundary and the second boundary, and determine the execution status of the second task based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; The periodic data packet generation module 204 is used to perform integrity verification and differentiation processing on the fused multimodal data based on a preset packaging period, and generate periodic data packets by combining the results of hierarchical tasks and the authorization index. The structured storage module 205 is used to perform structured storage of periodic data packets according to the results of the hierarchical tasks and the preset storage mode, and to read the corresponding multimodal data from the structured periodic data packets based on the first task or the second task to obtain the multimodal data acquisition results.

[0117] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0118] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0120] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0122] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method for synchronous fusion and acquisition of multimodal data from a robot, characterized in that, include: An authorization index is generated based on authorization verification. Initial multimodal data is collected synchronously according to the authorization index and bound to a unified global timestamp to obtain target multimodal data. The initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data. Based on a unified global timestamp, the target multimodal data is time-series aligned, and the time-series aligned target multimodal data is mapped through a preset coordinate transformation relationship; and based on the authorized index, sensitive information processing is performed on the coordinate-mapped target multimodal data to obtain fused multimodal data. Using audio data, the fused multimodal data is segmented into hierarchical tasks to determine the first boundary of the first task and the second boundary of the second task; a hierarchical task structure is constructed based on the first and second boundaries, and the execution status of the second task is determined based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; Based on a preset packaging cycle, integrity verification and differentiation processing are performed on the fused multimodal data, and periodic data packets are generated by combining the results of hierarchical tasks and authorized indexes. Based on the results of the hierarchical tasks and the preset storage mode, the periodic data packets are stored in a structured manner, and based on the first task or the second task, the corresponding multimodal data is read from the structured periodic data packets to obtain the multimodal data acquisition results.

2. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 1, characterized in that, The process involves generating an authorization index based on authorization verification, synchronously collecting initial multimodal data according to the authorization index, and binding it with a unified global timestamp to obtain the target multimodal data, which includes: Obtain notification logs, identity verification information, liveness verification information, confirmation audio information, and electronic signature data; The identity verification information, liveness verification information, and confirmation audio information are verified to obtain the authorization verification result; based on the authorization verification result, the notification letter viewing log and electronic signature data are stored as evidence to generate an authorization index; The authorization index is validated to obtain the permission validation result. The permission validation result is then used to drive the acquisition channel to collect data and obtain initial multimodal data. Based on the acquisition time corresponding to the acquisition channel, a unified global timestamp is bound to the visual data, audio data, joint force data, tactile data and motion trajectory data in the initial multimodal data to obtain the target multimodal data.

3. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 2, characterized in that, The process of validating the authorized index to obtain the permission verification result, and then using the permission verification result to drive the acquisition channel to acquire data and obtain initial multimodal data, includes the following steps: When the permission verification result indicates that the authorized index is valid, the preset pulse signal is unlocked; when the permission verification result indicates that the authorized index is invalid, the preset pulse signal is cut off and the current multimodal data acquisition process is terminated. Using the unlocked pulse signal, data acquisition is driven through the visual acquisition channel, audio acquisition channel, joint force acquisition channel, tactile acquisition channel and motion trajectory acquisition channel to obtain visual data, audio data, joint force data, tactile data and motion trajectory data respectively; Visual data, audio data, joint force data, tactile data, and motion trajectory data are aggregated and processed to obtain initial multimodal data.

4. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 1, characterized in that, The step of performing time-series alignment of target multimodal data based on a unified global timestamp, and mapping the time-series aligned target multimodal data through a preset coordinate transformation relationship, includes: Read the unified global timestamps corresponding to visual data, audio data, joint force data, tactile data and motion trajectory data in the target multimodal data respectively. Based on the unified global timestamps, establish the temporal correspondence between visual data, audio data, joint force data, tactile data and motion trajectory data to obtain the temporally aligned target multimodal data. Based on the preset coordinate transformation relationship, the visual target position corresponding to the time-aligned visual data, the joint force application point corresponding to the joint force data, and the tactile contact point corresponding to the tactile data are mapped to the robot base coordinate system respectively, so as to obtain the coordinate-mapped visual data, joint force data, and tactile data. Based on the time correspondence, the coordinate-mapped visual data, joint force data, and tactile data are associated with the time-aligned audio data and motion trajectory data to obtain the coordinate-mapped target multimodal data.

5. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 4, characterized in that, The process involves performing sensitive information processing on the target multimodal data after coordinate mapping, based on the authorized index, to obtain fused multimodal data, including: Based on the authorized index, determine the information processing methods corresponding to the visual and audio data in the target multimodal data after coordinate mapping; When the information processing method is raw data retention, visual data and audio data are identified as the primary target data; When the information processing method is desensitization, visual data and audio data are desensitized to obtain the second target data. Based on the time correspondence, the first target data or the second target data is associated with the coordinate-mapped joint force data, tactile data, and time-aligned motion trajectory data to obtain fused multimodal data.

6. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 1, characterized in that, The step of using audio data to perform hierarchical task segmentation of the fused multimodal data and determining the first boundary of the first task and the second boundary of the second task includes: Keyword recognition is performed on the audio data to obtain the first recognition result and the second recognition result; Based on the first recognition result, the start and end positions of the first task are determined, the first boundary is obtained, and the data range corresponding to the first task is determined from the fused multimodal data according to the first boundary. Within the data range corresponding to the first task, the start and end positions of the second task are determined based on the second recognition result, and the second boundary corresponding to each second task is obtained. Based on the second boundary, the data range corresponding to the second task is determined from the data range corresponding to the first task.

7. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 6, characterized in that, The process of constructing a hierarchical task structure based on the first and second boundaries, and determining the execution status of the second task based on fused multimodal data, to obtain the hierarchical task results includes: Based on the first boundary, establish the main task directory corresponding to the first task. Combined with the second boundary corresponding to the second task, establish the sub-task directory corresponding to the second task within the main task directory. Associate the data range corresponding to the second task with the corresponding sub-task directory to obtain a hierarchical task structure. Based on the data range corresponding to the second task, extract the state data corresponding to the second task from the fused multimodal data; State recognition is performed on the visual data, tactile data, joint force data, and audio data in the state data to obtain visual information, tactile information, joint information, and audio information; By associating visual information, tactile information, joint information, and audio information, the state determination information corresponding to the second task is obtained; Based on the status determination information, determine the success or failure status of the second task, and associate the success or failure status of the second task with the corresponding sub-task directory in the hierarchical task structure to obtain the hierarchical task result.

8. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 1, characterized in that, The process of generating a periodic data package based on a preset packaging cycle, performing integrity verification and differentiation processing on the fused multimodal data, and combining the results of hierarchical tasks and authorized indexes, includes the following steps: Within a preset packaging period, the fused multimodal data is divided into periods using a preset unified time base to obtain candidate period datasets, and then integrity verification and filtering are performed to obtain the target period dataset. The visual and audio data in the target period dataset are compressed, while the joint force data, tactile data, and motion trajectory data in the target period dataset are preserved with their original precision to obtain differentiated data. Based on the correspondence between the periodic time range corresponding to the target periodic dataset and the second boundary in the hierarchical task results, the second task corresponding to the target periodic dataset is determined, and the execution status of the second task is obtained to obtain the periodic task information. The differentiated data, periodic task information, and authorization index are encapsulated to obtain periodic data packets.

9. The method for synchronous fusion and acquisition of multimodal data of a robot according to claim 1, characterized in that, The process of structuring and storing periodic data packets according to the results of hierarchical tasks and a preset storage mode, and reading corresponding multimodal data from the structured periodic data packets based on the first task or the second task to obtain multimodal data acquisition results includes the following steps: From the hierarchical task results, obtain the task identifiers and start and end global times corresponding to the first and second tasks; When the preset storage mode is hierarchical directory storage mode, the main task directory corresponding to the first task and the sub-task directory corresponding to the second task are established according to the task identifier, and the periodic data packets are written to the corresponding sub-task directories; when the preset storage mode is fixed period unified storage mode, the periodic data packets are stored continuously according to the preset packaging period. Based on the start and end global times corresponding to the first and second tasks, a global index is established between the first and second tasks and the periodic data packets, and the periodic data packets and the global index are stored locally and in the cloud. Based on the first or second task, the corresponding start and end global times are obtained from the global index. Periodic data packets whose time ranges overlap with the start and end global times are filtered, and the filtered periodic data packets are subjected to time-series interception and sorting processing to obtain the corresponding multimodal data acquisition results.

10. A robot multimodal data synchronous fusion acquisition system, characterized in that, include: The synchronous acquisition module is used to generate an authorization index based on authorization verification, synchronously acquire initial multimodal data according to the authorization index, and bind a unified global timestamp to obtain target multimodal data; wherein, the initial multimodal data includes visual data, audio data, joint force data, tactile data, and motion trajectory data; The multimodal fusion processing module is used to perform time-series alignment of target multimodal data based on a unified global timestamp, map the time-series aligned target multimodal data through a preset coordinate transformation relationship, and perform sensitive information processing on the coordinate-mapped target multimodal data according to the authorized index to obtain fused multimodal data. The hierarchical task processing module is used to perform hierarchical task segmentation on the fused multimodal data using audio data, determine the first boundary of the first task and the second boundary of the second task; construct a hierarchical task structure based on the first and second boundaries, and determine the execution status of the second task based on the fused multimodal data to obtain the hierarchical task result; wherein, the first task includes at least one second task; The periodic data packet generation module is used to perform integrity verification and differentiation processing on the fused multimodal data based on a preset packaging period, and generate periodic data packets by combining the results of hierarchical tasks and authorized indexes. The structured storage module is used to perform structured storage of periodic data packets according to the results of hierarchical tasks and preset storage modes, and to read the corresponding multimodal data from the structured periodic data packets based on the first task or the second task to obtain the multimodal data acquisition results.

Images