A mobile dual-arm robot multi-modal data acquisition method and system

Abstract

The application discloses a kind of mobile dual-arm robot multi-modal data acquisition method and system, belong to robot data acquisition technical field.The method includes: constructing and executing two-stage acquisition process, first stage is carried out by traversing random seed task planning and execution attempt, and the target seed of task execution success and corresponding motion trajectory are screened out;Second stage is determined by using target seed reconstruction to simulation environment, and open loop replay is carried out by loading motion trajectory.In the process of open loop replay, the synchronous acquisition of multi-modal data is triggered with fixed time step.In the acquisition of spatial audio data, the physical process of sound propagation is rendered in real time by time step using spatial audio rendering model.The base motion increment and dual-arm joint action are spliced to construct joint action vector.Based on the method, the corresponding system is also proposed, and the application realizes complete acquisition of multi-modal data, accurate time synchronization and real-time rendering of high-fidelity spatial audio.

Description

Technical Field

[0001] This invention belongs to the field of robot data acquisition technology, and specifically relates to a method and system for multimodal data acquisition of a mobile dual-arm robot. Background Technology

[0002] In the current stage of development of embodied intelligence and robotics, high-quality multimodal datasets are crucial for advancing algorithmic research.

[0003] However, traditional data acquisition methods suffer from the following key problems. First, acquiring base motion data is difficult. Most current dual-arm robot datasets only record the motion information of the arm joints, lacking motion data of the moving base. For a mobile robotic arm, the displacement and orientation of the base in the horizontal plane are essential for a complete motion representation. The lack of these degrees of freedom makes the motion description of the dual-arm robot incomplete, making it difficult to support complex task learning involving base navigation. Second, temporal synchronization of modal data is difficult. When acquiring multiple data modalities including vision, audio, joints, and pose, the sampling rates and time steps of different modalities are often inconsistent, leading to difficulties in data alignment. In particular, the sampling rate of audio signals is much higher than that of visual images, making accurate alignment a technical challenge—a typical multi-source heterogeneous data synchronization problem in electronic digital data processing. Third, the realism of spatial audio acquisition is insufficient. When acquiring audio in a simulation environment, simple audio recording methods cannot accurately simulate acoustic effects in the real environment, such as room reverberation, sound attenuation, and delay changes caused by microphone position changes. Fourth, difficulties in environmental reproduction lead to low data acquisition efficiency. Random factors in the environment cause the initial conditions for each task execution to be different. Using a single-stage acquisition method requires independent task planning for each data sample, resulting in low acquisition efficiency and difficulty in obtaining sufficient effective data.

[0004] Therefore, existing technologies cannot solve the above problems simultaneously, and there is an urgent need for a multimodal data acquisition method and system that can completely capture base motion, achieve multimodal data time synchronization, acquire spatial audio with high fidelity, and improve acquisition efficiency. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a multimodal data acquisition method and system for a mobile dual-arm robot. This achieves complete acquisition of the three-degree-of-freedom motion of the base, precise time synchronization of multimodal data, real-time rendering of high-fidelity spatial audio, and a significant improvement in acquisition efficiency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes a method for multimodal data acquisition of a mobile dual-arm robot, comprising the following steps: A two-stage acquisition process is constructed and executed. In the first stage, random seeds are traversed to perform task planning and execution attempts in the simulation environment, and target seeds and corresponding motion trajectories for successful task execution are selected. In the second stage, the simulation environment is deterministically reconstructed using the target seeds, and the motion trajectories are loaded for open-loop replay. During the open-loop replay process, multimodal data is synchronously acquired at fixed time steps; the multimodal data includes: the three-degree-of-freedom motion increments of the robot base, the joint movements of the robot's two arms, visual image data, and spatial audio data; The spatial audio data is collected, and a preset spatial audio rendering model is used to render the physical process of sound propagation step by step in real time based on the real-time pose of the virtual microphone on the robot. By dimensionally splicing the three-degree-of-freedom motion increments of the robot base with the joint movements of the two arms, a joint motion vector describing the motion of all controllable degrees of freedom of the robot body within a single time step is constructed.

[0007] Furthermore, in the first stage, by traversing random seeds, task planning and execution attempts are performed in the simulation environment to select the target seeds for successful task execution and the corresponding motion trajectories. Specifically: Set different random seeds; For each seed, initialize all random number generators and randomly generate an initial environment based on that seed, including object position, lighting, and background texture; In the initial environment, the motion trajectory planning for base navigation and dual-arm manipulation is performed, and the task is attempted to be executed according to the planning results; If the task is successfully executed, the seed will be designated as the target seed, and the navigation trajectory and the dual-arm manipulation trajectory generated by the motion trajectory planning will be saved to the memory. Repeat the above process until the predetermined number of target seeds have been accumulated.

[0008] Furthermore, in the second stage, the simulation environment is deterministically reconstructed using the target seed, and the motion trajectory is loaded for open-loop replay, specifically as follows: Read the target seed list; For each target seed, use that seed again to initialize all random number generators to reproduce the initial environment that corresponds to that seed and is exactly the same as the first stage. Load the navigation trajectory and dual-arm manipulation trajectory corresponding to the seed from memory; The planning function is turned off, and the navigation trajectory and dual-arm manipulation trajectory are directly loaded and replayed. At the same time, multimodal data are collected synchronously during the replay process.

[0009] Furthermore, the planning of the motion trajectory includes the following steps: Perform a grid search in the configuration space to generate a high-level path from the starting position to the target position; Apply kinematic constraints to the high-level path to perform velocity planning and determine the allowable speed on the path; The path is time-parameterized to generate a sequence of trajectory points with timestamps.

[0010] Furthermore, the fixed time step is synchronized with the physical simulation step and the audio sampling step.

[0011] Furthermore, the physical process of sound propagation includes: Direct acoustic path delay and attenuation calculated based on the time-varying distance between the sound source and the microphone; The synthesis of early reflections and late reverberation determined by the geometric reflection characteristics of the room; where early reflections refer to reflections within a preset time length, and late reverberation refers to reverberation beyond the preset time length; And the superposition of environmental noise that conforms to spatial correlation characteristics.

[0012] Furthermore, the direct acoustic path delay is processed using a fractional delay interpolation method; The late reverberation was processed by stream convolution using an overlap-addition method. The ambient noise is colored noise with 1 / f power spectrum characteristics, and there is a preset spatial correlation coefficient between the noise received by the two virtual microphones.

[0013] Furthermore, the three-degree-of-freedom motion increments of the robot base are dimensionally concatenated with the joint movements of the two arms to construct a joint motion vector describing the robot body's motion across all controllable degrees of freedom within a single time step. Specifically, this includes: Obtain the current pose and the previous frame pose of the robot base, and calculate the relative transformation between the two; Three increments are extracted from the relative transformation: Axial displacement increment , Axial displacement increment and the increment of the steering angle about the vertical axis ; The , and The motion vectors are concatenated with the 14-dimensional joint motion vectors of both arms to form a 17-dimensional joint motion vector. The 14-dimensional joint motion vectors include motion information of 6 joints of the left arm, 1 degree of freedom of the left gripper, 6 joints of the right arm, and 1 degree of freedom of the right gripper.

[0014] Furthermore, the method also includes: after completing a single motion sequence, stacking the multimodal data of all acquired time frames along the time dimension and storing it in a structured file format; wherein, the joint action vector is organized in the structured file as a two-dimensional matrix of the number of time frames and the vector dimension.

[0015] Secondly, this invention also proposes a multimodal data acquisition system for a mobile dual-arm robot, comprising: The robot itself is equipped with a base, two arms, at least two virtual microphones, and at least one camera; The processor, built into the robot body, is configured to execute the aforementioned method for multimodal data acquisition of a mobile dual-arm robot. The memory is used to store the target seed, the motion trajectory, and the acquired multimodal data.

[0016] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a multimodal data acquisition method and system for a mobile dual-arm robot, belonging to the field of robot data acquisition technology. The method includes the following steps: constructing and executing a two-stage acquisition process. In the first stage, by traversing random seeds, task planning and execution attempts are performed in a simulation environment to select target seeds for successful task execution and their corresponding motion trajectories. In the second stage, the simulation environment is deterministically reconstructed using the target seeds, and the motion trajectories are loaded for open-loop replay. During the open-loop replay, multimodal data is synchronously acquired at fixed time steps. The multimodal data includes: three-degree-of-freedom motion increments of the robot base, joint movements of the robot's dual arms, visual image data, and spatial audio data. The spatial audio data is acquired, and a preset spatial audio rendering model is used to render the physical process of sound propagation in real-time, step-by-step, based on the real-time pose of the virtual microphone on the robot. The three-degree-of-freedom motion increments of the robot base and the joint movements of the dual arms are dimensionally concatenated to construct a joint motion vector describing the robot's body's motion of all controllable degrees of freedom within a single time step. Based on this method, a corresponding system is also proposed. This invention achieves complete acquisition of the three-degree-of-freedom motion of the base, precise time synchronization of multimodal data, real-time rendering of high-fidelity spatial audio, and a significant improvement in acquisition efficiency.

[0017] This invention employs an incremental representation method to calculate the relative transformation between the robot's base pose and the previous frame pose at each time step. It extracts the X-axis displacement increment, Y-axis displacement increment, and steering angle increment around the vertical axis. These three increments are then concatenated with the 14-dimensional joint motion vectors of the two arms to form a 17-dimensional joint motion vector. This technical solution is the first to completely record the three-degree-of-freedom motion information of the base in dual-arm robot data acquisition, enabling subsequent learning algorithms to recover the complete motion trajectory through integral increments. The use of incremental representation instead of absolute position representation avoids accumulated errors and exhibits good numerical stability. The 17-dimensional joint motion vector comprehensively describes all controllable degrees of freedom motion of the robot body within a single time step, providing a complete data foundation for learning complex tasks involving base navigation and dual-arm collaboration.

[0018] This invention employs a fixed time step that is synchronized with both the physical simulation step and the audio sampling step, ensuring that the physical simulation, audio signal rendering, and acquisition all occur on the same time base. This technical solution guarantees that within each fixed time step, physical simulation, audio rendering, and multimodal data acquisition are triggered synchronously, and all data have a unified timestamp reference. The audio signal sampling rate is precisely matched to the fixed time step, and the time span of the audio frame is equal to the physical simulation step, fundamentally solving the problem of cross-modal data alignment. The acquired multimodal data can be directly used for subsequent joint training without additional data preprocessing and alignment operations.

[0019] This invention integrates a spatial audio rendering model, which renders the physical process of sound propagation in real time, step-by-step, based on the real-time pose of a virtual microphone on a robot. For direct sound path calculation, the world coordinates of the virtual microphone are determined in real time, the Euclidean distance to the sound source is calculated, the propagation delay is calculated based on the speed of sound, and fractional delay interpolation is used to avoid signal distortion caused by discrete sampling. For reflection and reverberation synthesis, early reflections within a preset time length are treated as resolvable paths, and their delays and attenuations are calculated separately. Late reverberation beyond the preset time length is processed using an overlapping addition method with streaming convolution to simulate the statistical characteristics of complex multiple reflections. For noise addition, colored noise with 1 / f power spectral characteristics is added, and preset spatial correlation coefficients are set for the noise from the two virtual microphones to simulate noise characteristics in a real environment. This technical solution enables the audio signal to change dynamically with the robot's movement, and the delay caused by changes in microphone position is accurately simulated; the geometric reflection characteristics of the room determine the reverberation effect, giving the collected audio data a realistic sense of presence and space; fractional delay interpolation ensures the continuity and fault-free nature of the audio signal and avoids artificial noise; the collected spatial audio data can be used to train embodied intelligent models with auditory perception capabilities.

[0020] This invention employs a two-stage deterministic acquisition framework. In the first stage (planning and generation stage), different random seeds are traversed, and task planning and execution attempts are performed in a randomly initialized simulation environment. Target seeds and their corresponding motion trajectories that successfully complete the task are selected and saved to memory. In the second stage (high-fidelity acquisition stage), the exact same simulation environment is recreated using the target seeds recorded in the first stage. The saved motion trajectories are loaded and replayed in an open-loop manner, while high-fidelity multimodal data acquisition is performed during the replay process. This technical solution ensures that the deterministic reconstruction makes the environment in the second stage identical to that in the first stage, eliminating the impact of environmental randomness on data acquisition; open-loop replay avoids redundant planning, significantly improving data acquisition efficiency; the acquired multimodal data completely corresponds to the successful trajectories in the first stage, ensuring data quality consistency and reliability; the two-stage framework decouples exploratory planning from deterministic acquisition, guaranteeing both task success rate and data output.

[0021] The dataset collected in this invention can be widely used in fields such as reinforcement learning model training, behavior cloning and imitation learning, embodied AI basic model training, and navigation and manipulation research of mobile robotic arms, and has significant industrial practical value. Attached Figure Description

[0022] Figure 1 This is a flowchart of a multimodal data acquisition method for a mobile dual-arm robot proposed in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating a specific example of a multimodal data acquisition method for a mobile dual-arm robot proposed in Embodiment 2 of the present invention. Figure 3 This is a flowchart of spatial audio rendering proposed in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of a multimodal data acquisition system for a mobile dual-arm robot proposed in Embodiment 3 of the present invention. Detailed Implementation

[0023] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0024] Example 1 Embodiment 1 of this invention proposes a multimodal data acquisition method for a mobile dual-arm robot, which solves the problems of missing base motion data, asynchronous multimodal data time, insufficient spatial audio authenticity, and low acquisition efficiency caused by random environment in the existing mobile dual-arm robot data acquisition.

[0025] Figure 1 This is a flowchart of a multimodal data acquisition method for a mobile dual-arm robot proposed in Embodiment 1 of the present invention; In step S1, a two-stage acquisition process is constructed and executed. The first stage involves iterating through random seeds, performing task planning and execution attempts in the simulation environment, and selecting target seeds and their corresponding motion trajectories for successful task execution. The second stage uses the target seeds to perform deterministic reconstruction of the simulation environment and loads the motion trajectories for open-loop replay. Before executing step S1, it first checks whether a saved successful seed record file exists. If it does not exist, it proceeds to the first stage; if it does exist, it jumps to the second stage.

[0026] In the first phase (planning and generation phase), the following operations are performed: Set a series of different random seeds, such as seed 0 to N-1. For each seed, first initialize all random number generators using that seed, including pseudo-random number generators, random modules of deep learning frameworks, and random modules of standard libraries, to ensure that the same seed will produce a completely deterministic random sequence.

[0027] Then, an initial environment is randomly generated based on the seed. This initial environment includes: the position coordinates of the target object, the intensity and direction of ambient lighting, and the category and parameters of the room background texture. The processor loads a physics engine (such as the SAPIEN physics engine) to create a 3D scene containing basic elements such as tables and light sources, and loads a dual-arm robot model with 14 controllable degrees of freedom (6 joints in the left arm, 1 degree of freedom in the left gripper, 6 joints in the right arm, and 1 degree of freedom in the right gripper).

[0028] The processor also initializes three cameras, located at the robot's head, left wrist, and right wrist, respectively, with each camera set to a preset resolution (e.g., 1280×720 pixels). Simultaneously, it initializes the spatial audio rendering model, creates two virtual microphones located at the robot's left and right ears, loads the room acoustic model, and initializes noise parameters.

[0029] Path planning for base navigation and dual-arm manipulation is performed in the initial environment. Specifically, a random target position is first generated, and then a trajectory planning algorithm is used for planning: First, a grid search (such as the A* search algorithm) is performed in the configuration space to generate a high-level path from the starting position to the target position; second, kinematic constraints (including angular velocity constraints, lateral acceleration constraints, and longitudinal acceleration constraints) are applied to the high-level path to perform velocity planning and determine the maximum allowable speed at each point on the path; third, the path is time-parameterized to generate a sequence of trajectory points with timestamps, each trajectory point containing a timestamp, X coordinate, Y coordinate, heading angle, linear velocity, and angular velocity information. For dual-arm manipulation, the processor generates predefined manipulation trajectories, including the opening and closing of the gripper, the movement of the robotic arm to the pre-grasping position, the grasping action, and the manipulation action.

[0030] The robot attempts to execute the task according to the planning results. If the task is successfully executed (i.e., the robot successfully completes navigation and performs the specified manipulation task, such as clicking an alarm clock or grasping an object), the seed is designated as the target seed, and the planned navigation trajectory and the dual-arm manipulation trajectory are saved to the memory. If the task fails (e.g., the object is unstable, a collision occurs, or the task is not completed), the attempt is discarded, and the next seed is processed.

[0031] Repeat the above process until the predetermined number of target seeds have been accumulated.

[0032] In the second stage (high-fidelity acquisition stage), the processor performs the following operations: Read the list of target seeds from memory. For each target seed, use that seed again to initialize all random number generators to reproduce the initial environment corresponding to that seed, which is exactly the same as the first stage, including the precise position of objects, lighting parameters, background texture, and the initial velocity of objects, etc.

[0033] Load the navigation trajectory and dual-arm manipulation trajectory corresponding to the seed from memory, and set the planning flag to the off state, indicating that the loaded trajectory is used directly instead of replanning.

[0034] The navigation trajectory and the dual-arm manipulation trajectory are then loaded and replayed, while multimodal data are simultaneously acquired during the replay process.

[0035] In step S2, during the open-loop playback process, the synchronous acquisition of multimodal data is triggered with a fixed time step; the multimodal data includes: the three-degree-of-freedom motion increment of the robot base, the joint movements of the robot's two arms, visual image data, and spatial audio data; The fixed time step described in this application is synchronized with the physical simulation step and the audio sampling step. Specifically, the physical simulation step is set to T milliseconds, and the audio sampling rate is set to F hertz, so that the number of audio samples generated within each fixed time step is a preset fixed value, and the time span of the audio frame is equal to the physical simulation step.

[0036] In each physics simulation step, the processor performs the following operations: updates the position and orientation of the robot base, sets the robot base according to the loaded trajectory points; calls the step function of the physics engine to advance the physics simulation; updates the renderer to synchronize with the latest pose of the camera; and calls the spatial audio rendering model to perform audio rendering.

[0037] The processor triggers multimodal data acquisition once according to a preset sampling interval (e.g., every K physical steps). The acquired multimodal data includes: the three-degree-of-freedom motion increments of the robot base ( Axial displacement increment , Axial displacement increment and the increment of the steering angle about the vertical axis The data includes 14-dimensional joint movements of the robot's two arms (6 joint angles of the left arm, opening and closing of the left gripper, 6 joint angles of the right arm, and opening and closing of the right gripper), visual image data (RGB images from multiple cameras), and spatial audio data (audio signals from two virtual microphones).

[0038] In step S3, the spatial audio data is collected, and a preset spatial audio rendering model is used to render the physical process of sound propagation step by step in real time based on the real-time pose of the virtual microphone on the robot. The physical processes of sound propagation executed by the spatial audio rendering model include: First, the direct acoustic path delay and attenuation are calculated based on the time-varying distance between the sound source and the microphone. The processor determines the real-time position of the two virtual microphones in the world coordinate system, calculates the Euclidean distance from each microphone to the sound source, calculates the sound wave propagation delay based on the speed of sound, and uses a fractional delay interpolation method (such as Lagrange polynomial interpolation) to process the delay with subsample precision. At the same time, the amplitude attenuation of the sound wave is calculated in real time based on the distance.

[0039] Second, the synthesis of early reflections and late reverberation is determined by the room's geometric reflection characteristics. Early reflections refer to reflections within a preset time length (e.g., L milliseconds). The processor calculates the propagation delay and amplitude attenuation of multiple reflection paths and superimposes them onto the signal. Late reverberation refers to reverberation beyond the preset time length. The processor uses an overlapping addition method for streaming convolution processing to simulate the statistical characteristics of complex multiple reflections.

[0040] Third, the superposition of environmental noise conforming to spatial correlation characteristics. The processor generates colored noise with a power spectrum following the 1 / f characteristic, sets a preset spatial correlation coefficient between the noise signals of the two virtual microphones, and controls the overall signal-to-noise ratio to a preset value.

[0041] The output of the spatial audio rendering model is a two-dimensional array. The first dimension corresponds to the number of microphones (2), and the second dimension corresponds to the number of audio samples in the current time step. This output is precisely aligned with the physical simulation step size in time.

[0042] In step S4, the three-degree-of-freedom motion increments of the robot base are dimensionally spliced ​​with the joint movements of the two arms to construct a joint motion vector describing the motion of all controllable degrees of freedom of the robot body within a single time step.

[0043] The processor maintains two key variables in the robot's base tracking module: the current base pose and the base pose of the previous frame. At each time step, the processor calculates the relative transformation between the current pose and the previous pose. Three increments are extracted from this relative transformation: Axial displacement increment : Represents the distance the base moves in the forward or backward direction, in meters. This increment comes directly from the first component of the relative pose vector.

[0044] Axial displacement increment This represents the distance the base moves to the left or right, in meters. This is a significant improvement over traditional methods, which often neglect this lateral movement. This increment comes from the second component of the relative pose vector.

[0045] Increment of steering angle about the vertical axis This represents the change in angle of rotation of the base around the vertical axis, in radians. The increment of the steering angle is calculated from the quaternion of the relative transformation using inverse trigonometric functions.

[0046] The processor will , , The motion vector is concatenated with the 14-dimensional joint motion vectors of the two arms (6 left arm joints, 1 left gripper, 6 right arm joints, and 1 right gripper) to form a 17-dimensional joint motion vector. This vector completely records the robot's motion information in all degrees of freedom within that time step.

[0047] In step S5, after completing a single motion sequence, the multimodal data of all time frames acquired are stacked along the time dimension.

[0048] Specifically, the processor scans all frame data files in the cache directory, arranges them in order of frame index, and reads the RGB image, depth map, segmentation map, joint motion, end pose, audio signal and other data in each file in turn, and stacks these data into a multidimensional array along the time dimension.

[0049] The processor stores the stacked data in a structured file format (such as HDF5). The joint action vector is organized in the structured file as a two-dimensional matrix of time frame number and vector dimension, while other multimodal data are stored separately according to their respective data dimensions. The processor can optionally use compression algorithms (such as gzip compression) to compress the data to save storage space.

[0050] Embodiment 1 of this invention presents a complete process for implementing a multimodal data acquisition method for a mobile dual-arm robot. This method achieves complete acquisition of the three-degree-of-freedom motion of the base, precise time synchronization of multimodal data, real-time rendering of high-fidelity spatial audio, and a significant improvement in acquisition efficiency.

[0051] Example 2 Embodiment 2 of this invention provides a specific application example of Embodiment 1. A mobile dual-arm robot needs to navigate to a table in a simulated room and then use its arms to click an alarm clock. The robot's base has three degrees of freedom: forward, lateral, and turning. The arms have 14 controllable degrees of freedom (6 left arm joints, 1 left gripper, 6 right arm joints, and 1 right gripper). An RGB camera is mounted on the robot's head, left wrist, and right wrist, and a virtual microphone is mounted on each of the robot's left and right ears. The room contains a table with an alarm clock on it. The position and orientation of the alarm clock, as well as the room's lighting conditions and background texture, are all random variables.

[0052] Figure 2 This is the flowchart of a multimodal data acquisition method for a mobile dual-arm robot in Embodiment 2 of the present invention.

[0053] (1) System initialization When the system starts up, it first checks if the seed.txt file exists in the memory. If it does not exist, it means that the first phase of planning has not yet been carried out, and the system enters the first phase; if it exists, the system directly enters the second phase.

[0054] Create a SAPIEN physics engine instance and set the simulation time step to 4 milliseconds. This 4-millisecond choice is perfectly aligned with subsequent audio sampling: at a sampling rate of 16000 Hz, each audio frame contains 64 samples, corresponding to a time of 64 / 16000 = 4 milliseconds.

[0055] Load the 3D model of the room, create the table model, and set the light source. The system is configured in randomization mode, and the processor will randomly select the room background texture from the preset texture library, and randomize the light intensity, color temperature, and direction parameters.

[0056] Load the ALOHA robot model, set the robot's initial base position to [0, -1.85, 0], and randomly select the initial heading between 89 and 91 degrees.

[0057] Load three cameras, setting the resolution to 1280×720 pixels. The processor initializes the spatial audio rendering model (its structure is as follows). Figure 2 As shown in the figure, create two virtual microphones located at the left and right ear positions of the robot model, load the acoustic model of the room (including room size, wall material sound absorption coefficient, etc.), calculate the room impulse response, and initialize the noise model parameters.

[0058] Perform a stability check on the objects. Simulate 2000 physics steps to bring all objects to a stable state under gravity, then check if the rotation angle of the objects exceeds 3 degrees in the last 200 steps. If it does, the object is considered unstable, and the system re-initializes.

[0059] (2) First stage: Planning and generation The processor sets the random seed range to 0 to 149, for a total of 150 different random seeds.

[0060] For each seed, the processor performs the following operations: First, use this seed to initialize NumPy's random module, PyTorch's random module, and Python's standard library's random module simultaneously. This ensures that given the same seed, the system will make exactly the same random selection.

[0061] Then, the initial environment is randomly generated using the seed: the X coordinate of the alarm clock is randomized within the range of [-0.03, 0.03] meters, the Y coordinate is randomized within the range of [-0.67, -0.63] meters, and the orientation angle is randomized within the range of [0, 360) degrees; the light intensity, color temperature, and direction are randomized; and the room background texture is randomly selected from the preset texture library.

[0062] Next, the processor performs navigation trajectory planning. The system generates a random target location (X coordinate between -0.03 and 0.03 meters, Y coordinate between -0.67 and -0.63 meters, representing the area where the alarm clock is located). Trajectory planning consists of three steps: The first step is satellite A search. A grid search is performed in the configuration space with a grid resolution of 0.05 m × 0.05 m × 5 degrees (X, Y, heading angle) to find a high-level path from the starting position to the target area.

[0063] The second step is velocity planning. An angular velocity constraint (maximum 180 degrees / second), a lateral acceleration constraint (maximum 0.5 m / s²), and a longitudinal acceleration constraint (maximum 1.0 m / s²) are applied to the path to obtain the maximum allowable velocity at each point.

[0064] The third step is time parameterization. Time is assigned to each segment of the path, generating a trajectory with timestamps. The trajectory contains 1000 trajectory points, each containing a timestamp, X-coordinate, Y-coordinate, heading angle, linear velocity, and angular velocity.

[0065] The processor simultaneously plans the dual-arm manipulation trajectory, including: opening both grippers to the fully open state (opening degree 1.0), moving both arms to the pre-grabbing position, performing the gripping action (gripper opening degree changes from 1.0 to 0.0), performing the click operation, and then opening the grippers to return.

[0066] The processor attempts to execute the task according to the planned results. The robot moves its base sequentially along the navigation trajectory while simultaneously performing manipulations according to the trajectory of its arms. If no collision occurs during navigation, the alarm clock is successfully activated, and the object remains stable throughout the process, the task is considered successfully executed.

[0067] If the task is executed successfully, the processor records the seed value and saves the navigation trajectory as a pickle file in the _nav_data directory, and the dual-arm manipulation trajectory as a pickle file in the _arm_data directory. If planning fails or the task fails (for example, due to object instability or a collision), the system discards the attempt and continues to the next seed.

[0068] The system repeats the above process until 125 successful seeds are accumulated. Since the experimental success rate is 83%, the system actually needs to attempt approximately 150 times to obtain 125 successful seeds. The output of the first phase includes: a text file containing 125 successful seeds, a file set containing 125 navigation trajectories, and a file set containing 125 dual-arm manipulation trajectories.

[0069] Phase Two: High-Fidelity Data Acquisition In the second phase, the processor performs 125 high-fidelity multimodal data acquisitions.

[0070] For each successful seed in seed.txt, the processor performs the following operations: First, use this seed to reinitialize all random number generators, reproducing the exact same initial environment as in the first stage: the alarm clock's position is the same, the lighting is the same, the background texture is the same, and the initial velocity of the objects is the same.

[0071] Then, the navigation trajectory file and dual-arm manipulation trajectory file corresponding to the seed are loaded from memory, and the "Planning Required" flag is set to False, meaning the system will use the loaded trajectory instead of replanning.

[0072] The task execution process is then rapidly replayed along the same trajectory. During the replay, the processor performs multimodal data acquisition at each sampling interval (every 8 physical steps, corresponding to 32 milliseconds).

[0073] During each data acquisition, the processor calls the get_obs method to obtain the current observation, which includes: RGB image: 1280×720×3 color image from three cameras: head, left wrist, and right wrist; Depth map: 1280×720 depth images from three cameras; Segmentation image: 1280×720 segmented images from three cameras, with each pixel identifying different objects in the scene; Joint motion: 17-dimensional vector, including the 6 joint angles of the left arm, the opening of the left gripper, the 6 joint angles of the right arm, the opening of the right gripper, and the base. , , ; End-effector pose: The position and orientation of the left and right arm end-effectors in three-dimensional space (represented by quaternions); Audio signal: Audio data from the virtual microphones in the left and right ears.

[0074] Each frame of data collected is saved as a pickle file, and a complete task typically produces about 990 frames of data.

[0075] (3) Data fusion and storage After completing the data acquisition for a single motion sequence, the processor merges all pickle files into a single HDF5 file.

[0076] The processor scans all pickle files in the cache directory, sorting them by frame index number in the filename. Each pickle file is read sequentially, extracting RGB, depth, segmentation, joint motion, end-effector pose, and audio data. This data is then stacked into a multidimensional array along the time dimension. The final shape of the RGB data is (990, 1280, 720, 3, 3), where the first dimension is the number of time frames (990 frames), the second and third dimensions are the image size, the fourth dimension is the RGB channels, and the fifth dimension is the camera index. The joint motion data is stacked into a (990, 17) matrix, fully recording the 17-dimensional joint motion vector for each frame.

[0077] These multidimensional arrays are written to HDF5 files using gzip compression. Each HDF5 file is approximately 850 megabytes in size and contains 125 complete motion sequence loops.

[0078] The above process was executed on 125 successful seeds, resulting in 125 HDF5 files, totaling more than 40 million frames of high-quality multimodal data.

[0079] (4) Specific implementation of spatial audio rendering In this application example, the spatial audio rendering process is as follows: Figure 3 As shown, audio rendering is performed in each physical simulation step (4 milliseconds), and the specific process is as follows: Direct audio path calculation: The processor determines the current positions of the two virtual microphones in the world coordinate system. These positions are equal to the base position plus the microphone's offset relative to the base in the local coordinate system, obtained through a rotation matrix transformation. The Euclidean distance from each microphone to the alarm clock (sound source) is calculated, and the propagation delay is calculated based on the speed of sound (343 m / s). The processor uses a 4-point Lagrange polynomial for fractional delay interpolation to avoid signal distortion caused by rounding to integer samples. Simultaneously, amplitude attenuation is calculated based on the distance; the attenuation is inversely proportional to the square of the distance.

[0080] Reverb Processing: The processor divides the reverb into two parts. For the early reflections (0-80 milliseconds), the reflection paths from the alarm clock to each wall and then to the microphone are calculated separately, and the corresponding propagation delay and attenuation are calculated for each path, which are then directly superimposed on the signal. For the late reverb (80-200 milliseconds), streaming convolution is performed using the Overlap-Add (OLA) method, employing the pre-calculated late portion of the room impulse response, ensuring that the delay characteristics of the output signal are consistent with reality.

[0081] Noise addition: The processor adds colored noise with a 1 / f power spectrum characteristic, characterized by strong energy in the low-frequency range and weak energy in the high-frequency range. A spatial correlation coefficient of 0.4 is added to the noise of the two virtual microphones, ensuring that the noise received by the two microphones is neither completely independent nor identical. The overall signal-to-noise ratio is controlled at 20 dB, meaning the signal power is 100 times the noise power.

[0082] Audio Output: The audio rendering output is a 2×64 two-dimensional array, corresponding to 64 audio samples from the two microphones within the current 4-millisecond time step. At a sampling rate of 16000 Hz, the 64 samples correspond exactly to 4 milliseconds, perfectly aligned with the physical simulation step.

[0083] (5) Construction of 17-dimensional joint action vector The processor maintains two variables in the base tracking module: the current base pose and the base pose of the previous frame. At each time step, the processor calculates the inverse transformation between the current pose and the previous pose to obtain the relative pose.

[0084] Extracted from relative pose: Axial displacement increment (From the first component of the relative pose vector, unit: meters) Axial displacement increment (From the second component of the relative pose vector, in meters), and the increment of the steering angle about the vertical axis. (Calculated from the quaternion of relative pose using inverse trigonometric functions, unit: radians).

[0085] Will , , This vector is concatenated with the 14-dimensional joint motion vector to form a 17-dimensional joint motion vector. This vector fully describes the robot's motion in all degrees of freedom within this 4-millisecond time step.

[0086] The specific application examples disclosed in Embodiment 2 of the present invention fully demonstrate the effectiveness and feasibility of the multimodal data acquisition method for a mobile dual-arm robot proposed in Embodiment 1 of the present invention.

[0087] Example 3 Based on the multimodal data acquisition method for a mobile dual-arm robot proposed in Embodiment 1 of the present invention, Embodiment 3 of the present invention also proposes a multimodal data acquisition system for a mobile dual-arm robot. Figure 4 This is a schematic diagram of a multimodal data acquisition system for a mobile dual-arm robot proposed in Embodiment 3 of the present invention, comprising: The robot itself is equipped with a base, two arms (each with 6 joints and 1 gripper), two virtual microphones (located in the left and right ears) and three cameras (located in the head, left wrist and right wrist).

[0088] The processor, built into the robot itself (in actual implementation, this is the computer processor running the simulation environment), is configured to execute: Step S1: Construct and execute a two-stage acquisition process. The first stage involves iterating through random seeds, performing task planning and execution attempts in a simulation environment, and selecting target seeds and their corresponding motion trajectories for successful task execution. The second stage uses the target seeds to perform deterministic reconstruction of the simulation environment and loads the motion trajectories for open-loop replay. Step S2: During the open-loop playback process, the synchronous acquisition of multimodal data is triggered with a fixed time step; the multimodal data includes: the three-degree-of-freedom motion increment of the robot base, the joint movements of the robot's two arms, visual image data, and spatial audio data; Step S3: Collect the spatial audio data, and use a preset spatial audio rendering model to render the physical process of sound propagation step by step in real time based on the real-time pose of the virtual microphone on the robot. Step S4: Dimensionally concatenate the three-degree-of-freedom motion increments of the robot base with the joint movements of the two arms to construct a joint motion vector describing the motion of all controllable degrees of freedom of the robot body within a single time step.

[0089] Step S5: After completing a single motion sequence, stack the multimodal data of all time frames acquired along the time dimension.

[0090] The memory, coupled to the processor, is used to store the target seed (seed.txt), navigation trajectory, dual-arm manipulation trajectory, and 125 acquired HDF5 multimodal data files.

[0091] The description of the relevant parts of the multimodal data acquisition system for a mobile dual-arm robot provided in Embodiment 3 of this application can be found in the detailed description of the corresponding parts of the multimodal data acquisition method for a mobile dual-arm robot provided in Embodiment 1 of this application, and will not be repeated here.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0093] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for multimodal data acquisition of a mobile dual-arm robot, characterized in that, Includes the following steps: A two-stage acquisition process is constructed and executed. In the first stage, random seeds are traversed to perform task planning and execution attempts in the simulation environment, and target seeds and corresponding motion trajectories for successful task execution are selected. In the second stage, the simulation environment is deterministically reconstructed using the target seeds, and the motion trajectories are loaded for open-loop replay. During the open-loop replay process, multimodal data is synchronously acquired at fixed time steps; the multimodal data includes: the three-degree-of-freedom motion increments of the robot base, the joint movements of the robot's two arms, visual image data, and spatial audio data; The spatial audio data is collected, and a preset spatial audio rendering model is used to render the physical process of sound propagation step by step in real time based on the real-time pose of the virtual microphone on the robot. By dimensionally splicing the three-degree-of-freedom motion increments of the robot base with the joint movements of the two arms, a joint motion vector describing the motion of all controllable degrees of freedom of the robot body within a single time step is constructed.

2. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The first stage involves iterating through random seeds, attempting task planning and execution in a simulation environment, and selecting target seeds and their corresponding motion trajectories for successful task execution. Specifically: Set different random seeds; For each seed, initialize all random number generators and randomly generate an initial environment based on that seed, including object position, lighting, and background texture; In the initial environment, the motion trajectory planning for base navigation and dual-arm manipulation is performed, and the task is attempted to be executed according to the planning results; If the task is successfully executed, the seed will be designated as the target seed, and the navigation trajectory and the dual-arm manipulation trajectory generated by the motion trajectory planning will be saved to the memory. Repeat the above process until the predetermined number of target seeds have been accumulated.

3. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The second stage utilizes the target seed to perform deterministic reconstruction of the simulation environment and loads the motion trajectory for open-loop replay, specifically as follows: Read the target seed list; For each target seed, use that seed again to initialize all random number generators to reproduce the initial environment that corresponds to that seed and is exactly the same as the first stage. Load the navigation trajectory and dual-arm manipulation trajectory corresponding to the seed from memory; The planning function is turned off, and the navigation trajectory and dual-arm manipulation trajectory are directly loaded and replayed. At the same time, multimodal data are collected synchronously during the replay process.

4. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The planning of the motion trajectory includes the following steps: Perform a grid search in the configuration space to generate a high-level path from the starting position to the target position; Apply kinematic constraints to the high-level path to perform velocity planning and determine the allowable speed on the path; The path is time-parameterized to generate a sequence of trajectory points with timestamps.

5. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The fixed time step is synchronized with the physical simulation step and the audio sampling step.

6. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The physical process of sound propagation includes: Direct acoustic path delay and attenuation calculated based on the time-varying distance between the sound source and the microphone; The synthesis of early reflections and late reverberation determined by the geometric reflection characteristics of the room; where early reflections refer to reflections within a preset time length, and late reverberation refers to reverberation beyond the preset time length; And the superposition of environmental noise that conforms to spatial correlation characteristics.

7. The multimodal data acquisition method for a mobile dual-arm robot according to claim 6, characterized in that, The direct acoustic path delay is processed using a fractional delay interpolation method; The late reverberation was processed by stream convolution using an overlap-addition method. The ambient noise is colored noise with 1 / f power spectrum characteristics, and there is a preset spatial correlation coefficient between the noise received by the two virtual microphones.

8. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, By concatenating the three-degree-of-freedom motion increments of the robot base with the joint movements of the two arms, a joint motion vector describing the robot body's motion of all controllable degrees of freedom within a single time step is constructed, specifically including: Obtain the current pose and the previous frame pose of the robot base, and calculate the relative transformation between the two; Three increments are extracted from the relative transformation: Axial displacement increment , Axial displacement increment and the increment of the steering angle about the vertical axis ; The , and The motion vectors are concatenated with the 14-dimensional joint motion vectors of both arms to form a 17-dimensional joint motion vector. The 14-dimensional joint motion vectors include motion information of 6 joints of the left arm, 1 degree of freedom of the left gripper, 6 joints of the right arm, and 1 degree of freedom of the right gripper.

9. The multimodal data acquisition method for a mobile dual-arm robot according to claim 1, characterized in that, The method further includes: after completing a single motion sequence, stacking the multimodal data of all time frames acquired along the time dimension and storing it in a structured file format; wherein the joint action vector is organized in the structured file as a two-dimensional matrix of the number of time frames and the vector dimension.

10. A multimodal data acquisition system for a mobile dual-arm robot, characterized in that, include: The robot itself is equipped with a base, two arms, at least two virtual microphones, and at least one camera; The processor, built into the robot body, is configured to execute a multimodal data acquisition method for a mobile dual-arm robot according to any one of claims 1 to 9; The memory is used to store the target seed, the motion trajectory, and the acquired multimodal data.

Images