Multi-modal data processing fusion method and system oriented to IPU chip

Abstract

The invention relates to the technical field of data processing, in particular to a multi-modal data processing fusion method and system oriented to an IPU chip. The method comprises the following steps: acquiring multi-modal space-time sensing data; according to the multi-mode space-time sensing data, configuring the multi-mode processing parallelism degree of the IPU chip, and initializing a multi-mode parallel processing strategy of the IPU chip; based on an IPU chip multi-modal parallel processing strategy, performing multi-modal data parallel processing on the multi-modal time-space sensing data by using an IPU chip to obtain multi-dimensional sensing interaction feature data; identifying a candidate operation point set according to the multi-dimensional perception interaction feature data, and screening effective operation points; formulating a self-adaptive execution strategy according to the effective operation points; according to the method, the parallel computing capability of the IPU chip is utilized, so that the multi-modal data processing efficiency and real-time performance are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data processing fusion method and system for an IPU chip. Background Art

[0002] Traditional multimodal data processing methods primarily rely on general-purpose CPU or GPU architectures. While these processors possess certain parallel computing capabilities, they face significant bottlenecks when processing heterogeneous multimodal data. First, data from different modalities (such as images, touch, and posture) have distinct characteristics and processing requirements, making them difficult to optimize with general-purpose processors. Second, multimodal data exhibits complex spatiotemporal correlations, making it difficult for traditional serial processing architectures to effectively capture and utilize these correlated information. Third, scenarios with high real-time requirements are extremely sensitive to data processing latency, a requirement that existing technologies often struggle to meet. IPU chips offer higher parallel processing capabilities and a more flexible computing architecture, making them particularly well-suited for processing irregular data structures and complex graph computing tasks. Compared to traditional processors, IPU chips offer strong parallel processing capabilities, high memory access efficiency, and superior energy efficiency, making them particularly well-suited for processing heterogeneous data streams. However, the current application of IPU technology in multimodal data fusion is still in its infancy, lacking a systematic processing framework and approach. Existing IPU applications primarily focus on single-modal data processing or simple fusion scenarios, failing to fully leverage the IPU's advantages in multimodal parallel processing. Summary of the Invention

[0003] Based on this, the present invention provides a multimodal data processing fusion method and system for IPU chips to solve at least one of the above technical problems.

[0004] To achieve the above objectives, a multimodal data processing and fusion method for an IPU chip includes the following steps: Step S1: Acquire multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism according to the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; Step S2: Based on the IPU chip's multimodal parallel processing strategy, the IPU chip is used to perform multimodal data parallel processing on the multimodal spatiotemporal perception data, and multidimensional interaction feature fusion is performed to obtain multidimensional perception interaction feature data; wherein the multidimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; Step S3: Calculate a visual score based on the target visual feature data; calculate a tactile score based on the contact feature data; calculate a motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; generate multiple pose solutions based on the candidate operation point sets; Step S4: Filter effective operation points based on multiple groups of posture solutions, and formulate an adaptive execution strategy based on the effective operation points; use the adaptive execution strategy to perform operation control, and conduct a comprehensive stability evaluation to obtain real-time stability data.

[0005] Preferably, the present invention further provides a multimodal data processing fusion system for an IPU chip, which executes the multimodal data processing fusion method for an IPU chip as described above. The multimodal data processing fusion system for an IPU chip includes: The IPU multimodal initialization module is used to obtain multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism based on the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; A multi-dimensional feature fusion processing module is used to perform multi-modal data parallel processing on multi-modal spatiotemporal perception data using the IPU chip based on the IPU chip's multi-modal parallel processing strategy, and to perform multi-dimensional interactive feature fusion to obtain multi-dimensional perception interaction feature data; wherein the multi-dimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; The multimodal scoring planning module is used to calculate the visual score based on the target visual feature data; calculate the tactile score based on the contact characteristic data; calculate the motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; and generate multiple sets of pose solutions based on the candidate operation point sets. The adaptive execution monitoring module is used to screen effective operating points based on multiple sets of posture solutions and formulate adaptive execution strategies based on the effective operating points; it uses the adaptive execution strategies to perform operation control and conduct comprehensive stability evaluation to obtain real-time stability data.

[0006] This invention leverages the unique advantages of the IPU chip in parallel computing to effectively address the bottlenecks faced by traditional processors in multimodal data processing. By constructing a parallel processing strategy specifically for multimodal data, this method significantly improves the processing efficiency of heterogeneous data streams, reducing the cumulative latency of traditional serial processing by approximately 65%, significantly enhancing the system's real-time responsiveness, and overcoming the limitations of traditional architectures in processing data with diverse characteristics. Regarding the capture of spatiotemporal correlations, this solution enables simultaneous and parallel processing of multimodal data, fully mining and utilizing the spatiotemporal correlations between different modalities, such as visual, tactile, and posture data, and enhancing the system's environmental perception accuracy. Regarding real-time performance, this method leverages the highly parallel computing architecture of the IPU chip to significantly reduce the latency of multimodal data processing, meeting the stringent requirements of time-sensitive scenarios. The multidimensional interactive feature fusion mechanism innovatively integrates features from different modalities, generating richer and more reliable perceptual information than single-modality approaches. By comprehensively weighing the multidimensional scoring system (visual, tactile, and motion), the system can more intelligently identify the optimal operating point, avoiding the biased and unstable nature of single-modality evaluation. The introduction of an adaptive execution strategy gives the system the ability to dynamically adjust operating parameters according to real-time conditions, significantly improving its adaptability and robustness in complex and changing environments. The real-time stability assessment mechanism ensures the safety and reliability of the operation process. Through continuous monitoring and feedback, the system can adjust strategies in a timely manner to prevent potential risks. In terms of energy consumption, compared with traditional processing architectures, this solution fully utilizes the high-efficiency computing characteristics of the IPU chip, significantly reducing energy consumption while ensuring processing performance. It not only improves the quality and efficiency of data processing, but also expands the application scenarios of the system, making it more suitable for application deployment in resource-constrained scenarios, especially in areas such as robotic operation, intelligent manufacturing, industrial sorting, and assisted medical care that require the collaborative work of multi-source perception information. Therefore, the multimodal data processing and fusion method for IPU chips of the present invention divides independent buffers in the IPU chip memory to achieve efficient caching and synchronous processing of visual, tactile and position data, dynamically allocates weights and memory resources according to the quality score of multimodal data, optimizes the processing strategy, and then uses the parallel processing capability of the IPU chip for parallel processing, formulates an adaptive execution strategy, ensures the real-time and reliability of the operation, gives full play to the parallel computing advantages of the IPU chip, breaks through the bottleneck of the traditional architecture, and realizes efficient fusion and intelligent processing of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a schematic flow chart of the steps of a multimodal data processing and fusion method for an IPU chip according to the present invention; Figure 2 for Figure 1 Detailed implementation steps of step S2 in FIG. The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0008] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. It is obvious that the embodiments described are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0009] In addition, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor and / or microcontroller approaches.

[0010] It should be understood that although the terms "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the listed associated items.

[0011] To achieve this, please refer to Figures 1 to 2 The present invention provides a multimodal data processing and fusion method for an IPU chip, comprising the following steps: Step S1: Acquire multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism according to the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; Step S2: Based on the IPU chip's multimodal parallel processing strategy, the IPU chip is used to perform multimodal data parallel processing on the multimodal spatiotemporal perception data, and multidimensional interaction feature fusion is performed to obtain multidimensional perception interaction feature data; wherein the multidimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; Step S3: Calculate a visual score based on the target visual feature data; calculate a tactile score based on the contact feature data; calculate a motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; generate multiple pose solutions based on the candidate operation point sets; Step S4: Filter effective operation points based on multiple groups of posture solutions, and formulate an adaptive execution strategy based on the effective operation points; use the adaptive execution strategy to perform operation control, and conduct a comprehensive stability evaluation to obtain real-time stability data.

[0012] In an embodiment of the present invention, the multimodal data processing and fusion method for an IPU chip includes the following steps: Step S1: Acquire multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism according to the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; In an embodiment of the present invention, for example, an aluminum alloy crankshaft part is placed on the operating table, and the system first receives the initial grasping control instruction from the production execution system, which includes the target part ID, position coordinates (125mm, 240mm, 50mm) and grasping parameters. The control system starts the UR5 robotic arm, moves it from the initial position to the preparation point at a speed of 20mm / s, and starts sensor acquisition. The Intel RealSense D455 camera installed 15cm at the end captures the crankshaft RGB-D image at a frequency of 30Hz; the 8×8 tactile sensor array on the inside of the two-finger gripper records the contact pressure at a sampling rate of 100Hz; the six-axis joint encoder collects the joint status at 500Hz. When the robotic arm approaches the target to 50mm, the control system decelerates to 5mm / s, and the tactile sensor begins to detect the contact pressure. After contact is established, multi-channel perception data is transmitted to the processing unit via high-speed USB3.0 and EtherCAT buses. The IPU chip receives the trimodal data stream and performs time synchronization. It processes high-frequency data through weighted average downsampling and enhances low-frequency data through cubic spline interpolation to generate multimodal spatiotemporal perception data, and initializes the chip memory allocation and parallel processing strategy based on this.

[0013] Step S2: Based on the IPU chip's multimodal parallel processing strategy, the IPU chip is used to perform multimodal data parallel processing on the multimodal spatiotemporal perception data, and multidimensional interaction feature fusion is performed to obtain multidimensional perception interaction feature data; wherein the multidimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; In this embodiment of the present invention, the system performs multimodal data processing in parallel on a Graphcore IPU-M2000 chip, with four GC200 chips each handling different processing tasks. The first chip is dedicated to visual processing, dividing the crankshaft RGB-D data into 64 grid regions for parallel processing. Processing cores 0-511 are responsible for image preprocessing, including contrast adaptive equalization and bilateral filtering; processing cores 512-1023 execute a modified ResNet-50 network to extract 2048-dimensional visual features, capturing the crankshaft's geometric contours and surface texture. The second chip focuses on tactile data processing, distributing data from 64 tactile units to 973 processing cores for parallel computation. Each 15 cores analyzes the temporal changes of each tactile unit, using wavelet transforms for denoising and extracting contact pressure distribution and surface texture features. The third chip processes joint motion data, with 730 cores performing posture analysis: the first 365 cores perform joint angle coordinate system conversion, while the last 365 cores calculate the transformation matrix and perform end-effector posture analysis. The fourth chip performs feature fusion, using a tensor fusion network enhanced by an attention mechanism to integrate the three modal features according to their importance, and realizes inter-modal correlation analysis through a cross-attention weight calculation module, ultimately generating 512-dimensional multi-dimensional perceptual interaction feature data.

[0014] Step S3: Calculate a visual score based on the target visual feature data; calculate a tactile score based on the contact feature data; calculate a motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; generate multiple pose solutions based on the candidate operation point sets; In an embodiment of the present invention, the system identifies the optimal grasping position for a crankshaft based on multidimensional perceptual interaction feature data. First, a three-layer fully connected network calculates visual, tactile, and motion scores, respectively. These scores are then fused with a 4:3:3 weighting to generate a composite score. The system then performs uniform sampling on the crankshaft's 3D point cloud, generating 312 potential grasping points with a 5mm grid interval. Each point is scored, and a non-maximum suppression algorithm is applied to extract the local optimal point. The system retains the 27 highest-scoring candidate points to form the grasping candidate point set. For each candidate point, the system analyzes the crankshaft surface normal and generates seven grasping orientation options within a ±45° range at 15° intervals. Combining the candidate point positions and orientations, the system generates 189 initial grasping posture solutions. Each solution includes the gripper position coordinates, posture quaternion, and an estimated grasping score. The system then performs geometric verification, eliminating solutions that could result in collisions, retaining 132 valid grasping posture solutions. These plans are organized into a priority queue structure and sorted from high to low according to the comprehensive score. The high-scoring plans are executed first to ensure that the most stable part of the crankshaft is grasped.

[0015] Step S4: Filter effective operation points based on multiple groups of posture solutions, and formulate an adaptive execution strategy based on the effective operation points; use the adaptive execution strategy to perform operation control, and conduct a comprehensive stability evaluation to obtain real-time stability data.

[0016] In an embodiment of the present invention, a three-stage screening is performed on 132 groups of grasping posture solutions. First, the expected contact area is calculated, and it is found that the contact area of 43 groups of solutions is less than 5cm². These solutions are marked as unsuitable for grasping and eliminated. Then, the mechanical stability is analyzed, and the relative position of the line of action of the grasping force and the center of gravity of the crankshaft is calculated. It is found that the line of action of the force in 27 groups deviates from the center of gravity by more than 50% of the minimum width of the crankshaft. These solutions are marked as unstable grasping and eliminated. Finally, inverse kinematic calculations are performed to check the accessibility of the remaining 62 groups of solutions. 21 groups of solutions that exceed the joint limit are eliminated, and finally 41 groups of valid grasping points are retained. The system selects the grasping solution with the highest score (located in the cylindrical section in the middle of the crankshaft), identifies its surface as a smooth metal hard contact type, and sets a position control strategy of 15N grasping force and 5mm / s contact speed accordingly. During the grasping process, the IPU chip monitors the status of the robotic arm in real time, analyzes the tactile sensor data every 2ms, and calculates the contact uniformity and pressure distribution. When it detects that the pressure distribution is balanced and there is no sliding tendency, the system confirms that the grasping is stable and controls the robotic arm to move the crankshaft to the assembly position at a speed of 80mm / s, completing the entire grasping task.

[0017] Preferably, obtaining multimodal spatiotemporal perception data in step S1 includes: The IPU chip memory is divided into three independent buffers to obtain a sensor data buffer pool. The three independent buffers include a visual data buffer, a tactile data buffer, and a position data buffer. The size of the visual data buffer is 10MB, the size of the tactile data buffer is 2MB, and the size of the position data buffer is 1MB. Deploy multimodal data collection nodes; Based on the multimodal data acquisition node, visual, tactile and position information are continuously collected, and three-way perception signals are collected in parallel to be stored in the sensor data buffer pool; Extracting the original sensor data stream from the sensor data buffer pool; performing sampling frequency discrimination on the original sensor data stream based on a preset standard sampling frequency threshold, and obtaining high-frequency acquisition data and low-frequency acquisition data respectively; Perform weighted average downsampling processing on high-frequency collected data to obtain down-frequency synchronous data; perform interpolation and enhancement processing on low-frequency collected data to obtain up-frequency synchronous data; The down-converted synchronization data and the up-converted synchronization data are integrated into spatiotemporal perception data to generate multimodal spatiotemporal perception data.

[0018] In an embodiment of the present invention, the IPU chip adopts physical memory isolation technology and creates a memory partition table at the starting address 0x80000000 through hardware isolation instructions. The partition table contains the physical address range, access rights and buffer size identifiers of three independent buffers. The visual data buffer starts at address 0x80010000 and is divided into 10MB of space for storing RGB images and depth images; the tactile data buffer starts at address 0x80A10000 and is divided into 2MB of space for storing pressure matrix data; the position data buffer starts at address 0x80C10000 and is divided into 1MB of space for storing joint angles, velocities and accelerations. The DMA controller is used to ensure that buffer cross-access is avoided during data writing, and each buffer is configured with an independent data structure header to record the data write timestamp and format identifier. The multimodal data acquisition node consists of three types of hardware modules: (1) The visual acquisition module uses a binocular camera with a resolution of 1920×1080 and 60 frames per second, an F2.0 aperture, and a 77° field of view; (2) The tactile acquisition module uses a 16×16 array pressure sensor with a sensitivity of 0.01N and a sampling rate of 1000Hz; and (3) The joint position module uses an encoder with an accuracy of 0.01° and a sampling rate of 500Hz. All three types of hardware modules implement signal acquisition and preprocessing through FPGA integrated circuits. After adding timestamps to the collected data according to the IEEE-1588 time synchronization protocol, the collected data is connected to the IPU chip via a high-speed bus (transmission rate of 10Gbps) to establish a direct memory access channel to ensure that the data is written to the corresponding buffer in real time. Three sensory signals are collected in parallel using time-division multiplexing. Visual data is collected at a resolution of 640×480 pixels, a frame rate of 30 fps, and a bit depth of 24 bits, and is written directly to the visual data buffer via IPU chip DMA channel 0. Tactile data is collected using a 16×16 pressure array with a sampling rate of 100 Hz and 12-bit precision, and is written to the tactile data buffer via DMA channel 1. Joint data, including six degrees of freedom angles and corresponding torques, is collected at a sampling rate of 1000 Hz and 16-bit precision, and is written to the joint data buffer via DMA channel 2. The IPU chip's interrupt controller sets three data collection completion interrupts to trigger the data processing pipeline. Zero-copy technology is used to reduce data transfer overhead. After each data collection completion, the buffer header metadata is updated to record the timestamp, data size, and acquisition status. Block data is read from the three buffers to generate a timestamped raw sensor data stream. Each data frame is appended with a 16-byte metadata header containing data type, acquisition time, and length information. The system preset standard sampling frequency threshold is 500Hz. This threshold is determined by testing the system response time and processing capability: first, the processing delay of the IPU chip under different loads is measured, and then the critical response frequency is calculated based on the control system stability requirements.Based on this threshold, joint data (1000Hz) is identified as high-frequency data, while visual data (30Hz) and tactile data (100Hz) are identified as low-frequency data. The discrimination process uses a frequency detection state machine to calculate the actual sampling period for each data stream, compare it with the preset threshold, and then mark the data type flag. A weighted average downsampling process is applied to the high-frequency data (1000Hz joint data) using a sliding window method. The downsampling target frequency is set to 500Hz, the window size is two consecutive frames of raw data, and the weighting coefficients are 0.6 and 0.4, respectively. The downsampling calculation formula is: in, Represents the kth frame data after downsampling, and Represents the 2kth and 2k+1th frames in the original data sequence. A cubic spline interpolation algorithm is used to up-convert low-frequency data, increasing the visual data (30 Hz) and tactile data (100 Hz) to 500 Hz, respectively. The interpolation calculation formula is: Where t represents the interpolation time point, and are known adjacent sampling time points, 、 、 、 are the spline function coefficients, which are obtained by solving a set of linear equations to ensure that the interpolation function values at the nodes, the first-order derivatives, and the second-order derivatives are continuous. A timestamp index table is constructed to record the sampling time of each modal data with an accuracy of microseconds. Then, a sliding time window method is used with a window width of 2ms to integrate all the sampled data within the window. The integration adopts a weighted fusion strategy to assign weight coefficients to different modal data: visual data weight 0.4, tactile data weight 0.3, and joint data weight 0.3. The fusion calculation formula is: , where F(t) represents the fused multimodal spatiotemporal perception data, and V(t), T(t), and J(t) represent the visual, tactile, and joint data at time t, respectively. The integrated data is stored using a unified data structure consisting of a 32-byte header (describing the data source, timestamp, and integrity flag) and a variable-length data body (containing synchronized data from the three modalities), forming a multimodal spatiotemporal perception data packet.

[0019] Preferably, deploying a multimodal data acquisition node includes: A tactile sensor array is configured, with each sensing unit having a size of 5 mm × 5 mm, to deploy a tactile sensing device; Equipped with an RGB-D camera with a resolution of 1280×720 pixels, a depth resolution of 640×480 pixels, a field of view of 65°×40°, and a working distance range of 0.5-3m to deploy visual perception devices; Configuring position sensors to deploy motion sensing devices; A high-speed USB 3.0 interface was used to connect the visual perception device, an EtherCAT bus was used to connect the touch perception device, and a CAN bus was used to connect the joint encoder and torque sensor. The communication cycle was set to 1ms to obtain a multimodal data transmission channel. The multimodal data transmission channel is connected to the high-speed I / O port of the IPU chip, and the DMA channel is configured for zero-copy data transmission to obtain a multimodal data acquisition node.

[0020] In embodiments of the present invention, for example, including but not limited to, the tactile sensing deployment in the robotic gripper utilizes a customized installation process. A 1.2mm deep, 41mm×41mm mounting groove is pre-milled on the inner surface of the two-finger gripper. A TactArray P8×8 tactile sensor array (40mm×40mm) is embedded in the groove and secured with thermally conductive silicone to ensure the sensor surface is flush with the groove edge. A silicone protective layer with a thickness of 0.5±0.02mm, fabricated from polyurethane, is applied to the sensor surface via a vacuum injection molding process. The silicone has a Shore A hardness of 20, ensuring elastic transmission of contact force while protecting the sensor from mechanical damage. Each sensor unit collects pressure within a range of 0-10N and a sensitivity of 0.01N. It is connected to the gripper base signal processing circuit via a flexible printed circuit board, enabling 64-channel tactile data acquisition with a sampling frequency set at 100Hz. The visual perception device utilizes an Intel RealSense D455 RGB-D camera, mounted on the sixth axis of the robotic arm, 15cm from the end effector, via a custom aluminum alloy bracket. The bracket adopts a three-point fixed structure and is fastened with M3 screws. The installation error is controlled within ±0.1mm. The camera parameters are set as follows: RGB resolution 1280×720 pixels, 30 frames / second; depth resolution 640×480 pixels, frame rate 90 frames / second; field of view 65°×40°; working distance range 0.5-3m. The camera calibration toolbox and 11×8 format calibration plate are used for internal parameter calibration, and the reprojection error is controlled within 0.5 pixels; the eye-in-hand calibration method is used to determine the conversion matrix between the camera coordinate system and the robot base coordinate system, and the calibration error is less than 1mm. The camera installation angle is precisely adjusted to ensure that the angle between the optical axis and the gripping direction (Z axis) of the gripper is less than 2°, so as to achieve consistency in visual data acquisition and gripping operation space. The configuration of the joint motion sensing device first determines the physical position of the six-axis joint node of the robot arm UR5 model. Measure the coordinates of the center point of each joint axis and establish the DH parameter table of the connecting rod coordinate system: , , An ATI Axia80 torque sensor was then installed at each joint. The sensor has a thickness of 25 mm, an outer diameter of 80 mm, a measurement range of ±150 N·m, and a resolution of 0.03 N·m. A split connection method was used for installation: the original joint was split into a master and slave part, the sensor was placed in the middle and secured with 12 M6 bolts, and the installation torque was controlled at 25 N·m. The joint encoder uses an absolute encoder with a resolution of 27 bits (134, 217, 728 counts / rev), a maximum speed of 3000 rpm, and an RS-422 signal interface. It is connected to the encoder interface of the control cabinet via a dedicated cable. A layered communication architecture is used to construct the multimodal data transmission channel. The visual perception device is connected to the Type-C interface of the processing unit via a 5-meter long USB 3.1 Gen2 cable (transmission rate 10 Gbps). Asynchronous transmission mode is used, and the size of each RGB-D data packet is approximately 2.8 MB. The tactile sensing device utilizes the EtherCAT real-time industrial bus, configured in 100Mbps full-duplex transmission mode, with a cycle time set to 1ms and jitter controlled within 10μs. A distributed clock synchronization mechanism ensures data acquisition synchronization error of less than 1μs. The joint encoder and torque sensor are connected via a CAN-FD bus, configured with an extended frame format and a bit rate of 5Mbps. The communication mode uses periodic triggering, with identifiers assigned according to priority order: 0x100-0x105 for torque data and 0x200-0x205 for encoder data. All communication interfaces are electrically isolated and have a common-mode rejection ratio greater than 80dB, ensuring reliable signal transmission in industrial environments. The multimodal data acquisition node utilizes a Graphcore IPU-M2000 computing accelerator card, consisting of four GC200 IPU chips, each with 1216 processing cores and 900MB of on-chip memory. The data path first connects the USB 3.0 interface to the motherboard's PCIe 4.0 x4 slot via a PCIe adapter card, providing a bandwidth of 7.88GB / s. EtherCAT and CAN bus data are collected and aggregated to system memory via a dedicated fieldbus card. The IPU high-speed I / O interface is configured, and NVLink interconnection is enabled, achieving a bandwidth of 300GB / s. Multimodal data acquisition nodes are connected to the IPU chip using a high-speed circuit board-level connection. The IPU chip's high-speed I / O port provides a 32-channel parallel data interface, operating at a frequency of 500 MHz and a single-channel bandwidth of 4 Gbit / s. The USB 3.0 data stream is converted via a PCIe bridge and connected to channels 0-7 of the IPU; the EtherCAT data stream is connected to channels 8-15 of the IPU via the RGMII interface; and the CAN bus data is connected to channels 16-23 of the IPU via the SPI interface. Five independent DMA controllers are set up inside the IPU chip, each with a 64-megabyte ring buffer.DMA channel configuration uses a linked list of descriptors, with each descriptor containing four 32-bit fields: source address, destination address, transfer length, and link pointer. The DMA transfer mode is set to zero-copy write-through mode, where data is written directly from the I / O port to the designated memory area, bypassing the CPU cache and reducing data movement overhead. Upon completion of the transfer, the DMA controller notifies the IPU processing core via a hard interrupt, with interrupt vector numbers 0x40-0x44 and interrupt priority set to the highest level, 7.

[0021] Preferably, in step S1, configuring the IPU chip multimodal processing parallelism according to the multimodal spatiotemporal perception data and initializing the IPU chip multimodal parallel processing strategy includes: Calculate the signal-to-noise ratio and integrity of each modality based on multimodal spatiotemporal perception data to generate a data quality score; The weights of different modal data are assigned to the multimodal spatiotemporal perception data through data quality scoring to obtain the multimodal perception data weights; Evaluate the amount of data for each modality based on multimodal spatiotemporal perception data; Allocate IPU chip memory resources according to the amount of data in each modality, where the IPU chip memory resources include allocated visual data processing area, tactile data processing area, and position data processing area; Configure the IPU chip's multimodal processing parallelism based on the data quality score and generate a core task allocation table; Establish IPU inter-core communication strategy based on the core task allocation table; The multimodal parallel processing strategy is initialized based on the IPU chip memory resources, core task allocation table and IPU inter-core communication strategy to obtain the IPU chip multimodal parallel processing strategy.

[0022] In the embodiment of the present invention, multimodal spatiotemporal perception data is received, and the signal-to-noise ratio and integrity of the three modal data of vision, touch and joint are calculated respectively. The signal-to-noise ratio (SNR) calculation adopts the noise estimation method based on wavelet transform, and the 5-level db4 wavelet decomposition is applied to each modal data X, and the highest frequency subband coefficient σ_noise is calculated. , where σ_signal is the data standard deviation. The completeness index is calculated by the space-time missing rate, the formula is C=1-( ), N_missing represents the number of missing data points in the spatiotemporal dimension, and N_total represents the total number of theoretically complete data points. The visual modality detects pixel missing areas, the tactile modality detects sensor unit failure points, and the joint modality detects sampling time interval anomalies. The final quality score is calculated using a weighted geometric mean. , respectively, we get the visual quality score Q_vision, the tactile quality score Q_tactile and the joint quality score Q_joint. First, normalize the quality scores of the three modalities, and the calculation formula is The weight distribution adopts the exponential enhancement mechanism, the formula , α is the enhancement factor, set to 1.5, used to amplify the weight difference of high-quality data. At the same time, the modal prior importance coefficient P={P_vision=0.4, P_tactile=0.4, P_joint=0.2} is introduced to obtain the final weight . The system discretizes the calculated weight values and divides them into five weight levels of {0.1, 0.2, 0.3, 0.4, 0.5}, constructs a weight mapping table W_map, and finally outputs the multimodal perception data weight matrix, which is stored in the constant memory area of the IPU chip. The received multimodal spatiotemporal perception data is structurally analyzed to obtain the structural characteristics of each modality data. The visual data includes RGB images (3 channels × 1280 × 720 × 8 bits) and depth maps (1 channel × 640 × 480 × 16 bits). The data volume of a single frame is calculated as RGB_size = 1280 × 720 × 3 × 1 = 2,764,800 bytes, Depth_size = 640 × 480 × 2 = 614,400 bytes. The total data volume of 30 frames per second is Video_total = (2,764,800 + 614,400) × 30 = 101,376,000 bytes / second. Tactile data consists of 64 sensor units × 19 channels × 16-bit precision, with a sampling rate of 100 Hz. The data volume is calculated as Tactile_total = 64 × 19 × 2 × 100 = 243,200 bytes / second. Joint data consists of 6 joints × (position + velocity + torque) × 32-bit floating-point numbers, with a sampling rate of 500 Hz. The data volume is calculated as Joint_total = 6 × 3 × 4 × 500 = 36,000 bytes / second. The ratio of the data volume recorded by the system for each modality is [0.949, 0.046, 0.005]. The system uses a Graphcore GC200 IPU chip with a total on-chip SRAM capacity of 900 MB. Based on the modal data volume ratio [0.949, 0.046, 0.005] and the processing complexity factor [0.7, 0.2, 0.1] (the complexity factor is determined by offline analysis of the computational strength of each modal algorithm), the system calculates the weighted allocation ratio R = [0.949 × 0.7, 0.046 × 0.2, 0.005 × 0.1] = [0.664, 0.009, 0.0005]. After normalization, the result is [0.989, 0.013, 0.001]. Considering the minimum functional allocation requirement, the minimum value is adjusted, and the final memory allocation ratio is [0.8, 0.15, 0.05]. The system calls the allocateMemoryRegions() IPU memory management API function to create three data processing areas: the vision data processing area (720MB, address range 0x00000000-0x2CFFFFFF), the tactile data processing area (135MB, address range 0x2D000000-0x35FFFFFF), and the gesture data processing area (45MB, address range 0x36000000-0x38DFFFFF). Memory alignment requirements are taken into account during allocation, and memory is aligned on 2MB boundaries to ensure efficient access.Based on the quality scores Q = [Q_vision, Q_tactile, Q_joint] and the processing task importance matrix I = [0.5, 0.3, 0.2] (preset based on expert experience), the system calculates a processing priority P = Q × I. The system maps priority P to the IPU core allocation ratio. The IPU-M2000 contains four GC200 chips, each with 1216 processing cores, for a total of 4864 processing cores. Core allocation is tiered: the first tier allocates cores by modality type, with 3600 cores allocated to vision processing, 800 cores to tactile processing, and 464 cores to gesture processing. The second tier is further subdivided by task type: feature extraction in vision processing is allocated 2000 cores, object detection is allocated 1000 cores, and scene understanding is allocated 600 cores; pressure mapping in tactile processing is allocated 500 cores and slip detection is allocated 300 cores; and motion prediction in gesture processing is allocated 264 cores and collision detection is allocated 200 cores. The system generates a core task allocation table T_allocation, which contains a four-tuple structure consisting of [task ID, core ID starting value, number of cores, priority]. Four communication requirements are categorized: intra-modal communication, inter-modal fusion communication, control instruction communication, and result return communication. Intra-modal communication utilizes a tiled topology, grouping processing cores within the same modality into an n×n grid. Each core establishes direct connections with eight adjacent cores, enabling efficient local data exchange through a diffusion routing algorithm. Inter-modal fusion communication utilizes a star topology, with 64 dedicated fusion cores serving as transit nodes to aggregate processing results from each modality. Control instruction communication utilizes a broadcast mechanism, sending synchronization signals from the master control core to all processing cores. Result return communication utilizes a hierarchical convergence tree, merging data from leaf nodes to the root node. The system configures IPU communication primitive parameters: intra-modal communication bandwidth of 120 GB / s and latency of 0.5 μs; inter-modal communication bandwidth of 80 GB / s and latency of 2 μs; control broadcast latency of 1 μs; and result return throughput of 50 GB / s. The core task allocation table is converted into a Poplar computational graph structure, with each task mapped as an object. Based on the IPU's memory resource partitioning, the system creates three independent tensors, corresponding to the visual, tactile, and gesture data processing areas, and sets tensor access properties (read / write mode, caching strategy). The system configures inter-core data transfer primitives based on the kernel communication strategy, defining four communication operations: exchange (intra-modal exchange), broadcast, reduce, and gather. After initialization, the system constructs a complete execution graph and compiles it into IPU native code using GraphCompiler. During compilation, operator fusion, memory overlap, and pipeline parallelization optimizations are enabled to reduce memory access latency. The system loads the compiled code into the IPU device execution engine and monitors execution efficiency using hardware performance counters, enabling efficient parallel processing of multimodal data.

[0023] Preferably, step S2 includes: Step S21: Based on the multimodal parallel processing strategy of the IPU chip, the multimodal spatiotemporal perception data is subjected to multimodal data parallel processing control to obtain visual data to be processed, tactile data to be processed, and position data to be processed; Step S22: performing visual feature analysis based on the visual data to be processed to generate target visual feature data; Step S23: performing contact characteristic analysis based on the tactile data to be processed to obtain contact characteristic data; Step S24: performing coordinate system conversion on the position data to be processed to obtain spatial position data; Step S25: Calculate the transformation matrix based on the spatial position data to generate a pose matrix; Step S26: Perform multi-dimensional interactive feature fusion on the target visual feature data, contact characteristic data and posture matrix to generate multi-dimensional perception interactive feature data.

[0024] In an embodiment of the present invention, based on the initialized multimodal parallel processing strategy of the intelligent processing unit, there is a core task allocation table (clearly specifying that cores 0-49 process visual input, cores 50-59 process tactile input, and cores 60-64 process position input) and a memory mapping (the visual data buffer is located at address 0xA0000000, the tactile data buffer is located at 0xB0000000, and the position data buffer is located at 0xC0000000). When the system starts data acquisition, the original multimodal spatiotemporal perception data stream is input to the intelligent processing unit through the direct memory access controller. The scheduler of the intelligent processing unit activates cores 0-64 according to the strategy. Cores 0-49 read the visual data packet from the input port, parse the frame header information, and write continuous pixel data blocks to the memory area with a starting address of 0xA0000000 to form the visual data to be processed. Cores 50-59 receive samples from the tactile sensor array via the Serial Peripheral Interface bus, verify the data packets, and store them in the memory area starting at address 0xB0000000, generating the tactile data to be processed. Cores 60-64 parse the Global Positioning System (GPS) sentences received by the Universal Asynchronous Receiver / Transmitter (UART), extract the coordinates and timestamp information, and write them to the memory area starting at address 0xC0000000, generating the position data to be processed. Core group 70-369 (assigned by the previous core task assignment table) accesses the visual data to be processed (e.g., a frame of 1920x1080 resolution raw image data) stored at address 0xA0000000. These cores execute a pre-loaded convolutional neural network model (e.g., MobileNetV2) optimized for the intelligent processing unit. The image data is divided into multiple tiles and processed in parallel by different cores within the core group. Each core performs convolution, batch normalization, activation functions (e.g., rectified linear units), and pooling operations. Communication between cores occurs via the on-chip network (ONC), exchanging boundary pixel information or transferring intermediate layer feature maps. After the network's forward propagation, a 1280-dimensional target visual feature vector is generated at the network's end (e.g., after the global average pooling layer). This vector, or target visual feature data, is written by the core group to the pre-allocated shared memory address 0xD0000000. Core groups 400-419 access the tactile data to be processed (e.g., a 16-channel sequence of pressure values with 100 samples per channel) stored at address 0xB0000000. Cores 400-409 first execute digital filtering algorithms in parallel, such as applying a 5th-order Butterworth low-pass filter (with a preset cutoff frequency of 20 Hz and filter coefficients obtained via a table lookup) to each channel's time series to remove high-frequency noise.Cores 410-419 then analyze the filtered data: they calculate the average pressure value for each channel over the last 50 milliseconds; identify channels with pressure values exceeding a preset threshold (e.g., 0.05 Newtons, as determined by sensor calibration data) as contact points; calculate the centroid of all contact point pressure values (a weighted average of the contact point position coordinates, with the weights being the pressure values); and accumulate the pressure values of all contact points to obtain the total contact force. The calculated contact point positions, centroid coordinates (x, y), and total contact force values constitute the contact characteristic data, which is stored at memory address 0xD1000000. Core groups 500-504 access the position data to be processed (e.g., containing longitude, latitude, and altitude in WGS84 format) stored at address 0xC0000000. The system predefines the origin of the target local Cartesian coordinate system (e.g., the latitude and longitude of a specific geographic location) and the orientation of that coordinate system relative to magnetic north (e.g., with the X axis pointing due east and the Y axis pointing due north). Cores 500-504 perform coordinate conversion calculations. First, they apply the Transverse Mercator (UTM) or the simpler Equidistant Cylindrical projection formulas to convert the received longitude and latitude into metric coordinates (easting x', northing y') relative to a pre-determined origin. They then convert the altitude (typically relative to the WGS84 ellipsoid) to a height z' relative to the local horizontal plane using a geoid model (e.g., an EGM96 model lookup table preloaded in the core's local memory). The resulting (x', y', z') represents the spatial position data in the target local coordinate system and is stored at memory address 0xD2000000. Cores 505-509 read the spatial position data (local coordinates x', y', z') stored at address 0xD2000000. Simultaneously, they read real-time attitude data (Euler angles: roll φ, pitch θ, yaw ψ) from another memory area (e.g., address 0xC1000000) provided by the IMU processing core (assuming it is running in parallel). Cores 505-509 perform the following calculations: First, they calculate the 3x3 rotation matrix R from the Euler angles (φ, θ, ψ) using a standard formula (e.g., matrix multiplication using the ZYX rotation sequence). Then, they construct a 4x4 homogeneous transformation matrix, the pose matrix M. The upper-left 3x3 section of this matrix is filled with the calculated rotation matrix R, the upper-right 3x1 section is filled with the spatial position data [x'; y'; z'], and the last row is fixed to [0, 0, 0, 1]. All calculations are performed using the floating-point unit (FPU) built into the IPU core. Cores 600-649 are designated to perform this fusion task. They read the 1280-dimensional visual feature vector V from address 0xD0000000, the contact characteristic data T (including the center of mass coordinates and total force, constructed into a 3D vector) from address 0xD1000000, and the 4x4 pose matrix P from address 0xD3000000.First, cores 600-609 extract the translation vector (3D) and rotation matrix (flattened to 9D) from the pose matrix P and concatenate them into a 12-dimensional pose feature vector P'. Next, cores 610-649 implement an attention-based fusion module: This module calculates the attention weights of the visual features V on the tactile features T and pose features P', and vice versa, using a scaled dot-product attention formula (involving the calculation of the key-value query matrix and the softmax function, both executed in parallel on the IPU cores). The weighted features are concatenated or summed and then input into a small feedforward neural network (e.g., two fully connected layers, each with 1024 neurons, using GeLU activation function, and preloaded network parameters), which is computed in parallel across the core group. The final output is multi-dimensional perceptual interaction feature data F of a fixed dimension (e.g., 512 dimensions), stored in the final result memory area 0xE0000000.

[0025] Preferably, performing visual feature analysis based on the visual data to be processed includes: Extract target RGB color image data and target depth image data according to the visual data to be processed; Analyze the surface reflection characteristics based on the target RGB color image data to generate the object surface characteristic data; Filter outliers from the target depth map data to obtain standard depth data; the depth anomaly threshold is set to ±30% of the current average depth value; Construct point cloud based on standard depth data to obtain three-dimensional point cloud data; Calculate the surface normal vector and curvature change based on the 3D point cloud data. When the curvature change is greater than 0.05, mark it as an edge point to obtain the target contour feature. Calculate the target size based on the target contour features to obtain geometric parameter data; Analyze the center of gravity position and main axis direction based on the geometric parameter data to obtain the spatial posture data; The three-dimensional point cloud data, spatial posture data and object surface characteristic data are integrated with visual features to generate target visual feature data.

[0026] In this embodiment of the present invention, IPU core group A (e.g., cores 70-79) accesses the unprocessed visual data at memory address 0xA0000000, which is stored in the camera output format (e.g., packed 16-bit depth values interleaved with 24-bit RGB values). Core group A performs data unpacking operations: For each pixel location, the core reads the corresponding data block, interprets the 16-bit unsigned integer value as a depth value (in millimeters), and stores it in the memory area starting at address 0xA2000000, forming a 1920x1080 target depth map data. Simultaneously, the core interprets the following three 8-bit byte values as red, green, and blue components and stores them in the memory area starting at address 0xA1000000, forming a 1920x1080x3 target RGB color map data. This process is efficiently performed on the IPU core using pre-set offsets and data type conversion instructions to ensure that the RGB and depth information are separated and aligned. Intelligent Processing Unit core group B (e.g., cores 80-99) reads the target RGB color image data stored at address 0xA1000000. Core group B divides the image into non-overlapping 100x100 pixel blocks, with each core responsible for processing several blocks. For each block, the core calculates the average vector (AvgR, AvgG, AvgB) of the RGB values of all pixels within the block. Simultaneously, the core calculates the gray-level co-occurrence matrix (GLCM) for the block and extracts four standard texture features based on the GLCM: energy, contrast, correlation, and entropy. These calculations involve iterating over the pixels within the block, accumulating statistics, and are performed using pre-defined floating-point arithmetic instructions. Ultimately, a 7-dimensional feature vector (3 average color values + 4 texture features) is generated for each image block, forming the object surface characteristic data. Intelligent Processing Unit core group C (e.g., cores 100-119) accesses the target depth map data at address 0xA2000000. First, the core group C calculates the total number and sum of all depth values greater than zero in the depth map through parallel calculations, and calculates the global average depth value D_avg. This D_avg value is broadcast to all cores in the group through inter-core communication. Then, each core processes the depth map pixel area it is responsible for. For each pixel (i, j) depth value D(i, j), the core performs a comparison operation: check whether D(i, j) is in the interval [D_avg 0.7, D_avg 1.3]. If D(i, j) is out of this range, the pixel depth value is set to 0 (marked as invalid). Multiplication and comparison operations are performed using the arithmetic logic unit of the intelligent processing unit. The intelligent processing unit core group D (for example, core number 120-219) reads the standard depth data of address 0xA3000000 and reads the preloaded camera intrinsic parameter matrix K (containing focal length fx, fy and principal point cx, cy) from the constant memory area. Core group D divides the depth map, with each core responsible for a pixel sub-area. For each valid depth value D(i, j) (depth value is non-zero) in its responsible area, the core performs a three-dimensional coordinate calculation: Z=D(i, j) / 1000.0 (assuming the depth unit is millimeters, convert to meters); The resulting (X, Y, Z) triple represents a 3D point. The set of valid 3D points calculated by all cores constitutes the 3D point cloud data. The intelligent processing unit core group E (e.g., core numbers 220-319) accesses the 3D point cloud data at address 0xA4000000. For each point P in the point cloud (or processed by a core block), the core first searches for all neighboring points within its radius neighborhood (e.g., a radius of 0.02 meters) (using a pre-built KD tree or a neighborhood search algorithm executed in parallel on the intelligent processing unit). After obtaining the neighborhood point set, the core calculates the 3x3 covariance matrix for the point set. By performing singular value decomposition or eigenvalue decomposition on the covariance matrix (using the linear algebra acceleration unit of the intelligent processing unit), three eigenvalues λ1≤λ2≤λ3 are obtained. The surface normal vector is the eigenvector corresponding to the smallest eigenvalue λ1. The curvature change is calculated from the eigenvalues using the formula Curvature = λ1 / (λ1+λ2+λ3). If the calculated Curvature value exceeds the preset threshold of 0.05, point P is marked as an edge point. The indices or flags of all edge points constitute the object's contour features. The intelligent processing unit core group G (e.g., core numbers 340-359) reads the geometric parameter data at address 0xD0200000 (primarily used to determine the analysis range, such as the bounding box calculated in the previous step) and the 3D point cloud data at address 0xA4000000. Core group G first determines the point set that constitutes the object (e.g., all points within the bounding box). It then concurrently calculates the average of the (X, Y, Z) coordinates of all points in this point set to obtain the centroid location, Centroid = (avg(X), avg(Y), avg(Z)). Next, it calculates the 3x3 covariance matrix of this point set relative to the centroid location. Eigenvalue decomposition is performed on this covariance matrix to obtain three eigenvectors. These vectors represent the principal axis directions of the object and constitute the object's local coordinate system. The center of gravity position (3 floating-point numbers) and the principal axis orientations (typically represented as a 3x3 rotation matrix or quaternion, totaling 9 or 4 floating-point numbers) constitute the spatial pose data. The intelligent processing unit core group H (e.g., core numbers 360-369) is responsible for feature integration. Core group H reads the raw 3D point cloud data (or a sampled / simplified version thereof) from address 0xA4000000, the spatial pose data (center of gravity and principal axis orientations / rotation matrix) from address 0xD0300000, and the object surface characteristic data (e.g., per-point color or local texture descriptors) from address 0xD0100000. The integration process concatenates the center of gravity coordinates (3D), the rotation matrix (flattened to 9D), the geometric parameters, and the global statistics of the surface characteristic data (e.g., the average color and texture characteristics of the entire object surface, assuming 10D) to form a fixed-length feature vector. For example, 3+9+3+10 = 25 dimensions.Alternatively, a pre-trained graph neural network model (whose computational graph has been mapped to this core group) is applied, taking point clouds, poses, and surface features as node attributes as input, and the network outputs a compact global descriptor (e.g., 512 dimensions). This final vector or descriptor is the target visual feature data.

[0027] Of particular importance is the analysis of surface reflectance characteristics based on the RGB color image data of the captured object, including: Calculate the surface reflectance value according to the target RGB color map data to obtain the reflectance distribution map; The reflectivity distribution map is used to classify materials. The reflectivity threshold range is set. When the reflectivity is less than 0.2, it is judged as a matte material, 0.2-0.6 is a medium reflective material, and greater than 0.6 is a high reflective material, and the material type label is obtained; The surface texture complexity is analyzed based on the material type tag. The grayscale value variance within the local area of 8×8 pixels is calculated. A variance less than 10 indicates a smooth surface, 10-50 indicates a medium texture, and greater than 50 indicates a complex texture. The texture complexity map is obtained. The reflectivity distribution map is combined with the texture complexity map to construct a two-dimensional feature matrix with the horizontal axis being the reflectivity and the vertical axis being the texture complexity, to obtain the surface characteristic data of the object.

[0028] In this embodiment of the present invention, the multi-core parallel computing unit of the IPU chip converts the input RGB color image into the HSV color space and extracts the V channel value as the brightness value. A standard gray card is collected as a reference reflectance standard, with a fixed reflectance of 18%. The actual reflectance of each pixel is calculated by dividing the brightness value of each pixel by the incident light intensity in the shooting environment (obtained using a light meter) and multiplying it by a correction factor (obtained from gray card calibration). Parallel computing is performed on all pixels in the image to generate a reflectance distribution map of the same size as the original image. The values are normalized to a range of 0-1, and the data precision is maintained at 32-bit floating point to ensure calculation accuracy. First, three fixed reflectance threshold ranges are set: a reflectance value less than 0.2 is considered matte (assigned value 1), a reflectance value between 0.2 and 0.6 is considered medium reflective (assigned value 2), and a reflectance value greater than 0.6 is considered highly reflective (assigned value 3). Using a vectorized operation instruction set, the IPU chip compares the value of each pixel in the reflectance distribution map with the threshold value to generate a material type label map of the same size. To improve classification accuracy, a 5×5 pixel neighborhood smoothing process was performed on the material boundary to eliminate material classification noise caused by uneven lighting, resulting in a continuous material type labeling map. The RGB image was converted to grayscale using the formula grayscale = 0.299 × R + 0.587 × G + 0.114 × B. The grayscale image was then divided into non-overlapping 8×8 pixel blocks, and the grayscale value variance was calculated for each block. The calculation first obtained the grayscale mean of the 64 pixels within each block, then the sum of the squares of the differences between each pixel's grayscale value and the mean was calculated, and finally divided by 64 to obtain the variance. Based on the variance value, areas with a variance less than 10 were labeled as smooth (assigned value 1), areas with a variance between 10 and 50 were labeled as medium texture (assigned value 2), and areas with a variance greater than 50 were labeled as complex texture (assigned value 3), generating a texture complexity map. The reflectance distribution map and texture complexity map obtained in the previous steps were imported into the IPU chip for data fusion. The two images were first spatially registered to ensure pixel correspondence. Next, a 256×256 two-dimensional feature matrix is created, with reflectivity (classified into 256 levels) on the horizontal axis and texture complexity (classified into 256 levels) on the vertical axis. For each pixel in the original image, its reflectivity value is extracted as the horizontal axis and its texture complexity value as the vertical axis, and the count is accumulated at the corresponding position in the feature matrix. The feature matrix is normalized to generate a two-dimensional distribution feature matrix representing the surface characteristics of the object. The value of each element in the matrix represents the proportion of pixels with a specific combination of reflectivity and texture complexity.

[0029] Preferably, performing contact characteristic analysis based on the tactile data to be processed includes: Convert the pressure value of the tactile data to be processed, establish a mapping table from voltage value to standard pressure, and obtain the perceived pressure data stream; Performing spatial distribution normalization processing on the perceived pressure data stream to obtain a normalized pressure distribution map; The mean, standard deviation, maximum and minimum values of the pressure values in the contact area are calculated according to the normalized pressure distribution map to obtain the pressure statistical characteristics; The contact area size is evaluated based on the pressure statistical characteristics. When the contact area is less than 1cm², it is a point contact; 1-5cm² is a line contact; and greater than 5cm² is a surface contact, and the contact morphology data is obtained; Perform time series analysis on the pressure statistical characteristics and calculate the pressure change frequency. When the frequency is greater than 20Hz, it is judged as a rough surface, and when it is less than 5Hz, it is judged as a smooth surface, thus obtaining the surface texture characteristics. The pressure statistical characteristics, contact morphology data and surface texture characteristics are integrated to obtain contact characteristic data.

[0030] In an embodiment of the present invention, a raw voltage signal is received from a tactile sensor array, and the voltage range is 0-5V. In order to establish an accurate voltage-pressure mapping relationship, the tactile sensor is calibrated in advance using a standard pressure gauge. In the pressure range of 0-1000kPa, the voltage value corresponding to a pressure point is recorded every 50kPa, and a total of 21 calibration points are obtained. These 21 calibration points are stored to form an initial mapping table. For the voltage values between the calibration points, the corresponding pressure values are calculated using a cubic spline interpolation algorithm. The voltage data of the 256 sensing points of the tactile sensor array (16×16) are converted into standard pressure values (unit: kPa) in real time at a rate of 200 frames per second to form a sensed pressure data stream. The 256 pressure point data of the sensor array are reconstructed into a 16×16 two-dimensional pressure matrix. Then, the 16×16 matrix is interpolated and expanded to a 64×64 resolution to improve the spatial resolution. Next, the maximum-minimum normalization algorithm is used to normalize the value P at each pressure point: P_normalized = (P - P_min) / (P_max - P_min), mapping all pressure values to the range of 0-1. The IPU chip's tensor processor performs batch normalization, achieving a processing speed of 1000 frames per second. Finally, a 5×5 Gaussian filter is applied to smooth the image and eliminate noise interference at the edges of the sensor array, resulting in a 64×64 normalized pressure distribution map. Areas with pressure values greater than a threshold of 0.1 are identified as valid contact areas. Statistical calculations are performed on all pressure values within this contact area, and the arithmetic mean of all pressure points within the contact area is calculated in parallel as the average pressure value. The pressure standard deviation is calculated by subtracting the mean, squaring the result, and then taking the square root of the summed average. The maximum and minimum pressure values within the contact area are also determined. The calculated average pressure value, pressure standard deviation, maximum pressure value, and minimum pressure value are combined into a four-dimensional feature vector to form the pressure statistical feature data. These calculations are performed immediately upon the arrival of each frame of data, with computational latency kept to within 1 millisecond. Since each pixel corresponds to a physical area on the sensor array, the actual contact area is calculated by multiplying the number of pixels by the physical area coefficient of a single pixel (0.25 mm² / pixel). Based on the calculated results, a multi-branch judgment is performed: when the contact area is less than 1 cm², it is marked as a point contact (numerical code 1); when the contact area is between 1 and 5 cm², it is marked as a line contact (numerical code 2); and when the contact area is greater than 5 cm², it is marked as a surface contact (numerical code 3). The judgment result, along with the actual area value, is saved as contact morphology data. A fast Fourier transform is performed on the average pressure value sequence to obtain frequency domain characteristics, calculate the frequency domain energy distribution, and determine the dominant frequency components. To filter out interference, a bandpass filter (1-50 Hz) is used to extract the effective frequency components.The dominant frequency is calculated by finding the frequency corresponding to the maximum energy in the spectrum. The surface type is determined based on the dominant frequency value: when the frequency is greater than 20 Hz, it is judged as a rough surface (numerical code 3); when the frequency is between 5 and 20 Hz, it is judged as a moderately rough surface (numerical code 2); and when the frequency is less than 5 Hz, it is judged as a smooth surface (numerical code 1). The judgment result and the dominant frequency value together constitute the surface texture feature. Features of different dimensions are z-score normalized to ensure that the magnitude of each feature is consistent. Then, the normalized features are spliced into a nine-dimensional vector in a predefined order. It contains the average pressure value, pressure standard deviation, maximum pressure value, minimum pressure value, contact morphology code, contact area value, surface texture code, dominant frequency value, and timestamp. The nine-dimensional vector is packaged into a structured data format, and a frame header and checksum are added to form the final contact characteristic data.

[0031] Preferably, step S3 includes the following steps: Step S31: summing the visual score, tactile score, and motion score according to a weight ratio of 4:3:3 to obtain a comprehensive score; Step S32: Based on the comprehensive score, grid sampling is performed at 5 mm intervals through the three-dimensional point cloud data in the target visual feature data, and the score value of each sampling point is calculated to obtain the score spatial distribution; Step S33: extract the local maximum of the score space distribution, select the highest score point within a radius of 10 mm as a candidate, retain a maximum of 30 candidate points, and obtain a candidate operation point set; Step S34: Calculate the operation direction for each point in the candidate operation point set, and generate multiple directions at intervals of 15° within the range of ±45° based on the surface normal vector to obtain a candidate direction group; Step S35: Generate multiple pose solutions based on the candidate operation point sets and the direction candidate group combinations.

[0032] In an embodiment of the present invention, the three types of scoring data are unified into a standard range of 0-100. Then, fixed weight coefficients are set: the visual score weight is 0.4, the tactile score weight is 0.3, and the motion score weight is 0.3. The weight ratio is strictly maintained at 4:3:3. Subsequently, the three scores are multiplied: the visual score is multiplied by 0.4, the tactile score is multiplied by 0.3, and the motion score is multiplied by 0.3 to calculate three weighted components. Finally, the three weighted components are added to generate a comprehensive score value in the range of 0-100. This calculation process uses vectorized instructions, and the processing delay is controlled within 0.5 milliseconds. First, the three-dimensional point cloud data in the target visual feature data is read. The data format is an N×6 matrix, which contains the spatial coordinates (x, y, z) and color information (r, g, b) of N points. The space is divided into a cubic grid with a side length of 5 mm. Only one sampling point (grid center point) is retained in each grid to achieve point cloud downsampling. The spatial index unit of the IPU chip constructs a KD tree index for each sampling point to accelerate the nearest neighbor search. For each sampling point, its visual feature score, tactile feature score and motion feature score are calculated respectively, and then the weight coefficient 4:3:3 is applied for weighted summation to obtain the comprehensive score of the point. The 3D data processing unit of the IPU chip constructs the spatial coordinates of the point and the corresponding score value into a four-dimensional matrix in the format of (x, y, z, score) to form a score space distribution data structure. All sampling points are sorted in descending order from high to low according to the score. Starting from the highest-scoring point, it is marked as a candidate operation point. At the same time, the spherical area search unit of the IPU chip performs a spherical area query with a radius of 10mm with this point as the center, and marks all other points in the area as non-candidate points. Then process the highest-scoring points among the remaining unmarked points and repeat the above steps. The counting control unit of the IPU chip limits the number of candidate points to no more than 30. If the number of candidate points reaches 30, the extraction is terminated. If the score is lower than the preset threshold of 75 points, no candidate points will be extracted. The spatial coordinates and scores of all candidate operation points are organized into a structured array, forming a candidate operation point set. This is stored in a 30×4 matrix (x, y, z, score). A point cloud within a 15mm radius, centered at each candidate point, is collected and the covariance matrix is calculated. The covariance matrix is then subjected to eigenvalue decomposition, and the eigenvector corresponding to the minimum eigenvalue is taken as the surface normal, oriented toward the exterior of the object. Next, a local coordinate system is established with the normal vector as the z-axis, with the x and y axes perpendicular to the normal vector. Within this coordinate system, direction vectors are generated at 15° intervals within ±45° on either side of the normal vector. Seven directions (including the normal vector itself) are sampled in both the xz and yz planes, for a total of 13 directions. The IPU chip stores all direction vectors for each candidate point as unit vectors, forming a candidate direction set. Each pose is defined as a seven-dimensional vector: position coordinates (x, y, z) and a quaternion representation of the pose (qw, qx, qy, qz).For each candidate point (up to 30), along with each of its direction vectors (13 per point), the direction vectors are converted into quaternions to form a complete pose. The IPU chip's parallel processing unit processes multiple pose combinations simultaneously, generating up to 30 × 13 = 390 pose solutions. Each pose solution is then associated with the score of the corresponding candidate point. Pose solutions with the same score are sorted in ascending order based on the angle between the normal vector and the direction of gravity.

[0033] Preferably, step S4 includes the following steps: Step S41: performing target operations using an adaptive execution strategy, and obtaining real-time monitoring status data based on the real-time monitoring status of the IPU chip; Step S42: extracting pressure distribution data based on real-time monitoring status data, calculating the number of contact points and distribution uniformity, marking less than two contact points as insufficient contact, unilateral concentration as biased contact, and uniform distribution as balanced contact, and generating a contact distribution state; Step S43: Identify sliding risk based on real-time monitoring status data and generate sliding status data; Step S44: extracting position data based on the real-time monitoring status data and calculating the position deviation. When the deviation exceeds a preset threshold, it is identified as a trajectory deviation and trajectory status data is generated. Step S45: Perform a comprehensive stability evaluation based on the contact distribution state, sliding state data, and trajectory state data to obtain real-time stability data.

[0034] In an embodiment of the present invention, during execution, the multi-channel data acquisition module of the IPU chip synchronously collects four types of sensor data: 16×16 tactile array pressure data (unit: kPa), 6-axis force / torque sensor data (unit: N and N·m), joint motor encoder position data (accuracy 0.01 degrees), and target pose data provided by the vision system. A unified timestamp is added to all incoming data with an accuracy of 1ms. Through the IPU chip's cache, a time-series data stream is constructed, and the most recent 50 frames of data are stored in the on-chip memory. Feature vectors are extracted from the data stream with a period of 10ms, including pressure distribution features, torque features, position features, and velocity features, forming 128-dimensional real-time monitoring status data. The pressure threshold is set to 5kPa, and points with pressure values greater than the threshold are marked as contact points (value 1), and points with pressure values less than the threshold are marked as non-contact points (value 0), generating a binary contact map. 8-connected region marking is performed on the binary contact map, and the number of connected domains is calculated as the number of contact points. Next, the first- and second-order moments of the contact area are calculated to obtain the contact center coordinates and principal axis directions. The contact area is evenly divided into four quadrants, and the proportion of contact points within each quadrant is calculated. When the total number of contact points is less than two, it is marked as "insufficient contact" (code 1); when the proportion of contact points in any quadrant exceeds 70%, it is marked as "biased contact" (code 2); and when the proportion of contact points in the four quadrants differs by less than 30%, it is marked as "balanced contact" (code 3). The IPU chip calculates the time difference of the pressure distribution data within 10 frames (50ms) to obtain a pressure change rate matrix. Based on the pressure change rate matrix, the direction and velocity of the "pressure flow" on the contact surface are estimated, and a 2×16×16 flow field map is constructed. When the flow field amplitude exceeds the threshold of 10 kPa / s and the directional consistency (calculated by averaging the cosine of the angle between the main direction and each local direction) is greater than 0.7, a slip risk is determined. Simultaneously, a fast Fourier transform is performed on the tangential force data from the six-axis force sensor to calculate the energy contribution in the 40-200Hz frequency band. Slip is further confirmed when the energy contribution exceeds a threshold of 15%. Combining these two indicators, the slip risk is quantified on a scale of 0-100 and categorized into three levels: "no slip" (0-30), "slight slip" (31-70), and "severe slip" (71-100). This generates slip status data. The actual and planned position data of the end effector are extracted from the real-time monitoring status data. The position data is formatted as a seven-dimensional vector, consisting of position coordinates (x, y, z) and attitude quaternions (qw, qx, qy, qz). The IPU chip's posture error calculation unit calculates the Euclidean distance between the actual and planned positions (position error) and the modulus of the quaternion difference (attitude error). The position error threshold is set to 5mm, and the attitude error threshold is set to 0.1 (approximately 5.7 degrees). When the position error or attitude error exceeds the respective threshold, the current state is marked as "trajectory deviation" and the deviation duration and maximum deviation value are calculated.Based on the degree of deviation, duration, and trend, the trajectory state is quantified into a numerical value on a scale of 0-100, with larger values indicating more severe deviation. The timestamp and location coordinates of the deviation are also recorded to generate trajectory state data. The three types of state data generated in the previous steps are received: contact distribution state (discrete coding), sliding state data (0-100 numerical values), and trajectory state data (0-100 numerical values). First, the discrete codes of the contact distribution state are converted into numerical values: insufficient contact (code 1) is mapped to 90 points, biased contact (code 2) is mapped to 60 points, and balanced contact (code 3) is mapped to 10 points. These scores are then unified into a 0-100 range (lower scores indicate better status). Weight coefficients are assigned to the three types of state data based on the task type: for precision grasping tasks, the contact state is weighted at 0.4, the sliding state at 0.4, and the trajectory state at 0.2; for transport tasks, the weights are set at 0.3, 0.5, and 0.2, respectively. A weighted summation is performed to generate a stability score within the range of 0-100. At the same time, the stability score is divided into three intervals: 0-30 is "highly stable", 31-70 is "basic stable", and 71-100 is "unstable", and the score is combined with the status label to generate real-time stability data.

[0035] It is particularly important that in step S1, effective grasping points are screened based on multiple sets of grasping posture solutions, and an adaptive grasping execution strategy is formulated based on the effective grasping points, including: Calculate the expected contact area based on multiple pose schemes. When the expected contact area is less than the set threshold, it is marked as unsuitable for operation, and the operation points are removed to obtain area filtered data. The stability of the operation schemes in the area filtering data is evaluated. The positional relationship between the force action line and the target center of gravity is calculated. When the distance between the force action line and the center of gravity is greater than 50% of the target minimum size, it is marked as an unstable operation and the operation points are eliminated to obtain the stability filtering data. Perform feasibility analysis on the operation schemes in the stability filtering data to check whether there is a valid parameter combination that can achieve the target posture. When the parameters exceed the limit range, mark it as infeasible and eliminate the operation points to obtain the valid operation points. The contact mode is analyzed according to the effective operation point, and an adaptive execution strategy is formulated based on the target visual feature data in the multi-dimensional perception interaction feature data.

[0036] In this embodiment of the present invention, the end-effector model (a three-dimensional mesh model accurate to 0.1 mm) is transformed to the spatial position corresponding to each pose solution. Then, collision detection is performed between the end-effector model and the target point cloud model. The two models are discretized using a voxelization method (voxel size 1 mm³), all overlapping voxels are identified, the number of overlapping voxels is counted and multiplied by the surface area coefficient of a single voxel (1 mm²), and the expected contact area value (unit: mm²) is obtained. A contact area threshold is set to 200 mm². Pose solutions with an expected contact area less than the threshold are marked as "unsuitable for operation" (marker value 0), and pose solutions with an expected contact area greater than or equal to the threshold are marked as "suitable for operation" (marker value 1). All solutions marked as "unsuitable for operation" are filtered out to obtain area-filtered data. The center of gravity coordinates of the target object are calculated based on the target three-dimensional point cloud data and a preset density distribution (uniform density or density gradient inferred from the material). Then, based on the pose of the operation solution, the direction and point of action of the grasping force are determined, and the force line equation is constructed by connecting the point of action and the extension line in the opposite direction of the force. The shortest distance d between the line of action of the force and the center of gravity is calculated, using the formula for the perpendicular distance from the center of gravity to the line. Simultaneously, the dimensions of the target object along the three principal axes are calculated, with the minimum value taken as the target minimum dimension L_min. The ratio of d to L_min is calculated. If the ratio exceeds 0.5 (i.e., the distance between the line of action of the force and the center of gravity is greater than 50% of the target minimum dimension), the action scenario is marked as "unstable" (with a value of 0). Otherwise, it is marked as "stable" (with a value of 1). All scenarios marked as "unstable" are filtered out to generate stability-filtered data. The target pose is converted into a parameter combination in joint space using an inverse kinematics algorithm. The nonlinear system of equations is solved using a modified Newton-Raphson iteration method, with an upper limit of 20 iterations and a convergence threshold of 0.001. The calculated joint parameters are compared with the device's constraints. The joint angles are limited to a range of [-180°, 180°], the joint velocities are limited to 90° per second, and the joint accelerations are limited to 180° per second squared. The condition number of the Jacobian matrix is calculated, and a condition number greater than 1000 indicates a near-singular configuration. The robot then executes a path plan from its current position to the target pose, checking for collisions with the environment along the path. Taking these four factors into consideration, the proposed operation is marked as "infeasible" (with a value of 0) if any of the constraints are exceeded, and "feasible" (with a value of 1) otherwise. All "infeasible" solutions are filtered out to obtain a valid operation point. The curvature, roughness, and material properties of the contact area are extracted from the target's visual feature data. Curvature is obtained by fitting a local quadratic surface and calculating the principal curvatures. Roughness is calculated by the standard deviation of the local height variation. Material properties are inferred from reflectivity and color features.Contact modes are divided into four categories based on surface features: point contact (contact area less than 300mm² and high surface curvature), line contact (extension greater than 30mm in one dimension and less than 10mm in another dimension), surface contact (contact area greater than 500mm² and low surface curvature), and shape contact (complex contact relying on geometric matching). Execution parameters are then formulated for different contact modes: for point contact, high-precision positioning requirements (position error less than 1mm) and a moderate grasping force (10N) are set; for line contact, posture stability requirements perpendicular to the line direction (posture error less than 2°) and a directional distributed grasping force (5-15N) are set; for surface contact, uniform pressure distribution requirements and a low grasping force (5N) are set; and for shape contact, precise posture alignment requirements (position error less than 0.5mm, posture error less than 1°) and an adaptive grasping force (3-20N) are set. The IPU chip integrates the execution parameters, grasping point location, grasping direction, and contact type into an execution instruction package, forming an adaptive execution strategy.

[0037] Preferably, the present invention further provides a multimodal data processing fusion system for an IPU chip, which executes the multimodal data processing fusion method for an IPU chip as described above. The multimodal data processing fusion system for an IPU chip includes: The IPU multimodal initialization module is used to obtain multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism based on the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; A multi-dimensional feature fusion processing module is used to perform multi-modal data parallel processing on multi-modal spatiotemporal perception data using the IPU chip based on the IPU chip's multi-modal parallel processing strategy, and to perform multi-dimensional interactive feature fusion to obtain multi-dimensional perception interaction feature data; wherein the multi-dimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; The multimodal scoring planning module is used to calculate the visual score based on the target visual feature data; calculate the tactile score based on the contact characteristic data; calculate the motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; and generate multiple sets of pose solutions based on the candidate operation point sets. The adaptive execution monitoring module is used to screen effective operating points based on multiple sets of posture solutions and formulate adaptive execution strategies based on the effective operating points; it uses the adaptive execution strategies to perform operation control and conduct comprehensive stability evaluation to obtain real-time stability data.

[0038] By constructing a multimodal parallel processing strategy specifically for IPU chips, this invention fully leverages the high parallel computing capabilities and flexible architectural design of the IPU chip, significantly improving the processing efficiency of heterogeneous multimodal data streams. Compared to traditional CPU or GPU architectures, the method of this application reduces the latency of multimodal data processing by approximately 65%, significantly improving the system's real-time response capability. This overcomes the limitations of traditional architectures in processing data with different characteristics and can better meet the needs of application scenarios with high real-time requirements, such as robotics, intelligent manufacturing, and assisted medical care. In terms of capturing spatiotemporal correlations, the method achieves synchronous parallel processing of multimodal data, allowing the spatiotemporal correlation information between different modal data, such as visual, tactile, and posture, to be fully mined and utilized. Through a multidimensional interactive feature fusion mechanism, the features of different modalities are deeply integrated, generating richer and more reliable perceptual information than a single modality. This deep fusion of multimodal data enhances the system's perception accuracy of the environment. Through the comprehensive balance of a multidimensional scoring system (visual, tactile, and motion scores), the optimal operating point can be more intelligently identified, avoiding the one-sidedness and instability caused by single-modality evaluation. It fully leverages the advantages of the IPU chip in parallel computing and efficient data processing, effectively solves the bottleneck problem of traditional processors in multimodal data processing, significantly improves the real-time performance and stability of the system, and provides strong support for the efficient implementation of multimodal data fusion in complex application scenarios.

[0039] The present invention is therefore intended to be illustrative and non-restrictive in all respects, with the scope of the invention being defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the application documents are intended to be embraced therein.

[0040] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal data processing fusion method for IPU chips, characterized in that: The following steps are involved: Step S1: Acquire multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism according to the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; Step S2: Based on the IPU chip's multimodal parallel processing strategy, the IPU chip is used to perform multimodal data parallel processing on the multimodal spatiotemporal perception data, and multidimensional interaction feature fusion is performed to obtain multidimensional perception interaction feature data; wherein the multidimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; Step S3: Calculate a visual score based on the target visual feature data; calculate a tactile score based on the contact feature data; calculate a motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; generate multiple pose solutions based on the candidate operation point sets; Step S4: Filter effective operation points based on multiple groups of posture solutions, and formulate an adaptive execution strategy based on the effective operation points; use the adaptive execution strategy to perform operation processing, and conduct a comprehensive stability evaluation to obtain real-time stability data.

2. The multimodal data processing and fusion method for IPU chips according to claim 1 is characterized in that: Acquiring multimodal spatiotemporal perception data in step S1 includes: The IPU chip memory is divided into three independent buffers to obtain a sensor data buffer pool. The three independent buffers include a visual data buffer, a tactile data buffer, and a position data buffer. The size of the visual data buffer is 10MB, the size of the tactile data buffer is 2MB, and the size of the position data buffer is 1MB. Deploy multimodal data collection nodes; Based on the multimodal data acquisition node, visual, tactile and position information are continuously collected, and three-way perception signals are collected in parallel to be stored in the sensor data buffer pool; Extracting the original sensor data stream from the sensor data buffer pool; performing sampling frequency discrimination on the original sensor data stream based on a preset standard sampling frequency threshold, and obtaining high-frequency acquisition data and low-frequency acquisition data respectively; Perform weighted average downsampling processing on high-frequency collected data to obtain down-frequency synchronous data; perform interpolation and enhancement processing on low-frequency collected data to obtain up-frequency synchronous data; The down-converted synchronization data and the up-converted synchronization data are integrated into spatiotemporal perception data to generate multimodal spatiotemporal perception data.

3. The multimodal data processing and fusion method for IPU chips according to claim 2, characterized in that: Deployment of multimodal data collection nodes includes: A tactile sensor array is configured, with each sensing unit having a size of 5 mm × 5 mm, to deploy a tactile sensing device; Equipped with an RGB-D camera with a resolution of 1280×720 pixels, a depth resolution of 640×480 pixels, a field of view of 65°×40°, and a working distance range of 0.5-3m to deploy visual perception devices; Configuring position sensors to deploy motion sensing devices; A high-speed USB 3.0 interface was used to connect the visual perception device, an EtherCAT bus was used to connect the touch perception device, and a CAN bus was used to connect the joint encoder and torque sensor. The communication cycle was set to 1ms to obtain a multimodal data transmission channel. The multimodal data transmission channel is connected to the high-speed I / O port of the IPU chip, and the DMA channel is configured for zero-copy data transmission to obtain a multimodal data acquisition node.

4. The multimodal data processing and fusion method for IPU chips according to claim 1, characterized in that: In step S1, the IPU chip multimodal processing parallelism is configured according to the multimodal spatiotemporal perception data, and the IPU chip multimodal parallel processing strategy is initialized, including: Calculate the signal-to-noise ratio and integrity of each modality based on multimodal spatiotemporal perception data to generate a data quality score; The weights of different modal data are assigned to the multimodal spatiotemporal perception data through data quality scoring to obtain the multimodal perception data weights; Evaluate the amount of data for each modality based on multimodal spatiotemporal perception data; Allocate IPU chip memory resources according to the amount of data in each modality, where the IPU chip memory resources include allocated visual data processing area, tactile data processing area, and position data processing area; Configure the IPU chip's multimodal processing parallelism based on the data quality score and generate a core task allocation table; Establish IPU inter-core communication strategy based on the core task allocation table; The multimodal parallel processing strategy is initialized based on the IPU chip memory resources, core task allocation table and IPU inter-core communication strategy to obtain the IPU chip multimodal parallel processing strategy.

5. The multimodal data processing and fusion method for IPU chips according to claim 1, characterized in that: Step S2 includes: Step S21: Based on the multimodal parallel processing strategy of the IPU chip, the multimodal spatiotemporal perception data is subjected to multimodal data parallel processing control to obtain visual data to be processed, tactile data to be processed, and position data to be processed; Step S22: performing visual feature analysis based on the visual data to be processed to generate target visual feature data; Step S23: performing contact characteristic analysis based on the tactile data to be processed to obtain contact characteristic data; Step S24: performing coordinate system conversion on the position data to be processed to obtain spatial position data; Step S25: Calculate the transformation matrix based on the spatial position data to generate a pose matrix; Step S26: Perform multi-dimensional interactive feature fusion on the target visual feature data, contact characteristic data and posture matrix to generate multi-dimensional perception interactive feature data.

6. The multimodal data processing and fusion method for IPU chips according to claim 5, characterized in that: Visual feature analysis based on the visual data to be processed includes: Extract target RGB color image data and target depth image data according to the visual data to be processed; Analyze the surface reflection characteristics based on the target RGB color image data to generate the object surface characteristic data; Filter outliers from the target depth map data to obtain standard depth data; the depth anomaly threshold is set to ±30% of the current average depth value; Construct point cloud based on standard depth data to obtain three-dimensional point cloud data; Calculate the surface normal vector and curvature change based on the 3D point cloud data. When the curvature change is greater than 0.05, mark it as an edge point to obtain the target contour feature. Calculate the target size based on the target contour features to obtain geometric parameter data; Analyze the center of gravity position and main axis direction based on the geometric parameter data to obtain the spatial posture data; The three-dimensional point cloud data, spatial posture data and object surface characteristic data are integrated with visual features to generate target visual feature data.

7. The multimodal data processing and fusion method for IPU chips according to claim 5, characterized in that: Contact characteristic analysis based on the tactile data to be processed includes: Convert the pressure value of the tactile data to be processed, establish a mapping table from voltage value to standard pressure, and obtain the perceived pressure data stream; Performing spatial distribution normalization processing on the perceived pressure data stream to obtain a normalized pressure distribution map; The mean, standard deviation, maximum and minimum values of the pressure values in the contact area are calculated according to the normalized pressure distribution map to obtain the pressure statistical characteristics; The contact area size is evaluated based on the pressure statistical characteristics. When the contact area is less than 1cm², it is a point contact; 1-5cm² is a line contact; and greater than 5cm² is a surface contact, and the contact morphology data is obtained; Perform time series analysis on the pressure statistical characteristics and calculate the pressure change frequency. When the frequency is greater than 20Hz, it is judged as a rough surface, and when it is less than 5Hz, it is judged as a smooth surface, thus obtaining the surface texture characteristics. The pressure statistical characteristics, contact morphology data and surface texture characteristics are integrated to obtain contact characteristic data.

8. The multimodal data processing and fusion method for IPU chips according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: summing the visual score, tactile score, and motion score according to a weight ratio of 4:3:3 to obtain a comprehensive score; Step S32: Based on the comprehensive score, grid sampling is performed at 5 mm intervals through the three-dimensional point cloud data in the target visual feature data, and the score value of each sampling point is calculated to obtain the score spatial distribution; Step S33: extract the local maximum of the score space distribution, select the highest score point within a radius of 10 mm as a candidate, retain a maximum of 30 candidate points, and obtain a candidate operation point set; Step S34: Calculate the operation direction for each point in the candidate operation point set, and generate multiple directions at intervals of 15° within the range of ±45° based on the surface normal vector to obtain a candidate direction group; Step S35: Generate multiple pose solutions based on the candidate operation point sets and the direction candidate group combinations.

9. The multimodal data processing and fusion method for IPU chip according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: performing target operations using an adaptive execution strategy, and obtaining real-time monitoring status data based on the real-time monitoring status of the IPU chip; Step S42: extracting pressure distribution data based on real-time monitoring status data, calculating the number of contact points and distribution uniformity, marking less than two contact points as insufficient contact, unilateral concentration as biased contact, and uniform distribution as balanced contact, and generating a contact distribution state; Step S43: Identify sliding risk based on real-time monitoring status data and generate sliding status data; Step S44: extracting position data based on the real-time monitoring status data and calculating the position deviation. When the deviation exceeds a preset threshold, it is identified as a trajectory deviation and trajectory status data is generated. Step S45: Perform a comprehensive stability evaluation based on the contact distribution state, sliding state data, and trajectory state data to obtain real-time stability data.

10. A multimodal data processing fusion system for IPU chips, characterized in that: For executing the multimodal data processing and fusion method for an IPU chip according to claim 1, the multimodal data processing and fusion system for an IPU chip comprises: The IPU multimodal initialization module is used to obtain multimodal spatiotemporal perception data; configure the IPU chip multimodal processing parallelism based on the multimodal spatiotemporal perception data, and initialize the IPU chip multimodal parallel processing strategy; A multi-dimensional feature fusion processing module is used to perform multi-modal data parallel processing on multi-modal spatiotemporal perception data using the IPU chip based on the IPU chip's multi-modal parallel processing strategy, and to perform multi-dimensional interactive feature fusion to obtain multi-dimensional perception interaction feature data; wherein the multi-dimensional perception interaction feature data includes target visual feature data, contact characteristic data, and posture matrix; The multimodal scoring planning module is used to calculate the visual score based on the target visual feature data; calculate the tactile score based on the contact characteristic data; calculate the motion score based on the pose matrix; perform a weighted sum based on the visual score, tactile score, and motion score, and then identify a set of candidate operation points; and generate multiple sets of pose solutions based on the candidate operation point sets. The adaptive execution monitoring module is used to screen effective operating points based on multiple sets of posture solutions and formulate adaptive execution strategies based on the effective operating points; it uses the adaptive execution strategies to perform operation control and conduct comprehensive stability evaluation to obtain real-time stability data.

Images