Multi-modal neural network optimization method for intention prediction in human-robot collaboration

Abstract

This invention relates to the field of artificial intelligence and neural network optimization technology, and discloses a multimodal neural network optimization method for intention prediction in human-machine collaboration. The method includes: resampling to generate a multimodal state tensor and inputting it into the neural network computation graph; solving the cross-modal feature fusion tensor using a cross-attention gating network; calculating the intention uncertainty index based on the fused feature vector; adjusting the activation depth of the dynamic deep routing network using gating activation coefficients; triggering a change in the computation graph topology when the mean of the sliding queue monotonically crosses a preset threshold point, and maintaining a persistent allocation of GPU memory space when suspending higher-order refinement layers; outputting the prediction result to calculate the geometric boundary function of the safety domain space, and controlling the robot to avoid obstacles. This invention eliminates GPU memory reallocation jitter and inference latency fluctuations caused by neural network computing power switching, achieving low-latency, high-precision prediction and compliant robot control coordination.

Description

Technical Field

[0001] This invention relates to a multimodal neural network optimization method for intention prediction in human-computer collaboration, belonging to the field of artificial intelligence and neural network optimization technology. Background Technology

[0002] In human-machine shared workspaces, proactive safety protection typically relies on real-time prediction of human movement intentions and future trajectories. Existing methods generally integrate multimodal perception data collected by visual sensors and wearable inertial measurement units to extract changes in human skeletal posture and predict movement trends. This is used to adaptively adjust the robot's safety domain and achieve compliant force control interaction. In real-time perception and neural network inference, to reduce inference latency, a dynamic routing mechanism based on prediction uncertainty indices is usually adopted. This mechanism dynamically adjusts the activation depth of the neural network according to the certainty of input features to reduce computational overhead and maintain prediction accuracy as much as possible. However, during the data acquisition phase, the sampling periods of different modal sensors are not consistent, and the sampling timing is also asynchronous. The sampling times of visual sensors and wearable inertial measurement units within the same time window are difficult to directly correspond. If feature fusion is performed directly, the correspondence between different modalities is prone to deviation.

[0003] Besides the physical limitations of sensing data sampling and sensor hardware response, there are also shortcomings at the control algorithm and software strategy levels. For example, Chinese invention patent CN121756362B discloses a control method and system for a collaborative robot arm, which infers operational intent and uncertainty indices through multimodal data and adaptively updates impedance parameters and safe feasible regions based on the uncertainty indices. However, this scheme does not consider the underlying computational overhead caused by the neural network when performing computational power allocation and graph topology switching; during its dynamic adjustment of neural network depth, when the uncertainty index fluctuates drastically near the critical threshold, it leads to frequent reconstruction of the computation graph and memory depletion. Repeated requests between computation graphs lead to additional inference delays and fluctuations. This inference delay caused by computation graph reconstruction makes it difficult for upper-level impedance parameters and safety boundary calculations to match the actual working conditions in real time. This results in lag in physical control response and the risk of contact impact. Frequent computation graph reconstruction causes frequent switching of video memory context, clearing of instruction pipelines, and repeated allocation of tensor memory, which in turn causes additional inference delays and computational power loss, making it difficult to keep the inference time stable. If static pruning or fixed dimensionality reduction is simply used to reduce the amount of computation, it is easy to cause a significant increase in prediction error when there is limb occlusion or sensor signal drift, which further causes impedance matching imbalance at the robot end and lag in updating the safety buffer.

[0004] Therefore, how to eliminate computation graph reconstruction jitter and inference latency fluctuations in the computing power scheduling process of multimodal dynamic neural networks, and achieve low-latency, high-precision intention prediction and physical control coordination, has become the technical problem to be solved by this invention. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of the present invention is as follows: A multimodal neural network optimization method for intention prediction in human-computer collaboration, comprising the following steps: Step S1: Obtain the multimodal state tensor; the multimodal state tensor is generated by resampling the three-dimensional coordinate temporal tensor of human skeletal joints collected by the visual sensor and the motion feature temporal tensor collected by the wearable inertial measurement unit using the asynchronous time warp alignment operator. Step S2: Input the multimodal state tensor into the neural network computation graph containing the multimodal cross-attention gating network and the dynamic deep routing network, solve the cross-modal feature fusion tensor, and output the fused feature vector; Step S3: Calculate the intent uncertainty index based on the information entropy of the fused feature vector and the trace of the covariance matrix of the preceding trajectory, and solve the gated activation coefficients to adjust the activation depth of the dynamic deep routing network. Step S4: Maintain a sliding queue of gating activation coefficients. When the mean of the elements in the sliding queue exceeds a preset threshold, trigger a change in the topology of the neural network computation graph. Step S5: When the suspended state is triggered, the activation depth is limited to the basic feature extraction layer, the forward path of the higher-order cross-attention refinement layer is blocked, and its memory space is kept permanently allocated; when the suspended state is triggered, full-depth calculation is resumed. Step S6: The neural network computes the probability distribution vector of the operation intention and the three-dimensional motion trajectory prediction tensor, calculates the dynamic safety domain geometric boundary function and the end effector stiffness coefficient, and outputs them to the robot's underlying controller to control the robot to avoid obstacles.

[0006] Preferably, in step S1, the process of generating the multimodal state tensor includes the following steps: Step S11, based on the visual sampling period, extract data segments of the motion feature temporal tensor within the sliding time window, calculate the minimum weighted Euclidean distance between the visual joint motion velocity vector and the inertial acceleration integral velocity vector, and solve for the optimal temporal offset; Step S12, based on the optimal temporal offset, use a cubic spline interpolation algorithm to resample the motion feature temporal tensor to a temporal grid aligned with the visual feature frame, and combine to generate a multimodal state tensor containing preset dimensional features.

[0007] Preferably, in step S2, the process of solving the cross-modal feature fusion tensor includes: extracting the visual feature query matrix, the inertial feature key matrix, and the inertial eigenvalue matrix from the multimodal cross-attention gating network; constructing a time bias mask matrix for penalizing the time span feature association; multiplying the visual feature query matrix and the transpose of the inertial feature key matrix, dividing by the square root of the key feature dimension, and adding it to the time bias mask matrix; and weighting the inertial eigenvalue matrix using a normalized exponential function to generate the cross-modal feature fusion tensor.

[0008] Preferably, in steps S4 and S5, the process of allocating the activation depth of the dynamic deep routing network includes the following steps: Step S41, during multiple consecutive sampling periods, when the gating activation coefficient is less than 0.5, the weight calculation path and forward propagation path of the higher-order cross-attention refinement layer are cut off, and only the basic feature extraction layer is kept running; Step S51, in the cut-off state, the base address of the high-order cross-attention refinement layer is locked, and the pre-allocated memory space is kept in a resident occupied state to avoid inference delay jumps caused by memory re-allocation.

[0009] Preferably, in step S6, the process of calculating the dynamic safety domain geometric boundary function includes: constructing an inflatable capsule with the robot joint link as the axis, linearly accumulating the product of the basic safety radius, the human predicted motion speed and the current actual inference delay of the neural network, and the product of the intention uncertainty index and the uncertainty expansion coefficient, to calculate the dynamic expansion radius of the inflatable capsule and characterize the dynamic safety domain geometric boundary function.

[0010] Preferably, in step S6, the process of calculating the end effector stiffness coefficient includes: calculating the real-time spatial distance between the predicted human joint and the robot end effector; calculating the difference between the real-time spatial distance and the dynamic expansion radius of the inflatable capsule; inputting the difference and a preset transition gradient coefficient into a continuous S-shaped smooth transition function, and steplessly adjusting the end effector stiffness coefficient between a preset maximum free space stiffness and a minimum contact safety stiffness.

[0011] Preferably, the step S4 of triggering the change of the topology of the neural network computation graph includes the following steps: Step S42, in the sliding queue, determine whether the gating activation coefficients within a consecutive preset number of time steps show a monotonically changing trend; Step S43, when the mean of the elements in the sliding queue exceeds a preset threshold point, and the gating activation coefficients jump from below 0.5 to above or equal to 0.5, connect the forward computation path of the higher-order cross-attention refinement layer and restore full-depth computation.

[0012] Preferably, in step S1, when the visual sensor experiences partial occlusion or signal frame loss, the asynchronous time warp alignment operator automatically switches to single-mode degradation protection mode: in step S13, the motion feature time-series tensor collected by the wearable inertial measurement unit is used to calculate the inertial dead reckoning based on the time-domain offset memorized in the previous cycle, and the multimodal state tensor is completed.

[0013] Preferably, the process of outputting the end effector stiffness coefficient to the robot's underlying controller in step S6 to control the robot to avoid obstacles includes the following steps: Step S61, writing the calculated end effector stiffness coefficient into the impedance control register of the robot's underlying controller in real time, and correcting the drive torque of the end effector based on the end effector stiffness coefficient; Step S62, when the human body penetrates the dynamic safety domain, the end effector stiffness coefficient decreases with the spatial distance contraction to the minimum contact safety stiffness, so that the robot end effector exhibits compliant physical avoidance, and feeding back the real-time trajectory deviation during the avoidance process to the preceding prediction window to correct the three-dimensional motion trajectory prediction tensor of the next cycle.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. In the optimization of multimodal neural networks for intent prediction in human-computer collaboration, by using the intent uncertainty index for the computational allocation of dynamic deep routing networks and combining it with the sliding queue of gating activation coefficients for continuous state judgment, we can avoid frequent switching of neural network computation graphs when the input features fluctuate near the decision threshold. At the same time, by maintaining the resident allocation of GPU memory space when high-order cross-attention refinement layers are suspended, we can reduce the additional inference latency and computational power loss caused by repeated GPU memory allocation and instruction pipeline clearing. This makes the switching between the basic feature extraction path and the full-depth computation path smoother, reduces inference latency fluctuations, and improves the stability of the neural network computational power scheduling process.

[0015] 2. By using an asynchronous time warp alignment operator to perform time-series alignment of data from the visual sensor and the wearable inertial measurement unit, and by using a time deviation mask matrix to suppress feature correlation with large time spans, the impact of differences in sampling frequency and transmission delay between different sensors on multimodal feature fusion can be reduced, improving the temporal correspondence of data from different modalities. Even under conditions such as visual occlusion or signal drift of wearable devices, relatively continuous multimodal feature input can be maintained, providing a more stable data foundation for the calculation of intent uncertainty index and subsequent dynamic depth routing, thereby improving the anti-interference capability of the intent prediction process.

[0016] 3. By using the intent uncertainty index and the current actual inference delay of the neural network together to adjust the dynamic safety domain and end effector stiffness, the robot's underlying control can reflect the changes in the neural network's predicted state and inference time in a timely manner. When the prediction result is relatively certain and the inference delay is small, high operating efficiency can be maintained. When the prediction uncertainty increases or the inference delay increases, the safety domain can be expanded accordingly and the end effector stiffness can be smoothly reduced. This allows the changes in neural network inference delay to be reflected in the robot's safety control in a timely manner, reducing the safety buffer response lag caused by changes in computing power scheduling and improving the timeliness and compliance of avoidance in the human-machine collaboration process. Attached Figure Description

[0017] Figure 1 This is a flowchart of the multimodal intent prediction and dynamic routing control of the present invention; Figure 2 This is a diagram of the architecture of the perception fusion and robot compliance control system of the present invention.

[0018] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] A multimodal neural network optimization method for intention prediction in human-computer collaboration includes the following steps: Step S1: Obtain the multimodal state tensor; the multimodal state tensor is generated by resampling the three-dimensional coordinate temporal tensor of human skeletal joints collected by the visual sensor and the motion feature temporal tensor collected by the wearable inertial measurement unit using the asynchronous time warp alignment operator. Step S2: Input the multimodal state tensor into the neural network computation graph containing the multimodal cross-attention gating network and the dynamic deep routing network, solve the cross-modal feature fusion tensor, and output the fused feature vector; Step S3: Calculate the intent uncertainty index based on the information entropy of the fused feature vector and the trace of the covariance matrix of the preceding trajectory, and solve the gated activation coefficients to adjust the activation depth of the dynamic deep routing network. Step S4: Maintain a sliding queue of gating activation coefficients. When the mean of the elements in the sliding queue exceeds a preset threshold, trigger a change in the topology of the neural network computation graph. Step S5: When the suspended state is triggered, the activation depth is limited to the basic feature extraction layer, the forward path of the higher-order cross-attention refinement layer is blocked, and its memory space is kept permanently allocated; when the suspended state is triggered, full-depth calculation is resumed. Step S6: The neural network computes the probability distribution vector of the operation intention and the three-dimensional motion trajectory prediction tensor, calculates the dynamic safety domain geometric boundary function and the end effector stiffness coefficient, and outputs them to the robot's underlying controller to control the robot to avoid obstacles.

[0021] Preferably, in step S1, the process of generating the multimodal state tensor includes the following steps: Step S11, based on the visual sampling period, extract data segments of the motion feature temporal tensor within the sliding time window, calculate the minimum weighted Euclidean distance between the visual joint motion velocity vector and the inertial acceleration integral velocity vector, and solve for the optimal temporal offset; Step S12, based on the optimal temporal offset, use a cubic spline interpolation algorithm to resample the motion feature temporal tensor to a temporal grid aligned with the visual feature frame, and combine to generate a multimodal state tensor containing preset dimensional features.

[0022] Preferably, in step S2, the process of solving the cross-modal feature fusion tensor includes: extracting the visual feature query matrix, the inertial feature key matrix, and the inertial eigenvalue matrix from the multimodal cross-attention gating network; constructing a time bias mask matrix for penalizing the time span feature association; multiplying the visual feature query matrix and the transpose of the inertial feature key matrix, dividing by the square root of the key feature dimension, and adding it to the time bias mask matrix; and weighting the inertial eigenvalue matrix using a normalized exponential function to generate the cross-modal feature fusion tensor.

[0023] Preferably, in steps S4 and S5, the process of allocating the activation depth of the dynamic deep routing network includes the following steps: Step S41, during multiple consecutive sampling periods, when the gating activation coefficient is less than 0.5, the weight calculation path and forward propagation path of the higher-order cross-attention refinement layer are cut off, and only the basic feature extraction layer is kept running; Step S51, in the cut-off state, the base address of the high-order cross-attention refinement layer is locked, and the pre-allocated memory space is kept in a resident occupied state to avoid inference delay jumps caused by memory re-allocation.

[0024] Preferably, in step S6, the process of calculating the dynamic safety domain geometric boundary function includes: constructing an inflatable capsule with the robot joint link as the axis, linearly accumulating the product of the basic safety radius, the human predicted motion speed and the current actual inference delay of the neural network, and the product of the intention uncertainty index and the uncertainty expansion coefficient, to calculate the dynamic expansion radius of the inflatable capsule and characterize the dynamic safety domain geometric boundary function.

[0025] Preferably, in step S6, the process of calculating the end effector stiffness coefficient includes: calculating the real-time spatial distance between the predicted human joint and the robot end effector; calculating the difference between the real-time spatial distance and the dynamic expansion radius of the inflatable capsule; inputting the difference and a preset transition gradient coefficient into a continuous S-shaped smooth transition function, and steplessly adjusting the end effector stiffness coefficient between a preset maximum free space stiffness and a minimum contact safety stiffness.

[0026] Preferably, the step S4 of triggering the change of the topology of the neural network computation graph includes the following steps: Step S42, in the sliding queue, determine whether the gating activation coefficients within a consecutive preset number of time steps show a monotonically changing trend; Step S43, when the mean of the elements in the sliding queue exceeds a preset threshold point, and the gating activation coefficients jump from below 0.5 to above or equal to 0.5, connect the forward computation path of the higher-order cross-attention refinement layer and restore full-depth computation.

[0027] Preferably, in step S1, when the visual sensor experiences partial occlusion or signal frame loss, the asynchronous time warp alignment operator automatically switches to single-mode degradation protection mode: in step S13, the motion feature time-series tensor collected by the wearable inertial measurement unit is used to calculate the inertial dead reckoning based on the time-domain offset memorized in the previous cycle, and the multimodal state tensor is completed.

[0028] Preferably, the process of outputting the end effector stiffness coefficient to the robot's underlying controller in step S6 to control the robot to avoid obstacles includes the following steps: Step S61, writing the calculated end effector stiffness coefficient into the impedance control register of the robot's underlying controller in real time, and correcting the drive torque of the end effector based on the end effector stiffness coefficient; Step S62, when the human body penetrates the dynamic safety domain, the end effector stiffness coefficient decreases with the spatial distance contraction to the minimum contact safety stiffness, so that the robot end effector exhibits compliant physical avoidance, and feeding back the real-time trajectory deviation during the avoidance process to the preceding prediction window to correct the three-dimensional motion trajectory prediction tensor of the next cycle.

[0029] Example 1: Data processing is performed in an industrial collaborative robot human-robot shared assembly workstation. An RGB-D depth camera with a sampling frequency of 30Hz acquires the 3D coordinate temporal tensor of 21 joints of the human skeleton. A six-axis wearable inertial measurement unit with a sampling frequency of 200Hz acquires the 3D acceleration and 3D angular velocity temporal tensors. Within a sliding time window... Within a visual time span of 0.5 seconds, due to the heterogeneity in sampling frequency and transmission deviation between the visual sampling period and the inertial sampling period, the system utilizes an asynchronous time warp alignment operator to adjust the visual sampling time. As the reference point, where The time range extracted from the inertial data buffer queue is... to The optimal time-domain offset is obtained by calculating the minimum weighted Euclidean distance between the visual joint motion velocity vector and the inertial acceleration integral velocity vector from the data fragment. Its expression is as follows: ,in, This is the optimal time-domain offset. The velocity vector of the visual key point. The integral velocity vector is the inertial acceleration. For the first Frame visual sampling time, The time-domain offset variable is used; subsequently, a cubic spline interpolation algorithm is employed based on... The inertial data matrix is ​​resampled and mapped to a temporal grid that completely overlaps with 15 frames of visual data, generating a dimension of Multimodal state tensor ,in The number of frames in the sliding time window. The total feature dimension is composed of 63-dimensional features including the 3D coordinates of 21 key points and 6-dimensional features of aligned three-axis acceleration and three-axis angular velocity. When solving for the above minimum value, the calculation of weighted Euclidean distance introduces a modal signal-to-noise ratio weighting matrix. Specifically, the weight coefficient of the visual velocity vector is set to 0.60, and the weight coefficient of the inertial integral velocity vector is set to 0.40. This is used to balance the measurement confidence of the two heterogeneous sensors in different dynamic response ranges. At the same time, in order to prevent low-frequency zero-bias accumulation and numerical divergence from the second integral of inertial acceleration, the system introduces a zero-velocity correction and high-pass filtering mechanism in the integral calculation loop: in the static and smooth transition range of human movement, the zero-velocity state is extracted by monitoring the variance of angular velocity and the zero-point drift term of the accelerometer is deducted in real time; and a high-pass filter with a cutoff frequency of 0.05Hz is set on the integral path to eliminate DC integral accumulation error and ensure that the output velocity vector does not diverge.

[0030] Multimodal state tensor A multimodal cross-attention gating network, which shuns input to the neural network computation graph, generates a visual feature query matrix from visual features. Inertial feature key matrix generated from inertial features With inertial eigenvalue matrix The time deviation mask matrix associated with the penalty time span feature. Solving the cross-modal feature fusion tensor The calculation formula is as follows: ,in, For cross-modal feature fusion tensors, For visual feature query matrix, The inertial eigenkey matrix, The inertial eigenvalue matrix, For key feature dimension, The time-bias mask matrix is ​​used to calculate the mean of the cross-modal feature fusion tensor across 15 time steps, resulting in a 64-dimensional fusion feature vector. This vector preserves the transient features of the inertial data during high-speed hand movement and the spatial structure features of the visual data during slow torso movement. The mask element located in the i-th row and j-th column of the time-bias mask matrix is ​​determined by the absolute time difference between the visual sampling time and the inertial sampling time. The system calculates the absolute value of the index difference between the two times and uses it as the negative power of the exponent, multiplied by a decay factor of 0.10. As the time span of the two modal data increases, this mask element decays rapidly to a large negative value with a negative exponent, causing the corresponding weight of the normalized exponential function in the weighted calculation to approach 0. This imposes a forced penalty on heterogeneous temporal correlations with large spans, eliminating false cross-attention between asynchronous features.

[0031] The fused feature vector is input into the prediction decoder, and the neural network computation graph outputs the probability distribution vector of the operation intention at future time steps. It covers reaching out to retrieve items, handing over tools, remaining still, and yielding to others. The system calculates the user's intention and outputs a 3D motion trajectory prediction tensor for the hand joints within the next second. The entropy value formed by the probability of each intention in the intention probability distribution vector is used as the information entropy of the fused feature vector, combined with the covariance matrix of the preceding trajectory. traces Calculation of Intent Uncertainty Index The calculation formula is as follows: ,in, As an indicator of intent uncertainty, The total number of intent categories, For the first Class intent in The predicted probability at time t. The dimensional balance coefficient, The trace operator of a matrix. The preceding trajectory covariance matrix is ​​used to characterize the dispersion and trajectory fluctuation of the neural network prediction output in real time. In the above calculation process, the information entropy is a dimensionless pure numerical value. The trace of the preceding trajectory covariance matrix is ​​pre-divided by the nominal variance of the reference trajectory (0.01m) before being substituted into the calculation. 2Dimensionless normalization is performed to ensure that the normalized trajectory variance trace and information entropy are both on the same dimensionless order of magnitude. A dimensional balance coefficient of 0.15 is used to adjust the weighting ratio of the two in the dimensionless domain, thereby ensuring that the obtained intention uncertainty index is a self-consistent dimensionless scalar. Furthermore, when calculating the dynamic expansion radius of the inflating capsule, an uncertainty expansion coefficient of 0.20m is used as a spatial scaling conversion factor to linearly map the dimensionless intention uncertainty index into a meter-level spatial length increment, making it consistent with the physical dimensions of the basic safety radius of 0.15m and the meter-level displacement increment derived from the product of the human body's predicted motion speed and the current actual inference delay of the neural network, thus achieving a unified linear superposition on the spatial length dimension.

[0032] Dynamic deep routing networks based on intent uncertainty indicators Solving for gating activation coefficients To allocate the activation depth of the dynamic deep routing network, the formula for calculating the gating activation coefficient is as follows: ,in, The gating activation coefficient is... As an indicator of intent uncertainty, To preset the judgment threshold, The gain slope parameter is used to calculate the intent uncertainty index when the operator's action pattern is fixed and within a smooth reaching trajectory. If the value is below the preset threshold of 0.40, the gating activation coefficient is calculated. If the value is less than 0.5, the dynamic deep routing network cuts off the weight calculation path and forward propagation path of the last two high-order cross-attention refinement layers in the computation graph, limiting the activation depth of the computation graph to the first two basic feature extraction layers, thus reducing the single inference time of the neural network from 35ms to 11ms. When the high-order cross-attention refinement layer is in a suspended state, its memory space is maintained in a resident allocation to prevent repeated memory release and allocation from causing inference delay jumps. In the aforementioned parameter settings, the judgment threshold of 0.40 is the intersection point determined by fitting the discreteness of the assembly operation's smooth motion and the intentional sudden change critical state. If this threshold is less than 0.10, the system will still frequently trigger full-depth calculation under stable operating conditions, losing the effect of reducing computing power overhead. If this threshold is greater than 0.8... If the value is 0, the gating activation coefficient cannot be increased in time when limb occlusion or sudden action occurs, causing the higher-order cross-attention refinement layer to be suspended for a long time, resulting in a sharp increase in trajectory prediction error. The gain slope of 20.0 ensures that the activation coefficient produces a steep S-shaped transition on both sides of the threshold. If the gain slope is lower than 5.0, the activation coefficient is too sluggish in response to changes in uncertainty, causing topology switching lag. If the gain slope is higher than 50.0, it will cause violent oscillations in the value of the activation coefficient at the threshold boundary. The dimensional balance coefficient of 0.15 and the uncertainty expansion coefficient of 0.20m were both optimized on the test platform based on 1000 sets of offline action samples with the goal of minimizing the mean square value of prediction error, ensuring closed-loop self-consistency between computational overhead and physical safety boundary.

[0033] When an operator suddenly changes their operational intention during the assembly process, or their hand is obstructed by a workpiece, the probability distribution of that intention tends to become discrete, and the calculated uncertainty index of that intention rises to [a certain level]. Real-time system maintenance includes continuous A sliding queue of gated activation coefficients at each time step The mean of the elements in the sliding queue monotonically exceeds the preset threshold of 0.5, and the gating activation coefficient of the current step jumps to [value missing]. When the value is greater than or equal to 0.5, a change in the topology of the neural network computation graph is triggered. The dynamic deep routing network connects the forward computation path of the higher-order cross-attention refinement layer within 10ms, restores full-depth computation, and switches the single inference time of the neural network to 35ms, maintaining the trajectory prediction accuracy. Since the memory space of the higher-order cross-attention refinement layer is kept resident, there is no need to re-allocate this part of the memory space when the computation graph topology changes, thereby avoiding the jump in inference delay caused by repeated memory release and allocation.

[0034] The operation intent probability distribution vector, 3D motion trajectory prediction tensor, and intent uncertainty index output by the neural network computation graph Real-time mapping to the robot's underlying control constraints; the dynamic safety domain around each joint link of the robot is represented by an inflatable capsule, and the dynamic expansion radius of the inflatable capsule... The calculation formula is as follows: ,in, The dynamic expansion radius of the expanded capsule is expressed in meters (m). The basic safety radius is 0.15m. The predicted speed of human movement is 1.20 m / s; The actual inference delay of the neural network is expressed in seconds (s). The uncertainty expansion coefficient has a value of 0.20m. Under low uncertainty conditions, and Calculate the dynamic expansion radius When the uncertainty index of intent rises to Furthermore, the neural network switching back to the full depth path delays the actual inference process of the current neural network. At that time, dynamic expansion radius The inference delay variation and prediction uncertainty of the neural network computation graph are compensated in real time to the physical safety buffer.

[0035] The system predicts the real-time spatial distance between the joints and the robot's end effector based on the human body. Solving for end stiffness coefficients The calculation formula is as follows: ,in, This is the end stiffness coefficient, in N / m; The minimum stiffness for contact safety is 100 N / m; It has the highest stiffness in free space, with a value of 2000 N / m; Preset transition gradient coefficients; Predict the real-time spatial distance between the human body's joints and the robot's end effector. When the dynamic expansion radius is greater than 0.1992m, the end stiffness coefficient is... N / m, driving the robot to maintain high rigidity and high trajectory tracking accuracy; when a human approaches and penetrates the dynamic safety domain, the real-time spatial distance... When the dynamic expansion radius is less than or equal to 0.3220m, the end stiffness coefficient smoothly decreases to The N / m value enables the robot end effector to exhibit compliant physical avoidance. After calculating the end effector stiffness coefficient, it is written to the impedance control register of the robot's underlying controller in real time. The robot's underlying controller reads the current end effector stiffness coefficient within the impedance control cycle, updates the stiffness control quantity corresponding to the end effector position deviation, and corrects the drive torque of the end effector accordingly. During the avoidance process, the real-time trajectory deviation fed back to the preceding prediction window is the spatial vector difference between the actual physical position of the robot end effector and the original planned trajectory when it is squeezed by the human body or retreated by external force. This deviation vector represents the amount of physical encroachment exerted by the human body's actions on the shared workspace. The system converts this physical encroachment amount into the actual motion offset correction value of the human hand action through the inverse transformation of the spatial relative coordinate system between the robot and the human body, and inputs it as an exogenous observation variable into the preceding prediction window of the prediction decoder. In the next cycle, the three-dimensional motion trajectory prediction tensor is corrected to eliminate the trajectory bias caused by sudden changes in human body actions and the extrapolation lag of the prediction model, and realize the real feedback constraint of the physical retreat action on the intention prediction model.

[0036] Example 2: Verification experiments of the multimodal neural network optimization method were conducted on an industrial collaborative robot human-machine shared assembly workstation test platform. The test platform used an industrial control computer as the computing core, equipped with an AI inference accelerator card with a computing power of 275 TOPS, and the real-time operating system cycle was set to 10ms. The sensor side was equipped with an RGB-D depth camera with a sampling frequency of 30Hz and a six-axis wearable inertial measurement unit with a sampling frequency of 200Hz, worn on the operator's wrist. During the test, Gaussian white noise with a signal-to-noise ratio of 20dB and 50Hz power frequency interference harmonics were actively superimposed to simulate electromagnetic interference and partial visual occlusion. The experiment included comparative tests of the present invention's sample group, control group 1, control group 2, control group 3, and groups exceeding the lower and upper limits of the range. The present invention's sample group fully ran the asynchronous time-warped alignment operator, multimodal cross-attention gating network, dynamic depth routing network, persistent memory allocation, and dynamic safety domain impedance control mapping loop. The judgment threshold was set to... The gain slope is set to Control group 1 removed the dynamic deep routing network and ran in a fixed full-depth neural network topology mode; Control group 2 retained the dynamic deep routing network but removed the sliding queue hysteresis judgment and persistent memory allocation, adopted single-frame threshold hard cutoff, and released the memory of higher-order cross-attention refinement layers each time it was suspended; Control group 3 cut off the wearable inertial measurement unit input and used only single-modal visual keypoint data for intent prediction; The out-of-range lower limit group lowered the judgment threshold to The threshold for judging groups exceeding the upper limit will be raised to [a higher level]. .

[0037] Each group used the same data input sequence and ran continuously for 1000 sampling cycles. The input sequence covered smooth reaching for an object, violent avoidance, partial limb occlusion, and sudden change in intent. The single inference latency of the sample group under smooth action was 11ms. During continuous testing, the dynamic depth routing network adjusted the activation depth according to the intent uncertainty index and limited repeated switching near the threshold by a sliding queue, resulting in an average inference latency of 14.2ms and a latency variance of 1.8ms. 2 The topology switching jitter occurred 3 times. Under the same input, its cross-entropy loss was 0.21, the root mean square error of trajectory prediction was 0.023m, the safety domain response lag was 11.2ms, and the end-contact impact force was 32.5N. Control group 1 always used a fixed full-depth calculation path, so no computation graph topology switching occurred, and the number of topology switching jitters was 0. Its cross-entropy loss was 0.20, and the root mean square error of trajectory prediction was 0.022m, but the average inference delay reached 35.1ms, and the delay variance was 0.4ms. 2 After the reasoning process directly enters the calculation process of the dynamic safety domain, the response lag of the safety domain is 35.1ms. During the high-speed movement of the operator, the update of the dynamic expansion radius is delayed accordingly, the end stiffness coefficient fails to decrease in time as the human body approaches, and the end contact impact force is 88.6N.

[0038] Control group 2 retains the dynamic deep routing network but uses single-frame threshold switching and releases corresponding GPU memory when high-order cross-attention refinement layers are suspended. When the intent uncertainty index fluctuates around 0.40, the computation graph experiences 142 topology switching jitters during continuous testing. Memory reallocation and instruction pipeline clearing occur repeatedly with switching, resulting in a latency variance of 28.6ms. 2 The average inference latency was 22.8 ms; under the same input, the cross-entropy loss was 0.38, the root mean square error of trajectory prediction was 0.048 m, the safe domain response lag was 41.5 ms, and the end-effector impact force was 115.2 N. Control group 3 only retained visual keypoint data input, lacking the temporal tensor of motion features collected by the wearable inertial measurement unit to participate in multimodal feature fusion when there was partial limb occlusion; its average inference latency was 15.6 ms, and the latency variance was 2.1 ms. 2 The topology switching jitter was 5 times; the cross-entropy loss under occlusion conditions reached 0.89, the root mean square error of trajectory prediction was 0.115m, the safety domain response lag was 12.8ms, and the end contact impact force was 185.4N. The above data correspond to two different effects: the main changes in control group 2 are concentrated in computation graph switching and inference delay, while the main changes in control group 3 are concentrated in intent prediction and trajectory prediction results.

[0039] In the group exceeding the lower limit, the judgment threshold is set to Subsequently, under stable and highly deterministic operating conditions, the intention uncertainty index... The activation coefficient is still higher than the decision threshold, thus the gating activation coefficient is obtained. The dynamic deep routing network maintains a full-depth computation path, and continuous testing yielded an average inference latency of 33.8ms and a latency variance of 2.3ms. 2 The number of topology switching jitters was 2; its cross-entropy loss was 0.21, the root mean square error of trajectory prediction was 0.023m, the safety domain response lag was 33.8ms, and the end contact impact force was 85.1N. The prediction error of this group remained near the level of the sample group in this invention, but the lower decision threshold caused the higher-order cross-attention refinement layer to remain on during smooth operation, which increased the computation time accordingly.

[0040] In the group exceeding the upper limit, set the judgment threshold to Subsequently, when the operator makes a sharp change of direction or experiences obstruction, the intent uncertainty indicator... Still below the judgment threshold, the gating activation coefficient is With higher-order cross-attention refinement layers suspended and only the basic feature extraction layers running, the average inference latency for this group was 11.3ms, with a latency variance of 0.9ms. 2 The topology switching jitter occurred once, and the safety domain response lag was 11.3ms. As the full-depth calculation failed to recover when the action changed, the cross-entropy loss reached 0.76, the root mean square error of trajectory prediction reached 0.098m, and the end-contact impact force was 162.0N. Compared with the two out-of-range groups, when the decision threshold was 0.40, the dynamic depth routing network could switch the activation depth according to the gating activation coefficient between stable actions and action abrupt changes, so that the change of the calculation path was simultaneously constrained by the current prediction state. In the prototype of this invention, the asynchronous time warp alignment operator first performs time-series alignment on the data collected by the visual sensor and the wearable inertial measurement unit, and the multimodal cross-attention gating network completes feature fusion on this basis; the dynamic deep routing network adjusts the activation depth according to the intent uncertainty index, and uses a sliding queue to suppress repeated switching of the computation graph near the decision threshold, maintaining a persistent allocation of GPU memory space during the suspension of the high-order cross-attention refinement layer, thus reducing the actual inference latency of the neural network. Indicators of uncertainty of intent Continue to participate in dynamic expansion radius and end stiffness coefficient The adjustment allows the corresponding inference time and prediction state to continue to be transmitted to the robot's underlying control process after the neural network calculation path changes.

[0041] Example 3: During high-frequency continuous assembly operations at an industrial collaborative robot-human shared assembly workstation, the operator's hand movements frequently switch between steady-state extension and sudden retraction. The industrial control computer hardware platform is equipped with an RGB-D depth camera with a sampling frequency of 30Hz and a six-axis wearable inertial measurement unit with a sampling frequency of 200Hz, worn on the operator's wrist. Within a 0.5s sliding time window corresponding to a frame and a corresponding visual time span, the asynchronous time warp alignment operator resamples the two types of sensor data. During the initialization phase, the system imports an offline dataset containing 1000 sets of unobstructed standard motion sequences, setting the weight coefficient of the visual keypoint motion velocity vector to 0.60 and the weight coefficient of the inertial acceleration integral velocity vector to 0.40. The optimal temporal offset is calculated in real-time for each iteration. If the deviation of the minimum weighted Euclidean distance measured in three consecutive sampling periods exceeds 5ms, cubic spline interpolation resampling is initiated to map the high-frequency inertial data onto a temporal grid aligned with 15 frames of visual data, generating a temporally aligned multimodal state tensor with dimensions of 15×69. ,in, The number of frames in the sliding time window. The total feature dimension includes 63-dimensional features formed by the three-dimensional coordinates of 21 key points, and 6-dimensional features formed by the aligned three-axis acceleration and three-axis angular velocity.

[0042] Multimodal state tensor After inputting the neural network computation graph, the multimodal cross-attention gating network extracts features and outputs a cross-modal feature fusion tensor. The tensor is mean-valued dimension-wise over 15 time steps to obtain a 64-dimensional fused feature vector. The prediction decoder takes the fused feature vector as input and outputs a probability distribution vector of the operation intention at future time steps. And the three-dimensional motion trajectory prediction tensor, in which The operational intentions are categorized as reaching for an item, handing over a tool, remaining still, and withdrawing / avoiding. The entropy value formed by the probabilities of each intention in the probability distribution vector is used as the information entropy representation of the fused feature vector, and the intention uncertainty index is calculated by combining the trace of the covariance matrix of the preceding trajectory. ,against Within the preset judgment threshold The repeated changes in the topology of the neural network computation graph caused by nearby fluctuations necessitate a sliding queue for maintaining gating activation coefficients in the dynamic deep routing network. The system calculates the results of the preceding 100 sampling periods during the power-on self-test phase. The variance of the change; when the variance is less than 0.02, the sliding window length is determined to be... When the variance is greater than or equal to 0.02, the sliding window length is determined to be... ,exist At the same time, the gating activation coefficients of the five consecutive time steps in the queue are calculated in real time. The moving average is used, and the difference between the current step gating activation coefficient and the threshold point 0.5 is used as the deviation of the current step scale. When the moving average monotonically exceeds the preset threshold point 0.5, and the deviation of the current step scale is lower than -0.05 or higher than 0.05, the topology of the neural network computation graph is changed. Otherwise, the activation topology of the previous time step is maintained. For sudden avoidance situations, the dynamic deep routing network also retains the emergency preemption bypass mechanism. When the gating activation coefficient of the current single frame rises to above 0.90, and the difference between the activation coefficient of the current frame and the activation coefficient of the previous frame exceeds the mutation threshold of 0.50, the emergency preemption flag is directly activated, and the sliding queue mean monotonicity condition check is skipped in the current cycle and the forward computation path of the higher-order cross-attention refinement layer is connected. In the slowly changing critical situation where the emergency mutation condition is not met, the topology of the neural network computation graph continues to be controlled according to the sliding queue mean monotonicity condition.

[0043] When the gating activation coefficient satisfies At this time, the dynamic deep routing network cuts off the weight calculation path and forward propagation path of the higher-order cross-attention refinement layer. Simultaneously, the industrial control computer's underlying memory management module locks the pre-allocated 128MB base address of the memory for that layer, ensuring the memory space remains permanently allocated. During the forward propagation path cutoff, the pre-warm-up read instructions for the higher-order weight tensor continue to be maintained. When the gating activation coefficient jumps to... Furthermore, when the mean of the sliding queue monotonically exceeds 0.5, the forward computation path of the higher-order attention refinement layer is directly connected using the locked memory base address, restoring the full-depth neural network computation. This controls the parasitic latency of memory reallocation caused by changes in the topology of the neural network computation graph to within 0.1ms, thus reducing the actual inference latency of the neural network. To maintain a smooth transition, in the execution engine of the neural network computation graph, the forward computation nodes of higher-order cross-attention refinement layers are encapsulated within conditional control flow switching operators. When the gating activation coefficient is below a threshold, the switching operator transmits the input feature tensor to the output through a bypass identity mapping channel, skipping the matrix multiplication and activation function operations of the higher-order cross-attention refinement layers. Simultaneously, the memory management module marks the 128MB physical page table of the memory occupied by this layer as a protected locked state, prohibiting the garbage collection mechanism from releasing the corresponding physical memory. After the gating activation coefficient reaches the activation condition, the switching operator switches branch instructions within a single computation cycle, calling the locked physical base address of the memory to pass the tensor, so that the forward computation of the higher-order cross-attention refinement layers can be restored in the original memory space.

[0044] When the visual sensor experiences partial occlusion or signal frame loss, the asynchronous time warp alignment operator switches to single-mode degradation protection mode, locking the optimal time-domain offset memorized in the previous cycle. Inertial dead reckoning is performed using triaxial acceleration and triaxial angular velocity data collected by a six-axis wearable inertial measurement unit, completing the three-dimensional keypoint coordinates and generating a multimodal state tensor. When the single-modal degradation protection mode lasts for more than 5 visual sampling cycles, i.e., 166.65ms, the dimensional balance coefficient will be changed from... Adjusted to Accordingly, increase the uncertainty index of intent. The dynamic expansion radius of the driving expansion capsule body Extended; Updated dynamic expansion radius Compared to the current actual inference latency of neural networks Continue to participate in the end stiffness coefficient The system adjusts the robot's end effector stiffness to reduce when a human body approaches and penetrates the dynamic safety domain. In single-modal degradation protection mode, the system utilizes triaxial acceleration and triaxial angular velocity to first calculate the relative displacement and attitude angle change of the wrist joint at the current moment through forward kinematics recursion. Then, combined with the human skeleton topology tree structure saved offline in the previous cycle and the pre-calibrated fixed length constraints of each bone chain segment of the upper limb, the system uses the spatial displacement of the wrist joint as the end effector driving source and recursively solves the relative angles of the upper limb and trunk joints along the reverse kinematic chain from the wrist, elbow, and shoulder to the trunk. Subsequently, it performs forward attitude calculation according to the human skeleton chain, sequentially updating and completing the three-dimensional spatial coordinates of all 21 bone joints, expanding and mapping the 6-dimensional inertial vector into joint features containing 63-dimensional spatial position coordinates, and completing the dimensional completion of the multimodal state tensor.

[0045] Example 4: This example combines Figures 1 to 2 This paper explains the multimodal neural network optimization method for intention prediction in human-computer collaboration, such as... Figure 1As shown, step S1 is executed sequentially to obtain the multimodal state tensor and resample the temporal tensor of the three-dimensional coordinates of the human skeletal joints collected by the visual sensor and the temporal tensor of the motion features collected by the wearable inertial measurement unit using the asynchronous time warp alignment operator. Step S2 is executed to input the multimodal state tensor into a neural network computation graph containing a multimodal cross-attention gating network and a dynamic deep routing network to solve the cross-modal feature fusion tensor and output the fused feature vector. Step S3 is executed to calculate the intent uncertainty index based on the information entropy of the fused feature vector and the trace of the prior trajectory covariance matrix and solve the gating activation coefficient to adjust the activation of the dynamic deep routing network. The process involves several steps: Step S4 maintains a sliding queue of gated activation coefficients and triggers a change in the topology of the neural network computation graph when the mean of the elements in the sliding queue exceeds a preset threshold. Step S5 restricts the activation depth to the basic feature extraction layer, blocks the forward path of the higher-order cross-attention refinement layer, and maintains its permanent memory allocation when the suspension state is triggered. Full-depth computation is resumed when the suspension state is lifted. Step S6 outputs the operation intent probability distribution vector and the three-dimensional motion trajectory prediction tensor from the neural network computation graph and calculates the dynamic safety domain geometric boundary function and the end effector stiffness coefficient, which are then output to the robot's underlying controller to control the robot to avoid obstacles.

[0046] like Figure 2 As shown, the visual sensor acquires the three-dimensional coordinate temporal tensor of the human skeletal joints and inputs it into the asynchronous time warp alignment operator in the AI ​​inference acceleration card of the industrial control computer's real-time operating system. The wearable inertial measurement unit acquires the motion feature temporal tensor and inputs it into the asynchronous time warp alignment operator. The asynchronous time warp alignment operator outputs a multimodal state tensor to the neural network computation graph. The neural network computation graph contains a multimodal cross-attention gating network and a dynamic deep routing network and interacts with the underlying memory management module to maintain the permanent allocation of memory space. The neural network computation graph receives the real-time trajectory deviation and outputs the end effector stiffness coefficient and dynamic expansion radius to the impedance control register of the robot's underlying controller. The robot's underlying controller outputs the driving torque to the joint actuators and end effectors of the industrial collaborative robot according to the end effector stiffness coefficient.

[0047] Example 5: Before the start of the human-machine collaborative assembly operation, the system performs offline spatial pose and zero-bias calibration on the six-axis wearable inertial measurement unit worn by the operator to correct spatial installation pose deviation and sensor zero-point drift. The operator first maintains a static posture for 5 seconds, and then completes a 10-second preset trajectory arm extension movement. The industrial control computer collects the 200Hz raw acceleration and angular velocity sequence output by the six-axis wearable inertial measurement unit, calculates the static interval mean, and writes it into the hardware register as the acceleration zero-bias compensation vector and angular velocity drift compensation vector, respectively. At the same time, the system extracts the 30Hz wrist joint three-dimensional coordinates obtained by the RGB-D depth camera, solves the rotation matrix and translation vector from the inertial coordinate system to the visual coordinate system based on singular value decomposition, controls the coordinate transformation residual within 2mm, and establishes a physical space coordinate mapping benchmark.

[0048] During real-time human-machine collaboration, to address the fluctuations in ambient lighting at the assembly workstation and the noise envelope drift caused by long-term sensor operation, the system implements an adaptive calibration and parameter reconstruction procedure for background noise. It also maintains a sliding historical window with a time span of 300 seconds and periodically calculates the visual sampling missing rate and the inertial acceleration variance envelope. When the visual sampling missing rate is below 2% and the inertial acceleration variance is below 0.01 m / s², the system achieves the desired result. 2 When the operating condition is determined to be in a low-interference state, the preset judgment threshold is maintained. Dimensional balance coefficient When a sudden change in ambient lighting causes the visual sampling missing rate to rise to 5%, the intent uncertainty index is calculated over the preceding 50 sampling periods. The mean shift is adjusted by linearly proportionally to the dimensional balance coefficient. Simultaneously, the basic safety radius of the expanded capsule body was adjusted to m, and drive the end variable impedance controller to correct the end stiffness coefficient. .

[0049] Example 6: During the self-test phase of the human-machine shared assembly workstation system startup, the current industrial control computer calls the network topology and computing power configuration procedures to extract the memory addressing address and multimodal sensor interface status of the 275TOPS inference accelerator card. In the initial buffer of a frame corresponding to a visual time span of 0.5 seconds, the system offline collected 500 sets of reaching and tool delivery action sequences from 10 operators of different body types and operating habits on the assembly line, taking into account the extreme differences in operator motion acceleration, and predicted the motion speed based on the human body. An adaptive threshold lookup table is constructed based on the extreme values; when the predicted human hand movement speed is measured in real time and satisfies the condition... At m / s, the preset judgment threshold Take 0.35; when At m / s, according to the linear relationship Adjust it to 0.45 so that the corresponding judgment thresholds are used for high-speed motion and low-speed stable conditions respectively.

[0050] In the high-intensity assembly process of the assembly line, the multimodal state tensor Input a multimodal cross-attention gating network, where and The network outputs a cross-modal feature fusion tensor. After averaging the values ​​across 15 time steps, a 64-dimensional fused feature vector is obtained. Based on this, the prediction decoder outputs information including reaching for the item, submitting the tool, remaining still, and withdrawing to avoid collisions. Probability distribution vector of class operation intent And the 3D motion trajectory prediction tensor within the next 1 second, when the operator quickly switches workpieces at the assembly station, so that the covariance matrix of the preceding trajectory... traces Increased to 0.0080m 2 At that time, the indicator of intentional uncertainty It rose to 0.52, which is above the judgment threshold. The gating activation coefficient is obtained by dynamic deep routing network. and maintain continuous A sliding queue of gated activation coefficients at each time step When the mean of the elements in the sliding queue monotonically exceeds 0.5 and the deviation of the current step size is greater than 0.05, the forward computation path of the higher-order cross-attention refinement layer is activated. The weight tensor is directly loaded using the resident 128MB base address, changing the single inference time from 11ms for the basic feature extraction path to 35ms for the full-depth computation path. When the power frequency interference magnetic field causes the accelerometer output variance of the wearable inertial measurement unit to be greater than 0.05m / s², the forward computation path is activated. 2 At that time, the system shuts down the inertial feature key matrix. The gain path is reset, and the time offset mask matrix is ​​reset. The attenuation coefficient is used to limit the participation of disturbed inertial features in the cross-modal feature fusion tensor. The calculation.

[0051] The actual inference latency of the neural network at each cycle output by the neural network computation graph. Indicators of uncertainty of intent The robot's low-level control process is transmitted in real time, and the expanding capsule body adopts a basic safety radius. m, predicted human motion speed m / s and uncertainty expansion coefficient m calculates the dynamic expansion radius; under the full-depth calculation path. s and When the dynamic expansion radius is obtained, m, at this moment, is the real-time spatial distance between the human hand and the robot's end effector. The robot retracts to 0.15m and enters the dynamic safety domain, based on the highest stiffness in free space. N / m, minimum stiffness for contact safety N / m and preset transition gradient coefficient The end stiffness coefficient is obtained through a continuous S-shaped smooth relationship. The torque is reduced to approximately 266 N / m, thereby limiting the upper limit of the output torque of the robot joint actuators, so that the robot end effector remains compliant and yielding during physical contact and deviation from the spatial trajectory.

[0052] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal neural network optimization method for intention prediction in human-computer collaboration, characterized in that, Includes the following steps: Step S1: Obtain the multimodal state tensor; the multimodal state tensor is generated by resampling the three-dimensional coordinate temporal tensor of human skeletal joints collected by the visual sensor and the motion feature temporal tensor collected by the wearable inertial measurement unit using the asynchronous time warp alignment operator. Step S2: Input the multimodal state tensor into the neural network computation graph containing the multimodal cross-attention gating network and the dynamic deep routing network, solve the cross-modal feature fusion tensor, and output the fused feature vector; Step S3: Calculate the intent uncertainty index based on the information entropy of the fused feature vector and the trace of the covariance matrix of the preceding trajectory, and solve the gated activation coefficients to adjust the activation depth of the dynamic deep routing network. Step S4: Maintain a sliding queue of gating activation coefficients. When the mean of the elements in the sliding queue exceeds a preset threshold, trigger a change in the topology of the neural network computation graph. Step S5: When the suspended state is triggered, the activation depth is limited to the basic feature extraction layer, the forward path of the higher-order cross-attention refinement layer is blocked, and its memory space is kept permanently allocated; when the suspended state is triggered, full-depth calculation is resumed. Step S6: The neural network computes the probability distribution vector of the operation intention and the three-dimensional motion trajectory prediction tensor, calculates the dynamic safety domain geometric boundary function and the end effector stiffness coefficient, and outputs them to the robot's underlying controller to control the robot to avoid obstacles.

2. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In step S1, the process of generating the multimodal state tensor includes the following steps: Step S11, based on the visual sampling period, extract the data segment of the motion feature temporal tensor within the sliding time window, calculate the minimum weighted Euclidean distance between the motion velocity vector of the visual key point and the integral velocity vector of inertial acceleration, and solve for the optimal temporal offset; Step S12, based on the optimal temporal offset, use the cubic spline interpolation algorithm to resample the motion feature temporal tensor to a temporal grid aligned with the visual feature frame, and combine them to generate a multimodal state tensor containing preset dimensional features.

3. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In step S2, the process of solving the cross-modal feature fusion tensor includes: extracting the visual feature query matrix, inertial feature key matrix, and inertial eigenvalue matrix from the multimodal cross-attention gating network; constructing a time bias mask matrix for penalizing the time span feature association; multiplying the visual feature query matrix and the transpose of the inertial feature key matrix, dividing by the square root of the key feature dimension, and adding it to the time bias mask matrix; and then weighting the inertial eigenvalue matrix using a normalized exponential function to generate the cross-modal feature fusion tensor.

4. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In steps S4 and S5, the process of allocating the activation depth of the dynamic deep routing network includes the following steps: Step S41, within multiple consecutive sampling periods, when the gating activation coefficient is less than 0.5, the weight calculation path and forward propagation path of the higher-order cross-attention refinement layer are cut off, and only the basic feature extraction layer is kept running; Step S51, in the cut-off state, the base address of the high-order cross-attention refinement layer is locked, and the pre-allocated memory space is kept in a resident occupied state to avoid inference delay jumps caused by memory re-allocation.

5. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In step S6, the process of calculating the dynamic safety domain geometric boundary function includes: constructing an inflatable capsule with the robot joint link as the axis, linearly accumulating the product of the basic safety radius, the human predicted motion speed and the current actual inference delay of the neural network, and the product of the intention uncertainty index and the uncertainty expansion coefficient, to calculate the dynamic expansion radius of the inflatable capsule and characterize the dynamic safety domain geometric boundary function.

6. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In step S6, the process of calculating the end effector stiffness coefficient includes: calculating the real-time spatial distance between the predicted joint of the human body and the robot end effector; calculating the difference between the real-time spatial distance and the dynamic expansion radius of the inflatable capsule; inputting the difference and the preset transition gradient coefficient into a continuous S-shaped smooth transition function, and steplessly adjusting the end effector stiffness coefficient between the preset maximum stiffness in free space and the minimum stiffness in contact safety.

7. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, The steps to trigger the change of the topology of the neural network computation graph in step S4 include the following steps: Step S42, in the sliding queue, determine whether the gating activation coefficients within a consecutive preset number of time steps show a monotonically changing trend; Step S43, when the mean of the elements in the sliding queue exceeds the preset threshold point, and the gating activation coefficients jump from below 0.5 to above or equal to 0.5, connect the forward computation path of the higher-order cross-attention refinement layer and restore full-depth computation.

8. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, In step S1, when the visual sensor experiences partial occlusion or signal frame loss, the asynchronous time warp alignment operator automatically switches to single-mode degradation protection mode: In step S13, the motion feature time-series tensor collected by the wearable inertial measurement unit is used to calculate the inertial dead reckoning based on the time-domain offset memorized in the previous cycle, and the multimodal state tensor is completed.

9. The multimodal neural network optimization method for intention prediction in human-computer collaboration according to claim 1, characterized in that, The process of outputting the end effector stiffness coefficient to the robot's underlying controller in step S6 to control the robot's avoidance includes the following steps: Step S61, the calculated end effector stiffness coefficient is written into the impedance control register of the robot's underlying controller in real time, and the driving torque of the end effector is corrected according to the end effector stiffness coefficient; Step S62, when the human body penetrates the dynamic safety domain, the end effector stiffness coefficient decreases with the spatial distance contraction to the minimum contact safety stiffness, so that the robot end effector exhibits compliant physical avoidance, and the real-time trajectory deviation during the avoidance process is fed back to the preceding prediction window to correct the three-dimensional motion trajectory prediction tensor of the next cycle.

Images