Joint analysis method and device based on multi-modal perception and working condition adaptive fusion, robot joint module and robot

Abstract

The application relates to the technical field of robot control, in particular to a joint analysis method based on multi-modal perception and working condition adaptive fusion, which is executed by a processor integrated in a joint module driving control board and comprises the following steps: collecting multi-modal sensing data of the joint module in real time during operation, wherein the multi-modal sensing data at least comprises motor electrical data, mechanical vibration data and operation acoustic data; constructing a time sequence feature matrix (H) and a working condition vector (c) reflecting a current operation state based on the multi-modal sensing data; inputting the time sequence feature matrix (H) and the working condition vector (c) into a condition-gated attention fusion model; and generating and outputting a real-time torque estimation value (tau_est) of a joint output shaft and a mechanical health index based on fused high-level features. The application has the effect of dynamically adjusting the weight of different sensor modalities in decision-making according to real-time working conditions, thereby realizing high-precision, adaptive torque estimation and health prediction.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a joint analysis method, device, robot joint module, and robot based on multimodal perception and adaptive fusion of working conditions. Background Technology

[0002] In torque estimation and state monitoring of robot joints, traditional methods are mainly divided into two categories: physical model-based methods and data-driven methods. Physical model-based methods (such as torque estimation based on motor current) rely on accurate system parameters, but these parameters drift during actual operation, leading to a decrease in accuracy. Data-driven methods (such as those using neural networks) can learn complex mappings, but they are usually static and cannot adapt to changes in sensor reliability under different operating conditions.

[0003] In existing technologies, multi-sensor fusion often employs fixed-weighted averaging or simple fully connected neural networks, lacking explicit modeling of the crucial understanding that "different sensors have different reliability under different operating conditions." For example, at high speeds, vibration signals are more sensitive to bearing conditions; while at low speeds and high loads, current signals are more directly related to gear loads, and acoustic signals are more sensitive to abnormal friction. If the fusion strategy cannot be dynamically adjusted, the model's performance will become unstable under varying operating conditions.

[0004] Therefore, a fusion algorithm is needed that can automatically and reasonably allocate the modal weights of different sensors according to real-time operating conditions in order to improve the accuracy of torque estimation and system robustness. Summary of the Invention

[0005] In order to dynamically adjust the weights of different sensor modes in decision-making based on real-time operating conditions, thereby achieving high-precision and adaptive torque estimation and health prediction, this application provides a joint analysis method, device, robot joint module, and robot based on multimodal perception and adaptive fusion of operating conditions.

[0006] The above-mentioned objective of this application is achieved through the following technical solution:

[0007] A joint analysis method based on multimodal perception and adaptive working condition fusion, wherein the method is executed by a processor integrated into the joint module drive control board, comprising:

[0008] S1. Real-time acquisition of multimodal sensing data during the operation of the joint module, wherein the multimodal sensing data includes at least motor electrical data, mechanical vibration data and operating acoustic data;

[0009] S2. Based on the multimodal sensing data, construct a time-series feature matrix (H) and a working condition vector (c) reflecting the current operating state.

[0010] S3. Input the temporal feature matrix (H) and the working condition vector (c) into the condition-gated attention fusion model; wherein the model is configured to: dynamically generate at least one gating vector based on the working condition vector (c), and use the gating vector to modulate the weight allocation of the temporal feature matrix (H) in the attention calculation process, so as to output the fused high-level features;

[0011] S4. Based on the fused high-level features, generate and output the real-time torque estimate (τ_est) and mechanical health index of the joint output shaft.

[0012] In a preferred embodiment, this application can be further configured such that: in step S3, the conditional gated attention fusion model is a conditional gated multi-head attention network, the execution of which includes:

[0013] For the i-th attention head, the working condition vector (c) is mapped to a gated vector (g_i) using a gating generation function:

[0014] g_i=σ(W_{g_i}c+b_{g_i});

[0015] Where W_{g_i} and b_{g_i} are learnable parameters, and σ is the Sigmoid activation function;

[0016] The projection of the temporal feature matrix (H) is modulated using the gate vector (g_i) to calculate the condition-modulated attention weights.

[0017] In a preferred embodiment, this application can be further configured such that the step of modulation using the gate vector (g_i) includes:

[0018] The time-series feature matrix (H) is projected into a query matrix (Q_i), a key matrix (K_i), and a value matrix (V_i), respectively.

[0019] Calculate the modulated query matrix (Q_i') and key matrix (K_i'); where Q_i' = Q_i⊙g_i, K_i' = K_i⊙g_i, and ⊙ denotes element-wise multiplication;

[0020] The attention weight matrix (A_i) is calculated based on the modulated query matrix (Q_i') and key matrix (K_i').

[0021] In a preferred embodiment, this application can be further configured such that the method also includes an online self-calibration step:

[0022] S5. When the preset triggering conditions are met, control the joint to execute the preset calibration motion trajectory;

[0023] S6. During the calibration motion, the reference torque (τ_ref) is calculated by using the inverse dynamics model based on the ultra-high precision position information provided by the dual encoder system in the joint module.

[0024] S7. Using the reference torque (τ_ref) as a supervision signal, perform mini-batch gradient descent updates on the parameters used for torque estimation in the conditional gated attention fusion model, and freeze the feature extraction layer parameters of the model during the update process.

[0025] In a preferred embodiment, the present application may be further configured such that: the dual encoder system includes a first encoder disposed at the motor input end and a second encoder disposed at the joint output end; in step S6, the reference torque (τ_ref) is calculated based on the angular acceleration (α_out(t)) calculated from the output shaft position sequence (θ_out(t)) detected by the second encoder and the known load dynamic parameters.

[0026] The second objective of this invention is achieved through the following technical solution:

[0027] A joint analysis device based on multimodal perception and working condition adaptive fusion includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the method as described in any one of the above descriptions.

[0028] The third objective of this invention is achieved through the following technical solution:

[0029] A robot joint module, comprising:

[0030] case;

[0031] A hollow frameless torque motor and a multi-stage reducer are integrated within the housing;

[0032] The joint analysis device based on multimodal perception and adaptive fusion of working conditions, as described above;

[0033] In addition, a multimodal sensor group, which includes at least a current sensor for acquiring electrical data of the motor, a vibration sensor for acquiring mechanical vibration data, and an acoustic sensor for acquiring acoustic data of operation, wherein the multimodal sensor group is connected to the processor signal of a joint analysis device based on multimodal perception and adaptive fusion of operating conditions.

[0034] The fourth objective of this invention is achieved through the following technical solution:

[0035] A robot comprising the robot joint module described above.

[0036] In summary, this application includes at least one of the following beneficial technical effects:

[0037] 1. By employing multimodal information complementarity, time-series dynamic modeling, and condition-based adaptive fusion, the effects of single sensor failure, environmental interference, and nonlinear friction can be effectively suppressed. Experimental results show that under varying operating conditions and complex loads, the torque estimation error can be stabilized within ±1% FS (full scale), demonstrating a significant advantage over traditional current-based estimation methods (which typically have errors of ±5-10% FS).

[0038] 2. By integrating vibration and acoustic analysis with deep learning, the joint module possesses real-time mechanical health assessment and early fault warning capabilities without the need for additional expensive force sensors or dedicated vibration acquisition equipment. The MHI index can provide early warnings of potential faults such as minor gear wear and increased bearing clearance hundreds of hours in advance, enabling predictive maintenance.

[0039] 3. The same model can automatically adapt to joint operation under different speeds, loads, and temperatures without retraining or parameter adjustment, ensuring consistent and reliable performance across the robot's entire workspace. The design takes into account the constraints of the embedded platform (ARM+FPGA). Feature extraction (FFT, MFCC) can be parallelized, the attention mechanism can be highly optimized, and online fine-tuning computation is minimal. The entire algorithm can be completed within a 1ms control cycle, meeting the real-time control requirements of highly dynamic joints and achieving a balance between high-end performance and limited computing power platform deployment. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the implementation process of a joint analysis method based on multimodal perception and adaptive fusion of working conditions in one embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating the implementation process of a joint analysis method based on the fusion of multimodal perception and adaptive working conditions in another embodiment of this application. Detailed Implementation

[0042] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure.

[0044] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0045] like Figure 1 As shown, a joint analysis method based on multimodal perception and adaptive fusion of working conditions is used for torque estimation and health prediction of robot joints. The method is executed by a processor integrated into the joint module drive control board, and includes:

[0046] S1. Real-time acquisition of multimodal sensing data during joint module operation, including at least motor electrical data, mechanical vibration data and operating acoustic data;

[0047] The control cycle is 1ms. The control cycle refers to the fixed time interval within which the robot control system (in this invention, the drive control board integrated within the joint module) periodically executes a complete closed-loop process of data acquisition, algorithm calculation, and control command output. Multimodal sensing data includes:

[0048] Motor electrical data: Three-phase current is collected by current sensor and transformed to obtain q-axis current I_q; speed ω_m is obtained from encoder at motor end; motor winding temperature T_m and reducer housing temperature T_g are obtained from temperature sensor.

[0049] Mechanical vibration data: Raw vibration signals were acquired using a triaxial MEMS accelerometer (sampling rate 4kHz).

[0050] Acoustic data processing: Raw audio signals are acquired via a MEMS microphone (sampling rate 16kHz).

[0051] In one possible implementation, the motor electrical data includes the q-axis current I_q: 1.5A (representing the torque current component currently generated by the motor), motor speed ω_m: 2.0 rad / s (approximately 19.1 RPM), motor winding temperature T_m: 48.3℃, and reducer housing temperature T_g: 41.7℃. Mechanical vibration data includes, for example, a set of raw sampled values ​​(in g) output from a triaxial accelerometer, such as the X-axis: [0.011, 0.009, -0.005, 0.015] (4 sampling points). Operating acoustic data includes, for example, a set of raw audio sampled values ​​(in Pa) output from a MEMS microphone, such as [0.002, 0.001, -0.003, 0.000, ...].

[0052] Understandably, traditional solutions rely solely on motor current, resulting in a limited information dimension. This invention simultaneously collects three heterogeneous data sources: electrical, vibration, and acoustic. Electrical data (I_q, ω_m) directly reflects electromagnetic torque and motion state; vibration data contains microscopic mechanical states such as gear meshing and bearing rotation; and acoustic data includes broadband information on transient events such as friction and impact. The complementarity of multi-source data fundamentally overcomes the shortcomings of single sensors, such as susceptibility to interference and incomplete information, providing rich and comprehensive information input for subsequent fusion.

[0053] S2. Based on multimodal sensing data, construct a time-series feature matrix (H) and a working condition vector (c) that reflects the current operating state;

[0054] Construct the time series feature matrix H:

[0055] FFT was performed on the mechanical vibration data to extract the energy amplitude related to the gear meshing characteristic frequency, resulting in a 6-dimensional vibration feature vector F_vib.

[0056] The 26-dimensional MFCC features are calculated from the running acoustic data to obtain the acoustic feature vector F_audio.

[0057] The motor electrical data (I_q,ω_m,T_m), F_vib, and F_audio are concatenated to obtain a 40-dimensional feature vector f_t at a single time step.

[0058] Maintain a buffer of length 10 to store f_t for the most recent 10 control cycles (10ms). Stacking these buffers yields the time-series feature matrix H∈R^(10×40). Implement the construction of the time-series feature matrix (H).

[0059] Construct the working condition vector c:

[0060] Calculate the normalized rotational speed ω_norm and command torque τ_cmd_norm, and combine them with the normalized temperature T_m_norm and T_g_norm to form a 4-dimensional vector:

[0061] c=[ω_norm,τ_cmd_norm,T_m_norm,T_g_norm]^T.

[0062] Here, ^T is the standard notation in linear algebra, representing "transpose," which is the result of interchanging the rows and columns of a matrix. The resulting matrix is ​​called the transpose of the original matrix. After performing an FFT on the original vibration signal, the vibration energy (root mean square value) within a ±10Hz bandwidth around the two characteristic frequencies of 120Hz and 350Hz is extracted, yielding a 6-dimensional vibration feature vector F_vib. For example:

[0063] F_vib=[0.12,0.08,0.15,0.05,0.03,0.04]^T g.

[0064] Calculate the MFCC for a 16ms audio file to obtain the acoustic feature vector F_audio, for example, F_audio=[-5.2,1.3,0.8,-0.5,...,0.1]^T (26 dimensions).

[0065] The 40-dimensional feature vector f_t at a single time step is concatenated to obtain the following features:

[0066] f_t=Concat([I_q,ω_m,T_m],F_vib,F_audio)=[1.5,2.0,48.3,0.12,0.08,0.15,0.05,0.03,0.04,-5.2,1.3,...,0.1]^T (3+6+26=35 dimensions in total, supplemented with 5 zeros to reach 40 dimensions).

[0067] The temporal feature matrix (H) can be a cache of f_t from the most recent 10 time steps, forming a matrix. For example, the first row (the current time step) is the f_t above, the second row (1ms ago) is [1.48, 1.99, 48.2, 0.11, ...]^T, and so on. Ultimately, H is a 10-row × 40-column real number matrix.

[0068] "Maintaining a buffer of length 10" means storing data from the last 10 control cycles.

[0069] Assume the maximum motor speed ω_max = 3.14 rad / s and the maximum command torque τ_max = 10 Nm.

[0070] Calculate each value:

[0071] ω_norm=ω_m / ω_max=2.0 / 3.14≈0.637.

[0072] τ_cmd_norm=3.0 / 10=0.3 (assuming the current command torque is 3.0Nm).

[0073] T_m_norm=(48.3-25) / 70≈0.333.

[0074] T_g_norm=(41.7-25) / 70≈0.239.

[0075] The working condition vector is c = [0.637, 0.3, 0.333, 0.239]^T.

[0076] Dynamic process representation and operational state quantification were achieved, providing key inputs for adaptive fusion. Single-point features were extended into a time-series matrix H∈R^(T×D), enabling the model to learn and utilize the dynamic processes and historical dependencies of joint motion, rather than just static instants. This is crucial for smooth estimation of torque changes and early fault symptom identification. A condition vector c was constructed, quantizing the continuous and complex operating environment (speed, load, temperature) into a low-dimensional, dense feature vector.

[0077] S3. Input the temporal feature matrix (H) and the operating condition vector (c) into the condition-gated attention fusion model; wherein the model is configured to: dynamically generate at least one gating vector based on the operating condition vector (c), and use the gating vector to modulate the weight allocation of the temporal feature matrix (H) in the attention calculation process, so as to output the fused high-level features.

[0078] The model input consists of H and c obtained in step S2, which are input into a conditionally gated multi-head attention network. The number of attention heads is set to h=4.

[0079] The conditionally gated attention fusion model is a conditionally gated multi-head attention network, and its execution includes:

[0080] For the i-th attention head, the condition vector (c) is mapped to the gate vector (g_i) through a gating generation function:

[0081] g_i=σ(W_{g_i}c+b_{g_i}).

[0082] Where W_{g_i} and b_{g_i} are learnable parameters, and σ is the Sigmoid activation function.

[0083] The projection of the temporal feature matrix (H) is modulated using a gating vector (g_i) to calculate the condition-modulated attention weights.

[0084] For example, when c represents high-speed operating conditions, the W_{g_i} and b_{g_i} learned by the model can enable g_i to output high values ​​(such as above 0.9) in the dimension corresponding to the vibration characteristics.

[0085] Specifically, the projection of the temporal feature matrix (H) is modulated using a gate vector (g_i) to calculate the condition-modulated attention weights, including:

[0086] The time-series feature matrix (H) is projected into a query matrix (Q_i), a key matrix (K_i), and a value matrix (V_i), respectively.

[0087] Calculate the modulated query matrix (Q_i') and key matrix (K_i'); where Q_i' = Q_i⊙g_i, K_i' = K_i⊙g_i, and ⊙ denotes element-wise multiplication.

[0088] The attention weight matrix (A_i) is calculated based on the modulated query matrix (Q_i') and key matrix (K_i').

[0089] If the value of g_i in a certain dimension is 0.2, then when calculating attention, the influence of that dimension feature (which may correspond to a certain temperature feature) is weakened by 80%; if the value is 0.95, then the influence of that dimension feature (which may correspond to a certain key vibration feature) is fully preserved.

[0090] Attention weights are calculated using the modulated Qi' and Ki', and then weighted and summed with V_i to obtain the head output. Finally, the outputs of all heads are fused to obtain the fused features with adaptive modulation based on operating conditions. This collectively realizes the input to the conditionally gated attention fusion model and the output of the fused high-level features.

[0091] The following is a calculation example: Taking the first attention head (i=1) as an example, its parameter W_{g_1} is a 10×4 matrix, and b_{g_1} is a 10-dimensional vector. Assume its first row of parameters is:

[0092] W_{g_1}[1,:]=[0.1,0.2,-0.1,0.05],b_{g_1}[1]=0.0.

[0093] The `[1,:]` syntax is similar to array slicing or indexing in programming languages ​​like Python and MATLAB. It's used to extract specific rows or columns from a matrix (a two-dimensional array). The first number, 1, indicates the row index, and the colon `:` indicates "all" or "the entire range". For example, `W_{g_1}[1,:]` extracts the first row of matrix `W_{g_1}`, and `W_1^Q[:,1]` extracts the first column of matrix `W_1^Q`. Linear part:

[0094] z_1=W_{g_1}[1,:]·c+b_{g_1}[1]=0.1*0.637+0.2*0.3+(-0.1)*0.333+0.05*0.239+0.0=0.0637+0.06-0.0333+0.01195≈0.1024.

[0095] Activation part: g_1[1]=σ(z_1)=1 / (1+exp(-0.1024))≈1 / (1+0.9027)≈0.5256.

[0096] z_1 is an intermediate calculation variable, specifically referring to the calculation result of the linear transformation part when generating the gating vector g_1 of the first attention head. The magnitude of z_1 initially reflects the different initial importance that the model parameters W_{g_1} and b_{g_1} believe should be assigned to subsequent features under a given operating condition c. Then, z_1 is mapped to the (0, 1) interval through the Sigmoid function σ to obtain the final gating weight g_1 used for modulation. g_1[1]≈0.53 indicates that under the current operating condition c, the first attention head assigns an initial importance weight of about 0.53 to the first feature dimension in the H matrix. Assuming that this dimension corresponds to I_q, it means that under this operating condition, the current feature is given a moderate degree of attention.

[0097] The projection and modulation of query (Q), key (K), and value (V) (Q_i'=Q_i⊙g_i, K_i'=K_i⊙g_i) are calculated as follows:

[0098] Projection: H is transformed into Q_1 by the projection matrix W_1^Q (dimension 40×10). Example of calculating the first row and first column element of Q_1:

[0099] Q_1[1,1]=H[1,:]·W_1^Q[:,1](a 40-dimensional dot product).

[0100] Modulation: Assuming Q_1[1,1]=0.5 is calculated, modulation is performed using g_1[1]=0.5256 generated in the previous step:

[0101] Q_1'[1,1]=Q_1[1,1]*g_1[1]=0.5*0.5256=0.2628.

[0102] Comparative example: Suppose another attention head (i=2) has g_2[7]=0.95 (corresponding to a certain vibration feature dimension), and Q_2[1,7]=0.9, then after modulation Q_2'[1,7]=0.9*0.95=0.855, the feature is strongly preserved. This reflects the dynamic weight allocation: the vibration feature obtains a higher "confidence weight" under high-speed conditions.

[0103] Attention weight calculation: (A_i=Softmax((Q_i'(K_i')^T) / √(d_k)))

[0104] Assume that after modulation, the dot product (self-attention) of Q_1'[1,:] and K_1'[1,:] is 3.0, the dot product of Q_1'[2,:] and K_1'[2,:] is 1.0, and d_k=10.

[0105] Calculate the scaled dot product: s11 = 3.0 / √10 ≈ 0.949, s12 = 1.0 / √10 ≈ 0.316.

[0106] Calculate the Softmax weights:

[0107] A_1[1,1]=exp(0.949) / (exp(0.949)+exp(0.316)+...)≈2.583 / (2.583+1.372+...)≈0.65. This indicates that in the fused features at the current time step, the model considers its own time step features to contribute 65% of the weight. Here, √ represents the square root.

[0108] This step achieves adaptive and interpretable fusion of cross-modal information. Through the gated generation function g_i=σ(W_{g_i}c+b_{g_i}), the model learns to automatically adjust the "credibility" of each sensor feature according to the real-time operating condition c. For example, it learns that "the vibration feature weight g_i increases at high speeds, and the current feature weight g_i increases at low speeds and high loads," solving the performance degradation problem of fixed fusion strategies under varying operating conditions. Attention is calculated using the modulated Qi' and Ki', allowing the model to naturally focus on the most relevant feature dimensions and time segments under the current operating condition during feature association calculations, suppressing noise and irrelevant information interference, thereby extracting purer and more effective fused features. The gated vector g_i can serve as an intermediate variable, intuitively reflecting the model's decision-making basis under different operating conditions.

[0109] S4. Based on the fused high-level features, generate and output the real-time torque estimate (τ_est) and mechanical health index of the joint output axis.

[0110] The fused features output from step S3 are processed through two independent fully connected network branches:

[0111] One branch outputs a scalar, the real-time torque estimate τ_est. The other branch outputs a scalar between 0 and 1, the Mechanical Health Index (MHI), where a value closer to 1 indicates a worse health condition. This function generates and outputs the real-time torque estimate (τ_est) and the Mechanical Health Index for the joint's output axis.

[0112] The fused features are passed through a fully connected network, and the torque estimation branch is assumed to output: τ_est = 3.15 Nm.

[0113] Health score branch output: MHI=0.12 (far below the warning threshold of 0.7, indicating good health).

[0114] This step unifies the functions of "state awareness" and "health assessment" within a single model, achieving high efficiency and strong correlation. It shares underlying fusion features and outputs the torque estimate τ_est and the machine health index MHI in parallel. This design allows health assessment to utilize the high-quality dynamic features required for torque estimation, and conversely, torque estimation implicitly benefits from the model's understanding of the machine's state. The two are trained collaboratively and mutually reinforce each other, resulting in higher data utilization efficiency and intrinsic consistency compared to training two independent models.

[0115] Reference Figure 2 In another embodiment, the method further includes an online self-calibration step:

[0116] S5. When the preset triggering conditions are met, control the joint to execute the preset calibration motion trajectory.

[0117] Specifically, when the system detects the preset trigger condition of "continuous operation for 8 hours", the calibration process is automatically initiated. The control joint executes a preset uniform rotation trajectory at an extremely low speed of 0.05 rad / s under no-load conditions.

[0118] Specifically, it is triggered after running continuously for 8*3600=28800 seconds. The joint is controlled to rotate at a constant speed of ω_out=0.05rad / s.

[0119] S6. During the calibration motion, the reference torque (τ_ref) is calculated using the inverse dynamics model based on the ultra-high precision position information provided by the dual encoder system in the joint module.

[0120] The dual encoder system includes a first encoder located at the motor input end and a second encoder located at the joint output end; the reference torque (τ_ref) is calculated based on the angular acceleration (α_out(t)) obtained from the output shaft position sequence (θ_out(t)) detected by the second encoder and the known load dynamic parameters.

[0121] During the calibration trajectory execution, the dual encoder system initiates high-precision data acquisition. In particular, the second encoder (magnetic absolute type) located at the joint output provides position information θ_out(t).

[0122] Numerical differentiation of θ_out(t) yields the precise angular acceleration α_out(t). Since it is uniform motion, α_out(t)≈0.

[0123] Based on this, the reference torque is calculated using an inverse dynamics model:

[0124] τ_ref(t) = I_load * α_out(t) + f_visc * ω_out(t) + f_coul * sign(ω_out(t)). Since α_out(t) ≈ 0, τ_ref(t) mainly consists of small frictional torques and can be used as a high-confidence true value signal. The reference torque (τ_ref) is calculated using an inverse dynamics model.

[0125] For example, the second encoder measures the position sequence, and the angular acceleration α_out(t)≈0rad / s² is obtained by numerical differentiation.

[0126] Given the load moment of inertia I_load = 0.01 kg·m², the coefficient of viscous friction f_visc = 0.1 N·m·s / rad, and the Coulomb friction torque f_coul = 0.05 N·m.

[0127] Calculation: τ_ref=0.01*0+0.1*0.05+0.05*sign(0.05)=0.005+0.05=0.055Nm. This is the true value of the reference torque with high confidence.

[0128] S7. Using the reference torque (τ_ref) as the supervision signal, perform mini-batch gradient descent updates on the parameters used for torque estimation in the conditional gated attention fusion model, and freeze the feature extraction layer parameters of the model during the update process.

[0129] Specifically, multiple sets of data (H, c, τ_ref) generated during the calibration process are collected. Using τ_ref as the supervision signal, mini-batch gradient descent updates are performed only on the parameters used for moment estimation in the conditionally gated attention fusion model (i.e., the weights of the fully connected layers in the output layer). Simultaneously, the parameters of the feature extraction layers (such as FFT and MFCC computation logic) and the attention fusion layer itself are strictly frozen. An extremely low learning rate (1e-5) is used during the update process.

[0130] Continuing the previous example, a set of calibration data (H_calib, c_calib, τ_ref=0.055) is collected. Forward propagation: The current model output τ_est'=0.07Nm, with an error of 0.015Nm. Backward propagation: Using the loss function L=(0.07-0.055)^2=0.000225, the gradient is calculated and updated only for the weights of the torque output layer (e.g., a 40×1 matrix W_out). Assuming an element w in W_out originally has a value of 0.5, a learning rate η=1e-5, and a gradient ∂L / ∂w=0.006, then after the update, w'=0.5-1e-5*0.006=0.499994. The parameters of the feature extraction layer and attention layer are frozen and remain unchanged.

[0131] Traditional data-driven models experience performance degradation after deployment due to mechanical wear and component aging ("concept drift"). This invention utilizes high-precision position information provided by a dual-encoder system to generate a "true value" τ_ref under a specific calibration trajectory through inverse dynamics. This value is then used to update (fine-tune) limited parameters of the model, enabling the torque estimation output to track real-time changes in the system. Freezing the feature extraction layer ensures that the model's fundamental perception and fusion capabilities are not compromised; only the output mapping relationship is calibrated, achieving safe, efficient, and lifelong learning.

[0132] This application also provides a joint analysis device based on multimodal perception and working condition adaptive fusion, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements any of the methods described above.

[0133] Specific limitations regarding the joint analysis device based on multimodal perception and adaptive fusion of operating conditions can be found in the limitations of the joint analysis method based on multimodal perception and adaptive fusion of operating conditions mentioned above, and will not be repeated here. Each step of the joint analysis method based on multimodal perception and adaptive fusion of operating conditions can be implemented entirely or partially through software, hardware, or a combination thereof.

[0134] According to embodiments of this application, this application also provides a robot joint module, comprising:

[0135] case;

[0136] A hollow frameless torque motor and a multi-stage reducer are integrated into the housing;

[0137] Such as a joint analysis device based on multimodal perception and working condition adaptive fusion;

[0138] In addition, a multimodal sensor group, which includes at least a current sensor for acquiring electrical data of the motor, a vibration sensor for acquiring mechanical vibration data, and an acoustic sensor for acquiring acoustic data of operation, is connected to the processor signal of a joint analysis device based on multimodal perception and adaptive fusion of operating conditions.

[0139] According to embodiments of this application, this application also provides a robot comprising a robot joint module.

[0140] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A joint analysis method based on multimodal perception and adaptive working condition fusion, characterized in that, The method is executed by a processor integrated into the joint module driver control board, and includes: S1. Real-time acquisition of multimodal sensing data during the operation of the joint module, wherein the multimodal sensing data includes at least motor electrical data, mechanical vibration data and operating acoustic data; S2. Based on the multimodal sensing data, construct a time-series feature matrix (H) and a working condition vector (c) reflecting the current operating state. S3. Input the temporal feature matrix (H) and the working condition vector (c) into the condition-gated attention fusion model; wherein the model is configured to: dynamically generate at least one gating vector based on the working condition vector (c), and use the gating vector to modulate the weight allocation of the temporal feature matrix (H) in the attention calculation process, so as to output the fused high-level features; S4. Based on the fused high-level features, generate and output the real-time torque estimate (τ_est) and mechanical health index of the joint output shaft.

2. The joint analysis method based on multimodal perception and adaptive fusion of working conditions as described in claim 1, characterized in that, In step S3, the conditional gated attention fusion model is a conditional gated multi-head attention network, and its execution includes: For the i-th attention head, the working condition vector (c) is mapped to a gated vector (g_i) using a gating generation function: g_i=σ(W_{g_i}c+b_{g_i}); Where W_{g_i} and b_{g_i} are learnable parameters, and σ is the Sigmoid activation function; The projection of the temporal feature matrix (H) is modulated using the gate vector (g_i) to calculate the condition-modulated attention weights.

3. The joint analysis method based on multimodal perception and adaptive fusion of working conditions as described in claim 1, characterized in that, The step of modulation using the gate vector (g_i) includes: The time-series feature matrix (H) is projected into a query matrix (Q_i), a key matrix (K_i), and a value matrix (V_i), respectively. Calculate the modulated query matrix (Q_i') and key matrix (K_i'); where Q_i' = Q_i⊙g_i, K_i' = K_i⊙g_i, and ⊙ denotes element-wise multiplication; The attention weight matrix (A_i) is calculated based on the modulated query matrix (Q_i') and key matrix (K_i').

4. The joint analysis method based on multimodal perception and adaptive fusion of working conditions as described in claim 3, characterized in that, The method also includes an online self-calibration step: S5. When the preset triggering conditions are met, control the joint to execute the preset calibration motion trajectory; S6. During the calibration motion, the reference torque (τ_ref) is calculated by using the inverse dynamics model based on the ultra-high precision position information provided by the dual encoder system in the joint module. S7. Using the reference torque (τ_ref) as a supervision signal, perform mini-batch gradient descent updates on the parameters used for torque estimation in the conditional gated attention fusion model, and freeze the feature extraction layer parameters of the model during the update process.

5. The joint analysis method based on multimodal perception and adaptive fusion of working conditions as described in claim 4, characterized in that, The dual encoder system includes a first encoder disposed at the motor input end and a second encoder disposed at the joint output end; in step S6, the reference torque (τ_ref) is calculated based on the angular acceleration (α_out(t)) calculated from the output shaft position sequence (θ_out(t)) detected by the second encoder and the known load dynamic parameters.

6. A joint analysis device based on multimodal perception and adaptive fusion of working conditions, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 5.

7. A robot joint module, characterized in that, include: case; A hollow frameless torque motor and a multi-stage reducer are integrated within the housing; The joint analysis device based on multimodal perception and adaptive fusion of working conditions as described in claim 6; In addition, a multimodal sensor group, which includes at least a current sensor for acquiring electrical data of the motor, a vibration sensor for acquiring mechanical vibration data, and an acoustic sensor for acquiring acoustic data of operation, wherein the multimodal sensor group is connected to the processor signal of a joint analysis device based on multimodal perception and adaptive fusion of operating conditions.

8. A robot, characterized in that, It includes at least one robot joint module as described in claim 7.

Images