Sensor signal prediction method, system and equipment for humanoid robot

Abstract

The invention discloses a sensor signal prediction method, system and equipment for a humanoid robot. The method comprises the following steps: acquiring a multi-source sensor training sample of the humanoid robot; training the heterogeneous neural network fusion model according to the multi-source sensor training sample through a signal amplitude loss function, a frequency loss function and a transient characteristic loss function to obtain a signal prediction model; obtaining current sensor signals of the humanoid robot, wherein the current sensor signals comprise at least two of a joint angle signal, a motor current signal or an acceleration signal; and inputting the current sensor signal into the signal prediction model, and outputting a predicted sensor signal which is a force sense signal or a torque signal. Through multi-source sensor signal fusion and multi-domain feature learning, accurate prediction of key sensor signals is realized, the number of deployed sensors is reduced, the cost is reduced, the system reliability is improved, and the method can be widely applied to the technical field of robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a sensor signal prediction method, system and device for a humanoid robot. BACKGROUND

[0002] In the field of industrial automation, a humanoid robot needs to have high-precision motion control and environmental interaction capability, which relies on the cooperative work of a large number of sensors, such as joint position sensors, force sensors, acceleration sensors, and vision sensors. However, the intensive deployment of sensors has the following problems:

[0003] 1. High hardware cost: Industrial-grade force sensors, torque sensors, and other precision devices are expensive, and the sensor cost of a multi-joint humanoid robot can account for more than 30%;

[0004] 2. High system complexity: The increase in the number of sensors leads to complex wiring and increased data transmission delay, reducing system reliability;

[0005] 3. Difficult to maintain: The harsh industrial environment (such as dust and vibration) is prone to cause sensor failure, and the troubleshooting and replacement cost of a multi-sensor system is high;

[0006] 4. Kinetic coupling interference: Installing sensors at some key positions (such as end effectors) will change their inertia parameters, affecting motion accuracy.

[0007] In the prior art, the signal prediction for robots mostly uses a single neural network model, which only focuses on the time domain features of the signal, resulting in insufficient prediction accuracy and difficulty in meeting the high-precision requirements of industrial scenarios. SUMMARY

[0008] The main purpose of the embodiments of the present application is to propose a sensor signal prediction method, system and device for a humanoid robot with high prediction accuracy and low cost.

[0009] To achieve the above purpose, one aspect of the embodiments of the present application proposes a sensor signal prediction method for a humanoid robot, comprising the following steps:

[0010] Obtaining a multi-source sensor training sample of a humanoid robot, the multi-source sensor training sample comprising joint angle signals, motor current signals, acceleration signals, force signals, and torque signals;

[0011] Training a preset heterogeneous neural network fusion model according to the multi-source sensor training sample through a preset signal amplitude loss function, a frequency loss function, and a transient feature loss function, to obtain a signal prediction model;

[0012] obtaining a current sensor signal of the humanoid robot, the current sensor signal comprising at least two of the joint angle signal, the motor current signal, or the acceleration signal;

[0013] inputting the current sensor signal into the signal prediction model to output a predicted sensor signal, the predicted sensor signal being the force / torque signal.

[0014] In some embodiments, the obtaining of the multi-source sensor training sample of the humanoid robot specifically comprises:

[0015] collecting a plurality of original sensor signals;

[0016] performing noise reduction processing on each of the original sensor signals by improving a variational mode decomposition algorithm;

[0017] performing at least one or a combination of time domain alignment processing, frequency domain feature enhancement processing, or transient peak smoothing processing on the original sensor signals after noise reduction processing to obtain the multi-source sensor training sample.

[0018] In some embodiments, the performing of noise reduction processing on each of the original sensor signals by improving a variational mode decomposition algorithm specifically comprises:

[0019] obtaining a joint motion cycle of the humanoid robot, and determining a center frequency search range of an intrinsic mode function according to the joint motion cycle;

[0020] decomposing the original sensor signal into a plurality of intrinsic mode functions according to the center frequency search range;

[0021] calculating a sample entropy of each of the intrinsic mode functions, and performing screening on the intrinsic mode functions according to the sample entropy and a preset sample entropy threshold to obtain the original sensor signal after noise reduction processing.

[0022] In some embodiments, the heterogeneous neural network fusion model comprises at least two of:

[0023] a bidirectional long short-term memory network for processing time series features;

[0024] a convolutional neural network for extracting spatial correlation features;

[0025] a gradient boosting decision tree network for processing high-dimensional table features.

[0026] In some embodiments, the training of the preset heterogeneous neural network fusion model according to the multi-source sensor training sample by using the preset signal amplitude loss function, frequency loss function, and transient feature loss function to obtain a signal prediction model specifically comprises:

[0027] inputting the multi-source sensor training sample into the heterogeneous neural network fusion model, outputting a predicted signal, the multi-source sensor training sample comprising a sample label;

[0028] constructing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to the predicted signal and the sample label;

[0029] weighting and summing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to a proportion of a difference between the predicted signal and the sample label, to obtain a combined loss function;

[0030] training the heterogeneous neural network fusion model through the combined loss function, to obtain the signal prediction model.

[0031] In some embodiments, the signal amplitude loss function is calculated through a dynamic time warping algorithm, and a formula is as follows:

[0032]

[0033] wherein, L amp represents a signal amplitude loss, Y represents a sample label, represents a predicted signal, and DTW(·,·) represents a dynamic time warping distance calculation;

[0034] The frequency loss function is calculated through a main frequency difference after Fourier transform, and a formula is as follows:

[0035]

[0036] wherein, L fre represents a frequency loss, FFT(·) represents a fast Fourier transform, f max represents a main frequency component, and ‖·‖ represents an L1 norm;

[0037] The transient feature loss function is calculated based on a time difference and an amplitude difference of a signal mutation point, and a formula is as follows:

[0038]

[0039] wherein, L tra represents a transient feature loss, t i and A i respectively represent an i th mutation point time and amplitude of a sample label, and respectively represent an i th mutation point time and amplitude of a predicted signal, γ i represents a mutation point weight, θ represents an amplitude time conversion coefficient, and i is a natural number and i ∈ [1, n].

[0040] In some embodiments, the signal amplitude loss function, the frequency loss function and the transient feature loss function are weighted and summed to obtain a combined loss function according to a difference proportion of the predicted signal and the sample label, specifically comprising:

[0041] setting a difference proportion threshold;

[0042] calculating an amplitude difference proportion of the predicted signal and the sample label, and when the amplitude difference proportion exceeds the difference proportion threshold, obtaining a first weight combination, and then weighting and summing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to the first weight combination to obtain a first combined loss function;

[0043] calculating a frequency difference proportion of the predicted signal and the sample label, and when the frequency difference proportion exceeds the difference proportion threshold, obtaining a second weight combination, and then weighting and summing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to the second weight combination to obtain a second combined loss function;

[0044] calculating a transient feature difference proportion of the predicted signal and the sample label, and when the transient feature difference proportion exceeds the difference proportion threshold, obtaining a third weight combination, and then weighting and summing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to the third weight combination to obtain a third combined loss function.

[0045] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a sensor signal prediction system for a humanoid robot, comprising:

[0046] a training sample acquisition module configured to acquire multi-source sensor training samples of a humanoid robot, wherein the multi-source sensor training samples comprise joint angle signals, motor current signals, acceleration signals, force sense signals and torque signals;

[0047] a model training module configured to train a preset heterogeneous neural network fusion model according to the multi-source sensor training samples by using a preset signal amplitude loss function, a frequency loss function and a transient feature loss function, to obtain a signal prediction model;

[0048] a current signal acquisition module configured to acquire current sensor signals of the humanoid robot, wherein the current sensor signals comprise at least two of the joint angle signals, the motor current signals or the acceleration signals;

[0049] a signal prediction module configured to input the current sensor signal into the signal prediction model, and output a predicted sensor signal, the predicted sensor signal being the force / torque signal.

[0050] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method as described above when executing the computer program.

[0051] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method as described above.

[0052] The embodiments of the present application at least have the following beneficial effects: the sensor signal prediction method, system and device for humanoid robots provided by the present application first acquire a multi-source sensor training sample of a humanoid robot, then train a preset heterogeneous neural network fusion model according to the multi-source sensor training sample through a preset signal amplitude loss function, a frequency loss function and a transient feature loss function, obtain a signal prediction model, finally acquire a current sensor signal of the humanoid robot, input the current sensor signal into the signal prediction model, and output a predicted sensor signal. The signal prediction model for predicting the sensor signal is trained through the signal amplitude loss function, the frequency loss function and the transient feature loss function, which can fully exploit the correlation of the sensor signal in the time domain, the frequency domain and the transient feature, realize accurate prediction of a key signal based on an existing sensor signal, reduce the number of deployed sensors in an industrial scene, reduce the hardware cost and maintenance complexity, and improve the motion control accuracy and environmental adaptability of the humanoid robot. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application. It should be understood that the drawings introduced below are only for facilitating clear description of some embodiments in the technical solutions of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the premise that the drawings are not provided.

[0054] Figure 1 The step flow chart of the sensor signal prediction method for humanoid robots provided by an embodiment of the present application is shown in the following figure:

[0055] Figure 2 The flowchart of the sample acquisition stage provided by an embodiment of the present application is shown in the following figure:

[0056] Figure 3A structural schematic diagram of a heterogeneous neural network fusion model provided by an embodiment of the present application is shown in FIG. 1.

[0057] Figure 4 A flowchart of a model training stage provided by an embodiment of the present application is shown in FIG. 2.

[0058] Figure 5 A flowchart of a signal prediction stage provided by an embodiment of the present application is shown in FIG. 3.

[0059] Figure 6 A structural schematic diagram of a sensor signal prediction system for a humanoid robot provided by an embodiment of the present application is shown in FIG. 4.

[0060] Figure 7 A hardware structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0063] In the field of industrial automation, a humanoid robot needs to have high-precision motion control and environmental interaction capability, which relies on the cooperative work of a large number of sensors, such as joint position sensors, force sensors, acceleration sensors, vision sensors, etc. However, the intensive deployment of sensors has the following problems:

[0064] 1. High hardware cost: Industrial-grade force sensors, torque sensors, and other precision devices are expensive, and the sensor cost of a multi-joint humanoid robot can account for more than 30%;

[0065] 2. High system complexity: The increase in the number of sensors leads to complex wiring and increased data transmission delay, reducing system reliability;

[0066] 3. Difficult to maintain: The harsh industrial environment (such as dust, vibration) is prone to cause sensor failure, and the troubleshooting and replacement cost of a multi-sensor system is high;

[0067] 4. Kinematic coupling interference: Installing sensors on some key parts (such as end effectors) will change their inertia parameters, affecting motion accuracy.

[0068] In the prior art, signal prediction for robots mostly uses a single neural network model, only focusing on the time domain features of the signal, resulting in insufficient prediction accuracy and difficulty in meeting the high-precision requirements of industrial scenarios.

[0069] Therefore, an embodiment of the present application provides a sensor signal prediction method for a humanoid robot. First, a plurality of source sensor training samples of the humanoid robot are obtained. Then, a preset heterogeneous neural network fusion model is trained according to the plurality of source sensor training samples by using a preset signal amplitude loss function, a frequency loss function and a transient feature loss function, to obtain a signal prediction model. Finally, a current sensor signal of the humanoid robot is obtained, and the current sensor signal is input into the signal prediction model to output a predicted sensor signal. The signal prediction model for predicting the sensor signal is trained by using the signal amplitude loss function, the frequency loss function and the transient feature loss function, which can fully exploit the correlation of the sensor signal in the time domain, the frequency domain and the transient feature, realize accurate prediction of a key signal based on an existing sensor signal, reduce the number of sensors deployed in an industrial scenario, reduce hardware cost and maintenance complexity, and improve motion control accuracy and environmental adaptability of the humanoid robot.

[0070] Reference Figure 1 , Figure 1 A step flowchart of a sensor signal prediction method for a humanoid robot is provided in an embodiment of the present application. The sensor signal prediction method for a humanoid robot can include but is not limited to the following steps S101 to S104:

[0071] In step S101, a plurality of source sensor training samples of the humanoid robot are obtained. The plurality of source sensor training samples include joint angle signals, motor current signals, acceleration signals, force sense signals and torque signals.

[0072] Specifically, sensor signals of the humanoid robot in an industrial scenario (such as grasping and assembly) are collected, including joint angle, motor current, acceleration and force sense signals, etc. Then, the original sensor signals are preprocessed. The preprocessing process can use an improved variational modal decomposition algorithm for noise reduction, combine with joint motion cycle constraint to optimize intrinsic modal function decomposition, and retain effective signal components. Then, time domain alignment (based on motion synchronization clock), frequency domain feature enhancement (extracting main frequency and harmonic components), transient peak smoothing (suppressing sudden changes caused by measurement noise) and other processes are performed.

[0073] Further as optional implementation, step S101 can be further divided into steps S1011-S1013.

[0074] Step S1011, collect several original sensor signals;

[0075] Step S1012, perform noise reduction processing on each original sensor signal by an improved variational modal decomposition algorithm.

[0076] Further as optional implementation, step S1012 can be further divided into steps S10121-S10123.

[0077] Step S10121, obtain the joint motion cycle of the humanoid robot, and determine the center frequency search range of the intrinsic modal function according to the joint motion cycle;

[0078] Step S10122, decompose the original sensor signal into a plurality of intrinsic modal functions according to the center frequency search range.

[0079] Step S10123, calculate the sample entropy of each intrinsic modal function, and filter the intrinsic modal function according to the sample entropy and a preset sample entropy threshold value, to obtain the original sensor signal after noise reduction processing.

[0080] In some optional embodiments, the improved variational modal decomposition algorithm optimizes the intrinsic modal function decomposition by introducing joint motion cycle constraint. First, based on the joint motion cycle T of the humanoid robot, the center frequency search range of the intrinsic modal function is set as [1 / T, 5 / T], then the original sensor signal is decomposed into an intrinsic modal function set satisfying the center frequency search range, and then the noise component is filtered and removed through the sample entropy threshold value, and the effective signal component is retained.

[0081] Step S1013, at least one or a combination of time domain alignment processing, frequency domain feature enhancement processing or transient peak smoothing processing is performed on the original sensor signal after noise reduction processing, to obtain a multi-source sensor training sample.

[0082] Exemplarily, taking the humanoid robot performing a workpiece grabbing task as an example scenario, compliant control needs to be realized through an end force signal. The traditional scheme needs to install a six-dimensional force sensor at the end, while the embodiment of the application predicts the end force signal through the signals of the joint angle sensor and the motor current sensor, thereby eliminating the force sensor and reducing the hardware cost and maintenance complexity.

[0083] As Figure 2The flowchart shown is the sample acquisition stage. For signal acquisition and preprocessing, first install an angle sensor (sampling rate 1 kHz), a motor current sensor (sampling rate 1 kHz) at the wrist joint of the humanoid robot, and a reference force sensor (sampling rate 1 kHz) at the end, collect 500 sets of sensor signals during the grabbing process, each set containing: input signal X: joint angle θ(t), motor current I(t); label signal Y: end force F(t). Then use the improved variational modal decomposition algorithm to denoise the joint angle θ(t), motor current I(t) and end force F(t). Specifically: given the wrist joint motion period T = 0.5s (corresponding to a frequency of 2Hz), set the intrinsic modal function center frequency range to [2Hz, 10Hz]; decompose the original sensor signal into 5 intrinsic modal functions (signal components), calculate the sample entropy of each signal component, remove noise components with sample entropy > sample entropy threshold (such as 0.8), and retain effective signal components with energy ratio > 90%. Finally, perform feature enhancement processing on the denoised original sensor signal, which includes at least one or a combination of the following:

[0084] 1) Time domain processing: sliding window interception (window size 50ms) is performed on the joint angle θ(t) and motor current I(t) to construct samples containing time correlation;

[0085] 2) Frequency domain processing: fast Fourier transform is performed on the end force F(t) to extract 1-5Hz main frequency components, and the frequency domain features are integrated into the time domain signal through inverse transform;

[0086] 3) Transient processing: wavelet transform is used to identify the contact mutation point (force value change rate > 50N / s) in the end force F(t), and the signal details 100ms before and after the mutation are retained.

[0087] Step S102, train the preset heterogeneous neural network fusion model according to the multi-source sensor training sample through the preset signal amplitude loss function, frequency loss function and transient feature loss function, and obtain a signal prediction model;

[0088] Further, as an optional implementation, the heterogeneous neural network fusion model includes at least two of the following:

[0089] Bidirectional long short-term memory network for processing time series features;

[0090] Convolutional neural network for extracting spatial correlation features;

[0091] Gradient boosting decision tree network for processing high-dimensional table features.

[0092] Specifically, as Figure 3As shown is a structural schematic diagram of a heterogeneous neural network fusion model, the heterogeneous neural network fusion model comprising at least two subnetworks and an attention fusion layer:

[0093] 1) Bidirectional LSTM network (i.e., bidirectional long short-term memory network): processing time-dependent features, suitable for continuous change signals such as joint angles and currents;

[0094] 2) CNN network (i.e., convolutional neural network): extracting spatial correlation features, suitable for signals containing multi-dimensional vibration information such as accelerations;

[0095] 3) Gradient boosting decision tree network: processing high-dimensional feature interactions, suitable for nonlinear relationships of fused multi-source signals.

[0096] Attention fusion layer: weighted fusion of each subnetwork output through attention mechanism to obtain predicted signal.

[0097] Exemplarily, taking a humanoid robot performing a workpiece grasping task as an example scenario, compliant control needs to be achieved through an end effector force signal, the input signal X includes joint angles θ(t) and motor currents I(t), and the label signal Y is an end effector force F(t). The bidirectional LSTM subnetwork includes 2 layers of bidirectional LSTM, each layer has 64 neurons, the input is a time sequence of joint angles θ(t) (50 ms window, 50 sampling points), and the output is a 32-dimensional time feature vector. The CNN subnetwork includes 2 layers of convolution (3x3 convolution kernel, step 1) + pooling, the input is a vibration feature matrix of motor currents I(t) (50x3, including currents and first and second derivatives), and the output is a 32-dimensional frequency domain feature vector. The gradient boosting decision tree subnetwork includes a LightGBM model (100 trees), the input is statistical features (mean, peak, variance, etc. 10-dimensional features) of joint angles θ(t) and motor currents I(t), and the output is a 32-dimensional feature interaction vector. The attention fusion layer weights the outputs (32x3) of the three subnetworks through an attention mechanism, the weights are dynamically calculated through a softmax function, and finally the predicted force signal is output.

[0098] Further as an optional implementation, step S102 can be further divided into steps S1021 to S1024:

[0099] Step S1021, inputting a multi-source sensor training sample into a heterogeneous neural network fusion model to output a predicted signal, the multi-source sensor training sample comprising a sample label;

[0100] Step S1022, constructing a signal amplitude loss function, a frequency loss function, and a transient feature loss function according to the predicted signal and the sample label;

[0101] Specifically, the signal amplitude loss function constructed in the embodiment of the present application is used to calculate the time sequence shape difference between the predicted signal and the sample label based on the dynamic time warping algorithm; the frequency loss function is used to extract the main frequency component through Fourier transform to calculate the main frequency difference between the prediction and the label; and the transient feature loss function is used to identify the signal mutation point (such as the moment of grasping contact) to calculate the deviation of the mutation time and amplitude.

[0102] Further as an optional implementation, the signal amplitude loss function is calculated by the dynamic time warping algorithm, and the formula is as follows:

[0103]

[0104] wherein, L amp represents the signal amplitude loss, Y represents the sample label, represents the predicted signal, and DTW(·,·) represents the dynamic time warping distance calculation.

[0105] The frequency loss function is calculated by the main frequency difference after Fourier transform, and the formula is as follows:

[0106]

[0107] wherein, L fre represents the frequency loss, FFT(·) represents the fast Fourier transform, f max represents the main frequency component, and ‖·‖ represents the L1 norm.

[0108] The transient feature loss function is calculated based on the time difference and amplitude difference of the signal mutation point, and the formula is as follows:

[0109]

[0110] wherein, L tra represents the transient feature loss, t i and A i respectively represent the i th mutation point time and amplitude of the sample label, and respectively represent the i th mutation point time and amplitude of the predicted signal, γ i represents the mutation point weight, λ represents the amplitude-time conversion coefficient, and i is a natural number and i ∈ [1, n].

[0111] Exemplarily, taking the workpiece grasping task performed by a humanoid robot as an example scenario, the compliant control needs to be realized through the end force signal, the input signal X includes the joint angle θ (t) and the motor current I (t), and the label signal Y is the end force F (t). As shown in Figure 4 The flowchart of the model training stage is shown in the figure, and the amplitude loss L amp , the frequency loss Lfre and transient loss L tra Among them, the amplitude loss L amp The end force F(t) and the predicted end force are calculated using the DTW algorithm (Dynamic Time Warping algorithm). The difference is obtained, and the specific formula is as follows:

[0112]

[0113] in, For Euclidean distance, w i,j For dynamic programming of path weights.

[0114] Frequency loss L fre By analyzing the end force F(t) and the predicted end force Perform a Fourier transform to extract the dominant frequency component f. max and f′ max The specific formula is as follows:

[0115] L fre =|f max -f′ max |+0.1·|A max -A′ max |

[0116] Among them, A max A′ max The amplitude corresponding to the dominant frequency.

[0117] Transient loss L tra By identifying the contact mutation point t c (in F(t)) and ( The formula is as follows:

[0118]

[0119] Step S1023: Based on the proportion of difference between the predicted signal and the sample label, the signal amplitude loss function, frequency loss function, and transient feature loss function are weighted and summed to obtain the combined loss function;

[0120] Specifically, the dynamically weighted combined loss function is used to dynamically adjust the weights (α, β, γ) according to the difference ratio of the three types of loss, and then optimize the model parameters through backpropagation until the loss converges.

[0121] As an optional implementation, step S1023 can be further divided into the following steps S10231 to S10234:

[0122] Step S10231: Set the difference percentage threshold;

[0123] Step S10232, calculate the amplitude difference proportion of the predicted signal and the sample label, when the amplitude difference proportion exceeds the difference proportion threshold, obtain a first weight coefficient combination, and then according to the first weight coefficient combination, weight and sum the signal amplitude loss function, the frequency loss function and the transient feature loss function to obtain a first combined loss function;

[0124] Step S10233, calculate the frequency difference proportion of the predicted signal and the sample label, when the frequency difference proportion exceeds the difference proportion threshold, obtain a second weight coefficient combination, and then according to the second weight coefficient combination, weight and sum the signal amplitude loss function, the frequency loss function and the transient feature loss function to obtain a second combined loss function;

[0125] Step S10234, calculate the transient feature difference proportion of the predicted signal and the sample label, when the transient feature difference proportion exceeds the difference proportion threshold, obtain a third weight coefficient combination, and then according to the third weight coefficient combination, weight and sum the signal amplitude loss function, the frequency loss function and the transient feature loss function to obtain a third combined loss function.

[0126] Step S1024, train the heterogeneous neural network fusion model by the combined loss function to obtain a signal prediction model;

[0127] Specifically, the dynamically weighted combined loss function satisfies:

[0128] L = aL amp + bL fre + gL tra

[0129] Wherein, L amp is the amplitude loss, L fre is the frequency loss, L tra is the transient loss, a, b, g are weight coefficients, and a+b+g = 1.

[0130] Specifically, the difference proportion threshold, the first weight coefficient combination, the second weight coefficient combination and the third weight coefficient combination can be set according to actual needs. In the embodiment of the present application, the difference proportion threshold is set to 50%, when the amplitude difference proportion of the predicted signal and the sample label exceeds 50%, the first weight coefficient combination is set to a = 0.5, b = 0.3, g = 0.2; when the frequency difference proportion exceeds 50%, the second weight coefficient combination is set to a = 0.2, b = 0.5, g = 0.3; when the transient feature difference proportion exceeds 50%, the third weight coefficient combination is set to a = 0.3, b = 0.2, g = 0.5.

[0131] Exemplarily, taking the workpiece grabbing task performed by the humanoid robot as an example, compliant control needs to be realized through the end force signal, the input signal X includes the joint angle θ(t) and the motor current I(t); and the label signal Y is the end force F(t). As shown in Figure 4 the combined loss function is: L = aL amp + bL fre + gL tra , wherein the initial values of a, b and g are 0.4, 0.3 and 0.3; when L amp / L>0.5, a is adjusted to 0.5, b is adjusted to 0.3, and g is adjusted to 0.2, and the rest is similar. Then, the Adam optimizer (learning rate 0.001) is adopted, 100 rounds of iteration are set, the batch size is set to 32, the training is stopped when the loss of the verification set converges to below 0.5N, and the trained signal prediction model is obtained.

[0132] In step S103, the current sensor signal of the humanoid robot is acquired, and the current sensor signal includes at least two of the joint angle signal, the motor current signal or the acceleration signal.

[0133] In step S104, the current sensor signal is input into the signal prediction model, and a predicted sensor signal is output, the predicted sensor signal being a force signal or a torque signal.

[0134] Specifically, as shown in Figure 5 the flowchart of the signal prediction stage, at least two of the current joint angle signal, the motor current signal or the acceleration signal of the humanoid robot (i.e., the first sensor signal and the second sensor signal) are acquired first, the trained signal prediction model is input, and the predicted end effector force signal or torque signal (i.e., the third sensor signal) is output, so as to realize force control or torque control without additional force sensor or torque sensor.

[0135] Exemplarily, taking the workpiece grabbing task performed by the humanoid robot as an example, compliant control needs to be realized through the end force signal, the input signal X includes the joint angle θ(t) and the motor current I(t); and the label signal Y is the end force F(t). In the actual grabbing process, the joint angle θ(t) and the motor current I(t) are collected in real time, the trained signal prediction model is input, and the predicted force is output. The predicted force is used in the impedance control algorithm to realize compliant grabbing, and the force control accuracy can reach ±2N (meeting the requirements of industrial assembly).

[0136] The above describes the sensor signal prediction method for the humanoid robot according to the embodiments of the present application. It can be recognized that, compared with the existing signal prediction method, the embodiments of the present application have the following advantages:

[0137] I. Hardware cost reduction: Precise prediction of key signals based on existing sensor signals through trained signal prediction model can reduce 30%-50% of precise sensor deployment, reduce sensor failure risk, simplify wiring and maintenance process, while ensuring control accuracy;

[0138] II. Multi-domain feature fusion: The original sensor signals are processed in time domain alignment, frequency domain feature enhancement, transient peak smoothing, and the signal prediction model is trained through the constructed signal amplitude loss function, frequency loss function and transient feature loss function, which can simultaneously learn the time domain amplitude, frequency domain period and transient features of the sensor signals, fully exploit the correlation of the sensor signals in time domain, frequency domain and transient features, and improve the prediction accuracy of the model;

[0139] II. Dynamic weight adjustment: According to the prediction deviation type, the loss function is adaptively optimized, which can enhance the learning ability of the model to key features and improve the prediction accuracy of the model.

[0140] Reference Figure 6 The embodiment of the present application also provides a sensor signal prediction system for a humanoid robot, comprising:

[0141] A training sample acquisition module is configured to acquire multi-source sensor training samples of the humanoid robot, wherein the multi-source sensor training samples include joint angle signals, motor current signals, acceleration signals, force sense signals and torque signals.

[0142] A model training module is configured to train a preset heterogeneous neural network fusion model according to the multi-source sensor training samples through a preset signal amplitude loss function, a frequency loss function and a transient feature loss function, and obtain a signal prediction model.

[0143] A current signal acquisition module is configured to acquire current sensor signals of the humanoid robot, wherein the current sensor signals include at least two of joint angle signals, motor current signals or acceleration signals.

[0144] A signal prediction module is configured to input the current sensor signals into the signal prediction model and output predicted sensor signals, wherein the predicted sensor signals are force sense signals or torque signals.

[0145] It can be understood that the contents in the above method embodiments are applicable to the present system embodiment, the present system embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0146] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to realize the method described above. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0147] It can be understood that the contents in the method embodiments described above are applicable to the device embodiments. The device embodiments specifically realize the functions of the method embodiments described above, and achieve the same beneficial effects as the method embodiments described above.

[0148] Please refer to Figure 7 , Figure 7 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0149] The processor 1001 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0150] The memory 1002 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 1002 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to realize the method described above.

[0151] The input / output interface 1003 is used to realize information input and output.

[0152] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0153] The bus 1005 is used to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device.

[0154] The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are communicatively connected with each other through the bus 1005.

[0155] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method.

[0156] It can be understood that the contents in the method embodiments are applicable to the storage medium embodiment, the storage medium embodiment specifically realizes the functions of the method embodiments, and achieves the same beneficial effects as the method embodiments.

[0157] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the method.

[0158] It can be understood that the contents in the method embodiments are applicable to the program product embodiment, the program product embodiment specifically realizes the functions of the method embodiments, and achieves the same beneficial effects as the method embodiments.

[0159] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0160] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0161] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0162] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0163] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0164] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, without departing from the scope of the application, the embodiments described herein can be carried out in a different order than the one illustrated or described herein. In addition, the terms "comprising", "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that comprises a list of steps or units not necessarily limited to those explicitly listed, but can include other not expressly listed or inherent to such processes, methods, products or apparatus.

[0165] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0167] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they can be located in one place, or can be distributed to a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0168] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0169] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0170] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A sensor signal prediction method for a humanoid robot, characterized by, The method comprises the following steps: obtain a multi-source sensor training sample of a humanoid robot, the multi-source sensor training sample comprising joint angle signals, motor current signals, acceleration signals, force sense signals, and torque signals; train a preset heterogeneous neural network fusion model according to the multi-source sensor training sample through preset signal amplitude loss functions, frequency loss functions, and transient characteristic loss functions, to obtain a signal prediction model; obtain a current sensor signal of the humanoid robot, the current sensor signal comprising at least two of the joint angle signals, the motor current signals, or the acceleration signals; input the current sensor signal into the signal prediction model, and output a predicted sensor signal, the predicted sensor signal being the force sense signal or the torque signal.

2. The method of claim 1, wherein, The method of obtaining the multi-source sensor training sample of the humanoid robot specifically comprises: collecting a plurality of original sensor signals; performing noise reduction processing on each of the original sensor signals through an improved variational mode decomposition algorithm; performing at least one or a combination of time domain alignment processing, frequency domain feature enhancement processing, or transient peak smoothing processing on the original sensor signals after noise reduction processing, to obtain the multi-source sensor training sample.

3. The method of claim 2, wherein, The method of performing noise reduction processing on each of the original sensor signals through the improved variational mode decomposition algorithm specifically comprises: obtaining a joint motion cycle of the humanoid robot, and determining a center frequency search range of an intrinsic mode function according to the joint motion cycle; decomposing the original sensor signals into a plurality of intrinsic mode functions according to the center frequency search range; calculating sample entropy of each of the intrinsic mode functions, and screening the intrinsic mode functions according to the sample entropy and a preset sample entropy threshold, to obtain the original sensor signals after noise reduction processing.

4. The method of claim 1, wherein, The heterogeneous neural network fusion model comprises at least two of: a bidirectional long short-term memory network for processing time sequence features; a convolutional neural network for extracting spatial correlation features; a gradient boosting decision tree network for processing high-dimensional table features.

5. The method of claim 1, wherein, The method of training the preset heterogeneous neural network fusion model according to the multi-source sensor training sample through the preset signal amplitude loss functions, the frequency loss functions, and the transient characteristic loss functions, to obtain the signal prediction model, specifically comprises: inputting the multi-source sensor training sample into the heterogeneous neural network fusion model to output a predicted signal, the multi-source sensor training sample comprising a sample label; constructing the signal amplitude loss function, the frequency loss function, and the transient characteristic loss function according to the predicted signal and the sample label; weighting and summing the signal amplitude loss function, the frequency loss function, and the transient characteristic loss function according to a difference proportion of the predicted signal and the sample label, to obtain a combined loss function; training the heterogeneous neural network fusion model through the combined loss function, to obtain the signal prediction model.

6. The method of claim 1, wherein, The signal amplitude loss function is calculated through a dynamic time warping algorithm, and a formula is as follows: wherein L amp represents signal amplitude loss, Y represents sample label, represents predicted signal, DTW(·, ·) represents dynamic time warping distance calculation; The frequency loss function is calculated by the difference of the main frequency after Fourier transform, and the formula is: where L fre represents the frequency loss, FFT(·) represents the fast Fourier transform, f max represents the main frequency component, and ||·|| represents the L1 norm; The transient feature loss function is calculated based on the time difference and amplitude difference of the signal mutation point, and the formula is: wherein L tra represents the transient feature loss, t i and A i respectively represent the i-th mutation point time and amplitude of the sample label, and respectively represent the i-th mutation point time and amplitude of the predicted signal, γ i represents the mutation point weight, λ represents the amplitude time conversion coefficient, i is a natural number and i ∈ [1, n].

7. The method of claim 5, wherein, The combination loss function is obtained by weighting and summing the signal amplitude loss function, the frequency loss function and the transient feature loss function according to the difference proportion of the predicted signal and the sample label, and specifically includes: setting a difference proportion threshold; calculating the amplitude difference proportion of the predicted signal and the sample label, and when the amplitude difference proportion exceeds the difference proportion threshold, a first weight combination is obtained, and then the signal amplitude loss function, the frequency loss function and the transient feature loss function are weighted and summed according to the first weight combination to obtain a first combination loss function; calculating the frequency difference proportion of the predicted signal and the sample label, and when the frequency difference proportion exceeds the difference proportion threshold, a second weight combination is obtained, and then the signal amplitude loss function, the frequency loss function and the transient feature loss function are weighted and summed according to the second weight combination to obtain a second combination loss function; calculating the transient feature difference proportion of the predicted signal and the sample label, and when the transient feature difference proportion exceeds the difference proportion threshold, a third weight combination is obtained, and then the signal amplitude loss function, the frequency loss function and the transient feature loss function are weighted and summed according to the third weight combination to obtain a third combination loss function.

8. A sensor signal prediction system for a humanoid robot, characterized by, including: a training sample acquisition module configured to acquire multi-source sensor training samples of a humanoid robot, the multi-source sensor training samples including joint angle signals, motor current signals, acceleration signals, force sense signals and torque signals; a model training module configured to train a preset heterogeneous neural network fusion model according to the multi-source sensor training samples by using a preset signal amplitude loss function, a frequency loss function and a transient feature loss function, and obtain a signal prediction model; a current signal acquisition module configured to acquire current sensor signals of the humanoid robot, the current sensor signals including at least two of the joint angle signals, the motor current signals or the acceleration signals; a signal prediction module configured to input the current sensor signals into the signal prediction model and output predicted sensor signals, the predicted sensor signals being the force sense signals or the torque signals.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1-7.

Images