Methods, systems, equipment, and storage media for recognizing the operating style of electric robots.

Abstract

This invention relates to the field of power robot operation style recognition technology, and discloses a method, system, device, and storage medium for power robot operation style recognition. The method includes acquiring multimodal data of the power robot, performing cross-modal reasoning to obtain the task; obtaining a standard motion trajectory based on the task; inputting acceleration data into a feature extraction model to obtain aggression features; performing wavelet packet decomposition and spectral energy entropy calculation on the mechanical data to obtain refinement features; calculating the Lyapunov exponent and root mean square deviation of the trajectory based on the standard motion trajectory and trajectory data to obtain stability features; inputting the aggression, refinement, and stability features into a style scoring model to obtain a three-dimensional style score; and obtaining the operation style of the power robot based on the three-dimensional style score. This invention can accurately identify operation styles, reduce human-machine interaction conflicts through real-time matching of operation styles, and thus improve the execution efficiency of power tasks.

Description

Technical Field

[0001] This invention relates to the field of power robot operation style recognition technology, and in particular to a method, system, device and storage medium for power robot operation style recognition. Background Technology

[0002] In recent years, the demand for teleoperated robots in scenarios such as power inspection, fault repair, and preventive maintenance has been increasing. Teleoperation is a technology that enables humans to control remote machines or systems in real time through interactive devices. Its core lies in synchronously transmitting the operator's actions to the remote device. However, existing technologies have significant limitations in intelligent understanding and style recognition of operational behavior.

[0003] Current analyses of teleoperated robots mostly focus on task classification, neglecting the impact of style features on robot control accuracy and human-robot collaboration efficiency. Existing pattern recognition systems largely rely on single indicators to characterize operating styles, failing to comprehensively reflect operational characteristics and lacking end-to-end coverage from task recognition to style judgment, resulting in low style recognition accuracy. Furthermore, existing pattern recognition systems rely on fixed thresholds, making it difficult to adjust in real time to adapt to changes in operating styles. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method, system, device, and storage medium for recognizing the operating style of a power robot, enabling intelligent understanding of the entire process from task recognition to style judgment in a remotely operated power robot, improving the accuracy of operating style recognition, and thus enhancing human-machine collaboration efficiency.

[0005] In a first aspect, the present invention provides a method for recognizing the operating style of an electric robot, the method comprising: Acquire multimodal data of the electric robot performing the task, perform cross-modal reasoning on the multimodal data, and obtain the task of the electric robot. Motion data is extracted from the multimodal data, and a standard motion trajectory is obtained according to the task. The motion data includes acceleration data, mechanical data, and trajectory data. The acceleration data is input into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Wavelet packet decomposition and spectral energy entropy calculation are performed on the mechanical data to obtain the fineness features; Based on the standard motion trajectory and the trajectory data, the Lyapunov exponent and the root mean square deviation of the trajectory are calculated to obtain the stability characteristics. The radicality feature, the fineness feature, and the stability feature are input into a preset style scoring model to obtain a three-dimensional style score. Based on the three-dimensional style score, the operating style of the electric robot is obtained. The style scoring model is constructed based on a neural network model.

[0006] Furthermore, the step of acquiring multimodal data of the electric robot performing the task, and performing cross-modal reasoning on the multimodal data to obtain the task of the electric robot includes: Acquire multimodal data of the electric robot performing the task, the multimodal data including image data and motion data; Feature extraction is performed on the multimodal data to obtain multimodal features, which include image features and motion features; The multimodal features are input into a preset task recognition model to perform cross-modal reasoning on the multimodal data to obtain the operation task of the power robot. The task recognition model is constructed based on a multimodal large model.

[0007] Furthermore, the feature extraction model includes a temporal feature extraction module, a local feature extraction module, and a feature fusion module; The temporal feature extraction module is constructed using a long short-term memory neural network, and the local feature extraction module is constructed using a convolutional neural network. The temporal feature extraction module and the local feature extraction module are connected in parallel and then connected in series with the feature fusion module.

[0008] Further, the step of inputting the acceleration data into a preset feature extraction model to obtain the aggression feature includes: The acceleration data is input into the time-series feature extraction module to predict the sudden stop trend and obtain the time-series dynamic features. The acceleration data is input into the local feature extraction module to identify acceleration peaks and obtain local abrupt change features. The temporal dynamic features and the local mutation features are input into the feature fusion module for feature fusion, and the mean of the fused features is calculated using a sliding window mechanism to obtain the radicality features.

[0009] Furthermore, the step of performing wavelet packet decomposition and spectral energy entropy calculation on the mechanical data to obtain the fine-grained features includes: The mechanical data is preprocessed and decomposed into wavelet packets to obtain several sub-bands and corresponding wavelet coefficients. Based on the wavelet coefficients, calculate the energy of each sub-band and the total energy of all sub-bands; Based on the energy and the total energy, the spectral energy entropy is calculated, and the spectral energy entropy is used as a fineness feature.

[0010] Further, the step of calculating the Lyapunov exponent and root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data to obtain the stability characteristics includes: The trajectory data is reconstructed in phase space to obtain the Lyapunov exponent; Calculate the root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data; Based on the Lyapunov exponent and the root mean square deviation of the trajectory, a nonlinear function is used to obtain the stability characteristics.

[0011] Furthermore, the style scoring model includes a cascaded encoding module, a global feature fusion module, and a spatial projection module, wherein the encoding module is constructed using a Transformer encoder; The encoding module is used to globally model the radicality feature, the fineness feature, and the stability feature to obtain the hidden state sequence; The global feature fusion module is used to perform global average pooling on the hidden state sequence to obtain a global feature vector; The spatial projection module is used to perform projection matrix mapping and value range constraints on the global feature vector to obtain a three-dimensional style score.

[0012] Secondly, the present invention provides a power robot operation style recognition system, the system comprising: The cross-modal reasoning module is used to acquire multimodal data of the electric robot performing the task, and to perform cross-modal reasoning on the multimodal data to obtain the task of the electric robot. The operation feature extraction module is used to extract motion data from the multimodal data and obtain a standard motion trajectory according to the task. The motion data includes acceleration data, mechanical data and trajectory data. The acceleration data is input into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Wavelet packet decomposition and spectral energy entropy calculation are performed on the mechanical data to obtain the fineness features; Based on the standard motion trajectory and the trajectory data, the Lyapunov exponent and the root mean square deviation of the trajectory are calculated to obtain the stability characteristics. The operation style recognition module is used to input the aggression feature, the fineness feature and the stability feature into a preset style scoring model to obtain a three-dimensional style score, and to obtain the operation style of the electric robot based on the three-dimensional style score. The style scoring model is constructed based on a neural network model.

[0013] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0015] This invention provides a method, system, device, and storage medium for recognizing the operating style of power robots. Through multimodal fusion and inference, this invention can accurately identify the operational tasks of power robots. Through multi-dimensional style analysis, it can achieve accurate quantification and real-time matching of three-dimensional styles, improving the accuracy of operating style recognition. This invention enables real-time matching of operating styles, reduces human-machine interaction conflicts, and thus improves the execution efficiency of power tasks. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the electric robot operation style recognition method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the electric robot operation style recognition system in an embodiment of the present invention; Figure 3 This is an internal structural diagram of the computer device in an embodiment of the present invention.

[0017] Figure label: 10. Cross-modal reasoning module; 20. Operation feature extraction module; 30. Operation style recognition module. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1The first embodiment of the present invention proposes a method for recognizing the operating style of an electric robot, including steps S10 to S1: Step S10: Obtain multimodal data of the electric robot performing the task, and perform cross-modal reasoning on the multimodal data to obtain the task of the electric robot. Step S20: Extract motion data from the multimodal data and obtain a standard motion trajectory according to the task. The motion data includes acceleration data, mechanical data, and trajectory data. Step S30: Input the acceleration data into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Step S40: Perform wavelet packet decomposition and spectral energy entropy calculation on the mechanical data to obtain the fineness features; Step S50: Based on the standard motion trajectory and the trajectory data, calculate the Lyapunov exponent and the root mean square deviation of the trajectory to obtain the stability characteristics; Step S60: Input the radicality feature, the fineness feature and the stability feature into a preset style scoring model to obtain a three-dimensional style score, and obtain the operating style of the electric robot based on the three-dimensional style score. The style scoring model is constructed based on a neural network model.

[0020] This invention primarily targets the intelligent recognition of operator operating styles when remotely operated robots perform power-related tasks. Firstly, it uses a pre-set sensor array to perceive and collect multimodal data. In a preferred embodiment, the sensor array comprises a heterogeneous sensor network consisting of a 9-axis Inertial Measurement Unit (IMU) (1000Hz sampling), a 6-dimensional force / torque sensor (±200N range), and a binocular RGB-D camera (1280×720@60FPS). The IMU can be installed at the axial center of each joint drive unit of the robot's arm and near the end effector. The force / torque sensor is installed at the connection surface between the end effector and the flange, and at the output end of the wrist joint harmonic reducer. When the camera is installed, the binocular baseline is parallel to the Y-axis of the tool coordinate system, and it is equipped with an adjustable pitch mechanism, a ring LED array, and a polarizing filter illumination system. In addition, a spatiotemporal synchronization mechanism was added during multimodal data acquisition. The IEEE 1588 precision clock protocol was used to achieve μs-level alignment of multi-source data and construct a spatiotemporally unified operation flow. Image data, mechanical data, and IMU data (acceleration data and trajectory data) were recorded at the same time point to achieve effective correlation of multimodal data.

[0021] Then, cross-modal reasoning is performed on the collected multimodal data to identify the work tasks of the power robot. Specifically, the multimodal data of the work tasks are autonomously identified and labeled based on a large model. Here, the total work tasks are classified into three levels: scenario, task, and action. Scenarios include environmental type scenarios (such as substations / transmission lines / distribution rooms), working condition scenarios (such as normal inspections / fault repairs / preventive maintenance), and micro-environmental scenarios (such as damp surfaces / electromagnetic interference areas / confined spaces). Tasks are the work tasks that the power robot can perform in the current environment (such as live-line connection, replacing surge arresters, etc.). Actions are the atomic actions of the power robot (such as wire grasping, bolt tightening, etc.). Each work task is composed of multiple atomic actions in a certain time sequence.

[0022] Based on the above task classification, three main factors are identified through a multimodal large model. The reasoning and decision-making capabilities of this large model can then be used to analyze and extract structured labels for each task. For multimodal data, different feature extraction methods are employed. Preferably, image data can be extracted using image recognition techniques such as Convolutional Neural Networks (CNNs) to identify the feature codes of objects or scenes in the environment as image features. Mechanical data can be obtained through mechanical analysis, acquiring data such as the magnitude, direction, and change of force, and calculating statistical data such as mean and variance as mechanical features. IMU data includes acceleration data, angular velocity data, and trajectory data obtained through trajectory calculation. In this embodiment, these IMU data are used as trajectory features, and the mechanical and trajectory features are used as motion features. Then, these extracted multimodal features are input into a task recognition model built based on a large model (such as the CLIP-ViT model) for cross-modal reasoning to obtain the structured label of the task currently being performed by the power robot, such as a substation scenario—replacing a surge arrester—grabbing a conductor action. It should be noted that the task recognition model in this embodiment can be constructed using a conventional multimodal large model. Its reasoning process can refer to the reasoning process of the model used. Furthermore, the corresponding multimodal features are different for different task types. Therefore, the selection of multimodal features should be flexibly chosen based on the actual situation. No specific limitations are imposed on multimodal features and large models here.

[0023] In a preferred embodiment, to improve the dynamic adaptability of the task recognition model to meet the needs of different operator styles and complex scenarios, the present invention employs an autonomous evolution system combining a dynamic feature memory and a neural Turing machine to update the atomic action database used for training and inference of the task recognition model in real time. The steps include: The task recognition model is trained by a pre-set scene library and an atomic action library. The atomic action library includes a task type library and a corresponding feature library, and the atomic action library is stored using a dynamic feature memory library. The sensor data of the electric robot when performing tasks is recorded by a neural Turing machine. According to the feature dimension corresponding to the task category, the sensor data is calculated by a preset algorithm to obtain specific feature values. The calculated feature values ​​are categorized according to task type, the feature datasets for each task type are statistically analyzed, and task identification is performed based on the statistical results. The identification results include new feature points and operational anomalies. Add the new feature points to the corresponding feature library, update the feature data in the feature library, and delete the corresponding sensor data after the analysis is completed. The updated atomic action library is used to incrementally train the task recognition model.

[0024] In this embodiment, the scene library consists of different multimodal features and corresponding scene labels. The atomic action database also contains different atomic action labels and corresponding multimodal features. For example, the multimodal features are 5-7N continuous mechanical pressure, image line diameter, and circular motion at the end of the trajectory. The atomic action label is wire grasping. The model is trained based on the data in the database.

[0025] Preferably, the atomic action database adopts a dynamic feature memory (DMB). When constructing the DMB, the operation task classification system is first preset (such as screwing, wiring, grabbing wires, etc.), and the dimension / type of key features is preset for each category (such as screwing: force / torque mode; wiring: image features). The updating of the atomic action database adopts an autonomous learning mechanism that combines the dynamic feature memory (DMB) and the neural Turing machine (NTM).

[0026] Specifically, taking the screw-tightening task as an example, the "screw-tightening feature library" includes the following features: Force / Torque Features: The curves of the axial force applied by the screwdriver over time and the curves of the torque over time. These are the most crucial features. Speed ​​Features: The average speed of the screwdriver rotation and the range of speed variation. Path Features: The initial position deviation (small) when the screwdriver enters the screw head and the path stability during rotation. Stripping Features (Anomaly Detection): The statistical probability of abnormal fluctuations in torque or force (repeatedly rising and falling sharply in a short period of time) (used to determine the risk of stripping). Positioning Features: The time and / or number of turns required for the torque to reach a set threshold (indicating tightening). Differences in Features for Tightening Different Screws: For known different screw types (such as wood screws and metal self-tapping screws), the system learns their respective typical force / torque curve profiles. In short, the "screw-tightening" task should focus on force / torque, speed, and positioning signals.

[0027] The Neural Turing Machine (NTM) begins recording a stream of sensor data (such as torque change curves and image sequences) related to a task at the start of the task (or upon reaching a critical step). These are continuous, fine-grained, raw "operation logs." Records typically include timestamps and task type ID tags (e.g., Task_Tightening_Timestamp). For example, in a tightening screw task, the NTM records joint encoder data (position, velocity) throughout the entire tightening action, force / torque sensor readings (time series), high-frame-rate camera image / point cloud sequences before and after critical time points (e.g., the instant the screwdriver contacts the screw head, keyframes during the tightening process), and raw sampled data of the torque / force curve throughout the entire process.

[0028] The analysis algorithm in DMB periodically analyzes the data collected by NTM, examining all operation segments recorded by NTM over a past period (e.g., a day). For each recorded segment (corresponding to one actual operation), the algorithm extracts, classifies, aggregates, and statistically analyzes features according to preset feature dimensions for that task category (e.g., screw tightening). This identifies key changes or new patterns. Specifically, the algorithm calculates specific feature values ​​from the raw data (e.g., average force, maximum torque, speed variance, and whether there is a stripped thread peak in this screw tightening process). The calculated feature values ​​are categorized by task type (e.g., all screw tightening feature values ​​are clustered together). Then, statistical methods are applied to the aggregated feature datasets corresponding to each task type (e.g., for screw tightening tasks, analyzing the distribution of all historical "average force" values ​​to form a probability density function). If the "average force" value of a new operation falls into the high-probability region of this distribution, it is considered a "routine operation, consistent with existing knowledge"; if it falls into the low-probability region (anomaly), it may indicate a new situation (e.g., new screw material), operational error, or environmental change. Based on the statistical analysis results, it identifies which segments represent truly valuable new experiences (significant, reusable patterns) or operational anomalies.

[0029] Then, the DMB is updated through knowledge solidification, incorporating analyzed, valuable, and new feature data points (such as the successful tightening feature curve of a new type of screw) into the feature library of the corresponding category. The screw-tightening distribution model is updated with the new data points. Furthermore, if clusters of new features are consistently found to be significantly separated from existing clusters (e.g., the trend of completely different force patterns between plastic and metal screws becomes stable), a new subcategory may be created (e.g., a sub-library for "plastic screw tightening"). If the boundaries between two existing categories are blurred, they will be merged. Simultaneously, outdated or redundant feature data is removed (to prevent bloat). It is identified which information is trivial (e.g., a slight deviation in visual positioning due to temporary occlusion that does not affect the result), and this information is not solidified into the DMB.

[0030] After knowledge solidification is complete, the NTM is maintained. This means that after analysis and solidification, the value of the NTM segment for that time period is considered extracted, and its original content can be safely overwritten to make room for new real-time recordings. The task recognition model is then periodically incrementally trained using updated DMBs to improve its adaptability to new operational styles and environments.

[0031] After identifying the current task being performed by the teleoperated electric robot, the operator's operating style is identified through further analysis of its motion data. The motion data used in this embodiment includes acceleration data, mechanical data, and trajectory data. This motion data is used to analyze style characteristics across different dimensions. Acceleration data is used to analyze the operator's aggressiveness, mechanical data to analyze the operator's precision, and trajectory data to analyze the operator's stability. Specifically, for acceleration data, this embodiment employs a feature extraction model to extract aggressiveness features.

[0032] The feature extraction model in this embodiment adopts a cascaded spatiotemporal feature reconstruction structure, including a temporal feature extraction module, a local feature extraction module, and a feature fusion module. The temporal feature extraction module and the local feature extraction module are connected in parallel and then connected in series with the feature fusion module. The temporal feature extraction module is constructed using a long short-term memory neural network, and the local feature extraction module is constructed using a convolutional neural network. The feature extraction model extracts features from the input acceleration data to obtain the radicality features. The specific steps include: The acceleration data is input into the time-series feature extraction module to predict the sudden stop trend and obtain the time-series dynamic features. The acceleration data is input into the local feature extraction module to identify acceleration peaks and obtain local abrupt change features. The temporal dynamic features and the local mutation features are input into the feature fusion module for feature fusion, and the mean of the fused features is calculated using a sliding window mechanism to obtain the radicality features.

[0033] In this embodiment, the aggressive behavior is mainly characterized by transient abrupt changes, high-frequency oscillations, and instability. Transient abrupt changes refer to sharp jumps in the acceleration signal of the control lever / pedal, such as sudden starts / stops. High-frequency oscillations refer to abnormal high-frequency components in the signal caused by rapid reciprocating operations. Inconsistent stability refers to the alternation of stable and aggressive operating segments, with frequent state switching. Traditional single models cannot accurately identify these aggressive characteristics; therefore, this embodiment adopts a cascaded structure, using a temporal feature extraction module and a local feature extraction module for parallel processing to achieve dual-path collaborative detection, and then fusing the features from both paths.

[0034] Specifically, the temporal feature extraction module is constructed using a Long Short-Term Memory (LSMT) neural network, preferably a two-layer bidirectional LSTM with a hidden layer dimension of 64 to process historical N-step acceleration data. The output predictive probability vector for each time step is also the temporal dynamic feature, such as the "accelerated stop probability" at time t. The local feature extraction module is constructed using the convolutional layers of a Convolutional Neural Network (CNN). The convolutional layers detect local abrupt changes at each time step, such as the acceleration peak at time t, and represent them as feature values. Then, the temporal dynamic features and local abrupt changes are input into the feature fusion module for feature fusion. During feature fusion, a dynamic weighted fusion method can be used, multiplying the predictive probability vector output by the LSTM with the local feature vector output by the CNN element-wise. By adjusting the weights of each local feature, feature fusion is achieved.

[0035] For the fused features, a sliding window mechanism was used to calculate the mean of the fused feature values ​​to obtain the radicality feature: In the formula, aggr represents the aggression feature, T represents the time window length, which is the number of time steps contained in the time sequence data segment, N represents the CNN output feature dimension, reflecting the number of local features extracted from the acceleration signal, and OUT(t,n) represents the fused feature value of the i-th channel at the t-th time step.

[0036] A single CNN can capture "instantaneous spikes" (spatial local features) but cannot perceive historical trends, while a single LSTM can capture "continuous trends" (temporal dynamic features) but cannot detect instantaneous abrupt changes. Traditional serial modes require waiting for output transmission, resulting in response delays; while traditional parallel modes do not interact after parallelization, leading to high false positive rates. This embodiment solves the limitations of single modalities by cascading temporal-guided spatial interactions. After feature fusion, it retains the instantaneity of spikes and amplifies the feature weights of aggressive operations through historical trends, thus complementing temporal and spatial features and enhancing the model's expressive power.

[0037] This embodiment obtains fine-grained features by performing wavelet packet decomposition and spectral energy entropy calculation on mechanical data. The specific steps include: The mechanical data is preprocessed and decomposed into wavelet packets to obtain several sub-bands and corresponding wavelet coefficients. Based on the wavelet coefficients, calculate the energy of each sub-band and the total energy of all sub-bands; Based on the energy and the total energy, the spectral energy entropy is calculated, and the spectral energy entropy is used as a fineness feature.

[0038] In this embodiment, mechanical data, such as 6-dimensional mechanical signals (axial force along the X / Y / Z axes + torque around the X / Y / Z axes), is collected by mechanical sensors. Wavelet packet decomposition is performed on each dimension of each type of physical quantity to calculate the sub-band energy. Specifically, the 6-dimensional signals of linear force (F_x / F_y / F_z) and rotational torque (M_x / M_y / M_z) undergo data preprocessing, including denoising and normalization. db4 wavelet soft thresholding denoising can be used to eliminate electromagnetic interference from the substation (such as high-frequency spikes). Then, the amplitude of each dimension signal is normalized to eliminate magnitude differences. Wavelet packet decomposition is performed on the preprocessed single-dimensional data, preferably using db4 wavelet basis decomposition with 32 layers, resulting in 32 sub-bands and a wavelet coefficient matrix. Then, the sub-band energy is calculated, with the energy E of the d-th dimension and k-th sub-band being calculated. d,k for: In the formula, d is the dimension. wavelet_coeffs d,k,i It is the i-th wavelet coefficient of the k-th sub-band in the d-th dimension, and M represents the total number of wavelet coefficients.

[0039] The energy of a sub-band reflects the energy intensity of the mechanical signal in that band. The total energy in that dimension is obtained by summing the energies of all sub-bands, and then the frequency energy entropy H in dimension d is calculated. d : In the formula, K represents the total number of sub-bands. E d,total This represents the total energy of the d-th dimension.

[0040] If the mechanical signal is one-dimensional data, the corresponding spectral energy entropy can be calculated through the above steps. If multi-dimensional data is used, different weight values ​​can be set according to different tasks (for example, the torque dimension is more important when tightening bolts, while the weights of each dimension are the same in other general scenarios). Then, the multi-dimensional frequency energy entropy is weighted and summed to obtain the final frequency energy entropy, which is then used as a fineness feature.

[0041] In a preferred embodiment, the present invention characterizes stability features by calculating the Lyapunov exponent and root mean square deviation of trajectory data, specifically including the following steps: The trajectory data is reconstructed in phase space to obtain the Lyapunov exponent; Calculate the root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data; Based on the Lyapunov exponent and the root mean square deviation of the trajectory, a nonlinear function is used to obtain the stability characteristics.

[0042] In this embodiment, the trajectory data is first reconstructed in phase space to calculate the Lyapunov exponent, which characterizes the robot's dynamic stability. Then, a standard motion trajectory is extracted based on the identified task, and the root mean square deviation (RMSD) between the standard motion trajectory and the trajectory data is calculated. The RMSD reflects the trajectory tracking accuracy. The specific calculation process can refer to the conventional calculation process of the Lyapunov exponent and RMSD, and will not be elaborated here.

[0043] To better characterize the stability features of operators when manipulating the robot, this embodiment employs a synthetic stability index, quantifying the Lyapunov exponent and root mean square deviation of the trajectory as two separate indices. The quantization calculation utilizes a nonlinear function; by inputting the Lyapunov exponent and root mean square deviation of the trajectory into the nonlinear function, the stability feature Stab can be obtained. In the formula, λ represents the Lyapunov exponent, α represents the preset attenuation coefficient, which is used to control the weight of the influence of the root mean square deviation on stability. The preferred value range of α is [0.5, 1.2], and RMSE represents the root mean square deviation of the trajectory.

[0044] Then, the aggressiveness, refinement, and stability features are input into a pre-defined style scoring model to score the operational style. Based on the obtained operational style scores, the operational style of the electric robot is determined. The style scoring model can be constructed using a neural network model, such as a deep convolutional neural network or a multilayer perceptron model.

[0045] In a preferred embodiment, the present invention constructs a style scoring model based on the Transformer architecture. The style scoring model consists of an encoding module, a global feature fusion module, and a spatial projection module connected in series. The encoding module comprises a Transformer encoder, the global feature fusion module comprises a global average pooling layer, and the spatial projection module comprises a projection mapping layer and an activation function. Specifically, aggressiveness features, fineness features, and stability features are input into the encoding module. The self-attention mechanism of the Transformer encoder is used to globally model the dynamic relationships between features, outputting a sequence of hidden states. The hidden state sequence output by the Transformer encoder is input into the global feature fusion module. The global average pooling layer averages the hidden states at all time steps, integrating the global features of the entire operation to obtain a global feature vector, which is then input into the spatial projection module. The global feature vector can be represented as: In the formula, Represents the global feature vector. Indicates the first lThe hidden state vectors of time steps, with dimensions equal to the hidden state dimensions of the Transformer encoder, where L represents the total number of time steps.

[0046] Within the spatial projection module, the projection matrix W is first used... proj By performing a projection matrix mapping on the global feature vectors, the high-dimensional global feature vectors are mapped to a three-dimensional style space, resulting in three-dimensional feature vectors. : Then, the sigmoid activation function is used to constrain each element of the 3D feature vector to the interval [0,1], resulting in the final 3D style score: In the formula, σ(*) represents the sigmoid function, S aggr S represents the degree of aggression. fine S represents the refinement score. stab This indicates the stability score.

[0047] Finally, based on the 3D style score, the operator's operating style for the power robot is determined. Preferably, there are four operating styles: aggressive, refined, stable, and balanced. The operating style can be determined by preset scoring criteria. Specifically, these criteria include: if the highest score in the 3D style score is an aggressive score and the aggressive score is greater than a threshold score (e.g., 0.7), the operating style is considered aggressive; if the highest score in the 3D style score is a refined score and the refined score is greater than a threshold score (e.g., 0.7), the operating style is considered refined; if the highest score in the 3D style score is a stable score and the stable score is greater than a threshold score (e.g., 0.7), the operating style is considered stable; if none of the above three conditions are met, the operating style is considered balanced. It should be noted that each operator's operating style is not unique; an operator of a power robot can have multiple operating styles coexisting. For example, if there are two highest scores, both greater than the threshold, then there are two operating style types.

[0048] After obtaining the operating style of the electric robot operator, the robot's control parameters can be optimized based on the operating style. For example, the torque tolerance can be increased for electric robots with an aggressive operating style to achieve personalized robot control. Personnel skills can also be assessed based on the operating style to generate operation optimization suggestions. For example, for robots with a delicate operating style, redundant actions can be reduced. In addition, operators with suitable operating styles can be assigned to specific tasks to improve human-robot collaboration efficiency and task execution efficiency.

[0049] This embodiment provides a method for recognizing the operation style of a power robot. Through multimodal fusion and inference based on a large model, it can accurately identify the operation tasks of the power robot. By analyzing the style in multiple dimensions, it can solve the problem of insufficient representation by traditional single indicators, achieve accurate quantification and real-time matching of three-dimensional styles, and improve the accuracy of operation style recognition. This embodiment can match operation styles in real time, reduce human-computer interaction conflicts, and thus improve the execution efficiency of power tasks.

[0050] Please see Figure 2 Based on the same inventive concept, the second embodiment of the present invention proposes a power robot operation style recognition system, comprising: The cross-modal reasoning module 10 is used to acquire multimodal data of the electric robot performing the task, and perform cross-modal reasoning on the multimodal data to obtain the task of the electric robot. The operation feature extraction module 20 is used to extract motion data from the multimodal data and obtain a standard motion trajectory according to the task. The motion data includes acceleration data, mechanical data and trajectory data. The acceleration data is input into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Wavelet packet decomposition and spectral energy entropy calculation are performed on the mechanical data to obtain the fineness features; Based on the standard motion trajectory and the trajectory data, the Lyapunov exponent and the root mean square deviation of the trajectory are calculated to obtain the stability characteristics. The operation style recognition module 30 is used to input the aggression feature, the fineness feature and the stability feature into a preset style scoring model to obtain a three-dimensional style score, and to obtain the operation style of the electric robot based on the three-dimensional style score. The style scoring model is constructed based on a neural network model.

[0051] The technical features and effects of the electric robot operation style recognition system proposed in this embodiment of the invention are the same as those of the method proposed in this embodiment of the invention, and will not be repeated here. Each module in the above-mentioned electric robot operation style recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0052] Furthermore, embodiments of the present invention also propose a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0053] Please see Figure 3 The diagram illustrates the internal structure of a computer device in one embodiment. This computer device can specifically be a terminal or a server. The computer device includes a processor, memory, network interface, display, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for recognizing the operating style of an electric robot. The display screen of the computer device can be a liquid crystal display (LCD) or an e-ink display. The input devices of the computer device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse, etc.

[0054] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computing devices may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0055] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0056] In summary, the present invention provides a method, system, device, and storage medium for recognizing the operating style of an electric robot. The method acquires multimodal data of an electric robot performing a task, performs cross-modal reasoning on the multimodal data to obtain the robot's task; extracts motion data from the multimodal data and, based on the task, obtains a standard motion trajectory, the motion data including acceleration data, mechanical data, and trajectory data; inputs the acceleration data into a preset feature extraction model to obtain an aggression feature, the feature extraction model being constructed based on a long short-term memory neural network and a convolutional neural network; performs wavelet packet decomposition and spectral energy entropy calculation on the mechanical data to obtain a fineness feature; calculates the Lyapunov exponent and root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data to obtain a stability feature; inputs the aggression feature, the fineness feature, and the stability feature into a preset style scoring model to obtain a three-dimensional style score, and obtains the operating style of the electric robot based on the three-dimensional style score, the style scoring model being constructed based on a neural network model. This invention, through multimodal fusion and reasoning, can accurately identify the tasks of power robots. Through multi-dimensional style analysis, it can achieve accurate quantification and real-time matching of three-dimensional styles, improving the accuracy of operation style recognition. This invention can achieve real-time matching of operation styles, reduce human-computer interaction conflicts, and thus improve the execution efficiency of power tasks.

[0057] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0058] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A method for recognizing the operating style of an electric robot, characterized in that, include: Acquire multimodal data of the electric robot performing the task, perform cross-modal reasoning on the multimodal data, and obtain the task of the electric robot. Motion data is extracted from the multimodal data, and a standard motion trajectory is obtained according to the task. The motion data includes acceleration data, mechanical data, and trajectory data. The acceleration data is input into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Wavelet packet decomposition and spectral energy entropy calculation are performed on the mechanical data to obtain the fineness features; Based on the standard motion trajectory and the trajectory data, the Lyapunov exponent and the root mean square deviation of the trajectory are calculated to obtain the stability characteristics. The radicality feature, the fineness feature, and the stability feature are input into a preset style scoring model to obtain a three-dimensional style score. Based on the three-dimensional style score, the operating style of the electric robot is obtained. The style scoring model is constructed based on a neural network model.

2. The method for recognizing the operating style of a power robot according to claim 1, characterized in that, The steps of acquiring multimodal data of the electric robot performing the task, and performing cross-modal reasoning on the multimodal data to obtain the task of the electric robot include: Acquire multimodal data of the electric robot performing the task, the multimodal data including image data and motion data; Feature extraction is performed on the multimodal data to obtain multimodal features, which include image features and motion features; The multimodal features are input into a preset task recognition model to perform cross-modal reasoning on the multimodal data to obtain the operation task of the power robot. The task recognition model is constructed based on a multimodal large model.

3. The method for recognizing the operating style of a power robot according to claim 1, characterized in that, The feature extraction model includes a temporal feature extraction module, a local feature extraction module, and a feature fusion module; The temporal feature extraction module is constructed using a long short-term memory neural network, and the local feature extraction module is constructed using a convolutional neural network. The temporal feature extraction module and the local feature extraction module are connected in parallel and then connected in series with the feature fusion module.

4. The method for recognizing the operating style of an electric robot according to claim 3, characterized in that, The step of inputting the acceleration data into a preset feature extraction model to obtain the acceleration feature includes: The acceleration data is input into the time-series feature extraction module to predict the sudden stop trend and obtain the time-series dynamic features. The acceleration data is input into the local feature extraction module to identify acceleration peaks and obtain local abrupt change features. The temporal dynamic features and the local mutation features are input into the feature fusion module for feature fusion, and the mean of the fused features is calculated using a sliding window mechanism to obtain the radicality features.

5. The method for recognizing the operating style of a power robot according to claim 1, characterized in that, The steps of performing wavelet packet decomposition and spectral energy entropy calculation on the mechanical data to obtain the fine-grained features include: The mechanical data is preprocessed and decomposed into wavelet packets to obtain several sub-bands and corresponding wavelet coefficients. Based on the wavelet coefficients, calculate the energy of each sub-band and the total energy of all sub-bands; Based on the energy and the total energy, the spectral energy entropy is calculated, and the spectral energy entropy is used as a fineness feature.

6. The method for recognizing the operating style of a power robot according to claim 1, characterized in that, The step of calculating the Lyapunov exponent and root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data to obtain the stability characteristics includes: The trajectory data is reconstructed in phase space to obtain the Lyapunov exponent; Calculate the root mean square deviation of the trajectory based on the standard motion trajectory and the trajectory data; Based on the Lyapunov exponent and the root mean square deviation of the trajectory, a nonlinear function is used to obtain the stability characteristics.

7. The method for recognizing the operating style of a power robot according to claim 1, characterized in that, The style scoring model includes a cascaded encoding module, a global feature fusion module, and a spatial projection module, wherein the encoding module is constructed using a Transformer encoder. The encoding module is used to globally model the radicality feature, the fineness feature, and the stability feature to obtain the hidden state sequence; The global feature fusion module is used to perform global average pooling on the hidden state sequence to obtain a global feature vector; The spatial projection module is used to perform projection matrix mapping and value range constraints on the global feature vector to obtain a three-dimensional style score.

8. A power robot operation style recognition system, characterized in that, include: The cross-modal reasoning module is used to acquire multimodal data of the electric robot performing the task, and to perform cross-modal reasoning on the multimodal data to obtain the task of the electric robot. The operation feature extraction module is used to extract motion data from the multimodal data and obtain a standard motion trajectory according to the task. The motion data includes acceleration data, mechanical data and trajectory data. The acceleration data is input into a preset feature extraction model to obtain the aggression feature. The feature extraction model is constructed based on a long short-term memory neural network and a convolutional neural network. Wavelet packet decomposition and spectral energy entropy calculation are performed on the mechanical data to obtain the fineness features; Based on the standard motion trajectory and the trajectory data, the Lyapunov exponent and the root mean square deviation of the trajectory are calculated to obtain the stability characteristics. The operation style recognition module is used to input the aggression feature, the fineness feature and the stability feature into a preset style scoring model to obtain a three-dimensional style score, and to obtain the operation style of the electric robot based on the three-dimensional style score. The style scoring model is constructed based on a neural network model.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images