A power robot control method and system based on a visual speech action model

Abstract

The application relates to the technical field of electric power construction automation, and discloses an electric power robot control method and system based on a visual voice action model, the method of which improves the VLA visual voice action model by aiming at an electric power construction scene, strengthens the full connection layer of a feedforward neural network in a nonlinear transformation attention layer of the VLA visual voice action model, and replaces the full connection layer with a combination of a deep separable convolution layer and a full connection layer, so that the deep separable convolution layer can capture the local spatial continuity hidden in a Token sequence, the feedforward neural network can obtain a local spatial convolution mechanism similar to the local receptive field of a CNN, and the full connection layer can model global correlation, the spatial perception ability of the CNN and the global modeling ability of a Transformer are effectively combined, the spatial understanding ability of the VLA visual voice action model is improved, and the VLA visual voice action model is beneficial to more accurately controlling an electric power robot to complete a complex task in a complex electric power construction scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power construction automation, in particular to a power robot control method and system based on a visual voice action model. BACKGROUND

[0002] In the current substation equipment installation, power transmission line erection, power facility maintenance and other scenarios, the traditional manual assisted type power construction equipment and the preset program type power construction robot are usually used for construction.

[0003] For the traditional manual assisted type power construction equipment, the core is "manual operation + mechanical execution", such as aerial work platform, hydraulic wrench, etc. During construction, workers directly operate the equipment action through the console or remote controller. For example, in the power transmission line bolt tightening operation, workers need to first reach the work point by the aerial work platform, and then manually control the hydraulic wrench to complete the tightening operation. The core function is to expand human power and extend the work range through mechanical structure, but it completely depends on manual judgment of environment and operation opportunity, and has no autonomous perception and decision-making ability.

[0004] For the preset program type power construction robot, such as power transmission line tower material handling robot, the robot pre-records the path coordinates of the construction area, and relies on GPS positioning and infrared obstacle avoidance sensor to complete the grabbing and positioning of the tower material according to the preset program during work. The technical scheme is that the preset path data is stored in the robot body, the real-time position of the robot is obtained through GPS, and the moving direction is adjusted after comparing with the preset coordinates. When the infrared sensor detects an obstacle, a preset obstacle avoidance action (such as stopping or turning) is triggered. In addition, in the equipment installation scene, some robots can complete the positioning grabbing and placing of the mechanical arm through the pre-calibrated equipment coordinates, but cannot adapt to the dynamic changes such as equipment position offset and new obstacles in the construction scene.

[0005] In summary, the traditional manual assisted type power construction equipment completely depends on manual operation, which has the following disadvantages:

[0006] 1. High safety risk, workers need to work in high altitude and high voltage environment, which is easy to cause electric shock and falling accidents;

[0007] 2. Low construction efficiency, limited by workers' physical strength and skill proficiency, cannot work continuously;

[0008] 3. Poor quality stability, such as bolt tightening torque depending on workers' experience control, which is easy to cause equipment operation hidden danger due to operation error;

[0009] The preset program type power construction robot only realizes partial automation, which has the following disadvantages:

[0010] I. Poor environmental adaptability, only able to operate in pre-labeled fixed scenes, if the construction area has unrecorded obstacles (such as temporarily stacked tools), equipment position deviation, then operation interruption or misoperation will occur;

[0011] II. Rigid human-computer interaction, only supports fixed instructions (such as remote control keys, pre-set codes), cannot understand natural language instructions of construction personnel (such as "install the circuit breaker to the No. 3 cabinet"), the operation threshold is high, and professional personnel need to be trained before use;

[0012] III. Single function, mainly designed for single task (such as only carrying, only tightening), cannot complete the coherent construction process of "recognition-grabbing-installation-verification", complex tasks still need multiple devices to cooperate with manual connection, and the collaborative efficiency is low;

[0013] IV. No dynamic adjustment capability, no real-time feedback mechanism when executing actions, if the mechanical arm has too much grabbing force to cause equipment damage, or installation alignment deviation, cannot be automatically corrected, and manual intervention is needed.

[0014] Therefore, a more intelligent power robot control method is needed to better improve the power construction efficiency and safety. SUMMARY

[0015] The purpose of the present application is to provide a power robot control method and system based on a visual voice action model, which method improves the VLA visual voice action model for power construction scenes, replaces the fully connected layer of the feedforward neural network in the nonlinear transformation enhanced attention layer with a combination of a depth separable convolution layer and a fully connected layer, can capture the "local spatial continuity" implied in the Token sequence with the depth separable convolution layer, so that the feedforward neural network obtains a local spatial convolution mechanism similar to the local receptive field of CNN, and then uses the fully connected layer to model the global association, effectively combines the spatial perception ability of CNN and the global modeling ability of Transformer, improves the spatial understanding ability of the VLA visual voice action model, and is beneficial to the VLA visual voice action model to more accurately control the power robot to complete complex tasks in complex power construction scenes, and solves the technical problems of poor environmental adaptability, rigid human-computer interaction and single function of the existing technology using preset program type power construction robots.

[0016] The present application is realized by the following technical solutions:

[0017] In a first aspect, the present application discloses a power robot control method based on a visual voice action model, comprising the following steps:

[0018] Collecting power field visual data and power construction voice instructions and preprocessing to obtain visual information and structured task sequence;

[0019] Adopting cross attention network, similarity between equipment feature vector in visual information and language feature vector in structured task sequence is calculated to align semantics between target equipment and task, and joint feature is obtained;

[0020] The decoded joint feature is input into a nonlinear transformation reinforced attention layer to obtain a feature expression for predicting the next action, wherein the nonlinear transformation reinforced attention layer is configured such that the fully connected layer of the feedforward neural network thereof is replaced by a combination of a depth separable convolution layer and a fully connected layer, so that the feedforward neural network obtains a local spatial convolution mechanism;

[0021] According to the feature expression, the inverse kinematics algorithm of the robot arm and the numerical iteration method, the action sequence is obtained, and after the action parameters in the action sequence are verified, the action sequence is sent to the target power robot.

[0022] In order to better realize the present application, further, the joint feature is decoded by using a pre-established decoder, and the decoder is configured to preferentially focus on the Token continuous on the predefined neighborhood space when decoding the joint feature.

[0023] In order to better realize the present application, further, the method for decoding the joint feature comprises:

[0024] The spatial coordinates of each Token are obtained, and the spatial neighborhood mask M(i,j) is obtained according to the predefined neighborhood space rule, wherein if M(i,j) is 1, it indicates that Token i and Token j are continuous on the predefined neighborhood space and Token j is the spatial neighborhood Token of Token i, and if M(i,j) is 0, it indicates that Token i and Token j are not continuous on the predefined neighborhood space.

[0025] The joint feature is mapped into a joint feature query vector Q, a joint feature key vector K and a joint feature value vector V through linear transformation, and a decoding attention score matrix A is obtained. , wherein, is the dimension of the joint feature key vector K;

[0026] After element multiplication of the spatial neighborhood mask M and the decoding attention score matrix, Softmax normalization is performed to obtain a decoding attention weight matrix W. ;

[0027] ​The joint feature value vector V is weighted and summed using a decoding attention weight matrix to obtain a decoded joint feature .

[0028] To better realize the present application, further, the method for collecting power field visual data and power construction voice instructions and preprocessing to obtain visual information and structured task sequence, comprising:

[0029] Collecting power construction voice instructions of construction personnel from a voice input device, denoising and ASR voice-to-text transcription of the power construction voice instructions to obtain initial long task text instructions;

[0030] According to a preset long task subdivision rule, the initial long task text instructions are parsed to obtain a structured task sequence containing multiple short tasks executed in sequence;

[0031] Collecting visual data including power field image information and depth information from an RGBD camera, obtaining a three-dimensional reconstruction map according to the visual data, and denoising and size adjusting the visual data to obtain visual information.

[0032] To better realize the present application, further, the method for calculating the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence by using a cross-attention network to align the semantics between the target device and the task to obtain joint features, comprising:

[0033] extracting a device feature vector matrix in the visual information by a visual encoding model , and extracting a language feature vector matrix in the task sequence by a language encoding model , wherein B is a batch, is the number of device tokens, is the device feature dimension, is the number of language tokens, is the language feature dimension;

[0034] calculating a cross-modal interaction attention score matrix between the device feature vector matrix and the language feature vector matrix by using a cross-attention network , wherein is the cross-modal interaction feature dimension, is the cross-modal interaction query vector, is the cross-modal interaction key vector;

[0035] Softmax normalizing the cross-modal interaction attention score matrix to obtain a cross-modal interaction attention weight matrix ;

[0036] Based on the cross-modal interaction attention weight matrix, the mapped device feature vector matrix is ​​weighted, fused, and its features are output to obtain joint features. ,in, This is the mapped device feature vector matrix. This represents the mapped device feature dimension.

[0037] To better implement this invention, a method is further provided in which the decoded joint features are input into a nonlinear transformation-enhanced attention layer to obtain a feature representation for predicting the next action, comprising:

[0038] By replacing fully connected layers with a feedforward neural network combining depthwise separable convolutional layers and fully connected layers, the decoded joint features of the input are first subjected to a nonlinear transformation, followed by residual connections and LayerNorm normalization of the feature distribution to obtain the preceding feature representation. ,in, t is the time step, D is the dimension of the output, and W2 is the weight matrix of the feedforward neural network with nonlinear transformation to enhance the attention layer. This is a convolutional layer with a kernel size of 1×1. This is a convolutional layer with a kernel size of k×k, where k is an odd number greater than 1, and C is the number of channels. Let be the convolution kernel for the convolution operation corresponding to channel x. b1 is the convolution kernel used for pointwise convolution, b2 is the flattening operation, and b2 is the bias vector of the nonlinear transformation to enhance the attention layer feedforward neural network.

[0039] Slicing and autoregressive operations are performed on the preceding feature representation to obtain the feature representation used to predict the next action. Where [:,t,:] represents the slicing operation, W out To predict the weight matrix of the feedforward neural network in the output layer, b out To predict the bias vector of the output layer feedforward neural network, D action This refers to the actual dimension of the action.

[0040] To better realize this invention, a method for obtaining action sequences by performing motion planning based on feature representation, inverse kinematics algorithm of robotic arm, and numerical iteration method includes:

[0041] Obtain short tasks from a structured task sequence, and based on the relative position of the target device in the short task and the feature expression, obtain the target pose that represents the actual dimension and displacement of each dimension of the action corresponding to the short task.

[0042] According to the target pose, a Jacobian matrix is used to establish a relationship between the joint speed of the robot arm and the end effector speed of the robot arm , wherein, is an error correction direction, the error correction direction being a ratio of the desired linear speed and angular speed of the end effector of the robot arm, is the linear speed of the end effector in the y direction, is the linear speed of the end effector in the z direction, is the angular speed of the end effector in the x direction, is the angular speed of the end effector in the y direction, is the angular speed of the end effector in the z direction, is a 6*n Jacobian matrix, n being the degree of freedom of the joint of the robot arm, is a joint speed vector of the robot arm, q being a joint angle vector of the robot arm;

[0043] According to the relationship, the joint speed of the robot arm is solved as a correction amount using a derivation formula with a damping term added in the pseudo-inverse calculation, wherein the derivation formula is , is a damping term, I being a 6*6 unit matrix, x d is an end desired pose vector, q k is a joint angle vector of the robot arm at the kth iteration, J T (q k ) is a transpose matrix of the Jacobian matrix J(q k ), J(q k ) is a Jacobian matrix of the robot arm at the joint angle q k ;

[0044] According to the correction amount, a numerical iteration method is used for motion planning to obtain a first motion parameter for gradually approaching the target pose of the end effector of the robot arm, and according to the output parameter in the short task, the output value of the end effector of the robot arm is set as a second motion parameter;

[0045] According to the first motion parameter, the second motion parameter and the execution sequence corresponding to the short task, a motion sequence is obtained.

[0046] In order to better realize the present application, further, after the motion sequence is sent to the target power robot, it further comprises:

[0047] Actual execution data of the motion sequence is collected, and target execution data is obtained according to the structured task sequence;

[0048] The actual execution data and the target execution data are compared, if the deviation between the actual execution data and the target execution data is less than or equal to a preset deviation threshold, the control of the electric robot is ended this time, and a work report is generated, if the deviation between the actual execution data and the target execution data is greater than the preset deviation threshold, the joint feature is recalculated, the feature expression calculation and the action planning are performed according to the recalculated joint feature, until the deviation between the actual execution data and the target execution data is less than or equal to the preset deviation threshold.

[0049] In order to better realize the present application, further, by calling a preset electric power construction rule library, a structured task sequence is obtained or an action parameter in the action sequence is verified.

[0050] In the second aspect, the present application discloses an electric robot control system based on a visual voice action model, which is used to execute the electric robot control method based on the visual voice action model as any one of the first aspect, comprising:

[0051] A data acquisition and preprocessing module is configured to acquire electric power field visual data and electric power construction voice instructions and perform preprocessing to obtain visual information and a structured task sequence.

[0052] A multi-modal perception module is configured to calculate the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence by using a cross-attention network, to align the semantics between the target device and the task, and to obtain a joint feature.

[0053] A reinforced attention module is configured to decode the joint feature, input the decoded joint feature into a nonlinear transformation reinforced attention layer, and obtain a feature expression used for predicting the next action, wherein the nonlinear transformation reinforced attention layer is configured such that the fully connected layer of the feedforward neural network thereof is replaced by a combination of a depth separable convolution layer and a fully connected layer, so that the feedforward neural network obtains a local spatial convolution mechanism.

[0054] An action planning and compliance verification module is configured to perform action planning according to the feature expression, a kinematics inverse solution algorithm of a mechanical arm, and a numerical iteration method, to obtain an action sequence, and to send the action sequence to a target electric robot after the action parameter in the action sequence is verified.

[0055] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0056] The improved VLA visual voice action model end-to-end multimodal interaction ability makes the complex power construction voice instruction be understood by the VLA visual voice action model and be associated with the equipment in the power construction site, so that the power robot can perform power construction according to the structured task sequence parsed from the power construction voice instruction, and the problems of poor environmental adaptability, rigid human-computer interaction and single function of the preset program type power construction robot in the prior art are solved.

[0057] The improved VLA visual voice action model strengthens the attention layer through nonlinear transformation, strengthens the spatial understanding ability of the VLA visual voice action model, greatly improves the ability of the VLA visual voice action model to optimize the task order according to the spatial understanding ability during task planning, reduces the number of subdivided task steps, and greatly improves the path planning ability of the VLA visual voice action model during action generation, thereby improving the single-step operation efficiency.

[0058] Only when the action parameter in the action sequence is verified, the action sequence is sent to the target power robot, so that the damage of the equipment caused by excessive grabbing force of the mechanical arm is avoided, and the operation safety of the power robot is improved. BRIEF DESCRIPTION OF DRAWINGS

[0059] The present application is further illustrated in combination with the following drawings and examples, and all the innovative concepts of the present application should be regarded as the disclosed content and the protection scope of the present application.

[0060] Figure 1 It is a flowchart of embodiment 1 of the power robot control method based on the visual voice action model in the present application.

[0061] Figure 2 It is a flowchart of embodiment 4 of the power robot control method based on the visual voice action model in the present application.

[0062] Figure 3 It is a structure block diagram of the embodiment of the power robot control system based on the visual voice action model in the present application. DETAILED DESCRIPTION

[0063] Embodiment 1

[0064] As shown in the figure, an embodiment of the power robot control method based on the visual voice action model includes the following steps: Figure 1 Collecting power site visual data and power construction voice instructions and preprocessing to obtain visual information and structured task sequence;

[0065]

[0066] ​The cross-attention network is adopted to calculate the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence, so as to align the semantics between the target device and the task, and obtain the joint feature;

[0067] The decoded joint feature is input into the nonlinear transformation reinforced attention layer to obtain the feature expression for predicting the next action, wherein the nonlinear transformation reinforced attention layer is configured such that the fully connected layer of the feedforward neural network thereof is replaced by the combination of the depth separable convolution layer and the fully connected layer, so that the feedforward neural network obtains the local spatial convolution mechanism;

[0068] According to the feature expression, the inverse kinematics algorithm of the mechanical arm and the numerical iteration method, the action sequence is obtained, and after the action parameters in the action sequence are verified, the action sequence is sent to the target power robot.

[0069] Specifically, in the embodiment, a voice input device such as a microphone can be used to collect the power construction voice instructions of the construction personnel, and an RGBD camera can be used to collect the power site visual data. The RGBD camera can not only collect images, but also perform 3D scene reconstruction to ensure that the power robot can accurately perceive the distance. The pre-processing of the visual data includes image pre-processing operations such as format conversion, noise reduction, enhancement, size adjustment, ROI cropping, and depth information preprocessing. The pre-processing of the power construction voice instructions includes language pre-processing operations such as format processing, noise reduction, endpoint detection, ASR voice-to-text, text cleaning, word segmentation and entity recognition, task sequence structuring, and text standardization. The two types of pre-processing operations can be selected according to the actual working conditions.

[0070] The device feature vector includes the appearance, identification features and other features of each device, and the language feature vector includes the semantic features of each device type, each task type, the operation position and specific parameters;

[0071] Through the cross-attention network, after training and model iteration using data, the attention mechanism iterates the similarity of language and image (device) in the feature space, aligns the semantics between the target device and the task, and finally forms the joint feature of visual-language unified representation.

[0072] The decoded joint feature is captured by the improved feedforward neural network in the depth separable convolution layer for the local spatial continuity implied in the Token sequence, to obtain the local spatial convolution mechanism similar to the local receptive field of CNN. The fully connected layer connected to the depth separable convolution layer models the global correlation, and then the output of the feedforward neural network is converted into a prediction result in the action dimension through linear mapping, to obtain the feature expression for predicting the next action;

[0073] After obtaining the feature representation, motion planning can be performed using the inverse kinematics algorithm and numerical iteration method of the robotic arm to obtain the motion sequence. To avoid damage to the equipment caused by excessive gripping force of the robotic arm, the motion sequence is only sent to the target electric robot after the motion parameters in the motion sequence have been verified. In this way, an end-to-end electric robot control is completed.

[0074] Example 2

[0075] This embodiment further optimizes upon the above embodiment 1. In this embodiment, a pre-established decoder is used to decode the joint features. The decoder is configured to prioritize tokens that are consecutive in a predefined neighborhood space when decoding the joint features.

[0076] This implementation allows the VLA visual-speech-action model to prioritize tokens that are continuous in a predefined neighborhood space, reducing interference from irrelevant tokens. Combined with an improved nonlinear transformation to enhance the attention layer, the improved VLA visual-speech-action model further enhances its ability to optimize task order based on spatial understanding during task planning. Compared to the basic VLA visual-speech-action model, the number of subdivided task steps is reduced by more than 20%. Simultaneously, the improved VLA visual-speech-action model further enhances its path planning ability during action generation, improving single-step execution efficiency by 30% compared to the basic VLA visual-speech-action model in the same scenario. This solves the problem of insufficient spatial perception continuity in the basic VLA visual-speech-action model in power construction scenarios.

[0077] Furthermore, methods for decoding joint features include:

[0078] Obtain the spatial coordinates of each Token, and obtain the spatial neighborhood mask M(i,j) according to the predefined neighborhood space rules. If M(i,j) is 1, it means that Token i and Token j are continuous in the predefined neighborhood space and Token j is a spatial neighborhood Token of Token i. If M(i,j) is 0, it means that Token i and Token j are not continuous in the predefined neighborhood space.

[0079] Joint features are obtained through linear transformation The joint feature query vector Q, joint feature key vector K, and joint feature value vector V are mapped respectively to obtain the decoding attention score matrix. ,in, Let K be the dimension of the joint feature key vector;

[0080] The spatial neighborhood mask M is element-wise multiplied with the decoded attention score matrix and then subjected to Softmax normalization to obtain a decoded attention weight matrix ;

[0081] The decoded attention weight matrix is used to perform weighted summation on the joint feature vector V to obtain a decoded joint feature .

[0082] Specifically, the predefined neighborhood space rule can be a 4-neighborhood rule, that is, the spatial neighborhood Token of each Token is the Token above, below, left and right of the Token, or an 8-neighborhood rule, in which the Token in the diagonal direction is added as a spatial neighborhood Token based on the 4-neighborhood rule.

[0083] With the present embodiment, the problem of high dependence of operation of power equipment (such as circuit breakers and switch cabinets) on spatial position relationship is solved by setting the spatial neighborhood mask M to shield non-neighborhood Tokens, so that the VLA visual speech action model can focus on adjacent Tokens during decoding, the perception of spatial continuity of the VLA visual speech action model is strengthened, and the operation speed and accuracy of the VLA visual speech action model are improved.

[0084] Further, the method for collecting power site visual data and power construction speech instructions and preprocessing to obtain visual information and structured task sequence comprises:

[0085] Collecting power construction speech instructions of construction personnel from a speech input device, denoising and ASR speech-to-text transcription of the power construction speech instructions to obtain an initial long task text instruction;

[0086] According to a preset long task subdivision rule, the initial long task text instruction is analyzed to obtain a structured task sequence comprising a plurality of sequentially executed short tasks;

[0087] Collecting visual data including power site image information and depth information from an RGBD camera, obtaining a three-dimensional reconstruction map according to the visual data, and denoising and size adjusting the visual data to obtain visual information.

[0088] Further, the method for calculating the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence by using a cross-attention network to align the semantics between the target device and the task to obtain a joint feature comprises:

[0089] extracting a device feature vector matrix in the visual information by a visual encoding model , and extracting a language feature vector matrix in the task sequence by a language encoding model wherein B is batch, is the number of device tokens, is the dimension of device features, is the number of language tokens, is the dimension of language features.

[0090] The cross-attention network is used to calculate the cross-modal interaction attention score matrix between the device feature vector matrix and the language feature vector matrix , wherein, is the cross-modal interaction feature dimension, is the cross-modal interaction query vector, is the cross-modal interaction key vector.

[0091] The cross-modal interaction attention score matrix is subjected to Softmax normalization to obtain a cross-modal interaction attention weight matrix .

[0092] According to the cross-modal interaction attention weight matrix, the mapped device feature vector matrix is weighted and fused and the feature is output to obtain a joint feature , wherein, is the mapped device feature vector matrix, is the mapped device feature dimension.

[0093] In this embodiment, the cross-attention network is used to accurately extract two types of vector matrices, which facilitates subsequent decoding operation and feature expression calculation with the spatial neighborhood mask M.

[0094] Further, the decoded joint feature is input into a nonlinear transformation reinforced attention layer to obtain a method for obtaining a feature expression for predicting the next action, comprising:

[0095] The input decoded joint feature is subjected to nonlinear transformation by a feedforward neural network in which the fully connected layer is replaced by a combination of a depth separable convolution layer and a fully connected layer, and then subjected to residual connection and LayerNorm normalization of feature distribution to obtain a previous feature expression , wherein, , t is the time step, D is the dimension of the output, W2 is the weight matrix of the nonlinear transformation reinforced attention layer feedforward neural network, is a convolution layer with a convolution kernel size of 1x1, is a convolution layer with a convolution kernel size of kxk, k is an odd number greater than 1, and C is the number of channels, is the convolution kernel corresponding to the convolution operation of the x channel, ​flat is a flatten operation, b2 is a bias vector of the nonlinear transformation reinforced attention layer feedforward neural network;

[0096] slice and auto-regressive operation on the preceding feature representation to obtain a feature representation for predicting the next action wherein, [:, t, :] is a slice operation, W out is a weight matrix of the prediction output layer feedforward neural network, b out is a bias vector of the prediction output layer feedforward neural network, is the actual dimension of the action.

[0097] With the present embodiment, the feedforward neural network FNN obtains a local receptive field similar to a CNN, a local spatial convolution mechanism for efficient calculation through parameter sharing, by replacing the fully connected layer with a combination of a depth separable convolution layer and a fully connected layer;

[0098] The original output of the conventional feedforward neural network FNN is The corresponding original preceding feature representation is Compared with the conventional feedforward neural network FNN using only a fully connected layer, the feedforward neural network with a local spatial convolution mechanism in the present embodiment can efficiently extract features in the decoded joint features through the depth separable convolution layer, greatly reducing parameter redundancy, and then model global correlation through the fully connected layer, further improving calculation efficiency and accuracy.

[0099] Further, a method for action planning according to the feature representation, the inverse kinematics algorithm of the robot arm, and the numerical iteration method, includes:

[0100] Obtaining a target pose representing the actual dimension of the action corresponding to the short task and the displacement amount of each dimension according to the relative position of the target device in the short task and the feature representation;

[0101] According to the target pose, using the Jacobian matrix to establish the relationship between the joint speed of the robot arm and the end effector speed of the robot arm wherein, is an error correction direction, the error correction direction is the ratio of the desired linear speed and angular speed of the end effector of the robot arm, is the linear speed of the end effector in the y direction, is the linear speed of the end effector in the z direction, is the angular speed of the end effector in the x direction, is the angular speed of the end effector in the y direction, is the angular speed of the end effector in the z direction, is a Jacobian matrix, n is the degree of freedom of the joint of the robot arm, is a joint velocity vector of the robot arm, q is a joint angle vector of the robot arm;

[0102] The joint velocity of the robot arm is solved as a correction amount according to the relationship using a derivation formula in which a damping term is added in pseudo-inverse calculation, wherein the derivation formula is , is a damping term, I is a 6*6 unit matrix, x d is an end position vector, q k is a joint angle vector of the robot arm at the kth iteration, J T (q k ) is a transpose matrix of the Jacobian matrix J(q k ), J(q k ) is a Jacobian matrix of the robot arm at the joint angle q k ;

[0103] According to the correction amount, a numerical iteration method is used to plan the action to obtain a first motion parameter that makes the end effector of the robot arm gradually approach the target position, and according to the output parameter in the short task, the output value of the end effector of the robot arm is set as a second motion parameter;

[0104] According to the first motion parameter, the second motion parameter and the execution order corresponding to the short task, an action sequence is obtained.

[0105] By introducing the damping term, the singular point and the convergence stability problem can be solved when solving the correction amount.

[0106] In a specific embodiment, the preset long task subdivision rule includes subdividing the initial long task text instruction obtained by transcription according to four short task types of task type, target device, installation position and output parameter, for example, for the initial long task text instruction of "installing a 10kV SF6 circuit breaker to a No. 2 high-voltage cabinet, and tightening the bolt torque to 25N·m", four short tasks of "installing", "10kV SF6 circuit breaker", "No. 2 high-voltage cabinet" and "torque 25N·m" are parsed as a structured task sequence in order;

[0107] The RGBD camera performs real-time 3D scene reconstruction, identifies the positions of the 10kV SF6 circuit breaker and the No. 2 high-voltage cabinet in the material area through the RGB image, measures the XYZ coordinates and obstacle conditions through the depth image;

[0108] The reinforcement learning control algorithm and the feature expression obtained by the above embodiment are combined with the obtained 3D reconstruction scene to give a robot action motion planning, and the actual dimension Daction For 7, including navigation relative coordinates of six degrees of freedom x, y, z, u, v, w, and jaw working state, a 10kV SF6 circuit breaker is grabbed at the relative position [0.6, 0.3, 0.1], the inverse kinematics of the mechanical arm generates a grabbing trajectory, and the first motion parameter [1, 0.5, 0.2, 0, 0, 0, 0] of 7 dimensions is obtained, the first motion parameter is that the X axis moves 1 unit in the positive direction, the Y axis moves 0.5 units in the positive direction, and the Z axis moves 0.2 units in the positive direction, and the rest of the posture and the jaw posture and state remain unchanged, the unit is set according to the actual working condition, and 1 unit can be 1 cm or 10 cm (here it is 10 cm), according to the output parameter and the size of the 10kV SF6 circuit breaker identified, the second motion parameter of the tightening tool is set as a torque of 25N·m, and the second motion parameter of the jaw is set as an opening of 30cm;

[0109] After verifying that the tightening tool torque of 25N·m and the jaw opening of 30cm meet the safety specifications, the action sequence is output.

[0110] Embodiment 3

[0111] This embodiment is further optimized on the basis of the above-mentioned embodiment 2, and in this embodiment, after sending the action sequence to the target power robot, it further includes:

[0112] Collecting actual execution data of the action sequence, and obtaining target execution data according to the structured task sequence;

[0113] Comparing the actual execution data and the target execution data, if the deviation between the actual execution data and the target execution data is less than or equal to the preset deviation threshold, the control of the power robot this time is ended and a work report is generated, if the deviation between the actual execution data and the target execution data is greater than the preset deviation threshold, the joint feature is recalculated, the feature expression calculation and action planning are performed according to the recalculated joint feature, and the deviation between the actual execution data and the target execution data is less than or equal to the preset deviation threshold.

[0114] Specifically, referring to a specific implementation in embodiment 2, the mechanical arm of the power robot moves the 10kV SF6 circuit breaker to No. 2 high-voltage cabinet according to the action sequence, first detects the relative position deviation through the RGBD camera, if the relative position deviation is less than the relative position deviation threshold, such as 0.3mm, which is less than the relative position deviation threshold of 0.5mm, no adjustment is needed, when the torque reaches 25N·m is monitored through the force sensor, the tightening tool automatically stops, at the same time, the power robot broadcasts "10kV SF6 circuit breaker installation is completed, torque is qualified" through voice, and uploads the installation photo and torque data to the background management system.

[0115] Embodiment 4

[0116] This embodiment is further optimized on the basis of the above-mentioned embodiment 3. In this embodiment, a preset power construction rule library is called to obtain a structured task sequence or to verify the action parameters in the action sequence.

[0117] Specifically, the method for establishing the power construction rule library is as follows:

[0118] For the power construction scene, the common construction equipment categories are determined in communication with experts in the power construction field, such as circuit breakers, insulating blankets, wire breakers, transformers, insulators, and power transmission lines. Workers use handheld interactive devices to guide the robot to complete the work during the construction process by language instructions, record and collect the operation instructions, and obtain a domain data set containing the equipment type and specific instructions.

[0119] Referring to the specific implementation in embodiment 2, for the initial long task text instruction of “installing a 10kV SF6 circuit breaker to No. 2 high-voltage cabinet with a bolt fastening torque of 25N·m”, the power construction rule library is called to analyze the task type as “installation”, the target equipment as “10kV SF6 circuit breaker”, the installation position as “No. 2 high-voltage cabinet”, and the output parameter as “torque 25N·m”. Four short tasks in sequence are obtained as a structured task sequence, and it is verified whether the output parameter “torque 25N·m” meets the installation standard of the 10kV SF6 circuit breaker. If not, the voice broadcast is “the output parameter in the instruction is not verified, please re-input after checking”.

[0120] An electronic fence based on vision can also be used to monitor the relative position between the worker or the intruder and the target equipment or the power robot. When the worker or the intruder is too close to the target equipment or the power robot, i.e., the relative position between the worker and the target equipment is less than the safety distance threshold, it is determined that there is a safety risk in this operation. The power robot broadcasts the voice “monitoring a personnel intrusion risk in the operation range, this instruction cannot be executed, please check the operation area and re-input the instruction”.

[0121] By using the “perception-planning-execution-feedback” closed-loop mechanism, the key parameter deviations such as installation alignment and torque control are corrected in real time, the dynamic feedback closed-loop control is realized, the construction precision is controlled within ±0.5mm, and the problem that the existing technology adopts a preset program type power construction robot without dynamic adjustment capability and without real-time feedback mechanism when executing actions is solved, avoiding the situation that the installation alignment deviation cannot be automatically corrected and manual intervention is required.

[0122] Fusion of "human detection-output parameter monitoring-rule checking" triple security protection, from perception, decision, execution three levels to prevent safety accidents.

[0123] Further, after ending the control of the power robot this time and generating a work report, further comprising:

[0124] storing the task data generated in this task to a designated storage location;

[0125] obtaining task data within a specified time as an incremental data set, and fine-tuning the visual voice action model through the incremental data set.

[0126] Specifically, the task data includes all data generated from the initial step to the end of the task, the initial step is to collect power site visual data and power construction voice instructions and pre-process, the end of the task is to end the control of the power robot this time and generate a work report, the visual information, structured task sequence, joint features, feature expression, action sequence and other data generated therein will be used as task data for this task.

[0127] Specifically, as shown in Figure 2 a specific embodiment of the present embodiment, comprising the following steps:

[0128] Step S1, collecting the power construction voice instructions of the construction personnel from the voice input device, denoising and ASR voice-to-text transcription of the power construction voice instructions, obtaining the initial long task text instructions;

[0129] Step S2, according to the preset long task subdivision rule, analyzing the initial long task text instructions to obtain a structured task sequence containing multiple short tasks executed in sequence;

[0130] Step S3, collecting visual data including power site image information and depth information from the RGBD camera, obtaining a three-dimensional reconstruction map according to the visual data, and denoising and size adjusting the visual data to obtain visual information;

[0131] Step S4, extracting the device feature vector matrix in the visual information through the visual encoding model, and extracting the language feature vector matrix in the task sequence through the language encoding model;

[0132] Step S5, using cross-attention network to calculate the cross-modal interaction attention score matrix between the device feature vector matrix and the language feature vector matrix;

[0133] Step S6, Softmax normalizing the cross-modal interaction attention score matrix to obtain a cross-modal interaction attention weight matrix;

[0134] Step S7, according to the cross-modal interaction attention weight matrix, the mapped device feature vector matrix is weighted and fused, and a feature output is obtained, to obtain a joint feature;

[0135] Step S8, the input decoded joint feature is first subjected to a nonlinear transformation by a feedforward neural network in which a fully connected layer is replaced by a combination of a depth separable convolution layer and a fully connected layer, and then subjected to residual connection and LayerNorm normalization of feature distribution, to obtain a pre-sequence feature expression;

[0136] Step S9, the pre-sequence feature expression is subjected to slicing and autoregressive operation, to obtain a feature expression for predicting the next action;

[0137] Step S10, a short task in the structured task sequence is obtained, and according to the relative position of the target device in the short task and the feature expression, a target pose representing the actual dimension of the action corresponding to the short task and the displacement amount of each dimension is obtained;

[0138] Step S11, according to the target pose, a Jacobian matrix is used to establish a relationship between the joint speed of the robot arm and the end effector speed of the robot arm;

[0139] Step S12, using a derivation formula in which a damping term is added in pseudo-inverse calculation, the joint speed of the robot arm is solved as a correction amount according to the relationship;

[0140] Step S13, according to the correction amount, a numerical iterative method is used for action planning, to obtain a first motion parameter for making the end effector of the robot arm gradually approach the target pose, and according to the output parameter in the short task, an output value of the end effector of the robot arm is set as a second motion parameter;

[0141] Step S14, according to the first motion parameter, the second motion parameter and the execution order corresponding to the short task, an action sequence is obtained;

[0142] Step S15, actual execution data of the action sequence is collected, and target execution data is obtained according to the structured task sequence;

[0143] Step S16, the actual execution data and the target execution data are compared, if the deviation between the actual execution data and the target execution data is less than or equal to a preset deviation threshold, the control of the electric robot is ended and a work report is generated, if the deviation between the actual execution data and the target execution data is greater than the preset deviation threshold, the step S5 is returned.

[0144] Embodiment 5

[0145] As Figure 3As shown, an embodiment of a power robot control system based on a visual speech action model is used to perform the power robot control method based on the visual speech action model as described in any one of embodiments 1 to 4, comprising:

[0146] A data acquisition and preprocessing module is configured to acquire and preprocess power site visual data and power construction speech instructions to obtain visual information and a structured task sequence.

[0147] A multi-modal perception module is configured to use a cross-attention network to calculate the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence to align the semantics between the target device and the task, and obtain a joint feature.

[0148] A reinforcement attention module is configured to decode the joint feature and input the decoded joint feature into a nonlinear transformation reinforcement attention layer to obtain a feature representation for predicting the next action, wherein the nonlinear transformation reinforcement attention layer is configured such that the fully connected layer of its feedforward neural network is replaced by a combination of a depth separable convolution layer and a fully connected layer, so that the feedforward neural network obtains a local spatial convolution mechanism.

[0149] An action planning and compliance verification module is configured to perform action planning according to the feature representation, a kinematics inverse solution algorithm of the robot arm, and a numerical iteration method, obtain an action sequence, and send the action sequence to the target power robot after the action parameters in the action sequence are verified.

[0150] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change based on the technical essence of the present application to the above embodiment falls within the scope of protection of the present application.

Claims

1. A power robot control method based on a visual speech action model, characterized by, The method comprises the following steps: collecting and preprocessing power field visual data and power construction voice instructions to obtain visual information and a structured task sequence; using a cross-attention network to calculate the similarity between the equipment feature vector in the visual information and the language feature vector in the structured task sequence to align the semantics between the target equipment and the task, and obtaining joint features; decoding the joint features, inputting the decoded joint features into a nonlinear transformation reinforced attention layer to obtain feature expressions for predicting the next action, wherein the nonlinear transformation reinforced attention layer is configured such that the fully connected layer of the feedforward neural network thereof is replaced by a combination of a depth separable convolution layer and a fully connected layer, so that the feedforward neural network obtains a local spatial convolution mechanism; performing action planning according to the feature expressions, a kinematics inverse solution algorithm of a robot arm and a numerical iteration method, obtaining an action sequence, and sending the action sequence to a target power robot after the action parameters in the action sequence are verified; The method of using a cross-attention network to calculate the similarity between the equipment feature vector in the visual information and the language feature vector in the structured task sequence to align the semantics between the target equipment and the task, and obtaining joint features, comprises: extracting a device feature vector matrix in visual information through a visual encoding model and extracting a language feature vector matrix in a task sequence through a language encoding model wherein B is a batch, is a device token number, is a device feature dimension, is a language token number, is a language feature dimension; The device feature vector matrix is ​​calculated using a cross-attention network. and language feature vector matrix Cross-modal interaction attention score matrix ,in, For cross-modal interaction feature dimensions, For cross-modal interactive query vectors, It is a cross-modal interaction key vector; Softmax normalization is performed on the cross-modal interaction attention score matrix to obtain a cross-modal interaction attention weight matrix ; According to the cross-modal interaction attention weight matrix, a mapped device feature vector matrix is weighted and fused and a feature output is obtained to obtain a joint feature wherein, is a mapped device feature vector matrix, is a mapped device feature dimension The method of decoding the joint features comprises: using a pre-established decoder to decode the joint features, wherein the decoder is configured to preferentially focus on consecutive Tokens in a predefined neighborhood space when decoding the joint features, and the method of decoding the joint features specifically comprises: obtaining the spatial coordinates of each Token, and obtaining a spatial neighborhood mask M(i, j) according to a predefined neighborhood space rule, wherein if M(i, j) is 1, it indicates that Token i and Token j are continuous in the predefined neighborhood space and Token j is a spatial neighborhood Token of Token i, and if M(i, j) is 0, it indicates that Token i and Token j are not continuous in the predefined neighborhood space; The joint feature is mapped into a joint feature query vector Q, a joint feature key vector K, and a joint feature value vector V by a linear transformation, to obtain a decoding attention score matrix wherein, wherein, is a dimension of the joint feature key vector K. The spatial neighborhood mask M is element-wise multiplied with the decoded attention score matrix, and then normalized by Softmax to obtain a decoded attention weight matrix ; The joint feature value vector V is weighted and summed using the decoding attention weight matrix to obtain a decoded joint feature ; The method of inputting the decoded joint features into a nonlinear transformation reinforced attention layer to obtain feature expressions for predicting the next action comprises: The input decoded joint feature is first subjected to nonlinear transformation by a feedforward neural network replaced by a combination of a deep separable convolution layer and a full connection layer, then subjected to residual connection and LayerNorm normalization of feature distribution to obtain a pre-sequence feature expression wherein, , t is a time step, D is a dimension of output, W2 is a weight matrix of the nonlinear transformation reinforced attention layer feedforward neural network, is a convolution layer with a convolution kernel size of 1×1, is a convolution layer with a convolution kernel size of k×k, k is an odd number greater than 1, and C is a channel number, is a convolution kernel of a convolution operation corresponding to a channel of x, is a convolution kernel used for point-wise convolution, flat is a flattening operation, and b2 is a bias vector of the nonlinear transformation reinforced attention layer feedforward neural network; slicing and auto-regressive operations on the preceding feature representation to obtain a feature representation for predicting the next action where [:, t, :] is a slicing operation, W out is a weight matrix of the prediction output layer feed-forward neural network, b out is a bias vector of the prediction output layer feed-forward neural network, D action is the actual dimension of the action.

2. The power robot control method based on visual speech action model according to claim 1, characterized in that, The method of collecting and preprocessing power field visual data and power construction voice instructions to obtain visual information and a structured task sequence comprises: collecting power construction voice instructions of construction personnel from a voice input device, denoising and ASR voice-to-text transcription of the power construction voice instructions to obtain initial long task text instructions; analyzing the initial long task text instructions according to a preset long task subdivision rule to obtain a structured task sequence containing multiple short tasks executed in sequence; collecting visual data including power field image information and depth information from an RGBD camera, obtaining a three-dimensional reconstruction map according to the visual data, and denoising and resizing the visual data to obtain visual information.

3. The visual speech action model based power robot control method according to claim 1, wherein, The method of performing action planning according to the feature expressions, a kinematics inverse solution algorithm of a robot arm and a numerical iteration method to obtain an action sequence comprises: obtaining a target pose representing an actual dimension of an action corresponding to the short task and a displacement amount of each dimension according to a relative position of a target device in the short task and the feature expression; According to the target pose, a Jacobian matrix is used to establish a relationship between the joint speed of the robot arm and the end effector speed of the robot arm wherein, is an error correction direction, the error correction direction being a ratio of a desired linear velocity and angular velocity of the end effector of the robot arm, is a linear velocity of the end effector in the y direction, is a linear velocity of the end effector in the z direction, is an angular velocity of the end effector in the x direction, is an angular velocity of the end effector in the y direction, is an angular velocity of the end effector in the z direction, is a 6 x n Jacobian matrix, n being the degree of freedom of the joint of the robot arm, is a joint speed vector of the robot arm, q being a joint angle vector of the robot arm; The derivation formula with a damping term added in pseudo-inverse calculation is used to solve the joint speed of the mechanical arm as a correction amount according to the relationship, wherein the derivation formula is , is a damping term, I is a 6*6 unit matrix, x d is an end desired pose vector, q k is a joint angle vector of the mechanical arm at the kth iteration, J T (q k ) is a transpose matrix of the Jacobian matrix J(q k ), J(q k ) is a Jacobian matrix of the mechanical arm at the joint angle q k . obtaining a first motion parameter for gradually approaching the target pose by the end effector of the robot arm using a numerical iteration method according to the correction amount, and setting an output value of the end effector of the robot arm as a second motion parameter according to an output parameter in the short task; obtaining an action sequence according to the first motion parameter, the second motion parameter, and an execution order corresponding to the short task.

4. The power robot control method based on visual speech action model according to claim 3, characterized in that, After sending the action sequence to the target power robot, further comprising: collecting actual execution data of the action sequence, and obtaining target execution data according to the structured task sequence; comparing the actual execution data and the target execution data, if a deviation between the actual execution data and the target execution data is less than or equal to a preset deviation threshold, ending the control of the power robot this time and generating a work report, if the deviation between the actual execution data and the target execution data is greater than the preset deviation threshold, recalculating the joint feature, and performing feature expression calculation and action planning according to the recalculated joint feature until the deviation between the actual execution data and the target execution data is less than or equal to the preset deviation threshold.

5. The power robot control method based on the visual speech action model according to any one of claims 1 to 4, characterized in that: a preset power construction rule library is called to obtain a structured task sequence or to verify action parameters in the action sequence.

6. A visual speech action model based electric power robot control system for performing the visual speech action model based electric power robot control method according to any one of claims 1 to 5, characterized by, comprising: a data acquisition and preprocessing module for acquiring power site visual data and power construction speech instructions and preprocessing to obtain visual information and a structured task sequence; a multi-modal perception module for calculating the similarity between the device feature vector in the visual information and the language feature vector in the structured task sequence using a cross-attention network to align the semantics between the target device and the task, and obtaining a joint feature; a reinforced attention module for decoding the joint feature and inputting the decoded joint feature into a nonlinear transformation reinforced attention layer to obtain a feature expression for predicting the next action, wherein the nonlinear transformation reinforced attention layer is configured such that the fully connected layer of the feedforward neural network thereof is replaced by a combination of a depth separable convolution layer and a fully connected layer, so that the feedforward neural network obtains a local spatial convolution mechanism; an action planning and compliance verification module for performing action planning according to the feature expression, a robot arm kinematics inverse algorithm, and a numerical iteration method to obtain an action sequence and send the action sequence to the target power robot after the action parameters in the action sequence pass the verification.

Images