Intelligent robot safe operation method based on three-dimensional Gaussian and touch image fusion

Abstract

The invention provides an intelligent robot safe operation method based on three-dimensional Gaussian and touch image fusion, and relates to the technical field of intelligent robots. According to the mechanical arm, visual information and force sensor data are combined, the grabbing strategy of the clamping device is adjusted in real time, the operation process is optimized, and the reliability and efficiency of the mechanical arm in complex tasks are improved. For visual information acquisition, a three-dimensional space expression method based on three-dimensional Gaussian representation is introduced, the perception and understanding ability of the mechanical arm to the operation environment is remarkably enhanced, and therefore the success rate and accuracy of the mechanical arm during execution of various operation tasks are effectively improved. In addition, a dynamic feedback optimization control algorithm based on a safety area is introduced, and by combining real-time environment data and safety area constraints, it is ensured that the mechanical arm follows a reasoning track in the movement process and also avoids touching the boundary of the safety area, and therefore safety is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a safe operation method of an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images. Background Art

[0002] Traditional robots have a single working mode, poor manipulation flexibility, and lack of comprehension ability. They are unable to adapt to unplanned and random application scenarios, and it is difficult for them to meet the requirements of wide application and embodied intelligence.

[0003] In related technologies, with the rapid development of large-scale pre-trained models, embodied intelligence based on large models has achieved remarkable results in various tasks, showing strong generalization capabilities and broad application prospects. Embodied intelligence systems can perform complex tasks by perceiving, understanding and interacting with the physical world, becoming an important direction in the field of artificial intelligence.

[0004] For example, in the paper (Li P, Wu H, Huang Y, et al. GR-MG: Leveraging PartiallyAnnotated Data via Multi-Modal Goal Conditioned Policy[J]. arXiv preprintarXiv:2408.14368, 2024.), the GR-MG method is proposed to combine language instructions and target images to achieve general robot operations. In the training phase, GR-MG uses trajectory data with text and images for conditional training, and relies only on images when text is missing. In the reasoning phase, GR-MG generates target images based on language instructions through a diffusion image editing model, and then performs tasks based on text and generated images. In order to improve the accuracy of the target image, a progress-guided image generation model is designed, which significantly improves image fidelity and task execution performance. This method effectively utilizes a large amount of partially annotated data and enhances the robot's generalization ability in different environments.

[0005] However, traditional multimodal large model control schemes mainly rely on the third perspective and hand perspective to extract RGB features. The single visual information leads to insufficient gripper control stability, thus affecting the stability and efficiency of task execution. Summary of the invention

[0006] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a safe operation method for an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images, which solves the technical problem that the control stability of the gripper is insufficient due to single visual information.

[0007] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: A safe operation method for an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images, wherein a gripper is installed at the end of the mechanical arm of the intelligent robot; comprising: Collect RGB image sequences from the perspective of the robot hand and the third perspective, and extract image features of the RGB image sequences through the pre-trained ViT model; Based on the pre-saved query vector and the weight parameter of the cross attention, the cross attention calculation is performed on the image feature to obtain a high-dimensional feature based on the three-dimensional Gaussian; The contact force distribution between the left and right clamping jaws of the clamp and the operated object is sensed in real time by a force sensor to obtain a two-dimensional RGB tactile image sequence, and the tactile image features of the RGB tactile image sequence are extracted by the ViT model; The high-dimensional features and the tactile image features are respectively used as inputs of a resampler, and corresponding feature sequences are obtained and then spliced; The splicing result and the language instruction are used as the input of the feature fusion decoder, and the LSTM strategy head is used to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper; Based on the current posture of the gripper and the posture change, a target posture is calculated to determine the moving direction and distance of the gripper.

[0008] Preferably, in the process of calculating the target posture, a dynamic feedback optimization control algorithm based on a safe area is introduced, including: (1) Design of posture controller: Based on the pose change, a first position error between the target position and the current position is defined: Where, e1 is the first position error; is the change of posture; is the target pose of the gripper at the target position to be calculated; is the current posture of the gripper at the current position; A pose controller is used to calculate the control inputs to ensure that the gripper moves smoothly towards the target position: Among them, u1 is the output of the posture controller using the PD controller; k p1 , k d1 are the first proportional and first differential gains respectively; is the rate of change of the first position error; (2) Design of safe zone controller: The safety area is set as a plane area with a height of h. Outside this area, the free movement of the entire robot arm is not restricted. In this area, once the gripper approaches the area boundary, the safety area controller outputs a control amount away from the area boundary: Where u2 is the output of the safety zone controller; k p2 , k d2 are the second proportional and second differential gains respectively; e2 is the second position error between the current position and the safety area, and the component of e2 in the Z direction , d z is the distance from the end of the gripper to the boundary of the region; is the rate of change of the second position error; (3) Specify the evaluation function of the control objective: Based on the square of the difference between the target pose and the current pose, the first evaluation function of the pose controller is designed: Among them, t0 is the initial time point; T is the preset time span; ‖∙‖ is the modulus of the vector; t is the time variable; Based on the square of the difference between the distance from the end of the gripper to the boundary of the area and the height of the safety area, the second evaluation function of the safety area controller is designed: (4) Calculate the predicted evaluation gradient descent value and build a state feedback mechanism: Design saturation suppression function : in, To evaluate the gradient, i is 1 or 2; α is the upper limit of the gradient amplitude; Use gradient descent method to obtain the input after single objective optimization: Among them, u i is the basic control input; λ is the learning rate; K α is the saturation suppression function The corresponding gain factor; The state feedback mechanism is constructed to make the robot state converge quickly. The single-objective controller after adding state feedback is expressed as: Among them, K fb is the state feedback gain matrix, the subscript fb represents the state feedback, is the target state, x iis the current actual status; The combined output is: (5) Calculate a new safe target pose to ensure that the gripper avoids touching the region boundary: in, is the updated target pose.

[0009] Preferably, the process of obtaining the query vector and the cross-attention weight parameters includes: Select a fixed world coordinate system in the robot arm operation scene; Using P three-dimensional Gaussian distributions Represents the robot arm operation scene, each three-dimensional Gaussian is represented by a d-dimensional vector in the form of , where d = 10 + |C sem +2|, and m, s, r, and c are the mean, scale, rotation vector, and semantic category, respectively. is a set of real numbers, C sem is the number of semantic tags; initialization and the learnable vector ; where D is the dimension of the learnable vector, and D>>d, Used to represent the parameter properties of Gaussian, Q j g is a learnable three-dimensional spatial feature, the superscript g represents the query vector associated with the Gaussian representation, and the subscript j represents the index of the Gaussian kernel; According to the initialized Gaussian kernel, voxelization is performed and the occupancy prediction result of each voxel grid is calculated by the following formula: in, It is the sum of the contributions of each Gaussian distribution to any position p in the selected world coordinate system, which is used to characterize the occupancy prediction result; Predicted distribution for 3D Gaussian occupancy The value at point p; is the neighboring Gaussian set of point p; exp is the exponential function; Σ is the covariance matrix; S=diag(s) represents the scale matrix, diag is the function for extracting diagonal elements, and R=q2r(q) is the function for converting the quaternion q used to represent the rotation of three-dimensional space into a rotation matrix; If the prediction result of any voxel grid occupancy is greater than a certain threshold, the voxel grid is considered to be occupied, that is, the position v of any voxel grid is expressed as: Denote the set of occupied voxel grid positions as: Where |∙| is the size of the set; M is the number of voxel grid locations used; The features corresponding to the occupied voxel grid positions are represented as a sparse tensor: where F(v) is the sparse tensor at any voxel grid position v; F S is the set of sparse tensors corresponding to the set S; The P Gaussian kernels correspond to P specific voxel grid locations {v1,v2,…,v p}∈S, define the reserved voxel set P S for: P S Each voxel grid position v in j Apply the sparse convolution kernel to obtain the output feature F'(v): in, is the relative position set of the convolution kernel; W k is the convolution weight corresponding to the relative position k; is the relative position offset; σ is the nonlinear activation function ReLU; In the robot arm operation scene, RGB images of N perspectives are collected. For each perspective RGB image, the corresponding image features are obtained through ViT model encoding: in, is the image feature of the nth perspective; I n is the RGB image of the nth viewing angle; Initialize the query vector Query g and Perform the cross attention mechanism operation and get: Among them, cross_attn is cross attention; The query vector Query g Processed by a set of multi-layer perceptrons MLP, the updated Gaussian kernel parameters are obtained : in, They are the updated mean, scale, rotation vector and semantic category respectively; The updated Gaussian kernel directly replaces the original Gaussian kernel, and the new Gaussian kernel parameters It is expressed as: Will Embedded into a target voxel grid of size X×Y×Z; For each new 3D Gaussian, the radius of its neighborhood is calculated based on its scale property, and the occupancy prediction results of each voxel grid are recalculated; For each re-determined occupied voxel grid, calculate its semantic category corresponding label C label With the predicted value The cross entropy loss CE between: in, is the loss function; Represents the set of occupied voxel grids; the subscript label represents the label; The model parameters are updated through the loss function, and the query vector Query is saved after training. g And the weight parameters of Cross-Attn.

[0010] Preferably, the image features of the RGB image sequence are expressed as: in, is the image feature at time t, the superscript Indicates vision; are the RGB images of the robot arm hand view and the third view at time t respectively; ⊕ is the element-by-element addition symbol; Based on the pre-saved query vector and the weight parameter of the cross attention, the image feature is cross-attention calculated to obtain a high-dimensional feature based on a three-dimensional Gaussian; it is expressed as: in, is the high-dimensional feature based on three-dimensional Gaussian at time t.

[0011] Preferably, the tactile image features of the RGB tactile image sequence are extracted by the ViT model; expressed as: in, is the tactile image feature at time t; They are the RGB tactile images of the left and right jaws of the gripper at time t, and the superscripts lta and rta represent the left and right tactile perception, respectively.

[0012] Preferably, the high-dimensional features and the tactile image features are respectively used as inputs of a resampler, and corresponding feature sequences are obtained and then spliced; including: The high-dimensional features and the tactile image features are respectively input into the K and V fully connected layers of the resampler, and a set of learnable vectors q are input into the Q fully connected layer of the resampler: Among them, Q R ∈ is the query vector corresponding to the learnable parameters of the resampler, is the number of query vectors for the resampler, is the size of the hidden dimension, are the linear transformation matrices of keys and values, respectively, d g / ta Represents the dimension size of high-dimensional features or tactile image features; The transformed key and value vectors generated for the high-dimensional feature input respectively; The converted key and value vectors generated for the tactile image feature input, respectively; are feature sequences corresponding to high-dimensional features and tactile image features output by the resampler, respectively; softmax is the activation function; the superscript T is the transposition symbol; Concatenate the two obtained feature sequences: in, is the concatenation result at time t. The superscript g-ta indicates the fusion of Gaussian features and tactile features; concat is the concatenation operation.

[0013] Preferably, the splicing result and the language instruction are used as the input of the feature fusion decoder, and the LSTM strategy head is used to perform action prediction on the output of the feature fusion decoder to obtain the position change of the gripper and the next state of the gripper; including: The feature fusion process is performed by the feature fusion decoder and integrated with the language instruction to output ; The feature fusion process is expressed as: in, is the output of the lth cross attention layer at time t; Tanh is the hyperbolic tangent function; α∈ is a learnable parameter used to control the contribution of different features in the fusion process; A is the cross or self-attention operation; are the learnable parameters of the cross-attention layer respectively; are the outputs of the lth and l+1th self-attention layers at time t respectively; are the learnable parameters of the self-attention layer respectively; L is the sum of the number of alternating self-attention layers and cross-attention layers, and is an even number; N l is the number of language instructions; Use the LSTM strategy to output X t Perform motion prediction to obtain the position change of the gripper and the next state of the gripper; expressed as: Among them, MaxPooling is the maximum pooling layer, For its output; h t 、h t-1 are the hidden states at time t and t-1 respectively; They are respectively the predicted position change of the gripper and the next state of the gripper, and the state refers to whether the left and right jaws of the gripper are open or closed.

[0014] An intelligent robot safety operating system based on the fusion of three-dimensional Gaussian and tactile images, wherein a gripper is installed at the end of the mechanical arm of the intelligent robot; comprising: A collection module, used to collect RGB image sequences from the perspective of the robot hand and the third perspective, and extract image features of the RGB image sequences through a pre-trained ViT model; A calculation module, used for performing cross-attention calculation on the image features based on a pre-saved query vector and a cross-attention weight parameter to obtain a high-dimensional feature based on a three-dimensional Gaussian; an extraction module, configured to sense the contact force distribution between the left and right clamping jaws of the clamp and the operated object in real time through a force sensor, obtain a two-dimensional RGB tactile image sequence, and extract tactile image features of the RGB tactile image sequence through the ViT model; A sampling module, used to take the high-dimensional features and the tactile image features as inputs of a resampler respectively, obtain corresponding feature sequences and then perform splicing; A prediction module, used to use the splicing result and the language instruction as the input of the feature fusion decoder, and use the LSTM strategy head to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper; The determination module is used to calculate the target posture based on the current posture of the clamp and the posture change amount to determine the moving direction and distance of the clamp.

[0015] A storage medium stores a computer program for safe operation of an intelligent robot based on fusion of three-dimensional Gaussian and tactile images, wherein the computer program enables a computer to execute the safe operation method of the intelligent robot as described above.

[0016] An electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a program for executing the intelligent robot safety operation method as described above.

[0017] (III) Beneficial effects The present invention provides a safe operation method for an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images. Compared with the prior art, it has the following beneficial effects: In the present invention, firstly, the RGB image sequences of the manipulator hand view and the third view are collected to obtain high-dimensional features based on three-dimensional Gaussian; then, the contact force distribution between the end of the gripper and the operated object is sensed in real time by the force sensor to obtain a two-dimensional RGB tactile image sequence to obtain tactile image features; then, the high-dimensional features and the tactile image features are respectively used as the input of the resampler, and the corresponding feature sequences are obtained and spliced, and the splicing results and language instructions are used as the input of the feature fusion decoder, and the LSTM strategy head is used to predict the action of the output of the feature fusion decoder; finally, based on the current posture and posture change of the gripper, the target posture for determining the moving direction and distance of the gripper is calculated. By combining visual information with force sensor data, the gripping strategy of the gripper is adjusted in real time, the operation process is optimized, and the reliability and efficiency of the manipulator in complex tasks are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A block diagram of a safe operation method of an intelligent robot based on fusion of three-dimensional Gaussian and tactile images provided by an embodiment of the present invention; Figure 2 A flowchart of a safe operation method of an intelligent robot based on fusion of three-dimensional Gaussian and tactile images provided by an embodiment of the present invention; Figure 3 A flowchart of a three-dimensional space expression method based on three-dimensional Gaussian representation is provided in an embodiment of the present invention.

[0020] Figure 4 A schematic diagram of a scenario for safe operation of an intelligent robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The embodiment of the present application solves the technical problem of insufficient control stability of the gripper due to single visual information by providing a safe operation method for an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images.

[0023] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows: (1) Three-dimensional space expression method based on three-dimensional Gaussian representation and its application in precise operation of robotic arms: The embodiment of the present invention proposes a three-dimensional space expression method based on three-dimensional Gaussian representation, which aims to significantly improve the large model's ability to understand three-dimensional scenes. Through this method, the model can perceive and express three-dimensional space more accurately, thereby achieving precise operation and control of the robot arm, and improving the adaptability and flexibility of the robot arm in complex environments.

[0024] (2) Use tactile images to represent the force on the gripper and fine-tune it using a pre-trained model based on RGB: The embodiment of the present invention introduces a force sensor between the left and right jaws of the gripper to obtain a two-dimensional image of the force on the jaws in real time, thereby enhancing the model's perception of the jaw state and improving the accuracy of jaw control, thereby improving the stability and success rate of task execution. By combining force sensor data with visual information, the system can adjust the gripping strategy of the jaws in real time, optimize the operation process, and improve the reliability and efficiency of the robot arm in complex tasks.

[0025] (3) Dynamic safety control algorithm of robotic arm based on multi-modal large model: The embodiment of the present invention proposes a dynamic feedback optimization control algorithm based on the safety area, which aims to solve the safety hazards in the operation instructions of the machine learning model. By adjusting the action of the end effector of the robot arm (the gripper is selected in the embodiment of the present invention) in real time, obstacles are automatically identified and avoided, ensuring safe and stable operation, significantly reducing risks, improving the adaptability, flexibility and operation accuracy of the robot arm, and further enhancing the reliability and intelligence level of the system.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] Embodiment 1: like Figure 1 As shown, an embodiment of the present invention provides a safe operation method of an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images, wherein a gripper is installed at the end of the mechanical arm of the intelligent robot; comprising: S1. Collect RGB image sequences from the robot arm hand perspective and the third perspective, and extract image features of the RGB image sequences through the pre-trained ViT model; S2, based on the pre-saved query vector and the weight parameter of the cross attention, performing a cross attention calculation on the image feature to obtain a high-dimensional feature based on a three-dimensional Gaussian; S3, sensing the contact force distribution between the left and right clamping jaws of the clamper and the operated object in real time through a force sensor, acquiring a two-dimensional RGB tactile image sequence, and extracting tactile image features of the RGB tactile image sequence through the ViT model; S4, taking the high-dimensional features and the tactile image features as inputs of a resampler respectively, obtaining corresponding feature sequences and then splicing them; S5, taking the splicing result and the language instruction as the input of the feature fusion decoder, and using the LSTM strategy head to perform action prediction on the output of the feature fusion decoder to obtain the position change of the gripper and the next state of the gripper; S6. Calculate a target posture based on the current posture of the gripper and the posture change to determine the moving direction and distance of the gripper.

[0028] The embodiments of the present invention adjust the gripping strategy of the gripper in real time by combining visual information with force sensor data, optimize the operation process, and improve the reliability and efficiency of the robot arm in complex tasks.

[0029] In an optional implementation, during the process of calculating the target posture in S6, a dynamic feedback optimization control algorithm based on the safety area is also introduced. In summary: a fixed height plane is defined as the safety area, and the evaluation function, gradient optimization, and feedback mechanism for dynamically adjusting the target position are combined to adjust the robot's movement direction and amplitude in real time to effectively avoid exceeding the safety boundary and ensure the safety and stability of task execution.

[0030] like Figure 2 As shown, Figure 2 A flow chart of the provided intelligent robot safe operation method based on fusion of three-dimensional Gaussian and tactile images is given.

[0031] Next, we will combine Figure 2 The following are the steps of the above solution: In step S1, RGB image sequences of the robot arm hand perspective and the third perspective are collected, and image features of the RGB image sequences are extracted through a pre-trained ViT model.

[0032] It should be noted that in traditional large model-driven robotic arm operations, many large models rely on image input for training and reasoning. However, precise control of the robotic arm requires the model to have a full understanding of the three-dimensional space, which is difficult to achieve with only one or two RGB images. Although the introduction of depth maps can enhance the spatial perception ability of the model, it will also significantly increase the computational burden and affect the overall efficiency.

[0033] In contrast, the embodiment of the present invention proposes a new method combining 3D Gaussian to improve the ability of the large robot arm model to extract 3D spatial features through phased training. First, the 3D spatial feature extraction module is trained independently, and then the 3D features are learned using only RGB input, thereby maintaining computational efficiency while enhancing the model's environmental perception ability.

[0034] Accordingly, this step collects RGB image sequences of the robot arm hand perspective and the third perspective for the robot arm operation scene, and extracts image features of the RGB image sequence through the pre-trained ViT model; it is expressed as: in, is the image feature at time t, the superscript Indicates vision; They are the RGB images of the robot arm hand perspective and the third perspective at time t respectively; ⊕ is the element-by-element addition symbol.

[0035] In step S2, based on the pre-saved query vector and the weight parameters of the cross-attention, the image features are cross-attention calculated to obtain high-dimensional features based on three-dimensional Gaussian.

[0036] As mentioned above, the new method combined with three-dimensional Gaussian proposed in the embodiment of the present invention improves the ability of the large robot arm model to extract three-dimensional spatial features through staged training. First, the three-dimensional spatial feature extraction module is trained independently, which is specifically performed in this step based on the pre-saved query vector and the cross-attention weight parameters to perform cross-attention calculation on the image features to obtain high-dimensional features based on three-dimensional Gaussian.

[0037] Therefore, it is necessary to supplement the detailed method of obtaining the pre-saved query vector and the weight parameters of the cross attention. The relevant steps are as follows (wherein steps S400 to S900 are as follows Figure 3 shown): S100: Select a fixed world coordinate system in the robot arm operation scene.

[0038] S200, using P three-dimensional Gaussian distributions Represents the robot arm operation scene, each three-dimensional Gaussian is represented by a d-dimensional vector in the form of , where d = 10 + |C sem +2|, and m, s, r, and c are the mean, scale, rotation vector, and semantic category, respectively. is a set of real numbers, C sem is the number of semantic tags.

[0039] S300, Initialization and are learnable vectors ; where D is the dimension of the learnable vector, and D>>d, Used to represent the parameter properties of Gaussian, Q j g is a learnable three-dimensional spatial feature, the superscript g represents the query vector associated with the Gaussian representation, and the subscript j represents the index of the Gaussian kernel.

[0040] S400, voxel processing is performed according to the initialized Gaussian kernel, and the occupancy prediction result of each voxel grid is calculated by the following formula: in, It is the sum of the contributions of each Gaussian distribution to any position p in the selected world coordinate system, which is used to characterize the occupancy prediction result; Predicted distribution for 3D Gaussian occupancy The value at point p; is the set of neighboring Gaussians of point p; exp is the exponential function; Σ is the covariance matrix; S=diag(s) represents the scale matrix, diag is the function for extracting diagonal elements, and R=q2r(q) is the function for converting the quaternion q used to represent the rotation in three-dimensional space into a rotation matrix.

[0041] If the prediction result of any voxel grid occupancy is greater than a certain threshold, the voxel grid is considered to be occupied, that is, the position v of any voxel grid is expressed as: Denote the set of occupied voxel grid positions as: Where |∙| is the size of the set and M is the number of voxel grid locations used.

[0042] S500, represent the features corresponding to the occupied voxel grid positions as a sparse tensor: where F(v) is the sparse tensor at any voxel grid position v; F S is a set of sparse tensors corresponding to the set S.

[0043] The P Gaussian kernels correspond to P specific voxel grid locations {v1,v2,…,v p}∈S, define the reserved voxel set P S for: P S Each voxel grid position v in j Apply the sparse convolution kernel to obtain the output feature F'(v): in, is the relative position set of the convolution kernel (for example, a 3×3×3 convolution kernel corresponds to ); W k is the convolution weight corresponding to the relative position k; is the relative position offset; σ is the nonlinear activation function ReLU.

[0044] S600, collecting RGB images of N viewing angles in the robot arm operation scene, and obtaining corresponding image features through ViT model encoding for each viewing angle RGB image: in, is the image feature of the nth perspective; I n is the RGB image of the nth viewing angle.

[0045] It should be noted that the N perspectives here generally refer to N third perspectives.

[0046] S700: Initialize the query vector Query g and Perform the cross attention mechanism operation and get: Among them, cross_attn is cross attention.

[0047] S800: query vector Query g Processed by a set of multi-layer perceptrons MLP, the updated Gaussian kernel parameters are obtained : in, They are the updated mean, scale, rotation vector and semantic category respectively.

[0048] S900, directly replace the original Gaussian kernel with the updated Gaussian kernel, and the new Gaussian kernel parameters It is expressed as: Will Embedded in a target voxel grid of size X×Y×Z.

[0049] S1000, for each new three-dimensional Gaussian, calculate the radius of its neighborhood according to its scale attribute, and recalculate the occupancy prediction result of each voxel grid with reference to formulas (2) and (4).

[0050] S1100, for each re-determined occupied voxel grid, calculate its semantic category corresponding label C label With the predicted value The cross entropy loss CE between: in, is the loss function; Represents a set of occupied voxel grids; the subscript label represents the label.

[0051] S1200: Update the model parameters through the loss function, and save the query vector Query after training. g And the weight parameters of Cross-Attn.

[0052] How to obtain the query vector Query g Based on the weight parameters of Cross-Attn, in this step, the image features are cross-attention calculated to obtain high-dimensional features based on three-dimensional Gaussian; it is expressed as: in, is the high-dimensional feature based on three-dimensional Gaussian at time t.

[0053] In step S3, the contact force distribution between the left and right jaws of the clamper and the operated object is sensed in real time by a force sensor to obtain a two-dimensional RGB tactile image sequence, and the tactile image features of the RGB tactile image sequence are extracted by the ViT model.

[0054] It is understandable that the embodiment of the present invention can pre-arrange the force sensor at the end of the clamp to support this step to obtain the RGB value through the force sensor, sense the distribution of the contact force between the end of the clamp and the operated object in real time, and transmit the two-dimensional RGB tactile image in real time.

[0055] Next, this step extracts the tactile image features of the RGB tactile image sequence through the ViT model; it is expressed as: in, is the tactile image feature at time t; They are the RGB tactile images of the left and right jaws of the gripper at time t, and the superscripts lta and rta represent the left and right tactile perception, respectively.

[0056] In step S4, the high-dimensional features and the tactile image features are respectively used as inputs of a resampler, and corresponding feature sequences are obtained and then concatenated.

[0057] like Figure 2 As shown, in this step, the high-dimensional features and the tactile image features are respectively used as inputs of the resampler, and the corresponding feature sequences are obtained and then spliced; the relevant steps specifically include: First, the high-dimensional features and the tactile image features are respectively input into the K and V fully connected layers of the resampler, and a set of learnable vectors q are input into the Q fully connected layer of the resampler: Among them, Q R ∈ is the query vector corresponding to the learnable parameters of the resampler, is the number of query vectors for the resampler, is the size of the hidden dimension, are the linear transformation matrices of keys and values, respectively, d g / ta Represents the dimension size of high-dimensional features or tactile image features; The transformed key and value vectors generated for the high-dimensional feature input respectively; The converted key and value vectors generated for the tactile image feature input, respectively; are feature sequences corresponding to high-dimensional features and tactile image features output by the resampler, respectively; softmax is the activation function; and the superscript T is the transposition symbol.

[0058] Next, concatenate the two obtained feature sequences: in, is the concatenation result at time t. The superscript g-ta indicates the fusion of Gaussian features and tactile features; concat is the concatenation operation.

[0059] In step S5, the splicing result and the language instruction are used as the input of the feature fusion decoder, and the LSTM strategy head is used to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper.

[0060] As mentioned above, the resampler is used to combine the Gaussian features and the tactile image features, output the compressed features and input them into the feature fusion decoder.

[0061] In this step, the feature fusion decoder converts the language instructions (in Figure 2 The feature fusion decoder output is used to predict the action of the gripper, so as to obtain the position change of the gripper and the next state of the gripper. The relevant steps include: First, the feature fusion process is performed by the feature fusion decoder and integrated with the language instruction to output ; The feature fusion process is expressed as: in, is the output of the lth cross attention layer at time t; Tanh is the hyperbolic tangent function; α∈ is a learnable parameter used to control the contribution of different features in the fusion process; A is the cross or self-attention operation; are the learnable parameters of the cross-attention layer respectively; They are the outputs of the lth and l+1th self-attention layers at time t (the first ); are the learnable parameters of the self-attention layer respectively; L is the sum of the number of alternating self-attention layers and cross-attention layers, and is an even number; N l is the number of language instructions.

[0062] It should be noted that in the fusion process, the input type of attention layer is determined according to the input of Q and K. If they are the same, it is the self-attention layer Self-Attn, otherwise it is the cross-attention layer Cross-Attn. Since the two are alternately superimposed, it is exemplary to set both to 24 layers, a total of 48 layers, and the self-attention layer Self-Attn can use the pre-trained parameters of the large language model mpt-1b-redpajama-200b and freeze it, and the cross-attention layer Cross-Attn can use the pre-trained parameters of OpenFlamingo and fine-tune them.

[0063] Since the feature vector output by the feature fusion decoder needs to be converted into the underlying control instructions in the end, in order to facilitate the conversion, this step then uses the LSTM strategy head to output X t Perform motion prediction to obtain the position change of the gripper and the next state of the gripper, such as the 7 degrees of freedom (DoF) change of the gripper posture and the gripper state; specifically expressed as: Among them, MaxPooling is the maximum pooling layer, For its output; h t 、h t-1 are the hidden states at time t and t-1 respectively; They are respectively the predicted position change of the gripper and the next state of the gripper, and the state refers to whether the left and right jaws of the gripper are open or closed.

[0064] In particular, in order to improve the pre-trained attention mechanism backbone and LSTM strategy head, the embodiment of the present invention adopts the maximum likelihood objective for imitation learning. The goal is to optimize the relative pose through regression loss (for example, using mean square error MSE loss) and classify the state of the gripper through classification loss (using binary cross entropy BCE loss): in, is the total loss function, are the posture change of the gripper demonstrated at time t and the next state of the gripper, respectively, and λgripper is the weight factor of the gripper loss.

[0065] In step S6, based on the current posture of the gripper and the posture change, a target posture is calculated to determine the moving direction and distance of the gripper.

[0066] It can be understood that this step is used to determine the moving direction and distance target posture of the gripper by calculation, so as to ensure that the system completes the input language instruction. In addition, since the model reasoning is performed in a time series, the gripper is always in an appropriate posture state at every moment.

[0067] Furthermore, considering that the multimodal large model is unpredictable and random when generating output, the operation instructions in the robot arm control are disordered, uncertain, and even potentially dangerous. This uncertainty may generate inappropriate action sequences in critical mission stages, causing unexpected or dangerous operations, limiting the application of large models in safety-sensitive scenarios, reducing operational efficiency, and increasing risks.

[0068] Correspondingly, the embodiment of the present invention innovatively proposes a dynamic feedback optimization control algorithm based on the safety area, aiming to solve the safety hazards in the above-mentioned machine learning model operation instructions.

[0069] Specifically, the proposed algorithm includes: (1) Design of posture controller: Based on the pose change, a first position error between the target position and the current position is defined: Where, e1 is the first position error; is the change of posture; is the target pose of the gripper at the target position to be calculated; is the current pose of the gripper at its current position.

[0070] A pose controller is used to calculate the control inputs to ensure that the gripper moves smoothly towards the target position: Among them, u1 is the output of the posture controller using the PD controller; k p1 , k d1 are the first proportional and first differential gains respectively; is the rate of change of the first position error.

[0071] (2) Design of safe area controller: like Figure 4 As shown, the safety area is set as a plane area with a height of h. Outside this area, the free movement of the entire robot arm is not restricted. In this area, once the gripper approaches the area boundary, the safety area controller outputs a control amount away from the area boundary: Where u2 is the output of the safety zone controller; k p2 , k d2 are the second proportional and second differential gains respectively; e2 is the second position error between the current position and the safety area, and the component of e2 in the Z direction , d z is the distance from the end of the gripper to the boundary of the region; is the rate of change of the second position error.

[0072] (3) Specify the evaluation function of the control objective: Note that the expression for J is based on the state error, which is usually quadratic. This approach is crucial to ensure the convexity of the optimization problem and the singularity of the solution. It is assumed that every state of the system can be predicted based on a custom model.

[0073] Based on the above conditions, J for each target can be designed as follows: Based on the square of the difference between the target pose and the current pose, the first evaluation function of the pose controller is designed: Among them, t0 is the initial time point, usually the time when the control process starts; T is the preset time span; ‖∙‖ is the modulus of the vector; and t is the time variable.

[0074] Based on the square of the difference between the distance from the end of the gripper to the boundary of the area and the height of the safety area, the second evaluation function of the safety area controller is designed: (4) Calculate the predicted evaluation gradient descent value and build a state feedback mechanism: Design saturation suppression function : in, To evaluate the gradient (the gradient direction in the state space is the fastest convergence direction), i takes 1 or 2; α is the upper limit of the gradient amplitude.

[0075] Use gradient descent method to obtain the input after single objective optimization: Among them, u i is the basic control input; λ is the learning rate; K α is the saturation suppression function The corresponding gain factor.

[0076] Therefore, the state feedback mechanism is constructed to make the robot state converge quickly. The single-objective controller after adding state feedback is expressed as: Among them, K fb is the state feedback gain matrix, is the target state, x i The current actual status.

[0077] like Figure 4 As shown, the final comprehensive output is expressed as: (5) Calculate a new safe target pose to ensure that the gripper avoids touching the region boundary: in, is the updated target pose.

[0078] So far, the embodiment of the present invention has completed the entire process of the intelligent robot safe operation method based on the fusion of three-dimensional Gaussian and tactile images.

[0079] Embodiment 2: The embodiment of the present invention provides an intelligent robot safety operating system based on the fusion of three-dimensional Gaussian and tactile images, wherein a gripper is installed at the end of the mechanical arm of the intelligent robot; the intelligent robot comprises: A collection module, used to collect RGB image sequences from the perspective of the robot hand and the third perspective, and extract image features of the RGB image sequences through a pre-trained ViT model; A calculation module, used for performing cross-attention calculation on the image features based on a pre-saved query vector and a cross-attention weight parameter to obtain a high-dimensional feature based on a three-dimensional Gaussian; an extraction module, configured to sense the contact force distribution between the left and right clamping jaws of the clamp and the operated object in real time through a force sensor, obtain a two-dimensional RGB tactile image sequence, and extract tactile image features of the RGB tactile image sequence through the ViT model; A sampling module, used to use the high-dimensional features and the tactile image features as inputs of a resampler respectively, obtain corresponding feature sequences and then perform splicing; A prediction module, used to use the splicing result and the language instruction as the input of the feature fusion decoder, and use the LSTM strategy head to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper; The determination module is used to calculate the target posture based on the current posture of the clamp and the posture change amount to determine the moving direction and distance of the clamp.

[0080] Embodiment 3: An embodiment of the present invention provides a storage medium storing a computer program for safe operation of an intelligent robot based on fusion of three-dimensional Gaussian and tactile images, wherein the computer program enables a computer to execute the method for safe operation of an intelligent robot as described in Example 1.

[0081] Embodiment 4: An embodiment of the present invention provides an electronic device, including: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a program for executing the intelligent robot safety operation method as described in Example 1.

[0082] It can be understood that the intelligent robot safety operating system, storage medium and electronic device based on the fusion of three-dimensional Gaussian and tactile images provided in the embodiments of the present invention correspond to the intelligent robot safety operation method based on the fusion of three-dimensional Gaussian and tactile images provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the intelligent robot safety operation method, and will not be repeated here.

[0083] In summary, compared with the prior art, the present invention has the following beneficial effects: 1. The embodiment of the present invention uses a three-dimensional Gaussian model to express the three-dimensional space of the robot arm operation scene, providing stable and accurate three-dimensional spatial features for the multi-modal large model of the robot arm. These features significantly enhance the robot arm's ability to perceive and understand the operating environment, thereby effectively improving the success rate and accuracy of the robot arm in performing various operating tasks.

[0084] 2. The embodiment of the present invention adds force sensors at both ends of the gripper to obtain tactile feedback and integrate it with three-dimensional spatial features. The combination of force information and three-dimensional spatial features enables the robot arm to more accurately perceive the force changes during the grasping process, optimize the grasping strategy, and improve the operation accuracy and stability. This technology enhances the ability of the robot arm to interact with objects during operation.

[0085] 3. The embodiment of the present invention solves the problem of uncontrollable output of large models by introducing a control method of safety area constraints. Combining real-time environmental data with safety area constraints ensures that the robot arm follows the inference trajectory and avoids entering dangerous areas during movement, thereby improving safety. This method enhances the adaptability of the robot arm in complex environments, avoids collision risks, and ensures the stability and success rate of task execution.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A safe operation method for an intelligent robot based on the fusion of three-dimensional Gaussian and tactile images, characterized in that: The end of the mechanical arm of the intelligent robot is equipped with a gripper; comprising: Collect RGB image sequences from the perspective of the robot hand and the third perspective, and extract image features of the RGB image sequences through the pre-trained ViT model; Based on the pre-saved query vector and the weight parameter of the cross attention, the cross attention calculation is performed on the image feature to obtain a high-dimensional feature based on the three-dimensional Gaussian; The contact force distribution between the left and right clamping jaws of the clamp and the operated object is sensed in real time by a force sensor to obtain a two-dimensional RGB tactile image sequence, and the tactile image features of the RGB tactile image sequence are extracted by the ViT model; The high-dimensional features and the tactile image features are respectively used as inputs of a resampler, and corresponding feature sequences are obtained and then spliced; The splicing result and the language instruction are used as the input of the feature fusion decoder, and the LSTM strategy head is used to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper; Based on the current posture of the gripper and the posture change, a target posture is calculated to determine the moving direction and distance of the gripper.

2. The intelligent robot safe operation method according to claim 1, characterized in that: In the process of calculating the target posture, a dynamic feedback optimization control algorithm based on the safety area is introduced, including: (1) Design of posture controller: Based on the pose change, a first position error between the target position and the current position is defined: Where, e1 is the first position error; is the change of posture; is the target pose of the gripper at the target position to be calculated; is the current pose of the gripper at the current position; A pose controller is used to calculate the control inputs to ensure that the gripper moves smoothly towards the target position: Among them, u1 is the output of the posture controller using the PD controller; k p1 , k d1 are the first proportional and first differential gains respectively; is the rate of change of the first position error; (2) Design of safe zone controller: The safety area is set as a plane area with a height of h. Outside this area, the free movement of the entire robot arm is not restricted. In this area, once the gripper approaches the area boundary, the safety area controller outputs a control amount away from the area boundary: Where u2 is the output of the safety zone controller; k p2 , k d2 are the second proportional and second differential gains respectively; e2 is the second position error between the current position and the safety area, and the component of e2 in the Z direction , d z is the distance from the end of the gripper to the boundary of the region; is the rate of change of the second position error; (3) Specify the evaluation function of the control objective: Based on the square of the difference between the target pose and the current pose, the first evaluation function of the pose controller is designed: Among them, t0 is the initial time point; T is the preset time span; ‖∙‖ is the modulus of the vector; t is the time variable; Based on the square of the difference between the distance from the end of the gripper to the boundary of the area and the height of the safety area, the second evaluation function of the safety area controller is designed: (4) Calculate the predicted evaluation gradient descent value and build a state feedback mechanism: Design saturation suppression function : in, To evaluate the gradient, i is 1 or 2; α is the upper limit of the gradient amplitude; Use gradient descent method to obtain the input after single objective optimization: Among them, u i is the basic control input; λ is the learning rate; K α is the saturation suppression function The corresponding gain factor; The state feedback mechanism is constructed to make the robot state converge quickly. The single-objective controller after adding state feedback is expressed as: Among them, K fb is the state feedback gain matrix, the subscript fb represents the state feedback, is the target state, x i is the current actual status; The combined output is: (5) Calculate a new safe target pose to ensure that the gripper avoids touching the region boundary: in, is the updated target pose.

3. The intelligent robot safe operation method according to claim 1 or 2, characterized in that: The process of obtaining the query vector and the cross-attention weight parameters includes: Select a fixed world coordinate system in the robot arm operation scene; Using P three-dimensional Gaussian distributions Represents the robot arm operation scene, each three-dimensional Gaussian is represented by a d-dimensional vector in the form of , where d = 10 + |C sem +2|, and m, s, r, and c are the mean, scale, rotation vector, and semantic category, respectively. is a set of real numbers, C sem is the number of semantic tags; initialization and are learnable vectors ; where D is the dimension of the learnable vector, and D>>d, Used to represent the parameter properties of Gaussian, Q j g is a learnable three-dimensional spatial feature, the superscript g represents the query vector associated with the Gaussian representation, and the subscript j represents the index of the Gaussian kernel; According to the initialized Gaussian kernel, voxelization is performed and the occupancy prediction result of each voxel grid is calculated by the following formula: in, It is the sum of the contributions of each Gaussian distribution to any position p in the selected world coordinate system, which is used to characterize the occupancy prediction result; Predicted distribution for 3D Gaussian occupancy The value at point p; is the neighboring Gaussian set of point p; exp is the exponential function; Σ is the covariance matrix; S=diag(s) represents the scale matrix, diag is the function for extracting diagonal elements, and R=q2r(q) is the function for converting the quaternion q used to represent the rotation of three-dimensional space into a rotation matrix; If the prediction result of any voxel grid occupancy is greater than a certain threshold, the voxel grid is considered to be occupied, that is, the position v of any voxel grid is expressed as: Denote the set of occupied voxel grid positions as: Where |∙| is the size of the set; M is the number of voxel grid locations used; The features corresponding to the occupied voxel grid positions are represented as a sparse tensor: where F(v) is the sparse tensor at any voxel grid position v; F S is the set of sparse tensors corresponding to the set S; The P Gaussian kernels correspond to P specific voxel grid locations {v1,v2,…,v p }∈S, define the reserved voxel set P S for: P S Each voxel grid position v in j Apply the sparse convolution kernel to obtain the output feature F'(v): in, is the relative position set of the convolution kernel; W k is the convolution weight corresponding to the relative position k; is the relative position offset; σ is the nonlinear activation function ReLU; In the robot arm operation scene, RGB images of N perspectives are collected. For each perspective RGB image, the corresponding image features are obtained through ViT model encoding: in, is the image feature of the nth perspective; I n is the RGB image of the nth viewing angle; Initialize the query vector Query g and Perform the cross attention mechanism operation and get: Among them, cross_attn is cross attention; The query vector Query g Processed by a set of multi-layer perceptrons MLP, the updated Gaussian kernel parameters are obtained : in, They are the updated mean, scale, rotation vector and semantic category respectively; The updated Gaussian kernel directly replaces the original Gaussian kernel, and the new Gaussian kernel parameters It is expressed as: Will Embedded into a target voxel grid of size X×Y×Z; For each new 3D Gaussian, the radius of its neighborhood is calculated based on its scale property, and the occupancy prediction results of each voxel grid are recalculated; For each re-determined occupied voxel grid, calculate its semantic category corresponding label C label With the predicted value The cross entropy loss CE between: in, is the loss function; Represents the set of occupied voxel grids; the subscript label represents the label; The model parameters are updated through the loss function, and the query vector Query is saved after training. g And the weight parameters of Cross-Attn.

4. The intelligent robot safe operation method according to claim 3, characterized in that: The image feature of the RGB image sequence is expressed as: in, is the image feature at time t, the superscript Indicates vision; are the RGB images of the robot arm hand view and the third view at time t respectively; ⊕ is the element-by-element addition symbol; Based on the pre-saved query vector and the weight parameter of the cross attention, the image feature is cross-attention calculated to obtain a high-dimensional feature based on a three-dimensional Gaussian; it is expressed as: in, is the high-dimensional feature based on three-dimensional Gaussian at time t.

5. The intelligent robot safe operation method according to claim 4, characterized in that: The tactile image features of the RGB tactile image sequence are extracted by the ViT model; expressed as: in, is the tactile image feature at time t; They are the RGB tactile images of the left and right jaws of the gripper at time t, and the superscripts lta and rta represent the left and right tactile perception, respectively.

6. The intelligent robot safe operation method according to claim 5, characterized in that: The step of taking the high-dimensional features and the tactile image features as inputs of a resampler respectively, obtaining corresponding feature sequences and then splicing them; comprises: The high-dimensional features and the tactile image features are respectively input into the K and V fully connected layers of the resampler, and a set of learnable vectors q are input into the Q fully connected layer of the resampler: Among them, Q R ∈ is the query vector corresponding to the learnable parameters of the resampler, is the number of query vectors for the resampler, is the size of the hidden dimension, are the linear transformation matrices of keys and values, respectively, d g / ta Represents the dimension size of high-dimensional features or tactile image features; The transformed key and value vectors generated for the high-dimensional feature input respectively; The converted key and value vectors generated for the tactile image feature input, respectively; are feature sequences corresponding to high-dimensional features and tactile image features output by the resampler, respectively; softmax is the activation function; the superscript T is the transposition symbol; Concatenate the two obtained feature sequences: in, is the concatenation result at time t. The superscript g-ta indicates the fusion of Gaussian features and tactile features; concat is the concatenation operation.

7. The intelligent robot safe operation method according to claim 6, characterized in that: The splicing result and the language instruction are used as the input of the feature fusion decoder, and the LSTM strategy head is used to perform action prediction on the output of the feature fusion decoder to obtain the position change of the gripper and the next state of the gripper; including: The feature fusion process is performed by the feature fusion decoder and integrated with the language instruction to output ; The feature fusion process is expressed as: in, is the output of the lth cross attention layer at time t; Tanh is the hyperbolic tangent function; α∈ is a learnable parameter used to control the contribution of different features in the fusion process; A is the cross or self-attention operation; are the learnable parameters of the cross-attention layer respectively; are the outputs of the lth and l+1th self-attention layers at time t respectively; are the learnable parameters of the self-attention layer respectively; L is the sum of the number of alternating self-attention layers and cross-attention layers, and is an even number; N l is the number of language instructions; Use the LSTM strategy to output X t Perform motion prediction to obtain the position change of the gripper and the next state of the gripper; expressed as: Among them, MaxPooling is the maximum pooling layer, For its output; h t 、h t-1 are the hidden states at time t and t-1 respectively; They are respectively the predicted position change of the gripper and the next state of the gripper, and the state refers to whether the left and right jaws of the gripper are open or closed.

8. An intelligent robot safety operating system based on the fusion of three-dimensional Gaussian and tactile images, characterized in that: The end of the mechanical arm of the intelligent robot is equipped with a gripper; comprising: A collection module, used to collect RGB image sequences from the perspective of the robot hand and the third perspective, and extract image features of the RGB image sequences through a pre-trained ViT model; A calculation module, used for performing cross-attention calculation on the image features based on a pre-saved query vector and a cross-attention weight parameter to obtain a high-dimensional feature based on a three-dimensional Gaussian; an extraction module, configured to sense the contact force distribution between the left and right clamping jaws of the clamp and the operated object in real time through a force sensor, obtain a two-dimensional RGB tactile image sequence, and extract tactile image features of the RGB tactile image sequence through the ViT model; A sampling module, used to use the high-dimensional features and the tactile image features as inputs of a resampler respectively, obtain corresponding feature sequences and then perform splicing; A prediction module, used to use the splicing result and the language instruction as the input of the feature fusion decoder, and use the LSTM strategy head to perform action prediction on the output of the feature fusion decoder to obtain the posture change of the gripper and the next state of the gripper; The determination module is used to calculate the target posture based on the current posture of the clamp and the posture change amount to determine the moving direction and distance of the clamp.

9. A storage medium, characterized in that: It stores a computer program for safe operation of an intelligent robot based on fusion of three-dimensional Gaussian and tactile images, wherein the computer program enables a computer to execute the safe operation method of an intelligent robot as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include programs for executing the intelligent robot safe operation method as described in any one of claims 1 to 7.

Images