Fusion of tactile and visual robotic arm methods, systems, devices, and storage media

Abstract

The present application relates to the field of ocean engineering, in particular to a method, system, device and storage medium for fusing somatosensory and vision of a mechanical arm. The method for controlling the mechanical arm fusing somatosensory and vision comprises the following steps: obtaining target angle output by the mechanical arm, vision data and somatosensory device output data; and calculating the incremental change of the end position of the mechanical arm after each recursion according to the vision data and the somatosensory device output data by using adaptive weights. The present application fuses vision and somatosensory into the underwater mechanical arm, improves the degree of human-computer interaction, realizes the unification of vision and mechanical movement, and uses the somatosensory device to capture more information of the operator's body for control, so that the multi-degree-of-freedom mechanical arm movement is more in line with the human arm movement, which reduces the control difficulty.

Description

Technical Field

[0001] This invention relates to the field of marine engineering, and more specifically to robotic arm methods, systems, devices, and storage media that integrate haptic and visual sensing. Background Technology

[0002] The ocean is rich in resources, and human exploration of it has continued from ancient times to the present day. Underwater robots serve as important underwater research platforms for exploring the marine environment and developing marine technology. Underwater robotic arms are commonly used equipment on underwater robots, with wide applications in underwater heavy engineering infrastructure, confined space exploration, military mine clearance, marine life removal, and marine ecological protection. Among these applications, grasping is the core action of the underwater robotic arm, which requires accurate and rapid operation. Therefore, the efficiency and precision of the underwater robotic arm control system have become key issues that need to be addressed in this stage of underwater operations.

[0003] Currently, most underwater robotic arm control technologies on the market employ a master-slave control scheme. Operators manually control the robotic arm's movement via remote controls or host computers, leading to a series of problems such as high operational difficulty, low control efficiency, and reliance on manual operation for multi-joint robotic arms. As the requirements for underwater engineering operations increase, underwater robotic arms are becoming more complex in structure and more functional. Furthermore, operating via a host computer screen is not only slow and lacks human-machine interaction, but the two-dimensional image can also cause visual biases, leading to reduced accuracy or misoperation. In recent years, with the development of computer vision technology, underwater robotic arm control systems have gained a stronger ability to perceive their surroundings. Integrating positional information from visual measurements for grasping operations, and achieving hand-eye coordination or automatic control to assist manual operation, is the mainstream development direction for improving underwater robotic arm control. Summary of the Invention

[0004] In view of the problems mentioned in the prior art, the present invention proposes a robotic arm method, system, device and storage medium that integrates somatosensory perception and vision to solve the problems in the background art.

[0005] The present invention provides a robotic arm control method integrating body sensing and vision, comprising the following steps: S1: Acquire the target angle, visual data, and motion sensing device output data from the robotic arm; S2: Based on visual data and data output from motion sensing devices, adaptive weights are used to calculate the increment of the robotic arm's end-effector pose change after each recursion. S3: Update the real-time pose of the robotic arm end effector based on the incremental change in the pose of the robotic arm end effector in each recursive step. S4: Determine whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, return to step S2.

[0006] Preferably, the target angle output by the robotic arm in step S1 The process is as follows: the camera in the robotic arm identifies and measures the distance to the target object, and obtains the three-coordinate position data of the target position relative to the camera. Through coordinate system transformation, the position data collected by the camera is converted into the position data of the robotic arm coordinate system. The three-coordinate position data of the end of the robotic arm should coincide with it. Substituting it into the inverse kinematics solution, the target angle output by the robotic arm when the end of the robotic arm reaches the target position can be obtained.

[0007] Preferably, visual data is acquired in step S1. The target angle output by the robotic arm The difference between the real-time pose Ψ of the robotic arm end effector and the actual pose.

[0008] Preferably, in step S1, the data output by the motion sensing device is acquired. This includes the output angle of the robotic arm and the control angles of the robotic arm's upper and lower arms; The process of obtaining the output angle of the robotic arm is as follows: According to the angle of human hand rotation robotic arm rotation angle The proportionality coefficient is defined by the following formula:

[0009] The output angle of the robotic arm is calculated based on the proportional coefficient and the angular velocity corresponding to the output angle of the motion sensing device. The calculation formula is as follows:

[0010] In the formula: This is the angular velocity corresponding to the output angle of the motion sensing device; b = 1 / t0, where t0 is the serial port data transmission time interval. To obtain the difference between two adjacent angles; The process of obtaining the control angles of the robotic arm's forearm and upper arm is as follows: Establish motion models for the upper and lower arms of the robotic arm; Define the angle between the upper arm and the perpendicular line passing through the center of the circle as: The angle between the upper arm and the horizontal position is The data acquisition glove's pitch angle is defined as pitch, and the angle between the upper and lower arms is defined as... 3. The calculation formula is as follows:

[0011] In the formula: 3 is the angle between the forearm and upper arm; This is the increment of the angle between the upper arm and the perpendicular line passing through the center of the circle; This is the increment of the angle between the upper arm and the horizontal position.

[0012] Preferably, the formula for calculating the increment of the robotic arm's end-effector pose change after each recursion using adaptive weights in step S2 is as follows:

[0013] In the formula: This represents the increment of the end-effector pose change in each recursive iteration. This represents the difference in output data from the motion sensing device; This represents the adaptive weight that adjusts the ratio of the output data from the motion-sensing device to the visual data.

[0014] Preferably, the adaptive weights The calculation formula is as follows:

[0015] In the formula: Ψ represents the initial pose data of the robotic arm's end effector at the start of a grasping operation; Ψ represents the real-time pose data of the robotic arm's end effector. This represents the target angle output by the robotic arm.

[0016] Preferably, the calculation formula for updating the real-time pose of the robotic arm's end effector in step S3 is as follows: Ψ t =Ψ t-1 +

[0017] In the formula: Ψ t This represents the real-time pose data of the robotic arm's end effector. Ψ represents the increment of pose change at the end effector of the robotic arm in each recursive iteration; t-1 This represents the last real-time pose data of the robotic arm's end effector.

[0018] A robotic arm control system that integrates motion sensing and vision includes: The data acquisition module acquires the target position of the robotic arm, visual data, and data output from the motion sensing device. The calculation module uses adaptive weights to calculate the increment of the robotic arm's end-effector pose change after each recursion, based on visual data and data output from the motion sensing device. The update module updates the real-time pose of the robotic arm end effector based on the incremental change in pose of the robotic arm end effector with each recursive iteration. The judgment module determines whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, it returns to the calculation module for calculation.

[0019] Compared with the prior art, the present invention achieves the following technical effects: This invention integrates vision and motion sensing into an underwater robotic arm, achieving a unity of vision and mechanical movement. The use of motion sensing devices captures more information about the operator's body for control, making the multi-degree-of-freedom robotic arm movement more closely resemble the human arm's movements. This reduces control difficulty while improving the human-machine interface of the underwater robotic arm control. Due to the integration of visual information, the robotic arm control system can lock onto the target object's position, which reduces the time required for underwater robotic arm grasping operations and improves operational efficiency. Simultaneously, based on the information provided by vision, the robotic arm's posture moves towards a successful grasping posture, and ultimately each joint is fixed at the target output angle, improving grasping accuracy. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a diagram showing the correspondence between the output angle of the motion-sensing device and the robotic arm in this invention. Figure 3 This is a comparison diagram of the inventor's arm rotation angle and the robotic arm movement angle; Figure 4 This is a schematic diagram of the upper and lower arm motion model of the present invention; Figure 5 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0022] This invention provides a robotic arm control method that integrates body sensing and vision, such as... Figure 1 As shown, it includes the following steps: S1: Acquire the target position of the robotic arm, visual data, and output data from the motion sensing device; S2: Based on visual data and data output from motion sensing devices, adaptive weights are used to calculate the increment of the robotic arm's end-effector pose change after each recursion. S3: Update the real-time pose of the robotic arm end effector based on the incremental change in the pose of the robotic arm end effector in each recursive step. S4: Determine whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, return to step S2.

[0023] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: Step 1: Acquire data output from the motion-sensing device , This includes the output angle of the robotic arm and the control angles of the robotic arm's upper and lower arms.

[0024] The motion-sensing data glove used in this invention can collect the Euler angles of the upper arm, forearm, and hand of the robotic arm, as well as the flexion of the five fingers. However, the degrees of freedom of the robotic arm are actually much smaller than those of the human hand, and due to the manufacturing method of the motion-sensing device itself, the data output by the device cannot be directly mapped to the robotic arm for control. To clarify the correspondence between the output angles of the motion-sensing device and the control angles of the robotic arm, this invention specifies which control information is integrated into each joint, and the correspondence is as follows: Figure 2 As shown.

[0025] from Figure 2 As can be seen, angle 1 has a relatively large rotation range, and the yaw angle of the forearm module can better meet the requirements. Angle 2 almost completely corresponds to the upper arm pitch angle and does not need to be integrated with other angle information. Angle 3 is in a special position, being the angle between the upper and lower arm. This angle cannot be directly correlated with the angle output by any single device and must be integrated with the angle information of the upper and lower arms to be reflected. Angle 4 is the wrist rotation angle, which can be directly correlated with the row angle on the glove. Angle 5 represents the claw opening and closing angle, which allows for dexterous operation through finger control.

[0026] To ensure that the robotic arm does not interfere with the robot body during operation, the corresponding motion angles between the human hand and the robotic arm are as follows: Figure 3 As shown.

[0027] The unit angle of human arm rotation is set as The angle of rotation of the robotic arm is Here, a scaling factor is defined. This represents the magnitude conversion relationship between the movement angle of the robotic arm and the movement angle of the human arm.

[0028] Since the range of rotation angles of a human arm is greater than the range of motion of a robotic arm, the two cannot be directly mapped proportionally. Therefore, we choose... At that time, first use according to the measurement results The value should be reduced from the measurement result during actual use; to ensure the safety of the movement, It must be less than 1; A smaller value indicates that one unit of robotic arm rotation angle corresponds to a larger number of arm rotation angles, which means that the motion control of the robotic arm is more precise. Too small a size will also result in slow movement speed of the robotic arm and unclear response to control commands. The appropriate size needs to be selected based on the actual structure when using it.

[0029] In this embodiment, the device that collects angles in the motion sensing device is essentially a gyroscope. The angles collected by the gyroscope cannot be mapped proportionally using conventional methods because this would cause positioning problems. The positioning problem refers to the fact that when the robot starts working, the operator's face while wearing the motion-sensing data glove varies, resulting in different initial data from the glove. The yaw angle is particularly affected, changing with each startup, making it impossible to obtain a fixed angle and accurately control the movement at angle 1. To solve this problem, angular velocity control is employed.

[0030] Angular velocity derivation:

[0031] In the formula: Represents angular velocity; This represents the angle data recorded by the motion sensing device at time t; It represents the difference between two adjacent angles θ recorded in the time series; This represents the time interval between recording two angles.

[0032] Angular velocity control essentially involves differentiating the detected angle, multiplying the acquired angular velocity by a proportional parameter, and then outputting the control. In embedded systems, this differentiation process is approximated as discretization. As shown in the formula above, the final angular velocity is actually an angle difference divided by a time difference. The time difference is represented in the system as the serial port data reception period, which is also a constant. The final simplified formula is as follows:

[0033] In the formula: Angular velocity can be represented as a parameter multiplied by the angle difference.

[0034] The final output angle of the robotic arm is as follows:

[0035] In the formula: The angular velocity corresponding to the output angle of the motion sensing device; b = 1 / t0, where t0 is the serial port data transmission time interval; This is the difference between two adjacent angles.

[0036] As can be seen from the formula, the angle output by the robotic arm is independent of the original angle when the motion sensing device is turned on, and is also independent of the angle data displayed in real time by the gyroscope. It is only related to the magnitude of the angular velocity of the motion sensing device.

[0037] This embodiment Figure 2Joint 3 requires two control variables for coordinated control. This is significant because it allows the angle between the upper and lower arms to better match human movement, as a single angle cannot achieve a high level of human-computer interaction. In this embodiment, the upper arm rotation axis is used as the origin, and the upper arm length as the radius, to establish the motion model of the robotic arm's upper and lower arms. The angle between the upper arm and the perpendicular line passing through the center of the circle is defined as... The angle between the upper arm and the horizontal position is defined as The angle between the forearm and upper arm is defined as Models such as Figure 4 As shown: Figure 4 In the model, the red line represents the starting position, the blue line represents the forearm position after upper arm angle compensation, and the yellow line represents the actual position reached after adding pitch angle control. Compensated motion means that when the upper arm moves, the forearm maintains its original angle relative to the ground, a movement consistent with the motion of a human wearing a data glove. Geometric relationships reveal... and Relationship between them:

[0038] This can be simplified as follows:

[0039] In the formula: It is generated from the pitch angle data collected by the boom module.

[0040] By overlaying the forearm pitch angle data onto the compensated horizontal position, the current accurate position of the human arm can be obtained, and the control angle of the forearm-to-arm angle can be obtained:

[0041] In the formula: The angle between the forearm and upper arm; This is the increment of the angle between the upper arm and the perpendicular line passing through the center of the circle; This is the increment of the angle between the upper arm and the horizontal position.

[0042] The output angle of the robotic arm and the control angle between the upper and lower arms obtained above constitute the output data of the motion sensing device. .

[0043] Step 2: Obtain the target position of the robotic arm

[0044] In this embodiment, the target position of the robotic arm The specific process is as follows: The underwater binocular camera in the robotic arm control system identifies and measures the distance to the target object. The ranging information includes the target object's three-coordinate position data relative to the binocular camera. Kinematic calculations are performed on the actual robotic arm configuration to obtain the inverse kinematics solution for actual use. Through coordinate system transformation, the position data collected by the camera is converted into position data in the robotic arm's coordinate system. Assuming the three-coordinate position data of the robotic arm's end effector coincides with this, and substituting it into the inverse kinematics solution, the target angle output by each joint of the robotic arm when the end effector reaches the target object's position can be calculated. .

[0045] Step 3: Acquire visual data

[0046] This embodiment acquires visual data. The target position of the robotic arm The difference between the real-time pose Ψ of the robotic arm end effector and the actual pose.

[0047] Step 4: Integration of Motion and Vision The purpose of this fusion method is to improve the efficiency of grasping operations by converting simple master-slave control into semi-automatic control. The problem it solves is how to integrate information from both haptic and visual modalities and coordinate the two to control grasping.

[0048] For the data acquired by the vision system, based on the inverse kinematics of a four-degree-of-freedom robotic arm, the positioning information can be converted into the target output angle that each robotic arm joint should achieve when completing the grasping action. Because the camera in the vision system is constantly receiving and updating data, and the motion sensing device is also constantly outputting angle differences to adjust the robotic arm's output angle, the entire system is in a state of real-time change. Its entire schematic diagram is shown below. Figure 5 As shown, the following is based on Figure 5 Explain the specific steps and procedures.

[0049] Step 41. Based on the target position data acquired by the camera, obtain the target angle output by the robotic arm. The target angle output by the robotic arm The initial pose data of the robotic arm's end effector at the start of the grasping operation The difference in visual data ; Step 42. Obtain Then perform the first judgment, if That means the robotic arm has reached the target position, and the algorithm will stop running; otherwise, a new recursive calculation will be performed. Step 43. Combine the data obtained from Step 1 and Step 2 from the motion-sensing device input. The visual data obtained in step 43 Substitute into the following formula to calculate:

[0050] In the formula: This represents the increment of the end-effector pose change in each recursive iteration. This represents the difference in control data input from the motion sensing device; This represents the adaptive weight that adjusts the ratio of the output data from the motion-sensing device to the visual data.

[0051] Step 44. Update the real-time pose of the robotic arm end effector according to the incremental change in pose of the recursive robotic arm end effector, and use the pose obtained in step 43. Substitute into the following formula to calculate: Ψ t =Ψ t-1 +

[0052] In the formula: Ψ t This represents the real-time pose data of the robotic arm's end effector. Ψ represents the increment of pose change at the end effector of the robotic arm in each recursive iteration; t-1 This represents the last real-time pose data of the robotic arm's end effector.

[0053] Step 45. Output the target angle from the robotic arm. The difference between the data and Ψt obtained in step 44 yields the updated visual data. .

[0054] Step 46. Update the visual data Make a second judgment, if that If the robotic arm has reached the target position, the algorithm will stop running; otherwise, it will perform a new recursive calculation and return to step 43 to recalculate.

[0055] The most crucial part of the formula in step 43 is the adaptive weight. Its purpose is to better integrate visual and haptic information for coordinated grasping operations. Secondly, it maintains significant subjectivity in control, prioritizing haptic feedback over visual input. Furthermore, it automatically adjusts its size based on actual data, preventing a surge in the drive mechanism's output that could damage it. The specific formula is as follows:

[0056] In the formula: Ψ represents the initial pose data of the robotic arm's end effector at the start of the grasping operation; Ψ represents the real-time pose data of the robotic arm's end effector. This represents the target angle output by the robotic arm.

[0057] This represents the initial pose data of the robotic arm's end effector at the start of a grasping operation. We can see that `w` is composed of two parts multiplied together. For the first part, since the denominator is a constant, its weight increases as the robotic arm's end effector pose gets closer to the target pose (and further from the initial pose), indicating a greater proportion of visual information. The closer the robotic arm is to the target position, the more precise the required operation becomes, and the operational difficulty increases rapidly. Therefore, the increased operational difficulty can be handled by visual information. Simultaneously, the farther the initial pose of the robotic arm is from the target pose, the more significant the control adjustments required. Manual control is faster in this regard. This part reflects another characteristic: long-range control is primarily based on haptic feedback, while close-range control is primarily based on visual perception.

[0058] The second part of the formula serves the purpose of: the closer the initial pose of the robotic arm is to the target position, the faster the algorithm converges to the target pose. For grasping tasks that are very close to the target pose, visual information can be used to complete the task quickly, reducing the reliance on active control. However, for grasping tasks that are close to the target pose, active control is still needed for rapid movement to improve overall efficiency.

[0059] Based on the same inventive concept, this embodiment of the invention also provides a robotic arm control system that integrates somatosensory perception and vision. Since the principle of solving the problem by this robotic arm control system that integrates somatosensory perception and vision is similar to the aforementioned robotic arm control method that integrates somatosensory perception and vision, the implementation of this robotic arm control system that integrates somatosensory perception and vision can refer to the implementation of the robotic arm control method that integrates somatosensory perception and vision, and the repeated parts will not be described again.

[0060] In specific implementation, the robotic arm control system integrating body sensing and vision provided in this embodiment of the invention specifically includes: The data acquisition module acquires the target position of the robotic arm, visual data, and data output from the motion sensing device. The calculation module uses adaptive weights to calculate the increment of the robotic arm's end-effector pose change after each recursion, based on visual data and data output from the motion sensing device. The update module updates the real-time pose of the robotic arm end effector based on the incremental change in pose of the robotic arm end effector with each recursive iteration. The judgment module determines whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, it returns to step S2.

[0061] Accordingly, embodiments of the present invention also provide a robotic arm control device that integrates somatosensory perception and vision, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the robotic arm control method that integrates somatosensory perception and vision as provided in embodiments of the present invention.

[0062] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0063] Accordingly, embodiments of the present invention also provide a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the robotic arm control method integrating somatosensory perception and vision as provided in the embodiments of the present invention.

[0064] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0065] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0066] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0067] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] The above provides a detailed description of the robotic arm control method, system, device, and storage medium that integrates body sensing and vision provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A robotic arm control method integrating haptic feedback and vision, characterized in that, Includes the following steps: S1: Acquire the target angle, visual data output by the robotic arm, and data output by the motion sensing device; the motion sensing device output data This includes the output angle of the robotic arm and the control angles of the robotic arm's upper and lower arms; S2: Based on visual data and data output from the motion sensing device, an adaptive weighting method is used to calculate the increment of the robotic arm's end-effector pose after each recursion; the formula for calculating the adaptive weighting is as follows: In the formula: This represents the increment of the end-effector pose change in each recursive iteration. This represents the difference in output data from the motion sensing device; An adaptive weight representing the ratio of the output data from the motion-sensing device to the visual data; Representing visual data, the target angle output by the robotic arm. The difference between the real-time pose Ψ of the robotic arm end effector is obtained; Adaptive weights The calculation formula is as follows: In the formula: This represents the initial pose data of the robotic arm's end effector at the start of a grasping operation; Ψ represents the real-time pose data of the robotic arm's end effector. This represents the target angle output by the robotic arm; S3: Update the real-time pose of the robotic arm end effector based on the incremental change in the pose of the robotic arm end effector in each recursive step. S4: Determine whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, return to step S2.

2. The robotic arm control method integrating somatosensory perception and vision according to claim 1, characterized in that, The target angle output by the robotic arm in step S1 The process is as follows: the camera in the robotic arm identifies and measures the distance to the target object, and obtains the three-coordinate position data of the target position relative to the camera. Through coordinate system transformation, the position data collected by the camera is converted into the position data of the robotic arm coordinate system. The three-coordinate position data of the end of the robotic arm should coincide with it. Substituting it into the inverse kinematics solution, the target angle output by the robotic arm when the end of the robotic arm reaches the target position can be obtained.

3. The robotic arm control method integrating somatosensory perception and vision according to claim 1, characterized in that, The process of obtaining the output angle of the robotic arm is as follows: According to the angle of human hand rotation robotic arm rotation angle The proportionality coefficient is defined by the following formula: The output angle of the robotic arm is calculated based on the proportional coefficient and the angular velocity corresponding to the output angle of the motion sensing device. The calculation formula is as follows: In the formula: This is the angular velocity corresponding to the output angle of the motion sensing device; b = 1 / t0, where t0 is the serial port data transmission time interval. To obtain the difference between two adjacent angles; The process of obtaining the control angles of the robotic arm's forearm and main arm is as follows: Establish motion models for the upper and lower arms of the robotic arm; Define the angle between the upper arm and the perpendicular line passing through the center of the circle as: The angle between the upper arm and the horizontal position is The data acquisition glove's pitch angle is defined as pitch, and the angle between the upper and lower arms is defined as...

3. The calculation formula is as follows: In the formula: 3 is the angle between the forearm and upper arm; This is the increment of the angle between the upper arm and the perpendicular line passing through the center of the circle; This is the increment of the angle between the upper arm and the horizontal position.

4. The robotic arm control method integrating somatosensory perception and vision according to claim 1, characterized in that, The calculation formula for updating the real-time pose of the robotic arm's end effector in step S3 is as follows: P t =Ψ t-1 + In the formula: Ψ t This represents the real-time pose data of the robotic arm's end effector. Ψ represents the increment of pose change at the end effector of the robotic arm in each recursive iteration; t-1 This represents the last real-time pose data of the robotic arm's end effector.

5. A robotic arm control system integrating motion sensing and vision, characterized in that, include: The data acquisition module acquires the target position of the robotic arm, visual data, and data output from the motion sensing device; the motion sensing device output data... This includes the output angle of the robotic arm and the control angles of the robotic arm's upper and lower arms; The calculation module, based on visual data and data output from the motion sensing device, uses adaptive weighting to calculate the increment of the robotic arm's end-effector pose change after each recursion; the formula for the adaptive weighting calculation is as follows: In the formula: This represents the increment of the end-effector pose change in each recursive iteration. This represents the difference in output data from the motion sensing device; An adaptive weight representing the ratio of the output data from the motion-sensing device to the visual data; Representing visual data, the target angle output by the robotic arm. The difference between the real-time pose Ψ of the robotic arm end effector is obtained; Adaptive weights The calculation formula is as follows: In the formula: This represents the initial pose data of the robotic arm's end effector at the start of a grasping operation; Ψ represents the real-time pose data of the robotic arm's end effector. This represents the target angle output by the robotic arm; The update module updates the real-time pose of the robotic arm end effector based on the incremental change in pose of the robotic arm end effector with each recursive iteration. The judgment module determines whether the target position of the robotic arm is consistent with the updated end-effector pose. If so, the robotic arm reaches the target position; otherwise, it returns to the calculation module to recalculate.

6. A robotic arm control device integrating motion sensing and vision, characterized in that, It includes a processor and a memory, wherein the processor implements the robotic arm control method integrating somatosensory and visual perception as described in any one of claims 1 to 4 when executing a computer program stored in the memory.

7. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the robotic arm control method integrating somatosensory and visual perception as described in any one of claims 1 to 4.

Images