Data acquisition method of data acquisition system

Abstract

The invention provides a data acquisition method of a data acquisition system, the data acquisition system comprises a hardware device and a software program, the hardware device comprises a control device, a mechanical arm, an execution terminal, an interaction device, a first binocular camera and a second binocular camera, and the method comprises the following steps: acquiring the hand change of an operator sensed by the interaction device; analyzing data of relative displacement and relative posture change of the hands of the operator; according to the data of the relative displacement and the relative pose change of the hands of the operator, the movement of the mechanical arm, the pose change of the execution tail end and the action execution of the execution tail end are controlled; and recording multi-source data in the data acquisition system in real time. Real-time and high-precision multi-source data acquisition is carried out on the task execution process of the mechanical arm and the execution tail end, the movement track and the operation environment of the mechanical arm can be monitored in real time, and the multi-source data can be efficiently recorded and processed.

Description

[0001] This divisional application is a divisional application of Chinese Patent Application No. 202411774446.X titled "Data Acquisition System and Its Data Acquisition Method" filed on December 05, 2024. Technical Field

[0002] The present invention relates to the technical field of large data models based on deep learning, and particularly to a data acquisition method for a data acquisition system. Background Art

[0003] With the rapid development of artificial intelligence and machine learning, large models have become an important force driving technological progress. However, traditional data acquisition methods face many challenges in terms of efficiency and quality, especially in the application of multi-modal large models. Multi-modal large models can integrate data from different sensors, such as vision, hearing, touch, etc., to achieve more complex and efficient decision-making and operations.

[0004] In a robotic system, efficient data acquisition is crucial. These systems usually need to obtain real-time data from multiple sensors to support the training and iteration of machine learning models. However, existing data acquisition methods often have problems such as slow speed and uneven data quality, resulting in an extended model training cycle and limiting the flexibility and accuracy of large models in practical applications.

[0005] Therefore, there is an urgent need for an improved data acquisition system and data acquisition method that can improve the speed and efficiency of multi-source data acquisition while ensuring data quality. Summary of the Invention

[0006] The present invention is proposed in view of the above technical problems, and provides a data acquisition method that focuses on the requirements of multi-modal large models, ensures that the system can quickly and efficiently obtain multi-source data, and performs real-time and high-precision multi-source data acquisition on the process of the robotic arm and the execution end executing tasks, and can collect real-time and accurate multi-source data for the training and optimization of large models, providing a solid data foundation for intelligent decision-making.

[0007] According to the first aspect of the embodiments of the present invention, a data acquisition system is provided, which includes a hardware device and a software program. Among them, the hardware device includes a control device, a robotic arm, an execution end, an interaction device, a first binocular camera, and a second binocular camera. The first binocular camera is located at the end of the robotic arm to obtain data of the execution end, and the second binocular camera is used to obtain global data of the data acquisition system;

[0008] The control device sets the software program. The interactive device is worn by an operator for operation. The control device receives the status information of the robotic arm, the first binocular camera, the second binocular camera, and the interactive device, controls the actions of the robotic arm and the execution end according to the hand data information of the operator collected by the interactive device, and records the multi-source data in the data acquisition system in real time. The multi-source data includes the data of the first binocular camera, the second binocular camera, the robotic arm, and the execution end.

[0009] According to the second aspect of the embodiments of the present invention, a data acquisition system as in the first aspect is provided. Among them, the hardware devices of the data acquisition system further include an ultrasonic ranging sensor, a photoelectric sensor, and a proximity switch sensor. The ultrasonic ranging sensor is located on the robotic arm for obtaining information about the surrounding environment; the photoelectric sensor is located on the robotic arm for environmental monitoring; the proximity switch sensor is located on the execution end for monitoring the position of the execution end gripper and monitoring objects in the surrounding environment; the multi-source data further includes the data of the ultrasonic ranging sensor, the photoelectric sensor, and the proximity switch sensor.

[0010] According to the third aspect of the embodiments of the present invention, a method for data acquisition using the data acquisition system of the first and second aspects is provided. The method includes:

[0011] Turn on the hardware devices and software program of the data acquisition system. The operator wears the interactive device for control operations, and the control device and the interactive device are kept under the same local area network;

[0012] The interactive device senses the changes of the operator's hand, analyzes the data of the relative displacement and relative posture changes of the operator's hand, and transmits the data information to the control device. The control device controls the movement of the robotic arm and the posture change of the execution end according to the change data of the relative displacement and relative pose of the operator's hand;

[0013] The operator controls the movement of the robotic arm and the action of the execution end according to the target position to complete an execution task. The control device records the multi-source data in the data acquisition system in real time according to the control process of the operator;

[0014] When the operator controls the robotic arm and the execution end to complete an execution task, the control device records the data corresponding to the current task at a preset frequency, and at the same time increments the recorded task count by one. Then, it determines whether the task count stored in the current data acquisition system has reached the preset task count. If the saved task count reaches the preset task count, the operator stops controlling the robotic arm and the execution end to perform tasks, and the data acquisition system completes data acquisition. When the task count recorded after completing an execution task has not reached the preset task count, the operator continues to operate from the beginning to control the robotic arm and the execution end to perform a task again. At the same time, the control device records the data during the second execution task in real time, and so on, until the task count recorded by the control device reaches the preset task count, that is, data acquisition is completed.

[0015] According to the fourth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the third aspect, wherein the interaction device senses the change process of the operator's hand, and the specific implementation method is as follows:

[0016] When the operator uses a VR headset, the 3D camera configured in the interaction device identifies and locates the hand data of the operator. The VR headset tracks various hand data including the wrist, palm, and fingers. By using the obtained hand data, dynamic gestures are set, and then the robotic arm and the execution end are controlled using the set dynamic gestures. The dynamic gestures include finger pinching, forward and backward movement, upward lifting, and downward compression. The control device controls the robotic arm and the execution end to complete corresponding actions according to different dynamic gestures.

[0017] According to the fifth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the third aspect, wherein the interaction device senses the change process of the operator's hand, and the specific implementation method is as follows:

[0018] When the operator uses a handle to sense the hand, an inertial measurement unit including an accelerometer and a gyroscope is provided inside the handle. The inertial measurement unit measures the linear acceleration and angular velocity of the handle, thereby providing data on the motion state of the handle.

[0019] According to the sixth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the fourth or fifth aspect, wherein in the process of the operator controlling the movement and actions of the robotic arm according to the target position, the implementation method for the control device to record various data is as follows:

[0020] After the hardware device and software program of the data acquisition system are turned on, the interface of the software program displays the first binocular camera image. The operator clicks on the first binocular camera image, and the control device controls the execution end on the robotic arm to move to the target position.

[0021] When the execution end on the robotic arm moves to the target position, the operator manually controls the execution end of the robotic arm to complete the execution actions, and the execution actions include grasping, shearing and other refined actions;

[0022] The control device records various data of the robotic arm, the execution end, the first binocular camera and the second binocular camera in real time.

[0023] According to the seventh aspect of the embodiments of the present invention, there is provided a data acquisition method as in the sixth aspect, wherein the implementation manner of the control device controlling the execution end on the robotic arm to move to the target position is as follows:

[0024] Unify the robotic arm coordinate system, the execution end coordinate system, the first binocular camera coordinate system and the second binocular camera coordinate system to obtain the conversion formula corresponding to the three-dimensional points from the first binocular camera coordinate system to the robotic arm coordinate system;

[0025] The operator clicks on the first binocular camera screen on the software program interface to obtain pixel points, and the first binocular camera gives three-dimensional point information in the first binocular camera coordinate system according to the pixel point positions;

[0026] The control device calculates the three-dimensional position information in the robotic arm base coordinate system according to the conversion formula, and controls the execution end on the robotic arm to move to the target position according to the position information.

[0027] According to the eighth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the seventh aspect, wherein in the process of the operator manually controlling the execution end of the robotic arm to complete the execution actions, the distance moved by the operator's hand is proportionally reduced to control the distance moved by the robotic arm, and the angle of the pose change of the operator's hand is proportionally reduced to control the pose change of the execution end.

[0028] According to the ninth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the seventh aspect, wherein in the process of the control device controlling the execution end on the robotic arm to move to the target position, when the distance from the target position is far, the control device controls the robotic arm to move at a faster speed; when the distance from the target position is close, the control device controls the robotic arm to gradually decelerate and move.

[0029] According to the tenth aspect of the embodiments of the present invention, there is provided a data acquisition method as in the ninth aspect, wherein the control device controls the robotic arm and the execution end to adopt three motion modes: only moving the position without changing the pose; or, only changing the pose without moving the position; or, moving the position and changing the pose simultaneously.

[0030] The beneficial effects of the present invention are as follows: By performing real-time and high-precision multi-source data acquisition on the process of the robotic arm and the execution end performing tasks, it is not only possible to monitor the motion trajectory and operating environment of the robotic arm in real time, but also to efficiently record and process multi-source data, thereby providing more accurate multi-source data for the training and optimization of large models, and further improving the performance and reliability of large models in various application scenarios.

[0031] Referring to the following description and the accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the ways in which the principles of the present invention can be adopted.

[0032] It should be understood that the embodiments of the present invention are not limited thereby. Features described and / or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, combined with the features in other embodiments, or replace the features in other embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings included are used to provide a further understanding of the present invention, which form a part of the specification, illustrate the preferred embodiments of the present invention, and together with the written description are used to explain the principles of the present invention, wherein the same reference numerals are always used for the same elements.

[0034] In the drawings:

[0035] Figure 1 is a flowchart of the data acquisition method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Referring to the accompanying drawings, through the following description, the foregoing and other features of the present invention will become apparent. In the description and the drawings, specific embodiments of the present invention are disclosed, which show some embodiments in which the principles of the present invention can be adopted. It should be understood that the present invention is not limited to the described embodiments.

[0037] The present invention first provides a data acquisition system, which includes a hardware device and a software program. The hardware device includes a control device, a robotic arm, an execution end, an interaction device, and two vision cameras. In a preferred embodiment of the present invention, the vision cameras are a first RGB binocular camera and a second RGB binocular camera. A software program is set in the control device. The interaction device can be a VR headset, a game controller, or a mixed reality (MR) device, etc. The operator selects and wears various interaction devices for operation according to different application scenarios. The execution end is installed at the end of the robotic arm to perform specific tasks. For example, when it is a picking robot, a picking execution end is installed at the end of the robotic arm to perform specific picking tasks, such as picking various fruits. Of course, in other application scenarios, different execution ends can be replaced according to needs, and the robotic arm drives different execution ends to perform different tasks. The first RGB binocular camera is located at the end of the robotic arm to obtain the picture data of the execution end, and the second RGB binocular camera is set at the corresponding position according to needs to obtain the picture data of the entire data acquisition system. Preferably, the data acquisition system of the present invention is provided with multiple sensors, and multiple sensors are added to the hardware device to collect richer multi-source data. Among them, an ultrasonic ranging sensor is located on the robotic arm to obtain information about the surrounding environment and help the robotic arm make more intelligent decisions; a photoelectric sensor is located on the robotic arm, which can convert photoelectric signals to judge the lighting conditions and is used for environmental monitoring to adjust the operation of each mechanism in the system; a proximity switch sensor is located on the execution end, which can monitor the position of the gripper of the execution end to ensure that it reaches the correct opening and closing state when grasping or releasing an object, or by monitoring objects in the surrounding environment, it can prevent the execution end from accidentally colliding, thereby improving the safety of operation.

[0038] During the data acquisition process, the operator wears the interaction device for operation, turns on the software program of the control device. The control device receives the status information of the robotic arm, the first RGB binocular camera, the second RGB binocular camera, and the interaction device, controls the actions of the robotic arm and the execution end to complete the execution task according to the hand data information of the operator collected by the interaction device, and records the multi-source data of the first RGB binocular camera, the second RGB binocular camera, the robotic arm, and the execution end in real time. In a preferred implementation manner, the control device will correspondingly also receive the status information of each sensor and record richer multi-source data including each sensor in real time.

[0039] Correspondingly, the present invention also provides a data acquisition method using the data acquisition system. Refer to Figure 1 , Figure 1 which is the flowchart of the data acquisition method of the present invention, as shown in the figure:

[0040] The data acquisition method of the present invention is specifically implemented as follows:

[0041] First, turn on the hardware devices and software programs of the data acquisition system. The operator wears an interactive device for control operations. The control device and the interactive device are on the same local area network. The hardware devices include a control device, a robotic arm, an execution end, an interactive device, and two vision cameras, such as a first RGB binocular camera and a second RGB binocular camera. In a preferred embodiment, the hardware devices of the data acquisition system also include various sensors to collect more abundant multi-source data covering various sensor data.

[0042] Then, the interactive device senses the hand changes of the operator, analyzes the data of the relative displacement and relative pose changes of the operator's hand, and transmits the data information to the control device. The control device controls the movement of the robotic arm, the pose change of the execution end, and the action execution of the execution end (such as the opening and closing of the gripper) according to the data of the relative displacement and relative pose changes of the operator's hand.

[0043] Finally, the operator controls the movement of the robotic arm and the actions of the execution end according to the target position to complete an execution task. At the same time, the control device records the multi-source data in the data acquisition system in real time according to the control process of the operator. These multi-source data include the data of the robotic arm, the execution end, the first RGB binocular camera, the second RGB binocular camera, and various sensors, completing one data acquisition. For example, when the application scenario is a specific application scenario where a picking robot picks tomato fruits, the operator wears an interactive device to control the robotic arm and its gripper at the execution end of the robot to complete a picking task of picking tomato fruits. During this process, the control device collects the activity data of each hardware device in real time.

[0044] When the operator controls the robotic arm and the execution end to complete an execution task, the control device records the data corresponding to the current task at a preset frequency, and at the same time increments the recorded task count by one. Then it judges whether the number of tasks stored in the current data acquisition system reaches the preset task number. If the saved task number reaches the preset task number, the operator stops controlling the robotic arm and the execution end to execute the task, and the data acquisition system completes the data acquisition. When the number of recorded tasks does not reach the preset task number after completing an execution task, the operator continues to operate from the beginning to control the robotic arm and the execution end to execute a task again, and at the same time the control device records the data during the second execution task in real time, and so on, until the number of tasks recorded by the control device reaches the preset task number, that is, the data acquisition is completed. After completing the data acquisition for the preset number of times, the data acquisition work is completed. The data collected up to the preset task number can be used as the original data for training this task model, and thus can be input for the training and optimization of the large model.

[0045] According to a preferred embodiment of the present invention, the interactive device senses the change process of the operator's hand. The specific implementation method is as follows:

[0046] When the operator uses the VR headset, the 3D camera configured in the interactive device identifies and locates the hand data of the operator. The hand movement features are composed of the fingertip direction vector and the palm normal vector. The VR headset tracks various hand data including the operator's wrist, palm, fingers, etc. By using the obtained hand data, multiple dynamic gestures are set, and then these set dynamic gestures are used to control the robotic arm and the execution end. Various dynamic gestures include finger pinching, forward and backward movement, upward lifting, and downward compression, etc. The control device controls the robotic arm and the execution end to complete corresponding actions according to different dynamic gestures.

[0047] According to another preferred embodiment of the present invention, when the operator uses the handle to sense the hand, an inertial measurement unit (IMU) including an accelerometer and a gyroscope is provided inside the handle. The inertial measurement unit measures the linear acceleration and angular velocity of the handle, thereby providing data on the movement state of the handle. After algorithm processing, high-precision and low-latency handle tracking can be achieved. This means that even when moving quickly, the data acquisition system can accurately reflect the actual position of the handle. In addition, the handle also includes physical buttons, which can control the robotic arm or the execution end to perform corresponding actions according to the operator's use of different buttons.

[0048] According to an embodiment of the present invention, during the process of the operator controlling the movement of the robotic arm, the present invention optimizes the data acquisition process and sets the manual collection to semi-automatic collection. Specifically, during the process of the operator controlling the movement and actions of the robotic arm according to the target position, the implementation method for the control device to record various data is as follows:

[0049] After the hardware device and software program of the data acquisition system are turned on, the interface of the software program displays the first RGB binocular camera image. The operator clicks on the first RGB binocular camera image, and the control device controls the execution end on the robotic arm to move near the target position;

[0050] When the execution end on the robotic arm moves to the target position, the operator manually controls the execution end of the robotic arm to complete the execution actions, and the execution actions include grasping, shearing and other refined actions;

[0051] The control device records various data of the robotic arm, the execution end, the first RGB binocular camera, and the second RGB binocular camera in real time. When a execution task is completed, if the robotic arm needs to return to the preset position, a specific button or gesture can be set to control the robotic arm to return to the preset position, and a data acquisition is completed.

[0052] In the above steps, the implementation method for the control device to control the execution end on the robotic arm to move to the target position is as follows:

[0053] Unify the robotic arm coordinate system, the execution end coordinate system, the first RGB binocular camera coordinate system, and the second RGB binocular camera coordinate system to obtain the conversion formula corresponding to the three-dimensional points from the first RGB binocular camera coordinate system to the robotic arm coordinate system;

[0054] The operator clicks on the first RGB binocular camera image on the software program interface to obtain pixel points, and the first RGB binocular camera gives the three-dimensional point information in the first RGB binocular camera coordinate system according to the pixel point positions;

[0055] The control device calculates the three-dimensional position information in the robotic arm base coordinate system according to the obtained conversion formula, and controls the execution end on the robotic arm to move to the target position according to this position information.

[0056] When the present invention starts and ends, the position movement of the robotic arm is set to automatic operation, which can improve the quality of the collected data and speed up the speed of data collection. At the same time, during long-distance movement, there will inevitably be slight jitters in the movement of the operator's arm and hand, and this slight jitter will cause jitter in the robotic arm control, and the jitter in each data collection is also random, which will affect the learning efficiency of the model. Therefore, the semi-automatic collection of the present invention maximally avoids the influence brought by the body jitter of the operator.

[0057] According to the preferred embodiment of the present invention, when the operator manually controls the robotic arm to complete the refined execution actions, the present invention optimizes the control method from three aspects as follows:

[0058] First, in the process of the operator manually controlling the execution end of the robotic arm to complete the execution actions, the distance of the operator's hand movement is proportionally reduced to control the distance of the robotic arm movement, and the angle of the operator's hand pose change is proportionally reduced to control the pose change of the execution end. For example, a proportional reduction of 1:5 is set. If the hand moves relatively by 20 cm, the robotic arm is controlled to move 4 cm; if the hand pose changes by 20°, the execution end pose changes by 4°, which can effectively avoid unnecessary jitters of the robotic arm caused by the slight jitter of the hand, thereby making the movement of the robotic arm more stable.

[0059] Second, during the actual data collection of the robotic arm, it is necessary to ensure that the robotic arm can accurately and smoothly reach the target position when grasping the target and perform the corresponding end actions. To achieve the smooth operation of the robotic arm, the present invention designs a strategy for dynamically adjusting the movement speed. Specifically, during the process of the control device controlling the end effector on the robotic arm to move to the target position, when the distance from the target position is far, the control device controls the robotic arm to move at a faster speed; when the distance from the target position is close, the control device controls the robotic arm to gradually decelerate to ensure high-quality data collection.

[0060] Third, the present invention focuses on optimizing the control process of the robotic arm. The control device controls the robotic arm and the end effector to adopt three motion modes: only moving the position without changing the posture; or, only changing the posture without moving the position; or, moving the position and changing the posture simultaneously.

[0061] To simulate the natural movement of the human arm in activities such as object grasping, the present invention designs two different moving speed modes. When the end effector is far from the target, the robotic arm quickly approaches the target at a higher speed; when approaching the target, it switches to a lower speed to slowly and accurately complete the target docking. Combining multiple motion modes, this design can better simulate the motion characteristics of joints such as the human arm and wrist, thereby improving the performance and stability of the robotic arm in actual operations.

[0062] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings. Many features and advantages of these embodiments are clear from this detailed description. In addition, since those skilled in the art can easily think of many modifications and changes, the embodiments of the present invention are not limited to the exact structures and operations illustrated and described, but may cover all suitable modifications and equivalents.

Claims

1. A data acquisition method for a data acquisition system, the data acquisition system comprising a hardware device and a software program, wherein, The hardware device includes a control device, a robotic arm, an execution end, an interaction device, a first binocular camera, and a second binocular camera, and is characterized in that the method includes: Obtaining data on the relative displacement and relative pose change of the operator's hand parsed by the interaction device through sensing the hand change of the operator; Controlling the movement of the robotic arm, the pose change of the execution end, and the action execution of the execution end according to the data on the relative displacement and relative pose change of the operator's hand; Real-time recording of multi-source data in the data acquisition system, where the multi-source data includes various data of the first binocular camera, the second binocular camera, the robotic arm, and the execution end.

2. The method according to claim 1, wherein The method further includes: When the operator controls the robotic arm and the execution end to complete an execution task, recording the data collected in the current task at a preset frequency, adding one to the recorded task count at the same time, and then determining whether the current task count reaches the preset task count. If the current task count reaches the preset task count, data collection is completed; if the current task count does not reach the preset task count, execute a task again until the current task count reaches the preset task count.

3. The method according to claim 1, wherein The interaction device includes a VR headset, and a 3D camera is configured in the VR headset; the data on the relative displacement and relative pose change of the operator's hand parsed by the interaction device through sensing the hand change of the operator includes: When the operator uses the VR headset, using the 3D camera to identify and locate various hand data of the operator and set dynamic gestures; The controlling the movement of the robotic arm, the pose change of the execution end, and the action execution of the execution end according to the data on the relative displacement and relative pose change of the operator's hand includes: Controlling the robotic arm and the execution end to complete corresponding actions according to different dynamic gestures.

4. The method according to claim 1, wherein The interaction device includes a handle, and the handle includes an inertial measurement unit; the data on the relative displacement and relative pose change of the operator's hand parsed by the interaction device through sensing the hand change of the operator includes: Measuring the linear acceleration and angular velocity of the handle through the inertial measurement unit to provide data on the motion state of the handle, and thus parsing the data on the relative displacement and relative pose change of the operator's hand.

5. The method according to claim 3, wherein The controlling the movement of the robotic arm, the pose change of the execution end, and the action execution of the execution end according to the data on the relative displacement and relative pose change of the operator's hand further includes: Controlling the moving distance of the robotic arm by proportionally reducing the distance of the operator's hand movement, and controlling the pose change of the execution end by proportionally reducing the angle of the operator's hand pose change.

6. The method according to claim 5, wherein The controlling the movement of the robotic arm, the pose change of the execution end, and the action execution of the execution end according to the data on the relative displacement and relative pose change of the operator's hand further includes: When the execution end is far from the target position, controlling the robotic arm to move at a faster speed; when the execution end is close to the target position, controlling the robotic arm to gradually decelerate and move.

7. The method according to claim 1, characterized in that, Control the robotic arm and the execution end to adopt three motion modes: only move the position without changing the attitude; or, only change the attitude without moving the position; or, move the position and change the attitude simultaneously.

8. The method according to any one of claims 1 to 7, characterized in that, The hardware devices of the data acquisition system further include an ultrasonic ranging sensor, a photoelectric sensor, and a proximity switch sensor. The ultrasonic ranging sensor is located on the robotic arm to obtain information about the surrounding environment; the photoelectric sensor is located on the robotic arm for environmental monitoring; the proximity switch sensor is located on the execution end to monitor the position of the execution end gripper and the objects in the surrounding environment; the multi-source data further includes the data of the ultrasonic ranging sensor, the photoelectric sensor, and the proximity switch sensor. The multi-source data further includes the data of the ultrasonic ranging sensor, the photoelectric sensor, and the proximity switch sensor.

9. A data acquisition method for a data acquisition system, the data acquisition system comprising a hardware device and a software program, wherein, The hardware devices include a control device, a robotic arm, an execution end, an interaction device, a first binocular camera, and a second binocular camera. It is characterized in that the method includes: According to the position where the operator clicks on the software program interface of the first binocular camera, control the execution end on the robotic arm to move to the corresponding target position. When the execution end on the robotic arm moves to the target position, the operator manually controls the execution end of the robotic arm to complete the execution action. Record the data of the robotic arm, the execution end, the first binocular camera, and the second binocular camera in real time.

10. The method according to claim 9, wherein The step of controlling the execution end on the robotic arm to move to the corresponding target position according to the position where the operator clicks on the software program interface of the first binocular camera includes: Unify the robotic arm coordinate system, the execution end coordinate system, the first binocular camera coordinate system, and the second binocular camera coordinate system to obtain the conversion formula corresponding to the three-dimensional points from the first binocular camera coordinate system to the robotic arm coordinate system. Obtain the pixel points obtained by the operator clicking on the software program interface, and give the three-dimensional point information in the first binocular camera coordinate system according to the pixel point position. Calculate the three-dimensional position information in the robotic arm coordinate system according to the conversion formula, and control the execution end on the robotic arm to move to the target position according to the three-dimensional position information.

Images