High-precision control method and system for robotic arms based on MR and 3D point cloud modeling

Abstract

The present invention relates to the field of robotic arm control technology, and discloses a high-precision control method and system for a robotic arm based on MR and three-dimensional point cloud modeling. By acquiring real-time environmental data of three-dimensional point clouds and overall ocean current prediction associated data of a target operating area, regional ocean currents that may be formed in the target operating area are estimated. The completion time limit of the operating task and the operating accuracy of the operating task are used as a set of constraint conditions, and the minimum number of optimized updates of the operating action is used as the optimization goal. The operating execution strategy of the target operating area is optimized and solved. The execution sequence of the operating points in the operating execution strategy is used as guidance information for the robotic arm operation control, and is integrated with the picture data of the target operating area constructed using the real-time environmental data of the three-dimensional point cloud to generate a virtual scene picture of the target operating area. The operating force of the robotic arm when performing different operating actions at each operating point is adjusted in real time, thereby improving the operating accuracy and efficiency of remote operation of the robotic arm in marine operating scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm control, and in particular to a high-precision control method and system for a robotic arm based on MR and three-dimensional point cloud modeling. Background Art

[0002] Robotic arm teleoperation refers to a technology that allows a human operator, remotely located away from the robot arm, to manipulate the robot arm in real time to complete specific tasks using communication systems and human-machine interface devices. Its core is to establish a closed-loop control system between "human-communication link-robot arm," enabling the operator to perceive the remote environment and precisely control the robot arm's movements. The development and application of robotic arm teleoperation technology can transcend physical limitations, expand the scope of human activity, improve the accuracy and efficiency of complex tasks, enhance operational safety, and replace human operators in performing complex tasks in high-risk environments.

[0003] However, in actual applications, the application of special scenarios (such as deep-sea oil extraction, deep-sea mineral resource exploration, underwater archaeology, etc. in marine operation scenarios) makes the remote operation of the robotic arm still limited: First, when controlling the robotic arm to perform corresponding operation tasks in the ocean, due to the influence of the overall ocean currents and regional ocean currents, the robotic arm will perform actions such as base movement, robotic arm rotation and robotic arm rotation around the axis. There will be accuracy deviations, which makes it easy for the operator to remotely control the robotic arm and encounter problems such as motion offset and low control accuracy, affecting the execution efficiency of marine operation tasks; second, due to the influence of changes in the overall ocean currents and regional ocean currents, the operator may make adjustments to adaptive operation actions for marine operation tasks. The section standards are inconsistent, that is, the overall ocean currents and regional ocean currents at different operating locations and at different operating times may be different, which will result in different deviations when performing the same action at two operating points. Operators need to constantly trial and error based on experience to try to find the optimal control force for each operating point at the current time point, resulting in great control difficulty, low control accuracy and long operation time; thirdly, adjusting the operating power of the actuator corresponding to each operating action of the robotic arm is a method that can theoretically solve the accuracy deviation of the robotic arm's marine operating action, but how to accurately adjust the operating power of the actuator and minimize the number of adjustments as much as possible to avoid possible mechanical failure problems and operator adaptation problems is a difficulty.

[0004] Therefore, how to improve the operational accuracy and efficiency of remote operation of robotic arms in marine operation scenarios, while solving problems such as motion offset and low control accuracy caused by overall and regional ocean currents in the ocean, while reducing the control difficulty as much as possible and improving adaptability, is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of the present invention is to provide a high-precision control method and system for a robotic arm based on MR and three-dimensional point cloud modeling, aiming to solve at least one of the above technical problems.

[0006] To achieve the above objectives, the present invention provides a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling, the method comprising the following steps:

[0007] Obtaining operating environment information of the target operating area; wherein the operating environment information includes real-time three-dimensional point cloud environmental data and overall ocean current prediction related data;

[0008] Based on the real-time environmental data of the three-dimensional point cloud, a virtual scene of the target operation area is constructed, and the image data of the virtual scene is sent to the MR interactive device;

[0009] Based on the target operator's task plan in the target operation area, the overall ocean current prediction data for each operation location and the real-time 3D point cloud environmental data of the target operation area are considered. The task completion time limit and task operation accuracy are used as the constraint condition set, and the minimum number of operation action optimization updates is used as the optimization goal. The operation execution strategy of the target operator in the target operation area is optimized.

[0010] generating robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and sending the robot arm operation control guidance information to the MR interaction device;

[0011] Driving the MR interactive device to embed the received robot arm operation control guidance information into a virtual scene, generating a virtual scene image of a target operation area, so that the target operator performs a virtual operation action of the robot arm according to the virtual scene image presented by the MR interactive device;

[0012] The target operator's posture information when performing a virtual operation of the robot arm is obtained, and based on the posture information and the optimization ratio of the operation point operation in the operation execution strategy, the actual operation parameters of the robot arm are generated and the robot arm is controlled to perform the actual operation.

[0013] Optionally, the step of obtaining the operating environment information of the target operating area specifically includes:

[0014] Receive real-time 3D point cloud environmental data collected by 3D point cloud acquisition equipment deployed in the target operation area, access a pre-built overall ocean current prediction association database, and query the overall ocean current prediction association data for each operation position in the target operation area based on the area range information of the target operation area;

[0015] Based on the real-time environmental data of the three-dimensional point cloud of the target operation area and the overall ocean current prediction correlation data, the operating environment information of the target operation area is generated.

[0016] Optionally, the construction of a database related to overall ocean current prediction includes:

[0017] Obtaining several sets of overall ocean current correlation data and overall ocean current impact parameter sets for different overall ocean current monitoring areas, extracting overall ocean current prediction impact features from each set of overall ocean current impact parameter sets, and constructing several overall ocean current prediction samples using feature sets constructed using overall ocean current prediction impact features and corresponding overall ocean current correlation data;

[0018] The overall ocean current associated data includes the overall ocean current velocity and overall ocean current direction collected by the overall ocean current monitoring device set in the overall ocean current monitoring area; the overall ocean current influencing parameter set includes a plurality of overall ocean current influencing parameters in the associated overall ocean current area that are collected by the ocean observation station and affect the overall ocean current changes of the overall ocean current monitoring; the overall ocean current influencing parameters include at least one of sea surface temperature, sea surface height, seawater salinity and meteorological information;

[0019] Using several overall ocean current prediction samples, the pre-built initial convolutional neural network is trained. When the training reaches the target number of times or the training converges, a trained overall ocean current correlation data prediction model is obtained.

[0020] The overall ocean current impact parameter set of the target operation area corresponding to the target execution period recorded in the marine environment forecast information is extracted. The overall ocean current correlation data prediction model is used to predict the overall ocean current correlation data of the target operation area in each unit period within the target execution period, and an overall ocean current prediction correlation database is constructed.

[0021] Optionally, the step of constructing a virtual scene of the target operation area based on the real-time environmental data of the three-dimensional point cloud and sending the image data of the virtual scene to the MR interaction device specifically includes:

[0022] Based on the three-dimensional point cloud real-time environment data, perform virtual scene modeling of the target operation area, obtain the virtual scene of the target operation area, and send the image data of the virtual scene to the MR interaction device;

[0023] The scene configuration structure features of the virtual scene of the target operation area are analyzed, a scene configuration structure feature set of the virtual scene is established, and the scene configuration structure feature set is stored.

[0024] Optionally, based on the target operator's task plan in the target operation area, taking into account the overall ocean current prediction associated data of each operation location and the three-dimensional point cloud real-time environmental data of the target operation area, with the task completion time limit and task operation accuracy as the constraint condition set, and with the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy steps of the target operator in the target operation area are optimized, specifically including:

[0025] Receive the target operator's work task plan in the target work area, extract the work location, work content and work task plan time limit of several work points, and build a work task content list;

[0026] Considering the overall ocean current prediction associated data of the target operation area for each operation position and the three-dimensional point cloud real-time environmental data of the target operation area, with the task completion time limit and task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy of the target operator in the target operation area is optimized.

[0027] Optionally, the overall ocean current prediction associated data for each operation position in the overall ocean current prediction associated data of the target operation area and the real-time environmental data of the three-dimensional point cloud of the target operation area are considered, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, to optimize and solve the operation execution strategy steps of the target operator in the target operation area, specifically including:

[0028] Extracting a scene configuration structure feature set of a virtual scene constructed from the overall ocean current prediction associated data of the target operation area and the overall ocean current prediction associated data for each operation position and the three-dimensional point cloud real-time environmental data of the target operation area;

[0029] Based on the overall ocean current prediction correlation data of each operation location and the scene structure feature set of the target operation area, the regional ocean current type of the target operation area is matched in the comparison table between the scene structure features required for the formation of ocean currents in different regions and the overall ocean current correlation data, and the regional ocean current set of each unit time period of the target operation area within the target execution period is generated;

[0030] Recalling a pre-tested set of motion parameters for different regional ocean current types on a manipulator performing a task type corresponding to a target operation area, and extracting the drive power adjustment ratio for each regional ocean current on each operation action of the manipulator recorded in the motion parameter set; wherein the operation actions include base movement, manipulator arm rotation, and manipulator arm rotation around an axis;

[0031] Based on the operation location, operation content and operation task planning time limit of each operation point, the location of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement of the robotic arm are considered. With the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy of the target operator in the target operation area is optimized.

[0032] Optionally, based on the operation location, operation content, and operation task plan time limit of each operation point, the locations of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement of the manipulator are considered. With the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy steps of the target operator in the target operation area are optimized, specifically including:

[0033] Based on the operation location, operation content and operation task planning time limit of each operation point, the position of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement of the manipulator are considered;

[0034] The first constraint condition is that the total operation time determined by the operation time corresponding to the operation content of each operation point and the moving distance and standard moving speed between two adjacent operation points in the actual execution order of several operation points in the operation task plan is less than the operation task plan time limit. The second constraint condition is that the driving power adjustment ratio of each operation action of the robotic arm caused by the regional ocean current formed in the corresponding unit time period when the robotic arm executes the operation content at each operation point is less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of the operation point. The optimization goal is to minimize the sum of the number of times the driving power adjustment ratio of each operation action of the robotic arm is switched within the entire operation task plan. The operation point execution order of the operation task plan and the driving power adjustment ratio of each operation point for different operation actions are optimized.

[0035] Based on the execution sequence of the work points and the drive power adjustment ratio of each work point for different work actions, the work execution strategy for the target operator in the target work area is generated.

[0036] Optionally, the step of generating robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and sending the robot arm operation control guidance information to the MR interaction device specifically includes:

[0037] Generate robot arm operation control guidance information according to the operation point execution sequence in the operation execution strategy; wherein the robot arm operation control guidance information includes the operation position and operation content of each operation point arranged according to the operation point execution sequence;

[0038] The operation position and operation content of each operation point in the robot arm operation control guidance information are sent to the MR interactive device in sequence according to the execution order of the operation points.

[0039] Optionally, obtaining posture information of the target operator performing a virtual operation of the robot arm, generating actual operation parameters of the robot arm based on the posture information and the optimized ratio of the operation action of the operation point in the operation execution strategy, and controlling the robot arm to perform the actual operation action steps specifically includes:

[0040] Acquiring posture information of a target operator performing a virtual operation of the robotic arm, and converting the posture information into a standard driving power corresponding to the operation action;

[0041] The drive power adjustment ratio for each operation point in the operation execution strategy for different operation actions is used as the product of the operation point operation action optimization ratio and the standard drive power of the current operation action to generate the actual drive power of each operation action;

[0042] The actual driving power of each operating point for different operating actions is used as the actual operating parameter of the robot arm to control the robot arm to perform the actual operating action.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling, comprising:

[0044] An acquisition module is used to acquire operating environment information of a target operating area; wherein the operating environment information includes real-time environmental data of a three-dimensional point cloud and overall ocean current prediction related data;

[0045] A construction module, configured to construct a virtual scene of a target operation area based on the real-time environmental data of the three-dimensional point cloud, and send the image data of the virtual scene to an MR interactive device;

[0046] The optimization module is used to optimize the target operator's operation execution strategy in the target operation area based on the target operator's operation task plan in the target operation area, taking into account the overall ocean current prediction correlation data of each operation location and the real-time 3D point cloud environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal;

[0047] a sending module, configured to generate robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and send the robot arm operation control guidance information to the MR interaction device;

[0048] a generation module, configured to cause the MR interaction device to embed the received robot arm operation control guidance information into a virtual scene, thereby generating a virtual scene image of a target operation area, so that a target operator can perform a virtual operation action of the robot arm according to the virtual scene image presented by the MR interaction device;

[0049] The execution module is used to obtain the posture information of the target operator performing the virtual operation action of the robot arm, generate the actual operation parameters of the robot arm based on the posture information and the optimization ratio of the operation point operation action in the operation execution strategy, and control the robot arm to perform the actual operation action.

[0050] The beneficial effects of the present invention are as follows: a high-precision control method and system for a robotic arm based on MR and 3D point cloud modeling is proposed, by acquiring the real-time environmental data of the 3D point cloud and the overall ocean current prediction associated data of the target operation area, using the real-time environmental data of the 3D point cloud to construct the picture data of the target operation area, using the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction associated data and the real-time environmental data of the 3D point cloud, estimating the regional ocean currents that may be formed in the target operation area, taking the completion time limit of the operation task and the operation accuracy of the operation task as the constraint condition set, taking the minimum number of optimized updates of the operation action as the optimization goal, optimizing and solving the operation execution strategy of the target operator in the target operation area, and integrating the operation execution strategy into the target operation area. The execution sequence of the operation points is used as the guidance information for the robot arm's operation control and is integrated with the picture data of the target operation area to generate a virtual scene picture of the target operation area, guiding the operator to execute the operation content of each operation point in the operation task plan in sequence. At the same time, the operation point operation action optimization ratio in the operation execution strategy is used to adjust the operation force of the robot arm when performing different operation actions at each operation point in real time to adapt to the influence of problems such as action offset and low control accuracy caused by overall ocean currents and regional ocean currents in marine operation scenarios, improve the operation accuracy and operation efficiency of remote operation of the robot arm in marine operation scenarios, reduce the control difficulty as much as possible, and solve the mechanical failure problems and operator adaptation problems caused by frequent switching of action optimization ratios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling of the present invention;

[0052] Figure 2 This is a structural diagram of the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] The embodiment of the present invention provides a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling. Figure 1 , Figure 1This is a flow chart of an embodiment of a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to the present invention.

[0055] In this embodiment, a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling is provided, the method comprising the following steps:

[0056] S1: Acquire operating environment information of the target operating area; wherein the operating environment information includes real-time 3D point cloud environmental data and overall ocean current prediction related data;

[0057] S2: constructing a virtual scene of the target operation area based on the real-time environmental data of the three-dimensional point cloud, and sending the image data of the virtual scene to the MR interaction device;

[0058] S3: Based on the target operator's task plan in the target operation area, the overall ocean current prediction data for each operation location and the real-time 3D point cloud environmental data of the target operation area are considered. The task completion time limit and task operation accuracy are used as the constraint condition set, and the optimization goal is to minimize the number of operation action optimization updates. The operation execution strategy of the target operator in the target operation area is optimized.

[0059] S4: generating robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and sending the robot arm operation control guidance information to the MR interaction device;

[0060] S5: driving the MR interactive device to embed the received robot arm operation control guidance information into the virtual scene, generating a virtual scene image of the target operation area, so that the target operator performs the robot arm virtual operation action according to the virtual scene image presented by the MR interactive device;

[0061] S6: Obtain the posture information of the target operator performing the virtual operation action of the robot arm, generate the actual operation parameters of the robot arm based on the posture information and the optimization ratio of the operation action of the operation point in the operation execution strategy, and control the robot arm to perform the actual operation action.

[0062] It should be noted that the application of special scenarios (such as deep-sea oil extraction, deep-sea mineral resource exploration, underwater archaeology, etc. in marine operation scenarios) still has limitations on the remote operation of the robotic arm: First, when controlling the robotic arm to perform corresponding operation tasks in the ocean, due to the influence of the overall ocean currents and regional ocean currents, the robotic arm will perform actions such as base movement, robotic arm rotation and robotic arm rotation around the axis. There will be accuracy deviations, which makes it easy for the operator to remotely control the robotic arm. Problems such as motion offset and low control accuracy will occur, affecting the execution efficiency of marine operation tasks; second, due to the influence of changes in the overall ocean currents and regional ocean currents, the operator may make adaptive adjustments to the operation actions for marine operation tasks. The standards are inconsistent, that is, the overall ocean currents and regional ocean currents may be different at different operating locations and at different operating times, resulting in different deviations when performing the same action at two operating points. Operators need to constantly trial and error based on experience to try to find the optimal control force for each operating point at the current time point, resulting in great control difficulty, low control accuracy and long operation time; thirdly, adjusting the operating power of the actuator corresponding to each operating action of the robotic arm is a method that can theoretically solve the accuracy deviation of the robotic arm's marine operating action, but how to accurately adjust the operating power of the actuator and minimize the number of adjustments as much as possible to avoid possible mechanical failure problems and operator adaptation problems is a difficulty.

[0063] In order to solve the above problems, this embodiment obtains the real-time environmental data of the three-dimensional point cloud and the overall ocean current prediction related data of the target operation area, estimates the regional ocean currents that may be formed in the target operation area, takes the completion time limit of the operation task and the operation accuracy of the operation task as the constraint condition set, and takes the minimum number of operation action optimization updates as the optimization goal, optimizes and solves the operation execution strategy of the target operation area, and uses the execution order of the operation points in the operation execution strategy as the guidance information for the robot arm operation control and integrates it with the picture data of the target operation area constructed using the real-time environmental data of the three-dimensional point cloud to generate a virtual scene picture of the target operation area, and adjusts the operation force of the robot arm when performing different operation actions at each operation point in real time, so as to improve the operation accuracy and operation efficiency of the robot arm remote operation in the marine operation scene.

[0064] In a preferred embodiment, the step of obtaining the operating environment information of the target operating area specifically includes:

[0065] S110: Receive real-time 3D point cloud environmental data collected by 3D point cloud collection equipment deployed in the target operation area, access a pre-built overall ocean current prediction association database, and query overall ocean current prediction association data for each operation position in the target operation area based on the area range information of the target operation area;

[0066] S120: Based on the three-dimensional point cloud real-time environmental data of the target operating area and the overall ocean current prediction associated data, generate operating environment information of the target operating area.

[0067] Furthermore, the construction of the overall ocean current prediction database includes:

[0068] S101: Acquire several sets of overall ocean current correlation data and overall ocean current impact parameter sets for different overall ocean current monitoring areas, extract overall ocean current prediction impact features from each set of overall ocean current impact parameter sets, and construct several overall ocean current prediction samples using a feature set constructed using the overall ocean current prediction impact features and the corresponding overall ocean current correlation data;

[0069] The overall ocean current associated data includes the overall ocean current velocity and overall ocean current direction collected by the overall ocean current monitoring device set in the overall ocean current monitoring area; the overall ocean current influencing parameter set includes a plurality of overall ocean current influencing parameters in the associated overall ocean current area that are collected by the ocean observation station and affect the overall ocean current changes of the overall ocean current monitoring; the overall ocean current influencing parameters include at least one of sea surface temperature, sea surface height, seawater salinity and meteorological information;

[0070] S102: Using a number of overall ocean current prediction samples, a pre-built initial convolutional neural network is trained, and when the training reaches a target number of times or the training converges, a trained overall ocean current correlation data prediction model is obtained;

[0071] S103 extracts the overall ocean current impact parameter set corresponding to the target execution period of the target operation area recorded in the marine environment forecast information, uses the overall ocean current correlation data prediction model to predict the overall ocean current correlation data of the target operation area in each unit time period within the target execution period, and constructs an overall ocean current prediction correlation database.

[0072] In this embodiment, real-time 3D point cloud environmental data is collected by 3D point cloud acquisition equipment deployed in the target operation area. The target operation area's overall ocean current prediction correlation data is extracted by calling a pre-built overall ocean current prediction correlation database to construct the target operation area's operating environment information. In practical applications, the overall ocean current prediction correlation data can be obtained by using historically collected overall ocean current correlation data and a set of overall ocean current influencing parameters. A self-built convolutional neural network is trained to obtain an overall ocean current correlation data prediction model, and the overall ocean current correlation data is predicted for each unit time period within the target execution period of the target operation area.

[0073] In a preferred embodiment, the steps of constructing a virtual scene of the target operation area based on the real-time environmental data of the three-dimensional point cloud and sending the image data of the virtual scene to the MR interaction device specifically include:

[0074] S210: Execute virtual scene modeling of the target operation area based on the real-time three-dimensional point cloud environment data, obtain a virtual scene of the target operation area, and send image data of the virtual scene to the MR interaction device;

[0075] S220: Analyze the scene configuration structure features of the virtual scene of the target operation area, establish a scene configuration structure feature set of the virtual scene, and store the scene configuration structure feature set.

[0076] In this embodiment, the collection of real-time 3D point cloud environmental data is used to construct a virtual scene of the target operating area and transmit it to the MR interactive device for presentation. While constructing the virtual scene of the target operating area, the scene configuration structural features of the virtual scene are extracted and stored in a scene configuration structural feature set. This scene configuration structural feature set can then be used to assess the formation of local ocean currents within the target operating area.

[0077] In a preferred embodiment, based on the target operator's task plan in the target operation area, the overall ocean current prediction data for each operation location and the real-time three-dimensional point cloud environmental data of the target operation area are considered. With the task completion time limit and task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the steps of optimizing the target operator's task execution strategy in the target operation area are specifically as follows:

[0078] S310: Receive the target operator's work task plan in the target work area, extract the work location, work content and work task plan time limit of several work points, and build a work task content list;

[0079] S320: Considering the overall ocean current prediction associated data for each operation position in the overall ocean current prediction associated data of the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and with the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

[0080] Furthermore, considering the overall ocean current prediction data for each operation position in the target operation area and the real-time environmental data of the 3D point cloud of the target operation area, with the task completion time limit and task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy steps of the target operator in the target operation area are optimized, including:

[0081] S321: extracting a scene configuration structure feature set of a virtual scene constructed from the overall ocean current prediction associated data of the target operation area and the overall ocean current prediction associated data for each operation position and the three-dimensional point cloud real-time environment data of the target operation area;

[0082] S322: Based on the overall ocean current prediction association data of each operation location and the scene structure feature set of the target operation area, the regional ocean current type of the target operation area is matched in a comparison table between the scene structure features required for the formation of ocean currents in different regions and the overall ocean current association data, thereby generating a regional ocean current set for each unit period of the target operation area within the target execution period.

[0083] S323: Recalling a pre-tested set of motion effect parameters for different regional ocean current types on the manipulator performing a task type corresponding to the target operation area, and extracting the drive power adjustment ratio for each regional ocean current on each operation action of the manipulator recorded in the motion effect parameter set; wherein the operation action includes base movement, manipulator arm rotation, and manipulator arm rotation around an axis;

[0084] S324: Based on the operation location, operation content and operation task planning time limit of each operation point, considering the position of different regional ocean currents formed in different unit time periods in the target operation area and the standard moving speed of the base movement of the robot arm, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy of the target operator in the target operation area is optimized.

[0085] In practical applications, based on the operation location, operation content, and operation task planning time limit of each operation point, the location of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the robot arm's base movement are considered. With the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy steps of the target operator in the target operation area are optimized, including:

[0086] S3241: Based on the operation location, operation content and operation task plan time limit of each operation point, the location of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement of the manipulator are considered;

[0087] S3242: The first constraint condition is that the total operation duration, determined based on the operation duration corresponding to the operation content of each operation point and the moving distance and standard moving speed between two adjacent operation points, is less than the operation task plan time limit in the actual execution order of several operation points in the operation task plan. The second constraint condition is that the driving power adjustment ratio of each operation action of the manipulator caused by the regional ocean current formed in the corresponding unit time period when the manipulator executes the operation content at each operation point is less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of the operation point. The optimization objective is to minimize the sum of the number of times the driving power adjustment ratio of each operation action of the manipulator is switched within the entire operation task plan. The operation task plan's operation point execution order and the driving power adjustment ratio of each operation point for different operation actions are optimized.

[0088] S3243: Generate an operation execution strategy for a target operator in a target operation area according to the operation point execution sequence and the drive power adjustment ratio of each operation point for different operation actions.

[0089] In this embodiment, by obtaining the real-time environmental data of the three-dimensional point cloud and the overall ocean current prediction associated data of the target operation area, the real-time environmental data of the three-dimensional point cloud is used to construct the image data of the target operation area, and the operation task plan of the target operator in the target operation area is used to consider the overall ocean current prediction associated data and the real-time environmental data of the three-dimensional point cloud to estimate the regional ocean currents that may be formed in the target operation area. The operation task completion time limit and the operation task operation accuracy are used as the constraint condition set, and the optimization goal is to minimize the number of operation action optimization updates. The operation execution strategy of the target operator in the target operation area is optimized and solved. The operation point operation action optimization ratio in the operation execution strategy is used to adjust the operation force of the robot arm when performing different operation actions at each operation point in real time to adapt to the influence of the motion offset and low control accuracy caused by the overall ocean current and regional ocean current in the marine operation scenario, improve the operation accuracy and operation efficiency of the robot arm remote operation in the marine operation scenario, minimize the control difficulty, and solve the mechanical failure problem and operator adaptation problem caused by frequent switching of the action optimization ratio.

[0090] In a preferred embodiment, the steps of generating robot arm operation control guidance information according to the operation point execution order in the operation execution strategy and sending the robot arm operation control guidance information to the MR interaction device specifically include:

[0091] S410: Generating robot arm operation control guidance information according to the operation point execution sequence in the operation execution strategy; wherein the robot arm operation control guidance information includes the operation position and operation content of each operation point arranged according to the operation point execution sequence;

[0092] S420: Send the operation position and operation content of each operation point in the robot arm operation control guidance information to the MR interactive device in sequence according to the execution order of the operation points.

[0093] On this basis, the target operator's posture information of the robot arm's virtual operation action is obtained. Based on the posture information and the optimized ratio of the operation point operation action in the operation execution strategy, the actual operation parameters of the robot arm are generated and the robot arm is controlled to perform the actual operation action steps, which specifically include:

[0094] S610: Acquire posture information of a target operator performing a virtual operation of the robot arm, and convert the posture information into a standard driving power corresponding to the operation action;

[0095] S620: The drive power adjustment ratio for each operation point for different operation actions in the operation execution strategy is used as the product of the operation point operation action optimization ratio and the standard drive power of the current operation action to generate the actual drive power of each operation action;

[0096] S630: Using the actual driving power of each operating point for different operating actions as the actual operating parameter of the robot arm to control the robot arm to perform the actual operating action.

[0097] In this embodiment, the execution order of the operation points in the operation execution strategy is used as the robot arm operation control guidance information and is integrated with the picture data of the target operation area to generate a virtual scene picture of the target operation area, guiding the operator to perform the operation content of each operation point in the operation task plan in sequence. At the same time, by adjusting the operation force of the robot arm when performing different operation actions at each operation point in real time, it can adapt to the influence of problems such as motion offset and low control accuracy caused by overall ocean currents and regional ocean currents in marine operation scenarios, and improve the operation accuracy and operation efficiency of remote operation of the robot arm in marine operation scenarios.

[0098] Reference Figure 2 , Figure 2 This is a structural block diagram of an embodiment of a high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling of the present invention.

[0099] like Figure 2 As shown, the high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling proposed in an embodiment of the present invention includes:

[0100] An acquisition module 10 is used to acquire operating environment information of a target operating area; wherein the operating environment information includes real-time environmental data of a three-dimensional point cloud and overall ocean current prediction related data;

[0101] A construction module 20 is configured to construct a virtual scene of a target operation area based on the real-time environmental data of the three-dimensional point cloud, and send the image data of the virtual scene to an MR interaction device;

[0102] An optimization module 30 is configured to optimize the target operator's operation execution strategy in the target operation area based on the target operator's operation task plan in the target operation area, taking into account the overall ocean current prediction correlation data for each operation location and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal;

[0103] a sending module 40, configured to generate robot arm operation control guidance information according to the operation point execution sequence in the operation execution strategy, and send the robot arm operation control guidance information to the MR interaction device;

[0104] a generating module 50 for driving the MR interactive device to embed the received robot arm operation control guidance information into a virtual scene, generating a virtual scene image of a target operation area, so that a target operator can perform a virtual operation action of the robot arm according to the virtual scene image presented by the MR interactive device;

[0105] The execution module 60 is used to obtain the posture information of the target operator performing the virtual operation action of the robot arm, generate the actual operation parameters of the robot arm based on the posture information and the optimization ratio of the operation point operation action in the operation execution strategy, and control the robot arm to perform the actual operation action.

[0106] Other embodiments or specific implementations of the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0107] It should be understood that, in the description of this specification, reference to terms such as "one embodiment," "another embodiment," "other embodiments," or "first to Nth embodiments" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples.

[0108] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling, characterized in that: The method comprises the following steps: Obtaining operating environment information of the target operating area; wherein the operating environment information includes real-time 3D point cloud environmental data and overall ocean current prediction related data; specifically including: Receive real-time 3D point cloud environmental data collected by 3D point cloud acquisition equipment deployed in the target operation area, access a pre-built overall ocean current prediction association database, and query the overall ocean current prediction association data for each operation position in the target operation area based on the area range information of the target operation area; Generate operating environment information of the target operating area based on the real-time environmental data of the 3D point cloud of the target operating area and the overall ocean current prediction correlation data; Based on the real-time environmental data of the three-dimensional point cloud, a virtual scene of the target operation area is constructed, and the image data of the virtual scene is sent to the MR interactive device; Based on the target operator's task plan in the target operation area, the overall ocean current prediction data of each operation location and the real-time 3D point cloud environmental data of the target operation area are considered. With the task completion time limit and task operation accuracy as the constraint condition set and the minimum number of operation action optimization updates as the optimization goal, the operation execution strategy of the target operator in the target operation area is optimized. Specifically, it includes: Receive the target operator's work task plan in the target work area, extract the work location, work content and work task plan time limit of several work points, and build a work task content list; Extracting a scene configuration structure feature set of a virtual scene constructed from the overall ocean current prediction associated data of the target operation area and the overall ocean current prediction associated data for each operation position and the three-dimensional point cloud real-time environmental data of the target operation area; Based on the overall ocean current prediction correlation data of each operation location and the scene structure feature set of the target operation area, the regional ocean current type of the target operation area is matched in the comparison table between the scene structure features required for the formation of ocean currents in different regions and the overall ocean current correlation data, and the regional ocean current set of each unit time period of the target operation area within the target execution period is generated; Recalling a pre-tested set of motion parameters for different regional ocean current types on a manipulator performing a task type corresponding to a target operation area, and extracting the drive power adjustment ratio for each regional ocean current on each operation action of the manipulator recorded in the motion parameter set; wherein the operation actions include base movement, manipulator arm rotation, and manipulator arm rotation around an axis; Based on the operation location, operation content and operation task planning time limit of each operation point, the position of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement of the manipulator are considered; The first constraint condition is that the total operation time determined by the operation time corresponding to the operation content of each operation point and the moving distance and standard moving speed between two adjacent operation points in the actual execution order of several operation points in the operation task plan is less than the operation task plan time limit. The second constraint condition is that the driving power adjustment ratio of each operation action of the robotic arm caused by the regional ocean current formed in the corresponding unit time period when the robotic arm executes the operation content at each operation point is less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of the operation point. The optimization goal is to minimize the sum of the number of times the driving power adjustment ratio of each operation action of the robotic arm is switched within the entire operation task plan. The operation point execution order of the operation task plan and the driving power adjustment ratio of each operation point for different operation actions are optimized. Generate an operation execution strategy for the target operator in the target operation area based on the execution sequence of the operation points and the drive power adjustment ratio of each operation point for different operation actions; Generating robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and sending the robot arm operation control guidance information to the MR interaction device; specifically comprising: Generate robot arm operation control guidance information according to the operation point execution sequence in the operation execution strategy; wherein the robot arm operation control guidance information includes the operation position and operation content of each operation point arranged according to the operation point execution sequence; Sending the operation position and operation content of each operation point in the robot arm operation control guidance information to the MR interactive device in sequence according to the execution order of the operation points; Driving the MR interactive device to embed the received robot arm operation control guidance information into a virtual scene, generating a virtual scene image of a target operation area, so that the target operator performs a virtual operation action of the robot arm according to the virtual scene image presented by the MR interactive device; Obtaining the posture information of the target operator performing the virtual operation of the robot arm, generating the actual operation parameters of the robot arm based on the posture information and the optimization ratio of the operation point operation in the operation execution strategy, and controlling the robot arm to perform the actual operation; specifically including: Acquiring posture information of a target operator performing a virtual operation of the robotic arm, and converting the posture information into a standard driving power corresponding to the operation action; The drive power adjustment ratio for each operation point in the operation execution strategy for different operation actions is used as the product of the operation point operation action optimization ratio and the standard drive power of the current operation action to generate the actual drive power of each operation action; The actual driving power of each operating point for different operating actions is used as the actual operating parameter of the robot arm to control the robot arm to perform the actual operating action.

2. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, characterized in that: The construction of the overall ocean current prediction database includes: Obtaining several sets of overall ocean current correlation data and overall ocean current impact parameter sets for different overall ocean current monitoring areas, extracting overall ocean current prediction impact features from each set of overall ocean current impact parameter sets, and constructing several overall ocean current prediction samples using feature sets constructed using overall ocean current prediction impact features and corresponding overall ocean current correlation data; The overall ocean current associated data includes the overall ocean current velocity and overall ocean current direction collected by the overall ocean current monitoring device set in the overall ocean current monitoring area; the overall ocean current influencing parameter set includes a plurality of overall ocean current influencing parameters in the associated overall ocean current area that are collected by the ocean observation station and affect the overall ocean current changes of the overall ocean current monitoring; the overall ocean current influencing parameters include at least one of sea surface temperature, sea surface height, seawater salinity and meteorological information; Using several overall ocean current prediction samples, the pre-built initial convolutional neural network is trained. When the training reaches the target number of times or the training converges, a trained overall ocean current correlation data prediction model is obtained. The overall ocean current impact parameter set of the target operation area corresponding to the target execution period recorded in the marine environment forecast information is extracted. The overall ocean current correlation data prediction model is used to predict the overall ocean current correlation data of the target operation area in each unit period within the target execution period, and an overall ocean current prediction correlation database is constructed.

3. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, characterized in that: The steps of constructing a virtual scene of a target operation area based on the real-time environmental data of the three-dimensional point cloud and sending the image data of the virtual scene to the MR interaction device specifically include: Based on the three-dimensional point cloud real-time environment data, perform virtual scene modeling of the target operation area, obtain the virtual scene of the target operation area, and send the image data of the virtual scene to the MR interaction device; The scene configuration structure features of the virtual scene of the target operation area are analyzed, a scene configuration structure feature set of the virtual scene is established, and the scene configuration structure feature set is stored.

4. A high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling, used in the high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to any one of claims 1 to 3, characterized in that: include: An acquisition module is used to acquire operating environment information of a target operating area; wherein the operating environment information includes real-time environmental data of a three-dimensional point cloud and overall ocean current prediction related data; A construction module, configured to construct a virtual scene of a target operation area based on the real-time environmental data of the three-dimensional point cloud, and send the image data of the virtual scene to an MR interactive device; The optimization module is used to optimize the target operator's operation execution strategy in the target operation area based on the target operator's operation task plan in the target operation area, taking into account the overall ocean current prediction correlation data of each operation location and the real-time 3D point cloud environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal; a sending module, configured to generate robot arm operation control guidance information according to the operation point execution order in the operation execution strategy, and send the robot arm operation control guidance information to the MR interaction device; a generation module, configured to cause the MR interaction device to embed the received robot arm operation control guidance information into a virtual scene, thereby generating a virtual scene image of a target operation area, so that a target operator can perform a virtual operation action of the robot arm according to the virtual scene image presented by the MR interaction device; The execution module is used to obtain the posture information of the target operator performing the virtual operation action of the robot arm, generate the actual operation parameters of the robot arm based on the posture information and the optimization ratio of the operation point operation action in the operation execution strategy, and control the robot arm to perform the actual operation action.

Images