Mechanical arm high-precision control method and system based on MR and three-dimensional point cloud modeling

Abstract

The invention relates to the technical field of mechanical arm control, and discloses a mechanical arm high-precision control method and system based on MR and three-dimensional point cloud modeling, and the method comprises the steps: obtaining three-dimensional point cloud real-time environment data and overall ocean current prediction association data of a target operation region, estimating a region ocean current which is possibly formed in the target operation region, and carrying out the prediction of the overall ocean current in the target operation region; by taking a job task completion time limit and job task operation precision as a constraint condition set and taking the minimum operation action optimization updating frequency as an optimization target, optimizing and solving a job execution strategy of a target job area; the operation point execution sequence in the operation execution strategy serves as mechanical arm operation control guidance information to be fused with picture data of a target operation area constructed through the three-dimensional point cloud real-time environment data, and a virtual scene picture of the target operation area is generated; and the operation force when the mechanical arm executes different operation actions at each operation point is adjusted in real time, so that the operation precision and the operation efficiency of the teleoperation of the mechanical arm in the ocean operation scene are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm control, and particularly to a high-precision control method and system for a robotic arm based on MR and three-dimensional point cloud modeling. Background Art

[0002] Robotic arm teleoperation refers to a technology in which an operator remotely controls a robotic arm at a location far from the robotic arm, and with the aid of a communication system and a human-computer interaction device, the robotic arm is manipulated in real time to complete specific tasks. The core is to establish a closed-loop control system of "human-communication link-robotic arm", enabling the operator to perceive the remote environment and accurately control the actions of the robotic arm. The development and application of robotic arm teleoperation technology can break through the physical space limitations, expand the scope of human activities, improve the accuracy and efficiency of complex tasks, enhance operation safety, and replace the operator to perform complex operation tasks in high-risk environment operations.

[0003] However, in practical applications, the application in special scenarios (such as deep-sea oil exploitation, deep-sea mineral resource exploration, underwater archaeology, etc. in the ocean operation scenario) still makes robotic arm teleoperation have limitations: First, when controlling the robotic arm to perform corresponding operation tasks in the ocean, due to the influence of the overall ocean current and regional ocean current, there will be accuracy deviations when the robotic arm performs actions such as the movement of the execution base, the rotation of the robotic arm, and the rotation of the robotic arm around the axis. This makes it easy for the operator to have problems such as action deviation and low control accuracy when remotely operating the robotic arm, affecting the execution efficiency of ocean operation tasks; Second, due to the influence of the changes in the overall ocean current and regional ocean current, it may make the adjustment criteria for the operator to make adaptive operation actions for ocean operation tasks inconsistent, that is, the overall ocean current and regional ocean current at different operation positions and different operation times may be different, resulting in different deviations when performing the same action at two operation points. The operator needs to continuously try and error according to experience to find the best control force at each operation point at the current time point, resulting in high control difficulty, low control accuracy, and long operation time; Third, adjusting the operating power of the execution mechanism corresponding to each operation action of the robotic arm is a method that can theoretically solve the accuracy deviation problem of the robotic arm's ocean operation actions. However, how to accurately adjust the operating power of the execution mechanism and minimize the number of adjustments as much as possible to avoid possible mechanical failure problems and operator adaptation problems is a difficult point.

[0004] Therefore, how to improve the operation accuracy and operation efficiency of robotic arm teleoperation in the ocean operation scenario, while solving problems such as action deviation and low control accuracy caused by the overall ocean current and regional ocean current in the ocean, and reducing the control difficulty and improving the adaptability as much as possible is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The main objective of the present invention is to provide a high-precision control method and system for a robotic arm based on MR and three-dimensional point cloud modeling, aiming to solve at least one of the above technical problems.

[0006] To achieve the above objective, the present invention provides a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling. The method includes the following steps: Obtain the operation environment information of the target operation area; wherein, the operation environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; Based on the three-dimensional point cloud real-time environment data, construct a virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device; According to the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environment data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization objective, optimize and solve the operation execution strategy of the target operator in the target operation area; According to the operation point execution order in the operation execution strategy, generate robotic arm operation control guidance information, and send the robotic arm operation control guidance information to the MR interaction device; Drive the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene, generate a virtual scene screen of the target operation area, so that the target operator can perform robotic arm virtual operation actions according to the virtual scene screen presented by the MR interaction device; Obtain the posture information of the target operator performing the robotic arm virtual operation action, and based on the posture information and the operation point operation action optimization ratio in the operation execution strategy, generate the actual operation parameters of the robotic arm and control the robotic arm to perform actual operation actions.

[0007] Optionally, the step of obtaining the operation environment information of the target operation area specifically includes: Receive the three-dimensional point cloud real-time environment data collected by the three-dimensional point cloud acquisition device deployed in the target operation area, access the pre-constructed overall ocean current prediction correlation database, and query the overall ocean current prediction correlation data of each operation position in the target operation area according to the area range information of the target operation area; Based on the three-dimensional point cloud real-time environment data and the overall ocean current prediction correlation data of the target operation area, generate the operation environment information of the target operation area.

[0008] Optionally, the construction of the overall ocean current prediction correlation database specifically includes: Obtain several sets of overall ocean current correlation data and overall ocean current impact parameter sets for different overall ocean current monitoring regions, extract the overall ocean current prediction impact characteristics in each set of overall ocean current impact parameter sets, and construct several overall ocean current prediction samples with the feature set constructed from the overall ocean current prediction impact characteristics and the corresponding overall ocean current correlation data; Among them, the overall ocean current correlation data includes the overall ocean current speed and overall ocean current direction collected by the overall ocean current monitoring devices set in the overall ocean current monitoring region, and the overall ocean current impact parameter set includes several overall ocean current impact parameters in the associated overall ocean current region that affect the overall ocean current change monitored by the ocean observation station. The overall ocean current impact parameters include at least one of sea surface temperature, sea surface height, seawater salinity, and meteorological information; Use several overall ocean current prediction samples to train a pre-constructed initial convolutional neural network. When the training reaches the target number of times or converges, obtain a trained overall ocean current correlation data prediction model; Extract the overall ocean current impact parameter set corresponding to the target operation period of the target operation area recorded in the ocean environmental forecast information, and use the overall ocean current correlation data prediction model to predict the overall ocean current correlation data for each unit period within the target operation period of the target operation area, and construct an overall ocean current prediction correlation database.

[0009] Optionally, based on the three-dimensional point cloud real-time environmental data, constructing a virtual scene of the target operation area and sending the screen data of the virtual scene to the MR interaction device steps specifically include: Based on the three-dimensional point cloud real-time environmental data, perform virtual scene modeling of the target operation area to obtain a virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device; Analyze the scene configuration structure characteristics of the virtual scene of the target operation area, establish a scene configuration structure feature set of the virtual scene, and store the scene configuration structure feature set.

[0010] Optionally, according to the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, the steps of optimizing and solving the operation execution strategy of the target operator in the target operation area specifically include: Receive the operation task plan of the target operator in the target operation area, extract the operation positions, operation contents, and operation task plan time limits of several operation points, and construct an operation task content list; Considering the overall ocean current prediction correlation data for each operation location in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environment data of the target operation area, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of optimization update times of the operation actions as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

[0011] Optionally, considering the overall ocean current prediction correlation data for each operation location in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environment data of the target operation area, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of optimization update times of the operation actions as the optimization goal, the steps of optimizing and solving the operation execution strategy of the target operator in the target operation area specifically include: Extract the scene configuration structure feature set of the virtual scene constructed from the overall ocean current prediction correlation data for each operation location in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environment data of the target operation area; According to the overall ocean current prediction correlation data for each operation location and the scene construction structure feature set of the target operation area, perform regional ocean current type matching for the target operation area in the comparison table of the scene construction structure features required for the formation of different regional ocean currents and the overall ocean current correlation data, and generate the regional ocean current set for each unit time period within the target execution time period of the target operation area; Call the set of motion influence parameters obtained by pre-testing the influence of different regional ocean current types on the mechanical arm for performing the corresponding task type in the target operation area, and extract the driving power adjustment ratio of each regional ocean current to each operation action of the mechanical arm recorded in the set of motion influence parameters; wherein, the operation actions include base movement actions, mechanical arm rotation actions, and mechanical arm rotation around the axis actions; Based on the operation location, operation content, and operation task planned time limit of each operation point, considering the positions where different regional ocean currents are formed in different unit time periods within the target operation area and the standard movement speed of the base movement action of the mechanical arm, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of optimization update times of the operation actions as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

[0012] Optionally, based on the operation location, operation content, and operation task planned time limit of each operation point, considering the positions where different regional ocean currents are formed in different unit time periods within the target operation area and the standard movement speed of the base movement action of the mechanical arm, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of optimization update times of the operation actions as the optimization goal, the steps of optimizing and solving the operation execution strategy of the target operator in the target operation area specifically include: Based on the operation location, operation content, and operation task planned time limit of each operation point, consider the positions of the ocean currents in different unit time periods within the target operation area and the standard moving speed of the base movement of the robotic arm. Taking the total operation time determined by the operation duration corresponding to the operation content of each operation point, the moving distance between two adjacent operation points, and the standard moving speed in the actual execution order of several operation points in the operation task plan being less than the operation task plan time limit as the first constraint condition, and the adjustment ratio of the driving power of each operation action of the robotic arm by the regional ocean current formed in the corresponding unit time period when the robotic arm executes the operation content at each operation point being less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of this operation point as the second constraint condition, and the sum of the number of times of switching the driving power adjustment ratio of each operation action of the robotic arm within the entire operation task plan being minimized as the optimization objective, optimize and solve the execution order of the operation points of the operation task plan and the driving power adjustment ratio of each operation point for different operation actions. Generate the operation execution strategy of the target operator in the target operation area according to the operation point execution order and the driving power adjustment ratio of each operation point for different operation actions.

[0013] Optionally, the step of generating the robotic arm operation control guidance information according to the operation point execution order in the operation execution strategy and sending the robotic arm operation control guidance information to the MR interaction device specifically includes: Generate the robotic arm operation control guidance information according to the operation point execution order in the operation execution strategy; wherein, the robotic arm operation control guidance information includes the operation location and operation content of each operation point arranged according to the operation point execution order. Send the operation location and operation content of each operation point in the robotic arm operation control guidance information to the MR interaction device in sequence according to the operation point execution order.

[0014] Optionally, the step of obtaining the posture information of the target operator performing the virtual operation action of the robotic arm, generating the actual operation parameters of the robotic arm based on the posture information and the operation point operation action optimization ratio in the operation execution strategy, and controlling the robotic arm to perform the actual operation action specifically includes: Obtain the posture information of the target operator performing the virtual operation action of the robotic arm, and convert the posture information into the standard driving power corresponding to the operation action. Multiply the driving power adjustment ratio of each operation point for different operation actions in the operation execution strategy as the operation point operation action optimization ratio by the standard driving power of the current operation action to generate the actual driving power of each operation action. Take the actual driving power of each working point for different working actions as the actual operation parameters of the robotic arm to control the robotic arm to perform actual operation actions.

[0015] In addition, to achieve the above object, the present invention also provides a high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling, including: An acquisition module for acquiring the working environment information of the target working area; wherein, the working environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; A construction module for constructing a virtual scene of the target working area based on the three-dimensional point cloud real-time environment data and sending the screen data of the virtual scene to the MR interaction device; An optimization module for considering the overall ocean current prediction correlation data of each working position and the three-dimensional point cloud real-time environment data of the target working area according to the working task plan of the target operator in the target working area, taking the working task completion time limit and the working task operation accuracy as a set of constraint conditions, and taking the minimum number of optimization updates of the operation action as the optimization goal, and optimizing and solving the working execution strategy of the target operator in the target working area; A sending module for generating robotic arm operation control guidance information according to the execution order of the working points in the working execution strategy and sending the robotic arm operation control guidance information to the MR interaction device; A generation module for driving the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene to generate a virtual scene screen of the target working area, so that the target operator can perform robotic arm virtual operation actions according to the virtual scene screen presented by the MR interaction device; An execution module for acquiring the posture information of the target operator performing the robotic arm virtual operation action, generating actual operation parameters of the robotic arm based on the posture information and the operation action optimization ratio of the working points in the working execution strategy, and controlling the robotic arm to perform actual operation actions.

[0016] The beneficial effects of the present invention are as follows: a high-precision control method and system for a robotic arm based on MR and 3D point cloud modeling are proposed, by acquiring real-time 3D point cloud environmental data and overall ocean current prediction associated data of a target operating area, using the real-time 3D point cloud environmental data to construct image data of the target operating area, using the target operator's operating task plan in the target operating area, considering the overall ocean current prediction associated data and the real-time 3D point cloud environmental data, estimating the regional ocean currents that may be formed in the target operating area, taking the completion time limit of the operating task and the operating accuracy of the operating task as the constraint condition set, taking the minimum number of optimized updates of the operating action as the optimization goal, optimizing and solving the operating execution strategy of the target operator in the target operating area, and integrating the operating execution strategy into the target operating area. The execution sequence of the operating points is used as the guidance information for the robot arm's operation control and is integrated with the picture data of the target operation area to generate a virtual scene picture of the target operation area, guiding the operator to sequentially execute the operation content of each operating point in the operation task plan. At the same time, the operation point operation action optimization ratio in the operation execution strategy is used to adjust the operation force of the robot arm when performing different operation actions at each operation point in real time to adapt to the influence of movement offset and low control accuracy caused by overall ocean currents and regional ocean currents in marine operation scenarios, improve the operation accuracy and operation efficiency of remote operation of the robot arm in marine operation scenarios, reduce the control difficulty as much as possible, and solve the mechanical failure problems and operator adaptation problems caused by frequent switching of action optimization ratios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the high-precision control method of a robotic arm based on MR and three-dimensional point cloud modeling of the present invention; Figure 2 This is a structural diagram of the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] The embodiment of the present invention provides a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling, referring to Figure 1 , Figure 1 It is a flow chart of an embodiment of a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling of the present invention.

[0020] In this embodiment, a high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling is provided, and the method comprises the following steps: S1: Obtain the operation environment information of the target operation area; wherein, the operation environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; S2: Based on the three-dimensional point cloud real-time environment data, construct a virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device; S3: According to the operation task plan of the target operator in the target operation area, consider the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environment data of the target operation area, take the operation task completion time limit and operation task operation accuracy as the constraint condition set, and take the minimum number of operation action optimization updates as the optimization goal, and optimize and solve the operation execution strategy of the target operator in the target operation area; S4: Generate robotic arm operation control guidance information according to the operation point execution order in the operation execution strategy, and send the robotic arm operation control guidance information to the MR interaction device; S5: Drive the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene to generate a virtual scene screen of the target operation area, so that the target operator can perform robotic arm virtual operation actions according to the virtual scene screen presented by the MR interaction device; S6: Obtain the posture information of the target operator performing the robotic arm virtual operation action, and generate the actual operation parameters of the robotic arm and control the robotic arm to perform the actual operation action based on the posture information and the operation point operation action optimization ratio in the operation execution strategy.

[0021] It should be noted that the application in special scenarios (such as deep - sea oil exploitation, deep - sea mineral resource exploration, underwater archaeology, etc. in the marine operation scenario) still limits the remote operation of the robotic arm: First, when controlling the robotic arm to perform corresponding operation tasks in the ocean, due to the influence of the overall ocean current and regional ocean currents, there will be accuracy deviations when the robotic arm performs actions such as the movement of the execution base, the rotation of the robotic arm, and the rotation of the robotic arm around the axis. This makes it easy for the operator to have problems such as action deviation and low control accuracy when remotely operating the robotic arm, affecting the execution efficiency of marine operation tasks. Second, due to the influence of the changes in the overall ocean current and regional ocean currents, it may lead to inconsistent adjustment criteria for the operator to make adaptive operation actions for marine operation tasks. That is, the overall ocean current and regional ocean currents at different operation positions and different operation times may be different, resulting in different deviations when performing the same action at two operation points. The operator needs to continuously try and error according to experience to find the best control force at each operation point at the current time point, resulting in high control difficulty, low control accuracy, and long operation time. Third, adjusting the operating power of the execution mechanism corresponding to each operation action of the robotic arm is a method that can theoretically solve the accuracy deviation problem of the robotic arm's marine operation actions. However, how to accurately adjust the operating power of the execution mechanism and minimize the number of adjustment times to avoid possible mechanical failure problems and operator adaptation problems is a difficult point.

[0022] To solve the above problems, in this embodiment, by obtaining the three - dimensional point cloud real - time environment data and the overall ocean current prediction correlation data of the target operation area, predicting the possible regional ocean currents formed in the target operation area, taking the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and taking the minimum number of operation action optimization and update times as the optimization goal, optimizing and solving the operation execution strategy of the target operation area, fusing the operation point execution order in the operation execution strategy with the picture data of the target operation area constructed by using the three - dimensional point cloud real - time environment data to generate the virtual scene picture of the target operation area, and adjusting the operation force in real - time when the robotic arm performs different operation actions at each operation point, so as to improve the operation accuracy and operation efficiency of the robotic arm remote operation in the marine operation scenario.

[0023] In the preferred embodiment, the steps of obtaining the operation environment information of the target operation area specifically include: S110: Receive the three - dimensional point cloud real - time environment data collected by the three - dimensional point cloud acquisition device deployed in the target operation area, access the pre - constructed overall ocean current prediction correlation database, and query the overall ocean current prediction correlation data of each operation position in the target operation area according to the regional range information of the target operation area; S120: Generate the operation environment information of the target operation area based on the three - dimensional point cloud real - time environment data and the overall ocean current prediction correlation data of the target operation area.

[0024] Furthermore, the construction of the overall ocean current prediction correlation database specifically includes: S101: Obtain several sets of overall ocean current correlation data and overall ocean current impact parameter sets for different overall ocean current monitoring regions, extract the overall ocean current prediction impact features in each set of overall ocean current impact parameter sets, and construct several overall ocean current prediction samples with the feature sets constructed based on the overall ocean current prediction impact features and the corresponding overall ocean current correlation data; Among them, the overall ocean current correlation data includes the overall ocean current speed and overall ocean current direction collected by the overall ocean current monitoring devices set in the overall ocean current monitoring region, the overall ocean current impact parameter set includes several overall ocean current impact parameters in the associated overall ocean current region that affect the overall ocean current change monitored by the ocean observation station, and the overall ocean current impact parameter includes at least one of sea surface temperature, sea surface height, seawater salinity, and meteorological information; S102: Use several overall ocean current prediction samples to train a pre-constructed initial convolutional neural network, and when the training reaches the target number of times or converges, obtain a trained overall ocean current correlation data prediction model; S103: Extract the overall ocean current impact parameter set corresponding to the target operation period of the target operation region recorded in the ocean environmental forecast information, use the overall ocean current correlation data prediction model to predict the overall ocean current correlation data of each unit period in the target operation period of the target operation region, and construct the overall ocean current prediction correlation database.

[0025] In this embodiment, the three-dimensional point cloud real-time environment data is collected by the three-dimensional point cloud acquisition device deployed in the target operation region, and the overall ocean current prediction correlation data of the target operation region is extracted by calling the pre-constructed overall ocean current prediction correlation database, so as to construct the operation environment information of the target operation region. In practical applications, the acquisition of the overall ocean current prediction correlation data can use the historically collected overall ocean current correlation data and overall ocean current impact parameter sets, train an overall ocean current correlation data prediction model by self-constructing a convolutional neural network, and predict the overall ocean current correlation data of each unit period in the target operation period of the target operation region.

[0026] In a preferred embodiment, the steps of constructing a virtual scene of the target operation region based on the three-dimensional point cloud real-time environment data and sending the picture data of the virtual scene to the MR interaction device specifically include: S210: Based on the three-dimensional point cloud real-time environment data, perform virtual scene modeling of the target operation region to obtain the virtual scene of the target operation region, and send the picture data of the virtual scene to the MR interaction device; S220: Analyze the scene configuration structure features of the virtual scene in the target operation area, establish a set of scene configuration structure features for the virtual scene, and store the set of scene configuration structure features.

[0027] In this embodiment, the acquisition of three-dimensional point cloud real-time environmental data is used to construct a virtual scene of the target operation area and send it to the MR interaction device for presentation. While constructing the virtual scene of the target operation area, it is also possible to complete the storage by extracting the scene configuration structure features of the virtual scene and establishing a set of scene configuration structure features for the virtual scene. This set of scene configuration structure features can be used to evaluate the formation of local ocean currents in the target operation area later.

[0028] In a preferred embodiment, according to the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction correlation data for each operation position and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy steps of the target operator in the target operation area, specifically including: S310: Receive the operation task plan of the target operator in the target operation area, extract the operation positions, operation contents, and operation task plan time limits of several operation points, and construct an operation task content list; S320: Considering the overall ocean current prediction correlation data for each operation position in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

[0029] Furthermore, considering the overall ocean current prediction correlation data for each operation position in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy steps of the target operator in the target operation area, specifically including: S321: Extract the set of scene configuration structure features of the virtual scene constructed from the overall ocean current prediction correlation data for each operation position in the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area; S322: Based on the overall ocean current prediction correlation data of each operation position and the scene construction structure features of the target operation area, perform regional ocean current type matching for the target operation area in the comparison table of the scene construction structure features required for the formation of ocean currents in different regions and the overall ocean current correlation data, and generate the regional ocean current set for each unit time period within the target execution time period of the target operation area; S323: Invoke the set of motion influence parameters of different regional ocean current types on the manipulator for executing the corresponding task type in the target operation area obtained through pre-testing, and extract the driving power adjustment ratio of each regional ocean current to each operation action of the manipulator recorded in the set of motion influence parameters; wherein, the operation actions include base movement actions, manipulator rotation actions, and manipulator rotation actions around the axis; S324: Based on the operation position, operation content, and operation task plan time limit of each operation point, considering the positions where different regional ocean currents are formed in different unit time periods within the target operation area and the standard movement speed of the base movement action of the manipulator, with the operation task completion time limit and operation task operation accuracy as the set of constraint conditions, and the minimum number of operation action optimization and update times as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

[0030] In practical applications, based on the operation position, operation content, and operation task plan time limit of each operation point, considering the positions where different regional ocean currents are formed in different unit time periods within the target operation area and the standard movement speed of the base movement action of the manipulator, with the operation task completion time limit and operation task operation accuracy as the set of constraint conditions, and the minimum number of operation action optimization and update times as the optimization goal, the steps of optimizing and solving the operation execution strategy of the target operator in the target operation area specifically include: S3241: Based on the operation position, operation content, and operation task plan time limit of each operation point, consider the positions where different regional ocean currents are formed in different unit time periods within the target operation area and the standard movement speed of the base movement action of the manipulator; S3242: In the actual execution order of several operation points in the operation task plan, take the total operation duration determined according to the operation duration corresponding to the operation content of each operation point and the movement distance and standard movement speed between adjacent two operation points being less than the operation task plan time limit as the first constraint condition, and the driving power adjustment ratio of the regional ocean current formed in the corresponding unit time period when the manipulator executes the operation content at each operation point to each operation action of the manipulator being less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of this operation point as the second constraint condition, and the sum of the number of times of switching the driving power adjustment ratio of each operation action of the manipulator within the entire operation task plan being the minimum as the optimization goal, and optimize and solve the operation point execution order of the operation task plan and the driving power adjustment ratio of each operation point for different operation actions; S3243: Generate an operation execution strategy for the target operator in the target operation area according to the operation point execution sequence and the drive power adjustment ratio of each operation point for different operation actions.

[0031] In this embodiment, by acquiring the real-time 3D point cloud environmental data and the overall ocean current prediction correlation data of the target operation area, constructing the picture data of the target operation area using the real-time 3D point cloud environmental data, and considering the overall ocean current prediction correlation data and the real-time 3D point cloud environmental data using the operation task plan of the target operator in the target operation area, estimate the possible regional ocean currents formed in the target operation area. Taking the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and minimizing the number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area. Use the operation point operation action optimization ratio in the operation execution strategy to adjust the operation force of the robotic arm in real time when performing different operation actions at each operation point, so as to adapt to the problems such as action deviation and low control accuracy caused by the overall ocean current and regional ocean currents in the ocean operation scenario, improve the operation accuracy and operation efficiency of the robotic arm teleoperation in the ocean operation scenario, and minimize the control difficulty as much as possible and solve the mechanical failure problems and operator adaptation problems caused by frequently switching the action optimization ratio.

[0032] In a preferred embodiment, the step of generating the robotic arm operation control guidance information according to the operation point execution sequence in the operation execution strategy and sending the robotic arm operation control guidance information to the MR interaction device specifically includes: S410: Generate the robotic arm operation control guidance information according to the operation point execution sequence in the operation execution strategy; wherein, the robotic arm operation control guidance information includes the operation position and operation content of each operation point arranged according to the operation point execution sequence; S420: Send the operation position and operation content of each operation point in the robotic arm operation control guidance information to the MR interaction device in sequence according to the operation point execution sequence.

[0033] On this basis, the step of acquiring the posture information of the target operator performing the virtual operation action of the robotic arm, generating the actual operation parameters of the robotic arm based on the posture information and the operation point operation action optimization ratio in the operation execution strategy, and controlling the robotic arm to perform the actual operation action specifically includes: S610: Acquire the posture information of the target operator performing the virtual operation action of the robotic arm, and convert the posture information into the standard drive power corresponding to the operation action; S620: Multiply the drive power adjustment ratio of each operation point for different operation actions in the operation execution strategy as the operation point operation action optimization ratio by the standard drive power of the current operation action to generate the actual drive power of each operation action; S630: Using the actual driving power of each working point for different working actions as the actual operating parameter of the robotic arm to control the robotic arm to perform actual operating actions.

[0034] In this embodiment, the execution order of the working points in the operation execution strategy is used as the robotic arm operation control guidance information to be fused with the screen data of the target operation area, generating a virtual scene screen of the target operation area, guiding the operator to execute the operation content of each working point within the operation task plan in sequence. At the same time, by adjusting the operation force of the robotic arm in real time when performing different working actions at each working point, it can adapt to the problems such as action deviation and low control accuracy caused by the overall ocean current and regional ocean current in the ocean operation scenario, improving the operation accuracy and operation efficiency of the robotic arm teleoperation in the ocean operation scenario.

[0035] Refer to Figure 2 , Figure 2 which is the structural block diagram of the embodiment of the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling of the present invention.

[0036] As Figure 2 shown, the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling proposed in the embodiment of the present invention includes: An acquisition module 10, configured to acquire the operation environment information of the target operation area; wherein, the operation environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; A construction module 20, configured to construct a virtual scene of the target operation area based on the three-dimensional point cloud real-time environment data, and send the screen data of the virtual scene to the MR interaction device; An optimization module 30, configured to consider the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environment data of the target operation area according to the operation task plan of the target operator in the target operation area, taking the operation task completion time limit and the operation task operation accuracy as a set of constraint conditions, and taking the minimum number of operation action optimization updates as the optimization goal, and optimizing and solving the operation execution strategy of the target operator in the target operation area; A sending module 40, configured to generate robotic arm operation control guidance information according to the execution order of the working points in the operation execution strategy, and send the robotic arm operation control guidance information to the MR interaction device; A generation module 50, configured to drive the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene, generating a virtual scene screen of the target operation area, so that the target operator can perform robotic arm virtual operation actions according to the virtual scene screen presented by the MR interaction device; An execution module 60 is configured to obtain the posture information of a target operator performing a virtual operation action of a robotic arm, generate actual operation parameters of the robotic arm based on the posture information and the operation action optimization ratio in the operation execution strategy, and control the robotic arm to perform an actual operation action.

[0037] For other embodiments or specific implementation manners of the high-precision control system of the robotic arm based on MR and three-dimensional point cloud modeling of the present invention, reference may be made to the above method embodiments, which will not be elaborated herein.

[0038] It can be understood that in the description of this specification, the descriptions referring to terms such as "one embodiment", "another embodiment", "other embodiments", or "the first embodiment to the Nth embodiment" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0039] It should be noted that in this article, the term "comprising", "including", or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or system including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article, or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or system including the element.

[0040] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling, characterized in that The method includes the following steps: Obtain the operation environment information of the target operation area; wherein, the operation environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; Based on the three-dimensional point cloud real-time environment data, construct a virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device; According to the operation task plan of the target operator in the target operation area, consider the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environment data of the target operation area, take the operation task completion time limit and operation task operation accuracy as the constraint condition set, and take the minimum number of operation action optimization updates as the optimization goal, and optimize and solve the operation execution strategy of the target operator in the target operation area; Generate robotic arm operation control guidance information according to the operation point execution order in the operation execution strategy, and send the robotic arm operation control guidance information to the MR interaction device; Drive the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene to generate a virtual scene screen of the target operation area, so that the target operator can perform robotic arm virtual operation actions according to the virtual scene screen presented by the MR interaction device; Obtain the posture information of the target operator performing the robotic arm virtual operation action, and generate actual robotic arm operation parameters based on the posture information and the operation point operation action optimization ratio in the operation execution strategy, and control the robotic arm to perform actual operation actions.

2. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, wherein The step of obtaining the operation environment information of the target operation area specifically includes: Receive the three-dimensional point cloud real-time environment data collected by the three-dimensional point cloud collection device deployed in the target operation area, access the pre-constructed overall ocean current prediction correlation database, and query the overall ocean current prediction correlation data of each operation position in the target operation area according to the area range information of the target operation area; Generate the operation environment information of the target operation area based on the three-dimensional point cloud real-time environment data and the overall ocean current prediction correlation data of the target operation area.

3. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 2, characterized in that, The construction of the overall ocean current prediction correlation database specifically includes: Obtain several groups of overall ocean current correlation data and overall ocean current influence parameter sets for different overall ocean current monitoring areas, extract the overall ocean current prediction influence characteristics in each group of overall ocean current influence parameter sets, and construct several overall ocean current prediction samples with the feature set constructed by the overall ocean current prediction influence characteristics and the corresponding overall ocean current correlation data; Wherein, the overall ocean current correlation data includes the overall ocean current speed and overall ocean current direction collected by the overall ocean current monitoring device set in the overall ocean current monitoring area, the overall ocean current influence parameter set includes several overall ocean current influence parameters in the associated overall ocean current area that affect the overall ocean current change collected by the ocean observation station, and the overall ocean current influence parameter includes at least one of sea surface temperature, sea surface height, seawater salinity and meteorological information; Use several overall ocean current prediction samples to train the pre-constructed initial convolutional neural network, and when the training reaches the target number of times or converges, obtain the trained overall ocean current correlation data prediction model; Extract the set of overall ocean current impact parameters corresponding to the target execution period of the target operation area recorded in the ocean environmental forecast information. Use the overall ocean current correlation data prediction model to predict the overall ocean current correlation data of each unit period within the target execution period of the target operation area, and construct an overall ocean current prediction correlation database.

4. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, characterized in that Based on the three-dimensional point cloud real-time environmental data, construct a virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device. The steps specifically include: Based on the three-dimensional point cloud real-time environmental data, perform virtual scene modeling of the target operation area, obtain the virtual scene of the target operation area, and send the screen data of the virtual scene to the MR interaction device; Analyze the scene configuration structure features of the virtual scene of the target operation area, establish a set of scene configuration structure features of the virtual scene, and store the set of scene configuration structure features.

5. The high-precision control method of the robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, characterized in that, According to the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area. The steps specifically include: Receive the operation task plan of the target operator in the target operation area, extract the operation positions, operation contents, and operation task plan time limits of several operation points, and construct an operation task content list; Considering the overall ocean current prediction correlation data of each operation position in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

6. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 5, wherein Considering the overall ocean current prediction correlation data of each operation position in the overall ocean current prediction correlation data of the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area. The steps specifically include: Extract the set of scene configuration structure features of the virtual scene constructed from the overall ocean current prediction correlation data of each operation position in the target operation area and the three-dimensional point cloud real-time environmental data of the target operation area; According to the overall ocean current prediction correlation data of each operation position and the set of scene structure features of the target operation area, perform regional ocean current type matching of the target operation area in the comparison table of scene structure features required for the formation of ocean currents in different regions and the overall ocean current correlation data, and generate the regional ocean current set of the target operation area in each unit period within the target execution period; Call the parameter set of the motion influence of different regional ocean current types obtained through pre-tests on the manipulator for performing corresponding task types in the target operation area, and extract the driving power adjustment ratio of each regional ocean current in the motion influence parameter set for each operation action of the manipulator; wherein, the operation actions include base movement actions, manipulator rotation actions, and manipulator rotation actions around the axis; Based on the operation position, operation content, and operation task planned time limit of each operation point, considering the positions of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement action of the manipulator, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization and update times as the optimization goal, optimize and solve the operation execution strategy of the target operator in the target operation area.

7. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 6, characterized in that, Based on the operation position, operation content, and operation task planned time limit of each operation point, considering the positions of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement action of the manipulator, with the operation task completion time limit and operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization and update times as the optimization goal, the steps of optimizing and solving the operation execution strategy of the target operator in the target operation area specifically include: Based on the operation position, operation content, and operation task planned time limit of each operation point, consider the positions of different regional ocean currents formed in different unit time periods in the target operation area and the standard movement speed of the base movement action of the manipulator; Taking the total operation duration determined according to the operation duration corresponding to the operation content of each operation point and the movement distance and standard movement speed between two adjacent operation points in the actual execution order of several operation points in the operation task plan being less than the operation task planned time limit as the first constraint condition, and the driving power adjustment ratio of each operation action of the manipulator by the regional ocean current formed in the corresponding unit time period when the manipulator executes the operation content at each operation point being less than the driving power adjustment ratio threshold corresponding to the operation action included in the operation content of this operation point as the second constraint condition, and the sum of the number of times of switching the driving power adjustment ratio of each operation action of the manipulator within the entire operation task plan being the minimum as the optimization goal, optimize and solve the execution order of the operation points of the operation task plan and the driving power adjustment ratio of each operation point for different operation actions; Generate the operation execution strategy of the target operator in the target operation area according to the execution order of the operation points and the driving power adjustment ratio of each operation point for different operation actions.

8. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, wherein According to the execution order of the operation points in the operation execution strategy, generate the manipulator operation control guidance information, and the steps of sending the manipulator operation control guidance information to the MR interaction device specifically include: Generate the manipulator operation control guidance information according to the execution order of the operation points in the operation execution strategy; wherein, the manipulator operation control guidance information includes the operation position and operation content of each operation point arranged according to the execution order of the operation points; Send the operation position and operation content of each operation point in the manipulator operation control guidance information to the MR interaction device in sequence according to the execution order of the operation points.

9. The high-precision control method for a robotic arm based on MR and three-dimensional point cloud modeling according to claim 1, wherein Obtain the posture information of the target operator performing the virtual operation action of the robotic arm, and generate the actual operation parameters of the robotic arm and control the robotic arm to execute the actual operation action steps based on the posture information and the operation point operation action optimization ratio in the operation execution strategy, specifically including: Obtain the posture information of the target operator performing the virtual operation action of the robotic arm, and convert the posture information into the standard driving power corresponding to the operation action; Take the driving power adjustment ratio of each operation point in the operation execution strategy for different operation actions as the operation point operation action optimization ratio and multiply it by the standard driving power of the current operation action to generate the actual driving power of each operation action; Take the actual driving power of each operation point for different operation actions as the actual operation parameters of the robotic arm to control the robotic arm to execute the actual operation action.

10. A high-precision control system for a robotic arm based on MR and three-dimensional point cloud modeling, characterized in that, Including: An acquisition module, configured to acquire the operation environment information of the target operation area; wherein, the operation environment information includes three-dimensional point cloud real-time environment data and overall ocean current prediction correlation data; A construction module, configured to construct a virtual scene of the target operation area based on the three-dimensional point cloud real-time environment data, and send the picture data of the virtual scene to the MR interaction device; An optimization module, configured to optimize and solve the operation execution strategy of the target operator in the target operation area according to the operation task plan of the target operator in the target operation area, considering the overall ocean current prediction correlation data of each operation position and the three-dimensional point cloud real-time environment data of the target operation area, with the operation task completion time limit and the operation task operation accuracy as the constraint condition set, and the minimum number of operation action optimization updates as the optimization goal; A sending module, configured to generate robotic arm operation control guidance information according to the operation point execution sequence in the operation execution strategy, and send the robotic arm operation control guidance information to the MR interaction device; A generation module, configured to drive the MR interaction device to embed the received robotic arm operation control guidance information into the virtual scene to generate a virtual scene picture of the target operation area, so that the target operator can perform the virtual operation action of the robotic arm according to the virtual scene picture presented by the MR interaction device; An execution module, configured to obtain the posture information of the target operator performing the virtual operation action of the robotic arm, and generate the actual operation parameters of the robotic arm and control the robotic arm to execute the actual operation action based on the posture information and the operation point operation action optimization ratio in the operation execution strategy.

Images