Coordination control method for multi-perception fusion dexterous hand

Abstract

The invention discloses a multi-perception fusion dexterous hand coordination control method, and relates to the technical field of robot control, and the method comprises the steps: initializing and calibrating a visual sensor, a touch sensor, a force feedback sensor and a temperature sensor, loading a software configuration file, and waiting for receiving a task instruction; on the basis of environment description and state estimation, presetting key operation points, allocating tasks to the dexterous hand by using a distributed task allocation algorithm, and calculating an action path and sequence; starting an event camera to monitor a key operation point, activating a sensor to perform data acquisition and fusion to form feedback data, and adjusting the action path and sequence of the dexterous hand in real time according to the feedback data; and generating a control instruction according to the adjusted action path and sequence, and sending the control instruction to each joint of the dexterous hand and an actuator through a controller for coordinated control. According to the invention, through multi-sensor data fusion, deep learning and genetic algorithm optimization, efficient coordination control of the dexterous hand is realized.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a coordinated control method for a multi-sensory fusion dexterous hand. Background Art

[0002] With the continuous advancement of robotics, dexterous hands, as high-precision, highly flexible intelligent robotic devices, have gradually gained widespread application in fields such as medicine, manufacturing, and automated assembly. In these applications, the collaboration, perception, and precise control capabilities of dexterous hands are key research areas. In recent years, with the development of multimodal sensor technology and deep learning algorithms, an increasing number of dexterous hand systems have begun integrating multiple sensors, such as vision, touch, and force feedback, to achieve real-time perception and adaptation to the environment. Existing multimodal fusion technologies use data from different sensors to enhance the robot's perception capabilities, improving the operational stability and flexibility of dexterous hands in changing environments.

[0003] However, despite advances in perception, existing technologies still face shortcomings in coordinated control and real-time feedback adjustments. Traditional control methods often rely on simple feedback loops and are unable to fully process and integrate complex data from diverse sensors. This results in poor adaptability and low execution accuracy in dynamic environments. Furthermore, existing path planning and task allocation methods typically utilize centralized control algorithms that lack optimization for complex task allocation, making it difficult to achieve efficient and precise coordinated control in practical applications. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a coordinated control method for a multi-sensory fusion dexterous hand to solve the problems of insufficient multi-sensory data fusion, poor coordination during task execution, and unreasonable task allocation of the dexterous hand.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a coordinated control method for a multi-sensory fusion dexterous hand, which includes initializing and calibrating a visual sensor, a tactile sensor, a force feedback sensor, and a temperature sensor, loading a software configuration file, and waiting to receive a task instruction;

[0008] After receiving the task instructions, it parses them through the virtual agent interface to generate the location information, shape information and material information of the target object;

[0009] Through multimodal deep learning models, position information, shape information, and material information are processed, features are extracted, and data is fused to output environment description and state estimation;

[0010] Based on the environment description and state estimation, key operation points are preset, and tasks are assigned to the dexterous hand using a distributed task allocation algorithm, and the action path and sequence are calculated;

[0011] Start the event camera to monitor key operation points, activate the sensor to collect and fuse data, generate feedback data, and adjust the movement path and sequence of the dexterous hand in real time based on the feedback data;

[0012] Based on the adjusted motion path and sequence, control instructions are generated and sent to the various joints and actuators of the dexterous hand through the controller for coordinated control.

[0013] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand of the present invention, the following steps are performed: initializing and calibrating the visual sensor, tactile sensor, force feedback sensor and temperature sensor, loading the software configuration file and waiting to receive the task instruction.

[0014] Set initialization parameters and calibrate the visual sensor, tactile sensor, force feedback sensor, and temperature sensor respectively;

[0015] Store the initialization parameters and calibration results of each sensor in a unified software configuration file;

[0016] Load software configuration files through Python's standard library and verify the content;

[0017] After the software configuration file is loaded and verified, it waits for input from external task instructions.

[0018] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand of the present invention, after receiving the task instruction, the virtual agent interface is used to analyze and generate the position information, shape information and material information of the target object. The specific steps are as follows:

[0019] Receive external task instructions using the network interface and parse them through the virtual agent interface;

[0020] Based on the parsed information, the virtual agent interface further decomposes the task instructions into multiple subtasks;

[0021] Based on each subtask, generate the corresponding position information, shape information and material information of the target object;

[0022] The generated position information, shape information, and material information are integrated and passed as input to the multimodal deep learning model.

[0023] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand described in the present invention, the following specific steps are used to process position information, shape information, and material information, extract features, and fuse data, and output environment description and state estimation.

[0024] Standardize the input position information, shape information and material information;

[0025] Use PointNet++ to extract geometric features from the normalized shape information, and use a convolutional neural network to extract material features from the material information;

[0026] Apply the Attention mechanism to perform weighted fusion of geometric features and material features, and generate a comprehensive feature vector based on position information;

[0027] Use graph neural networks to process comprehensive feature vectors and construct a graph model of the target object and its surrounding environment;

[0028] The Bayesian optimization algorithm is used to optimize the graphical model of the target object and its surrounding environment, and output the environment description and state estimation.

[0029] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand described in the present invention, the following specific steps are taken: based on the environment description and state estimation, key operation points are preset, tasks are assigned to the dexterous hand using a distributed task allocation algorithm, and the action path and sequence are calculated.

[0030] Based on the output environment description and state estimation, the Q-learning algorithm is used to identify the key points of task execution and set them as key operation points;

[0031] Confirm the location and task allocation of each key operation point through distributed task allocation algorithm;

[0032] Based on the location and task allocation of each key operating point, a summation function F(x) is constructed, which is expressed as follows:

[0033]

[0034] Among them, x represents the current position, t is the integral variable, A i represents the importance score of the i-th key operation point, B i is the distance from the current position to the i-th key operation point, C i The amount of resources required to complete the task of the i-th key operation point, i is the key operation point index variable, and n is the total number of key operation points;

[0035] By calculating the summation function, combining the importance score, distance, and the amount of resources required to complete the key operation point task, the evaluation data of each action path and sequence is comprehensively obtained;

[0036] According to the evaluation data of each action path and sequence, the action path and sequence are calculated step by step using recursion and divide-and-conquer methods.

[0037] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand of the present invention, the following specific steps are taken: activating an event camera to monitor key operation points, activating sensors to collect and fuse data, generating feedback data, and adjusting the movement path and sequence of the dexterous hand in real time based on the feedback data.

[0038] A high-resolution event camera is activated to monitor dynamic changes in the dexterous hand's workspace using an event-based visual processing algorithm and capture change information.

[0039] While starting the high-resolution event camera, activate and collect data from the visual sensor, tactile sensor, force feedback sensor and temperature sensor, and pre-process the sensor data through the Kalman filter;

[0040] The preprocessed sensor data and the change information captured by the event camera are deeply fused into feedback data through the ConvLSTM network, and a comprehensive evaluation function G(x) is constructed to quantify the impact of the feedback data. The expression is as follows:

[0041]

[0042] Among them, D a is the importance score of the a-th sensor data association, σ is the standard deviation, B a is the distance from the current position to the target point associated with the a-th sensor data, C a The amount of resources required to complete the target of associating the a-th sensor data, H a Score the complexity of the target environment associated with the a-th sensor data, is the square root value of the state error at the jth iteration, m is the total number of sensors, k is the total number of iterations, j is the iteration index variable, and a is the sensor index variable;

[0043] According to the value of the calculated comprehensive evaluation function, the motion path and sequence of the dexterous hand are evaluated, and the motion path and sequence are adjusted using a genetic algorithm.

[0044] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand of the present invention, the following steps are performed: the motion path and sequence of the dexterous hand are evaluated according to the value of the calculated comprehensive evaluation function, and the motion path and sequence are adjusted using a genetic algorithm.

[0045] Use machine learning algorithms to analyze historical calculation data and preset thresholds;

[0046] Compare the actual value of the comprehensive evaluation function with the preset threshold, and filter out the action paths and sequences that are higher than the preset threshold;

[0047] The action paths and sequences that are higher than the preset threshold are used as the initial population of the genetic algorithm, and the solution space is explored repeatedly through the selection, crossover and mutation operations of the genetic algorithm.

[0048] After multiple iterations, the genetic algorithm converges to an approximately optimal solution, optimizing and adjusting the action path and sequence.

[0049] As a preferred solution of the coordinated control method of the multi-sensory fusion dexterous hand of the present invention, wherein: according to the adjusted motion path and sequence, a control instruction is generated and sent to each joint and actuator of the dexterous hand through the controller for coordinated control. The specific steps are as follows:

[0050] Decompose the action path and sequence adjusted by genetic algorithm optimization into a series of subtasks;

[0051] Each subtask is mapped to a control instruction using a fuzzy logic controller;

[0052] Encode the control instructions and transmit them to the controller of the dexterous hand using error correction coding technology;

[0053] The controller is used to send the encoded control instructions to the joints and actuators of the dexterous hand, and a multi-objective optimization strategy is used for coordinated control.

[0054] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the coordinated control method of the multi-sensory fusion dexterous hand as described in the first aspect of the present invention is implemented.

[0055] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the coordinated control method of the multi-sensory fusion dexterous hand as described in the first aspect of the present invention.

[0056] The beneficial effects of the present invention are as follows: the present invention uses the fusion of multiple sensors such as vision, touch, force feedback and temperature sensors through multi-sensory fusion technology to achieve more accurate environmental perception and state estimation. In terms of environmental modeling, through the application of PointNet++ and graph neural networks, the extraction accuracy of geometric features and material features is effectively improved, providing an accurate environmental description for the task execution of the dexterous hand. Based on this, the present invention adopts distributed task allocation algorithm optimization, and combines Q-learning and genetic algorithms to monitor key operation points by starting event cameras, and pre-processing sensor data with Kalman filters, and then deeply fuses the ConvLSTM network to form feedback data, thereby achieving real-time adjustment of the dexterous hand's action path and sequence, and effectively enhancing the dexterous hand's adaptability and operation accuracy in complex and dynamic environments. These improvement measures work together to significantly improve the dexterous hand's coordination, real-time feedback capability and task execution accuracy, so that the dexterous hand can be more widely used in fine operation tasks in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 This is a flow chart of the coordinated control method of the multi-sensory fusion dexterous hand in Example 1.

[0059] Figure 2 This is a flowchart of loading a software configuration file and waiting to receive a task instruction in Example 1. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0062] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0063] Example 1, with reference to Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a coordinated control method for a multi-sensory fusion dexterous hand, comprising the following steps:

[0064] S1. Initialize and calibrate the visual sensor, tactile sensor, force feedback sensor, and temperature sensor, load the software configuration file, and wait for task instructions.

[0065] Furthermore, initialization parameters are set and calibration is performed for the visual sensor, tactile sensor, force feedback sensor, and temperature sensor respectively;

[0066] Specifically, select a high-precision RGB-D camera, configure the focal length and exposure time, and perform image acquisition. Use the checkerboard calibration method to obtain the camera's internal and external parameters and perform distortion correction on the visual sensor. Select a multi-point tactile sensor and set the sensitivity parameters to ensure that the tactile sensor can respond to subtle force changes. Select an F / T force feedback sensor, set the sensitivity and perform zero-point calibration. By recording the force feedback sensor response under known torque, perform multi-point calibration to ensure linear response under different torque conditions. Select a thermocouple temperature sensor, set the measurement range and output voltage value, expose the temperature sensor to a standard temperature source, and use the calibration curve to calibrate the linear relationship between temperature and voltage.

[0067] Store the initialization parameters and calibration results of each sensor in a unified software configuration file;

[0068] The configuration file contains the operating mode, sensitivity, and data transmission protocol for all sensors. Storing all sensor initialization parameters and calibration results in a unified software configuration file enables efficient information management and rapid access. This simplifies the management complexity of multi-sensor perception, facilitates maintenance and updates, and facilitates rapid deployment and debugging, thereby optimizing resource allocation and reducing maintenance costs.

[0069] Load software configuration files through Python's standard library and verify the content;

[0070] Ideally, a range check is performed on each sensor's key parameters (focal length, exposure time, and sensitivity). If any data is found to be mismatched or missing, the program will raise an error and reload and modify the software configuration file. Programmatically automating the software configuration file checking process reduces the risk of human error and improves processing efficiency. This provides a reliable data foundation for subsequent operations, ensuring operational stability and security.

[0071] After the software configuration file is loaded and verified, it waits for input from external task instructions.

[0072] S2. After receiving the task instruction, the virtual agent interface is used to parse and generate the location information, shape information and material information of the target object.

[0073] Furthermore, the network interface is used to receive external task instructions and parse them through the virtual agent interface;

[0074] The virtual agent interface is a software middleware responsible for receiving, parsing, and distributing instructions. It utilizes a network interface to receive external task instructions, enabling efficient processing and conversion of complex external instructions. This allows for efficient task instruction parsing through the virtual agent interface, effectively resolving compatibility issues caused by diverse instruction formats.

[0075] Based on the parsed information, the virtual agent interface further decomposes the task instructions into multiple subtasks;

[0076] Optimally, based on the parsed information, the virtual agent interface can further break down complex task instructions into multiple specific subtasks, enabling detailed management of task instructions. This simplifies the task execution process, allowing each subtask to be more accurately assigned to the most appropriate processing unit and module. This improves the flexibility and targeting of task execution, helps optimize resource allocation, and ensures that each subtask is completed efficiently.

[0077] Based on each subtask, generate the corresponding position information, shape information and material information of the target object;

[0078] Specifically, the object's positional information is extracted from the subtasks within the decomposed task instructions as the target object's spatial positioning data for subsequent path planning and physical interaction calculations. The target object's geometry, size, and shape information are extracted from the subtasks. Based on this shape information, the object's geometric features are calculated to support subsequent task assignment and path planning. The target object's material information is extracted from the task instructions. Based on this material information, the object is assigned relevant physical properties, and the task execution strategy is adjusted based on the material characteristics.

[0079] The generated position information, shape information, and material information are integrated and passed as input to the multimodal deep learning model.

[0080] S3. Process the position information, shape information and material information through a multimodal deep learning model, extract features and fuse data, and output environment description and state estimation.

[0081] Furthermore, the input position information, shape information and material information are standardized;

[0082] Optimally, by standardizing the input position, shape, and material information, the impact of varying data scales and noise is significantly reduced, ensuring more accurate and consistent data processing. This is especially true when input data comes from different sensors, as standardization can make multimodal data fusion more efficient.

[0083] Use PointNet++ to extract geometric features from the normalized shape information, and use a convolutional neural network to extract material features from the material information;

[0084] It should be noted that PointNet++ is used to extract geometric features from standardized shape information, primarily processing point cloud data to extract the spatial structural characteristics of objects. Convolutional neural networks are applied to material information, capturing details such as texture and color through convolutional layers. Combining PointNet++ with convolutional neural networks extracts geometric structure from shape information, delving deeper into the details of material information and enhancing the multimodal deep learning model's understanding of the multidimensional characteristics of target objects. This joint feature extraction approach can significantly improve recognition accuracy, particularly in application scenarios where object shape and material information may complement each other, helping to comprehensively characterize the multimodal information of target objects.

[0085] Apply the Attention mechanism to perform weighted fusion of geometric features and material features, and generate a comprehensive feature vector based on position information;

[0086] The Attention mechanism is used to weight the extracted geometric and material features, enhancing focus on important information. The Attention mechanism automatically assigns different weights to each feature type in different contexts, automatically selecting the most important features for the task. By weightedly fusing geometric and material features and combining them with position information to generate a comprehensive feature vector, the state of the target object and changes in its surroundings can be more accurately determined.

[0087] Use graph neural networks to process comprehensive feature vectors and construct a graph model of the target object and its surrounding environment;

[0088] Specifically, a graph neural network is used to process the comprehensive feature vector. Each feature extracted from the position information, shape information and material information is regarded as a node. The comprehensive feature vector is used as the initial feature of each node. The edge is defined according to the spatial proximity between the nodes. The message passing mechanism of the graph neural network is applied. Through multiple iterative message passing processes, the feature representation of each node is refined. By aggregating the feature representation of each node, a graph model of the target object and its surrounding environment is constructed.

[0089] The Bayesian optimization algorithm is used to optimize the graphical model of the target object and its surrounding environment, and output the environment description and state estimation.

[0090] The Bayesian optimization algorithm is an intelligent search strategy that uses a Gaussian process as a proxy model to predict performance under different parameter combinations and select the optimal parameter combination. By identifying adjustable parameters, such as the number of layers in PointNet++ and the number of message passes in convolutional neural networks, a reasonable search range is set for these adjustable parameters. Random parameter combinations are tested and the results are input into the proxy model. After multiple test iterations and training, the optimal parameter combination is predicted and its performance is actually tested. By optimizing the parameters of the graphical model of the target object and its surroundings, an environmental description and state estimate are output.

[0091] S4. Based on the environment description and state estimation, key operation points are preset, tasks are assigned to the dexterous hand using a distributed task allocation algorithm, and the action path and sequence are calculated.

[0092] Furthermore, based on the output environment description and state estimation, the Q-learning algorithm is used to identify the key points of task execution and set them as key operation points;

[0093] Specifically, based on the output environment description and state estimation, the Q-learning algorithm in reinforcement learning (RL) is used to identify and determine the key operation points. The state space is defined as all possible positions in the environment description, and the action space is defined as all basic operations that the dexterous hand can perform. The Q-learning update formula is:

[0094] Q(s,c)←Q(s,c)+β[r+γmax c′ Q(s′,c′)-Q(s,c)]

[0095] Where Q(s,a) represents the expected reward of taking action c in state s, β is the learning rate, r is the immediate reward, γ is the discount factor, s′ is the next state after taking action c, and c′ is one of all possible actions in the next state s′. Through multiple iterations, the key operation point that maximizes the cumulative reward is found.

[0096] Confirm the location and task allocation of each key operation point through distributed task allocation algorithm;

[0097] The optimal distributed task allocation algorithm efficiently allocates tasks and locations to key operational points, ensuring coordination and resource optimization across multiple key operational points. This algorithm achieves global goals through local optimization, improving scalability and robustness while mitigating the impact of single points of failure. The task allocation and location allocation for each key operational point ensures efficiency while avoiding excessive resource consumption and achieving efficient coordination of task execution.

[0098] Based on the location and task allocation of each key operating point, a summation function F(x) is constructed, which is expressed as follows:

[0099]

[0100] Among them, x represents the current position, t is the integral variable, A i represents the importance score of the i-th key operation point, B i is the distance from the current position to the i-th key operation point, C i The amount of resources required to complete the task of the i-th key operation point, i is the key operation point index variable, and n is the total number of key operation points;

[0101] It's important to note that the constructed summation function F(x) combines both continuous (integral) and discrete (summation) considerations. The integral portion normalizes the contributions at different distances, while the summation directly considers factors such as importance score, distance, and resource consumption. This combination combines multiple factors from the environmental description and task requirements (such as the importance of key operating points, reach distance, and required resources). This formula accurately reflects the multi-dimensional requirements of task execution, provides a scientific basis for path planning, and improves prediction accuracy.

[0102] Among them, the integral part It is a common Gaussian distribution integral function (the cumulative distribution function of the standard normal distribution) used to measure the relative distance or importance between the current position and the task area or target. This part can be regarded as the cumulative effect of a factor in the task (such as time, space, and distance). It provides the ability to weight distance for subsequent path calculations and can play a weighted role in the task sequence. Used to calculate the score of each key operation point based on the specific requirements of the task and the actual situation of the key operation point.

[0103] By calculating the summation function, combining the importance score of each key operation point, the distance, and the amount of resources required to complete the key operation point task, the evaluation data of each action path and sequence is comprehensively obtained;

[0104] Specifically, for each key operation point on the action path, use the formula To calculate the score of this point, the score of each action path and sequence is the sum of the scores of each key operation point. Combined with the importance score, distance and the amount of resources required to complete the key operation point task of each key operation point, the comprehensive score of each action path and sequence is compared to obtain the evaluation data of each action path and sequence.

[0105] According to the evaluation data of each action path and sequence, the action path and sequence are calculated step by step using recursion and divide-and-conquer methods.

[0106] Recursion decomposes a problem into smaller subproblems, while divide-and-conquer methods break down complex problems into simpler ones, solving them step by step and then combining the results. Applied to task path and sequence calculations, this approach can effectively reduce computational complexity and improve task execution efficiency by gradually approaching the optimal solution. In path calculations, the divide-and-conquer approach can break down complex problems into easily solvable subproblems, gradually calculating action paths and sequences. This avoids repeated operations during the calculation process, improves algorithm execution efficiency, and effectively reduces computational complexity.

[0107] S5. Start the event camera to monitor key operation points, activate the sensor to collect and fuse data, form feedback data, and adjust the movement path and sequence of the dexterous hand in real time based on the feedback data.

[0108] Furthermore, a high-resolution event camera is activated to monitor dynamic changes in the dexterous hand's workspace using event-based visual processing algorithms and capture change information.

[0109] Specifically, we selected a high-resolution event camera based on a dynamic vision sensor, which can detect brightness changes at the pixel level and maintain high stability even under high-speed motion and fluctuating lighting conditions. We then initialized the high-resolution event camera, adjusting parameters such as exposure time and frame rate, and performed calibration to ensure data accuracy.

[0110] Using the event-based visual odometry algorithm, a high-resolution event camera is used to capture and parse real-time event streams (i.e., the timestamps and locations of brightness changes). Each event stream contains the time and space coordinates (x, y) and the direction of the brightness change (increase or decrease). A spatiotemporal filter is used to extract spatiotemporal features, filtering out noise and irrelevant events. Based on the extracted spatiotemporal features, an optimization algorithm is used to estimate the position and posture of the object. The event camera is then started to capture change information within the workspace.

[0111] While starting the high-resolution event camera, activate and collect data from the visual sensor, tactile sensor, force feedback sensor and temperature sensor, and pre-process the sensor data through the Kalman filter;

[0112] The Kalman filter effectively handles state estimation problems for linear systems, while also combining it with nonlinear transformations to improve its applicability. It preprocesses data from various sensors, eliminating noise and enhancing signal quality. When the event camera is activated, the dexterous hand's visual, tactile, force feedback, and temperature sensors are activated, and data from these sensors is collected synchronously.

[0113] The preprocessed sensor data and the change information captured by the event camera are deeply fused into feedback data through the ConvLSTM network, and a comprehensive evaluation function G(x) is constructed to quantify the impact of the feedback data. The expression is as follows:

[0114]

[0115] Among them, D a is the importance score of the a-th sensor data association, σ is the standard deviation, B a is the distance from the current position to the target point associated with the a-th sensor data, C a The amount of resources required to complete the target of associating the a-th sensor data, H a Score the complexity of the target environment associated with the a-th sensor data, is the square root value of the state error at the jth iteration, m is the total number of sensors, k is the total number of iterations, j is the iteration index variable, and a is the sensor index variable;

[0116] It is important to note that the preprocessed sensor data and the change information captured by the event camera are fed into the ConvLSTM network. Its convolutional layers and LSTM units extract spatiotemporal features, deeply fuse the multimodal data, and output feedback data that comprehensively describes the dexterous hand's operating environment. The construction of the comprehensive evaluation function G(x) provides a quantitative basis for evaluating the dexterous hand's motion paths and sequences. Based on environmental changes and sensor feedback, it accurately measures the impact of feedback data on the dexterous hand's task execution, providing a scientific reference for adjusting the dexterous hand's movements.

[0117] The comprehensive evaluation function G(x) is calculated based on the fused feedback data through the following three key factors:

[0118] Integral Item This reflects the smoothness or consistency of data in a time series. During evaluation, the primary measure is temporal stability, which refers to the changing trend of sensor or event camera data over a certain period of time. Excessive data fluctuations can affect the stability of the dexterous hand's movements, making this a crucial factor in system performance.

[0119] Weighted sum term The contribution of each sensor data to the task and the resources used for task execution are combined to provide a quantitative indicator for subsequent action adjustments.

[0120] Normalization term The square root error value of each iteration in the feedback process is taken into account. Through normalization processing, it is possible to avoid certain high error values ​​in multiple iterations from having too great an impact on the final result, thereby ensuring the stability of the optimization process.

[0121] According to the value of the calculated comprehensive evaluation function, the motion path and sequence of the dexterous hand are evaluated, and the motion path and sequence are adjusted using a genetic algorithm.

[0122] Genetic algorithms, as an optimization tool, can effectively explore the action space by simulating the process of natural selection, thereby adjusting action sequences and paths. This process dynamically adjusts the dexterous hand's action strategy to adapt to varying operational requirements and environmental changes, improving operational efficiency and task success rates. The introduction of genetic algorithms improves the dexterous hand's flexibility and robustness in actual operations by adaptively optimizing path selection in complex and dynamically changing environments, enabling it to cope with a variety of uncertainties and complex task requirements.

[0123] Specifically, a machine learning algorithm is used to analyze historical calculation data and preset thresholds;

[0124] Based on the machine learning algorithm's predictions, a threshold is set using a percentile approach, set at the 95th percentile value. This threshold corresponds to the optimal action path and sequence in historical data, ensuring that only those paths and sequences that are extremely optimal are considered for subsequent screening. The preset threshold is not static; as historical data continues to accumulate, the machine learning model can be retrained and the threshold updated. Dynamic adjustments can be made based on actual application conditions. This dynamic adjustment mechanism ensures high adaptability and optimization effectiveness despite changing environmental conditions.

[0125] Compare the actual value of the comprehensive evaluation function with the preset threshold, and filter out the action paths and sequences that are higher than the preset threshold;

[0126] Specifically, the comprehensive evaluation function value of each new action path and sequence is compared with a preset threshold. If the evaluation value of a certain action path and sequence is higher than the threshold, it is considered to meet the conditions and enters the subsequent genetic algorithm operation. If the evaluation value is lower than the threshold, the action path and sequence is filtered out to avoid wasting computing resources.

[0127] The action paths and sequences that are higher than the preset threshold are used as the initial population of the genetic algorithm, and the solution space is explored repeatedly through the selection, crossover and mutation operations of the genetic algorithm.

[0128] After multiple iterations, the genetic algorithm converges to an approximately optimal solution, optimizing and adjusting the action path and sequence.

[0129] After multiple iterations, genetic algorithms can achieve near-optimal solutions. Based on these near-optimal solutions, they can optimize and adjust the action paths and sequences, significantly improving the efficiency and accuracy of these optimization paths and sequences. Traditional optimization methods may not be able to efficiently solve complex nonlinear or high-dimensional problems, but genetic algorithms, through swarm search and adaptive adjustments, can still achieve near-optimal solutions under complex conditions.

[0130] S6. Based on the adjusted motion path and sequence, control instructions are generated and sent to the joints and actuators of the dexterous hand through the controller for coordinated control.

[0131] Furthermore, the action path and sequence after genetic algorithm optimization and adjustment are decomposed into a series of subtasks;

[0132] Specifically, the action path and sequence optimized by the genetic algorithm are decomposed into tasks based on time sequence, spatial region, and operation type. The action path and sequence are broken down into multiple levels of tasks. The top-level task (T0) executes the entire optimized path. Subtasks (T1, T2, ..., Tm) perform tasks based on time sequence, spatial region, and operation type. Atomic tasks (A1, A2, ..., Ak) provide specific control instructions, such as adjusting the angle of finger opening or arm movement.

[0133] Each subtask is mapped to a control instruction using a fuzzy logic controller;

[0134] Among them, fuzzy logic controllers, by fuzzifying input variables, can flexibly map fuzzy inputs into reasonable control instructions, thereby coping with environmental uncertainties. Especially when faced with complex tasks and unknown environments, control strategies can be dynamically adjusted to ensure the stability and accuracy of task execution. Furthermore, fuzzy logic controllers can handle nonlinear relationships among various input data, thereby improving overall performance. In complex environments, the operation of dexterous hands often requires coping with various uncertain factors, such as changes in object shape and fluctuations in friction. By using fuzzy logic controllers to map each subtask into a control instruction, uncertainty and ambiguity can be more flexibly addressed.

[0135] Encode the control instructions and transmit them to the controller of the dexterous hand using error correction coding technology;

[0136] Preferably, by encoding control instructions and transmitting them using error-correcting coding technology, data transmission reliability can be significantly improved. Control instructions need to be transmitted to the dexterous hand controller via a communication network. During this process, they may be affected by noise interference or signal attenuation, resulting in errors during transmission. The use of error-correcting coding technology can effectively detect and correct errors during transmission, ensuring that control instructions are not distorted or lost during transmission, thereby improving fault tolerance and reliability. This method can avoid task failures or inaccurate dexterous hand operations caused by transmission errors, especially during long-term, high-frequency task execution, and can ensure that the dexterous hand always performs according to the predetermined control instructions.

[0137] The controller is used to send the encoded control instructions to the joints and actuators of the dexterous hand, and a multi-objective optimization strategy is used for coordinated control.

[0138] This embodiment also provides a computer device suitable for the coordinated control method of multi-sensory fusion dexterous hands, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the coordinated control method of multi-sensory fusion dexterous hands proposed in the above embodiment.

[0139] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0140] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the coordinated control method for realizing a multi-sensory fusion dexterous hand as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0141] In summary, the present invention achieves more accurate environmental perception and state estimation through: multi-sensory fusion technology, using the fusion of multiple sensors such as vision, touch, force feedback and temperature sensors. In terms of environmental modeling, through the application of PointNet++ and graph neural networks, the extraction accuracy of geometric features and material features is effectively improved, providing an accurate environmental description for the task execution of the dexterous hand. Based on this, the present invention adopts distributed task allocation algorithm optimization, and combines Q-learning and genetic algorithms. By starting the event camera to monitor key operation points, and using the Kalman filter to pre-process the sensor data, and then deeply fusing the ConvLSTM network to form feedback data, it realizes real-time adjustment of the dexterous hand's action path and sequence, effectively enhancing the dexterous hand's adaptability and operation accuracy in complex and dynamic environments. These improvement measures work together to significantly improve the dexterous hand's coordination, real-time feedback capability and task execution accuracy, enabling the dexterous hand to be more widely used in fine operation tasks in different fields.

[0142] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A coordinated control method for a multi-sensory fusion dexterous hand, characterized by: include, Initialize and calibrate the visual sensor, tactile sensor, force feedback sensor, and temperature sensor, load the software configuration file, and wait for task instructions; After receiving the task instructions, it parses them through the virtual agent interface to generate the location information, shape information and material information of the target object; Through multimodal deep learning models, position information, shape information, and material information are processed, features are extracted, and data is fused to output environment description and state estimation; Based on the environment description and state estimation, key operation points are preset, and tasks are assigned to the dexterous hand using a distributed task allocation algorithm, and the action path and sequence are calculated; Start the event camera to monitor key operation points, activate the sensor to collect and fuse data, generate feedback data, and adjust the movement path and sequence of the dexterous hand in real time based on the feedback data; Based on the adjusted motion path and sequence, control instructions are generated and sent to the various joints and actuators of the dexterous hand through the controller for coordinated control.

2. The coordinated control method for a multi-sensory fusion dexterous hand according to claim 1, characterized in that: The steps of initializing and calibrating the visual sensor, tactile sensor, force feedback sensor and temperature sensor, loading the software configuration file and waiting to receive the task instruction are as follows: Set initialization parameters and calibrate the visual sensor, tactile sensor, force feedback sensor, and temperature sensor respectively; Store the initialization parameters and calibration results of each sensor in a unified software configuration file; Load software configuration files through Python's standard library and verify the content; After the software configuration file is loaded and verified, it waits for input from external task instructions.

3. The coordinated control method for a multi-sensory fusion dexterous hand according to claim 2, characterized in that: After receiving the task instruction, the virtual agent interface is used to analyze and generate the location information, shape information and material information of the target object. The specific steps are as follows: Receive external task instructions using the network interface and parse them through the virtual agent interface; Based on the parsed information, the virtual agent interface further decomposes the task instructions into multiple subtasks; Based on each subtask, generate the corresponding position information, shape information and material information of the target object; The generated position information, shape information, and material information are integrated and passed as input to the multimodal deep learning model.

4. The coordinated control method for a multi-sensory fusion dexterous hand according to claim 3, characterized in that: The multimodal deep learning model is used to process position information, shape information and material information, extract features and fuse data, and output environment description and state estimation. The specific steps are as follows: Standardize the input position information, shape information and material information; Use PointNet++ to extract geometric features from the normalized shape information, and use a convolutional neural network to extract material features from the material information; Apply the Attention mechanism to perform weighted fusion of geometric features and material features, and generate a comprehensive feature vector based on position information; Use graph neural networks to process comprehensive feature vectors and construct a graph model of the target object and its surrounding environment; The Bayesian optimization algorithm is used to optimize the graphical model of the target object and its surrounding environment, and output the environment description and state estimation.

5. The coordinated control method of the multi-sensory fusion dexterous hand according to claim 4, characterized in that: Based on the environment description and state estimation, key operation points are preset, tasks are assigned to the dexterous hand using a distributed task allocation algorithm, and the action path and sequence are calculated. The specific steps are as follows: Based on the output environment description and state estimation, the Q-learning algorithm is used to identify the key points of task execution and set them as key operation points; Confirm the location and task allocation of each key operation point through distributed task allocation algorithm; Based on the location and task allocation of each key operating point, a summation function F(x) is constructed, which is expressed as follows: Among them, x represents the current position, t is the integral variable, dt is the small increment of the integral variable t, A i represents the importance score of the i-th key operation point, B i is the distance from the current position to the i-th key operation point, C i is the amount of resources required to complete the i-th key operation point task, i is the key operation point index variable, n is the total number of key operation points, is the cumulative distribution function of the standard normal distribution; By calculating the summation function, combining the importance score of each key operation point, the distance, and the amount of resources required to complete the key operation point task, the evaluation data of each action path and sequence is comprehensively obtained; According to the evaluation data of each action path and sequence, the action path and sequence are calculated step by step using recursion and divide-and-conquer methods.

6. The coordinated control method of the multi-sensory fusion dexterous hand according to claim 5, characterized in that: The event camera is activated to monitor key operation points, sensors are activated to collect and fuse data, and feedback data is generated. The action path and sequence of the dexterous hand are adjusted in real time according to the feedback data. The specific steps are as follows: A high-resolution event camera is activated to monitor dynamic changes in the dexterous hand's workspace using an event-based visual processing algorithm and capture change information. While starting the high-resolution event camera, activate and collect data from the visual sensor, tactile sensor, force feedback sensor and temperature sensor, and pre-process the sensor data through the Kalman filter; The preprocessed sensor data and the change information captured by the event camera are deeply fused into feedback data through the ConvLSTM network, and a comprehensive evaluation function G(x) is constructed to quantify the impact of the feedback data. The expression is as follows: Among them, D a is the importance score of the a-th sensor data association, σ is the standard deviation, B a is the distance from the current position to the target point associated with the a-th sensor data, C a The amount of resources required to complete the target of associating the a-th sensor data, H a Score the complexity of the target environment associated with the a-th sensor data, is the square root value of the state error at the jth iteration, m is the total number of sensors, k is the total number of iterations, j is the iteration index variable, a is the sensor index variable, is part of the probability density function of the standard normal distribution; According to the value of the calculated comprehensive evaluation function, the motion path and sequence of the dexterous hand are evaluated, and the motion path and sequence are adjusted using a genetic algorithm.

7. The coordinated control method of the multi-sensory fusion dexterous hand according to claim 6, characterized in that: The steps of evaluating the motion path and sequence of the dexterous hand based on the value of the calculated comprehensive evaluation function and adjusting the motion path and sequence using the genetic algorithm are as follows: Use machine learning algorithms to analyze historical calculation data and preset thresholds; Compare the actual value of the comprehensive evaluation function with the preset threshold, and filter out the action paths and sequences that are higher than the preset threshold; The action paths and sequences that are higher than the preset threshold are used as the initial population of the genetic algorithm, and the solution space is explored repeatedly through the selection, crossover and mutation operations of the genetic algorithm. After multiple iterations, the genetic algorithm converges to an approximately optimal solution, optimizing and adjusting the action path and sequence.

8. The coordinated control method of the multi-sensory fusion dexterous hand according to claim 7, characterized in that: According to the adjusted motion path and sequence, control instructions are generated and sent to the joints and actuators of the dexterous hand through the controller for coordinated control. The specific steps are as follows: Decompose the action path and sequence adjusted by genetic algorithm optimization into a series of subtasks; Each subtask is mapped to a control instruction using a fuzzy logic controller; Encode the control instructions and transmit them to the controller of the dexterous hand using error correction coding technology; The controller is used to send the encoded control instructions to the joints and actuators of the dexterous hand, and a multi-objective optimization strategy is used for coordinated control.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the coordinated control method of the multi-sensory fusion dexterous hand according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the coordinated control method of the multi-sensory fusion dexterous hand according to any one of claims 1 to 7 are implemented.

Images