Single-machine task scenario information acquisition and single-agent motion control method

Abstract

This invention provides a method for acquiring scene information for a single-machine task and a method for single-agent motion control. The single-machine scene information acquisition method enables the acquisition of information for complex dynamic environments, serving as input and training basis for intelligent strategies. The single-agent motion control method employs a high-efficiency reinforcement learning strategy for robot operations under dynamic, low-sample task conditions, effectively solving the entire process problems of object simulation, strategy generation, task generalization, and experience transfer in robot operation learning tasks. This invention studies image capture and data acquisition technologies applicable to intelligent sensing data in single-machine task scenarios, and uses the acquired information as a training database for deep learning training, effectively serving the robot's perception system. The single-agent motion control method of this invention, employing a single-agent motion control strategy and based on deep reinforcement learning methods, studies robotic arm grasping techniques for different targets, directly supporting future intelligent space robots.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for acquiring scene information for a single-machine task and controlling the motion of a single intelligent agent. Background Technology

[0002] Artificial intelligence (AI) is a strategic technology leading the future and will be a disruptive force in the field of national security, with an impact comparable to that of nuclear, aerospace, information, and biotechnology. Advances in deep space exploration will, to some extent, depend on the development of intelligent technologies; without intelligent autonomy, progress will be impossible. Manned spacecraft are developing and configuring intelligent robotic assistants within their sealed cabins to assist astronauts in their on-orbit activities. On-orbit servicing spacecraft will develop non-cooperative satellite capture systems based on computer vision and cognitive reasoning. Intelligent autonomous operation or supervised autonomous operation is becoming an important development direction for future space systems. Intelligent military aerospace development integrates multiple cutting-edge intelligent technologies, and so on. Therefore, seizing the commanding heights of future space exploration, deeply exploring the compatibility of AI and aerospace technology, and pooling collective resources to vigorously develop cutting-edge software, hardware, and system technologies such as space intelligent perception, space intelligent decision-making and control, space swarm intelligence, space intelligent interaction, and space intelligent design—to support the future of intelligent aerospace—is imperative.

[0003] Currently, the new generation of artificial intelligence is experiencing its third boom, which is also the optimal time for AI to take root in aerospace. With the deepening research into AI theories and methods, represented by Deep Reinforcement Learning (DRL), and the future development of large-scale space projects such as deep space exploration, manned lunar landings, and on-orbit space servicing, intelligent robots will be humanity's most powerful assistants in the complex, dangerous, and uncertain environment of space. In the near future, humanity will achieve the goal of "solving intelligence and using intelligence to solve everything."

[0004] The ultimate goal of my country's manned lunar landing mission is to establish a base on the lunar surface. The construction of an extraterrestrial base is an unprecedented and massive project, requiring thorough preparation before implementation. A crucial aspect of this is the development of space robots capable of adapting to the lunar environment, paving the way for base establishment and ultimately, manned lunar landings. However, due to the unique mission environment and usage requirements, traditional robot operation and control methods face numerous challenges in space missions. On the one hand, compared to robots operating on Earth and near-Earth space stations, lunar robots face limited control by astronauts, an unknown environment, and strict communication constraints (limited communication windows, time delays, limited bandwidth, etc.), rendering many mature control methods successfully applied on Earth inapplicable. On the other hand, as the complexity of space missions continues to increase, the formulation of traditional model-based control strategies generally requires precise dynamic descriptions and boundary conditions. The dynamic modeling process for nonlinear systems is often complex, leading to variations in the same control method depending on the usage environment, resulting in a large amount of repetitive modeling work. Therefore, finding new solutions through new theories and approaches to study the motion control of lunar robots and the collaboration between humans and machines, as well as among multiple machines, is crucial for the application of my country's basic aerospace research results and is a bottleneck issue that the aerospace leadership unit urgently needs to overcome.

[0005] Currently, robotic systems generally suffer from "excessive capability but insufficient intelligence." The application of artificial intelligence (AI) technology to spacecraft systems faces challenges such as limited hardware resources, small data samples, unclear mechanisms (black-box computing), and low reliability. In the long term, thoroughly resolving issues related to large-scale onboard computing, transmission, storage, and energy consumption requires innovation in cutting-edge technologies. Although significant progress has been made in environmental understanding and scene cognition in recent years, AI has not yet reached the application level in decision-making and control under complex and constrained conditions across multiple scenarios. In practical systems, it remains difficult to overcome limitations imposed by factors such as data, security, and training efficiency to form an effective intelligent control system. Currently, the major technological bottlenecks that urgently need to be overcome in building intelligent aerospace systems are concentrated in the following aspects:

[0006] 1) There is a lack of information acquisition methods for complex dynamic environments, which serve as the input and training basis for intelligent strategies;

[0007] 2) Lack of efficient reinforcement learning strategies for robot operation under dynamic, few-shot task conditions;

[0008] 3) There is a lack of effective methods to solve the entire process of object simulation, strategy generation, task generalization, and experience transfer in robot operation learning tasks. Summary of the Invention

[0009] In view of this, the present invention provides a single-machine task scenario information acquisition and single-agent motion control method. The single-machine task scenario information acquisition method can acquire information for complex dynamic environments as input and training basis for intelligent strategies. The single-agent motion control method adopts a high-efficiency reinforcement learning strategy for robot operation under dynamic few-sample task conditions, effectively solving the whole process problems of object simulation, strategy generation, task generalization and experience transfer in robot operation learning tasks.

[0010] To achieve the above objectives, the present invention provides a method for acquiring single-machine task scenario information, comprising the following steps:

[0011] Step 1: Through moving target detection, separate the moving target of interest from the background in the video or image sequence to obtain the position information of the moving target and extract the target object;

[0012] Step 2: Based on the detection of moving targets, the features of the target are analyzed, the position of the target in the video image sequence is determined, the position of the target's centroid is recorded, and the trajectory of the moving target is calculated, thereby realizing continuous tracking of the target.

[0013] Step 3: Based on HOG features and SVM classification, target recognition is achieved through a learning phase and a detection phase, completing the acquisition of scene information for the single-machine task. The learning phase includes the following steps: First, positive and negative sample data are collected. Positive samples contain only the target object, and negative samples do not contain the target object. Second, the feature information of the samples is extracted into the feature vector space, and the feature information of the image is represented by a vector model to obtain the feature vector. Third, the feature vector is input into the classifier for training and learning to generate the target classifier. The detection phase includes the following steps: First, based on target detection, features are extracted from the detection window image. Second, after obtaining the target feature vector, it is input into the trained target classifier, and the target is classified by scanning the detection window and marked with a rectangle. Third, the output results are merged into rectangles, merging overlapping small rectangles into a large rectangle to achieve target recognition.

[0014] Among them, HOG features are those that can characterize the detected local object contours by the distribution of light intensity gradients or edge directions.

[0015] The specific implementation of HOG features is as follows: First, gradient calculation is performed on the object pixels. Then, the sample image is divided into individual cell units. Each cell consists of n*n pixels. The size of n varies for different sample images. Every 2*2 cell units form a block. The size of the cell is the sliding step size of the detection window. Each cell generates an oriented gradient histogram. The combination of the histograms represents the descriptor. The HOG features of all blocks are combined to form the HOG feature vector.

[0016] This involves normalizing the cell units, which normalizes all cell units within the block.

[0017] In step 2, Kalman filtering, extended Kalman filtering, particle filtering, or Mean Shift tracking algorithm is used for target tracking. During the tracking process, a filter is introduced to predict the possible location of the target based on its motion characteristics.

[0018] This invention also provides a single-agent motion control method, which uses the single-machine task scene information acquisition method described in this invention to acquire information, uses the acquired information as input for motion control, and adopts a single-agent motion control strategy to control a robotic arm to grasp different targets based on a deep reinforcement learning method.

[0019] This includes the following steps:

[0020] Step 31: Construct a simulation environment that includes a physical model of a six-degree-of-freedom robotic arm, a moving target to be captured, and the physical properties of the surrounding environment; sensor information of the robotic arm is added to the simulation environment in the form of plug-ins and displayed in a visual manner.

[0021] Step 32, Fitting the joint model; where, before assembling the physical robotic arm, data from the input control torque and joint end torque sensor of each joint are collected, and a fully connected network is trained for approximation; the neural network model fits the motor, model, reducer model and friction model of the joint;

[0022] Step 33: The position, contact friction, contact stiffness and contact damping parameters of the target to be captured are Gaussian randomized. The expected value of each randomized parameter is the nominal value of the real world, and negative values ​​that do not conform to physical meaning are removed. The target to be captured is initially outside the workspace of the robotic arm and passes through the workspace of the robotic arm in a straight line or free fall. Its direction, position and velocity are randomized within a given range.

[0023] Step 34: Train the controller's motion space using the PPO algorithm from deep reinforcement learning algorithms, selecting the torques of each joint:

[0024] a = [τ1, τ2, τ3, τ4, τ5, τ6] T

[0025] The state space is selected as the target's coordinates, the coordinates of the robotic arm's end effector, joint angles, and angular velocities:

[0026]

[0027] The reward function is designed as follows:

[0028]

[0029] Where, λ i (i = 1, 2, 3) represents the weight of each part of the reward function, which is adjusted by λ. i The relative magnitudes of the values ​​are used to control the relative importance of each indicator, while limiting the energy consumed; x1, y1, z1 represent the three-dimensional coordinates of the target to be captured in the Cartesian coordinate system, and x, y, z represent the coordinates of the end effector of the robotic arm; E represents the energy consumed by the robotic arm at the current moment, expressed as the sum of the products of the rotational speeds and output torques of the six motors; T s T is the sampling time. f The third item represents the maximum simulation time, indicating that the longer the capture time, the greater the negative reward.

[0030] Step 35: Conduct training in the simulation environment until the cumulative reward function reaches the required value or the maximum number of training cycles is reached;

[0031] Step 36: Transfer the trained algorithm to the actual robotic arm control board. Use the initial state of the robotic arm in the simulation environment as the input of the controller. Deploy the moving target detection algorithm and tracking algorithm on another control board, configure the camera system, convert it to the same coordinate system as the simulation environment, and transmit the target position calculation results to the robotic arm controller. The robotic arm controller realizes the motion control of the single intelligent agent.

[0032] Among them, the target object was launched using parameters within the same range in the simulation to test the robotic arm's ability to capture moving targets in a real environment.

[0033] Beneficial effects:

[0034] 1. The present invention provides a method for acquiring information in a single-machine task scenario, which studies image capture and data acquisition technology suitable for intelligent sensing data in a single-machine task scenario, and uses the acquired information as a training database that can be used for deep learning training, effectively serving the robot perception system.

[0035] 2. In the single-machine task scene information acquisition method of the present invention, in order to improve accuracy, cell cells are normalized, and all cell units in the block are normalized. The normalization process compresses the influence of illumination and shadow, making their influence smaller, and finally forms the feature vector of the sample image, which is used as the classification category and input into the learning algorithm.

[0036] 3. The single-agent motion control method of this invention adopts a single-agent motion control strategy and studies the technology of robotic arms grasping different targets based on deep reinforcement learning. The main contents include: Algorithm design: designing a high-efficiency deep reinforcement learning algorithm, while adopting certain strategies to improve the learning efficiency of sparse samples, improve generalization ability, and improve portability; Simulation training: training in a virtual environment to achieve technical indicators; Algorithm porting: studying the pruning and post-processing methods of the algorithm and porting it to the actual controller; Physical operation: porting the algorithm to the physical system, ensuring the performance of the algorithm after porting, and exploring the possibility of online learning.

[0037] 4. This invention can directly support future intelligent space robots, which must not only possess characteristics such as adaptability to various complex environments in space and on extraterrestrial bodies, and high reliability, but also have the ability to perform tasks such as intelligent perception, precise operation, compliant operation, multi-level collaboration, autonomous decision-making, and control in complex, variable, and uncertain environments. Ultimately, they can collaboratively complete complex space tasks such as on-orbit grasping, repair, maintenance, and construction, as well as extraterrestrial body exploration and base construction. Simultaneously, it can be extended to civilian applications, supporting professional technical fields such as collaborative patrol operations. Attached Figure Description

[0038] Figure 1 This is a flowchart of the spatial object recognition process of the present invention.

[0039] Figure 2 This is a schematic diagram of the HOG algorithm structure of the present invention, (a) 40*40 pixel sample, (b) 64*128 pixel sample.

[0040] Figure 3 This is a flowchart of the PPO algorithm of the present invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] This invention provides a method for acquiring information in a single-machine task scenario, and studies image capture and data acquisition technologies suitable for intelligent sensing data in single-machine task scenarios. The method for acquiring information in a single-machine task scenario of this invention includes the following steps:

[0043] Step 1, Moving target detection;

[0044] Specifically, moving object detection is one of the fundamental research topics in video processing technology and machine vision. The purpose of moving object detection is to separate moving objects of interest from the background in a video or image sequence, obtain information such as the position of the moving object, and extract the target object as a basis for subsequent tracking or recognition. Therefore, the detection result has a direct impact on subsequent processing, and many scholars have been continuously improving moving object detection algorithms to obtain accurate and effective foreground images. Based on the relationship between the camera and the moving object, moving object detection algorithms are currently divided into two main categories: those under static backgrounds and those under dynamic backgrounds. Many relatively mature and stable object detection algorithms already exist, such as background subtraction, inter-frame subtraction, Gaussian mixture modeling, optical flow, and codebook algorithms. To address the incomplete target extraction of inter-frame subtraction, some scholars have proposed object detection methods based on background estimation. Background estimation is suitable for object detection under static backgrounds. Early methods proposed background estimation based on color and texture, such as Gaussian distribution models, Gaussian mixture modeling, and codebook models. Firedman and Russell proposed a single Gaussian distribution. Stauffer et al. argued that in environments with complex backgrounds and varying lighting, the single Gaussian distribution cannot fully represent the background, and proposed Gaussian mixture background modeling to solve the interference from background elements such as swaying tree branches. Gaussian mixture background modeling involves building multiple Gaussian models for each pixel in the image to describe its features. As each frame changes, the model is continuously updated, and the matching principle between the actual pixel in the image and the Gaussian mixture model is determined. If they match, the pixel is considered a background point; otherwise, it is considered a foreground point. Its advantage is that it can accurately detect the foreground even in complex backgrounds with varying lighting. However, this method still suffers from high computational cost and difficulty in distinguishing shadows. Therefore, this invention introduces a moving target tracking algorithm on top of moving target detection for moving target tracking.

[0045] Step 2, moving target tracking;

[0046] Specifically, the core idea of ​​moving target tracking is to analyze certain features of the target (such as color and contour) based on moving target detection, determine the target's position in the video image sequence, record the target's centroid position, and calculate the moving target's trajectory, thereby achieving continuous tracking of the target. Since target tracking can also be viewed as a state estimation process, filters can be introduced to predict the target's possible location based on its motion characteristics. Currently, widely used moving target tracking algorithms include Kalman filtering, extended Kalman filtering, particle filtering, and Mean Shift tracking algorithms.

[0047] Step 3: Target recognition based on HOG features and SVM classification achieves single-machine task scene information acquisition. Moving target detection describes the target's position and shape information; however, with the development of spatial intelligence, moving target detection alone cannot achieve these functions. Therefore, machine learning-based moving target recognition technology has become a popular research topic. The basic process of moving target recognition is: first, target samples are collected and their features are described; then, a classifier is trained using the sample's feature vectors. The classifier learns from the samples to obtain a reference template, thereby completing the classification and recognition of the test samples. In the object recognition method of this invention, the main recognition process includes two stages: a learning stage and a detection stage. Figure 1 As shown, the learning phase includes three steps: First, collecting positive and negative sample data. Positive samples contain only the object to be tested and should not contain excessive background, while negative samples do not contain the object to be tested and generally contain background information. Second, extracting the feature information of the samples into the feature vector space and representing the image feature information using a vector model. Third, inputting the feature vectors into the classifier for training and learning to generate the target classifier. The detection phase also mainly consists of three steps: First, based on target detection, extracting features from the detection window image. Second, inputting the obtained target feature vectors into the trained classifier, classifying the target through detection window scanning, and marking it with a rectangular box. Third, merging the rectangular boxes in the output results, combining overlapping small rectangles into a large rectangle.

[0048] This invention proposes a spatial object target recognition algorithm based on Histogram of Oriented Gradients (HOG) features and Support Vector Machine (SVM). Specifically, HOG stands for Histogram of Oriented Gradients, which uses gradient information from local image regions to represent the edge features of objects. HOG features maintain geometric and optical transformation invariance, meaning they are unaffected by changes in sunlight when used for spatial object recognition and detection. The core idea of ​​HOG features is that the detected local object contour can be characterized by the distribution of light intensity gradients or edge directions. The implementation involves first calculating the gradient of object pixels, then segmenting the image into small connected regions, also called cells. Each cell consists of n*n pixels; the value of n varies depending on the sample, possibly 4 or 8. Every 2*2 cells form a block, where the cell size corresponds to the sliding step size of the detection window. Each cell generates an oriented gradient histogram, and the combination of these histograms represents a descriptor. To improve accuracy, this invention normalizes the cell units, normalizing all cell units within the block. The normalization process compresses the effects of illumination and shadow, minimizing their influence. Finally, a feature vector is formed from the sample image, which serves as the classification category and is input into the learning algorithm.

[0049] The specific algorithm flow for HOG feature extraction is as follows:

[0050] a) Global image normalization

[0051] For the same image, the brightness may vary at different locations. The purpose of global image normalization is to reduce the influence of lighting. First, the entire image is normalized, and then Gamma compression is used to reduce noise interference and the effects of local shadows and lighting.

[0052] b) Calculate image gradient

[0053] By calculating the gradient of each color channel, information such as the edge, contour structure, and shape of objects in the image can be represented, and the influence of local shadows and lighting in the image is further weakened.

[0054] c) Statistical analysis of local image gradient information

[0055] The gradient information of local images is statistically analyzed and quantized, and a feature description vector is calculated for each local image region. This feature description vector describes the local image information and is invariant to small changes in edges, contours, and shapes within that region. Specifically, the entire image is divided into several connected regions, or cells. A weighted gradient histogram is calculated for each cell. Figure 2 This is a schematic diagram of the HOG algorithm structure.

[0056] d) Cell normalization

[0057] Due to variations in lighting, the influence of local object shadows, and high similarity between the target and background, the gradient intensity of an image can change significantly. Therefore, gradient intensity normalization is necessary. Normalization further weakens the effects of lighting, shadows, and edge blurring. The specific method involves combining cell units, grouping every 2x2 cells into a large connected region, known as a block. A complete image consists of several blocks. For each cell within a block, its feature vector is calculated. The combination of all cell feature vectors within a block yields the HOG feature of that block. Since connected regions (blocks) overlap, the feature vectors of individual cells within each connected region will be represented differently in the HOG feature. Normalizing these cell feature vectors ultimately forms the descriptor of the connected region block, i.e., the HOG feature descriptor.

[0058] e) Generate the final feature vector

[0059] The HOG features of all blocks are combined to form the final feature vector, namely the HOG feature vector, which describes the image content in the detection window.

[0060] This invention also provides a single-agent motion control method. Employing a single-agent motion control strategy, it studies robotic arm grasping techniques for different targets based on deep reinforcement learning. The main contents include: Algorithm design: designing a high-efficiency deep reinforcement learning algorithm, while adopting certain strategies to improve the learning efficiency of sparse samples, enhance generalization ability, and improve portability; Simulation training: training in a virtual environment to achieve technical indicators; Algorithm porting: researching pruning and post-processing methods for the algorithm and porting it to an actual controller; Physical operation: porting the algorithm to a physical system, ensuring the performance of the ported algorithm, and exploring the possibility of online learning.

[0061] Specifically, the steps include the following:

[0062] Step 31, establish the simulation environment: use the open-source V-rep physics engine to build a rapid simulation environment for the quadruped robot, including the physical model of the six-degree-of-freedom robotic arm, the moving target to be captured, and the physical properties of the surrounding environment; the sensor information of the robotic arm is added to the simulation environment in the form of plug-ins and displayed in a visual way.

[0063] Specifically, the kinematic parameters and mass inertia parameters of the robotic arm are read from the mechanical model using CAD software, and the joint drive model is fitted using a neural network. The measurement of the target position uses parameters such as accuracy and sampling time that are close to those of an actual detection and tracking system.

[0064] Step 32: Fitting the joint model; to improve the transplantation effect, before assembling the physical robotic arm, data from the input control torque and joint end torque sensors of each joint are collected, and a 40*30 fully connected network is trained for approximation. The neural network model fits the parameters of the joint's motor, model, reducer model, and friction model.

[0065] Step 33, Randomize Training Parameters: The position, contact friction, contact stiffness, and contact damping of the target to be captured are Gaussian randomized. The expected value of each randomized parameter is the nominal value in the real world, and negative values ​​that do not conform to physical meaning are removed. The target to be captured initially enters the workspace of the robotic arm and passes through the workspace of the robotic arm in a straight line or free fall. Its direction, position, and velocity are randomized within a given range.

[0066] Step 34: Train the controller using the PPO algorithm from deep reinforcement learning algorithms. The PPO algorithm flowchart is shown below. Figure 3 As shown, the motion space is selected as the torque of each joint:

[0067] a = [τ1, τ2, τ3, τ4, τ5, τ6] T

[0068] The state space is selected as the target's coordinates, the coordinates of the robotic arm's end effector, joint angles, and angular velocities:

[0069]

[0070] The reward function is designed as follows:

[0071]

[0072] Where, λ i (i = 1, 2, 3) represents the weight of each part of the reward function, which is adjusted by λ. i The relative magnitudes of the values ​​are used to control the relative importance of each indicator, while limiting the energy consumed; x1, y1, z1 represent the three-dimensional coordinates of the target to be captured in the Cartesian coordinate system, obtained through the single-machine task scene information acquisition method described in this invention; x, y, z represent the coordinates of the end effector of the robotic arm; E represents the energy consumed by the robotic arm at the current moment, expressed as the sum of the products of the rotational speeds and output torques of the six motors. T s T is the sampling time. f The third term represents the maximum simulation time, indicating that the longer the capture time, the greater the negative reward.

[0073] Step 35: Conduct training in the simulation environment until the cumulative reward function reaches the required value or the maximum number of training cycles is reached.

[0074] Step 36: Transfer the trained algorithm to the actual robotic arm control board. Use the initial state of the robotic arm in the simulation environment as the input of the controller. Deploy the moving target detection algorithm and tracking algorithm on another control board, configure the camera system, convert it to the same coordinate system as the simulation environment, and transmit the target position calculation results to the robotic arm controller. The robotic arm controller realizes the motion control of the single intelligent agent.

[0075] Furthermore, the robotic arm's ability to capture moving targets in a real-world environment can be tested by launching target objects with parameters from the same range in the simulation.

[0076] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A single-agent motion control method, characterized by, An information acquisition method of a single robot task scene is adopted to acquire information; the information acquisition method of the single robot task scene comprises the following steps: Step 1, through motion target detection, separating the motion target of interest from the background in a video or image sequence, obtaining the position information of the motion target, and extracting the target object; Step 2, analyzing the features of the target and determining the position of the target in the video image sequence, recording the position of the target centroid and calculating the trajectory of the motion target; Step 3, based on HOG features and SVM classification, through a learning stage and a detection stage, the recognition of the target is realized, and the information acquisition of the single robot task scene is completed; the learning stage comprises the following steps: collecting positive and negative sample data; extracting the feature information of the sample into a feature vector space, representing the feature information of the image with a vector model to obtain a feature vector; inputting the feature vector into the classifier for training and learning to generate a target classifier; the detection stage comprises the following steps: on the basis of target detection, feature extraction is performed on the detection window image; after obtaining the target feature vector, inputting it into the trained target classifier, classifying the target through detection window scanning, and identifying it with a rectangular frame; the output result is merged with the rectangular frame, and small rectangles that overlap together are merged into a large rectangular frame to realize the recognition of the target; Wherein, the HOG feature is that the detected local object contour can be represented by the distribution of light intensity gradient or edge direction; The single intelligent agent motion control method comprises: taking the acquired information as the input of motion control, adopting a single intelligent agent motion control strategy, and controlling the robot arm to grasp different targets based on a deep reinforcement learning method; Comprising the following steps: Step 31, a simulation environment including a six-degree-of-freedom robot arm physical model, a motion target to be grasped, and physical properties of the surrounding environment is constructed; sensor information of the robot arm is added to the simulation environment in the form of a plug-in and displayed through visualization; Step 32, fitting a joint model; wherein, before the physical robot arm is assembled, the data of the input end control torque and the joint end torque sensor of each joint are collected, and a fully connected network is trained for approximation; the neural network model fits the motor, model, reducer model and friction model of the joint; Step 33, the position, contact friction, contact stiffness and contact damping parameters of the target to be grasped are subjected to Gaussian randomization processing, the expected values of the randomized parameters are the nominal values in the real world, and the negative values that do not conform to the physical meaning are removed; the target to be grasped is initially outside the working space of the robot arm, and enters the working space of the robot arm in a straight line or free fall manner, and the direction, position and speed of the shot are randomly processed within a given interval; Step 34, using the PPO algorithm in the deep reinforcement learning algorithm to train the controller action space selected as the torque of each joint: The state space is selected as the coordinates of the target, the coordinates of the robot arm end, the joint angle and the angular velocity: The reward function is designed as: Step 35, training in the simulation environment until the cumulative reward function reaches the required value or the maximum training cycle number is reached; in, ​ i ( i=1,2,3 () represents the weight of each part of the reward function, which can be adjusted... ​ i The relative magnitude of each indicator is used to control its relative importance, while also limiting the energy consumed. x 1 ，y 1 ，z 1 represents the three-dimensional coordinates of the target to be captured in the Cartesian coordinate system. ​ Represents the coordinate values ​​of the robotic arm's end effector; E The energy consumed by the robotic arm at the current moment is represented by the sum of the products of the rotational speeds and output torques of the six motors. Sampling time, The third item represents the maximum simulation time, indicating that the longer the capture time, the greater the negative reward. ​ Step 36, the trained whole algorithm part is migrated to the actual mechanical arm control board, the initial state of the mechanical arm in the simulation environment is used as the input of the controller, the motion target detection algorithm and the tracking algorithm are deployed on another control board, the camera system is configured, the coordinate system consistent with the simulation environment is converted, the target position calculation result is transmitted to the mechanical arm controller, and the motion control of the single agent is realized through the mechanical arm controller.

2. The method of claim 1, wherein, The specific implementation of the HOG feature is: first, the gradient of the object pixel point is calculated, then the sample image is divided into cell units, each cell is composed of n*n pixels, different sample images select different sizes of n, and each 2*2 cell units form a block, wherein the size of the cell is the sliding step length of the detection window, a direction gradient histogram is generated for each cell, the combination of the histogram represents the descriptor, and the HOG features of all blocks are combined to form a HOG feature vector.

3. The method of claim 2, wherein, The cell unit is normalized, and all cell units in the block are normalized.

4. The method of claim 1, wherein, The same interval parameter projectile target object in the simulation is used to test the dynamic target capture ability of the mechanical arm in the actual environment.

Images