A method for trajectory optimization of a robot arm based on reinforcement learning

Abstract

The application relates to an artificial intelligence technology and discloses a mechanical arm trajectory optimization method based on reinforcement learning. The method comprises the following steps: firstly, a connecting rod coordinate system is constructed according to joint information of the mechanical arm, and a kinematics model of the mechanical arm is established on the basis; secondly, a reinforcement learning environment is built, wherein the rotation angle of the mechanical arm and the target pose of the work are taken as input states, and the stability degree in the working process of the robot is taken as a key reward item; thirdly, a pre-trained mechanical arm work control model is used to execute a decision generation operation based on pose reward and centroid reward, so that robot action instructions are generated; finally, the target mechanical arm robot is accurately controlled according to the action instructions. Through the trajectory planning method, the stability of the robot movement is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a mechanical arm trajectory optimization method based on reinforcement learning. BACKGROUND

[0002] In modern manufacturing and power industry, the maintenance and repair work of transmission lines and related equipment are crucial, and traditional manual live-line work has great safety hazards. In particular, key components of transmission lines, such as transmission line bolts and nuts, need to be maintained regularly.

[0003] Due to the special working environment on the transmission line, the dual-arm robot base is a floating base, which may cause the inclination angle of the mechanical arm to change during the working process, affecting the motion stability and centroid distribution of the mechanical arm, and causing shaking or more serious situations. The existing mechanical arm control method has limited performance in complex and dynamic environments, and often relies on predefined trajectories and rules, which is difficult to adapt to changes and requirements of the environment. SUMMARY

[0004] The present application provides a mechanical arm trajectory optimization method based on reinforcement learning, which mainly improves the motion stability of the mechanical arm through reinforcement learning.

[0005] To achieve the above purpose, the present application provides a mechanical arm trajectory optimization method based on reinforcement learning, comprising:

[0006] Obtain the joint information of the target mechanical arm robot, model the target mechanical arm robot according to the joint information, and obtain the mechanical arm motion model;

[0007] Real-time acquisition of the rotation angle of each joint in the mechanical arm motion model, and positioning the execution end position of the mechanical arm motion model according to the rotation angle, to obtain the execution end real-time state vector;

[0008] Using the pre-constructed scanning device in the base of the target mechanical arm robot, obtaining the nearby line scanning image of the target mechanical arm robot, and performing operation target object recognition operation on the nearby line scanning image to obtain the operation target, and identifying the target operation position of the operation target;

[0009] Obtain the mechanical arm composition information of the target mechanical arm robot, calculate the centroid position of a single mechanical arm of the target mechanical arm robot according to the execution end real-time state vector and the mechanical arm composition information, and obtain the single-arm centroid position;

[0010] According to the execution end real-time state vector and the target operation position, an operation distance vector is obtained.

[0011] The execution end real-time state vector is constructed for posture excitation control of the preset end posture, and the single-arm centroid position is constructed for centroid excitation control of minimum centroid change rate;

[0012] The operation distance vector is subjected to robot decision generation operation based on the posture excitation control and the centroid excitation control in the pre-trained robot arm operation control model, robot operation instructions are obtained, and the target robot arm is controlled according to the robot operation instructions.

[0013] Optionally, the target robot arm has a base and two robot arms, wherein the robot arms are connected to the base, the robot arms have N joints, and an execution end of a preset service is arranged at the outermost end of the robot arm, and the execution end can be replaced according to the type of the preset service.

[0014] Optionally, the execution end position of the robot arm motion model is positioned according to the rotation angle to obtain an execution end real-time state vector, including:

[0015] The rotation angles of the joints in the robot arm in sequence from the base to the execution end are sorted to obtain an angle change vector sequence;

[0016] The base position information of the base based on a world coordinate system is obtained, and the angle change vector sequence and the base position information are spliced to obtain an execution end real-time state vector.

[0017] Optionally, the operation target is obtained by performing operation target object identification operation on the nearby line scanning image, and the target operation position of the operation target is identified, including:

[0018] The nearby line scanning image is subjected to grayscale processing to obtain a grayscale scanning image, and the grayscale scanning image is subjected to Gaussian filtering processing to obtain a noise reduction grayscale image;

[0019] The target operation demand input by a user is obtained, and an operation target object corresponding to the target operation demand is queried from a pre-constructed technical manual database;

[0020] The noise reduction grayscale image is subjected to feature extraction operation to obtain an image feature set, and the image feature set is subjected to binary classification judgment operation based on the operation target object to obtain an operation target, and the operation target is subjected to frame selection marking to obtain a target mark;

[0021] The target mark is subjected to visual recognition analysis according to a pre-constructed time-of-flight method to obtain a target operation position of the operation target based on the base.

[0022] Optionally, the operation distance vector is expressed as:

[0023] s=(θ1, θ2, θ3,…, θ N , x, y, z, α, β, γ)

[0024] Where s represents the operation distance vector; θ1, θ2, θ3, ..., θ N represents the real-time state vector of the execution end; x, y, z represent the position information of the base; α, β, γ represent the target operation position.

[0025] Optionally, the use of a pre-trained manipulator operation control model to perform a robot decision generation operation based on the posture excitation control and the center of mass excitation control on the operation distance vector to obtain a robot operation instruction, and controlling the target manipulator robot according to the robot operation instruction includes:

[0026] Using a pre-trained robotic arm operation control model, based on a pre-built Markov algorithm, randomly generate primary control instructions within a preset time period that make the operation distance vector approach zero;

[0027] Calculating the primary control instruction to control the primary motion change angle of the robotic arm;

[0028] Obtain the initial joint angle corresponding to the start time of the preset time period, and calculate the predicted joint angle at the end time of the preset time period based on the initial motion change angle and the initial joint angle, wherein the predicted joint angle is expressed as:

[0029] obs2=(q1+θ1, q2+θ2, q3+θ3,…, q N +θ N )

[0030] In the formula, θ1, θ2, θ3,..., θ N represents the initial joint angle; q1, q2, q3, ..., q N represents the primary motion change angle, where q i ∈[-1rad, 1rad], rad represents the radian unit, i∈[1, ..., N];

[0031] According to the predicted joint angle, the robot arm is calculated based on the execution end posture and the center of mass to obtain the execution end posture and the center of mass change curve of the robot arm;

[0032] According to the posture incentive control and the center of mass incentive control, the execution posture is standardized and scored to obtain an execution posture score, and the center of mass change curve is scored based on a mean center of mass change rate to obtain a center of mass change score.

[0033] The execution posture score and the center of mass change score corresponding to each primary operation instruction randomly generated by the Markov algorithm are sequentially sorted, and a primary operation instruction with the highest execution posture score and the smallest center of mass change score is selected as a robot operation instruction.

[0034] Optionally, the center of mass incentive control is represented as:

[0035] R com =-(x 2 +y 2 ×α+z 2 )×β

[0036] In the formula, R com represents the center of mass incentive control; x, y, and z represent the relative position of the center of mass of the mechanical arm to the base, wherein x and z represent the front-back direction and the up-down direction of the base, y represents the side direction of the base, and alpha and beta represent weight coefficients, alpha>1.

[0037] To solve the above problems, the application further provides a mechanical arm trajectory optimization device based on reinforcement learning, which comprises:

[0038] A mechanical arm modeling module is configured to acquire joint information of a target mechanical arm robot, model the target mechanical arm robot according to the joint information, and obtain a mechanical arm motion model.

[0039] A mechanical arm action positioning module is configured to acquire the rotation angle of each joint in the mechanical arm motion model in real time, and position the execution end position of the mechanical arm motion model according to the rotation angle to obtain an execution end real-time state vector.

[0040] A target recognition module is configured to acquire a nearby line scanning image of the target mechanical arm robot by using a pre-constructed scanning device in the base of the target mechanical arm robot, perform work target object recognition on the nearby line scanning image to obtain a work target, and identify the target work position of the work target.

[0041] The posture-centroid excitation control module is used for acquiring mechanical arm component information of the target mechanical arm robot, performing centroid position calculation on a single mechanical arm of the target mechanical arm robot according to the real-time state vector of the execution end and the mechanical arm component information, obtaining a single-arm centroid position, obtaining an operation distance vector according to the real-time state vector of the execution end and the target work position, constructing posture excitation control of the real-time state vector of the execution end on a preset end posture, constructing centroid excitation control of the single-arm centroid position on a minimum centroid change rate, and generating robot decision operation on the operation distance vector based on the posture excitation control and the centroid excitation control by using a pre-trained mechanical arm work control model to obtain a robot operation instruction, and controlling the target mechanical arm robot according to the robot operation instruction.

[0042] To solve the above problems, the present application further provides an electronic device, comprising:

[0043] at least one processor; and,

[0044] a memory connected with the at least one processor in communication; wherein,

[0045] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the above-mentioned reinforcement learning-based mechanical arm trajectory optimization method.

[0046] To solve the above problems, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned reinforcement learning-based mechanical arm trajectory optimization method.

[0047] To solve the problems in the background art, the present application first models a target mechanical arm robot, then positions the position of the execution end of the mechanical arm, then identifies a target work position of a work target, calculates the distance between the real-time state vector of the execution end and the target work position, and then controls the robot and its mechanical arm to move to the target work position; during the movement, the present application usually uses pre-constructed centroid excitation control and posture excitation control to excite the action of the mechanical arm, wherein the centroid excitation control can prevent the centroid from deviating due to the movement of the mechanical arm, causing the robot to lose balance, and the posture excitation control can ensure that the execution end has the posture or angle for participating in work when it reaches the target work position. Therefore, the reinforcement learning-based mechanical arm trajectory optimization method, device, equipment and storage medium provided by the present application embodiment can improve the motion stability of the mechanical arm through reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of a method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 1.

[0049] Figure 2 A schematic diagram of a simplified model of a dual-arm robot in the method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 2.

[0050] Figure 3 A schematic diagram of a joint coordinate system of a dual-arm robot in the method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 3.

[0051] Figure 4 A detailed flowchart of a step in the method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 4.

[0052] Figure 5 A schematic diagram of a reinforcement learning environment setting for a right arm in the method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 5.

[0053] Figure 6 A functional module diagram of a device for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 6.

[0054] Figure 7 A structural schematic diagram of an electronic device for implementing the method for trajectory optimization of a robot arm based on reinforcement learning provided by an embodiment of the present application is shown in FIG. 7.

[0055] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein merely exemplify the present application and are not intended to limit the present application.

[0057] The embodiment of the present application provides a method for optimizing the trajectory of a robotic arm based on reinforcement learning. In the embodiment of the present application, the execution subject of the method for optimizing the trajectory of a robotic arm based on reinforcement learning includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for optimizing the trajectory of a robotic arm based on reinforcement learning can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0058] Reference Figure 1 FIG. 1 is a flow chart of a method for optimizing a robot arm trajectory based on reinforcement learning according to an embodiment of the present invention. In this embodiment, the method for optimizing a robot arm trajectory based on reinforcement learning includes:

[0059] S1. Acquire joint information of a target manipulator robot, and model the target manipulator robot according to the joint information to obtain a manipulator motion model.

[0060] For details, refer to Figure 2 As shown in the simplified model picture of the dual-arm robot, in an embodiment of the present invention, the target robotic arm robot has a base and two robotic arms, wherein the robotic arms are connected to the base, the robotic arms have N joints and the execution end of the preset business is configured at the outermost end of the robotic arms, and the execution end can be replaced according to the type of the preset business.

[0061] Specifically, the robotic arm in this embodiment of the present invention has six joints. The present invention sequentially records the movable angles of the six joints from the inside out, then records the length and mass distribution of the subsections of the robotic arm connected to each joint to obtain joint information. Finally, numerically constrained mathematical modeling is used to obtain a robotic arm motion model.

[0062] S2. Obtain the rotation angle of each joint in the robot arm motion model in real time, and locate the execution end position of the robot arm motion model according to the rotation angle to obtain the real-time state vector of the execution end.

[0063] In the embodiment of the present invention, the rotation angle of each joint is directly used as the motion positioning description of the robot arm.

[0064] In detail, in the embodiment of the present application, the execution end position of the mechanical arm motion model is positioned according to the rotation angle, and an execution end real-time state vector is obtained, which comprises:

[0065] The rotation angles of each joint in the mechanical arm from the base to the execution end are sorted in sequence to obtain an angle change vector sequence;

[0066] The base position information of the base based on the world coordinate system is obtained, and the angle change vector sequence and the base position information are spliced to obtain an execution end real-time state vector.

[0067] Specifically, in the embodiment of the present application, according to Figure 3 The schematic diagram of the joint coordinate system of the dual-arm robot shown in FIG. 1 can show that the base of the dual-arm robot of the present application adopts the world coordinate system O w The intermediate joint part of the robot arm or the execution section of the robot arm end adopts the robot base coordinate system O r The position is positioned. Each joint has its own rotatable positive direction, which can be configured according to the business type.

[0068] S3, using the pre-constructed scanning device in the base of the target robot arm, obtaining a nearby line scanning image of the target robot arm, and performing a work target object recognition operation on the nearby line scanning image to obtain a work target and identify a target work position of the work target.

[0069] In the embodiment of the present application, an optical scanning device is used to take a picture of the area near the base to obtain a nearby line scanning image. Then, a neural network model is used to identify the target in the nearby line scanning image to obtain a work target, such as a screw, a switch blade, etc.

[0070] In detail, referring to FIG. 1, Figure 4 In the embodiment of the present application, the work target object recognition operation on the nearby line scanning image is performed to obtain a work target and identify a target work position of the work target, which comprises:

[0071] S31, performing a grayscale processing on the nearby line scanning image to obtain a grayscale scanning image, and performing a Gaussian filter processing on the grayscale scanning image to obtain a noise reduction grayscale image;

[0072] S32, obtaining a target work requirement input by a user, and querying a work target object corresponding to the target work requirement according to a pre-constructed technical manual database;

[0073] S33, a feature extraction operation is performed on the noise reduction grayscale image to obtain an image feature set, a binary classification judgment operation based on the job target object is performed on the image feature set to obtain a job target, and the job target is marked by framing to obtain a target mark;

[0074] S34, visual recognition analysis is performed on the target mark according to a pre-constructed time-of-flight method to obtain a target job position of the job target based on the base.

[0075] The grayscale processing refers to converting an RBG three-channel color image into black and white, which is used to reduce the order of magnitude of the nearby line scanning image and increase the image recognition speed.

[0076] Further, the Gaussian filtering processing refers to performing mean value calculation on each connected pixel in the image according to the Gaussian formula, thereby smoothing the pixel difference between each pixel, reducing the noise points in the grayscale scanning image, and obtaining a noise reduction grayscale image.

[0077] Further, the technical manual database is a training manual and matters needing attention for guiding related power operation business, and according to the technical manual database, the part knowledge and use and maintenance method designed according to each business requirement can be identified.

[0078] Then, in the embodiment of the application, according to the target job requirement, such as maintenance, assembly, detection and packaging, the correct screw, switch, button, slot and line are found. Finally, the trained neural network model is used to identify these switches, buttons and screws, so as to identify the job target.

[0079] Finally, in the embodiment of the application, the visual recognition analysis refers to marking the job target, performing light pulse detection on the job target according to a pre-constructed time-of-flight method, calculating the target job position of the job target based on the base according to the time interval. The time-of-flight method refers to a method of emitting a light pulse by a TOF camera and measuring the time of the light pulse from the camera to the target object and back, thereby calculating the distance of the target object.

[0080] S4, obtain the mechanical arm composition information of the target mechanical arm robot, and calculate the center of mass position of each single mechanical arm of the target mechanical arm robot according to the real-time state vector of the execution end and the mechanical arm composition information, to obtain the single-arm center of mass position.

[0081] In the embodiment of the application, the mechanical arm composition information can be extracted from the joint information, and the length and mass distribution of each part of the mechanical arm are mainly obtained. Therefore, the overall shape distribution of the mechanical arm can be inferred according to the real-time state vector of the execution end.

[0082] In the embodiment of the present application, for the convenience of calculation, it is assumed that the lengths of each part of the mechanical arm are the same and the mass distribution is uniform, thus the centroid position of a single mechanical arm can be calculated through the overall shape distribution.

[0083] S5, obtaining an operation distance vector according to the real-time state vector of the execution end and the target operation position.

[0084] In the embodiment of the present application, the operation distance vector is the difference between the real-time state vector of the execution end and the target operation position, for the convenience of calculation, the present application directly adopts the method of recording the initial-target position values for expression, and the specific calculation process can be calculated through the built-in program of the computer.

[0085] In detail, in the embodiment of the present application, the operation distance vector is expressed as:

[0086] s=(θ1, θ2, θ3, θ4, θ5, θ6, x, y, z, α, β, γ)

[0087] In the formula, s represents the operation distance vector; θ1, θ2, θ3,..., θ N represent the real-time state vector of the execution end; x, y, z represent the base position information; and α, β, γ represent the target operation position.

[0088] In the embodiment of the present application, the first half of the operation distance vector

θ1, θ2, θ3,..., θ6, x, y, z

[0089] S6, constructing a posture excitation control of the real-time state vector of the execution end for a preset end posture, and constructing a centroid excitation control of the centroid position of the single arm for the minimum centroid change rate.

[0090] It should be known that, on the one hand, the high-altitude power operation scene of the dual-arm robot, on the other hand, the operation mode of different operation objects is fixed. Therefore, the execution section needs to operate the operation object at a fixed posture and angle to avoid safety accidents caused by non-standard action.

[0091] Specifically, in the embodiment of the present application, since the operation environment involved in the present application is the operation environment on the power transmission line, the purpose is to remove key components such as wire clamp bolts and nuts. The mechanical arm needs to reach the target operation position at a specific posture when operating. The position of the end effector of the mechanical arm reaching the target operation position is as shown in Figure 5 The rotation axis direction is z' direction.

[0092] Further, in the embodiment of the present application, the single-arm centroid is analyzed, and the centroid position change of the mechanical arm is punished. The greater the centroid change is, the higher the punishment is.

[0093] In detail, in the embodiment of the present application, the centroid excitation control is represented as:

[0094] R com =-(x 2 +y 2 ×α+z 2 )×β

[0095] In the formula, R com represents the centroid excitation control; x, y, and z represent the relative position of the centroid of the robot arm to the base, wherein x and z represent the front-back direction and the up-down direction of the base, y represents the side direction of the base, and α and β represent weight coefficients, and α>1.

[0096] Specifically, the reward function is as shown in the formula of the centroid excitation control, x, y, and z are coordinate changes of the centroid in three axes, and since the y axis has the greatest impact on the centroid of the robot, a weight α (>1) is given in the y axis, and a β weight is given as a whole.

[0097] S7, using the pre-trained robot arm operation control model, generating an operation of the robot decision based on the posture excitation control and the centroid excitation control on the operation distance vector, obtaining a robot operation instruction, and controlling the target robot arm robot according to the robot operation instruction.

[0098] In the embodiment of the present application, the basic function of the robot arm operation control model is a distance reward function. The construction process of the distance reward function is as follows: a reverse proportional function is designed, the closer the distance between the robot end effector and the target, the higher the reward, and when the distance is less than a set threshold, an additional reward is added to speed up the robot to reach the target operation position.

[0099] In detail, in the embodiment of the present application, using the pre-trained robot arm operation control model, generating an operation of the robot decision based on the posture excitation control and the centroid excitation control on the operation distance vector, obtaining a robot operation instruction, and controlling the target robot arm robot according to the robot operation instruction, comprising:

[0100] Using the pre-trained robot arm operation control model, according to the pre-constructed Markov algorithm, a primary control instruction is randomly generated in a preset time period to make the operation distance vector tend to zero;

[0101] Calculating the primary motion change angle of the robot arm controlled by the primary control instruction;

[0102] Obtain the initial joint angle corresponding to the start time of the preset time period, and calculate the predicted joint angle at the end time of the preset time period based on the initial motion change angle and the initial joint angle, wherein the predicted joint angle is expressed as:

[0103] obs2=(q1+θ1, q2+θ2, q3+θ3,…, q N +θ N )

[0104] In the formula, θ1, θ2, θ3,..., θ N represents the initial joint angle; q1, q2, q3, ..., q N represents the primary motion change angle, where q i ∈[-1rad, 1rad], rad represents the radian unit, i∈[1, ..., N];

[0105] According to the predicted joint angle, the robot arm is calculated based on the execution end posture and the center of mass to obtain the execution end posture and the center of mass change curve of the robot arm;

[0106] According to the posture incentive control and the center of mass incentive control, the execution end posture is scored in a standardized manner to obtain an execution posture score, and the center of mass change curve is scored based on the mean of the center of mass change rate to obtain a center of mass change score;

[0107] The execution posture scores and center of mass change scores corresponding to the primary control instructions randomly generated by the Markov algorithm are sequentially sorted, and the primary control instruction with the highest execution posture score and the smallest center of mass change score is selected as the robot operation instruction.

[0108] The Markov algorithm is a randomized algorithm based on the Markov process. A Markov process is a random process with the property of no aftereffect (i.e., the future state is dependent only on the current state, not on past states). In a Markov algorithm, the state of a system transitions to the next state with a certain probability. The transition probability of each state depends only on the current state and is unaffected by the sequence of previous states. This property makes the Markov algorithm widely applicable in modeling and predicting systems with randomness.

[0109] Specifically, the application acquires a primary motion change angle, and then determines a final position of the mechanical arm according to a target work position. Since the final position of the mechanical arm can be directly detected through the posture excitation control, and considering the center of mass excitation control, each time node in the motion time of the mechanical arm needs to be monitored, therefore, the application views the change of the center of mass in the motion process of the mechanical arm to obtain a center of mass change curve. The average of the center of mass change rate is calculated as the score of the center of mass change.

[0110] Finally, the application constructs a basic mechanical arm control model according to the distance reward function and the Markov function, and then further limits the motion scheme instruction generated by the model according to the above-mentioned posture excitation control and center of mass excitation control, so as to obtain a robot operation instruction with a stable posture and a stable center of mass, so that the mechanical arm quickly approaches the target work object, thereby performing power work control on the target mechanical arm robot.

[0111] To solve the problems in the background art, the application first models the target mechanical arm robot, then positions the position of the execution end of the mechanical arm, then identifies the target work position of the work target, calculates the distance between the execution end real-time state vector and the target work position, and then controls the robot and its mechanical arm to move to the target work position. During the movement, the application usually uses the pre-constructed center of mass excitation control and posture excitation control to excite the action of the mechanical arm. The center of mass excitation control can prevent the center of mass from deviating due to the motion of the mechanical arm, causing the robot to be unbalanced, and the posture excitation control can ensure that the execution end has a working posture or angle when it reaches the target work position. Therefore, the application embodiment provides a kind of mechanical arm trajectory optimization method based on reinforcement learning, which can improve the motion stability of the mechanical arm through reinforcement learning.

[0112] As Figure 6 shown, it is a functional module diagram of the mechanical arm trajectory optimization device based on reinforcement learning provided by an embodiment of the application.

[0113] The mechanical arm trajectory optimization device based on reinforcement learning 100 can be installed in an electronic device. According to the functions implemented, the mechanical arm trajectory optimization device based on reinforcement learning 100 can include a mechanical arm modeling module 101, a mechanical arm action positioning module 102, a target identification module 103, and a posture-center of mass excitation control module 104. The modules of the application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, which are stored in the memory of the electronic device.

[0114] In this embodiment, the functions of each module / unit are as follows:

[0115] The mechanical arm modeling module 101 is configured to acquire joint information of a target mechanical arm robot, model the target mechanical arm robot according to the joint information, and obtain a mechanical arm motion model.

[0116] The mechanical arm action positioning module 102 is configured to acquire a rotation angle of each joint in the mechanical arm motion model in real time, and position an execution end position of the mechanical arm motion model according to the rotation angle to obtain an execution end real-time state vector.

[0117] The target identification module 103 is configured to acquire a nearby line scanning image of the target mechanical arm robot by using a scanning device pre-constructed in a base of the target mechanical arm robot, perform a work target object identification operation on the nearby line scanning image, obtain a work target, and identify a target work position of the work target.

[0118] The posture-center-of-mass excitation control module 104 is configured to acquire mechanical arm composition information of the target mechanical arm robot, calculate a center-of-mass position of a single mechanical arm of the target mechanical arm robot according to the execution end real-time state vector and the mechanical arm composition information to obtain a single-arm center-of-mass position, obtain an operation distance vector according to the execution end real-time state vector and the target work position, construct posture excitation control of the execution end real-time state vector on a preset end posture, construct center-of-mass excitation control of the single-arm center-of-mass position on a minimum center-of-mass change rate, perform robot decision generation operation on the operation distance vector based on the posture excitation control and the center-of-mass excitation control by using a pre-trained mechanical arm work control model to obtain a robot operation instruction, and control the target mechanical arm robot according to the robot operation instruction.

[0119] In detail, each module in the reinforcement learning-based mechanical arm trajectory optimization apparatus 100 in the embodiment of the present application uses the same technical means as the reinforcement learning-based mechanical arm trajectory optimization method in the above Figures 1 to 5 , and can produce the same technical effects, which will not be described herein again.

[0120] As Figure 7 shown is a structural schematic diagram of an electronic device 1 for implementing a reinforcement learning-based mechanical arm trajectory optimization method according to an embodiment of the present application.

[0121] The electronic device 1 can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program stored in the memory 11 and executable on the processor 10, such as a reinforcement learning-based mechanical arm trajectory optimization program.

[0122] The processor 10 may, in some embodiments, be composed of integrated circuits, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits of the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device 1, and connects various components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as an arm trajectory optimization program based on reinforcement learning), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0123] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 11 may, in some embodiments, be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. The memory 11 may, in other embodiments, also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 may include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as the code of the arm trajectory optimization program based on reinforcement learning, but also to temporarily store data that has been or will be output.

[0124] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0125] The communication interface 13 is used for communication between the electronic device 1 and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0126] Figure 7 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 7 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0127] For example, although not shown, the electronic device 1 can also include a power supply (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be realized through the power management device. The power supply can also include one or more direct current or alternating current power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0128] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by this structure.

[0129] The reinforcement learning-based robot arm trajectory optimization program stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, which, when executed in the processor 10, can achieve:

[0130] Obtaining joint information of a target robot arm, modeling the target robot arm according to the joint information to obtain a robot arm motion model;

[0131] Real-time acquisition of the rotation angle of each joint in the mechanical arm motion model, and positioning the execution end position of the mechanical arm motion model according to the rotation angle, to obtain an execution end real-time state vector;

[0132] Using a pre-constructed scanning device in the base of the target mechanical arm robot, acquiring a nearby line scanning image of the target mechanical arm robot, and performing a work target object identification operation on the nearby line scanning image to obtain a work target and identify a target work position of the work target;

[0133] Acquiring mechanical arm composition information of the target mechanical arm robot, and performing centroid position calculation on a single mechanical arm of the target mechanical arm robot according to the execution end real-time state vector and the mechanical arm composition information, to obtain a single-arm centroid position;

[0134] According to the execution end real-time state vector and the target work position, an operation distance vector is obtained;

[0135] The execution end real-time state vector is constructed for a preset end posture posture excitation control, and the single-arm centroid position is constructed for a centroid change rate minimum centroid excitation control;

[0136] Using a pre-trained mechanical arm work control model, a robot decision generation operation based on the posture excitation control and the centroid excitation control is performed on the operation distance vector, to obtain a robot operation instruction, and the target mechanical arm robot is controlled according to the robot operation instruction.

[0137] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the related steps in the corresponding embodiment of the accompanying drawings, which will not be described here.

[0138] Further, the modules / units integrated by the electronic device 1, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM).

[0139] The application also provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0140] Obtain joint information of a target robot arm, model the target robot arm according to the joint information, and obtain a robot arm motion model;

[0141] Obtain the rotation angle of each joint in the robot arm motion model in real time, and position the execution end position of the robot arm motion model according to the rotation angle, and obtain an execution end real-time state vector;

[0142] Obtain a nearby line scanning image of the target robot arm by using a pre-constructed scanning device in the base of the target robot arm, perform a work target object identification operation on the nearby line scanning image, obtain a work target, and identify a target work position of the work target;

[0143] Obtain robot arm composition information of the target robot arm, calculate the center of mass position of a single robot arm of the target robot arm according to the execution end real-time state vector and the robot arm composition information, and obtain a single-arm center of mass position;

[0144] Obtain an operation distance vector according to the execution end real-time state vector and the target work position;

[0145] Construct a posture excitation control of the execution end real-time state vector for a preset end posture, and construct a center of mass excitation control of the single-arm center of mass position for a minimum center of mass change rate;

[0146] Use a pre-trained robot arm work control model to generate robot decision-making operation based on the posture excitation control and the center of mass excitation control for the operation distance vector, obtain robot operation instructions, and control the target robot arm according to the robot operation instructions.

[0147] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner.

[0148] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the present embodiment scheme.

[0149] In addition, each of the functional modules in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a hardware plus software function module.

[0150] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0151] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application being defined by the appended claims and not by the above description, therefore all variations falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be considered as limiting the claims to which they relate.

[0152] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, for verifying the validity of the information (anti-fake) and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0153] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0154] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names and not to indicate any particular order.

[0155] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for trajectory optimization of a robot arm based on reinforcement learning, characterized in that, The method comprises: Obtain joint information of a target robot arm, model the target robot arm according to the joint information, and obtain a robot arm motion model; Obtain the rotation angle of each joint in the robot arm motion model in real time, and position the execution end position of the robot arm motion model according to the rotation angle to obtain an execution end real-time state vector; Obtain a nearby line scanning image of the target robot arm by using a scanning device pre-constructed in the base of the target robot arm, perform work target object identification on the nearby line scanning image to obtain a work target, and identify the target work position of the work target; Obtain robot arm composition information of the target robot arm, calculate the center of mass position of a single robot arm of the target robot arm according to the execution end real-time state vector and the robot arm composition information, and obtain a single-arm center of mass position; Obtain an operation distance vector according to the execution end real-time state vector and the target work position; Construct a posture excitation control of the execution end real-time state vector for a preset end posture, and construct a center of mass excitation control of the single-arm center of mass position for a minimum center of mass change rate; Generate a primary control instruction in a preset time period that makes the operation distance vector tend to zero by using a pre-trained robot arm work control model and according to a pre-constructed Markov algorithm; Calculate the initial motion change angle of the robot arm controlled by the primary control instruction; obtaining an initial joint angle corresponding to a start time of the preset time period, and calculating a predicted joint angle of an end time of the preset time period according to the initial motion change angle and the initial joint angle, wherein the predicted joint angle is represented as: wherein denotes the initial joint angle; denotes the initial motion change angle, wherein , denotes the radian unit, ; According to the predicted joint angle, perform execution end posture and center of mass calculation on the robot arm to obtain the execution end posture and the center of mass change curve of the robot arm; According to the posture excitation control and the center of mass excitation control, standardize the score of the execution end posture to obtain an execution posture score, and perform a scoring operation on the center of mass change curve based on the average center of mass change rate to obtain a center of mass change score; According to the execution posture score and the center of mass change score corresponding to each primary control instruction randomly generated by the Markov algorithm, perform an order to select a primary control instruction with the highest execution posture score and the smallest center of mass change score as a robot operation instruction. wherein the centroid excitation control is represented as: In the formula, indicates the center-of-mass excitation control; indicates the relative position of the center of mass of the robot arm with respect to the base, wherein, indicates the front-rear direction and the up-down direction of the base, indicates the side direction of the base, indicates the weight coefficient, .

2. The reinforcement learning-based robot arm trajectory optimization method of claim 1, wherein, The target robot arm has a base, two robot arms, wherein the robot arms are connected to the base, each robot arm has N joints, and a preset business execution end is arranged at the outermost end of the robot arm, and the execution end can be replaced according to the type of the preset business.

3. The reinforcement learning-based robot arm trajectory optimization method of claim 2, wherein, The execution end real-time state vector is obtained by positioning the execution end position of the robot arm motion model according to the rotation angle, comprising: Sort the rotation angles of each joint in the robot arm in sequence from the base to the execution end to obtain an angle change vector sequence; Obtain the base position information of the base based on the world coordinate system, splice the angle change vector sequence and the base position information to obtain the execution end real-time state vector.

4. The reinforcement learning-based robot arm trajectory optimization method of claim 3, wherein, The work target object identification operation is performed on the nearby line scanning image to obtain a work target, and the target work position of the work target is identified, comprising: The nearby line scanning image is subjected to grayscale processing to obtain a grayscale scanning image, and the grayscale scanning image is subjected to Gaussian filtering processing to obtain a noise-reduced grayscale image; Obtain the target operation demand input by the user, and query the operation target object corresponding to the target operation demand according to the pre-constructed technical manual database; Perform feature extraction on the noise-reduced grayscale image to obtain an image feature set, perform binary classification judgment on the image feature set based on the operation target object to obtain an operation target, and perform frame selection marking on the operation target to obtain a target mark; According to the pre-constructed time-of-flight method, the target mark is subjected to visual recognition analysis to obtain the target operation position of the operation target based on the base.

5. The reinforcement learning-based robot arm trajectory optimization method of claim 4, wherein, The operation distance vector is represented as: In the formula, represents an operation distance vector; represents the execution end real-time state vector; represents the base position information; represents the target work position.

6. A reinforcement learning based trajectory optimization apparatus for performing the method of claim 1, characterized by The device comprises: A mechanical arm modeling module is configured to obtain joint information of a target mechanical arm robot, model the target mechanical arm robot based on the joint information, and obtain a mechanical arm motion model; A mechanical arm action positioning module is configured to obtain the rotation angle of each joint in the mechanical arm motion model in real time, and position the execution end position of the mechanical arm motion model based on the rotation angle to obtain an execution end real-time state vector; A target recognition module is configured to obtain a nearby line scanning image of the target mechanical arm robot by using a pre-constructed scanning device in the base of the target mechanical arm robot, and perform operation target object recognition on the nearby line scanning image to obtain an operation target and identify the target operation position of the operation target; A posture-center-of-mass excitation control module is configured to obtain mechanical arm composition information of the target mechanical arm robot, calculate the center-of-mass position of a single mechanical arm of the target mechanical arm robot based on the execution end real-time state vector and the mechanical arm composition information to obtain a single-arm center-of-mass position, obtain an operation distance vector based on the execution end real-time state vector and the target operation position, construct posture excitation control of the execution end real-time state vector for a preset end posture, construct center-of-mass excitation control of the single-arm center-of-mass position for a minimum center-of-mass change rate, and generate robot decision-making based on the posture excitation control and the center-of-mass excitation control by using a pre-trained mechanical arm operation control model to obtain robot operation instructions, and control the target mechanical arm robot based on the robot operation instructions.

7. An electronic device, comprising: The electronic device comprises: At least one processor; and A memory connected in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the reinforcement learning-based mechanical arm trajectory optimization method of any one of claims 1 to 5.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the reinforcement learning-based mechanical arm trajectory optimization method of any one of claims 1 to 5.

Images