A Multi-Axis Robotic Arm Control Method Based on Image Processing

Abstract

This invention belongs to the field of robot control technology and provides a multi-axis robotic arm control method based on image processing. The method unifies the acquisition devices of the multi-axis robotic arm in space and time, acquiring static, three-dimensional, and dynamic data, and extracting feature parameters from each data set to form a collection. The feature collections from different acquisition cycles are normalized and missing terms are filled, then input into an image processing model to obtain predicted trajectories. Predicted trajectories from multiple acquisition cycles are combined to form a predicted sequence, which is then smoothed and trimmed with a unit tangent vector to obtain the predicted poses from multiple acquisition cycles. The deviation between the predicted pose and the actual executed pose is calculated, and a correction mode is selected based on the degree and type of deviation. This mode is input into a policy generation model to obtain policy probabilities, which are then sent to the multi-axis robotic arm to execute the policy. The actual feedback is used in conjunction with a loss function to update the parameters of the policy generation model.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically a multi-axis robotic arm control method based on image processing. Background Technology

[0002] Multi-axis robotic arms are robotic devices that are automatically controlled, reprogrammable, multi-degree-of-freedom, and versatile, playing an important role in fields such as industrial automation and precision manufacturing.

[0003] Chinese patent application number CN202510053682.0 discloses a control method for a multi-axis robotic arm. This method includes: acquiring relative position images of the actuators of the multi-axis robotic arm and a target object using sensing devices, and extracting visual features; inputting a reinforcement learning model for the target task, and having the reinforcement learning model output a corresponding control strategy; generating a corresponding first control command based on the control strategy and the current image, and controlling the multi-axis robotic arm and actuators to perform actions on the target object according to the first control command. This solves the problems of high difficulty in operating multi-axis robotic arms and poor accuracy in controlling complex tasks in related technologies.

[0004] In robot control technology, although reinforcement learning models are used to train and correct multi-axis robotic arms based on output control strategies, addressing the high difficulty of manipulating multi-axis robotic arms, in the assembly of flexible electronic devices, flexible materials are prone to deformation or twisting during manipulation, and devices may also experience positional deviations due to thermal or stress factors. Static robotic arms struggle to respond instantly, leading to misalignment between the robotic arm's execution path and the actual contact point. Therefore, a method based on image processing for dynamically adjusting the robotic arm while simultaneously achieving high-precision, flexible, and adaptive control is needed. Summary of the Invention

[0005] This invention provides a multi-axis robotic arm control method based on image processing, aiming to solve the problem that statically set multi-axis robotic arms are difficult to dynamically adjust according to deformation when assembling flexible electronic devices, thus compensating for the accuracy issue of the robotic arm's execution path and the actual contact point. The technical solution adopted by this invention to solve the above-mentioned technical problem is to provide a multi-axis robotic arm control method based on image processing: acquiring multi-dimensional images of flexible devices, extracting and analyzing device feature parameters, constructing an image analysis model based on the parameters, predicting the motion trajectory of the flexible device, dynamically correcting the trajectory through strategies, and feeding back the results to iterate the strategy mapping network model.

[0006] As a preferred implementation, the specific steps for acquiring multi-dimensional images of flexible devices are as follows: An image acquisition device is installed in the working area of ​​the multi-axis robotic arm; a checkerboard calibration plate is placed at different positions; the captured checkerboard images are used to generate corresponding world coordinates; a calibration algorithm is called to solve for internal and external parameters; multiple acquisition devices are calibrated to achieve spatial uniformity; a master clock and slave clock are established using PTP (Precision Time Protocol); data packets are sent to obtain round-trip delay and the device's own timestamp; and the timestamps of the acquisition devices are unified. For the positioned flexible device, static images are captured from top and side views using the acquisition device, recording the device's contour and texture data to generate static data. The acquisition device simultaneously captures RGB images and depth maps; a color depth map is generated through matching calibration; and the RGB images are fused with a disparity map obtained through an algorithm to obtain a three-dimensional model. RGB images and depth maps of the flexible device are continuously acquired, sorted by timestamp to form an image stream, creating dynamic data. The static data, three-dimensional model, and dynamic data are integrated to constitute the multi-dimensional data of the flexible device.

[0007] As a preferred embodiment, the specific steps for extracting and analyzing the feature parameters of the device are as follows: denoising the RGB image in the static data using a Gaussian filter; performing median filtering on the corresponding depth map; cropping the RGB image to the same size as the predefined acquisition area; applying the Canny edge detection algorithm to the image using Gaussian filtering to obtain a smooth image; calculating the gradient magnitude and direction of each pixel; setting high and low thresholds to filter pixels to extract the edge pixel set; calculating the edge pixel contour area; selecting the one with the largest contour area from the edge pixel set and setting it as the main contour; fitting an outer matrix to record the center point and rotation angle; selecting the main contour of the depth map pixel set; and combining the main contours corresponding to the RGB image set and the depth map set {(x i ,y i ,d i )}, where x, y represent image coordinates, d represents depth map depth, and i represents RGB image i; using least squares plane fitting, the equation of the reference plane is established as follows:

[0008] z = ax + by + c

[0009] Where a, b, and c are the first, second, and third undetermined coefficients, respectively; the warp of the RGB image is calculated using the following formula:

[0010] warp i =d i -z(x i ,y i ),

[0011] Where d i warp represents the depth of RGB image i.i The warp of RGB image i is represented by z(x). i ,y i ) indicates that the RGB image i is on the reference plane (x i ,y i The value corresponding to the point;

[0012] Combining warp, the center point of the circumscribed matrix, and the rotation angle, a static feature set is formed. The average value of the coordinates in the 3D model is calculated to obtain the model centroid. The difference between the coordinates of each point in the model and the centroid coordinates is used to construct the covariance matrix and solve for the direction vector. Combining the direction vector and the centroid coordinates, a 3D feature set is formed. Combining the static feature set of continuous time periods and the corresponding 3D model centroid, the motion angular velocity and centroid velocity are calculated. The Farneback algorithm is used on two consecutive frames. The parameters are set and the corresponding grayscale image is input. Each pixel is split into components to obtain the pixel motion vector. Combining the motion angular velocity, centroid velocity, and pixel motion vector, a dynamic feature set is formed.

[0013] As a preferred implementation, the specific steps for constructing an image analysis model based on parameters are as follows: Collect feature sets from multiple periods, concatenate them into feature vectors in static, three-dimensional, and dynamic order, normalize each dimension within the feature vectors, fill in missing values ​​using the mean of previous and subsequent periods for feature items, select a multilayer perceptron model, use the aforementioned feature vectors as input, divide them into training data, validation data, and test data in an 8:2:2 ratio, calculate the mean and standard deviation of each dimension in the training data, set the mean square error as the loss function, calculate the loss value using the validation data, set the parameter with the smallest loss value as the multilayer perceptron model parameter, calculate the mean error and root mean square error using the test data, update the multilayer perceptron model parameters, export the qualified multilayer perceptron model parameters to ONNX (Open Neural Network Exchange) format, introduce an inference engine into the architecture of the multi-axis robotic arm controller, load the ONNX format multilayer perceptron model parameters, and after the acquisition device obtains the feature vectors, use the inference engine in the controller architecture to obtain the inference results.

[0014] As a preferred embodiment, the specific steps for predicting the motion trajectory of the flexible device are as follows: recording the initial position of the current period t, synchronously acquiring the feature set of period t, and sequentially aggregating them into a feature vector p. t The feature vector is normalized and input into the multilayer perceptron model. It is then mapped in the hidden and output layers of the model, projected onto the bias space, and propagated forward to obtain the prediction bias vector Δp. t The predicted vector is calculated from the feature vector using the following formula:

[0015] p′ t=p t +Δp t ,

[0016] Where p′ t p represents the prediction vector for period t. t Δp represents the eigenvector with period t. t The prediction deviation vector represents the period t;

[0017] Repeat the above steps to obtain prediction sequences for multiple periods. Extract a two-dimensional coordinate set from the prediction sequence, connect adjacent points of the two-dimensional coordinate set in chronological order to obtain a preliminary motion trajectory prediction curve, and smooth the prediction curve using a sliding window algorithm. Utilize the yaw angle θ corresponding to the two-dimensional coordinates in the prediction sequence... t Convert to a unit tangent vector:

[0018] V t =[cosθ t sinθ t ],

[0019] Where θ t V represents the yaw angle during period t. t Represents the unit tangent vector with period t;

[0020] For each inflection point on the predicted curve, the angle error between the curve's tangent and orientation is eliminated by correcting with a unit tangent vector, thus obtaining the predicted pose for multiple cycles.

[0021] As a preferred implementation, the specific steps of dynamically correcting the trajectory through the strategy are as follows: the deviation between the predicted pose and the actual pose at period k is recorded as the prediction difference, which is divided into positional deviation and angular deviation; and a correction mode δ is selected according to the type and degree of the prediction difference. k Each correction mode is pre-mapped with a corresponding label value, parameter value μ. k , will (δ k ,μ k As a correction decision for period k, the correction decisions for multiple periods are combined with the corresponding predicted poses to form a training set. A new policy mapping network is built in the deep learning framework, and the training set is used as the input parameter vector. The linear transformation unit in the multilayer perceptron model linearly transforms the input parameter vector, and the compensation component is obtained for each term of the prediction difference. The output policy probability is activated by Softmax, and the formula is:

[0022]

[0023] Where C represents the strategy probability, j represents the current acquisition period, m represents the m-th period, n represents the total number of acquisition periods, p represents the feature vector, and z(δ) j) represents the label value corresponding to the correction mode of period j, exp represents the exponential function, and p j p represents the eigenvector of period j. m This represents the eigenvector of the m-th period;

[0024] The compensation component of the prediction difference is combined with the output policy probability to form the loss function, as shown in the formula:

[0025]

[0026] Where α1 and α2 are the feature difference weight and the correction mode weight, respectively, and α1 + α2 = 1, h is the total number of compensation components, s represents the index of the current compensation component, and Δp s This represents the s-th true feature vector. Let represent the s-th predicted feature vector, u represent the total number of correction modes, and g represent the index of the current correction mode.

[0027] The predicted pose of the cycle is input into the strategy mapping network to obtain the real-time corrected trajectory. The corrected trajectory is then sent to the built-in controller of the multi-axis robotic arm, which drives the robotic arm according to the corrected trajectory.

[0028] As a preferred implementation, the specific steps of the feedback result iterative strategy mapping network model are as follows: After each acquisition of the corrected trajectory, the acquisition device records the actual trajectory of the multi-axis robotic arm and calculates it with the corrected trajectory issued by the strategy mapping network model. The deviation vector and the feature vector corresponding to the acquisition cycle are statistically analyzed as feedback data. A deviation threshold is set, and feedback data exceeding the deviation threshold is marked. The feedback data is standardized, and iterative training is performed on the feedback data using a small learning rate. The parameters of the strategy mapping network model after iteration are retained, and the current inference version is replaced. After deploying the strategy mapping network model parameters, the performance and feedback data are monitored in real time. The performance and feedback data are compared with the historical model parameter execution. If the performance and feedback data are not as good as the previous version, the system is rolled back to the stable version and the rollback log is recorded.

[0029] The beneficial effects of this invention are:

[0030] 1. It can respond to flexible deformation in real time. Through static, three-dimensional and dynamic feature data, it can perceive the deformation of flexible electronic devices in real time. It can automatically correct and optimize the robotic arm using a strategy mapping network model, without the need for manual adjustment of the statically set multi-axis robotic arm.

[0031] 2. The pose deviation of the multi-axis robotic arm is predicted in real time using a predictive model, the deviation value between the model and the actual running trajectory is calculated, and the trajectory is dynamically corrected to reduce fitting misalignment and improve the accuracy of flexible electronic device assembly.

[0032] 3. By mapping network model parameters through version management strategies and implementing a dynamic rollback mechanism for parameters that do not meet the requirements, the optimal model can be quickly verified and deployed, reducing the risk of production stoppage due to model failure.

[0033] Legend

[0034] Figure 1 This is a flowchart of a multi-axis robotic arm control method based on image processing. Detailed Implementation

[0035] To make the technical means, creative features, objectives, and effects of this invention easier to understand, the invention is further described below with reference to specific embodiments. However, the following embodiments are merely preferred embodiments of this invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments described herein without creative effort are all within the protection scope of this invention.

[0036] Example 1, such as Figure 1This is a multi-axis robotic arm control method based on image processing. It includes acquiring multi-dimensional images of a flexible device, extracting and analyzing device feature parameters, constructing an image analysis model based on these parameters, predicting the motion trajectory of the flexible device, dynamically correcting the trajectory through a strategy, and feeding back the results to iterate the strategy mapping network model. The specific implementation steps are as follows: The specific steps for acquiring multi-dimensional images of a flexible device in this image processing-based multi-axis robotic arm control method are as follows: Image acquisition devices are installed in the working area of ​​the multi-axis robotic arm; a checkerboard calibration board is placed at different positions; the captured checkerboard images are used to generate corresponding world coordinates; a calibration algorithm is called to solve for internal and external parameters; multiple acquisition devices are calibrated to achieve spatial uniformity; and a Precision Time Protocol (RTP) is used. The Protocol for Persistence (PTP) establishes a master clock and slave clock, sends data packets to obtain round-trip time and its own timestamp, and unifies the timestamps of the acquisition devices. For the located flexible device, the acquisition devices capture static images from top and side views, recording data such as the device's outline and texture to generate static data. The acquisition devices simultaneously capture RGB images and depth maps, generate a color depth map through matching and calibration, and fuse it with the disparity map obtained by the algorithm from the RGB images to obtain a 3D model. The RGB images and depth maps of the flexible device are continuously acquired, sorted by timestamp to form an image stream, forming dynamic data, and integrating static data. Data, 3D models, and dynamic data constitute the multidimensional data of flexible devices. The specific steps for extracting and analyzing device feature parameters are as follows: denoising the RGB images in the static data using a Gaussian filter; performing median filtering on the corresponding depth maps; cropping the RGB images to the same size as the predefined acquisition area; using the Canny edge detection algorithm to extract the edge pixel set; calculating the edge pixel contour area; selecting the largest contour area from the edge pixel set and setting it as the main contour; fitting an outer matrix to record the center point and rotation angle; selecting the main contour of the depth map pixel set; and combining the main contours corresponding to the RGB image set and the depth map set {(x i ,y i ,d i )}, where x, y represent image coordinates, d represents depth map depth, and i represents RGB image i; using least squares plane fitting, the equation of the reference plane is established as follows:

[0037] z = ax + by + c

[0038] Where a, b, and c are the first, second, and third undetermined coefficients, respectively; the warp of the image is calculated using the following formula:

[0039] warp i =d i -z(x i ,y i ),

[0040] Where d i warp represents the depth of RGB image i. i The warp of RGB image i is represented by z(x). i ,y i ) indicates that the RGB image i is on the reference plane (x i ,y i The value corresponding to the point;

[0041] Combining warp, the center point of the circumscribed matrix, and the rotation angle, a static feature set is formed. The average value of the coordinates in the 3D model is calculated to obtain the model centroid. The difference between the coordinates of each point in the model and the coordinates of the centroid is used to construct the covariance matrix and solve for the direction vector. Combining the direction vector and the centroid coordinates, a 3D feature set is formed. Combining the static feature set of continuous time periods and the corresponding 3D model centroid, the motion angular velocity and centroid velocity are calculated. The Farneback algorithm is used to calculate the pixel motion vector for two consecutive frames. Combining the motion angular velocity, centroid velocity, and pixel motion vectors, a dynamic feature set is formed.

[0042] Based on the above steps, the specific steps for constructing an image analysis model according to the parameters are as follows: Collect feature sets from multiple periods, concatenate them into feature vectors in static, 3D, and dynamic order, normalize each dimension within the feature vectors, fill in missing values ​​using the mean of previous and subsequent periods for feature terms, select a multilayer perceptron model, use the above feature vectors as input, and divide them into training data, validation data, and test data in an 8:2:2 ratio. Calculate the mean and standard deviation of each dimension in the training data, set the mean square error as the loss function, calculate the loss value using the validation data, and set the parameter with the smallest loss value as the multilayer perceptron model parameter. Calculate the mean error and root mean square error using the test data, update the multilayer perceptron model parameters, and export the parameters of the successfully trained multilayer perceptron model as ONNX (Open Neural Networks). NetworkExchange (an open neural network exchange format) is introduced into the architecture of a multi-axis robotic arm controller, using an inference engine. The ONNX format multilayer perceptron model parameters are loaded, and after the acquisition device obtains feature vectors, the inference engine within the controller architecture retrieves the inference results. The specific steps for predicting the motion trajectory of the flexible device are: recording the initial position of the current period t, synchronously acquiring the feature set of period t, and sequentially aggregating them into a feature vector p. t The feature vector is normalized and input into the multilayer perceptron model to obtain the prediction bias vector Δp. t The predicted vector is calculated from the feature vector using the following formula:

[0043] p′ t =p t +Δp t ,

[0044] Where p′ t p represents the prediction vector for period t. t Δp represents the eigenvector with period t. t The prediction deviation vector represents the period t;

[0045] Repeat the above steps to obtain prediction sequences for multiple periods. Extract a two-dimensional coordinate set from the prediction sequence, connect adjacent points of the two-dimensional coordinate set in chronological order to obtain a preliminary motion trajectory prediction curve, and smooth the prediction curve using a sliding window algorithm. Utilize the yaw angle θ corresponding to the two-dimensional coordinates in the prediction sequence... t Convert to a unit tangent vector:

[0046] V t =[cosθ t sinθ t ],

[0047] Where θ t V represents the yaw angle during period t. t Represents the unit tangent vector with period t;

[0048] For each inflection point on the predicted curve, the angle error between the curve's tangent and orientation is eliminated by correcting with a unit tangent vector, thus obtaining the predicted pose for multiple cycles.

[0049] Based on the above steps, the specific steps for dynamically correcting the trajectory using the strategy are as follows: The deviation between the predicted pose and the actual pose at period k is recorded as the prediction difference, which is divided into positional deviation and angular deviation. Based on the type and degree of the prediction difference, a correction mode δ is selected. k Each correction mode is pre-mapped with a corresponding label value, parameter value μ. k , will (δ k ,μ k As a correction decision for period k, the correction decisions for multiple periods are combined with the corresponding predicted poses to form a training set. A new policy mapping network model is built in the deep learning framework. This is a multi-task neural network architecture used to map the feature parameters extracted from multi-dimensional images to the execution policy of the robotic arm. The training set is used as the input parameter vector. After one forward propagation through a shared hidden layer, the numerical compensation amount and discrete policy decision are output. The linear transformation unit in the policy mapping network model linearly transforms the input parameter vector, and the compensation component is obtained for each item of the prediction difference. The output policy probability is activated by Softmax, and the formula is:

[0050]

[0051] Where C represents the strategy probability, j represents the current acquisition period, m represents the m-th period, n represents the total number of acquisition periods, p represents the feature vector, and z(δ)j ) represents the label value corresponding to the correction mode of period j, exp represents the exponential function, and p j p represents the eigenvector of period j. m This represents the eigenvector of the m-th period;

[0052] The compensation component of the prediction difference is combined with the output policy probability to form the loss function, as shown in the formula:

[0053]

[0054] Where α1 and α2 are the weights of each item, and α1 + α2 = 1, h is the total number of compensation components, s represents the index of the current compensation component, and Δp s This represents the s-th true feature vector. Let represent the s-th predicted feature vector, u represent the total number of correction modes, and g represent the index of the current correction mode.

[0055] The mean squared error and cross-entropy loss of various branch parameters are calculated, and the parameter gradients are jointly calculated with the loss function. The parameters are updated using mini-batch gradient descent until convergence, thus completing the optimization of the policy mapping network model. The predicted pose of the cycle is input into the optimized policy mapping network model to obtain the real-time corrected trajectory. The corrected trajectory is sent to the built-in controller of the multi-axis robotic arm, and the controller drives the robotic arm according to the corrected trajectory. The specific steps of iterating the policy mapping network model based on the feedback results are as follows: After each acquisition of the corrected trajectory, the acquisition device records the actual trajectory of the multi-axis robotic arm and calculates it with the corrected trajectory issued by the policy mapping network model. The deviation vector and the feature vector corresponding to the acquisition cycle are statistically analyzed as feedback data. A deviation threshold is set, and feedback data exceeding the deviation threshold is marked. The feedback data is standardized, and iterative training is performed on the feedback data using a small learning rate. The optimal policy mapping network model parameters after iteration are retained and replaced with the current inference version. After deploying the policy mapping network model parameters, the performance and feedback data are monitored in real time and compared with the historical model parameter execution. If the performance and feedback data are not as good as the previous version, the system is rolled back to the stable version and the rollback log is recorded.

[0056] Example 2, based on Example 1 above, describes the practical application of the image processing-based multi-axis robotic arm control method in a flexible electronic component assembly scenario, specifically as follows:

[0057] Step 1: Select an RGB camera and a depth camera for data acquisition. Use a 9×6 checkerboard calibration board for intrinsic and extrinsic parameter calibration to establish a unified world coordinate system. For static 2D data, first use the Canny algorithm to extract the pixel set, fit the outer matrix to obtain the matrix length and width, and combine the depth map with the fitted plane to calculate the warping. For 3D data, after calculating the center mass point, construct the covariance matrix, and decompose it to obtain mutually orthogonal unit eigenvectors. For dynamic data, record the data of 10 consecutive frames as a set, calculate the linear velocity and angular velocity between every two frames, calculate the mean to obtain the eigenvector, and concatenate the above three types of eigendata into a complete eigenvector.

[0058] Step 2: For multiple different acquisition cycles, feature sets are obtained from the image acquisition system. The outer matrix length, width, and warpage of the static feature set, the centroid coordinates of the 3D feature set, and the average linear velocity and average angular velocity of the dynamic feature set are normalized and concatenated into a feature vector as input. The predicted trajectory is obtained by the multilayer perceptron model, and the deviation vector is calculated between it and the actual motion trajectory. A loss function is set between the deviation vector and the feature vector, and the multilayer perceptron model is forward mapped. The mean squared error between the deviation vector and the true deviation vector is set as the loss value. The smaller the loss value, the more accurate the prediction. The multilayer perceptron model is updated, and the optimized predicted trajectory is output. The trajectories of multiple acquisition cycles are connected to form a trajectory sequence.

[0059] Step 3: By recording the trajectory sequence of the multilayer perceptron model generated in the control system, for each acquisition cycle of the trajectory sequence, each dimension is normalized and input into the pre-trained policy mapping network model. The input feature vector is passed through several fully connected layers to become the hidden representation. A linear mapping is performed on the hidden representation, and the output policy vector is concatenated, which includes numerical compensation and discrete policy decisions. The pose and parameters of the trajectory sequence are corrected and sent to the multi-axis robotic arm for execution. The execution result is compared with the sent-out situation, and the mean error and variance are calculated. The mean absolute error is set as the mean error threshold, and the residual variance is calculated and set as the variance threshold. For policies that exceed the threshold, they are recorded and returned to the policy mapping network model for calculation. Iterative training is performed on the feedback set with a small learning rate, and the parameters of the policy mapping network model are updated. For policies that do not exceed the threshold, the threshold determination is re-executed periodically on the collected feedback set.

[0060] The embodiments of the present invention described above are subject to modification and change of method by those skilled in the art without departing from the embodiments and broader aspects of the present invention. The appended claims are intended to include all such modifications and changes of method that do not depart from the present invention.

Claims

1. A multi-axis robotic arm control method based on image processing, characterized in that: Acquiring multi-dimensional images of flexible devices and extracting and analyzing device feature parameters involves unifying the acquisition equipment of a multi-axis robotic arm in space and time, integrating static data, 3D models, and dynamic data, calculating the warp of RGB images, combining the warp, the center point of the circumscribed matrix, and the rotation angle to form a static feature set, calculating pixel motion vectors, and combining the motion angular velocity, the centroid velocity, and the pixel motion vectors to form a dynamic feature set. Based on the parameters, an image analysis model is constructed to predict the motion trajectory of flexible devices. This involves collecting feature sets from multiple different periods, concatenating them into feature vectors in static, 3D, and dynamic order, introducing an inference engine into the architecture of the multi-axis robotic arm controller, normalizing the feature vectors, inputting them into a multilayer perceptron model, obtaining a prediction bias vector, calculating the prediction vector with the feature vectors, extracting a 2D coordinate set, converting the yaw angle corresponding to the 2D coordinates in the prediction sequence into a unit tangent vector, and correcting it with the unit tangent vector to obtain the predicted pose for multiple periods. The strategy dynamically corrects the trajectory, and the feedback results iterate the strategy mapping network model. This is done by calculating the deviation between the predicted pose and the actual executed pose, selecting a correction mode based on the degree and type of deviation, inputting it into the strategy generation model to obtain the strategy probability, and updating the parameters of the strategy generation model with the joint loss function of the actual feedback.

2. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps for acquiring multidimensional images of flexible devices are as follows: Image acquisition devices are installed in the working area of ​​a multi-axis robotic arm. A checkerboard calibration plate is placed in different positions, and the captured checkerboard images are used to generate corresponding world coordinates. A calibration algorithm is called to solve for internal and external parameters, and multiple acquisition devices are calibrated to achieve spatial uniformity. The acquisition devices are then unified with timestamps via the PTP protocol. For the positioned flexible device, static images are captured from top and side views using the acquisition devices. The device's outline and texture data are recorded to generate static data. The acquisition devices simultaneously capture RGB images and depth maps. A color depth map is generated through matching calibration and fused with the disparity map obtained from the RGB images using an algorithm to obtain a 3D model. RGB images and depth maps of the flexible device are continuously acquired and sorted by timestamp to form an image stream, creating dynamic data. The static data, 3D model, and dynamic data are integrated to form multidimensional data of the flexible device.

3. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps for extracting and analyzing the characteristic parameters of the device are as follows: In static data, RGB images are denoised using a Gaussian filter, and the corresponding depth maps are processed with median filtering. The RGB images are cropped to the same size as the predefined acquisition area. The Canny edge detection algorithm is used to extract the set of edge pixels, and the contour area of ​​the edge pixels is calculated. The main contour is selected from the edge pixel set based on the contour area. An outer matrix is ​​fitted to record the center point and rotation angle. The main contour of the depth map pixel set is selected, and the main contours corresponding to the RGB image set and the depth map set are combined. i ,y i ,d i )}, where x, y represent image coordinates, d represents depth map depth, and i represents RGB image i; using least squares plane fitting, the equation of the reference plane is established as follows: z = ax + by + c Where a, b, and c are the first, second, and third undetermined coefficients, respectively; the warp of the RGB image is calculated using the following formula: warp i =d i -z(x i ,y i ), Where d i warp represents the depth of RGB image i. i The warp of RGB image i is represented by z(x). i ,y i ) indicates that the RGB image i is on the reference plane (x i ,y i The value corresponding to the point; Combining warp, the center point of the circumscribed matrix, and the rotation angle forms a set of static features.

4. The multi-axis robotic arm control method based on image processing according to claim 3, characterized in that: The specific steps for extracting and analyzing device feature parameters also include: The average value of the coordinates in the 3D model is calculated to obtain the centroid of the model. The covariance matrix is ​​constructed by subtracting the coordinates of each point in the model from the coordinates of the centroid and solving for the direction vector. The direction vector and the centroid coordinates are combined to form a 3D feature set. The motion angular velocity and centroid velocity are calculated by combining the static feature set of continuous time periods and the corresponding 3D model centroid. The Farneback algorithm is used to calculate the pixel motion vector for two consecutive frames. The motion angular velocity, centroid velocity, and pixel motion vector are combined to form a dynamic feature set.

5. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps for constructing an image analysis model based on parameters are as follows: Normalization is performed on each dimension of the feature vector. For feature terms with missing values, the mean of the preceding and following periods is used for filling. A multilayer perceptron model is selected, and the above feature vector is used as input, divided into training data, validation data, and test data in an 8:2:2 ratio. The mean and standard deviation of each dimension in the training data are calculated, and the mean square error is set as the loss function. The loss value is calculated using the validation data, and the multilayer perceptron model parameters are set according to the loss value. The mean error and root mean square error are calculated using the test data, and the multilayer perceptron model parameters are updated. The parameters of the successfully trained multilayer perceptron model are exported in ONNX format. An inference engine is introduced into the architecture of the multi-axis robotic arm controller. The ONNX format multilayer perceptron model parameters are loaded, and after the acquisition device obtains the feature vector, the inference engine in the controller architecture obtains the inference results.

6. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps for predicting the motion trajectory of the flexible device are as follows: Record the initial position of the current period t, synchronously collect the feature set of period t, and aggregate them into a feature vector p in sequence. t The feature vector is normalized and input into the multilayer perceptron model to obtain the prediction bias vector Δp. t The predicted vector is calculated from the feature vector using the following formula: p' t =p t +Δp t , Where p ' t p represents the prediction vector for period t. t Δp represents the eigenvector with period t. t The prediction deviation vector represents the period t; Repeat the above steps to obtain prediction sequences for multiple periods. Extract a two-dimensional coordinate set from the prediction sequence, connect adjacent points of the two-dimensional coordinate set in chronological order to obtain a preliminary motion trajectory prediction curve, and smooth the prediction curve using a sliding window algorithm. Utilize the yaw angle θ corresponding to the two-dimensional coordinates in the prediction sequence... t Convert to a unit tangent vector: V t =[cosθ t ,sinθ t ], Where θ t V represents the yaw angle during period t. t Represents the unit tangent vector with period t; For each inflection point on the predicted curve, the angle error between the curve's tangent and orientation is eliminated by correcting with a unit tangent vector, thus obtaining the predicted pose for multiple cycles.

7. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps for dynamically correcting the trajectory through the strategy are as follows: the deviation between the predicted pose and the actual pose at period k is recorded as the prediction difference, which is divided into position deviation and angle deviation. Based on the type and degree of the prediction difference, the correction mode δ is selected. k Each correction mode is pre-mapped with a corresponding label value, parameter value μ. k , will (δ k ,μ k As a correction decision for period k, the correction decisions for multiple periods are combined with the corresponding predicted poses to form a training set.

8. The multi-axis robotic arm control method based on image processing according to claim 7, characterized in that: The specific steps for dynamically correcting the trajectory through the strategy also include: In a deep learning framework, a new policy mapping network is created. The training set is used as the input parameter vector. The linear transformation unit in the policy mapping network model linearly transforms the input parameter vector, obtaining a compensation component for each prediction difference. The output policy probability is then activated by Softmax, as shown in the formula: Where C represents the strategy probability, j represents the current acquisition period, m represents the m-th period, n represents the total number of acquisition periods, p represents the feature vector, and z(δ) j ) represents the label value corresponding to the correction mode of period j, exp represents the exponential function, and p j p represents the eigenvector of period j. m This represents the eigenvector of the m-th period.

9. The multi-axis robotic arm control method based on image processing according to claim 7, characterized in that: The specific steps for dynamically correcting the trajectory through the strategy also include: The compensation component of the prediction difference is combined with the output policy probability to form the loss function, as shown in the formula: Where α1 and α2 are the feature difference weight and the correction mode weight, respectively, and α1 + α2 = 1, h is the total number of compensation components, s represents the index of the current compensation component, and Δp s This represents the s-th true feature vector. Let represent the s-th predicted feature vector, u represent the total number of corrected modes, and g represent the index of the current corrected mode; The predicted pose of the cycle is input into the strategy mapping network to obtain the real-time corrected trajectory. The corrected trajectory is then sent to the built-in controller of the multi-axis robotic arm, which drives the robotic arm according to the corrected trajectory.

10. The multi-axis robotic arm control method based on image processing according to claim 1, characterized in that: The specific steps of the feedback result iterative strategy mapping network model are as follows: Each time a corrected trajectory is acquired, the acquisition device records the actual trajectory executed by the multi-axis robotic arm, calculates it with the corrected trajectory issued by the strategy mapping network model, summarizes the deviation vector with the feature vector corresponding to the acquisition cycle, and statistically analyzes it into feedback data. A deviation threshold is set for the feedback data, and feedback data that exceeds the deviation threshold is marked. The feedback data is standardized, and iterative training is performed on the feedback data using a small learning rate to replace the current inference version. After deploying the policy mapping network model parameters, the performance and feedback data status are monitored in real time.

Images