Picking robot control method based on improved yolov5 algorithm and picking robot

Abstract

The application provides a picking robot based on an improved yolov5 algorithm and a picking robot and method, which comprises the following steps: building an orchard simulation environment, generating a target path of the picking robot, detecting error values of the path in real time through a sensor, calculating a proportional parameter P according to the error values, and adjusting the motion state of the robot through a control signal of the proportional parameter P; adopting a video stream processing framework to load a yolov5 algorithm model to construct a video processing channel, configuring an inference engine for the video processing channel to execute an inference acceleration task; collecting a target video stream as an input of the yolov5 algorithm model, respectively adopting transpose convolution, a global context block and C3GHOST convolution to optimize the yolov5 algorithm model, outputting pixel coordinates of a picking target in the target video stream, and obtaining relative coordinates according to coordinate conversion; calculating static motion parameters of a mechanical arm through inverse kinematics, solving dynamic motion parameters of the mechanical arm through polynomial interpolation, and controlling the picking robot to execute a picking task based on the motion parameters and the relative coordinates.

Description

Technical Field

[0001] This invention relates to the field of harvesting robot technology, specifically to a harvesting robot control method and a harvesting robot based on an improved YOLOv5 algorithm. Background Technology

[0002] With the development of the times, the degree of automation in all walks of life is constantly increasing. However, the existing agriculture has a low level of mechanization and intelligence in some areas. If harvesting is done manually, most of the fruit still needs to be harvested after it is ripe. The manual labor intensity is high and the harvesting efficiency is low, which cannot meet the growing demand.

[0003] Therefore, automated picking robots have been proposed. For example, patent number 202210311178.2 discloses a strawberry picking method and robot system based on binocular vision. It identifies strawberry fruits in preprocessed images according to a preset recognition model. Based on the strawberry fruit recognition results, the picking point position is determined by the lowest point of the strawberry and the centroid of the strawberry, thereby improving the recognition accuracy and solving the problem of positioning and distance measurement. Patent number 202210936842.2 discloses a fully automatic fruit picking robot and picking method based on vision technology. It uses a combination of binocular camera and inertial navigation module to construct an orchard map. The algorithm determines the height of the lifting platform and the initial pose of the robotic arm for each picking, which can increase the picking success rate and picking range.

[0004] However, the aforementioned existing technologies still have the following drawbacks:

[0005] (1) Deep learning models are slow in reasoning when faced with high-resolution images, which cannot meet the real-time performance requirements of target detection. Directly improving detection performance will lead to increased resource consumption, which will be a significant burden for small tools such as harvesting robots.

[0006] (2) Target detection algorithms are easily affected by factors such as lighting and occlusion when facing complex scenes, resulting in insufficient accuracy and stability of target detection results;

[0007] (3) The combination of lifting platform and robotic arm end effector results in low degree of freedom and flexibility in the picking process, especially in the face of complex picking scenarios, and cannot meet the requirements of precise positioning and attitude control of the target picking position.

[0008] Therefore, there is an urgent need for a harvesting robot control method based on the improved YOLOv5 algorithm to solve the above problems. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a harvesting robot control method and a harvesting robot based on an improved YOLOv5 algorithm.

[0010] The first aspect of this invention discloses a control method for a harvesting robot based on an improved YOLOv5 algorithm, comprising:

[0011] An orchard simulation environment is built, a target picking task is generated, the target path of the robot is determined, the error value between the robot's movement path and the target path is detected in real time by sensors, the proportional parameter P is calculated based on the error value, and the robot's motion state is adjusted by feeding back the control signal of the proportional parameter P through a PID controller.

[0012] A video processing channel is constructed by loading the YOLOv5 algorithm model using a video stream processing framework, and an inference engine is configured to execute inference acceleration tasks on the video processing channel.

[0013] The target video stream is acquired as input to the YOLOv5 algorithm model. The YOLOv5 algorithm model is optimized by transposed convolution, global context block, and C3GHOST convolution, respectively. The target recognition result, pixel coordinates, and the ratio of measured pixel distance to actual distance in the target video stream are output. The relative coordinates of the pixel coordinates are obtained according to the coordinate transformation.

[0014] The static motion parameters of the robotic arm corresponding to the relative coordinates are calculated by inverse kinematics, and the dynamic motion parameters of the robotic arm are solved by fifth-order polynomial interpolation. Based on the static motion parameters, dynamic motion parameters, and relative coordinates, the picking robot is controlled to perform the target picking task.

[0015] In an optional embodiment, the process of building an orchard simulation environment, generating a target picking task, and determining the robot's target path includes:

[0016] Orchard map information is obtained by collecting early images of the orchard to simulate the orchard map environment. The starting point and ending point of the target picking task are planned according to the picking needs. One or more target paths are generated based on the starting point and ending point information.

[0017] In an optional embodiment, the step of detecting the error value between the robot's movement path and the target path in real time using sensors, calculating a proportional parameter P based on the error value, and adjusting the robot's motion state by feeding back the proportional parameter P through a PID controller includes:

[0018] The robot loads the target path and moves according to the starting point-to-end point map trajectory of the target picking task. By detecting and feeding back the actual movement path parameters in real time, it calculates the error value between the target path and the actual movement path in real time for each unit of time. Based on the error value and the target path in real time, it updates the value of the proportional parameter P to output the motion control parameters for the current time. The robot adjusts its current motion state through the motion control parameters until the proportional parameter P is adjusted to the optimal value. Based on the optimal value of the proportional parameter P and the error value at the current time, the robot continuously outputs motion control parameters to control its motion state until it reaches the destination.

[0019] In an optional embodiment, the step of using a video stream processing framework to load a YOLOv5 algorithm model to construct a video processing channel, and configuring an inference engine for the video processing channel to accelerate the object detection task, includes:

[0020] A video processing channel was built on the Jetson Nano embedded development board configured on the robot using the DeepStream video stream processing framework, and the TensorRT inference engine dependency was configured for the video processing channel. The yolov5 algorithm model was loaded into the DeepStream video stream processing framework.

[0021] The YOLOv5 algorithm model is trained on a cloud server, outputting a .pt weight file. The .pt weight file is then transferred to a Jetson Nano embedded development board using the WINSCP graphical client. The TensorRT inference engine's conversion program converts the .pt weight file to a .wts weight file to optimize the YOLOv5 algorithm model's parameters. The .wts weight file is then compiled into an .engine file and imported into the target path of the DeepStream video stream processing framework. This converts the YOLOv5 algorithm model into an inference engine configured in the video processing channel for one or more tasks, including data input, data preprocessing, inference acceleration, post-processing, and output. The output is the YOLOv5 algorithm model performing object detection and tracking on each frame of the input video stream. The GPU of the Jetson Nano embedded development board executes the inference acceleration task of the YOLOv5 algorithm model through the TensorRT inference engine.

[0022] In an optional embodiment, the transposed convolution optimizes the YOLOv5 algorithm model, including:

[0023] A transposed convolutional layer, `transposed_con`, is defined in the header structure of the YOLOv5 algorithm model using the `nn.ConvTranspose2d` class. The `torch.randn` function generates a random tensor to simulate the input image, which serves as the input for the upsampling operation of the YOLOv5 algorithm model. The input random tensor is transposed using the inverse convolution kernel of the `transposed_con` layer to generate an output tensor, which is then used for training to expand the feature map space dimension of each frame in the target video stream. The shape of the random tensor is represented as (batch_size, input_channels, input_height, input_width), where `batch_size` represents the batch size of the input, `input_channels` represents the number of input channels, `input_height` represents the height of the input, and `input_width` represents the width of the input.

[0024] In an optional embodiment, optimizing the YOLOv5 algorithm model using a global context block includes:

[0025] The Global Context Block defines a 1x1 convolutional layer, a softmax layer, and a feature transformation module in sequence. The feature transformation module includes multiple convolutional layers, normalization layers, and activation functions.

[0026] The input size of the input feature map in each frame of the target video stream is extracted. The input feature map is converted into a single-channel feature map through the 1x1 convolutional layer to obtain the global context information of the input feature map. The input size includes the batch size, number of channels, height h, and width w of the input feature map. The single-channel feature map is flattened and its shape is represented as (batch, 1, h, w), where 1 represents the number of channels, batch represents the batch size, h represents the height, and w represents the width.

[0027] The single-channel feature map is normalized by the softmax layer to calculate the weight distribution. The single-channel feature map, which is used to represent global context information after flattening, is fused with the weight distribution through channel multiplication to calculate the global context vector. The shape of the global context vector is (batch,c,1,1).

[0028] The feature transformation module applies a linear transformation and nonlinear activation to the global context vector to generate a feature map with the same shape as the input feature map. The input feature map is then added point by point to the feature map after the global context vector transformation to obtain the feature representation of the input feature map enhanced by the global context block.

[0029] In an optional embodiment, the C3GHOST convolution optimizes the YOLOv5 algorithm model, including:

[0030] A lightweight model is constructed by applying Ghost convolution in the backbone feature extraction network of the YOLOv5 algorithm model using GhostBottleneck. The GhostBottleneck class defines a residual block that includes GhostConv class, DWConv class and direct path residual connection.

[0031] The GhostConv-based convolutional structure uses a 1x1 convolutional kernel to convert the number of input channels to half the number of hidden channels. After feature extraction using a 5x5 convolutional kernel, the feature outputs from the two convolutional kernel stages are concatenated along the channel dimension to add a Ghost Channel to each input channel, which is a linear combination of the original input channels, to represent the feature information of the original input channel.

[0032] The DWConv class performs downsampling based on stride parameters, and depthwise separable convolution is used to perform model computation and feature extraction simultaneously.

[0033] The information flow and feature reuse within the model are constructed through direct path residual connections using the GhostConv and DWConv classes.

[0034] In an optional embodiment, obtaining the relative coordinates of the pixel coordinates according to the coordinate transformation includes:

[0035] Based on the pixel coordinates (x, y, w, h) determined by the recognition results, the coordinates of the image midpoint (cx, cy) and the target midpoint (zx, zy) corresponding to the recognition results are calculated respectively. The formulas for calculating the image midpoint coordinates and the target midpoint coordinates are as follows:

[0036]

[0037] The coordinates of the midpoint in the image are set as the origin. The target midpoint coordinates (zx, zy) are transferred relative to the origin using the ratio (bx, by) between the measured pixel distance and the actual distance to generate target coordinates (nx, ny). Based on the ratio (bx, by), the target coordinates (nx, ny) are converted into relative coordinates (rx, ry) measured in centimeters. The formulas for calculating the target coordinates and relative coordinates are as follows:

[0038]

[0039] The relative coordinates help the robot locate and navigate the target to be picked.

[0040] In an optional embodiment, the step of calculating the static motion parameters of the robotic arm corresponding to the relative coordinates through inverse kinematics includes:

[0041] Construct the mapping relationship between the joint angles of the three-degree-of-freedom robotic arm and the base of the robotic arm, and collect the bottom height H1, upper arm length L1, lower arm length L2 and arm-direction compensation B of the robotic arm;

[0042] The centimeter coordinates (X, Y) between the target and the robotic arm base are calculated based on the relative coordinates (rx, ry). The static motion parameters of the robotic arm are then calculated based on the robotic arm parameters and the centimeter coordinates (x1, y1). The formula for calculating the static motion parameters is as follows:

[0043]

[0044] The static motion parameters include vertical projection T, base angle J0, upper arm angle J1, forearm angle J2, short side D, and hypotenuse X0.

[0045] In an optional embodiment, the method of solving the dynamic motion parameters of the robotic arm using fifth-order polynomial interpolation includes:

[0046] Define the starting angle INIT, target angle TARGET, initial angular velocity W0, target angular velocity W1, initial angular acceleration AC0, target angular acceleration AC1, total motion time TF, time step TS, and motion time point T for the dynamic motion of the robotic arm servo motor. Calculate the servo motor's initial angle A0, initial angular velocity A1, half of the initial angular velocity A2, acceleration A3, square of acceleration A4, and cube of acceleration A5 using these parameters. Based on the robotic arm servo motor's angles Theta, WC0, and ACC0, calculate the servo motor's angle, angular velocity, and angular acceleration positions at different time points to provide dynamic motion parameters for smooth servo motor motion. The calculation formula is as follows:

[0047] A0 = INIT

[0048] A1 = W0

[0049]

[0050] A3=(20*TARGET-20*INIT-(8*W1+12*W0)*TF-(3*AC0-AC1)*TF 2 / (2*TF 3 )

[0051] A4=(30*TARGET-30*INIT+(14*W1+16*W0)*TF-(3*AC0-2*AC1)*TF 2 / (2*TF 4)

[0052] A5=(12*TARGET-12*INIT-(6*W1+6*W0)*TF-(AC0-AC1)*TF 2 / (2*TF 5 )

[0053] Theta = A0 + A1*T + A2*T 2 +A3*T 3 +A4*T 4 +A5*T 5

[0054] WC0=A1+2*A2*T+3*A3*T 2 +4*A4*T 3 +5*A5*T 4

[0055] ACC0 = 2*A2 + 6*A3*T + 12*A4*T 2 +20*A5*T 3

[0056] A second aspect of the present invention discloses a harvesting robot for performing the steps of a harvesting robot control method based on an improved YOLOv5 algorithm as disclosed in the first aspect of the present invention, including:

[0057] A mobile platform component, a robotic arm component mounted on the mobile platform, and a control system mounted within the mobile platform;

[0058] The mobile platform assembly includes a chassis consisting of a top plate, a bottom plate, and a first support column, a Mecanum wheel located at the bottom of the bottom plate, and a geared motor that drives the Mecanum wheel;

[0059] The robotic arm assembly includes a tray, an arm, a wrist, a gripper, a lead screw that drives the extension and retraction of the arm, a servo motor that drives the rotation of the wrist, a drive motor that drives the gripper to operate, and a camera mounted at the rear end of the gripper for acquiring target video streams.

[0060] The control system includes a main control module, a wireless control module, a sensor module, a motor drive module, and a power supply module. The main control module includes an STM32RCT6 microcontroller for controlling the robot's motion trajectory and wireless control, a JETSON NANO embedded development board for controlling target video stream image processing, and a PCA9685 module for controlling the robotic arm based on the image processing results from the JETSON NANO embedded development board. The PCA9685 module uses an I2C interface to control the angle of the robotic arm's servo motors. The JETSON NANO embedded development board accelerates the inference of the YOLOv5 algorithm model through the DeepStream video stream processing framework and the TensorRT inference engine. The STM32RCT6 microcontroller and the JETSON NANO embedded development board are interconnected through TX and RX pins to enable bidirectional information transmission between the main control module and the robot.

[0061] The wireless control module is configured as an HC-05 Bluetooth module, which communicates with mobile devices via Bluetooth to remotely control the harvesting robot.

[0062] The sensor module is configured as an MPU6050 module integrating a three-axis gyroscope and a three-axis accelerometer, for measuring the rotation angle and rotational acceleration of the robotic arm;

[0063] The motor drive module is configured as a TB6612 dual H-bridge driver to control one or more motors mounted on the harvesting robot for motor drive.

[0064] The power supply module includes a 12V power supply for the STM32RCT6 microcontroller via a DC socket, a 5V 4A DC power supply configured on the JETSON NANO embedded development board, and an LM2596 voltage regulator module that provides a stable 5V power supply for the HC-05 Bluetooth module, MPU6050 module, and PID tracking module. The JETSON NANO embedded development board and the PCA9685 module are connected via a step-down module that outputs 5V to the PCA9685 module to execute the robotic arm servo control.

[0065] In an optional embodiment, the control system further includes a display module configured to visualize the robot's status information on an OLED display screen and provide click buttons for adjusting the tracking parameters of the PID tracking module. The status information includes the robot's elevation angle, depression angle, yaw angle, type of target to be picked, and number of targets to be picked.

[0066] A third aspect of this invention discloses a harvesting robot control device based on an improved YOLOv5 algorithm, comprising:

[0067] At least one processor, and,

[0068] A memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform a harvesting robot control method based on the improved YOLOv5 algorithm as disclosed in any of the first aspects of the present invention.

[0070] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a harvesting robot control method based on the improved YOLOv5 algorithm as disclosed in any of the first aspects of the present invention.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] (1) This invention employs DeepStream, a high-performance, low-latency real-time video stream processing framework, combined with the TensorRT inference engine (a deep learning inference acceleration library) to perform efficient inference on models such as YOLOv5. TensorRT utilizes GPUs to accelerate the inference process, improving the inference speed of the model and meeting the real-time performance requirements of object detection, while reducing the resource consumption of object detection, making it particularly suitable for small devices such as harvesting robots.

[0073] (2) This invention enhances the model’s perception ability and classification accuracy by introducing global context information through global context blocks, improves the accuracy and stability of target detection by using transposed convolution in the model head structure, and reduces the number of model parameters and computation by introducing Ghost convolution in the Neck part of the model, thereby improving the feature extraction ability and optimizing the YOLOv5 algorithm model. This improves the model’s perception ability, accuracy and computational efficiency, and reduces the impact of interference factors in complex scenes on the target detection results.

[0074] (3) This invention optimizes the motion control of a three-axis robotic arm by combining inverse kinematics and polynomial interpolation. The inverse kinematics algorithm calculates the final joint angle based on the kinematic characteristics of the robotic arm and the target position, which helps the three-axis robotic arm to accurately locate the target position and control its posture, thereby enabling the execution of complex picking tasks. Polynomial interpolation uses the continuity of the local interpolation polynomial and the continuity of the first derivative to achieve the fitting of a smooth curve, which can effectively approximate the target motion trajectory in the servo control system and obtain a continuous and smooth servo trajectory, making the servo motion more stable. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a flowchart of the harvesting robot control method based on the improved YOLOv5 algorithm of the present invention;

[0077] Figure 2 This is a schematic diagram of the harvesting robot in this invention.

[0078] Figure label:

[0079] 1-Top plate, 2-Bottom plate, 3-Support column, 4-Mecanum wheel, 5-Gear motor, 6-Panel, 7-Arm, 8-Wrist, 9-Gripper, 10-Lead screw, 11-Servo motor, 12-Drive motor, 13-Camera. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0081] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0082] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0083] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0084] Example 1

[0085] See Figure 1 This invention discloses a control method for a harvesting robot based on an improved YOLOv5 algorithm. The first aspect of this invention discloses a control method for a harvesting robot based on an improved YOLOv5 algorithm, comprising:

[0086] An orchard simulation environment is built, a target picking task is generated, the target path of the robot is determined, the error value between the robot's movement path and the target path is detected in real time by sensors, the proportional parameter P is calculated based on the error value, and the robot's motion state is adjusted by feeding back the control signal of the proportional parameter P through a PID controller.

[0087] A video processing channel is constructed by loading the YOLOv5 algorithm model using a video stream processing framework, and an inference engine is configured to execute inference acceleration tasks on the video processing channel.

[0088] The target video stream is acquired as input to the YOLOv5 algorithm model. The YOLOv5 algorithm model is optimized by transposed convolution, global context block, and C3GHOST convolution, respectively. The target recognition result, pixel coordinates, and the ratio of measured pixel distance to actual distance in the target video stream are output. The relative coordinates of the pixel coordinates are obtained according to the coordinate transformation.

[0089] The static motion parameters of the robotic arm corresponding to the relative coordinates are calculated by inverse kinematics, and the dynamic motion parameters of the robotic arm are solved by fifth-order polynomial interpolation. Based on the static motion parameters, dynamic motion parameters, and relative coordinates, the picking robot is controlled to perform the target picking task.

[0090] Specifically, the design concept of this invention's harvesting robot control method based on the improved YOLOv5 algorithm involves using a four-wheeled robot, constructing a simple simulated orchard, and setting an orchard route. To achieve efficient motion control, this invention selects the powerful STM32RCT6 as the main controller, which can perform precise motion control and path planning for the robot. For image processing, this invention incorporates a JETSON NANO and is equipped with a high-definition camera module. The JETSON NANO communicates with the STM32RCT6 via a serial port, enabling data exchange and collaboration between the two. The PCA9685 module uses an I2C interface to control the angles of the four servos. The JETSON NANO uses DeepStream and TensorRT to accelerate YOLOv5 inference, thereby enabling real-time detection of the fruit by the camera. Simultaneously, the system uses the detected fruit coordinates and performs inverse calculations to determine the required rotation angle of the servos, harvesting the target fruit to the designated area. The system achieves fully automated harvesting.

[0091] In an optional embodiment, the process of building an orchard simulation environment, generating a target picking task, and determining the robot's target path includes:

[0092] Orchard map information is obtained by collecting early images of the orchard to simulate the orchard map environment. The starting point and ending point of the target picking task are planned according to the picking needs. One or more target paths are generated based on the starting point and ending point information.

[0093] In an optional embodiment, the step of detecting the error value between the robot's movement path and the target path in real time using sensors, calculating a proportional parameter P based on the error value, and adjusting the robot's motion state by feeding back the proportional parameter P through a PID controller includes:

[0094] The robot loads the target path and moves according to the starting point-to-end point map trajectory of the target picking task. By detecting and feeding back the actual movement path parameters in real time, it calculates the error value between the target path and the actual movement path in real time for each unit of time. Based on the error value and the target path in real time, it updates the value of the proportional parameter P to output the motion control parameters for the current time. The robot adjusts its current motion state through the motion control parameters until the proportional parameter P is adjusted to the optimal value. Based on the optimal value of the proportional parameter P and the error value at the current time, the robot continuously outputs motion control parameters to control its motion state until it reaches the destination.

[0095] Specifically, when using the PID algorithm for robot motion control, it is based on three main control parameters: proportional, integral, and derivative. In this robot motion control, the PID algorithm is combined with the MPU6050 to control the robot. The data from the MPU6050 can be used in the feedback loop to monitor the robot's attitude and motion state in real time. This real-time feedback helps the robot respond more accurately to external changes and make corresponding adjustments. The attitude and motion data provided by the MPU6050 sensors can monitor the robot's roll and pitch angles in real time. This data can serve as feedback signals for PID control, used to calculate errors, and to achieve precise motion control by adjusting the control parameters.

[0096] Furthermore, in robot motion control, adjusting only the P value can achieve excellent control results. The proportional parameter (P) determines the response speed and the degree of error adjustment. By appropriately adjusting the P value, the robot can respond quickly to errors and accurately control its posture and motion state. In this robot motion control, by reading data from the MPU6050, the error between the target angle and the actual angle is calculated, and deviations occurring during robot motion are corrected.

[0097] In an optional embodiment, the step of using a video stream processing framework to load a YOLOv5 algorithm model to construct a video processing channel, and configuring an inference engine for the video processing channel to accelerate the object detection task, includes:

[0098] A video processing channel was built on the Jetson Nano embedded development board configured on the robot using the DeepStream video stream processing framework, and the TensorRT inference engine dependency was configured for the video processing channel. The yolov5 algorithm model was loaded into the DeepStream video stream processing framework.

[0099] The YOLOv5 algorithm model is trained on a cloud server, outputting a .pt weight file. The .pt weight file is then transferred to a Jetson Nano embedded development board using the WINSCP graphical client. The TensorRT inference engine's conversion program converts the .pt weight file to a .wts weight file to optimize the YOLOv5 algorithm model's parameters. The .wts weight file is then compiled into an .engine file and imported into the target path of the DeepStream video stream processing framework. This converts the YOLOv5 algorithm model into an inference engine configured in the video processing channel for one or more tasks, including data input, data preprocessing, inference acceleration, post-processing, and output. The output is the YOLOv5 algorithm model performing object detection and tracking on each frame of the input video stream. The GPU of the Jetson Nano embedded development board executes the inference acceleration task of the YOLOv5 algorithm model through the TensorRT inference engine.

[0100] Specifically, a video analytics application is built using the DeepStream framework. Secondly, the YOLOv5 model is loaded into DeepStream and optimized and accelerated using TensorRT. Finally, the optimized YOLOv5 model is applied to the video stream through the DeepStream pipeline, achieving efficient real-time object detection and tracking. When using DeepStream and TensorRT to accelerate YOLOv5 inference, the DeepStream framework must first be installed and configured, ensuring that dependencies such as TensorRT and CUDA are correctly installed. Then, DeepStream plugins and modules are used to build a video processing pipeline, including steps such as data source input, preprocessing, inference acceleration, post-processing, and result output. After loading the YOLOv5 model, it is optimized using TensorRT. TensorRT can automatically optimize the model structure and use GPUs to accelerate the inference process, thereby improving inference speed and efficiency. Through TensorRT optimization, the YOLOv5 model can be transformed into a high-efficiency inference engine, reducing computational resource consumption and achieving faster inference speed. Once optimization is complete, the optimized YOLOv5 model is integrated into the DeepStream pipeline. In the pipeline, the model receives a video stream as input and performs object detection and tracking on each frame. Thanks to TensorRT acceleration, the inference process is more efficient and meets real-time performance requirements. Finally, DeepStream can display, store, or transmit the processed results through different output modules. For example, detected targets can be plotted on the video stream and displayed on the screen in real time, or the results can be saved to a file for subsequent analysis and processing. The combination of DeepStream and TensorRT enables efficient and fast object detection and tracking applications suitable for various video analysis scenarios, such as fruit picking and real-time object detection tasks.

[0101] Furthermore, deep learning is used on the Jetson Nano to determine the type of fruit being inspected. The Jetson Nano is trained on fruit images using the YOLOv5 algorithm. The trained model is then used by the Jetson Nano to perform artificial intelligence detection using the corresponding algorithm. The Jetson Nano communicates with the STM32 microcontroller via serial port, sending relevant data to the STM32. The STM32 microcontroller then performs the next step based on the acquired data.

[0102] In an optional embodiment, the transposed convolution optimizes the YOLOv5 algorithm model, including:

[0103] A transposed convolutional layer, `transposed_con`, is defined in the header structure of the YOLOv5 algorithm model using the `nn.ConvTranspose2d` class. The `torch.randn` function generates a random tensor to simulate the input image, which serves as the input for the upsampling operation of the YOLOv5 algorithm model. The input random tensor is transposed using the inverse convolution kernel of the `transposed_con` layer to generate an output tensor, which is then used for training to expand the feature map space dimension of each frame in the target video stream. The shape of the random tensor is represented as (batch_size, input_channels, input_height, input_width), where `batch_size` represents the batch size of the input, `input_channels` represents the number of input channels, `input_height` represents the height of the input, and `input_width` represents the width of the input.

[0104] Specifically, by using transposed convolution in the YOLOv5 header structure, the size of the feature map can be enlarged, thereby improving the accuracy and stability of object detection. Transposed convolution can better preserve feature information and learn richer and more complex feature representations, thereby improving the model's ability to recognize objects. In addition, transposed convolution can effectively reduce the number of model parameters and computational cost, improve the model's running efficiency and speed, especially in deployment scenarios on mobile and embedded devices, where it has better applicability and performance. The advantages of transposed convolution include: (1) Transposed convolution can enlarge low-resolution feature maps to the size of the original input image, thereby increasing the resolution and receptive field of the feature map. (2) Similar to ordinary convolution operations, transposed convolution also has the characteristic of parameter sharing, which can share weights during the learning process, thereby reducing the number of model parameters and computational cost. (3) Transposed convolution can be flexibly applied to different structures of neural networks, including autoencoders, generative adversarial networks (GANs) and convolutional neural networks (CNNs), which can be combined and adjusted according to actual needs.

[0105] Furthermore, in fruit-picking robots, transposed convolution can upsample low-resolution feature maps to high resolution, enriching the details of the feature maps. This is particularly important for fruit detection, as higher resolution captures the detailed features of the fruit, thereby improving detection accuracy. Simultaneously, high-resolution feature maps help to finely locate the edges and contours of the fruit, which is crucial for accurate fruit positioning and grasping. Transposed convolution can be used in the decoding part of the detection network to help recover the spatial details of the input image.

[0106] In an optional embodiment, optimizing the YOLOv5 algorithm model using a global context block includes:

[0107] The Global Context Block defines a 1x1 convolutional layer, a softmax layer, and a feature transformation module in sequence. The feature transformation module includes multiple convolutional layers, normalization layers, and activation functions.

[0108] The input size of the input feature map in each frame of the target video stream is extracted. The input feature map is converted into a single-channel feature map through the 1x1 convolutional layer to obtain the global context information of the input feature map. The input size includes the batch size, number of channels, height h, and width w of the input feature map. The single-channel feature map is flattened and its shape is represented as (batch, 1, h, w), where 1 represents the number of channels, batch represents the batch size, h represents the height, and w represents the width.

[0109] The single-channel feature map is normalized by the softmax layer to calculate the weight distribution. The single-channel feature map, which is used to represent global context information after flattening, is fused with the weight distribution through channel multiplication to calculate the global context vector. The shape of the global context vector is (batch,c,1,1).

[0110] The feature transformation module applies a linear transformation and nonlinear activation to the global context vector to generate a feature map with the same shape as the input feature map. The input feature map is then added point by point to the feature map after the global context vector transformation to obtain the feature representation of the input feature map enhanced by the global context block.

[0111] Specifically, the Global Context Block (GLC) is a structure used to enhance the perception and representation capabilities of Convolutional Neural Networks (CNNs). It improves the network's understanding and analysis of input data by introducing global contextual information. This module effectively encodes global contextual information into feature maps through global average pooling and channel-dimensional attention mechanisms. By fusing and rescaling with the original feature maps, the global contextual information is integrated into the network's intermediate layers, making the network more focused on global semantics and contextual relationships, thus enhancing its perception and representation capabilities. Using the GLC to optimize the YOLOv5 algorithm model offers the following advantages:

[0112] (1) The Global Context Block can enhance the perceptual ability of the YOLOv5 algorithm model. The Global Context Block enhances the perceptual ability of the convolutional neural network by introducing global contextual information. It can capture the global semantics and contextual relationships of the input data, making the network more capable of understanding and analyzing the input data. This helps improve the model's performance on complex tasks with contextual dependencies.

[0113] (2) The Global Context Block can improve the classification accuracy of the YOLOv5 algorithm model. By introducing global context information, the Global Context Block can make the network more focused on global semantics and important features, thereby improving classification accuracy. It can help the network better distinguish the feature differences between different categories, thereby reducing misclassification.

[0114] (3) The global context block can improve the localization accuracy of the YOLOv5 algorithm model, and the global context module is particularly beneficial for target localization tasks. It can enhance the network's understanding of the contextual information around the target, providing more accurate target localization and bounding box prediction. This helps reduce target localization errors and improve localization accuracy. It can enhance global semantics. The global context module incorporates global semantic information into the feature map through global average pooling and channel-dimensional attention mechanisms. This helps the network better understand the overall semantics and context of the input data, and is more advantageous in handling the overall scene and global dependencies.

[0115] Furthermore, the code implementation of the Global Context Block in the YOLOv5 algorithm model is as follows: The `GlobalContextBlock` class is defined to implement a generic Global Context Block (GCB) used to capture the global context information of the feature map. During initialization, based on the number of channels in the input feature map, the pooling method ('att' for attention pooling, 'avg' for average pooling), and the fusion method ('channel_add' for channel addition, 'channel_mul' for channel multiplication), relevant convolutional and pooling layers are defined, and the `initialize_weights` method is called to initialize the weights. The `initialize_weights` method is specifically used to initialize the weights of the attention pooling convolutional layers and the fusion convolutional layers. The `compute_context` method is responsible for calculating the global context information of the input feature map: for attention pooling, the attention mask is calculated through convolutional layers and softmax layers, and then weighted and summed to obtain the context vector; for average pooling, the context vector is calculated using an adaptive average pooling layer. During the forward propagation process, the compute_context method is first called to calculate the context vector. Then, the context vector is added and multiplied by channel_add_conv and channel_mul_conv respectively, and the results are applied to the input feature map to finally obtain the output feature map.

[0116] In an optional embodiment, the C3GHOST convolution optimizes the YOLOv5 algorithm model, including:

[0117] A lightweight model is constructed by applying Ghost convolution in the backbone feature extraction network of the YOLOv5 algorithm model using GhostBottleneck. The GhostBottleneck class defines a residual block that includes GhostConv class, DWConv class and direct path residual connection.

[0118] The GhostConv-based convolutional structure uses a 1x1 convolutional kernel to convert the number of input channels to half the number of hidden channels. After feature extraction using a 5x5 convolutional kernel, the feature outputs from the two convolutional kernel stages are concatenated along the channel dimension to add a Ghost Channel to each input channel, which is a linear combination of the original input channels, to represent the feature information of the original input channel.

[0119] The DWConv class performs downsampling based on stride parameters, and depthwise separable convolution is used to perform model computation and feature extraction simultaneously.

[0120] The information flow and feature reuse within the model are constructed through direct path residual connections using the GhostConv and DWConv classes.

[0121] Specifically, Ghost convolution is a lightweight convolution operation designed to reduce the number of parameters and computational cost of the model while maintaining good performance. It is achieved by introducing a small number of Ghost Channels on each channel, that is, adding a corresponding Ghost Channel on top of each input channel. The Ghost Channel is generated by a linear combination of the original channels and can approximate the feature information of the original channels with fewer parameters. The C3Ghost module applies Ghost convolution to the C3 module, which is one of the modules in YOLOv5 used to extract and fuse features at different scales. By introducing C3Ghost, the model's feature extraction ability can be enhanced while maintaining its lightweight nature, thus improving the model's performance and efficiency. The specific advantages of introducing Ghost convolution into the C3 module of YOLOv5 C3 are as follows: (1) Ghost convolution can share parameters on each channel, thereby reducing the number of parameters in the model and reducing the complexity and computational cost of the model. (2) Since Ghost convolution introduces Ghost Channels, the feature information of the original channels can be represented with fewer parameters, thus making the model more lightweight. Furthermore, by introducing Ghost convolutions into the C3 module, the model's ability to extract features at different scales can be enhanced, improving the model's perception and generalization capabilities. (3) Since Ghost convolutions reduce the number of model parameters and computational cost, they can accelerate the model's training and inference processes, improving the model's efficiency and speed. Experiments have shown that, because the Backbone is at a critical position for extracting data features, introducing C3Ghost into the Backbone will result in less feature extraction due to the less dense lightweight convolutional structure, leading to poor evaluation results. Therefore, it is necessary to introduce C3Ghost into the Neck part to optimize the YOLOv5 algorithm model.

[0122] Furthermore, the code implementation of C3GHOST convolution applied to the YOLOv5 algorithm model is as follows:

[0123] In the code, the GhostConv class implements a special convolutional structure that enhances feature extraction capabilities through a two-stage operation: first, a 1x1 convolutional kernel is used to convert the number of input channels to half the number of hidden channels; then, a 5x5 convolutional kernel is used for feature extraction. The outputs of the two stages are concatenated along the channel dimension to increase non-linear expressiveness. Next, the DWConv class implements depthwise separable convolution, determining whether to perform downsampling based on the stride parameter, thus effectively extracting features while maintaining computational efficiency. Further, the GhostBottleneck class defines a residual block structure, containing a combination of GhostConv and DWConv, as well as residual connections with direct paths, achieving efficient information flow and feature reuse. Finally, the C3Ghost class inherits from C3 and stacks multiple GhostBottleneck instances on top of it to construct a complete neural network structure.

[0124] Furthermore, the design of GhostConv and DWConv classes enables convolutional operations to effectively extract features of different scales and complexities in images while maintaining high computational efficiency. For fruit detection, this means the network can accurately capture features of fruits of various sizes and shapes, effectively detecting and recognizing both small fruits like strawberries and large fruits like oranges. The GhostBottleneck structure also allows for efficient information transfer and reuse within the network. This design helps improve the network's perceptual and representational capabilities, enabling the fruit detector to maintain stable detection performance under different background, lighting, and occlusion conditions. The combined effect of these advantages allows the network model to achieve high accuracy and stability in fruit detection tasks.

[0125] As an example, experiments were conducted to verify the YOLOv5 algorithm model optimized with transposed convolution, global context block, and C3GHOST convolution. The labeled dataset was used as the training and test sets in the experiment. The experimental results were evaluated using Precision and Recall. The same training and validation sets were used for training and testing of the transposed convolution, global context block, and C3GHOST convolution optimized YOLOv5 algorithm models.

[0126] Furthermore, the CSPDarknet53 network and Upsample upsampling layer of the original YOLOv5 algorithm model were used as control groups to compare and demonstrate the optimization results with Global Context Block, Transposed Convolution, and C3Ghost, as shown in Table 1 below:

[0127]

[0128] Table 1 shows that when using Global Context Block, Transposed Convolution, and C3Ghost, a significant increase in precision is observed, while recall decreases. Although this may reduce recall, in scenarios such as fruit picking, higher precision is more important than higher recall. The focus should be on accuracy rather than missed detections. Therefore, optimizations to the YOLOv5 algorithm model's backbone network and head structure greatly improve the application requirements for target detection in fruit picking.

[0129] Furthermore, regarding C3Ghost, the Jetson Nano on which the robot is equipped is a small computer designed for embedded devices, which has limitations in power consumption. The C3Ghost-optimized model requires less computation than the original model, reducing the device's power consumption and extending its lifespan. It also boasts higher computational efficiency, enabling higher real-time performance on the Jetson Nano. This is crucial for tasks requiring real-time video streaming or other real-time detection. While making the model lightweight, performance and accuracy are not sacrificed. On the contrary, the C3Ghost-optimized model maintains high accuracy while achieving higher speed and efficiency. This significantly improves the efficiency of deploying YOLOv5 models on devices like the Jetson Nano without compromising performance. Therefore, applying the C3Ghost-optimized weight file to the Jetson Nano fruit recognition system greatly simplifies model parameters and GFLops.

[0130] In an optional embodiment, obtaining the relative coordinates of the pixel coordinates according to the coordinate transformation includes:

[0131] Based on the pixel coordinates (x, y, w, h) determined by the recognition results, the coordinates of the image midpoint (cx, cy) and the target midpoint (zx, zy) corresponding to the recognition results are calculated respectively. The formulas for calculating the image midpoint coordinates and the target midpoint coordinates are as follows:

[0132]

[0133] The coordinates of the midpoint in the image are set as the origin. The target midpoint coordinates (zx, zy) are transferred relative to the origin using the ratio (bx, by) between the measured pixel distance and the actual distance to generate target coordinates (nx, ny). Based on the ratio (bx, by), the target coordinates (nx, ny) are converted into relative coordinates (rx, ry) measured in centimeters. The formulas for calculating the target coordinates and relative coordinates are as follows:

[0134]

[0135] The relative coordinates help the robot locate and navigate the target to be picked.

[0136] Specifically, the coordinate transformation process described above ensures that the captured fruit coordinates accurately represent its actual position in the image, in centimeters. In the fruit-picking robot, a USB high-definition camera serves as the vision module, and the JETSON NANO is responsible for processing and transforming the fruit's coordinate information in real time. By transmitting the fruit's actual coordinates to the program controlling the robotic arm, the robot can precisely locate and grasp the fruit, providing accurate positional information and thus improving operational efficiency and precision.

[0137] In an optional embodiment, the step of calculating the static motion parameters of the robotic arm corresponding to the relative coordinates through inverse kinematics includes:

[0138] Construct the mapping relationship between the joint angles of the three-degree-of-freedom robotic arm and the base of the robotic arm, and collect the bottom height H1, upper arm length L1, lower arm length L2 and arm-direction compensation B of the robotic arm;

[0139] The centimeter coordinates (X, Y) between the target and the robotic arm base are calculated based on the relative coordinates (rx, ry). The static motion parameters of the robotic arm are then calculated based on the robotic arm parameters and the centimeter coordinates (x1, y1). The formula for calculating the static motion parameters is as follows:

[0140]

[0141] The static motion parameters include vertical projection T, base angle J0, upper arm angle J1, forearm angle J2, short side D, and hypotenuse X0.

[0142] Furthermore, when the height of the object being picked is lower than the height of the base, the formula for calculating the forearm angle J2 is:

[0143] J2 = tan -1 (T / D)

[0144] When the height of the object being picked is equal to the height of the base, the forearm angle J2 is 0. When the height of the object being picked is greater than the height of the base, the formula for calculating the forearm angle J2 is:

[0145] J2 = tan -1 (D / T)

[0146] Furthermore, the inverse kinematics algorithm for a three-axis robotic arm is based on a mathematical model and geometric derivation. By analyzing the structure and kinematic characteristics of the robotic arm, a mapping relationship between joint angles and end effector coordinates is established. In this process, it is assumed that the robotic arm has three degrees of freedom, i.e., three joints. First, the structural parameters of the robotic arm are determined, including the base height, upper arm length, lower arm length, and arm-direction compensation. These parameters are obtained through measurement. Then, for a given target position and orientation of the end effector, we need to calculate the angle of each joint. Based on the established mathematical model, we can calculate the vertical projection using formulas: the vertical distance from the end effector to the robotic arm base is calculated based on the end effector coordinates; the base angle is calculated based on the end effector coordinates; the upper arm angle is calculated based on the coordinates of the end effector to the robotic arm base; the possible values ​​of the upper arm angle are calculated using the cosine theorem and inverse cosine function; and the possible values ​​of the lower arm angle are calculated using the cosine theorem and inverse cosine function. Finally, different calculation formulas are used to obtain the final joint angles depending on the kinematic characteristics of the robotic arm and the different target positions. This inverse kinematics algorithm can help a three-axis robotic arm achieve accurate positioning and attitude control of a target, thereby enabling the execution of complex tasks.

[0147] In an optional embodiment, the method of solving the dynamic motion parameters of the robotic arm using fifth-order polynomial interpolation includes:

[0148] Define the starting angle INIT, target angle TARGET, initial angular velocity W0, target angular velocity W1, initial angular acceleration AC0, target angular acceleration AC1, total motion time TF, time step TS, and motion time point T for the dynamic motion of the robotic arm servo motor. Calculate the servo motor's initial angle A0, initial angular velocity A1, half of the initial angular velocity A2, acceleration A3, square of acceleration A4, and cube of acceleration A5 using these parameters. Based on the robotic arm servo motor's angles Theta, WC0, and ACC0, calculate the servo motor's angle, angular velocity, and angular acceleration positions at different time points to provide dynamic motion parameters for smooth servo motor motion. The calculation formula is as follows:

[0149] A0 = INIT

[0150] A1 = W0

[0151]

[0152] A3=(20*TARGET-20*INIT-(8*W1+12*W0)*TF-(3*AC0-AC1)*TF 2 / (2*TF 3)

[0153] A4 (30*TARGET-30*INIT+(14*W1+16*W0)*TF-(3*AC0-2*AC1)*TF 2 / (2*TF 4 )

[0154] A5(12*TARGET-12*INIT-(6*W1+6*W0)*TF-(AC0-AC1)*TF 2 / (2*TF 5 )

[0155] Theta=A0+A1*T+A2*T 2 +A3*T 3 +A4*T 4 +A5*T 5

[0156] WC0=A1+2*A2*T+3*A3*T 2 +4*A4*T 3 +5*A5*T 4

[0157] ACC0=2*A2+6*A3*T+12*A4*T 2 +20*A5*T 3

[0158] Furthermore, to achieve smooth control of the servo motor of the robotic arm, a fifth-order polynomial was designed to optimize the motion control of the servo motor. Five-term spline interpolation is an interpolation method that approximates the data by inserting smooth spline curves between given data points. By utilizing the continuity of the local interpolation polynomial and the continuity of the first derivative, the smooth curve can be fitted, which can effectively approximate the target motion trajectory in the servo motor control system. By interpolating between known data points, a continuous and smooth servo motor trajectory can be obtained, making the servo motor movement more stable. Using the fifth-order polynomial trajectory to control the servo motor has the following advantages: (1) Smoothness: The fifth-order polynomial trajectory can achieve a smooth motion process. The servo motor will not experience sudden changes or jitters during the motion, thus ensuring the stability and comfort of the system. (2) Accuracy: By accurately controlling the coefficients of the polynomial, the position, speed, and acceleration of the servo motor can be accurately controlled to achieve the expected target position, thereby meeting the high requirements of the system for position control. (3) Controllability: The shape and motion parameters of the polynomial trajectory can be adjusted according to specific needs, including the starting angle, target angle, starting angular velocity, target angular velocity, and acceleration, so that the motion process of the servo can flexibly adapt to different application scenarios. (4) Real-time performance: The fifth-order polynomial trajectory calculation is simple and efficient, and can quickly generate control commands in real-time systems to achieve real-time response of the servo and meet the needs of real-time control. (5) Ease of implementation: The fifth-order polynomial trajectory control algorithm is relatively simple, does not require complex control logic and a large amount of calculation, is easy to implement and deploy, and is suitable for various embedded systems and control platforms.

[0159] This invention employs DeepStream, a high-performance, low-latency real-time video streaming framework, combined with the TensorRT inference engine (a deep learning inference acceleration library), to perform efficient inference on models such as YOLOv5. TensorRT utilizes GPUs to accelerate the inference process, improving the model's inference speed and meeting the real-time performance requirements of object detection, while reducing the resource consumption of object detection, making it particularly suitable for small devices such as harvesting robots.

[0160] This invention optimizes the YOLOv5 algorithm model by employing three methods: global context block, transposed convolution, and C3Ghost. This improves the model's perception capability, accuracy, and computational efficiency, while reducing the impact of interference factors in complex scenes on the target detection results.

[0161] This invention optimizes the motion control of a three-axis robotic arm by combining inverse kinematics and polynomial interpolation. The inverse kinematics algorithm calculates the final joint angles based on the kinematic characteristics of the robotic arm and the target position, helping the three-axis robotic arm to accurately locate the target position and control its posture, thereby enabling the execution of complex harvesting tasks. Polynomial interpolation utilizes the continuity of the local interpolation polynomial and the continuity of the first derivative to achieve smooth curve fitting, which can effectively approximate the target motion trajectory in the servo control system, resulting in a continuous and smooth servo trajectory, making the servo movement more stable.

[0162] like Figure 2 As shown, a second aspect of the present invention discloses a harvesting robot, comprising:

[0163] A mobile platform component, a robotic arm component mounted on the mobile platform, and a control system mounted within the mobile platform;

[0164] The mobile platform assembly includes a chassis consisting of a top plate 1, a bottom plate 2 and a first support column 3, a Mecanum wheel 4 disposed at the bottom of the bottom plate 2, and a reduction motor 5 that drives the Mecanum wheel 4;

[0165] The robotic arm assembly includes a tray 6, an arm 7, a wrist 8, a gripper 9, a lead screw 10 for driving the extension and retraction of the arm 7, a servo motor 11 for driving the rotation of the wrist 8, a drive motor 12 for driving the gripper 9 to operate, and a camera 13 mounted on the rear end of the gripper 9 for acquiring target video streams.

[0166] The control system includes a main control module, a wireless control module, a sensor module, a motor drive module, and a power supply module. The main control module includes an STM32RCT6 microcontroller for controlling the robot's motion trajectory and wireless control, a JETSON NANO embedded development board for controlling target video stream image processing, and a PCA9685 module for controlling the robotic arm based on the image processing results from the JETSON NANO embedded development board. The PCA9685 module uses an I2C interface to control the angle of the robotic arm's servo motors. The JETSON NANO embedded development board accelerates the inference of the YOLOv5 algorithm model through the DeepStream video stream processing framework and the TensorRT inference engine. The STM32RCT6 microcontroller and the JETSON NANO embedded development board are interconnected through TX and RX pins to enable bidirectional information transmission between the main control module and the robot.

[0167] Specifically, the robot's control scheme is a crucial component in the system's hardware and software design. The harvesting robot uses an STM32RCT6 and a Jetson Nano as its main controllers. The STM32RCT6 handles motion control for tasks such as line following and Bluetooth connectivity; its low power consumption and high performance make it an ideal choice for embedded systems. The Jetson Nano handles image processing and controls the PCA9685 for servo motor control; its GPU-accelerated computing capabilities provide strong support for the robot's perception and decision-making. To achieve effective communication between the two, a serial port connection is used. Specifically, the STM32RCT6's serial port TX pin is connected to the Jetson Nano's serial port RX pin, and vice versa, ensuring bidirectional information transmission and enabling overall system collaboration.

[0168] The wireless control module is configured as an HC-05 Bluetooth module, which communicates with mobile devices via Bluetooth to remotely control the harvesting robot.

[0169] Specifically, the HC-05 Bluetooth module is used to build a mobile app for remote control of the harvesting robot. The HC-05 has a simple serial communication interface, providing a standard AT command set, simplifying configuration and control. The HC-05 supports basic Bluetooth protocols and is compatible with most Bluetooth devices, enabling wireless communication with smartphones, tablets, and other devices. The HC-05 features long-distance transmission, with a transmission range exceeding 10 meters. Simultaneously, the HC-05 supports a low-power mode, entering sleep mode when idle to save power and extend battery life.

[0170] The sensor module is configured as an MPU6050 module integrating a three-axis gyroscope and a three-axis accelerometer, for measuring the rotation angle and rotational acceleration of the robotic arm;

[0171] Specifically, the MPU6050 module integrates a three-axis gyroscope and a three-axis accelerometer, providing measurements of an object's rotation and acceleration. It offers high measurement accuracy. Data from the gyroscope and accelerometer can be measured at 16-bit resolution, providing high precision and accuracy.

[0172] The motor drive module is configured as a TB6612 dual H-bridge driver to control one or more motors mounted on the harvesting robot for motor drive.

[0173] Specifically, in the motor drive control scheme, an H-bridge driver is used as the motor driver. Among the two motor drivers, the L298N and TB6612, the TB6612 was chosen. The reasons for selecting the TB6612 over the L298N are mainly twofold: package size and space layout considerations. First, the TB6612 uses a smaller package compared to the larger multi-pin package of the L298N, making it more suitable for integration and layout within limited space. The small package of the TB6612 makes it easier to install in limited space, reducing its footprint on the circuit board and leaving more space for other components. Second, the TB6612 is comparable to the L298N in performance and functionality, and even has some advantages. The TB6612 is a dual H-bridge driver capable of controlling two DC motors or stepper motors. It has high operating voltage and current capabilities, providing stable and reliable motor drive. Furthermore, the TB6612 also features safety functions such as overcurrent protection, overtemperature protection, and short-circuit protection, effectively protecting the motor and circuitry. Therefore, in terms of performance and functionality, the TB6612 can meet the needs of most motor drives.

[0174] The power supply module includes a 12V power supply for the STM32RCT6 microcontroller via a DC socket, a 5V 4A DC power supply configured on the JETSON NANO embedded development board, and an LM2596 voltage regulator module that provides a stable 5V power supply for the HC-05 Bluetooth module, MPU6050 module, and PID tracking module. The JETSON NANO embedded development board and the PCA9685 module are connected via a step-down module that outputs 5V to the PCA9685 module to execute the robotic arm servo control.

[0175] Furthermore, the PCB shape and size of the above modules are designed according to the size and structure of the robot. The layout of each module is designed according to its function and connection relationship, and their positions and directions are arranged reasonably to ensure short-distance connection of signal lines and stable power supply of power lines. A double-layer board design is adopted for the PCB board.

[0176] In an optional embodiment, the control system further includes a display module configured to visualize the robot's status information on an OLED display screen and provide click buttons for adjusting the tracking parameters of the PID tracking module. The status information includes the robot's elevation angle, depression angle, yaw angle, type of target to be picked, and number of targets to be picked.

[0177] Specifically, the working principle of the harvesting robot disclosed in the second aspect of this invention through the control method is as follows:

[0178] After the harvesting robot is powered on and initialized, the Bluetooth module waits to receive serial signals from mobile devices (such as a mobile app). Once a signal is received, the robot initiates the PID line-following algorithm to stay within the lane. When the robot encounters a left-side intersection, it performs a left-side translation until it reaches the destination and stops. At this point, the STM32RCT6 sends a serial signal to the JETSON NANO. Upon receiving this signal, the JETSON NANO begins reading the image captured by the camera and performing fruit recognition. The fruit's coordinate information is stored in a TXT file along the predetermined path. The JETSON NANO performs coordinate transformation and inverse kinematics calculations on the read coordinate information to determine the required rotation angles for each axis of the robotic arm. Simultaneously, the JETSON NANO controls the PCA9685 module to begin the grasping operation, moving the robotic arm to the appropriate position to grasp the fruit. After grasping, the robotic arm places the fruit in the designated location. The task ends after the entire program is completed. This solution combines the driving control of the harvesting robot with the image processing and robotic arm control of the JETSON NANO, achieving automated fruit-grabbing tasks. Through serial communication and coordinated control, the harvesting robot and the JETSON NANO achieve information exchange and task collaboration, improving the system's intelligence and autonomy.

[0179] A third aspect of this invention discloses a harvesting robot control device based on an improved YOLOv5 algorithm, comprising:

[0180] At least one processor, and,

[0181] A memory communicatively connected to the at least one processor; wherein,

[0182] The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform a harvesting robot control method based on the improved YOLOv5 algorithm as disclosed in any of the first aspects of the present invention.

[0183] The computer device can be a terminal, comprising a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a harvesting robot control method based on an improved YOLOv5 algorithm. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0184] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a harvesting robot control method based on the improved YOLOv5 algorithm as disclosed in any of the first aspects of the present invention.

[0185] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes described in the embodiments of the harvesting robot control method based on the improved YOLOv5 algorithm. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0186] Alternatively, if the above-mentioned modules of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, RAM, ROM, magnetic disks, or optical disks.

[0187] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A picking robot control method based on an improved yolov5 algorithm, characterized in that, The method includes: An orchard simulation environment is built, a target picking task is generated, the target path of the robot is determined, the error value between the robot's movement path and the target path is detected in real time by sensors, the proportional parameter P is calculated based on the error value, and the robot's motion state is adjusted by feeding back the control signal of the proportional parameter P through a PID controller. A video processing channel is constructed by loading the YOLOv5 algorithm model using a video stream processing framework, and an inference engine is configured to execute inference acceleration tasks on the video processing channel. The target video stream is acquired as input to the YOLOv5 algorithm model. The YOLOv5 algorithm model is optimized by transposed convolution, global context block, and C3GHOST convolution, respectively. The target recognition result, pixel coordinates, and the ratio of measured pixel distance to actual distance in the target video stream are output. The relative coordinates of the pixel coordinates are obtained according to the coordinate transformation. The static motion parameters of the robotic arm corresponding to the relative coordinates are calculated by inverse kinematics, and the dynamic motion parameters of the robotic arm are solved by fifth-order polynomial interpolation. Based on the static motion parameters, dynamic motion parameters and relative coordinates, the picking robot is controlled to perform the target picking task. The transposed convolution optimizes the YOLOv5 algorithm model, including: A transposed convolutional layer, `transposed_con`, is defined in the header structure of the YOLOv5 algorithm model using the `nn.ConvTranspose2d` class. The `torch.randn` function generates a random tensor to simulate the input image, which serves as the input for the upsampling operation of the YOLOv5 algorithm model. The input random tensor is transposed using the inverse convolution kernel of the `transposed_con` layer to generate an output tensor, which is then used for training to expand the feature map space dimension of each frame in the target video stream. The shape of the random tensor is represented as (batch_size, input_channels, input_height, input_width), where `batch_size` represents the batch size of the input, `input_channels` represents the number of input channels, `input_height` represents the height of the input, and `input_width` represents the width of the input. The optimization of the YOLOv5 algorithm model using global context blocks includes: The Global Context Block defines a 1x1 convolutional layer, a softmax layer, and a feature transformation module in sequence. The feature transformation module includes multiple convolutional layers, normalization layers, and activation functions. The input size of the input feature map in each frame of the target video stream is extracted. The input feature map is converted into a single-channel feature map through the 1x1 convolutional layer to obtain the global context information of the input feature map. The input size includes the batch size, number of channels, height h, and width w of the input feature map. The single-channel feature map is flattened and its shape is represented as (batch, 1, h, w), where 1 represents the number of channels, batch represents the batch size, h represents the height, and w represents the width. The single-channel feature map is normalized by the softmax layer to calculate the weight distribution. The single-channel feature map, which is used to represent global context information after flattening, is fused with the weight distribution by channel multiplication to calculate the global context vector. The shape of the global context vector is (batch, c, 1, 1). The feature transformation module applies linear transformation and nonlinear activation to the global context vector to generate a feature map with the same shape as the input feature map. The input feature map is then added point by point to the feature map after the global context vector transformation to obtain the feature representation of the input feature map enhanced by the global context block. The C3GHOST convolution optimizes the YOLOv5 algorithm model, including: A lightweight model is constructed by applying Ghost convolution in the backbone feature extraction network of the YOLOv5 algorithm model using GhostBottleneck. The GhostBottleneck class defines a residual block that includes GhostConv class, DWConv class and direct path residual connection. The GhostConv-based convolutional structure uses a 1x1 convolutional kernel to convert the number of input channels to half the number of hidden channels. After feature extraction using a 5x5 convolutional kernel, the feature outputs from the two convolutional kernel stages are concatenated along the channel dimension to add a Ghost Channel to each input channel, which is a linear combination of the original input channels, to represent the feature information of the original input channel. The DWConv class performs downsampling based on stride parameters, and depthwise separable convolution is used to perform model computation and feature extraction simultaneously. The information flow and feature reuse within the model are constructed through direct path residual connections using the GhostConv and DWConv classes.

2. The picking robot control method based on the improved yolov5 algorithm according to claim 1, characterized in that, The process of building a simulated orchard environment, generating a target harvesting task, and determining the robot's target path includes: Orchard map information is obtained by collecting early images of the orchard to simulate the orchard map environment. The starting point and ending point of the target picking task are planned according to the picking needs. One or more target paths are generated based on the starting point and ending point information.

3. The harvesting robot control method based on the improved YOLOv5 algorithm according to claim 2, characterized in that, The step of detecting the error value between the robot's movement path and the target path in real time using sensors, calculating a proportional parameter P based on the error value, and adjusting the robot's motion state by feeding back the proportional parameter P through a PID controller includes: The robot loads the target path and moves according to the starting point-to-end point map trajectory of the target picking task. By detecting and feeding back the actual movement path parameters in real time, it calculates the error value between the target path and the actual movement path in real time for each unit of time. Based on the error value and the target path in real time, it updates the value of the proportional parameter P to output the motion control parameters for the current time. The robot adjusts its current motion state through the motion control parameters until the proportional parameter P is adjusted to the optimal value. Based on the optimal value of the proportional parameter P and the error value at the current time, the robot continuously outputs motion control parameters to control its motion state until it reaches the destination.

4. The harvesting robot control method based on the improved YOLOv5 algorithm according to claim 1, characterized in that, The process involves using a video stream processing framework to load a YOLOv5 algorithm model to construct a video processing channel, and configuring an inference engine for the video processing channel to accelerate the object detection task, including: A video processing channel was built on the Jetson Nano embedded development board configured on the robot using the DeepStream video stream processing framework, and the TensorRT inference engine dependency was configured for the video processing channel. The yolov5 algorithm model was loaded into the DeepStream video stream processing framework. The YOLOv5 algorithm model is trained on a cloud server, outputting a .pt weight file. The .pt weight file is then transferred to a Jetson Nano embedded development board using the WINSCP graphical client. The TensorRT inference engine's conversion program converts the .pt weight file to a .wts weight file to optimize the YOLOv5 algorithm model's parameters. The .wts weight file is then compiled into an .engine file and imported into the target path of the DeepStream video stream processing framework. This converts the YOLOv5 algorithm model into an inference engine configured in the video processing channel for one or more tasks, including data input, data preprocessing, inference acceleration, post-processing, and output. The output is the YOLOv5 algorithm model performing object detection and tracking on each frame of the input video stream. The GPU of the Jetson Nano embedded development board executes the inference acceleration task of the YOLOv5 algorithm model through the TensorRT inference engine.

5. The harvesting robot control method based on the improved YOLOv5 algorithm according to claim 1, characterized in that, The step of obtaining the relative coordinates of the pixel coordinates based on coordinate transformation includes: Based on the pixel coordinates (x, y, w, h) determined by the recognition results, the coordinates of the image midpoint (cx, cy) and the target midpoint (zx, zy) corresponding to the recognition results are calculated respectively. The formulas for calculating the image midpoint coordinates and the target midpoint coordinates are as follows: ； The origin is set as the coordinates of the midpoint in the image. The target midpoint coordinates (zx, zy) are transferred relative to the origin using the ratio (bx, by) between the measured pixel distance and the actual distance to generate target coordinates (nx, ny). Based on the ratio (bx, by), the target coordinates (nx, ny) are converted into relative coordinates (rx, ry) measured in centimeters. The formulas for calculating the target coordinates and relative coordinates are as follows: ； The relative coordinates help the robot locate and navigate the target to be picked.

6. The harvesting robot control method based on the improved YOLOv5 algorithm according to claim 5, characterized in that, The calculation of the static motion parameters of the robotic arm corresponding to the relative coordinates through inverse kinematics includes: Construct the mapping relationship between the joint angles of the three-degree-of-freedom robotic arm and the base of the robotic arm, and collect the bottom height H1, upper arm length L1, lower arm length L2 and arm-direction compensation B of the robotic arm; Calculate the centimeter coordinates (X, Y) between the target and the robotic arm base based on the relative coordinates (rx, ry), and then use the robotic arm parameters and centimeter coordinates as a basis. Calculate the static motion parameters of the robotic arm, and the formula for calculating the static motion parameters is as follows: ； The static motion parameters include vertical projection T, base angle J0, upper arm angle J1, forearm angle J2, short side D, and hypotenuse X0.

7. The harvesting robot control method based on the improved YOLOv5 algorithm according to claim 6, characterized in that, The method of solving the dynamic motion parameters of the robotic arm using fifth-order polynomial interpolation includes: Define the starting angle INIT, target angle TARGET, initial angular velocity W0, target angular velocity W1, initial angular acceleration AC0, target angular acceleration AC1, total motion time TF, time step TS, and motion time point T for the dynamic motion of the robotic arm servo motor. Calculate the servo motor's initial angle A0, initial angular velocity A1, half of the initial angular velocity A2, acceleration A3, square of acceleration A4, and cube of acceleration A5 using these parameters. Based on the robotic arm servo motor's angles Theta, WC0, and ACC0, calculate the servo motor's angle, angular velocity, and angular acceleration positions at different time points to provide dynamic motion parameters for smooth servo motor motion. The calculation formula is as follows: 。 8. A harvesting robot, used to perform the steps of the harvesting robot control method based on the improved YOLOv5 algorithm as described in any one of claims 1-7, characterized in that, include: A mobile platform component, a robotic arm component mounted on the mobile platform, and a control system mounted within the mobile platform; The mobile platform assembly includes a chassis consisting of a top plate (1), a bottom plate (2) and a support column (3), a Mecanum wheel (4) located at the bottom of the bottom plate (2) and a geared motor (5) that drives the Mecanum wheel (4). The robotic arm assembly includes a tray (6), an arm (7), a wrist (8), a gripper (9), a lead screw (10) that drives the extension and retraction of the arm (7), a servo motor (11) that drives the rotation of the wrist (8), a drive motor (12) that drives the gripper (9) to operate, and a camera (13) mounted on the rear end of the gripper (9) for acquiring target video streams. The control system includes a main control module, a wireless control module, a sensor module, a motor drive module, and a power supply module. The main control module includes an STM32RCT6 microcontroller for controlling the robot's motion trajectory and wireless control, a JETSON NANO embedded development board for controlling target video stream image processing, and a PCA9685 module for controlling the robotic arm based on the image processing results from the JETSON NANO embedded development board. The PCA9685 module uses an I2C interface to control the angle of the robotic arm's servo motors. The JETSON NANO embedded development board accelerates the inference of the YOLOv5 algorithm model through the DeepStream video stream processing framework and the TensorRT inference engine. The STM32RCT6 microcontroller and the JETSON NANO embedded development board are interconnected through TX and RX pins to enable bidirectional information transmission between the main control module and the robot. The wireless control module is configured as an HC-05 Bluetooth module, which communicates with mobile devices via Bluetooth to remotely control the harvesting robot. The sensor module is configured as an MPU6050 module integrating a three-axis gyroscope and a three-axis accelerometer, for measuring the rotation angle and rotational acceleration of the robotic arm; The motor drive module is configured as a TB6612 dual H-bridge driver to control one or more motors mounted on the harvesting robot for motor drive. The power supply module includes a 12V power supply for the STM32RCT6 microcontroller via a DC socket, a 5V 4A DC power supply configured on the JETSON NANO embedded development board, and an LM2596 voltage regulator module that provides a stable 5V power supply for the HC-05 Bluetooth module, MPU6050 module, and PID tracking module. The JETSON NANO embedded development board and the PCA9685 module are connected via a step-down module that outputs 5V to the PCA9685 module to execute the robotic arm servo control.

9. The harvesting robot according to claim 8, characterized in that, The control system also includes a display module, configured as an OLED display screen to visualize the robot's status information and to provide click buttons for adjusting the tracking parameters of the PID tracking module. The status information includes the robot's elevation angle, depression angle, yaw angle, type of target to be picked, and number of targets to be picked.

10. A harvesting robot control device based on an improved YOLOv5 algorithm, characterized in that, include: At least one processor, and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the harvesting robot control method based on the improved YOLOv5 algorithm as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the harvesting robot control method based on the improved YOLOv5 algorithm as described in any one of claims 1 to 7.

Images