Grape picking robot viewpoint planning method based on deep reinforcement learning

Abstract

The invention discloses a viewpoint planning method based on deep reinforcement learning. The method comprises the following steps: acquiring a sheltered image and a camera attitude and obtaining state information; inputting the state information into a trained DQN network to obtain a camera motion action at the next moment; and controlling the mechanical arm to move to adjust the camera position. In order to improve the training efficiency of the DQN network, a training strategy of global trend guide learning is creatively provided. The global trend guided learning strategy refers to a method for forming a global trend by recording viewpoint areas of successfully detected fruit stems in the same scene in a learning process, and then guiding a network learning process by using the global trend. According to the invention, viewpoint planning is realized by a brand-new technical means, so that the problem of shielding of the grape picking robot in the picking operation process is solved. According to the method, the viewpoint planning time is greatly shortened under the condition that the high picking success rate is kept, and a new technical scheme is provided for the field of active vision technology application.

Description

Technical Field

[0001] The present invention belongs to the technical field of active vision of grape picking robots, and in particular relates to a viewpoint planning method based on deep reinforcement learning. Background Art

[0002] In order to solve the occlusion problem faced by grape picking robots during the picking process and improve the robot's autonomous operation ability, researchers have proposed applying active vision technology to picking robots. The so-called active vision technology refers to the technology that the visual system can actively select the movement of the camera according to the current task requirements and obtain the corresponding image from the appropriate perspective. According to the definition of active vision, the key to the implementation of active vision technology in the picking robot lies in how to perform viewpoint planning. The present invention is directed to a grape picking robot. The robot is equipped with a 6-degree-of-freedom robotic arm and is installed on a crawler chassis. A depth camera is installed at the end of the robotic arm, and the camera viewing angle is controlled by controlling the posture of the end of the robotic arm.

[0003] The invention patent "A viewpoint planning method based on active visual strategy and its picking system" (publication number: CN116619388A) discloses a viewpoint planning method based on spatial random sampling. This method randomly generates a number of candidate viewpoints around the picking point area, constructs a three-dimensional voxel map of the grape picking point area, and then uses a scoring function based on the spatial occlusion rate to calculate the score of each candidate viewpoint to predict the ideal viewpoint, thereby adjusting the position of the camera on the robotic arm. Although this type of viewpoint planning method is relatively simple to implement, the short-term decision-making method of evaluating candidate viewpoints using a scoring function can easily cause the planned viewpoint to fall into a local optimum. Secondly, three-dimensional mapping is required for viewpoint evaluation, which makes the entire planning time too long.

[0004] Deep reinforcement learning technology has been proven to be suitable for long-term decision-making processes because it considers cumulative rewards to make current decisions. At present, there is little research based on deep reinforcement in the field of viewpoint planning for grape picking robots. The reason is that the deep reinforcement learning model is still unclear in specific viewpoint planning tasks. Secondly, the learning process of deep reinforcement learning networks is essentially trial and error. How to design reward functions and learning strategies to improve learning efficiency is also an issue that needs to be studied in depth. In order to realize the robot's autonomous picking behavior, it is urgent to study the viewpoint planning method of deep reinforcement learning. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a viewpoint planning method based on deep reinforcement learning. The method of the present invention is different from the viewpoint planning method based on random sampling. Instead, the viewpoint planning process is set as a continuous action decision process. The main goal is to find an action strategy that enables the grape picking machine to find the ideal viewing angle for detecting the fruit stem in a highly occluded environment according to the principle of maximizing the expected cumulative reward.

[0006] The technical solution of the present invention to solve the above problem comprises the following steps:

[0007] Step 1: Collect images and camera pose data from the grape picking scene to create a dataset for deep reinforcement learning training;

[0008] Step 2: According to the requirements of the grape picking viewpoint planning task, determine the state space and design the reward function for training the network;

[0009] Step 3: Build a deep reinforcement learning network structure, use the prepared data set and the designed reward function to train the network to obtain a trained action strategy network;

[0010] Step 4: Place the picking robot in a real picking scene, obtain the image taken by the depth camera, and process the image to obtain the ROI image. At the same time, obtain the position of the camera in the viewpoint planning process;

[0011] Step 5: Input the ROI image into the Mask R-CNN network to detect the area where the leaves block the fruit stem and obtain the detection frame of the blocked area;

[0012] Step 6: Input the current view state into the trained action strategy network to obtain the camera motion action at the next moment;

[0013] Step 7: Determine whether the total number of actions executed exceeds the maximum allowed number. If so, end the viewpoint planning. Otherwise, control the robot to execute the motion output by the action strategy network, thereby adjusting the position of the depth camera on the robot to obtain a new viewpoint; then determine whether the new viewpoint can detect the fruit stem. If so, end the viewpoint planning and proceed with the picking operation. Otherwise, continue to adjust the depth camera position according to steps 4 to 7.

[0014] Furthermore, in step 1, the following steps are specifically included:

[0015] Step 1.1. Collect data from three grape picking scenes. The occluded leaves in the three scenes are located at different occlusion angles on the left, middle, and right. In each scene, the camera viewpoint is sampled on a sphere with the grape center point Q in space as the origin and R as the radius. In the spherical coordinate system, the depth camera position p = [R, θ, ψ] TIn order to make the camera always face the target grape, the direction vector is calculated according to the center point Q and the depth camera position to get the camera posture. It is divided into finite rectangular areas in the ∑ψOθ coordinate system, defined as Regions=[R0,R1,...,R i ]. Any region R i The length and width of region R are Δψ and Δθ respectively. i The center point is (ψ i ,θ i ), R i The area can be expressed as

[0016] R i＝ {(ψ i +x,θ i +y)|-Δψ / 2≤x≤Δψ / 2,-Δθ / 2≤y≤Δθ / 2}

[0017] The sampling process first randomly selects 5 points from the R0 region in Regions as the initial sampling points. Then, starting from each initial sampling point, all vertices on the rectangular grid formed in the ∑ψOθ coordinate system with a horizontal step size of Δψ and a vertical step size of Δθ are used as sampling points. Finally, the robotic arm is controlled to move the camera to each sampling point on the spherical surface to obtain RGB images and data such as the position of the camera relative to the center of mass of the grapes.

[0018] Step 1.2: In the ∑ψOθ coordinate system, record the sampling points corresponding to the +Δψ, -Δψ, +Δθ, and -Δθ actions of each sampling point, and establish the connection between the sampling points. If the sampling point exceeds the range of the robot arm after executing the action, an invalid sampling point is recorded.

[0019] Furthermore, in step 2, the following steps are specifically included:

[0020] Step 2.1, state space mainly considers visual state and spatial state. The visual state includes the image of the occluded ROI area and the detection frame of the occluded area; the spatial state includes the position of the camera relative to the center GC of the grapes. First, use the Mask R-CNN network to segment the grape bunches. Assume that the center of the ROI area is located on the centroid line of the fruit bunch perpendicular to the ground. The centroid of the fruit bunch is usually the geometric center of the fruit bunch on the 2D image. After the Mask R-CNN network outputs the semantic segmentation result, the centroid point PC (u c ,v c ), and calculate the vertex of the grape contour as T(u t ,v t The detection box width output by the Mask R-CNN network is w. Considering the working space margin of the shearing mechanism, D(u c , vt -3*w / 8) as the center coordinate, and a rectangular area with a side length of L = 1.5w as the ROI area. The ROI area is intercepted on the acquired RGB image to obtain the ROI image. Secondly, after obtaining the ROI image, the Mask R-CNN network is used again to detect and segment the ROI image to obtain the detection frame of the occluded area. Finally, the center point Q of the grape in space is determined based on the centroid point of the target grape bunch and the internal and external parameters of the depth camera. And in the spherical coordinate system with Q as the origin, the depth camera spatial position p = [r, θ, ψ] is obtained. T Considering that the change of the sphere radius r will not have a substantial impact on the view planning, r is set to a fixed value R, and the position of the depth camera can be represented by the parameters ψ and θ.

[0021] Step 2.2: To evaluate the current state S t Perform different actions A t , define the action scoring function And given by the following formula in and They represent the occlusion area reduction factor and the occlusion area increase factor on one side of the center line. At the same time, ω∈[0,1] represents the weight coefficient. From the definition of When it is greater than 0, this indicates that execution A t The state S after the action t+1 The reward is adjusted by the action step length L and the action scoring function. The reward function is set to

[0022]

[0023] Furthermore, in step 3, the following steps are specifically included:

[0024] Step 3.1, the action strategy network is implemented using a DQN network, which consists of state input information, a preprocessed ResNet-50 network, and a multi-layer perceptron MLP. The state input information includes the ROI image, the detection box of the occluded area, and the ψ and θ of the current viewpoint position. In order to extract the features of the ROI image, a pre-trained ResNet-50 network is used. The network is trained based on the ImageNet dataset. Unlike the original Resnet50 network, the SoftMax layer of the residual network is discarded to directly output a 2048-dimensional feature vector. In addition, the input features also include 2-dimensional information of the camera position state and 4-dimensional information of the normalized occlusion detection box. All features are fused and input into a multi-layer perceptron MLP. The multi-layer perceptron consists of an input layer, a hidden layer, and an output layer. Since the state features are a 2054-dimensional feature vector, the number of neurons in the input layer is 2054; the number of neurons in the hidden layer is set to 256; the output action of the DQN network for viewpoint planning is A = {+Δθ, -Δθ, +Δψ, -Δψ}, so the MLP output layer contains 4 neurons. The Relu activation function is used between each layer in the MLP.

[0025] Step 3.2: Define a cache D to record the viewpoint position V of the fruit stem that is successfully detected during the learning process. p =[r,θ,ψ] T , where r = R is a fixed radius. In order to reduce the storage space of D, the space is divided into Regions. The recording process only needs to store Region R i The number i is recorded in D.

[0026] At the beginning of training, the initial viewpoint V is randomly sampled in the invisible area of ​​the fruit stem init Each learning cycle first obtains the init The state S t And input to the evaluation network Q eval Get action A t . Successfully executed action A t Afterwards, we get the reward r(S t ,A t ) and new perspectives V t+1 . Then select the ideal viewpoint area from the cache D and get the global trend T. This process is divided into three steps. First, in the ∑ψOθ coordinate system, calculate the distance d from the center of each viewpoint area in D to the current position. The distance d calculation formula is given by the following formula:

[0027]

[0028] Among them, (ψ, θ) represents the current camera position C, (ψ i ,θ i) represents the center point of the viewpoint area. Secondly, the area with the shortest distance is selected as S t Finally, the center point of this area is selected as the target viewpoint P(θ T ,ψ T ), with vector Approximately represents the global trend T. Define action A t The direction vector is New reward new_r(S t ,A t ) is given by the following formula:

[0029]

[0030] in It means that the global trend is inconsistent with the direction of movement, so a penalty of ε is given in the new reward to speed up the training process. t ,A t ,new_r(S t ,A t ),S t+1 ) and other data are stored in the playback buffer B. Each time learning, n groups of data are taken from B, and then the loss Loss is calculated. The loss calculation formula is given by the following formula

[0031] Loss=(new_r(S t ,A t )+γmaxQ target (S t+1 ,A t+1； θ target )-Q eval (S t ,A t θ eval )) 2

[0032] Among them, γ is the loss factor. After calculating the loss, Q is updated according to the gradient descent method eval The parameters of the network are eval After each learning cycle, if the fruit stem is successfully detected, the viewpoint area in cache D is updated. When the learning cycle reaches 150 times, Q eval The network parameters are updated to Q target network.

[0033] Beneficial Effects

[0034] Compared with the existing method, the advantages of the present invention are:

[0035] 1. The technical solution of the present invention provides a viewpoint planning method based on deep reinforcement learning. Different from the viewpoint planning method based on random sampling, the viewpoint planning process is set as a continuous action decision process. The main goal is to find an action strategy that enables the grape picking machine to find the ideal viewing angle for detecting the fruit stem in a highly occluded environment according to the principle of maximizing the expected cumulative reward. This method effectively solves the problem that the short-term decision-making method of evaluating candidate viewpoints using a scoring function easily causes the planned viewpoint to fall into a local optimum.

[0036] 2. In order to shorten the viewpoint planning decision time, the technical solution of the present invention abandons the time-consuming three-dimensional mapping process and makes decisions based on the method of extracting features from two-dimensional images. Compared with the viewpoint planning method based on three-dimensional mapping of the picking point area, the running time is greatly shortened. Secondly, the action network adopts a simpler MLP network structure to further shorten the action decision time.

[0037] 3. In order to realize the training process of the action strategy network, the technical solution of the present invention produces a data set for the grape picking robot viewpoint planning application scenario, and designs a reward function for training by integrating the action step length, action scoring function and motion constraints. The proposed reward function can achieve a higher success rate than the reward function based on the maximum action score.

[0038] 4. In a further preferred embodiment of the present invention, a global learning strategy is proposed to improve the learning efficiency of the action strategy network. The so-called global trend guided learning strategy means that in the learning process, a global trend T is formed by recording the viewpoint area where the fruit stalk is successfully detected in the same scene, and then the global trend is used to guide the network learning method. The principle is that when S t In this state, execute action A t When the movement direction of is inconsistent with T, a penalty factor is added to the original reward as a new reward input into the network for learning, thereby achieving the purpose of speeding up the network training process.

[0039] In summary, the present invention uses a new technical means to achieve viewpoint planning to solve the occlusion problem faced by grape picking robots during the picking process. The method greatly shortens the viewpoint planning time while maintaining a high picking success rate, providing a new technical solution for the application field of active vision technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the present invention are briefly introduced below. Figure 1 A system framework diagram of a deep reinforcement learning viewpoint planning method provided by an embodiment of the present invention;

[0041] Figure 2This is a flowchart of the viewpoint planning method based on deep reinforcement learning;

[0042] Figure 3 is a schematic diagram of a state space provided by an embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the changes in the picking point area before and after viewpoint planning;

[0044] Figure 5 is a network structure diagram provided by an embodiment of the present invention;

[0045] Figure 6 It is a schematic diagram of global trend guidance. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] Embodiment 1:

[0048] like Figure 1 As shown, the picking system used in Example 1 of the present invention is equipped with a 6-DOF robotic arm and is installed on a crawler chassis. A depth camera is installed at the end of the robotic arm, and the camera angle of view is controlled by controlling the posture of the end of the robotic arm. The system obtains the robotic arm movement action by inputting ROI image and robot status information into a DQN network to achieve the purpose of changing the angle of view. The viewpoint planning process is a continuous action process, and it stops until the angle of view that can detect the fruit stem is obtained.

[0049] Embodiment 2:

[0050] like Figure 2 As shown, the viewpoint planning method based on deep reinforcement learning of the present invention has the following specific process:

[0051] Step 1: Collect images and camera pose data from the grape picking scene to create a dataset for deep reinforcement learning training;

[0052] Step 2: According to the requirements of the grape picking viewpoint planning task, determine the state space and design the reward function for training the network;

[0053] Step 3: Build a deep reinforcement learning network structure, use the prepared data set and the designed reward function to train the network to obtain a trained action strategy network;

[0054] Step 4: Place the picking robot in a real picking scene, obtain the image taken by the depth camera, and process the image to obtain the ROI image. At the same time, obtain the position of the camera in the viewpoint planning process;

[0055] Step 5: Input the ROI image into the Mask R-CNN network to detect the area where the leaves block the fruit stem and obtain the detection frame of the blocked area;

[0056] Step 6: Input the current view state into the trained action strategy network to obtain the camera motion action at the next moment;

[0057] Step 7: Determine whether the total number of actions executed exceeds the maximum allowed number. If so, end the viewpoint planning. Otherwise, control the robot to execute the motion output by the action strategy network, thereby adjusting the position of the depth camera on the robot to obtain a new viewpoint; then determine whether the new viewpoint can detect the fruit stem. If so, end the viewpoint planning and proceed with the picking operation. Otherwise, continue to adjust the depth camera position according to steps 4 to 7.

[0058] Furthermore, in step 1, the following steps are specifically included:

[0059] Step 1.1. Collect data from three grape picking scenes. The occluded leaves in the three scenes are located at different occlusion angles on the left, middle, and right. In each scene, the camera viewpoint is sampled on a sphere with the grape center point Q in space as the origin and R as the radius. In the spherical coordinate system, the depth camera position p = [R, θ, ψ] T In order to make the camera always face the target grape, the direction vector is calculated according to the center point Q and the depth camera position to get the camera posture. It is divided into finite rectangular areas in the ∑ψOθ coordinate system, defined as Regions=[R0,R1,...,R i ]. Any region R i The length and width of region R are Δψ and Δθ respectively. i The center point is (ψ i ,θ j ), R i The area can be expressed as

[0060] R i＝ {(ψ i +x,θ j +y)|-Δψ / 2≤x≤Δψ / 2,-Δθ / 2≤y≤Δθ / 2}

[0061] The sampling process first randomly selects 5 points from the R0 region in Regions as the initial sampling points. Then, starting from each initial sampling point, all vertices on the rectangular grid formed in the ∑ψOθ coordinate system with a horizontal step size of Δψ and a vertical step size of Δθ are used as sampling points. Finally, the robotic arm is controlled to move the camera to each sampling point on the spherical surface to obtain RGB images and data such as the position of the camera relative to the center of mass of the grapes.

[0062] Step 1.2: In the ∑ψOθ coordinate system, record the sampling points corresponding to the +Δψ, -Δψ, +Δθ, and -Δθ actions of each sampling point, and establish the connection between the sampling points. If the sampling point exceeds the range of the robot arm after executing the action, an invalid sampling point is recorded.

[0063] Furthermore, in step 2, the following steps are specifically included:

[0064] Step 2.1: Figure 3 As shown in the figure, the state space mainly considers the visual state and the spatial state. The visual state includes the image of the occluded ROI area and the detection frame of the occluded area; the spatial state includes the position of the camera relative to the center GC of the grapes. First, the Mask R-CNN network is used to segment the grape bunches. Assume that the center of the ROI area is located on the centroid line of the fruit bunch perpendicular to the ground. The centroid of the fruit bunch is usually the geometric center of the fruit bunch on the 2D image. After the Mask R-CNN network outputs the semantic segmentation result, the centroid point PC (u c ,v c ), and calculate the vertex of the grape contour as T(u t ,v t The detection box width output by the Mask R-CNN network is w. Considering the working space margin of the shearing mechanism, D(u c , v t -3*w / 8) as the center coordinate, and a rectangular area with a side length of L = 1.5w as the ROI area. The ROI area is intercepted on the acquired RGB image to obtain the ROI image. Secondly, after obtaining the ROI image, the MaskR-CNN network is used again to detect and segment the ROI image to obtain the detection frame of the occluded area. Finally, the center point Q of the grapes in space is determined based on the centroid of the target grape bunch and the internal and external parameters of the depth camera. And in the spherical coordinate system with Q as the origin, the depth camera spatial position p = [r, θ, ψ] is obtained. T Considering that the change of the sphere radius r will not have a substantial impact on the view planning, r is set to a fixed value R, and the position of the depth camera can be represented by the parameters ψ and θ.

[0065] Step 2.2: To evaluate the current state S t Perform different actions A t , define the action scoring function And given by the following formula

[0066]

[0067] in and They represent the occlusion area reduction factor and the occlusion area increase factor on one side of the center line. At the same time, ω∈[0,1] represents the weight coefficient. Figure 4 As shown, according to From the definition of When it is greater than 0, this indicates that execution A t The state S after the action t+1 The reward is adjusted by the action step length L and the action scoring function. The reward function is set to

[0068]

[0069] Furthermore, in step 3, the following steps are specifically included:

[0070] Step 3.1: Figure 5 As shown in the figure, the action policy network is implemented using a DQN network, which consists of state input information, a preprocessed ResNet-50 network, and a multi-layer perceptron MLP. The state input information includes the ROI image, the detection box of the occluded area, and the θ and ψ of the current viewpoint position. In order to extract the features of the ROI image, a pre-trained ResNet-50 network is used. The network is trained based on the ImageNet dataset. Unlike the original Resnet50 network, the SoftMax layer of the residual network is discarded to directly output a 2048-dimensional feature vector. In addition, the input features also include 2-dimensional information of the camera position state and 4-dimensional information of the normalized occluded detection box. All features are fused and input into a multi-layer perceptron MLP. The multi-layer perceptron consists of an input layer, a hidden layer, and an output layer. Since the state features are a 2054-dimensional feature vector, the number of neurons in the input layer is 2054; the number of neurons in the hidden layer is set to 256; the output action of the DQN network for viewpoint planning is A = {+Δθ, -Δθ, +Δψ, -Δψ}, so the MLP output layer contains 4 neurons. The Relu activation function is used between each layer in the MLP.

[0071] Step 3.2: Define a cache D to record the viewpoint position V of the fruit stem that is successfully detected during the learning process. p =[r,θ,ψ] T , where r = R is a fixed radius. In order to reduce the storage space of D, the space is divided into Regions. The recording process only needs to store Region R i The number i is recorded in D.

[0072] At the beginning of training, the initial viewpoint V is randomly sampled in the invisible area of ​​the fruit stem init Each learning cycle first obtains the init The state S tAnd input to the evaluation network Q eval Get action A t . Successfully executed action A t Afterwards, we get the reward r(S t ,A t ) and new perspectives V t+1 . Then select the ideal viewpoint area from the cache D and get the global trend T. This process is divided into three steps. First, in the ∑ψOθ coordinate system, calculate the distance d from the center of each viewpoint area in D to the current position. The distance d calculation formula is given by the following formula:

[0073]

[0074] Among them, (θ, ψ) represents the current camera position C, (θ i ,ψ i ) represents the center point of the viewpoint area. Secondly, the area with the shortest distance is selected as S t Finally, the center point in this area is selected as the target viewpoint P(θ T ,ψ T ), with vector Approximately represents the global trend T. Define action A t The direction vector is New reward new_r(S t ,A t ) is given by the following formula:

[0075]

[0076] in Indicates that the global trend is inconsistent with the direction of movement, such as Figure 6 As shown. Therefore, a penalty of ε is given in the new reward to speed up the training process. t ,A t ,new_r(S t ,A t ),S t+1 ) and other data are stored in the playback buffer B. Each time learning, n groups of data are taken from B, and then the loss Loss is calculated. The loss calculation formula is given by the following formula

[0077] Loss=(new_r(S t ,A t )+γmaxQ target (S t+1 ,A t+1； θ target )-Q eval (S t ,A t θeval )) 2

[0078] Among them, γ is the loss factor. After calculating the loss, Q is updated according to the gradient descent method eval The parameters of the network are eval After each learning cycle, if the fruit stem is successfully detected, the viewpoint area in cache D is updated. When the learning cycle reaches 150 times, Q eval The network parameters are updated to Q target network.

[0079] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A grape picking robot viewpoint planning method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect images and camera pose data from the grape picking scene to create a dataset for deep reinforcement learning training; Step 2: According to the requirements of the grape picking viewpoint planning task, determine the state space and design the reward function for training the network; Step 3: Build a deep reinforcement learning network structure, use the prepared data set and the designed reward function to train the network to obtain a trained action strategy network; Step 4: Place the picking robot in a real picking scene, obtain the image taken by the depth camera, and process the image to obtain the ROI image. At the same time, obtain the position of the camera in the viewpoint planning process; Step 5: Input the ROI image into the Mask R-CNN network to detect the area where the leaves block the fruit stem and obtain the detection frame of the blocked area; Step 6: Input the current view state into the trained action strategy network to obtain the camera motion action at the next moment; Step 7: Determine whether the total number of actions executed exceeds the maximum allowed number. If so, end the viewpoint planning. Otherwise, control the robot to execute the motion output by the action strategy network, thereby adjusting the position of the depth camera on the robot to obtain a new viewpoint; Then determine whether the new viewpoint can detect the fruit stem. If so, end the viewpoint planning and proceed with the picking operation. Otherwise, continue to adjust the depth camera position according to steps 4 to 7.

2. According to the grape picking robot viewpoint planning method based on deep reinforcement learning according to claim 1, the specific steps of step 1 are: Step 1.

1. Collect data from three grape picking scenes. The occluded leaves in the three scenes are located at different occlusion angles on the left, middle, and right. In each scene, the camera viewpoint is sampled on a sphere with the grape center point Q in space as the origin and R as the radius. In the spherical coordinate system, the depth camera position p = [R, θ, ψ] T In order to make the camera always face the target grape, the direction vector is calculated according to the center point Q and the depth camera position to get the camera posture. It is divided into finite rectangular areas in the ∑ψOθ coordinate system, defined as Regions=[R0,R1,...,R i ]. Any region R i The length and width of region R are Δψ and Δθ respectively. i The center point is (ψ i ,θ i ), R i The area can be expressed as R i＝ {(ψ i +x,θ j +y)|-Δψ / 2≤x≤Δψ / 2,-Δθ / 2≤y≤Δθ / 2} The sampling process first randomly selects 5 points from the R0 region in Regions as the initial sampling points. Then, starting from each initial sampling point, all vertices on the rectangular grid formed in the ∑ψOθ coordinate system with a horizontal step size of Δψ and a vertical step size of Δθ are used as sampling points. Finally, the robotic arm is controlled to move the camera to each sampling point on the spherical surface to obtain RGB images and data such as the position of the camera relative to the center of mass of the grapes. Step 1.2: In the ∑ψOθ coordinate system, record the sampling points corresponding to the +Δψ, -Δψ, +Δθ, and -Δθ actions of each sampling point, and establish the connection between the sampling points. If the sampling point exceeds the range of the robot arm after executing the action, an invalid sampling point is recorded.

3. According to the grape picking robot viewpoint planning method based on deep reinforcement learning according to claim 1, the specific steps of step 2 are: Step 2.1, state space mainly considers visual state and spatial state. The visual state includes the image of the occluded ROI area and the detection frame of the occluded area; the spatial state includes the position of the camera relative to the center GC of the grapes. First, use the Mask R-CNN network to segment the grape bunches. Assume that the center of the ROI area is located on the centroid line of the fruit bunch perpendicular to the ground. The centroid of the fruit bunch is usually the geometric center of the fruit bunch on the 2D image. After the Mask R-CNN network outputs the semantic segmentation result, the centroid point PC (u c ,v c ), and calculate the vertex of the grape contour as T(u t ,v t The detection box width output by the Mask R-CNN network is w. Considering the working space margin of the shearing mechanism, D(u c , v t -3*w / 8) as the center coordinate, and a rectangular area with a side length of L = 1.5w as the ROI area. The ROI area is intercepted on the acquired RGB image to obtain the ROI image. Secondly, after obtaining the ROI image, the Mask R-CNN network is used again to detect and segment the ROI image to obtain the detection frame of the occluded area. Finally, the center point Q of the grape in space is determined based on the centroid point of the target grape bunch and the internal and external parameters of the depth camera. And in the spherical coordinate system with Q as the origin, the depth camera spatial position p = [r, θ, ψ] is obtained. T Considering that the change of the sphere radius r will not have a substantial impact on the view planning, r is set to a fixed value R, and the position of the depth camera can be represented by the parameters ψ and θ. Step 2.2: To evaluate the current state S t Perform different actions A t , define the action scoring function And given by the following formula in and They represent the occlusion area reduction factor and the occlusion area increase factor on one side of the center line. At the same time, ω∈[0,1] represents the weight coefficient. From the definition of When it is greater than 0, this indicates that execution A t The state S after the action t+1 The reward is adjusted by the action step length L and the action scoring function. The reward function is set to 4. According to the grape picking robot viewpoint planning method based on deep reinforcement learning according to claim 1, the specific steps of step 3 are: Step 3.1, the action strategy network is implemented using a DQN network, which consists of state input information, a preprocessed ResNet-50 network, and a multi-layer perceptron MLP. The state input information includes the ROI image, the detection box of the occluded area, and the ψ and θ of the current viewpoint position. In order to extract the features of the ROI image, a pre-trained ResNet-50 network is used. The network is trained based on the ImageNet dataset. Unlike the original Resnet50 network, the SoftMax layer of the residual network is discarded to directly output a 2048-dimensional feature vector. In addition, the input features also include 2-dimensional information of the camera position state and 4-dimensional information of the normalized occlusion detection box. All features are fused and input into a multi-layer perceptron MLP. The multi-layer perceptron consists of an input layer, a hidden layer, and an output layer. Since the state features are a 2054-dimensional feature vector, the number of neurons in the input layer is 2054; the number of neurons in the hidden layer is set to 256; the output action of the DQN network for viewpoint planning is A = {+Δθ, -Δθ, +Δψ, -Δψ}, so the MLP output layer contains 4 neurons. The Relu activation function is used between each layer in the MLP. Step 3.2: Define a cache D to record the viewpoint position V of the fruit stem that is successfully detected during the learning process. p =[r,θ,ψ] T , where r = R is a fixed radius. In order to reduce the storage space of D, the space is divided into Regions. The recording process only needs to store Region R i The number i is recorded in D. At the beginning of training, the initial viewpoint V is randomly sampled in the invisible area of ​​the fruit stem init Each learning cycle first obtains the init The state S t And input to the evaluation network Q eval Get action A t Action A is successfully executed t Afterwards, we get the reward r(S t ,A t ) and new perspectives V t+1 . Then select the ideal viewpoint area from the cache D and get the global trend T. This process is divided into three steps. First, in the ∑ψOθ coordinate system, calculate the distance d from the center of each viewpoint area in D to the current position. The distance d calculation formula is given by the following formula: in, (ψ,θ) represents the current camera position C, (ψ i ,θ i ) represents the center point of the viewpoint area. Secondly, the area with the shortest distance is selected as S t Finally, the center point in this area is selected as the target viewpoint P(θ T ,ψ T ), with vector Approximately represents the global trend T. Define action A t The direction vector is New reward new_r(S t ,A t ) is given by the following formula: in It means that the global trend is inconsistent with the direction of movement, so a penalty of ε is given in the new reward to speed up the training process. t ,A t ,new_r(S t ,A t ),S t+1 ) and other data are stored in the playback buffer B. Each time learning, n groups of data are taken from B, and then the loss Loss is calculated. The loss calculation formula is given by the following formula Loss=(new_r(S t ,A t )+γmaxQ target (S t+1 ,A t+1； i target )-Q eval (S t ,A t ;θ eval )) 2 Among them, γ is the loss factor. After calculating the loss, Q is updated according to the gradient descent method eval The parameters of the network are eval After each learning cycle, if the fruit stem is successfully detected, the viewpoint area in cache D is updated. When the learning cycle reaches 150 times, Q eval The network parameters are updated to Q target network.

Images