Fruit and vegetable transport vehicle, follow-up transfer control method, electronic equipment and storage medium

Abstract

The invention discloses a fruit and vegetable transport vehicle, a follow-up transfer control method, electronic equipment and a storage medium. The autonomous follow-up transfer control method of the fruit and vegetable transport vehicle comprises the following steps of performing gesture recognition training; constructing a map; after the first target gesture is detected, the transport vehicle follows the picking worker in the (N-1) th working space; in the picking process, it is detected that the picking worker makes a second target gesture, the fruit and vegetable transport vehicle exits from the (N-1) th working space and returns to the unloading subpackaging place, and after unloading is completed, the fruit and vegetable transport vehicle returns to the basket full return point after a third target gesture is detected; or, the fruit and vegetable transport vehicle moves to the operation end point position of the (N-1) th working space, and returns to the unloading and sub-packaging point position; and after recognizing the third target gesture, the fruit and vegetable transport vehicle located at the unloading sub-packaging point position returns to the working starting point position of the Nth working space. The control method can meet the intelligent harvesting and transporting requirements of fruits and vegetables in the multi-span glass greenhouse.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of agricultural robots, and more specifically, to a fruit and vegetable transport vehicle and a follow-up transport control method, an electronic device and a storage medium. Background Art

[0002] In existing fruit and vegetable planting areas, such as multi-span glass greenhouses, the picking and transportation of fruits and vegetables is usually still done manually by entering between rows to pick them, placing them in a portable storage device, and then transporting them to designated loading locations by manual handling or remote-controlled rail transport vehicles. Moreover, the rail transport vehicle needs to be paved with standard tracks between the fruit and vegetable planting rows for manual driving of the rail lifting vehicle. Due to the paving of the tracks, the conditions for manual entry between rows for picking and transportation are more complicated, the labor intensity is increased, and the efficiency is reduced. In addition, the use of manual remote control to operate the remote-controlled rail transport vehicle is complicated to operate and cannot meet the needs of intelligent harvesting and transportation of fruits and vegetables in multi-span glass greenhouses. Summary of the invention

[0003] The purpose of the present invention is to provide a new technical solution for a fruit and vegetable transport vehicle and a follow-up transfer control method, an electronic device and a storage medium, which can solve at least one of the technical problems in the prior art such as complex picking and transportation operation conditions, increased labor intensity, and reduced efficiency.

[0004] In a first aspect of the present invention, a method for autonomous following and transporting control of a fruit and vegetable transport vehicle is provided, comprising the following steps: selecting a first target gesture, a second target gesture and a third target gesture for gesture recognition training to obtain a gesture recognition algorithm model, and deploying the gesture recognition algorithm model in the control system of the fruit and vegetable transport vehicle; constructing a map, the map comprising information of an operation area and a crop planting row, the operation area comprising at least one crop planting column, the crop planting column comprising a plurality of crop planting rows, the space between two adjacent crop planting rows being a workspace, the workspace being divided into the first workspace ... the Nth workspace, N being an integer not less than 1; marking the row starting position and the row ending position of each of the workspaces in the map; selecting the operation area currently required to be picked in the map, and writing the operation starting position, the operation ending position and the unloading and sub-packaging point position in the operation area into the task control module of the fruit and vegetable transport vehicle, the operation starting position coincides with the row starting position of the first workspace, and the operation ending position coincides with the row ending position of the Nth workspace; starting the autonomous navigation and positioning function module, the gesture The recognition module and the task control module; the fruit and vegetable transport vehicle stops moving after moving from the unloading and sub-packing point to the operation starting position of the N-1th workspace, and at the same time, the camera of the fruit and vegetable transport vehicle starts to detect the gesture information of the picking staff; after detecting that the picking staff makes the first target gesture, the transport vehicle autonomously follows the picking staff to carry out picking and transportation operations in the N-1th workspace; after the picking process in the N-1th workspace has not reached the end point and the picking basket is full, it is detected that the picking staff makes the second target gesture, and the fruit and vegetable transport vehicle returns from the full basket to the workspace. The vehicle exits the N-1th workspace by returning to the point where the goods are unloaded and packaged. After unloading, the vehicle recognizes the third target gesture and autonomously navigates back to the full basket return point of the N-1th workspace to continue the picking and transportation work in the N-1th workspace. Alternatively, the vehicle moves to the end point of the operation in the N-1th workspace, and the vehicle autonomously returns to the unloading and packaging point. After the vehicle is located at the unloading and packaging point and recognizes the third target gesture, it autonomously navigates to the starting point of the operation in the Nth workspace.

[0005] Optionally, when constructing the gesture recognition algorithm model, a YOLO v5 algorithm based on a PyTorch framework is used for gesture recognition pre-training, and the LD ConGR dataset is used for training.

[0006] Optionally, when constructing the map, the chassis movement of the fruit and vegetable transport vehicle is controlled by a ROS keyboard node, and a two-dimensional grid map is constructed using laser SLAM technology.

[0007] Optionally, when the fruit and vegetable transport vehicle autonomously follows the picking staff in the workspace, the fruit and vegetable transport vehicle and the picking staff maintain a preset safe distance through laser radar ranging in front of the fruit and vegetable transport vehicle.

[0008] Optionally, when the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle located at the unloading and loading point completes unloading.

[0009] Optionally, when the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle located at the unloading and loading point is not greater than a preset weight value.

[0010] Optionally, the operation starting point is far away from the unloading and loading point, and the operation ending point is close to the unloading and loading point.

[0011] A second aspect of the present invention provides a fruit and vegetable transport vehicle, comprising: a vehicle body; a fruit and vegetable storage basket, the fruit and vegetable storage basket being arranged on the vehicle body; a control system, the control system being arranged on the vehicle body, the control system executing any of the above-mentioned autonomous following and transportation control methods for a fruit and vegetable transport vehicle; and a camera, the camera being arranged on the vehicle body, the camera detecting gesture information of a picking staff member.

[0012] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor executes any one of the methods described above.

[0013] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes any of the above-mentioned methods.

[0014] According to the autonomous following and transportation control method of the fruit and vegetable transport vehicle according to the embodiment of the present invention, by integrating autonomous navigation technology and combining gesture recognition technology based on deep learning target detection algorithm, the autonomous following and transportation function of the fruit and vegetable transport vehicle facilities in the workspace can be realized. During the picking operation in the workspace, the pickers can control the functions of the fruit and vegetable transport vehicle to autonomously follow in the row through several different gestures; not only does it make the interaction between the pickers and the fruit and vegetable transport vehicle simpler and more intelligent, but it can also free the pickers from the additional transport vehicle control, thereby improving the overall picking efficiency.

[0015] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] Figure 1 is a distribution diagram of crop planting rows according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a fruit and vegetable transport vehicle according to an embodiment of the present invention; Figure 3 It is a flow chart of constructing a model of an autonomous following and transporting control system for a facility fruit and vegetable transport vehicle according to an embodiment of the present invention; Figure 4 This is an operation flow chart of an autonomous following and transportation control system for a facility fruit and vegetable transport vehicle according to an embodiment of the present invention; Figure 5 It is a flow chart of a facility fruit and vegetable transport vehicle and a picking worker maintaining a safe distance according to an embodiment of the present invention; Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0018] Reference numerals: Crop planting column 10; Crop planting rows 20; Workspace 30; row start position 31; row end position 32; Operation starting point position 41; operation end point position 42; unloading and loading point position 43; Fruit and vegetable transport vehicle 50; Car body 51; Fruit and vegetable storage basket 52; Camera 53; Lidar 54; Electronic device 200; Processor 201; Memory 202; operating system 2021; application program 2022; Network interface 203; Input device 204; Hard disk 205; Display device 206. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0024] The following first describes in detail the autonomous following transportation control method of the fruit and vegetable transport vehicle 50 according to an embodiment of the present invention with reference to the accompanying drawings.

[0025] like Figures 1 to 5 As shown, the autonomous following transportation control method of the fruit and vegetable transport vehicle 50 according to an embodiment of the present invention includes the following steps: The first target gesture, the second target gesture and the third target gesture are selected for gesture recognition training to obtain a gesture recognition algorithm model, and the gesture recognition algorithm model is deployed in the control system of the fruit and vegetable transport vehicle 50. By deploying the gesture recognition algorithm model, the camera 53 of the fruit and vegetable transport vehicle 50 can recognize the gestures of the fruit and vegetable pickers.

[0026] A map is constructed, and the map includes information about the operation area and the crop planting rows 20. The operation area includes at least one crop planting column 10, and the crop planting column 10 includes multiple crop planting rows 20. The space between two adjacent crop planting rows 20 is used as a work space 30, and can also be defined as a ridge between two crop planting rows 20. The work space 30 can be divided into the first work space 30...the Nth work space 30, where N is an integer not less than 1. For example, the operation area includes two crop planting columns 10 distributed on the left and right. The crop planting column 10 on the right is taken as an example. The crop planting column 10 includes 10 crop planting rows 20 distributed in the front and back directions. The crop planting row 20 in front is taken as the first crop planting row 20, and there is a first work space 30 between the first crop planting row 20 and the second crop planting row 20. When the fruit and vegetable transport vehicle 50 is located in the first work space 30, the pickers can pick and advance the first crop planting row 20 and the second crop planting row 20 at the same time. A second working space 30 is set between the second crop planting row 20 and the third crop planting row 20. When the fruit and vegetable transport vehicle 50 is located in the second working space 30, the picking worker can pick and advance the second crop planting row 20 and the third crop planting row 20 at the same time. Similarly, in the backward direction, the third working space 30, the fourth working space 30, the fifth working space 30, the sixth working space 30 ... the ninth working space 30 are sequentially set.

[0027] The row start position 31 and the row end position 32 of each workspace 30 are marked in the map. For example, the row start position 31 and the row end position 32 between each crop planting row 20 are determined and marked in the two-dimensional grid map information. For another example, the workspace 30 extends in the left-right direction, and the row start position 31 is set at the left end of the workspace 30, and the row end position 32 is set at the right end.

[0028] Select the current work area where picking is required in the map, write the work start position 41, work end position 42 and unloading and sub-packing point position 43 in the work area into the task control module of the fruit and vegetable transport vehicle 50, the work start position 41 coincides with the row start position 31 of the first workspace 30, and the work end position 42 coincides with the row end position 32 of the Nth workspace 30. For example, the work start position 41 coincides with the row start position 31 of the first workspace 30, and the work end position 42 coincides with the row end position 32 of the ninth workspace 30.

[0029] Start the autonomous navigation and positioning function module, gesture recognition module and task control module of the fruit and vegetable transport vehicle 50. The autonomous navigation and positioning function module here can realize autonomous navigation, for example, navigating from the unloading and sub-packing point position 43 to the operation starting point position 41, and navigating from any position of the workspace 30 to the unloading and sub-packing point position 43.

[0030] The fruit and vegetable transport vehicle 50 moves from the unloading and subpacking point position 43 to the operation starting position 41 of the N-1th workspace 30 and then stops moving, and at the same time, the camera 53 of the fruit and vegetable transport vehicle 50 starts to detect the gesture information of the picking staff. For example, after the autonomous navigation and positioning function module, gesture recognition module and task control module of the fruit and vegetable transport vehicle 50 are started, the fruit and vegetable transport vehicle 50 starts from the unloading and subpacking point position 43, moves to the row starting position 31 of the first workspace 30 and then stops moving, and at the same time, the front camera 53 starts to detect the gesture information of the picking staff.

[0031] After detecting that the picking staff makes the first target gesture, the transport vehicle autonomously follows the picking staff to carry out the picking and transporting operation in the N-1th workspace 30. For example, after the picking staff is in place and the camera 53 in front of the transport vehicle successfully detects and recognizes the first target gesture of the picking staff, the transport vehicle begins to autonomously follow the picking staff to carry out the picking and transporting operation.

[0032] During the picking process of the N-1th workspace 30, if the end point has not been reached and the picking basket is full, the picking staff is detected to make a second target gesture, and the fruit and vegetable transport vehicle 50 exits the N-1th workspace 30 from the basket full return point, returns to the unloading and sub-packaging site, and after completing the unloading, after recognizing the third target gesture, autonomously navigates back to the basket full return point of the N-1th workspace 30, and continues the picking and transportation work of the N-1th workspace. Alternatively, the fruit and vegetable transport vehicle 50 moves to the operation end position 42 of the N-1th workspace 30, and the fruit and vegetable transport vehicle 50 autonomously returns to the unloading and sub-packaging point position 43. In addition, after the fruit and vegetable transport vehicle 50 at the unloading and sub-packaging point position 43 recognizes the third target gesture, it autonomously navigates to the operation starting position 41 of the Nth workspace 30. For example, after the staff completes the unloading, the camera 53 in front of the transport vehicle successfully detects and recognizes the third target gesture of the unloading staff, and the fruit and vegetable transport vehicle 50 autonomously navigates to the row starting position 31 of the next workspace 30. For example, during the current row of picking, when the camera 53 in front of the fruit and vegetable transport vehicle 50 detects that the picking staff makes the second target gesture, the fruit and vegetable transport vehicle 50 automatically exits the current workspace 30 along the original path and returns to the unloading and sub-packaging point position 43 determined before the start of the operation, waiting for the staff to complete the unloading. For another example, when the fruit and vegetable transport vehicle 50 moves to the operation end position 42 corresponding to the current workspace 30, the fruit and vegetable transport vehicle 50 automatically returns to the designated unloading and sub-packaging point position 43, waiting for the staff to complete the unloading.

[0033] It can be seen that in this embodiment, by integrating autonomous navigation technology and combining gesture recognition technology based on deep learning target detection algorithm, the autonomous following transportation function of the fruit and vegetable transport vehicle 50 in the workspace 30 can be realized. During the picking operation in the workspace 30, the pickers can control the autonomous following of the fruit and vegetable transport vehicle in the row through several different gestures; not only does it make the interaction between the pickers and the fruit and vegetable transport vehicle simpler and more intelligent, but it can also free the pickers from the additional transport vehicle control, thereby improving the overall picking efficiency.

[0034] According to one embodiment of the present invention, Figure 3 As shown in the figure, when constructing the gesture recognition algorithm model, the YOLO v5 algorithm based on the PyTorch framework is used for gesture recognition pre-training, and the training uses the LD ConGR dataset. For example, the YOLO v5 algorithm based on the PyTorch framework is used for gesture recognition pre-training, and the training uses the LD ConGR dataset released by the Institute of Software, Chinese Academy of Sciences in 2022. Three gestures with obvious features and large differences are selected as the first target gesture, the second target gesture, and the third target gesture. After the training is completed, the gesture recognition algorithm model is obtained and deployed on the industrial computer of the fruit and vegetable transport vehicle with autonomous following function.

[0035] In some specific embodiments of the present invention, when constructing a map, the chassis movement of the fruit and vegetable transport vehicle 50 is controlled by the ROS keyboard node, and the laser SLAM technology is used to construct a two-dimensional grid map. That is, the chassis movement of the transport vehicle is controlled by the ROS keyboard node, and the laser SLAM technology is used to construct a two-dimensional grid map in the facility. The constructed map should include information such as various operating areas and crop planting rows that the robot can pass.

[0036] According to one embodiment of the present invention, when the fruit and vegetable transport vehicle 50 autonomously follows the picking staff in the workspace 30, the laser radar 54 in front of the fruit and vegetable transport vehicle 50 measures the distance, so that the fruit and vegetable transport vehicle 50 and the picking staff maintain a preset safe distance. In other words, when the fruit and vegetable transport vehicle 50 autonomously follows the picking staff in the workspace 30, the laser radar 54 in front of the fruit and vegetable transport vehicle 50 measures the distance, so that the fruit and vegetable transport vehicle 50 and the picking staff maintain a safe distance. For example, when the distance between the fruit and vegetable transport vehicle 50 and the picking staff is less than the safe distance, the fruit and vegetable transport vehicle 50 stops moving; when the distance between the picking staff and the fruit and vegetable transport vehicle 50 is greater than the safe distance, the fruit and vegetable transport vehicle 50 automatically follows the picking staff. In this embodiment, the laser radar 54 is used to achieve a more humane and intelligent interactive control between the picking staff and the fruit and vegetable transport vehicle 50 without increasing the hardware cost.

[0037] In addition, when the prior art uses infrared sensors, after the fruits and vegetables are stored between rows, they need to be transported to the designated packaging location in the greenhouse in conjunction with the sensor systems such as electromagnetic wires, signal generators, and line patrol systems that have been laid in advance. This lacks interactive flexibility and the overall maintenance cost of the system is relatively high. In this embodiment, by using a laser radar 54, the interactive flexibility can be improved and the overall maintenance cost can be reduced.

[0038] In some specific embodiments of the present invention, when the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle 50 at the unloading and sub-packing point position 43 completes unloading. That is, during the picking process in the current workspace 30, when the front camera 53 of the fruit and vegetable transport vehicle 50 detects that the picking staff makes the second target gesture, the fruit and vegetable transport vehicle 50 automatically exits the current workspace 30 along the original path, and returns to the unloading and sub-packing point position 43 determined before the start of the operation, waiting for the staff to complete unloading.

[0039] According to an embodiment of the present invention, when the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle 50 located at the unloading and loading point 43 is not greater than a preset weight value, which can assist in determining whether the unloading is completed.

[0040] In some specific embodiments of the present invention, Figure 1 As shown, the operation starting point 41 is far away from the unloading and loading point 43, and the operation end point 42 is close to the unloading and loading point 43. For example, the operation area includes two crop planting rows 10 distributed on the left and right, and the unloading and loading point 43 is located between the two crop planting rows 10. Taking the right crop planting row 10 as an example, the crop planting row 10 includes 10 crop planting rows 20 distributed along the front-to-back direction. The row starting point 31 of the first workspace 30 is located on the left side of the workspace 30, and the row end point 32 of the ninth workspace 30 is located on the right side of the workspace 30. The unloading and loading point 43 is located on the left side of the row starting point 31 of the ninth workspace 30, and is closest to the row starting point 31 of the ninth workspace 30 relative to the nine workspaces 30. In this embodiment, by moving the operation starting point 41 away from the unloading and loading point 43 and the operation ending point 42 close to the unloading and loading point 43, it is convenient to control the picking order so that the staff does not need to walk a long distance to reach the unloading and loading point 43 after completing the picking of the last crop planting row 20.

[0041] The autonomous following transportation control method of the fruit and vegetable transport vehicle 50 according to the embodiment of the present invention is described in detail below in conjunction with specific embodiments.

[0042] Step S1, implement the YOLO v5 algorithm based on the PyTorch framework for gesture recognition pre-training. The training uses the LD ConGR dataset released by the Institute of Software, Chinese Academy of Sciences in 2022. Three gestures with obvious features and large differences are selected from the dataset, defined as the first target gesture, the second target gesture, and the third target gesture. In order to be more in line with the actual operation scene, ten picking workers were selected to collect 50 first target gestures, 50 second target gestures, and 50 third target gestures respectively during the actual picking operation, a total of 1500 picture data, which are used as supplementary data sets for model training. After training, the gesture recognition algorithm model is obtained and deployed with the 50 industrial computers in the autonomously following fruit and vegetable transport vehicle.

[0043] Step S2, control the fruit and vegetable transport vehicle 50 to move in the ridge path through the ROS keyboard node, carrying the 3D laser radar 54, controller, IMU and photoelectric encoder, etc., and use the scanning data of the laser radar and the fused odometer data of the IMU and encoder to use the Gmapping algorithm to establish a two-dimensional grid map of the picking operation area in the facility, and save the map; the constructed map should include information such as various operation areas and crop planting rows that the robot can pass.

[0044] Step S3: Use the rviz plug-in to open the two-dimensional grid map, determine and record the row starting position 31 information of the working space 30 between each crop planting row in the map P line_si ( x line_si , y line_si ) and the row end position 32 information, P line_ei ( x line_ei , y line_ei ),in i=1~n 1, n 1 represents the total number of rows that need to be picked; 43 information on the location of the unloading and sub-loading point P o ( x o , y o ),in j=1~n 2, n 2 represents the total number of rows and ridges that need to be picked, that is, the total number of workspaces 30; the information of the starting position 41 of the operation P sj ( x sj , y sj )、Operation end position 42 information P ej( x ej , y ei ).

[0045] Step S4: determine the current picking operation area in the map, and write the operation starting point position 41 information in the area determined in step S3 into the task control module. P sj ( x sj , y sj )、Operation end position 42 information P ej ( x ej , y ei )、Unloading and loading point location 43 information P o ( x o , y o ).

[0046] Step S5, start each node of the autonomous navigation and positioning function module, the gesture recognition module and the task control module in the ROS robot operating system, and control the fruit and vegetable transport vehicle 50 to autonomously navigate from the unloading and sub-packaging point position 43 to the row starting position 31 of the working space 30 between the first crop planting row 20 and the second crop planting row 20.

[0047] Step S6, according to the current positioning information of the fruit and vegetable transport vehicle 50 released in real time by the autonomous navigation and positioning function module, after the task control module motion detects that the fruit and vegetable transport vehicle 50 has reached the row starting position 31 of the workspace 30, the fruit and vegetable transport vehicle 50 is controlled to stop at the position and wait, and at the same time, the task control module turns on the front camera 53 to start collecting gesture image information of the picking staff.

[0048] Step S7, after the picking staff are in place, the fruit and vegetable transport vehicle 50 uses the gesture recognition algorithm model of the gesture recognition module to process and identify the staff gesture image data successfully collected by the front camera 53. After stably and successfully identifying the first target gesture, the autonomous navigation and positioning function module sends an autonomous following instruction to control the fruit and vegetable transport vehicle 50 to autonomously follow the picking staff to carry out picking and transportation operations.

[0049] Step S8: During the operation of the fruit and vegetable transport vehicle 50 autonomously following the picking staff in the workspace 30, the task control module automatically controls the fruit and vegetable transport vehicle 50 and the picking staff to always maintain a preset safe and convenient distance range for the staff to place fruits and vegetables according to the real-time distance between the fruit and vegetable transport vehicle 50 and the picking staff fed back by the laser radar 54 in front of the fruit and vegetable transport vehicle 50. D ( d min ，d max ). When the distance between the fruit and vegetable transport vehicle 50 and the picking staff is less than the lower limit of the safe distance d min When the distance between the picking staff and the fruit and vegetable transport vehicle 50 is greater than the upper limit of the safety distance, the autonomous navigation and positioning function module issues a stop motion instruction to control the fruit and vegetable transport vehicle 50 to stop moving; d max When the autonomous navigation and positioning function module sends an autonomous following instruction to control the fruit and vegetable transport vehicle 50 to autonomously follow the picking staff.

[0050] Step S9: During the picking process of the current workspace 30, when the storage box of the fruit and vegetable transport vehicle 50 is full, the picking staff makes a target second target gesture to the camera 53 in front of the fruit and vegetable transport vehicle 50. After the gesture recognition module detects the target second target gesture, the task control module controls the fruit and vegetable transport vehicle 50 to autonomously exit the current workspace 30 from the basket full return point along the original path, and autonomously return to the operation packaging position. P o ( x o , y o ), waiting for the staff to complete the unloading.

[0051] Step S10: After unloading is completed, the third target gesture is recognized, and the fruit and vegetable transport vehicle 50 directly returns to the basket full return point of step S9, that is, the two conditions of step S9 and step S10 (the second target gesture and the third target gesture) are met at the same time, and the fruit and vegetable transport vehicle 50 directly returns to the basket full return point of step S9 to continue the picking and transportation work.

[0052] Step S9', the current workspace 30 has been completed, and the task control module detects that the fruit and vegetable transport vehicle 50 has reached the end position 32 of the current workspace 30, and the task control module controls the fruit and vegetable transport vehicle 50 to autonomously exit the current workspace 30 along the original path and autonomously return to the operation packaging position P o ( x o , y o ), waiting for the staff to complete the unloading.

[0053] Step S10', at the subpackaging position, after the staff completes unloading, the staff makes a target third target gesture to the camera 53 in front of the fruit and vegetable transport vehicle 50. After the gesture recognition module detects the target third target gesture, the task control module controls the fruit and vegetable transport vehicle 50, and the autonomous navigation and positioning function module controls the fruit and vegetable transport vehicle 50 to autonomously navigate from the subpackaging point to the row starting position 31 of the next workspace 30.

[0054] Repeat the above steps until all the selected target operation areas are harvested.

[0055] In summary, in the embodiment of the present invention, the gesture recognition algorithm in the deep learning technology is combined with the multi-sensor fusion navigation technology. For example, the laser radar 54 and the IMU (inertial measurement unit) and encoder and other sensor data used in step S2 can realize a more humane and intelligent interactive control between the pickers and the fruit and vegetable transport vehicle 50 without increasing the hardware cost. It not only overcomes the difficulties that these existing technologies are difficult to solve in actual engineering applications, but also frees the pickers from the control of the additional fruit and vegetable transport vehicle 50, improves the overall picking work efficiency, and provides a reference for the research and development and industrial application of intelligent harvesting equipment for facility fruits and vegetables.

[0056] The present invention further provides a fruit and vegetable transport vehicle 50 , which includes a vehicle body 51 , a fruit and vegetable storage basket 52 , a control system and a camera 53 .

[0057] Specifically, the fruit and vegetable storage basket 52 is arranged on the vehicle body 51, the control system is arranged on the vehicle body 51, the control system executes the autonomous following and transportation control method of the fruit and vegetable transport vehicle 50 described in any of the above embodiments, and the camera 53 is arranged on the vehicle body 51, and the camera 53 detects the gesture information of the picking staff.

[0058] Since the autonomous following and transportation control method of the fruit and vegetable transport vehicle 50 in the embodiment of the present invention can realize the autonomous following and transportation function of the fruit and vegetable transport vehicle 50 in the working space 30, the fruit and vegetable transport vehicle 50 in the embodiment of the present invention also has the advantages of intelligence and ease of use, which will not be elaborated here.

[0059] The present invention further provides an electronic device 200, comprising: a processor 201 and a memory 202, wherein the memory 202 stores computer program instructions, wherein when the computer program instructions are executed by the processor 201, the processor 201 executes the steps of the method in the above embodiment.

[0060] Furthermore, if Figure 6 As shown, the electronic device 200 further includes a network interface 203 , an input device 204 , a hard disk 205 , and a display device 206 .

[0061] The above-mentioned interfaces and devices can be interconnected through a bus architecture. The bus architecture can include any number of interconnected buses and bridges. Specifically, one or more central processing units 201 (CPUs) represented by the processor 201 and various circuits of one or more memories 202 represented by the memory 202 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art, so they will not be described in detail herein.

[0062] The network interface 203 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, and save it in the hard disk 205.

[0063] The input device 204 can receive various instructions input by the operator and send them to the processor 201 for execution. The input device 204 can include a keyboard or a pointing device (for example, a mouse, a trackball, a touch pad or a touch screen, etc.).

[0064] The display device 206 can display the result obtained by the processor 201 executing the instruction.

[0065] The memory 202 is used to store programs and data necessary for the operation of the operating system 2021, as well as data such as intermediate results during the calculation process of the processor 201.

[0066] It is understood that the memory 202 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. The memory 202 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory 202.

[0067] In some implementations, the memory 202 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 2021 and application programs 2022 .

[0068] The operating system 2021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 2022 includes various application programs 2022, such as a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 2022.

[0069] The processor 201 executes the steps of the method according to the above embodiment when calling and executing the application 2022 and data stored in the memory 202, specifically, the program or instructions stored in the application 2022.

[0070] The method disclosed in the above embodiment of the present invention can be applied to the processor 201, or implemented by the processor 201. The processor 201 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 201 or the instruction in the form of software. The above processor 201 can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general processor can be a microprocessor or the processor 201 can also be any conventional processor 201, etc. The steps of the method disclosed in conjunction with the embodiment of the present invention can be directly embodied as a hardware decoding processor to be executed, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 202, and the processor 201 reads the information in the memory 202 and completes the steps of the above method in combination with its hardware.

[0071] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present invention or a combination thereof.

[0072] For software implementation, the technology herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions herein. The software code can be stored in the memory 202 and executed by the processor 201. The memory 202 can be implemented in the processor 201 or outside the processor 201.

[0073] Specifically, the processor 201 is also used to read the computer program and execute the following steps: the method predicts and outputs the answer to the question asked by the user.

[0074] The present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 201, the processor 201 executes the steps of the method in the above embodiment.

[0075] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0076] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0077] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0078] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. An autonomous following and transporting control method for a fruit and vegetable transport vehicle, characterized in that: The steps include: Selecting a first target gesture, a second target gesture, and a third target gesture for gesture recognition training to obtain a gesture recognition algorithm model, and deploying the gesture recognition algorithm model in a control system of a fruit and vegetable transport vehicle; Constructing a map, the map including information of an operation area and a crop planting row, the operation area including at least one crop planting row, the crop planting row including a plurality of crop planting rows, the space between two adjacent crop planting rows as a workspace, the workspace being divided into a first workspace ... an Nth workspace, where N is an integer not less than 1; Marking the starting point and the ending point of each workspace on the map; The operation area currently required for picking is selected in the map, and the operation starting position, operation end position and unloading and sub-packaging point position in the operation area are written into the task control module of the fruit and vegetable transport vehicle, wherein the operation starting position coincides with the row starting position of the first workspace, and the operation end position coincides with the row end position of the Nth workspace; Starting the autonomous navigation and positioning function module, gesture recognition module and task control module of the fruit and vegetable transport vehicle; The fruit and vegetable transport vehicle moves from the unloading and sub-loading point to the operation starting point of the N-1th workspace and then stops moving, and at the same time, the camera of the fruit and vegetable transport vehicle starts to detect the gesture information of the picking staff; After detecting that the picking staff makes the first target gesture, the transport vehicle autonomously follows the picking staff to carry out picking and transport operations in the N-1th workspace; During the picking process in the N-1th workspace, if the end point has not been reached and the picking basket is full, it is detected that the picking staff has made the second target gesture, and the fruit and vegetable transport vehicle exits the N-1th workspace from the basket full return point and returns to the unloading and packaging site. After unloading is completed, the fruit and vegetable transport vehicle recognizes the third target gesture and autonomously navigates back to the basket full return point of the N-1th workspace to continue the picking and transportation work in the N-1th workspace; or, the fruit and vegetable transport vehicle moves to the operation end position of the N-1th workspace, and the fruit and vegetable transport vehicle autonomously returns to the unloading and packaging point. After the fruit and vegetable transport vehicle at the unloading and packaging point recognizes the third target gesture, it autonomously navigates to the operation starting position of the Nth workspace.

2. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1 is characterized in that: When constructing the gesture recognition algorithm model, the YOLO v5 algorithm based on the PyTorch framework is used for gesture recognition pre-training, and the LD ConGR dataset is used for training.

3. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1 is characterized in that: When constructing the map, the chassis movement of the fruit and vegetable transport vehicle is controlled by the ROS keyboard node, and the laser SLAM technology is used to construct a two-dimensional grid map.

4. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1 is characterized in that: When the fruit and vegetable transport vehicle autonomously follows the picking staff in the work space, the laser radar in front of the fruit and vegetable transport vehicle measures the distance, so that the fruit and vegetable transport vehicle and the picking staff maintain a preset safe distance.

5. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1 is characterized in that: When the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle located at the unloading and dispensing point completes unloading.

6. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1, characterized in that: When the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle located at the unloading and dispensing point is not greater than a preset weight value.

7. The autonomous following and transporting control method of a fruit and vegetable transport vehicle according to claim 1, characterized in that: The operation starting point is far away from the unloading and loading point, and the operation ending point is close to the unloading and loading point.

8. A fruit and vegetable transport vehicle, characterized in that: include: Vehicle body; A fruit and vegetable storage basket, the fruit and vegetable storage basket being arranged on the vehicle body; A control system, wherein the control system is arranged on the vehicle body, and the control system executes the autonomous following and transporting control method of the fruit and vegetable transport vehicle according to any one of claims 1 to 7; A camera is arranged on the vehicle body, and detects gesture information of the picking staff.

9. An electronic device, characterized in that: include: A processor and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images