Fruit and vegetable transport vehicle and following transfer control method, electronic device and storage medium

Abstract

The application discloses a fruit and vegetable transport vehicle, a following and transfer control method, an electronic device and a storage medium. The autonomous following and transfer control method of the fruit and vegetable transport vehicle comprises the following steps: gesture recognition training is performed; a map is constructed; after a first target gesture is detected, the transport vehicle follows a picking staff in an N-1th working space; during picking, after a second target gesture made by the picking staff is detected, the fruit and vegetable transport vehicle exits the N-1th working space and returns to a unloading and sub-packaging position, after unloading is completed, a third target gesture is detected, and the fruit and vegetable transport vehicle returns to a basket full return point; or, the fruit and vegetable transport vehicle moves to a work end position of the N-1th working space, and the fruit and vegetable transport vehicle returns to the unloading and sub-packaging point position; after the fruit and vegetable transport vehicle located at the unloading and sub-packaging point position recognizes the third target gesture, the fruit and vegetable transport vehicle returns to a work start position of the Nth working space. The control method can meet the intelligent picking and transport requirements of fruits and vegetables in a multi-span glass greenhouse.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control of agricultural robots, and more particularly to a fruit and vegetable transport vehicle, a following and transfer control method, an electronic device and a storage medium. BACKGROUND

[0002] In the existing fruit and vegetable planting area, such as a multi-span glass greenhouse, the picking and transporting of fruits and vegetables is still manually performed by entering the rows to pick the fruits and vegetables and placing them in a storage device carried by the picker, and then the fruits and vegetables are manually transported or transported by a remote control track transport vehicle to a designated loading location. Moreover, the track transport vehicle needs to be matched with a standard track laid in the fruit and vegetable planting rows for driving the track lifting operation vehicle, and the laying of the track makes the picking and transporting operation conditions more complex, increases the labor intensity and reduces the efficiency. In addition, the remote control track transport vehicle is operated manually, which is complex and cannot meet the intelligent picking and transporting requirements of fruits and vegetables in the multi-span glass greenhouse. SUMMARY

[0003] The present application aims to provide a new technical solution for a fruit and vegetable transport vehicle, a following and transfer control method, an electronic device and a storage medium, which can solve at least one of the technical problems of complex picking and transporting operation conditions, increased labor intensity and reduced efficiency in the prior art.

[0004] In a first aspect, the present application provides an autonomous following transfer control method for a fruit and vegetable transport vehicle, comprising the following steps: selecting a first target gesture, a second target gesture and a third target gesture for gesture recognition training to obtain a gesture recognition algorithm model, and deploying the gesture recognition algorithm model to a control system of the fruit and vegetable transport vehicle; constructing a map, wherein the map comprises information of a work area and crop planting rows, the work area comprises at least one crop planting column, the crop planting column comprises a plurality of crop planting rows, a space between two adjacent crop planting rows is used as a work space, the work space is divided into a first work space, a second work space,..., and an Nth work space, N is an integer not less than 1; marking a row start position and a row end position of each work space in the map; selecting a work area currently requiring picking in the map, and writing a work start position, a work end position and an unloading and sub-packaging point position in the work area to a task control module of the fruit and vegetable transport vehicle, wherein the work start position coincides with the row start position of the first work space, and the work end position coincides with the row end position of the Nth work space; starting an autonomous navigation positioning function module, a gesture recognition module and a task control module of the fruit and vegetable transport vehicle; the fruit and vegetable transport vehicle stops moving after moving from the unloading and sub-packaging point position to the work start position of the Nth-1 work space, and a camera of the fruit and vegetable transport vehicle starts detecting gesture information of a picking worker; after detecting that the picking worker makes the first target gesture, the transport vehicle autonomously follows the picking worker to carry out picking and transport work in the Nth-1 work space; after detecting that the picking worker makes the second target gesture during picking in the Nth-1 work space, the fruit and vegetable transport vehicle exits the Nth-1 work space from a full basket return point and returns to the unloading and sub-packaging point, and after completing unloading, the fruit and vegetable transport vehicle autonomously navigates to the full basket return point of the Nth-1 work space to continue picking and transport work in the Nth-1 work space; or, the fruit and vegetable transport vehicle moves to the work end position of the Nth-1 work space, and the fruit and vegetable transport vehicle autonomously returns to the unloading and sub-packaging point position, and after detecting the third target gesture, the fruit and vegetable transport vehicle autonomously navigates to the work start position of the Nth work space.

[0005] Optionally, when constructing the gesture recognition algorithm model, a YOLO v5 algorithm based on a PyTorch framework is used for gesture recognition pre-training, and an LD ConGR data set is used for training.

[0006] Optionally, when constructing the map, a chassis of the fruit and vegetable transport vehicle is controlled through a ROS keyboard node, and a two-dimensional grid map is constructed by using a laser SLAM technology.

[0007] Optionally, during the autonomous following of the picking worker by the fruit and vegetable transport vehicle in the working space, the fruit and vegetable transport vehicle keeps a preset safe distance from the picking worker by laser radar ranging in front of the fruit and vegetable transport vehicle.

[0008] Optionally, when the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle located at the unloading and sub-packaging point position completes unloading.

[0009] Optionally, when the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle located at the unloading and sub-packaging point position is not greater than a preset weight value.

[0010] Optionally, the work starting point position is away from the unloading and sub-packaging point position, and the work ending point position is close to the unloading and sub-packaging point position.

[0011] In a second aspect, the present application provides a fruit and vegetable transport vehicle, comprising: a vehicle body; a fruit and vegetable storage basket arranged in the vehicle body; a control system arranged in the vehicle body, the control system performing the autonomous following and transfer control method of the fruit and vegetable transport vehicle described in any of the preceding aspects; and a camera arranged in the vehicle body, the camera detecting gesture information of a picking worker.

[0012] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory having computer program instructions stored therein, wherein when the computer program instructions are executed by the processor, the processor executes the method described in any of the preceding aspects.

[0013] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to execute the method described in any of the preceding aspects.

[0014] The autonomous following and transfer control method of the fruit and vegetable transport vehicle according to the embodiments of the present application can realize the autonomous following and transfer function of the fruit and vegetable transport vehicle facility in the working space by fusing autonomous navigation technology and gesture recognition technology realized by a deep learning target detection algorithm. The picking worker can control the fruit and vegetable transport vehicle to follow autonomously in the line by several different gestures during the picking operation in the working space. This not only makes the interaction between the picking worker and the fruit and vegetable transport vehicle more simple and intelligent, but also liberates the picking worker from additional transport vehicle control, thereby improving the overall picking efficiency.

[0015] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0017] Figure 1 is a schematic diagram of distribution of crop planting rows of an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of structure of a fruit and vegetable transport vehicle of an embodiment of the present application;

[0019] Figure 3 is a flow chart of model construction of a facility fruit and vegetable transport vehicle autonomous following transfer control system of an embodiment of the present application;

[0020] Figure 4 is a flow chart of operation of a facility fruit and vegetable transport vehicle autonomous following transfer control system of an embodiment of the present application;

[0021] Figure 5 is a flow chart of maintaining a safe distance by a facility fruit and vegetable transport vehicle from a picking worker of an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of an electronic device of an embodiment of the present application.

[0023] REFERENCE NUMERALS:

[0024] crop planting row 10;

[0025] crop planting row 20;

[0026] work space 30; row start position 31; row end position 32;

[0027] work start position 41; work end position 42; unloading and sorting point position 43;

[0028] fruit and vegetable transport vehicle 50;

[0029] vehicle body 51;

[0030] fruit and vegetable storage basket 52;

[0031] camera 53;

[0032] laser radar 54;

[0033] electronic device 200;

[0034] processor 201;

[0035] memory 202; operating system 2021; application program 2022;

[0036] a network interface 203;

[0037] an input device 204;

[0038] a hard disk 205;

[0039] a display device 206. DETAILED DESCRIPTION

[0040] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in these embodiments are not limiting to the scope of the present application unless specifically stated otherwise.

[0041] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application its application or uses.

[0042] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the specification.

[0043] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the exemplary embodiments can have different values.

[0044] It should be noted that like references and characters herein relate to like items throughout the figures, and once an item is defined in one figure, it need not be discussed further in subsequent figures.

[0045] The autonomous following transfer control method of the fruit and vegetable transport vehicle 50 according to the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0046] As shown in the drawings, the autonomous following transfer control method of the fruit and vegetable transport vehicle 50 according to the embodiments of the present application includes the following steps: Figures 1 to 5 The first target gesture, the second target gesture, and the third target gesture are selected for gesture recognition training to obtain a gesture recognition algorithm model, and the gesture recognition algorithm model is deployed in the control system of the fruit and vegetable transport vehicle 50. By deploying the gesture recognition algorithm model, the camera 53 of the fruit and vegetable transport vehicle 50 can recognize the gestures of the fruit and vegetable picking personnel.

[0047]

[0048] ​The map is constructed, and the map includes information of a work area and crop planting rows 20. The work area includes at least one crop planting column 10, and the crop planting column 10 includes a plurality of crop planting rows 20. A space between two adjacent crop planting rows 20 is a work space 30, which can also be defined as a ridge way between two crop planting rows 20. The work space 30 can be divided into a first work space 30...an Nth work space 30, where N is an integer not less than 1. For example, the work area includes two crop planting columns 10 distributed on the left and right sides. Taking the right crop planting column 10 as an example, the crop planting column 10 includes 10 crop planting rows 20 distributed along the front-back direction. The first crop planting row 20 is in front, and the first crop planting row 20 and the second crop planting row 20 have a first work space 30. When the fruit and vegetable transport vehicle 50 is located in the first work space 30, the picker can simultaneously pick and push forward the first crop planting row 20 and the second crop planting row 20. The second crop planting row 20 and the third crop planting row 20 have a second work space 30. When the fruit and vegetable transport vehicle 50 is located in the second work space 30, the picker can simultaneously pick and push forward the second crop planting row 20 and the third crop planting row 20. Similarly, in the rear direction, a third work space 30, a fourth work space 30, a fifth work space 30, a sixth work space 30...a ninth work space 30 are sequentially arranged.

[0049] In the map, the row start position 31 and the row end position 32 of each work space 30 are marked. For example, in the two-dimensional grid map information, the row start position 31 and the row end position 32 between each crop planting row 20 are determined and marked. For another example, the work space 30 extends along the left-right direction, and the row start position 31 is arranged at the left end of the work space 30, and the row end position 32 is arranged at the right end of the work space 30.

[0050] In the map, a work area currently needed for picking is selected, and the work start position 41, the work end position 42, and the unloading and sub-packaging point position 43 in the work area are written in the task control module of the fruit and vegetable transport vehicle 50. The work start position 41 coincides with the row start position 31 of the first work space 30, and the work end position 42 coincides with the row end position 32 of the Nth work space 30. For example, the work start position 41 coincides with the row start position 31 of the first work space 30, and the work end position 42 coincides with the row end position 32 of the ninth work space 30.

[0051] The autonomous navigation positioning function module, the gesture recognition module, and the task control module of the fruit and vegetable transport vehicle 50 are started. The autonomous navigation positioning function module herein can realize autonomous navigation, such as navigating from the unloading and sub-packaging point position 43 to the work start position 41, and navigating from any position in the work space 30 to the unloading and sub-packaging point position 43.

[0052] The fruit and vegetable transport vehicle 50 stops moving after moving from the unloading and sorting point position 43 to the starting point position 41 of the N-1th working space 30, and the camera 53 of the fruit and vegetable transport vehicle 50 starts detecting the gesture information of the picking worker. For example, after starting the autonomous navigation positioning function module, the gesture recognition module and the task control module of the fruit and vegetable transport vehicle 50, the fruit and vegetable transport vehicle 50 starts from the unloading and sorting point position 43, moves to the starting point position 31 of the 1st working space 30, and stops moving, and the front camera 53 starts to detect the gesture information of the picking worker.

[0053] After detecting that the picking worker makes the first target gesture, the transport vehicle autonomously follows the picking worker to carry out the picking and transporting work in the N-1th working space 30. For example, after the picking worker is in place and the front camera 53 of the transport vehicle successfully detects and recognizes the first target gesture of the picking worker, the transport vehicle starts to autonomously follow the picking worker to carry out the picking and transporting work.

[0054] When the picking process in the N-1th working space 30 has not reached the end point and the basket is full, the second target gesture is detected, the fruit and vegetable transport vehicle 50 exits the N-1th working space 30 from the basket full return point and returns to the unloading and sorting point, after unloading is completed, the third target gesture is recognized, and the autonomous navigation returns to the basket full return point of the N-1th working space 30, and the picking and transporting work in the N-1th working space is continued. Alternatively, the fruit and vegetable transport vehicle 50 moves to the end point position 42 of the N-1th working space 30, and the fruit and vegetable transport vehicle 50 autonomously returns to the unloading and sorting point position 43. In addition, the fruit and vegetable transport vehicle 50 located at the unloading and sorting point position 43 recognizes the third target gesture and autonomously navigates to the starting point position 41 of the Nth working space 30. For example, after the worker completes unloading, the front camera 53 of the transport vehicle successfully detects and recognizes the third target gesture of the unloading worker, and the fruit and vegetable transport vehicle 50 autonomously navigates to the starting point position 31 of the next working space 30. For example, during the current picking process, when the fruit and vegetable transport vehicle 50 front camera 53 detects that the picking worker makes the second target gesture, the fruit and vegetable transport vehicle 50 autonomously exits the current working space 30 along the original path and returns to the unloading and sorting point position 43 determined before the start of the work, and waits for the worker to complete the unloading. For example, when the fruit and vegetable transport vehicle 50 moves to the end point position 42 of the current working space 30, the fruit and vegetable transport vehicle 50 autonomously returns to the designated unloading and sorting point position 43 and waits for the worker to complete the unloading.

[0055] It can be seen that, in the embodiment, by fusing the autonomous navigation technology, in combination with the gesture recognition technology realized based on the deep learning target detection algorithm, the autonomous following and transporting function of the fruit and vegetable transporting vehicle 50 in the picking of the work space 30 can be realized. The picking personnel can control the fruit and vegetable transporting vehicle in the row to follow autonomously and the like through several different gestures in the picking work process in the work space 30. Not only the interaction between the picking personnel and the fruit and vegetable transporting vehicle is more simplified and intelligent, but also the picking personnel is liberated from the additional transporting vehicle control, and the overall picking work efficiency is improved.

[0056] According to one embodiment of the present application, as shown in Figure 3 When the gesture recognition algorithm model is constructed, the YOLO v5 algorithm based on the PyTorch framework is used for gesture recognition pre-training, and the LD ConGR data set is used for training. For example, the YOLO v5 algorithm based on the PyTorch framework is used for gesture recognition pre-training, and the LD ConGR data set released by the Institute of Software, Chinese Academy of Sciences in 2022 is used for training. Three gestures with obvious features and large differences are selected as the first target gesture, the second target gesture, and the third target gesture. After training, the gesture recognition algorithm model is obtained, and it is deployed in the industrial computer of the fruit and vegetable transporting vehicle with autonomous following function.

[0057] In some specific embodiments of the present application, when the map is constructed, the chassis movement of the fruit and vegetable transporting vehicle 50 is controlled through the ROS keyboard node, and a two-dimensional grid map is constructed by using the laser SLAM technology. That is, the chassis movement of the transporting vehicle is controlled through the ROS keyboard node, and a two-dimensional grid map in the facility is constructed by using the laser SLAM technology. The constructed map should include information such as various work areas and crop planting rows that the robot can pass through.

[0058] According to one embodiment of the present application, during the autonomous following of the fruit and vegetable transporting vehicle 50 in the work space 30 by the picking personnel, the distance between the fruit and vegetable transporting vehicle 50 and the picking personnel is measured by the laser radar 54 in front of the fruit and vegetable transporting vehicle 50, so that the fruit and vegetable transporting vehicle 50 and the picking personnel maintain a preset safe distance. That is, during the autonomous following of the fruit and vegetable transporting vehicle 50 in the work space 30 by the picking personnel, the distance between the fruit and vegetable transporting vehicle 50 and the picking personnel is measured by the laser radar 54 in front of the fruit and vegetable transporting vehicle 50, so that the fruit and vegetable transporting vehicle 50 and the picking personnel maintain a safe distance. For example, when the distance between the fruit and vegetable transporting vehicle 50 and the picking personnel is less than the safe distance, the fruit and vegetable transporting vehicle 50 stops moving; when the distance between the fruit and vegetable transporting vehicle 50 and the picking personnel is greater than the safe distance, the fruit and vegetable transporting vehicle 50 automatically follows the picking personnel to move forward. In the embodiment, the laser radar 54 is used, which can realize more humanized and intelligent interaction control between the picking personnel and the fruit and vegetable transporting vehicle 50 without increasing the hardware cost.

[0059] In addition, when the prior art uses an infrared sensor, after the fruits and vegetables are stored between the rows, the fruits and vegetables are transported to the designated unloading point in the greenhouse, which requires the cooperation of the sensor system such as the electromagnetic wire, signal generator, and line tracking system laid in advance. The system lacks interactive flexibility and has relatively high overall maintenance costs. In the present embodiment, the use of the laser radar 54 can improve interactive flexibility and reduce overall maintenance costs.

[0060] In some embodiments of the present application, when the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle 50 located at the unloading and packaging point position 43 completes unloading. That is, during the picking process in the current work space 30, when the front camera 53 of the fruit and vegetable transport vehicle 50 detects that the picking worker makes the second target gesture, the fruit and vegetable transport vehicle 50 autonomously exits the current work space 30 along the original path and returns to the unloading and packaging point position 43 determined before the start of the work, and waits for the worker to complete the unloading.

[0061] According to an embodiment of the present application, when the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle 50 located at the unloading and packaging point position 43 is not greater than a preset weight value, which can assist in determining whether the unloading is completed.

[0062] In some embodiments of the present application, as shown in Figure 1 The work start point position 41 is away from the unloading and packaging point position 43, and the work end point position 42 is close to the unloading and packaging point position 43. For example, the work area includes two crop planting rows 10 distributed on the left and right, and the unloading and packaging point position 43 is located between the two crop planting rows 10. Taking the right crop planting row 10 as an example, the crop planting row 10 includes 10 crop planting rows 20 distributed along the front and back directions. The row start point position 31 of the first work space 30 is located on the left side of the work space 30, and the row end point position 32 of the ninth work space 30 is located on the right side of the work space 30. The unloading and packaging point position 43 is located on the left side of the row start point position 31 of the ninth work space 30, and is closest to the row start point position 31 of the ninth work space 30 relative to the nine work spaces 30. In the present embodiment, by arranging the work start point position 41 away from the unloading and packaging point position 43 and the work end point position 42 close to the unloading and packaging point position 43, the control of the picking sequence is facilitated, and the worker does not need to walk a long distance to reach the unloading and packaging point position 43 after completing the picking of the last crop planting row 20.

[0063] The autonomous following and transfer control method of the fruit and vegetable transport vehicle 50 according to the embodiment of the present application will be described in detail below in conjunction with specific embodiments.

[0064] Step S1, gesture recognition pre-training is implemented based on the YOLO v5 algorithm in the PyTorch framework, and the LD ConGR dataset released by the Institute of Software, Chinese Academy of Sciences in 2022 is used for training. Three gestures with obvious features and large differences are selected from the dataset and defined as the first target gesture, the second target gesture and the third target gesture. To better meet the actual working scenario, ten picking personnel each collected 50 pictures of the first target gesture, the second target gesture and the third target gesture respectively during the actual picking operation, a total of 1500 pictures, as a supplement dataset together with the model training, complete the training to obtain the gesture recognition algorithm model, and deploy it with the industrial computer in the autonomous following fruit and vegetable transport vehicle 50.

[0065] Step S2, the fruit and vegetable transport vehicle 50 carries the 3D laser radar 54, the controller, the IMU and the photoelectric encoder and walks in the ridge way by controlling the ROS keyboard node. The scanning data of the laser radar and the fusion odometer data of the IMU and the encoder are used to establish a two-dimensional grid map of the picking operation area in the facility by using the Gmapping algorithm, and the map is saved. The constructed map should include information such as various work areas and crop planting rows that the robot can pass through.

[0066] Step S3, the rviz plug-in is used to open the two-dimensional grid map, and the row start position 31 information of the work space 30 between each crop planting row in the map is determined and recorded P line_si x line_si , y line_si and the row end position 32 information, P line_ei x line_ei , y line_ei , wherein i = 1 ~ n 1, n 1 represents the total number of rows that need to be picked; the unloading and sub-packaging point position 43 information P o x o , y o , wherein j = 1 ~ n 2, n 2 represents the total number of ridge ways between the rows that need to be picked, i.e. the total number of work spaces 30; the work start position 41 information P sj x sj , y sj , the work end position 42 information P ej ​​​​( x ej , y ei ).

[0067] Step S4, determine the current picking operation area in the map, and write the work start point position 41 information in the task control module in the area determined in step S3 P sj ( x sj , y sj ), work end point position 42 information P ej ( x ej , y ei ), unloading and sub-packaging point position 43 information P o ( x o , y o ).

[0068] Step S5, start the nodes of the autonomous navigation positioning function module, gesture recognition module and task control module in the ROS robot operating system, and control the fruit and vegetable transport vehicle 50 to autonomously navigate from the unloading and sub-packaging point position 43 to the row start point position 31 of the work space 30 between the first crop planting row 20 and the second crop planting row 20.

[0069] Step S6, according to the real-time published current positioning information of the fruit and vegetable transport vehicle 50 by the autonomous navigation positioning function module, the task control module detects that the fruit and vegetable transport vehicle 50 reaches the row start point position 31 of the work space 30, and controls the fruit and vegetable transport vehicle 50 to stop at the position and wait, while the task control module opens the front camera 53 to start collecting the gesture image information of the picking staff.

[0070] Step S7, after the picking staff is in place, the fruit and vegetable transport vehicle 50 processes and recognizes the staff gesture image data successfully collected by the front camera 53 using the gesture recognition algorithm model of the gesture recognition module, and after successfully recognizing the first target gesture, the autonomous navigation positioning function module sends an autonomous following instruction to control the fruit and vegetable transport vehicle 50 to autonomously follow the picking staff to carry out picking and transportation operations.

[0071] Step S8, during the autonomous following of the fruit and vegetable transport vehicle 50 in the working space 30 by the picking worker, the task control module automatically controls the fruit and vegetable transport vehicle 50 to keep a preset safe and convenient distance range with the picking worker according to the real-time distance between the fruit and vegetable transport vehicle 50 and the picking worker fed back by the laser radar 54 in front of the fruit and vegetable transport vehicle 50 D ( d min ，d max When the distance between the fruit and vegetable transport vehicle 50 and the picking worker is less than the lower limit of the safe distance d min , the autonomous navigation positioning function module sends a stop motion instruction to control the fruit and vegetable transport vehicle 50 to stop moving; when the distance between the picking worker and the fruit and vegetable transport vehicle 50 is greater than the upper limit of the safe distance d max , the autonomous navigation positioning function module sends an autonomous following instruction to control the fruit and vegetable transport vehicle 50 to autonomously follow the picking worker to move forward.

[0072] Step S9, during the picking process in the current working space 30, when the storage box of the fruit and vegetable transport vehicle 50 has been filled, the picking worker makes a target second target gesture in front of the camera 53 of the fruit and vegetable transport vehicle 50, and after the gesture recognition module detects the target second target gesture, the task control module controls the fruit and vegetable transport vehicle 50 to autonomously exit the current working space 30 from the full basket return point along the original path and autonomously return to the work sub-packaging position P o ( x o , y o , and waits for the worker to complete unloading.

[0073] Step S10, after unloading is completed, a third target gesture is recognized, and the fruit and vegetable transport vehicle 50 directly returns to the full basket return point in step S9, that is, both conditions (the second target gesture and the third target gesture) of step S9 and step S10 are met, and the fruit and vegetable transport vehicle 50 directly returns to the full basket return point in step S9 to continue the current picking and transporting work.

[0074] Step S9', after the current working space 30 has completed picking, and the task control module detects that the fruit and vegetable transport vehicle 50 reaches the end position 32 of the current working space 30, the task control module controls the fruit and vegetable transport vehicle 50 to autonomously exit the current working space 30 along the original path and autonomously return to the work sub-packaging position P o ( x o , y o , and waits for the worker to complete unloading.

[0075] Step S10', at the distribution position, after the worker completes unloading, the worker makes a target third target gesture to the camera 53 in front of the fruit and vegetable transport vehicle 50, and after the gesture recognition module detects the target third target gesture, the task control module controls the fruit and vegetable transport vehicle 50, and the autonomous navigation positioning function module controls the fruit and vegetable transport vehicle 50 to autonomously navigate from the distribution point to the starting position 31 of the row of the next work space 30.

[0076] The above steps are repeated until all the selected target work areas are picked.

[0077] In summary, in the embodiment of the present application, the gesture recognition algorithm in the deep learning technology is combined with the multi-sensor fusion navigation technology, such as the sensor data of the laser radar 54 and the IMU (inertial measurement unit) and the encoder used in step S2, without increasing the hardware cost, the interaction control between the picking personnel and the fruit and vegetable transport vehicle 50 is more humanized and intelligent, not only overcoming the difficulties in the practical engineering application of the prior art, but also liberating the picking personnel from the additional fruit and vegetable transport vehicle 50 control, improving the overall picking efficiency, and providing a reference for the development and industrialization application of the intelligent fruit and vegetable collecting equipment.

[0078] The present application also provides a fruit and vegetable transport vehicle 50, which comprises a vehicle body 51, a fruit and vegetable storage basket 52, a control system and a camera 53.

[0079] Specifically, the fruit and vegetable storage basket 52 is arranged on the vehicle body 51, the control system is arranged on the vehicle body 51, the control system executes the autonomous following transfer control method of the fruit and vegetable transport vehicle 50 described in any of the above embodiments, the camera 53 is arranged on the vehicle body 51, and the camera 53 detects the gesture information of the picking personnel.

[0080] Since the autonomous following transfer control method of the fruit and vegetable transport vehicle 50 of the embodiment of the present application can realize the autonomous following transportation function of the fruit and vegetable transport vehicle 50 facility in the work space 30 picking, the fruit and vegetable transport vehicle 50 of the embodiment of the present application also has the advantages of intelligence, easy to use, etc., which will not be repeated here.

[0081] The present application also provides an electronic device 200, comprising a processor 201 and a memory 202, and the memory 202 stores computer program instructions, wherein when the computer program instructions are run by the processor 201, the processor 201 executes the steps of the method in the above embodiments.

[0082] Further, as shown in the figure, Figure 6 The electronic device 200 further comprises a network interface 203, an input device 204, a hard disk 205, and a display device 206.

[0083] The various interfaces and devices described above can be interconnected through a bus architecture. The bus architecture can include any number of interconnecting buses and bridges. Various circuits can be connected to one or more central processing units (CPU) represented by the processor 201, along with various memories represented by the memory 202. The bus architecture also can include various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is to be understood that the bus architecture is used to facilitate communication among these components and is not intended to limit the scope of the application. The bus architecture can include a data bus to facilitate the transfer of data, a control bus to facilitate the transfer of control information, and a state signal bus to facilitate the transfer of status information.

[0084] The network interface 203 can be connected to a network (e.g., the Internet, a local area network, etc.) to obtain relevant data from the network and can store the data in the hard disk 205.

[0085] The input device 204 can receive various instructions input by an operator and send the instructions to the processor 201 for execution. The input device 204 can include a keyboard or a pointing device (e.g., a mouse, a trackball, a touchpad, or a touchscreen, etc.).

[0086] The display device 206 can display the results obtained by the processor 201 executing the instructions.

[0087] The memory 202 can store programs and data necessary for the operation of the operating system 2021, as well as intermediate results and other data generated during the computation of the processor 201.

[0088] It is to be understood that the memory 202 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. The memory 202 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable type of memory 202.

[0089] In some embodiments, the memory 202 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 2021 and an application program 2022.

[0090] The operating system 2021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application programs 2022 include various application programs 2022, such as a browser, and the like, for implementing various application services. Programs implementing the method of the embodiments of the present application can be included in the application programs 2022.

[0091] The processor 201 described above, when invoking and executing the application programs 2022 and data stored in the memory 202, specifically, programs or instructions stored in the application programs 2022, executes the steps of the method according to the above embodiments.

[0092] The method disclosed in the above embodiments of the present application can be applied to the processor 201 or implemented by the processor 201. The processor 201 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 201. The processor 201 described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor 201 can also be any conventional processor 201 and the like. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor. The software modules can be located in the random access memory (RAM), the flash memory, the read-only memory (ROM), the programmable read-only memory (PROM), the electrically erasable programmable memory (EEPROM), the register, and other mature storage mediums in the art. The storage medium is located in the memory 202, and the processor 201 reads information in the memory 202 and combines it with the hardware to complete the steps of the above method.

[0093] It can be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions of the present application, or a combination thereof.

[0094] For software implementation, the techniques herein can be implemented by means of a module for performing the function, such as procedures, functions, and so on. Software codes can be stored in the memory 202 and executed by the processor 201. The memory 202 can be implemented in the processor 201 or outside the processor 201.

[0095] In particular, the processor 201 is further configured to read a computer program and perform the following steps: the method predicts and outputs the question answer of the user's question.

[0096] The application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by the processor 201, the processor 201 performs the steps of the method of the above embodiment.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0098] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can be a separate physical unit, or two or more units can be integrated in a unit. The above integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0099] The integrated unit implemented in the form of software function units can be stored in a computer readable storage medium. The above software function unit stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute part of the steps of the transmitting and receiving method of the embodiments of the present application. The above storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0100] While specific embodiments of the application have been described in detail, those skilled in the art will appreciate that the examples provided are for illustrative purposes only and are not meant to limit the scope of the application. Those skilled in the art will appreciate that modifications can be made to the described embodiments without departing from the scope and spirit of the application. The scope of the application is defined by the appended claims.

Claims

1. An autonomous follow-up transfer control method of a fruit and vegetable transport vehicle, characterized by, The method comprises the following steps: selecting a first target gesture, a second target gesture and a third target gesture for gesture recognition training to obtain a gesture recognition algorithm model, and deploying the gesture recognition algorithm model to a control system of the fruit and vegetable transport vehicle; constructing a map, the map comprising information of a work area and crop planting rows, the work area comprising at least one crop planting column, the crop planting column comprising a plurality of crop planting rows, a space between adjacent two crop planting rows serving as a work space, the work space being divided into a first work space to an Nth work space, N being an integer not less than 1; labeling a row start position and a row end position of each work space in the map; selecting a work area currently requiring picking in the map, and writing a work start position, a work end position and an unloading and sub-packaging position in the work area to a task control module of the fruit and vegetable transport vehicle, the work start position coinciding with the row start position of the first work space, and the work end position coinciding with the row end position of the Nth work space; starting an autonomous navigation positioning function module, a gesture recognition module and a task control module of the fruit and vegetable transport vehicle; the fruit and vegetable transport vehicle moving from the unloading and sub-packaging position to the work start position of the Nth-1 work space and then stopping moving, while a camera of the fruit and vegetable transport vehicle starts detecting gesture information of a picking worker; after detecting that the picking worker makes the first target gesture, the transport vehicle autonomously follows the picking worker to carry out picking and transport work in the Nth-1 work space; after detecting that the picking worker makes the second target gesture during picking in the Nth-1 work space and the picking basket is full, the fruit and vegetable transport vehicle exits the Nth-1 work space from a full basket return point and returns to the unloading and sub-packaging position, and after completing unloading and recognizing the third target gesture, the fruit and vegetable transport vehicle autonomously navigates to the full basket return point of the Nth-1 work space to continue picking and transport work in the Nth-1 work space; or, the fruit and vegetable transport vehicle moves to the work end position of the Nth-1 work space, and the fruit and vegetable transport vehicle autonomously returns to the unloading and sub-packaging position, and after recognizing the third target gesture, the fruit and vegetable transport vehicle autonomously navigates to the work start position of the Nth work space.

2. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, When constructing the gesture recognition algorithm model, a YOLO v5 algorithm based on a PyTorch framework is used for gesture recognition pre-training, and an LD ConGR data set is used for training.

3. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, When constructing the map, a chassis of the fruit and vegetable transport vehicle is controlled through a ROS keyboard node, and a two-dimensional grid map is constructed by using a laser SLAM technology.

4. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, During autonomous following of the picking worker in the work space, a laser radar in front of the fruit and vegetable transport vehicle measures a distance, so that the fruit and vegetable transport vehicle keeps a preset safety distance from the picking worker.

5. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, When the first target gesture or the third target gesture is recognized, the fruit and vegetable transport vehicle located at the unloading and distribution point position completes unloading.

6. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, When the first target gesture or the third target gesture is recognized, the weight of the fruit and vegetable transport vehicle located at the unloading and distribution point position is not greater than a preset weight value.

7. The autonomous follow-up transfer control method of the fruit and vegetable transport cart according to claim 1, characterized by, The work starting point position is away from the unloading and distribution point position, and the work ending point position is close to the unloading and distribution point position.

8. A fruit and vegetable transport vehicle characterized by, The fruit and vegetable transport vehicle comprises: a vehicle body; a fruit and vegetable storage basket arranged on the vehicle body; a control system arranged on the vehicle body, the control system performing the autonomous follow-up transfer control method of the fruit and vegetable transport vehicle according to any one of claims 1-7; a camera arranged on the vehicle body, the camera detecting gesture information of a picking worker.

9. An electronic device, comprising: The fruit and vegetable transport vehicle comprises: a processor and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor performs the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the method according to any one of claims 1-7.

Images