Cucumber picking working method and picking robot thereof

Abstract

The invention provides a cucumber picking working method and a picking robot thereof, and belongs to the technical field of robots. The method comprises the following steps: identifying a first environment image to obtain first positions of a plurality of first fruits, wherein the first environment image is collected by a global camera; the robot is controlled to move to a preset parking position corresponding to the first fruit, a second environment image is recognized to obtain the existence state of each first fruit, and the second environment image is collected by a local camera of the robot; determining a positioning algorithm of each first fruit according to the existence state of each first fruit and the corresponding relationship between the existence state and the positioning method; and positioning a second position of each first fruit according to the positioning algorithm of each first fruit, and picking each first fruit according to the second position. According to the invention, the positioning accuracy and the picking efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robots, and particularly relates to a cucumber picking work method and a picking robot. BACKGROUND

[0002] With the acceleration of agricultural modernization, the demand for mechanization and intelligentization in the field of fruit and vegetable planting continues to rise, especially in the scenario of facility agriculture (such as greenhouse and greenhouse), efficient picking operation becomes a key link to improve production efficiency and reduce labor cost. As an economic vegetable widely planted in the world, cucumber is mainly cultivated in a dense mode, the fruit growth position is scattered and the shape is curved, and there are characteristics of "great maturity difference and easy breakage", which puts forward higher requirements for the accuracy and flexible operation of picking operation. Under this background, it is particularly important to develop a picking robot specially adapted to the fruit characteristics and planting scene of crop planting.

[0003] In the related art, the robot moves along a fixed path, and then picks the recognized fruits through image recognition. However, in the actual picking process, the fruits may be blocked by leaves, which leads to incorrect recognition and inaccurate positioning. Therefore, the positioning accuracy of the related art is poor. SUMMARY

[0004] To solve the above technical problems, the application provides a cucumber picking work method and a picking robot to solve the technical problem of incorrect recognition and incorrect positioning of fruits caused by leaf blocking in the prior art, and to solve the problem of decreased positioning accuracy.

[0005] In a first aspect, the application provides a cucumber picking work method, which adopts the following technical solution: A cucumber picking work method, comprising: identifying a first environment image to obtain a first position of a plurality of first fruits, the first environment image being collected by a global camera; controlling a robot to move to a preset parking position corresponding to the first fruit, and identifying a second environment image to obtain an existence state of each first fruit, the second environment image being collected by a local camera of the robot; determining a positioning algorithm of each first fruit according to the correspondence between the existence state of each first fruit and the existence state and the positioning algorithm; positioning a second position of each first fruit according to the positioning algorithm of each first fruit, and picking each first fruit according to the second position.

[0006] In an embodiment, the presence state is presence or absence, the positioning algorithm of each first fruit is determined according to a corresponding relationship between the presence state of each first fruit, the presence state and the positioning method, and the positioning algorithm comprises: If the presence state is the presence, a second position of each first fruit is obtained based on the second environment image and a preset coordinate conversion algorithm; If the presence state is the absence, a fruit distribution ellipsoid is constructed based on the first position of the first fruit; An inscribed sphere of the fruit distribution ellipsoid is scanned to obtain the second position of the first fruit.

[0007] In an embodiment, the fruit distribution ellipsoid is constructed based on the first position of the first fruit, and the method comprises: A plurality of covariance matrix data are obtained, and a position error covariance is calculated according to each covariance matrix data; Camera parameters of the robot and mechanical arm parameters of the robot are obtained, and an inscribed sphere radius of the fruit distribution ellipsoid is calculated according to the camera parameters and the mechanical arm parameters; The inscribed sphere radius of the fruit distribution ellipsoid is calculated according to the following formula: , r is a distribution radius, k is a distribution coefficient (3≤k≤3.5), is a pixel positioning noise, is a depth noise, is a repeatability of the robot, is a joint transmission backlash, is a mechanical arm end deflection of the robot; The fruit distribution ellipsoid is constructed based on the first position and the distribution radius, wherein the first position is a sphere center of the inscribed sphere of the fruit distribution ellipsoid.

[0008] In an embodiment, before the existence state of each first fruit is obtained by identifying the second environment image, the method further comprises: A robot position of the robot is obtained, wherein the robot position is a position of the robot after movement; A picking sequence of each first fruit is determined according to the robot position and the first position of each first fruit; Correspondingly, the existence state of each first fruit is obtained by identifying the second environment image, and the method comprises: The second environment image is identified, and the existence state of each first fruit is obtained in sequence according to the picking sequence of each first fruit.

[0009] In an embodiment, the method further comprises, after picking each of the first fruits according to the second positions: When the robot moves to the first fruit of the next picking sequence, identifying the second fruit in the second environment image, the second fruit being the fruit identified in the process of moving the local camera; Obtaining a third position of the second fruit, and matching the third position of the second fruit with the first position; If the third position of the second fruit is different from the first position, determining that the second fruit is a missed fruit, and generating a missed fruit signal; If the third position of the second fruit is the same as the first position, determining that the second fruit is not the missed fruit.

[0010] In an embodiment, the method further comprises, before picking each of the first fruits according to the second positions: Obtaining a current growth period; Determining a target environment image recognition model corresponding to the current growth period based on a correspondence relationship between the current growth period, the growth period, and the environment image recognition model; Identifying the second fruit in the second environment image using the target environment image recognition model.

[0011] In an embodiment, the method further comprises, before picking each of the first fruits according to the second positions: Identifying a fruit image of each of the first fruits to obtain a fruit size, and determining whether the fruit size is less than a preset fruit size; Obtaining a bending degree of each of the first fruits, and determining whether the bending degree is greater than a preset bending degree threshold; If the fruit size is less than the preset fruit size, or the bending degree is greater than the preset bending degree threshold, determining that the second fruit is a problem fruit, and generating a problem fruit signal, the problem fruit signal being used to remind that there is a problem with the fruit.

[0012] In a second aspect, the application provides a cucumber picking robot, which adopts the following technical solution: A cucumber picking robot, comprising: A mobile chassis for moving the robot; A mechanical arm arranged above the mobile chassis for approaching a first fruit by free movement; A vision system arranged on the mechanical arm for searching and positioning the first fruit; A controller is electrically connected with the mobile chassis, the mechanical arm and the vision system, and configured to control the mobile chassis, the mechanical arm and the vision system.

[0013] In an embodiment, the mechanical arm further comprises an execution device arranged at the end of the mechanical arm, and configured to hold the first fruit to separate the first fruit from the plant.

[0014] In an embodiment, the vision system comprises a global camera and a local camera, the global camera is arranged in the lifting mechanism of the mechanical arm, and the local camera is arranged on the execution device and configured to locate the first fruit by following the movement of the mechanical arm.

[0015] In summary, the present application has the following beneficial technical effects: 1. The first position of the first fruit is identified from the first environment image collected by the global camera to achieve coarse positioning of the first fruit. Then the robot is controlled to move to the first position to obtain the existence state of each first fruit. The existence state of the first fruit is confirmed on the basis of a relatively close distance, which can effectively improve the accuracy of the existence state confirmation of the first fruit. On the basis of accurate confirmation of the existence state of the first fruit, different positioning algorithms are used for re-positioning of the first fruit, which further improves the precision of the positioning of the first fruit. In addition, re-positioning of the first fruit according to the existence state can also avoid the situation that other interference objects are mistakenly identified as fruits and then continue to be positioned and picked, thereby effectively improving the positioning precision and picking efficiency.

[0016] 2. When the second fruit exists, the second fruit is directly fine-positioned and picked. When the second fruit does not exist, the effective search range is expanded by constructing a fruit distribution circle to achieve fine positioning and picking of the second fruit. Different fine positioning methods are used for different situations, which effectively improves the fine positioning and search efficiency of the second fruit. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a cucumber picking robot working method provided by an embodiment of the present application; Figure 2 A working space and a corresponding preset parking position provided by an embodiment of the present application; Figure 3 A coordinate origin provided by an embodiment of the present application; Figure 4 A flowchart of a cucumber picking robot cycle picking provided by an embodiment of the present application; Figure 5 A structure diagram of a cucumber picking robot provided by an embodiment of the present application; Figure 6A structural schematic diagram of a controller provided by an embodiment of the present application.

[0018] In the drawings, various reference numerals refer to various identical or similar elements. 10, mobile chassis; 20, mechanical arm; 21, execution device; 31, global camera; 32, local camera; 301, processor; 302, bus; 303, memory; 304, transceiver. DETAILED DESCRIPTION

[0019] The following will be described in detail with reference to the accompanying drawings. Figure 1 to the accompanying drawings Figure 6 The present application will be described in further detail.

[0020] The specific embodiments are merely illustrative of the present application, and are not intended to limit the present application. Those skilled in the art can make modifications to the embodiments without creative contribution, according to the needs after reading the present specification, and as long as the modifications are within the scope of the present application, they are protected by the patent law.

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative contribution are within the scope of protection of the present application.

[0022] In addition, the term "and / or" in this paper is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper, unless otherwise specified, generally represents an "or" relationship between the associated objects before and after it.

[0023] The embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0024] The working method of the cucumber picking robot provided by the embodiments of the present application is executed by a controller, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application. Figure 1 As shown in the figure, the method comprises steps S1, S2, S3 and S4, wherein: Step S1: identifying a first position of a plurality of first fruits in a first environment image, the first environment image being captured by a global camera.

[0025] Specifically, the first environment image is an environment image captured by the robot before moving, and all individual fruits identified through the first environment image are defined as first fruits. In the embodiments of the present application, the fruits can be cucumbers.

[0026] The position of each first fruit in the first environment image can be identified after receiving an identification request. A monitoring program is pre-integrated in the robot, and the monitoring program is used to monitor the triggering behavior of the identification request. Once it is monitored that the identification request is triggered, the controller controls the global camera to capture the first environment image, and the controller identifies the first environment image.

[0027] After obtaining the first positions of the plurality of first fruits, a target working space of the robot is determined. The specific process of determining the target working space of the robot includes: using a preset image recognition algorithm to identify the number of fruits of the first fruits in a plurality of first environment images, comparing the number of fruits corresponding to each first environment image, and determining the position corresponding to the first environment image with the maximum number of fruits as the target working space. Each first environment image does not have image overlap, and each first environment image corresponds to a working space. It can be understood that by determining the working space with the maximum number of fruits as the target working space of the robot, the picking robot can pick as many first fruits as possible at one time. At the same time, by determining the target working space, the coarse positioning of the first fruits can be realized, and a data basis for picking by the mechanical arm can be provided.

[0028] Further, the specific process of identifying the first positions of the first fruits in the target working space in the first environment image includes: pre-processing the first environment image to reduce the interference of background objects, and then inputting the first environment image into a preset identification model, the preset identification model outputting two-dimensional pixel coordinates of each first fruit; establishing a coordinate system with any point in the first environment image as the coordinate origin, and then performing coordinate conversion according to the global camera parameters, the coordinate system and the two-dimensional pixel coordinates to obtain the first positions (i.e. actual space coordinates) of the first fruits. The preset identification model is obtained by training a neural network by technical personnel, and the specific training process and the specific process of coordinate conversion are not limited in the embodiments of the present application.

[0029] Step S2: controlling the robot to move to a preset parking position corresponding to the first fruits, and identifying a second environment image to obtain the existence state of each first fruit, the second environment image being captured by a local camera of the robot.

[0030] Specifically, the mobile chassis of the robot drives the robot to move to a preset parking position of any first fruit in the target workspace, and the preset parking position corresponding to each workspace is preset, i.e., the preset parking positions corresponding to all first fruits in the target workspace are the same, and each workspace corresponds to a preset parking position, as shown in Figure 2 The working space and the corresponding preset parking position provided by the embodiment of the application are shown in the schematic diagram. After reaching the target workspace, the robot stops moving, and the local camera of the robot collects an environmental image when the robot is in the target workspace, i.e., a second environmental image, to achieve close-range collection of the first fruits. The existence state of each first fruit is identified from the second environmental image, including: the robot moves the mechanical arm to the front of the first fruit (i.e., a preset distance from the first position), the controller controls the local camera to obtain the second environmental image, and the controller inputs the obtained second environmental image into a preset neural network recognition model, and the neural network recognition model outputs the existence state of the first fruit. The existence state of the first fruit is that the first fruit exists or the first fruit does not exist. It can be understood that, in the coarse positioning process, the global camera of the robot is far away from the fruit plant, and the number of first fruits appearing in the field of view of the global camera is large. At the same time, due to the characteristics of the global camera, as the distance between the global camera and the first fruit increases, the area covered by the field of view of the global camera becomes larger, resulting in lower accuracy of the first fruit positioning based on the first environmental image. Therefore, in order to avoid missing the first fruit, it is necessary to confirm the first fruit photographed in the coarse positioning stage again after the robot moves to the target workspace.

[0031] Step S3: determining the positioning algorithm of each first fruit according to the correspondence between the existence state of each first fruit and the existence state and the positioning algorithm.

[0032] Specifically, the correspondence between the existence state and the positioning algorithm is preset by the technician. In the present application, the content of the correspondence between the existence state and the positioning algorithm includes: when the existence state is existence, the corresponding positioning algorithm is the coordinate conversion algorithm, and when the existence state is non-existence, the corresponding positioning algorithm is the fruit distribution ellipsoid circumscribed sphere algorithm.

[0033] Step S4: positioning the second position of each first fruit according to the positioning algorithm of each first fruit, and picking each first fruit according to the second position.

[0034] Specifically, the specific process of positioning the second position of each first fruit according to the coordinate conversion algorithm includes: inputting the second environmental image into a preset recognition model, and the preset recognition model outputs the two-dimensional pixel coordinates of each first fruit, and converting the two-dimensional pixel coordinates into the second position of each first fruit through the coordinate conversion algorithm.

[0035] If the state is non-existent, a fruit distribution ellipsoid is constructed based on the first position of the first fruit, and an outer sphere of the fruit distribution ellipsoid is scanned to obtain a second position of the first fruit. It can be understood that when the first fruit is non-existent, it indicates that the global camera may have a wrong shot, or the first position of the first fruit is inaccurate, so that the local camera cannot shoot the fruit image of the first fruit. At this time, the search range needs to be expanded to further determine whether the first fruit exists, so it is necessary to construct a fruit distribution ellipsoid according to the first position, and scan the outer sphere of the fruit distribution ellipsoid as the scanning range to determine the existence of the first fruit again. Wherein, the coordinate origin of the first position and the coordinate origin of the second position are located at the mechanical arm of the picking robot, but since the picking robot moves the chassis during picking, the coordinate origin of the first position and the coordinate origin of the second position are different, as shown in the coordinate origin diagram. Figure 3

[0036] It can be understood that in the context of "local linear and zero mean light tail noise", the confidence ellipsoid defined by the covariance matrix can cover the real fruit position with a high coverage probability. In actual application, various errors such as camera, distance, mechanical gap and environmental factors will form a "cloud cluster" close to the ellipsoid around the predicted point, that is, when the coverage ratio of 95% is selected, it is obtained that "the fruit will fall within this ellipsoid under the condition of 95%", therefore, the fruit ellipsoid distribution algorithm is adopted in the embodiment of the application, compared with direct search, the effective search range can be quickly delimited, and searching for the fruit in the range with a high search success probability can effectively improve the search efficiency and avoid prolonging the search time.

[0037] Wherein, the fruit distribution ellipsoid is constructed based on the first position of the first fruit, comprising: Obtaining a plurality of covariance matrix data, and calculating the position error covariance according to each covariance matrix data; Obtaining the camera parameters of the robot and the mechanical arm parameters of the robot, and calculating the outer sphere radius of the fruit distribution ellipsoid according to the camera parameters and the mechanical arm parameters; Wherein, the outer sphere radius of the fruit distribution ellipsoid, the calculation formula is: , r is the distribution radius, k is the distribution coefficient (3≤k≤3.5), is the pixel positioning noise, is the depth noise, is the repeatability of the robot, is the joint transmission backlash, is the deflection of the end of the mechanical arm of the robot; ​The first position is the center of a circumscribed sphere of the fruit distribution ellipsoid.

[0038] Specifically, the covariance matrix data can be obtained from a preset information library, and in the present application, the covariance matrix data includes a global camera pixel positioning covariance matrix, a depth measurement covariance matrix, a robot mobile chassis positioning covariance matrix, a robot joint transmission backlash covariance matrix, and a mechanical arm end deflection covariance matrix, and the calculation formula is: , wherein, is the global camera pixel positioning covariance matrix, is the depth measurement covariance matrix, is the robot mobile chassis positioning covariance matrix, is the robot joint transmission backlash covariance matrix, is the mechanical arm end deflection covariance matrix.

[0039] The camera parameter is a camera parameter of a local camera, and the camera parameter and the mechanical arm parameter can be obtained from the information library, and in the present application, the circumscribed sphere radius of the fruit distribution ellipsoid is the maximum principal axis radius.

[0040] wherein, the preset fruit distribution ellipsoid calculation formula is: , wherein, the degree of freedom is 3, p is the chi-square quantile with a confidence level of p , x the second position of the first fruit.

[0041] Further, the controller controls the execution device of the robot mechanical arm to move to the second position of the first fruit, and picks the second fruit according to the picking instruction. The center of the fruit distribution ellipsoid is the same as the center of the circumscribed sphere of the fruit distribution ellipsoid.

[0042] Based on the above embodiment, the first environment image captured by the global camera is recognized to obtain the first position of the first fruit, so as to realize the coarse positioning of the first fruit; then the robot is controlled to move to the first position to obtain the existence state of each first fruit, and the existence state of the first fruit is confirmed on the basis of the close distance with the first fruit, so that the accuracy of the existence state confirmation of the first fruit can be effectively improved; on the basis of accurately confirming the existence state of the first fruit, different positioning algorithms are selectively used for repositioning the first fruit, so that the positioning accuracy of the first fruit is further improved, and the repositioning of the first fruit according to the existence state can also avoid the situation that other interference objects are incorrectly identified as fruits and then continue to be positioned and picked, so that the positioning accuracy and picking efficiency are effectively improved.

[0043] In a possible implementation of the embodiment of the application, before the existence information of the second fruit is determined according to the first position of the second fruit, the method further includes: obtaining a robot position of the robot, the robot position being a position of the robot after movement; determining a picking sequence of each first fruit according to the robot position and the first position of the first fruit; Correspondingly, the existence state of each first fruit is obtained by recognizing the second environment image, including: The second environment image is recognized, and the existence state of each first fruit is obtained in sequence according to the picking sequence of the first fruit.

[0044] Specifically, the coordinate system can be established by a coordinate system conversion algorithm to obtain the robot position, and in the embodiment of the application, the robot position is the end position of the mechanical arm of the robot. The picking sequence of each first fruit can be determined by using a greedy nearest neighbor initialization combined with a 2-opt improved algorithm, specifically including: first constructing an initial solution by using a nearest neighbor heuristic, and then optimizing the solution by using a 2-opt algorithm; the specific process of solution optimization is not limited in the application. Correspondingly, in the process of determining the existence state of the first fruit, the mechanical arm moves to the first position of each first fruit in sequence according to the picking sequence, so as to determine whether the first fruit exists. It can be understood that, in the process of determining the existence state of the first fruit, the picking sequence of the first fruit is determined on the basis of the shortest distance, so that the picking time of the mechanical arm can be effectively shortened, and the picking efficiency is further improved.

[0045] Based on the above embodiment, the robot position is obtained, the picking distance between the robot and the first fruit is determined by taking the robot position and the first position as references, and the picking sequence is determined according to the picking distance, so that the picking efficiency can be improved by minimizing the picking path when the second fruit is picked in sequence according to the picking sequence.

[0046] In a possible implementation of the embodiment of the application, after the first fruit is picked according to the second position, the method further includes: When the mechanical arm moves to the first fruit of the next picking sequence, the second environment image is recognized to obtain a second fruit in the second environment image, the second fruit being a fruit recognized in the local camera movement process; A third position of the second fruit is obtained, and the third position of the second fruit is matched with the first position; If the third position of the second fruit is different from the first position, it is determined that the second fruit is a missed fruit, and a missed picking signal is generated; If the third position of the second fruit is the same as the first position, it is determined that the second fruit is not a missed fruit.

[0047] Specifically, the second fruit is a fruit other than the first fruit and recognized in the movement process. When the mechanical arm completes the picking of the current first fruit and moves to the first fruit corresponding to the next picking sequence, the local camera collects and recognizes the second environment image in the movement process in real time, and determines whether there is a fruit that is not recognized in the coarse positioning stage in the movement process by recognizing the second environment image. Further, when the second fruit is recognized in the movement process, the third position of the second fruit is obtained, and if the second fruit is not recognized in the movement process, the movement to the first fruit of the next picking sequence is continued to perform the picking work. Wherein, the specific process of recognizing the second fruit includes: the third position of the second fruit can be obtained by a coordinate conversion algorithm, and the specific process of obtaining the fruit position by the coordinate conversion algorithm is not limited in the embodiments of the application. The third position of the second fruit recognized in the movement process is matched with the first position of the second fruit to determine whether the second fruit is a fruit that has not been recognized before. If the third position of the second fruit is different from the first position of the second fruit, it indicates that the second fruit is a fruit that has not been recognized before, i.e., the fruit is not recognized in the coarse positioning stage, so the fruit is determined to be a missed fruit, a missed picking signal is generated, and the missed picking signal is sent to the management personnel mobile side device. If the third position of the second fruit is the same as the first position of the second fruit, it indicates that the second fruit is not a missed fruit, which is the second fruit corresponding to the next picking sequence.

[0048] Further, when it is determined that the second fruit is a missed fruit, the execution device of the robot mechanical arm picks the second fruit, and after completing the picking of the second fruit, the first fruit of the next picking sequence is continued to be picked.

[0049] Based on the above embodiment, when the mechanical arm picks up the next first fruit, whether there is a missed identification fruit, i.e., a second fruit, is determined by recognizing the second environment image, and a third position of the second fruit is obtained, so as to match the third position of the second fruit and the first position of the second fruit, and whether the second fruit is the second fruit in the next picking sequence is determined through the matching, if the third position of the second fruit does not match the first position, it is determined that the second fruit is not the first fruit in the next picking sequence, i.e., the second fruit is a missed identification fruit, and a missed picking signal is generated; the missed identification fruit is confirmed in real time through image scanning during the movement of the mechanical arm, which can effectively reduce the number of missed picking fruits.

[0050] In a possible implementation manner of the embodiment of the application, the second fruit in the second environment image is identified, including: obtaining a current growth period; determining a target environment image recognition model corresponding to the current growth period based on a corresponding relationship between the current growth period, the growth period and the environment image recognition model; using the target environment image recognition model to identify the second environment image to obtain the second fruit.

[0051] Specifically, the current date and the planting date of each first fruit can be obtained, the date difference between the current date and the planting date is calculated, and the date difference, the fruit growth period and the corresponding relationship of the date difference are matched to determine the current growth period of the fruit. It can be understood that the fruit characteristics of the first fruit in different growth periods are different, and compared with direct identification, the fruit characteristics in the current growth period are used as the reference basis for identification in the case of determining the fruit growth period, which can effectively improve the identification efficiency and the identification accuracy. The corresponding relationship between the growth period and the environment image recognition model is pre-set by the technical personnel and input into the electronic device. It can be understood that when the fruit is in different growth periods, the size, shape and color of the fruit are different, in order to ensure the fruit identification accuracy, therefore, the corresponding target environment image recognition model needs to be used according to the fruit growth period. The environment image recognition model is obtained by the technical personnel through a plurality of sample data and a neural network model, and the environment image recognition model is not limited in the embodiment of the application. The second environment image is input into the target environment image recognition model, and the target environment image recognition model inputs the identification result to obtain the second fruit.

[0052] Based on the above embodiment, the fruit shape in different growth periods is different, and therefore the environment image recognition model is used according to the fruit in different growth periods, which can effectively improve the accuracy of determining the second fruit existence information.

[0053] In a possible implementation manner of the embodiment of the application, before picking up each first fruit based on the second position, the method further includes: obtaining a fruit size of the first fruit by recognizing the fruit image of the first fruit, and determining whether the fruit size is less than a preset fruit size; obtaining a bending degree of the first fruit, and determining whether the bending degree is greater than a preset bending degree threshold; If the fruit size is less than the preset fruit size, or the bending degree is greater than the preset bending degree threshold, it is determined that the second fruit is a problem fruit, and a problem fruit signal is generated, which is used to remind that there is a problem with the fruit.

[0054] Specifically, the fruit size includes a fruit length and a fruit diameter, and the specific process of obtaining the fruit size by recognizing the fruit image is not limited in the embodiments of the present application.

[0055] The specific process of obtaining the bending degree of the first fruit includes: using a semantic segmentation algorithm to extract a contour mask of the fruit from the front view of the fruit, selecting head and tail end points of the first fruit, and using a straight line to connect the two end points, selecting all points on the fruit contour, and calculating the perpendicular distance of each point to the straight line, and then determining the point corresponding to the maximum perpendicular distance as the midpoint, connecting the two end points to the midpoint respectively, calculating the angle between the two line segments, and determining the above angle as the bending degree. The preset fruit size includes the shortest fruit length and the smallest fruit diameter, and the preset fruit size and the preset bending degree threshold are set by the technical personnel according to the fruit test standard. Further, when the fruit length of the first fruit is less than the shortest fruit length, when the fruit diameter of the first fruit is less than the smallest fruit diameter, or when the bending degree is greater than the preset bending degree threshold, it is determined that the first fruit is a problem fruit (i.e., the first fruit is a fruit with unqualified quality), and therefore a problem fruit signal can be generated and sent to the management personnel side device to remind the management personnel. It can be understood that detecting and marking the fruit quality of the first fruit before picking the fruit can effectively improve the subsequent sorting efficiency.

[0056] Based on the above embodiments, the fruit size and the bending degree are obtained by recognizing the fruit image, and it is determined whether the fruit size is less than the preset fruit size, and whether the bending degree is greater than the preset bending degree threshold. When the fruit size is less than the preset fruit size, or the bending degree is greater than the preset bending degree threshold, it indicates that the second fruit has a quality problem and needs to be reported for subsequent timely detection.

[0057] Further, after the robot completes the picking of all the first fruits according to the picking order, the first environment image after picking is obtained again, and the first environment image after picking is recognized to determine the number of fruits after picking. If the number of fruits of the first fruit is a preset fruit number, the robot ends picking, wherein the preset fruit number is 0. If the number of fruits of the first fruit is greater than the preset fruit number, steps S2 to S4 are executed again until the number of fruits of the first fruit is 0, such as Figure 4As shown, a flowchart of a cycle picking process of a cucumber picking robot is provided in the embodiment of the present application.

[0058] The above embodiment introduces a cucumber picking method from the perspective of method flow, and the following embodiment introduces a cucumber picking robot.

[0059] The embodiment of the present application provides a cucumber picking robot, as shown in the figure. Figure 5 The cucumber picking robot includes a mobile chassis 10, a mechanical arm 20, a vision system, and a controller. The mobile chassis 10 is used to drive the robot to move. The mechanical arm 20 is arranged above the mobile chassis 10 and is used to approach the first fruit by free movement. The vision system is arranged on the mechanical arm 20 and is used to search and locate the first fruit. The controller is electrically connected with the mobile chassis 10, the mechanical arm 20, and the vision system, and is used to control the mobile chassis 10, the mechanical arm 20, and the vision system. In the embodiment of the present application, the mechanical arm 20 has one lifting movement degree of freedom and three rotation degrees of freedom. The mechanical arm 20 includes a hollow lifting column connected to the mobile chassis 10 through a lifting mechanism, and an arm assembly with three rotation degrees of freedom connected to the lifting column.

[0060] In a possible implementation manner of the embodiment of the present application, the mechanical arm 20 further includes an execution device 21 arranged at the end of the arm assembly of the mechanical arm 20 and used to clamp the first fruit to make the first fruit separate from the plant.

[0061] In a possible implementation manner of the embodiment of the present application, the vision system includes a global camera 31 and a local camera 32. The global camera 31 is arranged at the top of the lifting column of the mechanical arm 10, and the global camera 31 can be an eye-out-of-hand camera. The local camera 32 is arranged on the execution device 21 and is used to locate the first fruit by following the movement of the mechanical arm 10. The local camera 32 can be an eye-on-hand camera.

[0062] Specifically, after receiving the identification request, the controller controls the global camera 31 of the robot to collect a first environment image, and identifies the collected first environment image to obtain a first position of each first fruit. The controller controls the mobile chassis 10 of the robot to stop moving after moving to the first fruit, and controls the local camera 32 of the robot to collect a second environment image. The controller analyzes the collected second environment image to determine the existence state of the first fruit, adopts different positioning algorithms for the first fruit with different existence states for repositioning, effectively improves the positioning accuracy of the first fruit, and controls the execution device 21 to pick the first fruit according to the second position of the first fruit.

[0063] Further, the embodiment of the present application provides a controller, as shown in the figure. Figure 6 Figure 6 ​The controller shown includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, through a bus 302. Optionally, the controller can also include a transceiver 304. It should be noted that the transceiver 304 is not limited to one in actual applications, and the structure of the controller does not constitute a limitation on the embodiments of the present application.

[0064] The processor 301 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor 301 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0065] The bus 302 can include a path for transmitting information between the above-mentioned components. The bus 302 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 302 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience, Figure 6 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or one type of bus.

[0066] The memory 303 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0067] The memory 303 is configured to store application program codes for implementing the solutions of the present application, and the processor 301 is configured to control the execution. The processor 301 is configured to execute the application program codes stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0068] Figure 6 The controller shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0069] Compared with the related art, the first environment image captured by the global camera is recognized to obtain the first positions of the first fruits, so as to realize rough positioning of the first fruits. Then, the robot is controlled to move to the first positions to obtain the existence states of the first fruits. The existence states of the first fruits are confirmed on the basis of the close distance to the first fruits, which can effectively improve the accuracy of the existence state confirmation of the first fruits. On the basis of the accurate confirmation of the existence states of the first fruits, different positioning algorithms are used for repositioning of the first fruits, which further improves the precision of the positioning of the first fruits. In addition, the repositioning of the first fruits according to the existence states can also avoid the situation that other interference objects are incorrectly recognized as fruits and then continue to be positioned, so that the subsequent picking is performed, and the positioning precision and picking efficiency are effectively improved.

[0070] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0071] The above only describes some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for harvesting cucumbers, characterized in that, include: The first location of multiple first fruits is obtained by identifying the first environmental image, which is captured by a global camera; The robot is controlled to move to the preset stopping position corresponding to the first fruit, and the existence status of each first fruit is obtained by recognizing the second environmental image, which is captured by the robot's local camera; The positioning algorithm for each first fruit is determined based on the correspondence between the existence state of each first fruit and the positioning algorithm. The second position of each first fruit is located according to the positioning algorithm of each first fruit, and the first fruit is harvested according to the second position.

2. The cucumber harvesting method as described in claim 1, characterized in that, The state of existence is either present or absent. The step of locating the second position of each of the first fruits according to the positioning algorithm of each of the first fruits includes: If the existence state is "existence", then the second position of each of the first fruits is obtained based on the second environmental image and the preset coordinate transformation algorithm; If the existence state is "not present", then construct a fruit distribution ellipsoid based on the first position of the first fruit; The second position of the first fruit is obtained by scanning the circumsphere of the fruit distribution ellipsoid.

3. The cucumber harvesting method as described in claim 2, characterized in that, Constructing a fruit distribution ellipsoid based on the first position of the first fruit includes: Multiple covariance matrix data are obtained, and the position error covariance is calculated based on each of the covariance matrix data. Obtain the camera parameters and robotic arm parameters of the robot, and calculate the circumsphere radius of the fruit distribution ellipsoid based on the camera parameters and robotic arm parameters; The formula for calculating the circumradius of the fruit distribution ellipsoid is as follows: ， r is the distribution radius, and k is the distribution coefficient (3≤k≤3.5). To locate noise for pixels, For depth noise, The repeatability accuracy of the robot. This refers to the backlash in the joint transmission. The deflection at the end of the robot's robotic arm; The fruit distribution ellipsoid is constructed based on the first position and the distribution radius, wherein the first position is the center of the circumsphere of the fruit distribution ellipsoid.

4. The cucumber harvesting method as described in claim 1, characterized in that, Before obtaining the existence status of each first fruit by recognizing the second environmental image, the method further includes: Obtain the robot's position, which is the position the robot has moved to; The picking order of each of the first fruits is determined based on the robot's position and the first position of each of the first fruits. Accordingly, the step of identifying the existence state of each of the first fruits from the second environmental image includes: The second environmental image is identified, and the existence status of each of the first fruits is obtained sequentially according to the picking order of each of the first fruits.

5. The cucumber harvesting method as described in claim 1, characterized in that, After harvesting each of the first fruits according to the second position, the method further includes: When the robot moves to the first fruit in the next picking sequence, it identifies the second fruit in the second environmental image, and the second fruit is the fruit identified by the local camera during the movement. Obtain the third position of the second fruit and match the third position of the second fruit with the first position; If the third position of the second fruit is different from the first position, then the second fruit is determined to be a missed fruit, and a missed fruit signal is generated; If the third position of the second fruit is the same as the first position, then the second fruit is determined not to be the missed fruit.

6. The cucumber harvesting method as described in claim 1, characterized in that, The step of identifying the second fruit in the second environment image includes: Get the current growth cycle; Based on the correspondence between the current growth cycle, the growth cycle and the environmental image recognition model, determine the target environmental image recognition model corresponding to the current growth cycle; The second environment image is identified using the target environment image recognition model to obtain the second fruit.

7. The cucumber harvesting method as described in claim 1, characterized in that, Before harvesting each of the first fruits according to the second position, the method further includes: The fruit size is obtained by identifying the fruit image of each of the first fruits, and it is determined whether the fruit size is smaller than a preset fruit size; Obtain the curvature of each of the first fruits and determine whether the curvature is greater than a preset curvature threshold. If the fruit size is smaller than the preset fruit size, or the curvature is greater than the preset curvature threshold, then the second fruit is determined to be a problem fruit, and a problem fruit signal is generated. The problem fruit signal is used to remind that there is a problem with the fruit.

8. A cucumber harvesting robot, characterized in that, include: A mobile chassis (10) is used to move the robot. A robotic arm (20) is positioned above the movable chassis (10) for moving freely to approach the first fruit; A vision system, mounted on the robotic arm (20), is used to search for and locate the first fruit; The controller is electrically connected to the mobile chassis (10), the robotic arm (20) and the vision system, and is used to control the mobile chassis (10), the robotic arm (20) and the vision system.

9. A cucumber harvesting robot as described in claim 8, characterized in that, The robotic arm (20) also includes: An actuator (21) is provided at the end of the robotic arm (20) for gripping the first fruit to detach the first fruit from the plant.

10. A cucumber harvesting robot as described in claim 9, characterized in that, The vision system includes: A global camera (31) is installed in the lifting mechanism of the robotic arm (10); A local camera (32), mounted on the actuator (21), is used to position the first fruit by following the movement of the robotic arm (10).

Images