Vision control method, device and electronic equipment based on agricultural robot

Abstract

Embodiments of the application disclose a vision control method and device based on an agricultural robot and an electronic device, and relate to the technical field of agricultural robots. The method applies the agricultural robot to an agricultural work area, positions a fruit area based on a vision image, and determines a fruit variety in the fruit area, so as to trigger replacement of a gripper of the agricultural robot according to the fruit variety, detect the adaptation of the gripper of the agricultural robot to the fruit, and determine a gripping plan based on the multiple fruits in the vision image and the positioning of the fruits. The gripping plan is used to regulate a moving track of the gripper of the agricultural robot, and the gripper is synchronously regulated to avoid obstacles, thereby improving the safety of gripping the fruits by the agricultural robot. In addition, when the gripper grips the fruits, the positions of other fruits in the current environment are synchronously monitored, a falling state is determined based on the offset track of the other fruits, the safety of gripping the fruits by the agricultural robot is further ensured, and the other fruits in the falling state can be timely controlled to realize preferential gripping.

Description

Technical Field

[0001] This invention relates to the technical field of agricultural robots, and more particularly to a vision control method, device, and electronic device based on agricultural robots. Background Technology

[0002] With the development of technology, agricultural robots are gradually being applied to agricultural production bases, such as fruit planting bases. In these bases, the robots pick up fruit and locate the fruit by capturing images of the planting area. However, when the robot's gripper picks up one fruit, it can affect the position of other fruits, or even cause them to fall. This results in low safety for the current agricultural robots in picking up fruit. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a vision control method, device, and electronic device for agricultural robots. The method regulates the movement trajectory of the gripper head of the agricultural robot according to the gripping plan and simultaneously performs avoidance control, improving the safety of the agricultural robot's gripping of fruit. Furthermore, while the gripper head is gripping fruit, it simultaneously monitors the positional shifts of other fruits in the current environment and determines their falling status based on their shift trajectories, further ensuring the safety of the agricultural robot's gripping of fruit. It also enables timely control of other fruits in a falling state, achieving priority gripping.

[0004] In a first aspect, embodiments of the present invention provide a vision control method based on an agricultural robot, comprising:

[0005] When an agricultural robot enters an agricultural work area, it acquires visual images of the agricultural work area.

[0006] Based on visual images, locate fruit areas and determine the fruit varieties within those areas;

[0007] The agricultural robot's gripper is changed based on the fruit variety, and the compatibility between the current gripper and the fruit is detected.

[0008] If the current fit between the gripper and the fruit meets the preset threshold, the gripping plan is determined based on multiple fruits and their positions in the visual image.

[0009] The movement trajectory of the gripper of the agricultural robot is controlled according to the gripping plan, and the gripper is simultaneously controlled to avoid interference.

[0010] While the clamp is picking up fruit, it simultaneously monitors the positional shifts of other fruits in the current environment and determines the falling status based on the shift trajectories of other fruits.

[0011] According to a specific implementation of an embodiment of the present invention, the step of locating the fruit region based on a visual image and determining the fruit variety located in the fruit region includes:

[0012] The visual image is divided into regions, and the preliminary investigation area is located based on the color difference between the fruit color and the green leaf color.

[0013] Based on the initial area identification of the fruit area, the area corresponding to the color of the fruit is now the fruit area.

[0014] The fruit areas are identified individually, and the fruit varieties in the fruit areas are determined based on the outer contour and color of individual fruits within the fruit areas.

[0015] Obtain the current month's quarter and the corresponding sweetness level of the fruit, and further confirm the fruit variety located in the fruit region.

[0016] According to a specific implementation of an embodiment of the present invention, the step of triggering the changing of the gripper of the agricultural robot based on the fruit variety and detecting the compatibility between the current gripper and the fruit includes:

[0017] Obtain fruit varieties;

[0018] The gripper of the agricultural robot is determined based on a table of fruit varieties and grippers, wherein the table of grippers includes the correspondence between fruit varieties and grippers.

[0019] The chuck replacement of the agricultural robot is triggered based on the fruit variety, and the progress of the chuck replacement is monitored.

[0020] After the gripper of the agricultural robot is replaced, the compatibility between the gripper and the fruit variety is tested. At this point, the gripping area of ​​the gripper covers the outer contour of the fruit.

[0021] According to a specific implementation of an embodiment of the present invention, the step of triggering the changing of the gripper of the agricultural robot based on the fruit variety and detecting the compatibility between the current gripper and the fruit further includes:

[0022] Obtain the outer contour of the fruit;

[0023] Wear detection is performed based on the indentations in the outer contour of the fruit.

[0024] If the fruit has abrasions, locate the abrasions by establishing a coordinate system based on the fruit's center of gravity and determining the coordinates of the abrasions.

[0025] Adjust the gripping position of the clamp according to the abrasion points on the fruit, and avoid the positioning coordinates of the abrasion points on the fruit.

[0026] According to a specific implementation of an embodiment of the present invention, if the fit between the current gripper and the fruit meets a preset threshold, then determining a gripping plan based on multiple fruits and their positions in a visual image includes:

[0027] Get the current compatibility between the clamp and the fruit;

[0028] If the current clamp and the fruit are compatible with the preset threshold, then the current clamp is suitable for picking up the fruit.

[0029] The positional relationships between multiple fruits are constructed based on the visual image, and the movement space between multiple fruits is calculated based on the positional relationships between multiple fruits;

[0030] The gripping plan is determined based on the positional relationship between multiple fruits and the current position of the gripper. The gripping plan prioritizes gripping based on the distance between the gripper and multiple fruits.

[0031] According to a specific implementation of an embodiment of the present invention, if the fit between the current gripper and the fruit meets a preset threshold, then determining the gripping plan based on multiple fruits and their positions in the visual image further includes:

[0032] Obtain individual images of multiple fruits;

[0033] A maturity test was conducted based on individual images of multiple fruits, and the maturity of multiple fruits was determined.

[0034] The ripeness of multiple fruits is recorded sequentially and labeled to the corresponding fruits;

[0035] The ripeness of the fruit and the current position of the gripper are used as parameters for the gripping plan, and these parameters affect the gripping plan.

[0036] According to a specific implementation of an embodiment of the present invention, when the gripper picks up the fruit, it simultaneously monitors the positional shift of other fruits in the current environment and determines the falling state based on the shift trajectories of the other fruits, including:

[0037] While the clamp is picking up fruit, it captures dynamic images of other fruits in real time;

[0038] The location shift of other fruits in the current environment is monitored synchronously based on dynamic images;

[0039] The offset trajectory is determined based on the positional changes of other fruits in the dynamic image;

[0040] Judge the degree of drooping of other fruits based on their offset trajectories;

[0041] If the drooping degree of other fruits meets the preset drooping degree, then the falling state of other fruits is determined.

[0042] According to a specific implementation of an embodiment of the present invention, when the gripper picks up the fruit, it simultaneously monitors the positional shift of other fruits in the current environment and determines the falling state based on the shift trajectories of the other fruits, including:

[0043] If other fruits are in a falling state, the picking plan will be temporarily adjusted, and the fruits in a falling state will be picked up first.

[0044] The gripping order of the clamps is dynamically adjusted according to the degree of drooping of other fruits, and the gripping plan is also dynamically adjusted.

[0045] Secondly, embodiments of the present invention provide a vision control device based on an agricultural robot, comprising:

[0046] The acquisition module is used to acquire visual images of the agricultural work area when the agricultural robot enters the agricultural work area;

[0047] The positioning module is used to locate fruit areas based on visual images and determine the fruit varieties located within those areas.

[0048] The replacement module is used to trigger the replacement of the gripper of the agricultural robot according to the fruit variety and to detect the compatibility between the current gripper and the fruit.

[0049] The gripping module is used to determine the gripping plan based on multiple fruits and their positions in the visual image if the current gripper's fit with the fruit meets a preset threshold.

[0050] The control module is used to control the movement trajectory of the gripper of the agricultural robot according to the gripping plan, and simultaneously control the gripper to avoid obstacles.

[0051] The status module is used to simultaneously monitor the positional shifts of other fruits in the current environment while the gripper is picking up fruit, and to determine the falling status based on the shift trajectories of other fruits.

[0052] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed within the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the method described in any of the foregoing implementations.

[0053] Fourthly, embodiments of the present invention also provide an application program that is executed to implement the method described in any embodiment of the present invention.

[0054] This invention provides a vision control method, device, and electronic device for agricultural robots. The method applies the agricultural robot to an agricultural work area, locates fruit areas based on visual images, and identifies the fruit varieties within those areas. This allows the robot to change its gripper based on the fruit variety. The compatibility between the robot's gripper and the fruit is detected, and a gripping plan is determined based on multiple fruits and their locations in the visual image. The robot controls the movement trajectory of its gripper according to the gripping plan and simultaneously implements avoidance control, improving the safety of the robot's fruit gripping. Furthermore, while the gripper is gripping fruit, it simultaneously monitors the positional shifts of other fruits in the current environment and determines their falling status based on their shift trajectories, further ensuring the safety of the robot's fruit gripping and enabling timely control of other fruits in a falling state, allowing for priority gripping. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating the vision control method based on agricultural robots in an embodiment of the present invention.

[0057] Figure 2 This is a flowchart illustrating step S11 of the vision control method based on agricultural robots in an embodiment of the present invention.

[0058] Figure 3 This is a flowchart illustrating step S12 of the vision control method based on agricultural robots in an embodiment of the present invention.

[0059] Figure 4 This is a flowchart illustrating S13 of the vision control method based on agricultural robots in an embodiment of the present invention.

[0060] Figure 5 This is a flowchart illustrating S14 of the vision control method based on agricultural robots in an embodiment of the present invention.

[0061] Figure 6 This is a flowchart illustrating S16 of the vision control method based on agricultural robots in an embodiment of the present invention.

[0062] Figure 7 This is a schematic diagram of the device composition of the vision control device based on agricultural robots in an embodiment of the present invention;

[0063] Figure 8 This is a hardware diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0064] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0065] This embodiment provides a vision control method based on agricultural robots, so that different network filters can form their own adaptive control combinations to improve the fault handling efficiency of each network filter.

[0066] Figure 1 This is a flowchart illustrating the vision control method for agricultural robots according to Embodiment 1 of the present invention, as shown below. Figures 1 to 7 As shown, this embodiment of the invention provides a vision control method for agricultural robots, the method comprising:

[0067] S11. When the agricultural robot enters the agricultural work area, acquire visual images of the agricultural work area;

[0068] S12. Locate the fruit area based on visual images and determine the fruit variety in the fruit area;

[0069] S13. Trigger the replacement of the gripper of the agricultural robot according to the fruit variety, and detect the compatibility between the current gripper and the fruit;

[0070] S14. If the current fit between the gripper and the fruit meets the preset threshold, then determine the gripping plan based on multiple fruits and their positions in the visual image.

[0071] S15. Adjust the movement trajectory of the gripper of the agricultural robot according to the gripping plan, and simultaneously adjust the gripper to avoid interference.

[0072] S16. When the clamp picks up fruit, it simultaneously monitors the positional shift of other fruits in the current environment and determines the falling status based on the shift trajectory of other fruits.

[0073] This embodiment applies to agricultural robots. The method applies the agricultural robot to an agricultural work area, locates fruit areas based on visual images, and identifies the fruit varieties within those areas. This allows the robot to change its gripper based on the fruit variety. The compatibility between the robot's gripper and the fruit is detected, and a gripping plan is determined based on multiple fruits and their locations in the visual image. The robot's gripper movement trajectory is adjusted according to the gripping plan, and avoidance mechanisms are simultaneously implemented, improving the safety of the robot's fruit gripping. Furthermore, while the gripper is gripping fruit, the positional shifts of other fruits in the current environment are monitored simultaneously, and their falling status is determined based on their shift trajectories. This further ensures the safety of the robot's fruit gripping and allows for timely control of other fruits in a falling state, enabling priority gripping.

[0074] The following is combined with Figures 2 to 7 This invention provides a detailed description of a vision control method for agricultural robots, comprising:

[0075] S11. When the agricultural robot enters the agricultural work area, acquire visual images of the agricultural work area;

[0076] In this process, the agricultural robot enters the agricultural work area from the outside to trigger the robot to grasp the fruit. At this time, visual detection is performed on the agricultural work area, and visual images of the agricultural work area are acquired to enable visual control based on the visual images of the agricultural work area.

[0077] S111, Agricultural robots trigger visual detection when entering agricultural work areas;

[0078] S112. Acquire visual images within the agricultural work area. At this time, the visual images are 360° images, formed by the agricultural robot's camera.

[0079] Among them, the agricultural robot, as a fruit-grabbing robot, moves to the agricultural work area. When the agricultural robot enters the agricultural work area, it triggers corresponding visual detection to enable the agricultural robot to acquire the environment of the agricultural work area and obtain visual images of the agricultural work area. At this time, the visual images are 360° images, which are formed by the detection of the agricultural robot's camera. At this time, the agricultural robot's camera rotates 360° as the agricultural robot moves or stops, and forms visual images.

[0080] S12. Locate the fruit area based on visual images and determine the fruit variety in the fruit area;

[0081] This involves further processing the visual image and locating the fruit region within it to facilitate fruit recognition within that region.

[0082] S121. Divide the visual image into regions and locate the preliminary investigation area based on the color difference between the fruit color and the green leaf color.

[0083] S122. Based on the preliminary investigation of the area, locate the fruit area. At this time, the area corresponding to the color of the fruit is the fruit area.

[0084] S123. Identify the fruit area separately and determine the fruit variety in the fruit area based on the outer contour and color of the individual fruit in the fruit area.

[0085] S124. Obtain the current month's quarter and the corresponding sweetness of the fruit, and further confirm the fruit variety in the fruit region.

[0086] In the further processing of the visual image, the visual image is divided into regions with different colors. At this point, the preliminary screening area is located based on the color difference between the fruit color and the green leaf color, so as to facilitate further calculations on the preliminary screening area and thus locate the fruit area. At this point, the area corresponding to the fruit color is taken as the fruit area, and the location is quickly performed based on the color difference between the fruit color and the green leaf color, so as to speed up the determination of the fruit area.

[0087] Specifically, fruit areas are identified individually, and the fruit variety within each area is determined based on the outline and color of individual fruits. The current month's season and the corresponding sweetness of the fruit are obtained to further confirm the fruit variety within the area. At this point, the fruit variety is not limited to apples, pomegranates, or lychees. The fruit variety is initially identified through the outline and color of individual fruits within the area. To further ensure the certification of the fruit variety, the current month's season and the corresponding sweetness of the fruit are used as auxiliary certification factors to ensure the accuracy of the fruit variety certification.

[0088] S13. Trigger the replacement of the gripper of the agricultural robot according to the fruit variety, and detect the compatibility between the current gripper and the fruit;

[0089] Different fruit varieties are matched with different grippers. By using the appropriate gripper to pick up the corresponding fruit, the smoothness and stability of the agricultural robot's fruit picking is ensured.

[0090] S131. Obtain fruit varieties;

[0091] S132. Determine the gripper of the agricultural robot based on a fruit variety and gripper reference table, wherein the gripper reference table includes the correspondence between fruit varieties and grippers.

[0092] S133. Trigger the chuck replacement of the agricultural robot according to the fruit variety, and monitor the progress of the chuck replacement of the agricultural robot.

[0093] S134. After the gripper of the agricultural robot is replaced, the compatibility between the gripper and the fruit variety is tested. At this time, the gripping area of ​​the gripper covers the outer contour of the fruit.

[0094] The process involves matching fruit varieties with grippers. A fruit variety and gripper mapping table is used to determine the appropriate gripper for the agricultural robot. This table lists the correspondence between fruit varieties and grippers, ensuring compatibility. The gripper is then adjusted based on this mapping. Gripper replacement is triggered based on the fruit variety, and the replacement progress is monitored. After the gripper replacement is complete, the compatibility between the gripper and the fruit variety is tested. At this point, the gripping area covers the outer contour of the fruit, ensuring stable gripping by the gripper.

[0095] In addition, the step of triggering the changing of the gripper of the agricultural robot based on the fruit variety and detecting the compatibility between the current gripper and the fruit also includes: acquiring the outer contour of the fruit; performing wear detection based on the depressions in the outer contour of the fruit; if there are wear areas on the fruit, locating the wear areas, wherein a coordinate system is established based on the center of gravity of the fruit, and the positioning coordinates of the wear areas are determined; adjusting the gripping position of the gripper according to the wear areas of the fruit, and avoiding the positioning coordinates of the wear areas, thereby avoiding contact between the gripper and the wear areas of the fruit, preventing further wear on the wear areas of the fruit, and ensuring the safety of the gripper in grasping the fruit.

[0096] S14. If the current fit between the gripper and the fruit meets the preset threshold, then determine the gripping plan based on multiple fruits and their positions in the visual image.

[0097] S141. Obtain the current compatibility between the clamp and the fruit;

[0098] S142. If the current clamp and the fruit are compatible with the preset threshold, then the current clamp is suitable for picking up the fruit.

[0099] S143. Construct the positional relationship between multiple fruits based on the visual image, and calculate the movement space between multiple fruits based on the positional relationship between multiple fruits;

[0100] S144. Determine a gripping plan based on the positional relationship between multiple fruits and the current position of the gripper, wherein the gripping plan prioritizes gripping based on the distance between the gripper and multiple fruits.

[0101] If the current clamp and the fruit's fit meets a preset threshold, then the current clamp is suitable for gripping the fruit, ensuring the fit between the current clamp and the fruit. At this point, the positional relationship between multiple fruits is constructed based on the visual image, and the movement space between multiple fruits is calculated based on the positional relationship between multiple fruits. The movement range of the clamp can be constructed through the movement space, so as to determine the gripping plan based on the positional relationship between multiple fruits and the current position of the clamp. The gripping plan prioritizes gripping based on the distance between the clamp and multiple fruits, ensuring the principle of proximity priority, and under this principle, gripping planning can be performed on fruits in local areas.

[0102] In addition, if the compatibility between the current clamp and the fruit meets a preset threshold, then determining the clamping plan based on multiple fruits and their positions in the visual image further includes: acquiring individual images of multiple fruits; performing a maturity test based on the individual images of multiple fruits and determining the maturity of multiple fruits; sequentially registering the maturity of multiple fruits and marking them to the corresponding fruits; using the maturity of the fruit and the position of the current clamp as a parameter for the clamping plan, and influencing the clamping plan. At this point, the maturity of multiple fruits is introduced, and the maturity of multiple fruits is used as a planning factor, thereby using the maturity of the fruit and the position of the current clamp as a parameter for the clamping plan, and influencing the clamping plan, thereby adjusting the clamping order of the clamp.

[0103] S15. Adjust the movement trajectory of the gripper of the agricultural robot according to the gripping plan, and simultaneously adjust the gripper to avoid interference.

[0104] At this point, the gripping plan is obtained; the movement trajectory of the gripper of the agricultural robot is adjusted according to the gripping plan, and the gripping order of each fruit is determined; the gripper of the agricultural robot moves along the gripper movement trajectory, and the gripper is controlled to avoid obstacles. At this time, the gripper avoids obstacles and grips each fruit in the order of gripping.

[0105] S16. When the clamp picks up the fruit, it simultaneously monitors the positional shift of other fruits in the current environment and determines the falling status based on the shift trajectory of other fruits.

[0106] S161. While the clamp is picking up fruit, acquire dynamic images of other fruits in real time;

[0107] S162. Based on dynamic images, synchronously monitor the positional shifts of other fruits in the current environment;

[0108] S163. Determine the offset trajectory based on the positional changes of other fruits in the dynamic image;

[0109] S164. Determine the degree of drooping of other fruits based on their offset trajectories;

[0110] S165. If the degree of drooping of other fruits meets the preset degree of drooping, then determine the falling state of other fruits.

[0111] In this process, while the clamp is picking up a fruit, other fruits are observed simultaneously, and dynamic images of the other fruits are acquired in real time to facilitate monitoring of these dynamic images. At this time, the positional shift of other fruits in the current environment is monitored synchronously based on the dynamic images. Other fruits undergo positional changes during the picking process, and the offset trajectory is determined based on the positional changes of other fruits in the dynamic images.

[0112] At this point, the degree of drooping of other fruits is determined based on their offset trajectories. If the degree of drooping of other fruits meets the preset drooping degree, the falling state of other fruits is determined. If other fruits are in a falling state, the clamping plan is temporarily adjusted, and fruits in a falling state are clamped first. The clamping order of the clamp is dynamically adjusted according to the degree of drooping of other fruits, and the clamping plan is dynamically adjusted.

[0113] like Figure 7 As shown in the figure, this embodiment of the invention also provides a vision control device based on an agricultural robot, the device comprising:

[0114] The acquisition module 21 is used to acquire visual images of the agricultural work area when the agricultural robot enters the agricultural work area;

[0115] The positioning module 22 is used to locate the fruit area based on the visual image and determine the fruit variety in the fruit area;

[0116] The replacement module 23 is used to trigger the replacement of the gripper of the agricultural robot according to the fruit variety and to detect the compatibility between the current gripper and the fruit.

[0117] The gripping module 24 is used to determine the gripping plan based on multiple fruits and their positions in the visual image if the current gripper and the fruit meet a preset threshold.

[0118] The control module 25 is used to control the movement trajectory of the gripper of the agricultural robot according to the gripping plan, and simultaneously control the gripper to avoid obstacles.

[0119] The status module 26 is used to simultaneously monitor the positional shift of other fruits in the current environment when the clamp is picking up the fruit, and to determine the falling status based on the offset trajectory of other fruits.

[0120] Figure 8This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention, which implements the present invention. Figure 1-3 The process of the illustrated embodiment is as follows: Figure 7 As shown, the aforementioned electronic device includes: a housing 41, a processor 42, a memory 43, a circuit board 44, and a power supply circuit 45. The circuit board 44 is disposed inside the space enclosed by the housing 41, and the processor 42 and the memory 43 are disposed on the circuit board 44. The power supply circuit 45 is used to supply power to the various circuits or devices of the aforementioned electronic device. The memory 43 is used to store executable program code. The processor 42 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 43, for executing the vision control method based on agricultural robots described in any of the foregoing embodiments.

[0121] For details on the specific execution process of the above steps by processor 42, and the steps further executed by processor 42 through running executable program code, please refer to the present invention. Figure 1-3 The description of the illustrated embodiments will not be repeated here.

[0122] This electronic device exists in various forms, including but not limited to:

[0123] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0124] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0125] (3) Portable entertainment devices: These devices display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0126] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0127] (5) Other electronic devices with data interaction functions.

[0128] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores one or more programs that can be executed by one or more processors to implement the aforementioned vision control method based on agricultural robots. For example, the computer-readable storage medium is a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0129] In addition, embodiments of the present invention also provide an application program that is executed to implement the methods provided in any embodiment of the present invention.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0132] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0133] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module are implemented in one or more software and / or hardware.

[0134] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments is accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0135] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A vision control method for agricultural robots, characterized in that, include: When an agricultural robot enters an agricultural work area, it acquires visual images of the agricultural work area. Based on visual images, locate fruit areas and determine the fruit varieties within those areas; The agricultural robot's gripper is changed based on the fruit variety, and the compatibility between the current gripper and the fruit is detected. If the current fit between the gripper and the fruit meets the preset threshold, the gripping plan is determined based on multiple fruits and their positions in the visual image. The movement trajectory of the gripper of the agricultural robot is controlled according to the gripping plan, and the gripper is simultaneously controlled to avoid interference. While the clamp is picking up fruit, the positional shift of other fruits in the current environment is monitored simultaneously, and the falling status is determined based on the shift trajectory of other fruits. The process of simultaneously monitoring the positional shifts of other fruits in the current environment while the clamp is picking up the fruit, and determining the falling status based on the shift trajectories of the other fruits, includes: While the clamp is picking up fruit, it captures dynamic images of other fruits in real time; The location shift of other fruits in the current environment is monitored synchronously based on dynamic images; The offset trajectory is determined based on the positional changes of other fruits in the dynamic image; Judge the degree of drooping of other fruits based on their offset trajectories; If the drooping degree of other fruits meets the preset drooping degree, then the falling state of other fruits is determined; If other fruits are in a falling state, the picking plan will be temporarily adjusted, and the fruits in a falling state will be picked up first. The gripping order of the clamps is dynamically adjusted according to the degree of drooping of other fruits, and the gripping plan is also dynamically adjusted.

2. The vision control method for agricultural robots according to claim 1, characterized in that, The step of locating fruit regions based on visual images and determining the fruit varieties within those regions includes: The visual image is divided into regions, and the preliminary investigation area is located based on the color difference between the fruit color and the green leaf color. Based on the initial area identification of the fruit area, the area corresponding to the color of the fruit is now the fruit area. The fruit areas are identified individually, and the fruit varieties in the fruit areas are determined based on the outer contour and color of individual fruits within the fruit areas. Obtain the current month's quarter and the corresponding sweetness level of the fruit, and further confirm the fruit variety located in the fruit region.

3. The vision control method for agricultural robots according to claim 2, characterized in that, The process of triggering the changing of the agricultural robot's gripper based on the fruit variety and detecting the compatibility between the current gripper and the fruit includes: Obtain fruit varieties; The gripper of the agricultural robot is determined based on a table of fruit varieties and grippers, wherein the table of grippers includes the correspondence between fruit varieties and grippers. The chuck replacement of the agricultural robot is triggered based on the fruit variety, and the progress of the chuck replacement is monitored. After the gripper of the agricultural robot is replaced, the compatibility between the gripper and the fruit variety is tested. At this point, the gripping area of ​​the gripper covers the outer contour of the fruit.

4. The vision control method for agricultural robots according to claim 3, characterized in that, The step of triggering the changing of the agricultural robot's gripper based on the fruit variety and detecting the compatibility between the current gripper and the fruit also includes: Obtain the outer contour of the fruit; Wear detection is performed based on the indentations in the outer contour of the fruit. If the fruit has abrasions, locate the abrasions by establishing a coordinate system based on the fruit's center of gravity and determining the coordinates of the abrasions. Adjust the gripping position of the clamp according to the abrasion points on the fruit, and avoid the positioning coordinates of the abrasion points on the fruit.

5. The vision control method for agricultural robots according to claim 4, characterized in that, If the current gripper's fit with the fruit meets a preset threshold, then a gripping plan is determined based on multiple fruits and their positions in the visual image, including: Get the current compatibility between the clamp and the fruit; If the current clamp and the fruit are compatible with the preset threshold, then the current clamp is suitable for picking up the fruit. The positional relationships between multiple fruits are constructed based on the visual image, and the movement space between multiple fruits is calculated based on the positional relationships between multiple fruits; The gripping plan is determined based on the positional relationship between multiple fruits and the current position of the gripper. The gripping plan prioritizes gripping based on the distance between the gripper and multiple fruits.

6. The vision control method for agricultural robots according to claim 5, characterized in that, If the fit between the current gripper and the fruit meets a preset threshold, then determining the gripping plan based on multiple fruits and their positions in the visual image further includes: Obtain individual images of multiple fruits; A maturity test was conducted based on individual images of multiple fruits, and the maturity of multiple fruits was determined. The ripeness of multiple fruits is recorded sequentially and labeled to the corresponding fruits; The ripeness of the fruit and the current position of the gripper are used as parameters for the gripping plan, and these parameters affect the gripping plan.

7. A vision control device based on agricultural robots, characterized in that, include: The acquisition module is used to acquire visual images of the agricultural work area when the agricultural robot enters the agricultural work area; The positioning module is used to locate fruit areas based on visual images and determine the fruit varieties located within those areas. The replacement module is used to trigger the replacement of the gripper of the agricultural robot according to the fruit variety and to detect the compatibility between the current gripper and the fruit. The gripping module is used to determine the gripping plan based on multiple fruits and their positions in the visual image if the current gripper's fit with the fruit meets a preset threshold. The control module is used to control the movement trajectory of the gripper of the agricultural robot according to the gripping plan, and simultaneously control the gripper to avoid obstacles. The status module is used to simultaneously monitor the positional shift of other fruits in the current environment while the clamp is picking up fruit, and to determine the falling status based on the shift trajectory of other fruits. The process of simultaneously monitoring the positional shifts of other fruits in the current environment while the clamp is picking up the fruit, and determining the falling status based on the shift trajectories of the other fruits, includes: While the clamp is picking up fruit, it captures dynamic images of other fruits in real time; The location shift of other fruits in the current environment is monitored synchronously based on dynamic images; The offset trajectory is determined based on the positional changes of other fruits in the dynamic image; Judge the degree of drooping of other fruits based on their offset trajectories; If the drooping degree of other fruits meets the preset drooping degree, then the falling state of other fruits is determined; If other fruits are in a falling state, the picking plan will be temporarily adjusted, and the fruits in a falling state will be picked up first. The gripping order of the clamps is dynamically adjusted according to the degree of drooping of other fruits, and the gripping plan is also dynamically adjusted.

8. An electronic device, characterized in that, The electronic device includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the vision control method based on agricultural robots as described in any one of claims 1 to 6.

Images