Banana bud suction cutting point rapid and accurate positioning method based on machine vision

Abstract

The invention discloses a method for quickly and accurately positioning banana bud suction cutting points based on machine vision. The method specifically comprises the steps that real-time RGB images and depth data of bananas are collected, the RGB images and the depth data are aligned, the aligned image data are input into a trained optimal key point detection model, and two auxiliary key points are detected through the optimal key point detection model; calculating a midpoint of the two auxiliary key points as a cutting point of banana bud suction; utilizing a neighbor interpolation algorithm to extract depth values of the cutting points from the aligned depth data; and the image coordinates and the depth values of the cutting points are converted into three-dimensional coordinates in a camera coordinate system, and the converted three-dimensional coordinates are transmitted to a central control system, so that the operation robot is controlled to carry out banana suction bud cutting work. According to the method for rapidly and accurately positioning the cutting points of the banana suction buds, the cutting points of the banana suction buds can be rapidly and accurately positioned, so that the cutting efficiency and the cutting precision are improved.

Description

Technical Field

[0001] The invention belongs to the field of agricultural automation, and in particular relates to a method for quickly and accurately locating a banana sucking bud cutting point based on machine vision. Background Art

[0002] Bananas have the habit of growing suckers, which compete with the mother plant for nutrients, thus affecting the growth of the mother plant. In banana planting management, regular removal of banana suckers is essential for the healthy growth of banana plants and increasing yields. The traditional method of cutting banana suckers relies on manual experience, and the cutting accuracy is not high, which can easily hurt the mother plant by mistake, thus affecting the growth of the mother plant. In addition, manual operation is inefficient, and in the case of large-scale planting, it is difficult to meet the requirements of modern agriculture for efficient and sustainable production. Therefore, it is urgent to adopt innovative management and technical means to improve the efficiency and accuracy of sucker removal.

[0003] Machine vision technology is widely used in automated harvesting to accurately locate the position of the fruit and guide the robot to perform efficient harvesting operations. In the existing banana sucker removal technology, machine vision technology has not yet been used to accurately locate the sucker cutting point. Therefore, it is particularly important to develop a cutting point positioning method based on machine vision. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision. The method for quickly and accurately locating the cutting point of a banana sucking bud can quickly and accurately locate the cutting point of the banana sucking bud, thereby improving the cutting efficiency and accuracy.

[0005] The technical solution of the present invention to solve the above technical problems is:

[0006] A method for quickly and accurately locating a banana bud cutting point based on machine vision comprises the following steps:

[0007] S1: Use an image acquisition device to collect real-time RGB images and depth data of the banana, align the RGB image with the depth data, and obtain aligned image data;

[0008] S2: Input the aligned image data into the trained optimal key point detection model, and detect two auxiliary key points through the optimal key point detection model;

[0009] S3: Calculate the midpoint of the two auxiliary key points as the cutting point of the banana sucker;

[0010] S4: using the nearest neighbor interpolation algorithm, extracting the depth value of the cutting point of the banana sucker from the aligned depth data;

[0011] S5: using the camera's internal parameters, converting the image coordinates and depth values ​​of the cutting points of the banana sucking buds into three-dimensional coordinates in the camera coordinate system;

[0012] S6: The converted three-dimensional coordinates in the camera coordinate system are transmitted to the central control system, and the central control system controls the working robot to perform the removal of the banana sucking buds based on the three-dimensional coordinates in the camera coordinate system.

[0013] Preferably, in step S1, the image acquisition device is a binocular depth camera.

[0014] Preferably, in step S1, the RGB image is aligned with the depth data by using the SDK provided by the binocular depth camera to ensure that the RGB value and depth value of each pixel point correspond to the same physical position.

[0015] Preferably, in step S2, the steps of constructing the preferred key point detection model are:

[0016] S201: Collect banana images and pre-process the banana images to construct an image dataset;

[0017] S202: Divide the image dataset into a training set, a test set, and a validation set in a ratio of 8:1:1;

[0018] S203: labeling key points of banana images in the training set and the validation set;

[0019] S204: setting corresponding hyperparameters, and sending the labeled training set to multiple different types of YOLO neural network models; training each YOLO neural network model through the training set, and then evaluating it through the test set, comparing the mAP@.5 values ​​of different types of YOLO neural network models in the banana sucking bud key point detection task, so as to select the YOLO neural network model with the best performance in the banana sucking bud key point detection task as the preferred key point detection model.

[0020] Preferably, in step S201, the preprocessing process is: randomly cropping the collected original image according to a 1:1 aspect ratio.

[0021] Preferably, in step S203, the key point markings include standard sucker key point markings and regenerated sucker key point markings.

[0022] Preferably, in step S204, the YOLO neural network models used include YOLOv3, YOLOv5, YOLOv6, YOLOv8 and YOLOv9.

[0023] Preferably, in step S204, the hyper parameters are: the training batch size is 32, the training rounds are 300, the optimizer is AdamW, and the learning rate is 0.001429.

[0024] An electronic device comprises a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for quickly and accurately locating a banana sucking bud cutting point based on machine vision.

[0025] A computer-readable storage medium stores a computer program implemented according to the method for quickly and accurately locating a banana sucking bud cutting point based on machine vision in the form of computer-readable instructions. When the computer program is called and run by a computer, the steps included in the corresponding method are executed.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] 1. The method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision of the present invention can quickly and accurately locate the cutting point of a banana sucking bud, thereby improving cutting efficiency and cutting accuracy.

[0028] 2. The method for quickly and accurately locating the cutting point of banana sucking buds based on machine vision of the present invention improves the cutting accuracy and efficiency, is adaptable to banana sucking buds of different shapes and sizes, has high real-time performance and robustness, and is suitable for modern agricultural automation management.

[0029] 3. The method for quickly and accurately locating the cutting point of banana sucking buds based on machine vision of the present invention uses machine vision technology to achieve accurate positioning of the cutting point of banana sucking buds, performs key point detection through a deep learning model, and combines the RGBD data obtained by the depth camera to calculate the three-dimensional coordinates of the cutting point, providing accurate operation guidance for the automation equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flow chart of a method for quickly and accurately locating a banana sucking bud cutting point based on machine vision.

[0031] Figure 2 Schematic diagram with key points marked.

[0032] Figure 3 Schematic diagram of the Mixup data enhancement process.

[0033] Figure 4 This is the network structure diagram of YOLOv8-pose.

[0034] Figure 5 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0036] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0038] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playback function, or a smart TV, a set-top box and other devices.

[0039] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0040] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0041] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0042] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0043] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0044] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0045] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0046] See also Figure 1-Figure 4 The method for quickly and accurately locating the cutting point of a banana bud based on machine vision of the present invention comprises the following steps:

[0047] S1: Use an image acquisition device to collect real-time RGB images and depth data of the banana, align the RGB image with the depth data, and obtain aligned image data;

[0048] In this embodiment, the Intel Realsense D534 binocular depth camera is used to collect real-time RGB images and depth data to ensure data synchronization. At the same time, the SDK provided by the camera is used to align the RGB image with the depth data to ensure that the RGB value and depth value of each pixel correspond to the same physical position. The aligned data will be used for subsequent key point detection and depth information extraction.

[0049] S2: Input the aligned image data into the trained optimal key point detection model, and detect two auxiliary key points through the optimal key point detection model;

[0050] In this embodiment, the two auxiliary key points are l_cut and r_cut, and their image coordinates are (x1, y1) (x2, y2) respectively.

[0051] S3: Calculate the midpoint of the two auxiliary key points as the cutting point of the banana sucker; the calculation formula of the midpoint is:

[0052] ;

[0053] S4: Since the nearest neighbor interpolation algorithm can effectively process holes and noise in the depth data, thereby providing a more accurate depth value, the nearest neighbor interpolation algorithm is used to extract the depth value of the cutting point of the banana sucker from the aligned depth data;

[0054] S5: using the camera's internal parameters (such as focal length, principal point coordinates), converting the image coordinates and depth values ​​of the cutting point of the banana sucking bud into three-dimensional coordinates in the camera coordinate system;

[0055] ;

[0056] ;

[0057] ;

[0058] Where (u, v) is the image coordinate, are the principal point coordinates of the image, , is the focal length of the camera, Z is the depth value, and (X, Y, Z) is the three-dimensional coordinate in the camera coordinate system.

[0059] S6: The converted three-dimensional coordinates in the camera coordinate system are transmitted to the central control system, and the central control system controls the working robot to perform the removal of the banana sucking buds based on the three-dimensional coordinates in the camera coordinate system.

[0060] Through the above steps, the present invention can realize the precise positioning of the cutting point of the banana sucker bud and provide reliable coordinate information for the automated banana sucker bud cutting, thereby improving the precision and efficiency of the banana sucker bud cutting, and is suitable for the practical application of agricultural automation equipment.

[0061] In step S2, the steps of constructing the preferred key point detection model are:

[0062] S201: Collect banana images and pre-process the banana images to construct an image dataset;

[0063] In this embodiment, a Sony a5100 digital camera is used for data acquisition, and the resolution of the collected original image is 6000x4000. The exposure mode of the camera is set to automatic exposure during shooting. 507 original images are taken in a real banana plantation, but because the original image is too large, it will be compressed when input into the deep learning network, and a large number of detailed features are lost. Therefore, the original image needs to be randomly cropped according to the aspect ratio of 1:1 to obtain 1202 images.

[0064] S202: Divide the image dataset into a training set, a test set, and a validation set in a ratio of 8:1:1;

[0065] In this embodiment, the obtained 1202 images are divided into a training set, a test set and a validation set in a ratio of 8:1:1, which have 964, 119 and 119 images respectively.

[0066] S203: labeling key points of banana images in the training set and the validation set;

[0067] By analyzing the botanical morphology of suckers and simulating the artificial removal of suckers, the above-ground part of the suckers is removed as much as possible. The constructed image dataset contains two categories: standard suckers and regenerated suckers, and the key point annotation positions are slightly different. Figure 2 Mark the key points with a schematic diagram, such as Figure 2 As shown in the figure, the key points of the standard sucker are located at the left and right points where it intersects with the ground, and the key points of the regenerated sucker are located at the two points where the regenerated part intersects with the original sucker. The image annotation work is completed in the X-AnyLabeling open source software, and the annotation file is saved in .txt format. Each line in the file saves the category of a target (c), the normalized center coordinates and width and height of the rectangular box (cx, cy, w, h) and the normalized coordinates of two sets of key points (x1, y1, x2, y2).

[0068] S204: setting corresponding hyperparameters, and sending the labeled training set to multiple different types of YOLO neural network models; training each YOLO neural network model through the training set, and then evaluating it through the test set, comparing the mAP@.5 values ​​of different types of YOLO neural network models in the banana sucking bud key point detection task, so as to select the YOLO neural network model with the best performance in the banana sucking bud key point detection task as the preferred key point detection model.

[0069] like Figure 3 As shown in the figure, the mixup data enhancement strategy is used to mix two images and their labels to generate new training samples; this method helps to improve the generalization ability of the model in complex agricultural environments; key point detection heads are added to YOLOv3, YOLOv5, YOLOv6, YOLOv8, and YOLOv9 respectively, and the above image datasets are used for training; in this process, in order to achieve efficient detection in a resource-limited environment while maintaining low computational cost and fast inference speed, the n-scale model is selected for training;

[0070] In this embodiment, the training hardware device is an Nvidia RTX 3090 graphics card with 24G video memory, the training batch size is 32, the training rounds are 300, the optimizer uses AdamW, the learning rate is 0.001429, the mixup data enhancement parameter α is 0.4, and the other parameters are default parameters.

[0071] Evaluated through the test set, Compare Different types of YOLO neural network models In the key point detection task mAP@.5 values, the comparison results are shown in Table 1 As shown:

[0072] Table 1: mAP@0.5 for different types of YOLO neural network models

[0073] Model Bud sucking mAP@0.5 Resurrection bud mAP@0.5 Bounding box mAP@0.5 YOLOv3t-pose 0.852 0.885 0.869 YOLOv5n-pose 0.881 0.860 0.871 YOLOv6n-pose 0.901 0.865 0.883 YOLOv9t-pose 0.802 0.697 0.75 YOLOv8n-pose 0.884 0.881 0.883

[0074] As can be seen from Table 1, in this embodiment, the model YOLOv8n-pose (whose network structure is as follows) which performs best in the banana sucking bud key point detection task is selected. Figure 4 ), as the core detection model for subsequent steps.

[0075] Finally, the method for quickly and accurately locating the cutting point of banana sucking buds based on machine vision of the present invention has the following advantages:

[0076] (1) Strong adaptability: By analyzing the morphological structure of banana suckers and traditional cutting methods, a detection and positioning strategy suitable for this task is designed, enabling the system to adapt to banana suckers of different shapes and sizes.

[0077] (2) High robustness: The combination of depth cameras and deep learning models enables the system to maintain high detection accuracy in the complex environment of banana plantations, enhancing the robustness of the system.

[0078] (3) Strong real-time performance: The YOLOv8-pose model completes detection in a single forward propagation. Combined with the rapid processing of depth information, the entire system has strong real-time performance and can respond quickly in dynamic environments.

[0079] Based on any embodiment of this application, please refer to Figure 5 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 5 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0080] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for quickly and accurately locating the banana sucking bud cutting point based on machine vision as described in any embodiment of the present application.

[0081] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision as described in any embodiment of the present application.

[0082] A person skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiments of the present application can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0083] The above is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited to the above content. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for quickly and accurately locating the cutting point of banana buds based on machine vision, characterized in that: The following steps are involved: S1: Use an image acquisition device to collect real-time RGB images and depth data of the banana, align the RGB image with the depth data, and obtain aligned image data; S2: Input the aligned image data into the trained optimal key point detection model, and detect two auxiliary key points through the optimal key point detection model; S3: Calculate the midpoint of the two auxiliary key points as the cutting point of the banana sucker; S4: using the nearest neighbor interpolation algorithm, extracting the depth value of the cutting point of the banana sucker from the aligned depth data; S5: using the camera's internal parameters, converting the image coordinates and depth values ​​of the cutting points of the banana sucking buds into three-dimensional coordinates in the camera coordinate system; S6: The converted three-dimensional coordinates in the camera coordinate system are transmitted to the central control system, and the central control system controls the working robot to perform the removal of the banana sucking buds based on the three-dimensional coordinates in the camera coordinate system.

2. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 1, characterized in that: In step S1, the image acquisition device is a binocular depth camera.

3. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 1, characterized in that: In step S1, the RGB image and the depth data are aligned by using the SDK provided by the binocular depth camera to ensure that the RGB value and depth value of each pixel point correspond to the same physical position.

4. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 1, characterized in that: In step S2, the steps of constructing the preferred key point detection model are: S201: Collect banana images and pre-process the banana images to construct an image dataset; S202: Divide the image dataset into a training set and a validation set in a ratio of 8:2; S203: labeling key points of banana images in the training set and the validation set; S204: setting corresponding hyperparameters, and sending the labeled training set to multiple different types of YOLO neural network models; training each YOLO neural network model through the training set, and then evaluating it through the test set, comparing the mAP@.5 values ​​of different types of YOLO neural network models in the banana sucking bud key point detection task, so as to select the YOLO neural network model with the best performance in the banana sucking bud key point detection task as the preferred key point detection model.

5. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 4, characterized in that: In step S201, the preprocessing process is: randomly cropping the collected original image according to a 1:1 aspect ratio.

6. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 4, characterized in that: In step S203, the key point marking includes standard sucker key point marking and regenerated sucker key point marking.

7. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 4, characterized in that: In step S204, the YOLO neural network models used include YOLOv3, YOLOv5, YOLOv6, YOLOv8 and YOLOv9.

8. The method for quickly and accurately locating the cutting point of banana buds based on machine vision according to claim 4, characterized in that: In step S204, the hyper parameters are: the training batch size is 32, the training rounds are 300, the optimizer is AdamW, and the learning rate is 0.001429.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer program implemented by the method for quickly and accurately locating the cutting point of a banana sucking bud based on machine vision according to any one of claims 1 to 8 is stored in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images