Method, device and system for identifying and positioning oil palm clusters and storage medium
By combining image processing technology with RGBD and RGB cameras, along with target and maturity recognition models, the problem of identifying and locating oil palm fruit ears during mechanical harvesting has been solved. This has enabled automated and accurate fruit ear identification and maturity determination, improving harvesting efficiency and accuracy.
Patent Information
- Application Number
- CN202511741659.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-02-27
AI Technical Summary
Mechanical harvesting of oil palm fruit bunches requires a large amount of manual identification and operation, and it is difficult to avoid missed or incorrect harvesting due to inconsistent human judgment standards.
The first image, which includes RGB channel image and D channel depth data, is acquired using an RGBD camera. The second image, which has a wider dynamic range, is then acquired using an RGB camera. The target recognition and maturity recognition models are used to identify, locate, and determine the maturity of the oil palm ears. Image registration technology is then used for precise positioning.
It enables simple and efficient identification and positioning of oil palm fruit bunches, reduces manual intervention, improves harvesting efficiency and accuracy, and avoids missed or incorrect harvesting.
Smart Images
Figure CN121582785A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the agricultural field, and in particular to a method, apparatus, system, and storage medium for identifying and locating oil palm fruit ears. Background Technology
[0002] Mechanical harvesting of oil palm fruit bunches is more efficient and saves manpower compared to traditional manual harvesting. However, it still requires personnel to manually identify the oil palm fruit bunches before manually operating the machinery to harvest them.
[0003] However, the efficiency of mechanical harvesting depends on human identification and operation of the machinery, which still requires a large amount of manpower.
[0004] Moreover, it is difficult to avoid the under-harvesting or misharvesting of oil palm due to different workers' different understanding of the maturity judgment criteria. Summary of the Invention
[0005] In view of the above-mentioned problems in the prior art, the purpose of the present invention is to provide a method, device, system and storage medium for identifying and locating oil palm fruit ears, which can easily and efficiently identify and locate oil palm fruit ears.
[0006] To address the aforementioned problems, the first aspect of this invention provides a method for identifying and locating oil palm fruit ears, the method comprising:
[0007] Step S1: Obtain a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data.
[0008] Step S2: Based on the first image, identify and locate the oil palm fruit bunch.
[0009] Furthermore, the identification and positioning method also includes:
[0010] Step S3: Acquire a second image containing the oil palm fruit bunch, and identify and determine the maturity of the oil palm fruit bunch based on the second image. The second image is captured by an RGB camera, whose dynamic range is greater than that of the RGBD camera.
[0011] When the oil palm fruit bunch is determined to be mature, the mature oil palm fruit bunch is located based on the first image and the second image.
[0012] Further, step S2 includes:
[0013] The oil palm fruit bunches in the first image are identified, and the identified oil palm fruit bunches are labeled.
[0014] Step S3 includes:
[0015] Obtain the second image;
[0016] The oil palm fruit bunches in the second image are identified, and the identified oil palm fruit bunches are labeled a second time.
[0017] Determine whether the oil palm fruit bunches marked in the second annotation are mature;
[0018] When it is determined that the oil palm fruit bunch in the second annotation is in a mature state, find the oil palm fruit bunch in the first annotation in the RGB channel image corresponding to the mature oil palm fruit bunch, and use it as the predetermined fruit bunch;
[0019] The predetermined ear of fruit is located by mapping the RGB channel image and the D channel depth data of the predetermined ear of fruit.
[0020] Furthermore, the oil palm fruit ears in the RGB channel image are identified by the first target recognition model, and the oil palm fruit ears in the second image are identified by the second target recognition model.
[0021] Furthermore, the first target recognition model is trained using the following method:
[0022] Acquire multiple first image samples;
[0023] Label the oil palm fruit ears in each of the first image samples;
[0024] Deep learning is performed on the RGB channel images of the labeled ears of fruit to obtain the first target recognition model;
[0025] The second target recognition model was trained using the following method:
[0026] Acquire multiple second image samples;
[0027] Label the oil palm fruit bunches in each of the second image samples;
[0028] Deep learning is performed on the second image after the ear of fruit is labeled to obtain the second target recognition model.
[0029] Furthermore, the maturity identification model is used to determine whether the oil palm fruit ears labeled in the second step are mature.
[0030] Furthermore, the maturity recognition model is formed through the following training method:
[0031] Acquire multiple oil palm fruit bunch image samples;
[0032] The maturity level of the oil palm fruit bunches is marked according to the maturity classification standard.
[0033] Deep learning was performed on the oil palm fruit ear image samples after maturity labeling to obtain a maturity recognition model.
[0034] A second aspect of the present invention provides an identification and positioning device for oil palm fruit ears, the identification and positioning device comprising:
[0035] The first acquisition module is used to acquire a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data.
[0036] The first identification and positioning module is used to identify and locate the oil palm fruit ears based on the first image.
[0037] A third aspect of the present invention provides an identification and positioning system for oil palm fruit ears, the identification and positioning system comprising:
[0038] An RGBD camera is used to acquire a first image of the oil palm, the first image including RGB channel image and D channel depth data;
[0039] The processor is configured to acquire a first image containing oil palm fruit ears, the first image being captured by an RGBD camera, the first image including RGB channel image and D channel depth data; and to identify and locate the oil palm fruit ears based on the first image.
[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the oil palm fruit ear identification and positioning method described in any one of the preceding claims.
[0041] Due to the above technical solution, the present invention has the following beneficial effects:
[0042] According to an embodiment of the present invention, the method for identifying and locating oil palm fruit ears involves capturing images of the oil palm fruit ears using an RGBD camera to obtain a first image. The first image includes RGB channel image data and D channel depth data. The oil palm fruit ears are identified using the RGB channel image data and located using the D channel depth data. This method enables simple and efficient identification and location of oil palm fruit ears, facilitating automated harvesting by harvesting equipment. Attached Figure Description
[0043] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.
[0044] Figure 1 This is a flowchart of a method for identifying and locating oil palm fruit ears according to an embodiment of the present invention;
[0045] Figure 2 This is a flowchart of a method for identifying and locating oil palm fruit ears according to another embodiment of the present invention;
[0046] Figure 3 This is a schematic diagram of an oil palm fruit bunch identification and positioning device according to an embodiment of the present invention;
[0047] Figure 4 This is a schematic diagram of an oil palm fruit bunch identification and positioning device according to another embodiment of the present invention. Detailed Implementation
[0048] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0050] To make the objectives, technical solutions, and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. The acquisition, storage, use, and processing of data in the technical solutions of the embodiments of the present invention all comply with the relevant provisions of national laws and regulations.
[0051] The following describes the method for identifying and locating oil palm fruit ears according to an embodiment of the present invention.
[0052] like Figure 1 and Figure 2 As shown, the method for identifying and locating oil palm fruit ears according to an embodiment of the present invention includes:
[0053] Step S1: Acquire a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data. The RGBD camera is a known technology and will not be described further here.
[0054] An RGBD camera photographs the area containing oil palm fruit bunches, producing the first image. The first image includes RGB channel image data and D channel depth data.
[0055] Step S2: Based on the first image, identify and locate the oil palm fruit bunches.
[0056] Oil palm fruit ears are identified using RGB channel images and located using D channel depth data.
[0057] The above method for identifying and locating oil palm fruit bunches involves capturing images of the bunches using an RGBD camera to obtain a first image. This first image includes RGB channel image data and D channel depth data. The oil palm fruit bunches are identified using the RGB channel image and located using the D channel depth data. This method provides a simple and efficient way to identify and locate oil palm fruit bunches, facilitating automated harvesting by harvesting equipment.
[0058] In some embodiments of the present invention, such as Figure 2 As shown, the identification and positioning method further includes step S3. Step S3 involves acquiring a second image containing the oil palm fruit ear, and identifying and determining the maturity of the oil palm fruit ear based on the second image. The second image is captured by an RGB camera, whose dynamic range is greater than that of an RGBD camera. When the oil palm fruit ear is determined to be mature, the mature oil palm fruit ear is located based on the first and second images.
[0059] A second image is obtained by photographing the area containing oil palm fruit ears using an RGB camera. The RGB camera has a wider dynamic range than the RGBD camera, resulting in a second image with greater contrast in brightness and darkness compared to the RGB channel image. The second image displays details in both the brightest and darkest parts of the image, exhibiting high color fidelity. Therefore, the maturity of the oil palm fruit ears can be determined more accurately based on the second image. The second image, captured by an RGB camera with a wider dynamic range, is more accurate in determining the maturity of oil palm fruit ears than the RGB channel image captured by an RGBD camera with a narrower dynamic range.
[0060] When the oil palm fruit bunch is determined to be mature, the mature oil palm fruit bunch is located by combining the first image and the second image, so as to facilitate the harvesting equipment to harvest the mature oil palm fruit bunch.
[0061] Further, step S2 includes identifying oil palm fruit bunches in the first image and making a first annotation on the identified oil palm fruit bunches.
[0062] Step S3 includes: acquiring a second image; identifying oil palm fruit ears in the second image and adding a second annotation to the identified oil palm fruit ears; determining whether the oil palm fruit ears with the second annotation are mature; when it is determined that the oil palm fruit ears with the second annotation are mature, finding the oil palm fruit ears with the first annotation in the RGB channel image corresponding to the mature oil palm fruit ears, and using this as the predetermined fruit ears; mapping the RGB channel image and D channel depth data of the predetermined fruit ears to locate the predetermined fruit ears.
[0063] Among them, the correspondence can be established by using image registration technology or by using relative position markers, and the corresponding oil palm fruit ear in the first label can be found based on the second label.
[0064] For example, a first image is taken of region A containing oil palm fruit ears using an RGBD camera, and a second image is taken of region A using an RGB camera.
[0065] A first image is acquired, and its RGB channels are analyzed to identify five oil palm fruit clusters, which are then labeled as fruit 01, fruit 02, fruit 03, fruit 04, and fruit 05. A second image is acquired, and its RGB channels are also analyzed to identify five oil palm fruit clusters, which are then labeled as fruit 11, fruit 12, fruit 13, fruit 14, and fruit 15. Fruit 11 corresponds to fruit 01, fruit 12 to fruit 02, fruit 13 to fruit 03, fruit 14 to fruit 04, and fruit 15 to fruit 05.
[0066] The maturity of the five oil palm fruit bunches in the second image was determined, and only fruit bunches 12 and 15 were found to be mature.
[0067] Based on the mature fruit 12 and fruit 15 in the second image, find the corresponding fruit 02 and fruit 05 in the RGB channel image.
[0068] Map the RGB channel images of fruit 02 and fruit 05 to the corresponding D channel depth data of fruit 02 and fruit 05 to locate fruit 02 and fruit 05, making it easier for the harvesting device to pick fruit 02 and fruit 05 from the first image.
[0069] This allows harvesting equipment to accurately pick mature oil palm fruit ears.
[0070] In some embodiments of the present invention, an oil palm fruit bunch in an RGB channel image is identified by a first target recognition model, and an oil palm fruit bunch in a second image is identified by a second target recognition model.
[0071] Using the first target recognition model and the second target recognition model, it is possible to intelligently identify oil palm fruit ears in the RGB channel image and oil palm fruit ears in the second image.
[0072] The first and second target recognition models can be target recognition models such as Faster-RCNN and YOLO.
[0073] Optionally, the first target recognition model is trained by the following method: acquiring multiple first image samples; labeling the oil palm fruit ears in each first image sample; and performing deep learning on the RGB channel images of the labeled fruit ears to obtain the first target recognition model.
[0074] Therefore, an accurate first target recognition model can be obtained.
[0075] Optionally, the second target recognition model is trained by the following method: acquiring multiple second image samples; labeling the oil palm fruit ears in each second image sample; and performing deep learning on the labeled second image to obtain the second target recognition model.
[0076] Therefore, an accurate second target recognition model can be obtained.
[0077] In some embodiments of the present invention, a maturity recognition model is used to determine whether the second-labeled oil palm fruit ears are mature.
[0078] The maturity identification model can be a deep learning model (such as ResNet) or a machine learning model (such as Support Vector Machine (SVM)).
[0079] The maturity recognition model can accurately determine whether oil palm fruit ears are mature.
[0080] Furthermore, the maturity recognition model is formed through the following training method: acquiring multiple oil palm fruit ear image samples; labeling the oil palm fruit ear images with maturity according to the maturity classification criteria of oil palm fruit ears; and performing deep learning on the mature palm fruit ear image samples to obtain the maturity recognition model.
[0081] Therefore, an accurate maturity identification model can be obtained.
[0082] The identification and positioning device 1000 for oil palm fruit ears according to an embodiment of the present invention will be described below.
[0083] like Figure 3 As shown, the oil palm fruit ear identification and positioning device 1000 of this embodiment of the invention includes a first acquisition module 1001 and a first identification and positioning module 1002.
[0084] The first acquisition module 1001 is used to acquire a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data.
[0085] The first identification and positioning module 1002 is used to identify and locate oil palm fruit ears based on the first image.
[0086] Furthermore, such as Figure 4 As shown, the oil palm fruit ear identification and positioning device 1000 also includes a second identification and positioning module 1003.
[0087] The second identification and positioning module 1003 is used to acquire a second image containing the oil palm fruit ear, and to identify and determine the maturity of the oil palm fruit ear based on the second image. The second image is captured by an RGB camera, and the dynamic range of the RGB camera is greater than that of the RGBD camera. When the oil palm fruit ear is determined to be mature, the mature oil palm fruit ear is located based on the first image and the second image.
[0088] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process can be found in the corresponding method embodiments, which will not be repeated here.
[0089] The identification and positioning system for oil palm fruit ears according to an embodiment of the present invention will be described below. The identification and positioning system includes an RGBD camera and a processor.
[0090] An RGBD camera is used to acquire the first image of the oil palm, which includes RGB channel image and D channel depth data.
[0091] The processor is used to acquire a first image containing oil palm fruit ears, which is captured by an RGBD camera and includes RGB channel image and D channel depth data; based on the first image, the oil palm fruit ears are identified and located.
[0092] Furthermore, the oil palm fruit ear identification and positioning system also includes an RGB camera.
[0093] An RGB camera is used to acquire a second image containing the oil palm fruit bunch, and to identify and determine the maturity of the oil palm fruit bunch based on the second image. The second image is captured by an RGB camera, whose dynamic range is greater than that of an RGBD camera.
[0094] The processor is also used to locate mature oil palm ears based on the first and second images when the oil palm ears are determined to be mature.
[0095] The following describes a computer-readable storage medium according to embodiments of the present invention.
[0096] A computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the oil palm fruit ear identification and positioning method as described above.
[0097] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for identifying and locating oil palm fruit ears, characterized in that, The identification and positioning method includes: Step S1: Obtain a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data. Step S2: Based on the first image, identify and locate the oil palm fruit bunch.
2. The method for identifying and locating oil palm fruit ears according to claim 1, characterized in that, The identification and positioning method further includes: Step S3: Acquire a second image containing the oil palm fruit bunch, and identify and determine the maturity of the oil palm fruit bunch based on the second image. The second image is captured by an RGB camera, whose dynamic range is greater than that of the RGBD camera. When the oil palm fruit bunch is determined to be mature, the mature oil palm fruit bunch is located based on the first image and the second image.
3. The method for identifying and locating oil palm fruit ears according to claim 2, characterized in that, Step S2 includes: The oil palm fruit bunches in the first image are identified, and the identified oil palm fruit bunches are labeled. Step S3 includes: Obtain the second image; The oil palm fruit bunches in the second image are identified, and the identified oil palm fruit bunches are labeled a second time. Determine whether the oil palm fruit bunches marked in the second annotation are mature; When it is determined that the oil palm fruit bunch in the second annotation is in a mature state, find the oil palm fruit bunch in the first annotation in the RGB channel image corresponding to the mature oil palm fruit bunch, and use it as the predetermined fruit bunch; The predetermined ear of fruit is located by mapping the RGB channel image and the D channel depth data of the predetermined ear of fruit.
4. The method for identifying and locating oil palm fruit ears according to claim 3, characterized in that, The first target recognition model identifies the oil palm fruit ears in the RGB channel image, and the second target recognition model identifies the oil palm fruit ears in the second image.
5. The method for identifying and locating oil palm fruit ears according to claim 4, characterized in that, The first target recognition model was trained using the following method: Acquire multiple first image samples; Label the oil palm fruit ears in each of the first image samples; Deep learning is performed on the RGB channel images of the labeled ears of fruit to obtain the first target recognition model; The second target recognition model was trained using the following method: Acquire multiple second image samples; Label the oil palm fruit bunches in each of the second image samples; Deep learning is performed on the second image after the ear of fruit is labeled to obtain the second target recognition model.
6. The method for identifying and locating oil palm fruit ears according to claim 3, characterized in that, The maturity identification model is used to determine whether the oil palm fruit ears labeled in the second category are mature.
7. The method for identifying and locating oil palm fruit ears according to claim 6, characterized in that, The maturity recognition model is formed through the following training method: Acquire multiple oil palm fruit bunch image samples; The maturity level of the oil palm fruit bunches is marked according to the maturity classification standard. Deep learning was performed on the oil palm fruit ear image samples after maturity labeling to obtain a maturity recognition model.
8. A device for identifying and locating oil palm fruit ears, characterized in that, The identification and positioning device includes: The first acquisition module is used to acquire a first image containing oil palm fruit ears. The first image is captured by an RGBD camera and includes RGB channel image and D channel depth data. The first identification and positioning module is used to identify and locate the oil palm fruit ears based on the first image.
9. A system for identifying and locating oil palm fruit ears, characterized in that, The identification and positioning system includes: An RGBD camera is used to acquire a first image of the oil palm, the first image including RGB channel image and D channel depth data; The processor is configured to acquire a first image containing oil palm fruit ears, the first image being captured by an RGBD camera, the first image including RGB channel image and D channel depth data; and to identify and locate the oil palm fruit ears based on the first image.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the oil palm fruit ear identification and positioning method as described in any one of claims 1-7.