Palm fruit and leaf image segmentation method

Abstract

The invention discloses a palm fruit and leaf image segmentation method. The method comprises the following steps: acquiring RGB color space images of palm fruits and leaves; converting the image into an HSV color space image; extracting an H component, and drawing an H component histogram; analyzing H component thresholds of the fruits and the leaves under different illumination; on the basis of a traditional FCM clustering segmentation algorithm, weight coefficients are added, calculation of each pixel point is converted into calculation of H component frequency, an improved FCM clustering algorithm is obtained, the H component of the image is segmented, and an interested area is obtained and serves as a result; morphological processing is carried out on the result, the influence of interference factors such as noise is reduced, the obtained mask is applied to an original image, and segmented fruit and leaf images are obtained. According to the method, on the premise that the adverse effect of uneven illumination is reduced, the images of the palm fruits and the leaves can be quickly and accurately segmented, and a basis is provided for work such as three-dimensional positioning and fruit quality detection.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to a method for segmenting palm fruit and leaf images. Background Art

[0002] Palm fruits play a crucial role in the current global agricultural and economic landscape. As the fruits borne on the oil palm trees of the palm family, their significance spans multiple key areas. From the perspective of the agricultural industry, it is one of the pillars of the tropical agricultural economy, providing a major source of income for farmers. In the food industry, the palm oil extracted from palm fruits is irreplaceable. With relatively stable chemical properties, moderate prices, and wide applicability, it has become an important part of the global food processing industry. From an energy perspective, with the increasing exploration of renewable energy globally, palm fruits also show new potential. Through advanced conversion technologies, their extracts are expected to become high-quality raw materials for biodiesel, contributing to alleviating the energy crisis and reducing carbon emissions. Therefore, harvesting palm fruits is of inestimable significance for the current and future development of the world.

[0003] However, the current palm fruit harvesting faces a series of severe problems, posing a major challenge to the sustainable development of the palm oil industry. In terms of harvesting efficiency, in some current regions, traditional manual harvesting methods are still used. Workers need to climb tall palm trees and use simple tools such as machetes and sickles for harvesting. This method not only has a high labor intensity but also low efficiency. For example, in India, farmers rely on their professional knowledge and agility to climb palm trees with minimal equipment to pick fruits, which is a time-consuming and laborious process. Even in some regions where simple mechanical assistance is introduced, the efficiency improvement is relatively limited and difficult to meet the increasing harvesting demands of large-scale plantations. Moreover, manual harvesting is greatly affected by factors such as weather and the physical condition of workers. In case of bad weather, such as heavy rain and high temperature, the harvesting work often cannot be carried out normally.

[0004] In response to the above phenomena, scholars at home and abroad have studied and optimized the palm fruit picking method. The robotic arm of the palm fruit picking machine invented by Wang Jianyong et al. has a rotating column formed by the upper and lower columns that can rotate in the horizontal plane and move forward and backward on the chassis, realizing the rotation, advancement, and retreat functions of the robotic arm. Its mast pitching mechanism enables the robotic arm to perform pitching movements within 0 to 72° and can be locked at any position, realizing functions such as the advancement, retreat, pitching, telescoping, and horizontal rotation of the robotic arm. Finally, a tool joint connects the tool for cutting. This mechanism has a simple structure and is easy to use, greatly reducing the difficulty of manual picking and improving the picking efficiency. However, the automation level of this invention is relatively low and still relies on manual operation. Yu Shishun et al. invented a fruit picking mechanism for an automatic fruit picking machine, which mainly consists of a fruit picking workbench, a fruit dropping box, and a rotating mechanism. During operation, the staff enters the workbench and stands in front of the main support plate. Then the main support plate rises, and the auxiliary support plate naturally hangs below the main support plate. The staff stands on the workbench to cut the palm fruits, and the cut palm fruits naturally fall into the fruit dropping box. Although this invention is highly practical and convenient, it still requires manual cutting, has a low degree of automation, and is difficult to achieve large-scale production. Jayaselan et al. designed a five-degree-of-freedom palm picking robotic arm and related control circuits and control software. The operator only needs to input simple instructions on the operation interface according to the actual growth conditions of the palm tree, and the software can quickly analyze and command the robotic arm to execute corresponding actions, greatly reducing the operation difficulty. Compared with traditional manual picking and some simple picking tools, this robotic arm significantly improves the palm fruit picking efficiency and reduces labor costs due to its advantages of high automation and precision. However, this robotic arm has poor environmental adaptability. For example, in the case of some foliage blocking, it is difficult to accurately reach the target position for picking, resulting in a reduction in picking efficiency or the inability to complete the task. At the same time, it cannot fully achieve fully automated picking and cannot truly liberate human labor. Summary of the Invention

[0005] To solve the existing problems in palm fruit picking and achieve picking automation, visual recognition of palm fruits and leaves is a key link. In response to this important link, the present invention proposes a method for segmenting palm fruit and leaf images.

[0006] To achieve the above technical objectives, the technical solution adopted by the present invention is as follows: A method for segmenting palm fruit and leaf images, which specifically includes the following steps:

[0007] S1: Collect multiple palm fruit images and leaf images with different maturities at different angles and under different lighting conditions through an RGB camera;

[0008] S2: Convert the RGB image obtained in S1 into an HSV image through the conversion formula for converting the RGB color space to the HSV color space;

[0009] S3: Extract the H component from the HSV image obtained in S2, draw a histogram of the H component, analyze the peak distribution of the H component under different illuminations, and thus determine the H component as the segmentation object;

[0010] S4: Based on the FCM clustering segmentation algorithm, by adding a weight coefficient, convert its original calculation for each pixel point into a calculation of the frequency of the H component, thereby obtaining an improved FCM clustering segmentation algorithm;

[0011] S5: Use the improved FCM clustering segmentation algorithm obtained in S4 to perform clustering segmentation on the H component of the converted HSV image to obtain the segmented fruit and leaf images;

[0012] S6: Convert the segmented image obtained in S5 into a binary image, perform morphological processing to obtain the final region mask, and apply this mask to the original image to obtain the final segmentation result.

[0013] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0014] A computer-readable storage medium stores a computer program thereon. When the program is executed by a processor, the steps of the above method are implemented.

[0015] A computer program product includes a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a basis for the automation of palm fruit picking to a certain extent, and solves the problems of low automation and low efficiency of palm fruit picking at the present stage to a certain extent; and compared with the existing methods for fruit image segmentation at the present stage, the present invention is easy to set up, does not require a large amount of data for training, and can reduce the influence of the adverse factor of uneven illumination; in addition, at the algorithm level, the present invention improves the FCM algorithm, greatly improves the calculation efficiency of the algorithm, and further improves the real-time performance of the system. Description of the Drawings

[0017] Figure 1 is a flowchart of the method of the present invention.

[0018] Figure 2 is an example of the original palm fruit image obtained in the embodiment of the present invention (sufficient illumination).

[0019] Figure 3 It is an original image example of palm fruit image obtained in the embodiment of the present invention (insufficient light).

[0020] Figure 4 It is the H-component histogram of the image under sufficient light conditions in the embodiment of the present invention.

[0021] Figure 5 It is the H-component histogram of the image under insufficient light conditions in the embodiment of the present invention.

[0022] Figure 6 It is an example of improved FCM segmentation of palm fruit image in the embodiment of the present invention (sufficient light).

[0023] Figure 7 It is an example of improved FCM segmentation of palm fruit image in the embodiment of the present invention (insufficient light).

[0024] Figure 8 It is the mask after morphological processing of the palm fruit image after improved FCM segmentation in the embodiment of the present invention (sufficient light).

[0025] Figure 9 It is the mask after morphological processing of the palm fruit image after improved FCM segmentation in the embodiment of the present invention (insufficient light).

[0026] Figure 10 It is the segmentation result of the palm fruit image in the embodiment of the present invention (sufficient light).

[0027] Figure 11 It is the segmentation result of the palm fruit image in the embodiment of the present invention (insufficient light).

[0028] Figure 12 It is the segmentation result of the leaf image in the embodiment of the present invention (sufficient light).

[0029] Figure 13 It is the segmentation result of the leaf image in the embodiment of the present invention (insufficient light). Detailed implementation manners

[0030] The present invention provides a method for segmenting palm fruit and leaf images, which includes: obtaining RGB color space images of palm fruit and leaves; converting the obtained images into HSV color space images according to the calculation formula for converting RGB color space to HSV color space; for the obtained HSV color space images, extracting their H components and plotting the H component histogram; by analyzing the peak distribution of the H component histogram under different lighting conditions, analyzing the H component thresholds of fruits and leaves under different lighting; based on the traditional FCM clustering segmentation algorithm, adding a weight coefficient to change its original calculation for each pixel point to the calculation of the H component frequency, thereby obtaining an improved FCM clustering algorithm and greatly increasing the algorithm efficiency; using the above improved FCM clustering segmentation algorithm to segment the H component of the image to obtain the region of interest as the result; performing morphological processing on this result to reduce the influence of interference factors such as noise, and applying the obtained mask to the original image to obtain the segmented palm fruit and leaf images. The embodiments of the present invention can quickly and accurately segment the images of palm fruit and leaves on the premise of reducing the adverse effect of uneven lighting, providing a basis for work such as three-dimensional positioning and fruit quality detection.

[0031] In order to enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] As Figure 1 shown, a method for segmenting palm fruit and leaf images of the present invention, the specific scheme of this method includes the following steps:

[0033] S1: Use an RGB camera to collect multiple palm fruit images and leaf images of different maturities under different angles, different lighting and other factors.

[0034] The specific operation is: use an RGB camera to collect a total of 493 palm fruit images under different angles, different degrees of being blocked by leaves, different lighting and different maturities. The lighting conditions include sufficient lighting and insufficient lighting, and the maturities include four maturities: unripe, semi-ripe, ripe and over-ripe. The image size is 416*416. Image examples are Figure 2 and Figure 3 .

[0035] S2: Convert the RGB images obtained in S1 into HSV images through the conversion formula for converting RGB color space to HSV color space.

[0036] The specific operation is: extract three indicators in RGB, and their values are in the range of [0, 255], and perform normalization processing on them, that is

[0037] R′ = R / 255, G′ = G / 255, B′ = B / 255

[0038] Calculate the maximum, minimum, and the difference between the maximum and minimum values among R′, G′, and B′

[0039] C max = max(R′, G′, B′)

[0040] C min = min(R′, G′, B′)

[0041] Δ = C max - C min

[0042] Calculate the values of H, S, and V

[0043]

[0044] V = C max

[0045] Through the above algorithm, the model of the palm fruit picture in the HSV color space can be obtained for subsequent segmentation tasks.

[0046] S3: Extract the H component from the HSV image obtained in S2, draw the histogram of the H component, analyze the peak distribution of the H component under different illuminations, and determine the H component as the segmentation object accordingly.

[0047] The specific operation is as follows: Obtain the histograms of the H component under different illuminations, such as Figure 4 and Figure 5 . It can be seen from the peak distributions of the two histograms that under different illumination conditions, their peak distributions are basically the same. Therefore, selecting the H component as the segmentation object can greatly reduce the influence of the factor of uneven illumination.

[0048] S4: Based on the traditional FCM clustering segmentation method, by adding a weight coefficient, convert its original calculation for each pixel point into a calculation of the frequency of the H component, thereby obtaining an improved FCM clustering segmentation algorithm.

[0049] For the traditional FCM clustering segmentation selected in S4, set the number of clusters to 9, the fuzzy index to 1.5, the maximum number of iterations to 100, and the stopping condition to 0.005. The introduced frequency of the H component is ω(k), which means the frequency corresponding to when the H component is k.

[0050] S5: Use the improved FCM clustering segmentation algorithm obtained in S4 to perform clustering segmentation on the H component of the converted HSV image to obtain the segmented fruit and leaf images.

[0051] For the traditional FCM algorithm, its algorithm steps are as follows:

[0052] (1) Given the initial number of clusters \(c\), the iteration stop condition \(\varepsilon\), and the fuzziness index \(m\), and perform initialization; in the initialization stage, random cluster centers \(v\) will be generated k to form the cluster center matrix \(V\), and at the same time, the membership matrix \(U\) is initialized; in the membership matrix \(U\), \(u\) ik is the membership degree of the data point \(x\) i to the \(k\)-th cluster. At this time, each \(u\) ik is a random value, and the sum of each row is 1, indicating that the sum of the membership degrees of each data point to all cluster centers is 1. While initializing the membership matrix \(U\), let the iteration number \(N = 1\).

[0053] (2) Calculate the cluster center matrix:

[0054]

[0055] (3) Calculate and update the membership matrix:

[0056]

[0057] where \(D\) ik represents the distance from the \(i\)-th data point to the \(k\)-th cluster, and its relationship with the data point \(x\) i is as follows:

[0058] \(D\) ik =\(\vert\vert x\) i - v\) k \(\vert\vert\) 2

[0059] (4) If the stop iteration condition \(\vert\vert U\) (N) - U\) (N-1) \(\vert\vert\lt\varepsilon\) is satisfied, stop the calculation; otherwise, the iteration number \(N\) is incremented by 1, and the cluster center matrix is calculated again according to the updated membership matrix, and then the membership matrix is updated again until the final iteration stop condition is satisfied. And use the finally generated membership matrix and cluster center matrix as the segmentation basis for segmentation, allowing each data point to belong to multiple clusters and having different membership degrees.

[0060] For the improved FCM algorithm, the \(H\) component of the image is converted into a grayscale image, and the grayscale value distribution of this grayscale image is \([0, 255]\), and the conversion formula is as follows:

[0061]

[0062] Use the frequency of each grayscale value in the whole picture as the weight coefficient; use \(h\) to represent the grayscale value, and \(f(h)\) to represent the frequency of the pixel with grayscale value \(h\) appearing, and it satisfies the following equal relationship:

[0063]

[0064] The obtained gray value h and its frequency f(h) replace the calculation part for each pixel point x in the traditional FCM algorithm i The improved FCM algorithm steps are as follows:

[0065] (1) Given the initial number of clusters c, the iteration stop condition ε, and the fuzziness index m, and perform initialization. In the initialization stage, random cluster centers v k are formed into a cluster center matrix V, and at the same time, the membership degree matrix U is initialized. In the membership degree matrix U, u hk is the membership degree of the gray value h for the k-th cluster. At this time, each u hk is a random value, and the sum of each row is 1, indicating that the sum of the membership degrees of each data point for all cluster centers is 1. While initializing the membership degree matrix U, let the iteration number N = 1.

[0066] (2) Calculate the cluster center matrix:

[0067]

[0068] Here, different from the original cluster center matrix update algorithm, the calculation for each pixel point x i is changed to the calculation of the gray value h and its frequency f(h).

[0069] (3) Calculate and update the membership degree matrix:

[0070]

[0071] Among them, D hk represents the distance between the gray value h and the cluster center k, D hj Similarly, that is:

[0072] D hk = ||h - v k || 2

[0073] (4) If the stop condition ||U (N) - U (N-1) || < ε is satisfied, stop the calculation. If not, calculate the cluster center matrix and the membership degree matrix until the membership degree matrix satisfies the stop condition.

[0074] For the improved algorithm, the calculation for each data point is changed to the calculation of the gray value and its frequency, significantly reducing the clustering samples, thereby improving the algorithm cutting speed. The improved FCM algorithm reduces the influence of points with smaller gray value frequencies and focuses on points with larger frequencies. Therefore, compared with the traditional FCM algorithm, its clustering effect will not be greatly affected. The example diagram of the segmentation process is as Figure 6 andFigure 7 。

[0075] S6: Convert the segmented image obtained in S5 into a binary image, perform morphological processing to obtain the final region mask, and apply this mask to the original image to obtain the final segmentation result.

[0076] The specific operation is as follows: For the segmented mask, select the size of the kernel used for dilation to be 2*2, select the size of the kernel used for erosion to be 3*3, and define the iteration times of the dilation and erosion operations to be 3 and 2 respectively. First, perform two erosion operations, and then perform three dilation operations. The final mask is as Figure 8 and Figure 9 。After being applied to the final image, the obtained fruit image examples are as Figure 10 and Figure 11 。In addition, the leaf image examples cut out by using the above method are as Figure 12 and Figure 13 。

[0077] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for segmenting palm fruit and leaf images, characterized in that Including: S1: Collect RGB images of palm fruit fruits and leaf images with different maturities at different angles and under different illuminations through an RGB camera; S2: Convert the RGB images obtained in S1 into HSV images through the conversion formula for converting the RGB color space to the HSV color space; S3: Extract the H component from the HSV images obtained in S2, draw a histogram of the H component, analyze the peak distribution of the H component under different illuminations, and thereby determine the H component as the segmentation object; S4: Based on the FCM clustering segmentation algorithm, by adding a weight coefficient, convert its original calculation for each pixel point into a calculation of the frequency of the H component, thereby obtaining an improved FCM clustering segmentation algorithm; S5: Use the improved FCM clustering segmentation algorithm obtained in S4 to perform clustering segmentation on the H component of the converted HSV image to obtain the segmented fruit and leaf images; S6: Convert the segmented images obtained in S5 into binary images, perform morphological processing to obtain the final region mask, and apply this mask to the original image to obtain the final segmentation result.

2. The segmentation method of palm fruit and leaf images according to claim 1, characterized in that: The size of the RGB image obtained in step S1 is 416*416 pixels.

3. A method for segmenting palm fruit and leaf images according to claim 1, characterized in that: In step S2, extract the three indicators in RGB, whose values are in the range of [0, 255], and perform normalization processing on them, that is R′ = R / 255, G′ = G / 255, B′ = B / 255 Calculate the maximum value C among R′, G′, and B′ max , the minimum value C min and the difference Δ between the maximum value and the minimum value: C max = max(R′, G′, B′) C min = min(R′, G′, B′) Δ = C max -C min Calculate the values of H, S, and V V = C max。 4. A method for segmenting palm fruit and leaf images according to claim 1, characterized in that: In step S4, the selected FCM clustering segmentation method sets the number of clusters to 9, the fuzzy index to 1.5, the maximum number of iterations to 100, and the stop condition to 0.005; the introduced frequency of the H component is ω(k), indicating the frequency corresponding to when the H component is k.

5. A method for segmenting palm fruit and leaf images according to claim 1, characterized in that: Step S5: Use the improved FCM clustering segmentation algorithm obtained in S4 to perform clustering segmentation on the H component of the converted HSV image to obtain the segmented fruit and leaf images, specifically: Convert the H component of the image into a grayscale image, and the grayscale value distribution of this grayscale image is [0, 255]. The conversion formula is as follows: Use the frequency of each grayscale value in the entire picture as the weight coefficient; use h to represent the grayscale value, and f(h) to represent the frequency of the pixel with the grayscale value h. It satisfies the following equivalent relationship: Obtain the grayscale value h and its frequency f(h); the steps of the improved FCM algorithm are as follows: (1) Given the initial number of clusters c, the iteration stop condition ε, and the fuzziness index m, and perform initialization; in the initialization stage, generate random cluster centers v k to form the cluster center matrix V, and at the same time initialize the membership matrix U; in the membership matrix U, u hk is the membership degree of the gray value h for the k-th cluster. At this time, each u hk is a random value, and the sum of each row is 1, indicating that the sum of the membership degrees of each data point for all cluster centers is 1; while initializing the membership matrix U, let the iteration number N = 1; (2) Calculate the cluster center matrix: (3) Calculate and update the membership degree matrix: Among them, D hk represents the distance between the gray value h and the cluster center k, that is: D hk = ||h - v k || 2 (4) If the stop condition ||U (N) -U (N-1) || < ε is satisfied, stop the calculation; if not, calculate the cluster center matrix and the membership degree matrix until the membership degree matrix satisfies the stop condition.

6. The segmentation method of palm fruit and leaf images according to claim 1, characterized in that: For the morphological operation in step S6, select the size of the kernel used for dilation to be 2*2, select the size of the kernel used for erosion to be 3*3, and at the same time define the number of iterations for dilation and erosion operations to be 3 and 2 respectively; first perform the erosion operation, and then perform the dilation operation.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1-6.

Images