An apple recognition and positioning method suitable for a complex open environment of an orchard

Abstract

The application discloses an apple recognition and positioning method suitable for a complex open environment of an orchard, and comprises the following steps: high-light area detection and high-light compensation are performed on an image of an apple to be recognized; shadow area adaptive screening is performed on B and R channels separated from the image, and then standard logarithmic transformation is performed; after the image is enhanced by high shadow, A components in a LAB color space and improved light adaptive I2 components in an OHTA color space are extracted, information fusion is performed based on color features of the apple in the two space components, and an RGB three-channel image containing only all visible apple areas is obtained; binaryzation, morphological operation and median filtering are performed, and an apple area binary image is obtained; the apple area binary image is used to separate the adhered apples; and the image after the adhered apples are separated is used for apple recognition and positioning.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an apple recognition and localization method suitable for complex open environments in orchards. Background Technology

[0002] Apples require intensive harvesting during their ripening period, and harvesting is one of the most time-consuming and labor-intensive aspects of apple cultivation. With an aging population and the loss of agricultural labor, the labor costs for apple harvesting are increasing. Therefore, researching automated and intelligent apple-harvesting robots to replace manual labor in apple picking is urgently needed.

[0003] Apple recognition and localization based on computer vision is currently a key research focus for apple-picking robots. However, in open outdoor environments, due to lighting conditions, captured images often contain numerous shadow and highlight areas. Without preprocessing, these areas severely impact the effectiveness of subsequent apple recognition and localization. Furthermore, real-world red apples also contain many cyan areas, making it difficult to separate the apple from other regions using a single segmentation method. Therefore, researching a method that can compensate for highlights and enhance shadows during preprocessing, while simultaneously leveraging the apple's color characteristics and different color space properties to extract the complete visible area of ​​the apple, and then performing apple segmentation to ultimately achieve apple recognition and localization is of great significance. Summary of the Invention

[0004] The purpose of this invention is to address the issue that detection images acquired in complex open environments are affected by highlights and shadows, and that red apples may have cyan areas and be adhered together. This invention proposes an apple identification and localization method suitable for complex open environments in orchards. After performing highlight compensation and shadow enhancement preprocessing on the original image, this invention utilizes the characteristics of apples in different color spaces to completely extract the visible area of ​​the apple, and on this basis, achieves apple identification and localization based on shape features.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An apple identification and localization method suitable for complex open environments in orchards includes the following steps:

[0007] S1, detect the highlight areas on the apples in the original RGB image containing the apple tree, perform highlight compensation on the detected highlight areas, and obtain the highlight-compensated image;

[0008] S2, the R channel, G channel and B channel of the image after highlight compensation are separated. The ratio of B component to R component is compared with the set threshold to detect the area that needs shadow compensation and perform shadow enhancement.

[0009] S3. For the image after shadow enhancement, extract the A component in the LAB color space and the improved illumination-adaptive I2 component in the OHTA color space. Perform global threshold segmentation on each component. Combine the segmentation results of the I2 component and the A component to remove all non-apple regions from the original RGB image, resulting in an RGB three-channel image containing only all visible apple regions.

[0010] S4. Binarize the RGB three-channel image containing only all visible apple regions. Use multiple consecutive morphological operations and median filtering to eliminate regions in the current image that are smaller than 0.2 times the size of a single apple, fill in the holes inside the apple regions, smooth the boundaries of the apple regions, and obtain a binary image of the apple regions.

[0011] S5, use the binary map of the apple region to separate the adhered apples;

[0012] S6 performs apple recognition and localization on the image after separating the adhered apples.

[0013] Preferably, in S1, the set highlight determination conditions for the apple region are used to determine each pixel in the original RGB image, thereby determining the highlight region on the apple:

[0014] The criteria for determining whether a point in the original RGB image is a highlight point in the apple region are as follows:

[0015]

[0016] Where r, g, and b are the values ​​of the r component, g component, and b component, respectively; thre_1 is the highlight threshold.

[0017] If the point meets the determination condition, the point is determined to be a highlight point of the apple region; otherwise, the point is not a highlight point of the apple region.

[0018] When performing highlight compensation on the detected highlight areas, a preset color is used for compensation. The preset color compensation value is [129, 56, 76]. If the highlight point is not an apple area, no compensation is performed.

[0019] Preferably, in S2, the R, G, and B channels of the highlight-compensated image are separated, and the ratio of the B component to the R component is compared with a set threshold to detect areas requiring shadow compensation: the R, G, and B channels of the highlight-compensated image are separated, and the r, g, and b components of each pixel are extracted. A light intensity coefficient threshold thre_2 is set, and the criteria for determining pixels requiring shadow enhancement are as follows:

[0020]

[0021] If a pixel meets the criteria for determining the pixel requiring shadow enhancement, then shadow enhancement is performed using a normalized logarithmic transformation. The formula for the normalized logarithmic transformation is as follows:

[0022]

[0023] Where r, g, and b are the values ​​of the r component, g component, and b component, respectively;

[0024] If a pixel does not meet the criteria for determining pixels requiring shadow enhancement, then shadow enhancement will not be performed.

[0025] Preferably, in S3, the process of fusing the segmentation results of I2 and A components to remove all non-apple regions from the original RGB image, resulting in an RGB three-channel image containing only all visible apple regions, includes the following steps:

[0026] The image after shadow enhancement is converted to LAB space, the A channel image is extracted, and global thresholding is performed. Then, an image Image_a of the same size and type as the A channel image is created. When the value of a pixel in the A channel is greater than the value of a pixel in the Image_a image, the value of the single-channel image Image_a at that point is set to 255, otherwise it is set to 0. Finally, a binary image that retains the red area on the apple is extracted.

[0027] The image after shadow enhancement was subjected to channel separation to extract the B and R channels. Then, each pixel of the B and R images was processed using the improved I2 channel image extraction algorithm in the OHTA color space to obtain an I2 channel image that can adapt to lighting conditions. The I2 image extraction formula is as follows:

[0028]

[0029] In the formula, Image_i2 is the image to be extracted for the I2 channel, (x,y) are the position coordinates of the Image_i2 image, r is the pixel value of the R channel image at position (x,y), and b is the pixel value of the B channel image at position (x,y).

[0030] OTSU adaptive segmentation is performed on the I2 channel image to obtain a binary image that retains all apple regions and some leaf regions;

[0031] The background of the shadow-enhanced image is removed using the binary image segmented from the I2 channel image, resulting in an RGB image that retains only all apple regions and some leaf regions. Channel separation is then performed on this RGB image. Based on the characteristic that the R channel value is the smallest in the cyan apple RGB image, the deviation of corresponding pixels in the three channels from the G and B channels is calculated. Pixels that do not belong to the apple are then removed. The formula for calculating the deviation of the R channel from the G and B channels is as follows:

[0032]

[0033] In the formula, err1 is the difference between the corresponding pixel values ​​in the G channel and the R channel, err2 is the difference between the corresponding pixel values ​​in the B channel and the R channel, r is the pixel value of the pixel in the R channel image, b is the pixel value of the pixel in the B channel image, and g is the pixel value of the pixel in the G channel image.

[0034] When err1>0 or err2>0, the pixel is determined to be a non-apple pixel, and all three channels of the pixel are set to 0; finally, a three-channel image that retains most of the apple area is obtained;

[0035] The original RGB image is processed using a binary image segmented from the A-channel image and a three-channel image segmented from the I2-channel image: when the pixel value of the binary image segmented from the A-channel image is 0 and the pixel value of the three-channel image segmented from the I2-channel image is [0,0,0], the original RGB image is [0,0,0], resulting in an RGB three-channel image that contains only all visible apple regions.

[0036] Preferably, in S4, after binarizing the RGB three-channel image containing only all visible apple regions, three morphological operations are performed based on a circular convolution kernel. The three morphological operations are morphological opening, morphological closing, and morphological opening in sequence. In an image with a size of 1920×1080, the shapes and sizes of the convolution kernels used are: a 10×10 rectangular convolution kernel, a circular convolution kernel with a radius of 10, and a 5×5 rectangular convolution kernel. Then, a 3×3 rectangular convolution kernel is used to perform median filtering on the image, and finally, a binary image of the apple region with the elimination of excessively small regions, the breaking of small connections, and the smoothing of edges is obtained.

[0037] Preferably, the specific process of S5 includes the following steps:

[0038] S501, extract the contours of the binary image of the apple region, traverse all the extracted contours in turn, and perform convex hull detection on each contour;

[0039] S502, when there is a convex hull and the number of convex hulls is greater than 5 and the number of pixels in the contour of the convex hull is greater than 130, convex defect detection is performed on the convex hull. When the number of convex defects is greater than 2, it proves that there are overlapping apples in the contour and they need to be separated.

[0040] S503: Sort the detected convex defects according to their depth, and take the two pixels with the largest defect depth as the starting and ending points of the dividing line, respectively.

[0041] S504, repeat S501 to S503 to perform iterative detection and separation of overlapping apples, and take the condition of not finding a convex hull that satisfies the condition in S502 as the end of the iteration.

[0042] Preferably, in S6, when performing apple recognition and localization on the image after separating the adhered apples, Hough circle detection is performed on the binary image after separating the adhered apples, and the detection results are corrected using minimum diameter constraints and color satisfaction constraints to eliminate erroneous screening areas.

[0043] Among them, the Canny operator with an upper threshold of 45 pixels and a lower threshold of 25 pixels is used for edge extraction. Hough circle detection is performed on the extracted contours. On a 1920×1080 image, the minimum center distance threshold of the circle to be detected is set to 60 pixels, the minimum circle radius is 10 pixels, the maximum radius is 60 pixels, and the circle center determination threshold is 60 pixels.

[0044] When correcting the detection results, a minimum diameter of 40 pixels for the fitted circle is set as a constraint.

[0045] This invention also provides an apple identification and positioning system suitable for complex open environments in orchards, comprising:

[0046] Highlight compensation module: used to detect the highlight areas on the apples in the original RGB image containing the apple tree, perform highlight compensation on the detected highlight areas, and obtain the highlight-compensated image;

[0047] Shadow enhancement module: Used to separate the R, G and B channels of the image after highlight compensation. By comparing the ratio of the B component to the R component with a set threshold, the area that needs shadow compensation is detected and shadow enhancement is performed.

[0048] Image fusion module: It is used to extract the A component in the LAB color space and the improved illumination adaptive I2 component in the OHTA color space from the image after shadow enhancement, and perform global threshold segmentation on each. The segmentation results of I2 and A components are fused to remove all non-apple regions from the original RGB image, resulting in an RGB three-channel image containing only all visible apple regions.

[0049] Apple Region Binary Image Processing Module: This module binarizes an RGB three-channel image containing only all visible apple regions. It uses multiple morphological operations and median filtering to eliminate regions smaller than 0.2 times the size of a single apple in the current image, fills in holes inside the apple regions, and smooths the boundaries of the apple regions to obtain a binary image of the apple regions.

[0050] Adhesive Apple Separation Module: Used to separate adhesive apples using the binary map of the apple region;

[0051] Apple recognition and positioning module: used to recognize and locate apples in images after they have been separated from each other.

[0052] The present invention also provides an electronic device, comprising:

[0053] One or more processors;

[0054] A storage device on which one or more programs are stored;

[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the apple identification and location method of the present invention as described above, which is applicable to complex open environments in orchards.

[0056] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the apple identification and positioning method of the present invention as described above, applicable to complex open environments in orchards.

[0057] The present invention has the following beneficial technical effects:

[0058] This invention relates to an apple identification and localization method applicable to complex open environments in orchards. It preprocesses images of apples to be detected that have local highlights and shadows in complex environments, and uses the fusion of LAB and OHTA color space information for segmentation. This method can identify green apples, red apples, and green-red mixed apples. This identification method has the advantages of strong environmental adaptability and weak influence from apple color. Attached Figure Description

[0059] Figure 1 This is a flowchart of an apple identification and location method applicable to complex open environments in orchards, as described in an embodiment of the present invention.

[0060] Figure 2(a) is the original RGB image to be processed in an embodiment of the present invention, and Figure 2(b) is the image of Figure 2(a) after grayscale processing in Word.

[0061] Figure 3(a1) is the original image of an apple with highlights in an embodiment of the present invention; Figure 3(a2) is the image of Figure 3(a1) after grayscale processing in Word; Figure 3(b1) is the effect image after highlight compensation in an embodiment of the present invention; Figure 3(b2) is the image of Figure 3(b1) after grayscale processing in Word.

[0062] Figure 4(a1) is the original image of the apple with shadows in an embodiment of the present invention; Figure 4(a2) is the image of Figure 4(a1) after grayscale processing in Word; Figure 4(b1) is the image of the shadow enhancement effect in an embodiment of the present invention; Figure 4(b2) is the image of Figure 4(b1) after grayscale processing in Word.

[0063] Figure 5(a) is a binary image of the apple region segmented using the A component in the LAB space in an embodiment of the present invention. Figure 5(b1) is an RGB image of the apple region extracted using the I2 component in the improved OHTA space in an embodiment of the present invention. Figure 5(b2) is a grayscale image of Figure 5(b1) after grayscale processing in Word.

[0064] Figure 6(a) is an image of the apple region extracted after fusing the two spatial segmentation results in an embodiment of the present invention; Figure 6(b) is a grayscale image of Figure 6(a) in Word.

[0065] Figure 7 In this embodiment of the invention, the final apple region extracted image (i.e., Figure 6) is binarized and subjected to multiple morphological operations, resulting in a binary image after smoothing.

[0066] Figure 8(a) is the original image of the adhered apple in the embodiment of the present invention, and Figure 8(b) is the effect image of separating the adhered apple using convex defect detection in the embodiment of the present invention.

[0067] Figure 9(a) is the effect diagram of Hough circle detection in the embodiment of the present invention, and Figure 9(b) is the grayscale image of Figure 9(a) in Word.

[0068] Figure 10(a) is the final apple identification and positioning result after removing the mis-selected area in the embodiment of the present invention, and Figure 10(b) is the grayscale image of Figure 10(a) in Word. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more explicit definition of the scope of protection of the present invention.

[0070] Example

[0071] See Figure 1The apple identification and location method applicable to complex open environments in orchards provided by this invention includes the following steps:

[0072] Step 1) Detect the highlight areas on the apple in the original RGB image and perform highlight compensation on the detected areas;

[0073] Step 2) Separate the R, G, and B channels of the image after highlight compensation. Use the ratio of the B component to the R component to compare with the set threshold to detect the area that needs shadow compensation, and use normalized logarithmic transformation to enhance the shadow.

[0074] Step 3) For the image after highlight compensation and shadow enhancement, extract the A component in the LAB color space and the improved illumination-adaptive I2 component in the OHTA color space, and perform global thresholding segmentation on each. Based on the color features of the apple in the two color space components, information fusion is performed, and all non-apple regions are removed from the original RGB image to obtain an RGB three-channel image containing only all visible apple regions;

[0075] Step 4) Binarize the image of the extracted apple region, use multiple morphological operations and median filtering to eliminate regions in the current image that are smaller than 0.2 times the size of a single apple, fill in the holes inside the apple region, and smooth the boundaries of the apple region.

[0076] Step 5) Separate the stuck apples from the image after morphological operations: extract the contour edges, perform convex hull detection on the contours, perform iterative convex defect detection on the contours with convex hulls and contour points that meet the threshold condition, and use the convex defects to separate the overlapping apples.

[0077] Step 6) Perform Hough circle detection on the binary image after separating the overlapping apples, and set constraints to correct the detection results, eliminate erroneous screening areas, and realize apple recognition and localization in the complex open environment of the orchard.

[0078] Figure 2(a) shows the initial image to be identified obtained in a real apple orchard environment in the embodiment; Figure 3(a1) is an example image of an apple with a highlight area in the original image. In step 1), all pixels in the original RGB image are traversed, and the values ​​of r, g, and b components of each pixel are extracted. The highlight threshold thre_1 is set to 200. The relationship between r, g, b and the highlight threshold thre_1 is used to determine whether each pixel is a highlight. The determination conditions are as follows:

[0079]

[0080] If it is a highlight, the point is compensated using the preset three-channel color compensation value [129,56,76]. The compensated result is shown in Figure 3(b1). If it is not a highlight in the apple area, no operation is performed.

[0081] Figure 4(a1) is an example image of an apple with a shadow area in the image after illumination compensation in this embodiment. In step 2) above, channel separation is performed on the image after highlight compensation, and the r, g, b components of each pixel are extracted. The light intensity coefficient threshold thre_2 is set to 0.7. The criteria for determining pixels that need shadow enhancement are as follows: Once the pixel meets the requirements, the values ​​of the three channels of that pixel are normalized logarithmically transformed, and the resulting image of the apple region with enhanced shadows is shown in Figure 4(b1).

[0082] When a pixel meets the requirements, the values ​​of its three channels are subjected to a standardized logarithmic transformation, as shown in the following formula:

[0083]

[0084] Figure 5(a) shows the apple region segmented by the A component in the LAB space, and Figure 5(b1) shows the apple region extracted using the I2 component in the improved OHTA space. Step 3) specifically includes the following steps:

[0085] Step 301) Convert the image after highlight compensation and shadow enhancement to LAB space, extract the image of the A channel, and perform global thresholding segmentation with a segmentation threshold of thre_a = 140. Then create an image Image_a of the same size and type as the A channel image. When the pixel value of the A channel is greater than 140, set the value of the single-channel image Image_a to 255 at that point; otherwise, set it to 0. Finally, extract the binary image that retains the red area on the apple.

[0086] Step 302) Separate the highlight-compensated and shadow-enhanced images by channel extraction, extracting the B and R channels. Then, process each pixel of the B and R images using the improved I2 channel image extraction algorithm in the OHTA color space to obtain an I2 channel image that adapts to lighting conditions. The I2 image extraction formula is as follows:

[0087]

[0088] In Equation 1, Image_i2 is the image whose I2 channel is to be extracted, (x,y) is the position coordinate of the Image_i2 image, r is the pixel value of the R channel image at position (x,y), and b is the pixel value of the B channel image at position (x,y).

[0089] Next, the OTSU segmentation algorithm is used to segment out a binary image that retains only the apple region and part of the leaf region.

[0090] Step 303) Using a binary image that retains only the apple region and part of the leaf region, background removal is performed on the image after highlight compensation and shadow enhancement, resulting in an RGB image that retains only all apple regions and part of the leaf region. Channel separation is then performed on this image. Based on the characteristic that the R channel value is the smallest in the cyan apple RGB image, the deviation of the corresponding pixel in the three channels on the R channel and the G and B channels is calculated, and pixels that do not belong to the apple are removed. The method for calculating the deviation on the R channel and the G and B channels is as follows:

[0091]

[0092] In equation (4), err1 is the difference between the corresponding pixel values ​​in the G channel and the R channel, err2 is the difference between the corresponding pixel values ​​in the B channel and the R channel, r is the pixel value of the pixel in the R channel image, b is the pixel value of the pixel in the B channel image, and g is the pixel value of the pixel in the G channel image.

[0093] When err1>0 or err2>0, the point is determined to be a non-apple pixel, and all three channels of the point are set to 0. This ultimately yields a three-channel image that retains most of the apple region.

[0094] See Figure 6(a), which is the result of merging the apple regions extracted from Figure 5(a) and Figure 5(b1) in different color spaces.

[0095] Step 304) Process the original RGB image Image_origin using the binary image ImageA_bianry segmented from the A-channel image and the three-channel image ImageI2_thre segmented from the I2-channel image: When the pixel value ImageA_bianry(x,y) = 0 at a certain point in the binary image and the pixel value ImageI2_thre(x,y) = [0,0,0] at the corresponding position in the three-channel image segmented from the I2-channel image, then set the pixel value of the original RGB image at that point to Image_origin(x,y) = [0,0,0]; otherwise, do not make any changes. The final result is a three-channel image containing only the visible apple region.

[0096] See Figure 7 , Figure 7To binarize the extracted visible apple region image and perform multiple morphological operations, resulting in a smoothed binary image, step 4) of this embodiment is as follows: After binarizing the three-channel image of the extracted visible apple region, three morphological operations are performed: morphological opening, morphological closing, and morphological opening. In an image with a size of 1920×1080, the convolution kernel shapes and sizes used are: a 10×10 rectangular convolution kernel, a circular convolution kernel with a radius of 10, and a 5×5 rectangular convolution kernel, respectively. Next, a 3×3 rectangular convolution kernel is used to perform median filtering on the image, ultimately obtaining a binary image of the apple region that eliminates excessively small regions, breaks micro-connections, and has smoothed edges.

[0097] Figure 8(a) shows the original binary image of the region where the apples are stuck together, and Figure 8(b) shows the effect of separating the stuck apples after segmentation using convex defect detection. Step 5) of this embodiment is as follows:

[0098] Step 501) Extract the contours of the binary image of the apple region after morphological processing, traverse all extracted contours in turn, and perform convex hull detection on each contour.

[0099] Step 502) When a convex hull exists and the number of convex hulls is greater than 5, and the number of pixels on the contour where the convex hull is located is greater than 130, perform convex defect detection on the convex hull. When the number of convex defects is greater than 2, it proves that there are overlapping apples on the contour and they need to be separated.

[0100] Step 503) Sort the detected convex defects in descending order of defect depth, and take the two pixels with the largest defect depth as the starting and ending points of the dividing line.

[0101] Step 504) Repeat steps 501) to 503) above to perform iterative detection and separation of overlapping apples. The iteration ends and the separation of the adhered apples is completed when no convex hull that meets the conditions is found.

[0102] Referring to Figure 9(a), Figure 9(a) shows the result after Hough circle detection. In step 6) of this embodiment, edge extraction is performed using the Canny operator with upper and lower thresholds of 45 pixels and 25 pixels respectively. Hough circle detection is then performed on the extracted contours. On a 1920×1080 image, the minimum center distance threshold for the circle to be detected is set to 60 pixels, the minimum circle radius to 10 pixels, the maximum radius to 60 pixels, and the circle center determination threshold to 10 pixels. The final result is an image with misidentified areas.

[0103] Referring to Figure 10(a), Figure 10(a) shows the final apple recognition and localization result after removing falsely selected areas. A condition was set that the minimum diameter of the fitted circle is 40 pixels to delete apple areas that are far from the camera or severely occluded. A color satisfaction condition for the area within the circle was set. Each detection result was iterated through in the preprocessed image, and the pixel values ​​in the three channels of each point were extracted. When the R channel pixel value was the maximum or minimum, the point was determined to belong to the apple area and counted. The count value was divided by the total number of pixels within the circle. If it was less than 50%, it indicated that the apple area was in the minority of the detection results, indicating false detection or severe occlusion of the apple. This detection result was then deleted, resulting in a reliable apple recognition and localization result.

[0104] This invention proposes a preprocessing method for highlight compensation and shadow enhancement in apple regions of images to be detected. By traversing each pixel in the image, the R, G, and B channel values ​​of each pixel are extracted and compared with a set highlight threshold. When all three channel values ​​are greater than the threshold, and the R channel value is dominant, the pixel value is set as a preset highlight compensation value. The ratio of the B and R channels is compared with a light intensity coefficient threshold; when it is less than the threshold, it is identified as a shadow point and enhanced using a standard logarithmic transformation. This preprocessing method ensures that the apple recognition and localization algorithm can still achieve good segmentation results even when highlights and shadows exist in the apple region.

[0105] This invention provides a vision-based method for recognizing and locating red and green apples in complex open environments. It utilizes the excellent segmentation effect of the A component of the LAB color space on the red region of the apple to extract the red area. In the OHTA space, an improved I2 component extraction algorithm is used to extract the I2 component, followed by global threshold segmentation to segment the green region of the apple. Logical operations are performed on the segmentation results to finally extract all visible areas of the apple. Furthermore, iterative convex defect detection is used to separate adhered apples, and multiple constraints are applied to evaluate the Hough circle detection results, ultimately yielding reliable detection results. Compared with general apple recognition algorithms, this multi-color space fusion segmentation method greatly reduces dependence on apple color. Iterative convex defect detection separates adhered apples, making the method adaptable to adhered apples as well. The re-evaluation of the Hough circle detection results also significantly improves the reliability of the recognition and location results.

[0106] The foregoing has shown and described the basic principles, main features and advantages of the present invention.

[0107] The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An apple recognition and positioning method suitable for complex open environment of orchard, characterized in that, The method comprises the following steps: S1, detecting the highlight area on the apples in the original RGB image containing apple trees, performing highlight compensation on the detected highlight area, and obtaining a highlight compensated image; S2, separating the highlight compensated image into R, G and B channels, comparing the ratio of B component to R component with a set threshold value, detecting the area requiring shadow compensation, and performing shadow enhancement; S3, extracting the A component in the LAB color space and the improved light adaptive I2 component in the OHTA color space from the image after shadow enhancement, respectively performing global threshold segmentation, fusing the segmentation results of I2 and A component, and removing all non-apple areas from the original RGB image to obtain an RGB three-channel image containing only all visible apple areas; specifically, converting the image after shadow enhancement to the LAB space, extracting the A channel image, and performing global threshold segmentation; then creating an image Image_a of the same size and type as the A channel image, setting the single-channel image Image_a to 255 at the point where the value of the A channel pixel point is greater than the value of the image Image_a pixel point, and setting it to 0 otherwise, and finally extracting a binary image retaining the red area on the apples; performing channel separation on the image after shadow enhancement, extracting the B channel and R channel images, processing each pixel point of the B image and R image according to the extraction algorithm of the improved I2 channel image in the OHTA color space, obtaining an I2 channel image that can adapt to the light, and the I2 image extraction formula is as follows: In the formula, Image_i2 is the image to be extracted I2 channel, (x, y) is the position coordinate of the Image_i2 image, r is the pixel value of the R channel image at the (x, y) position, and b is the pixel value of the B channel image at the (x, y) position; performing OTSU adaptive segmentation on the I2 channel image to obtain a binary image retaining all apple areas and part of the leaf area; performing background removal on the image after shadow enhancement using the binary image segmented by the I2 channel image to obtain an RGB image retaining only all apple areas and part of the leaf area, performing channel separation on the RGB image retaining only all apple areas and part of the leaf area, calculating the deviation of the corresponding pixel points on the R channel, G channel and B channel according to the characteristic that the R channel value is the smallest in the RGB image of the green apple, removing the pixel points not belonging to the apples, and the deviation calculation formula of the R channel, G channel and B channel is as follows: In the formula, err1 is the difference between the corresponding pixel point values on the G channel and the R channel, err2 is the difference between the corresponding pixel point values on the B channel and the R channel, r is the pixel value of the pixel point on the R channel, b is the pixel value of the pixel point on the B channel, and g is the pixel value of the pixel point on the G channel; when err1>0 or err2>0, it is determined that the pixel point is a non-apple pixel point, and the three channels of the pixel point are set to 0; finally, a three-channel image retaining most of the apple area is obtained; The original RGB image is processed by using the binary image segmented by the A channel image and the three-channel image segmented by the I2 channel: when the pixel value of the binary image segmented by the A channel image = 0 and the pixel value of the three-channel image segmented by the I2 channel = [0, 0, 0], the original RGB image = [0, 0, 0], and a RGB three-channel image containing only all visible apple regions is obtained; S4, the RGB three-channel image containing only all visible apple regions is binarized, and continuous morphological operations and median filtering are used to eliminate regions with a size less than 0.2 times the size of a single apple in the current image, fill holes in the apple region, smooth the apple region boundary, and obtain an apple region binary image; S5, the apple region binary image is used to separate the adhered apples; S6, apple recognition and positioning are performed on the image after the adhered apples are separated.

2. The apple recognition and positioning method suitable for complex open environment of orchard according to claim 1, characterized in that, In S1, the set apple region highlight judgment condition is used to judge each pixel point in the original RGB image to realize the judgment of the highlight region on the apple: The judgment condition of whether a point in the original RGB image is an apple region highlight point is as follows: wherein r, g and b are the numerical values of the r component, the g component and the b component, respectively; thre_1 is a high light point threshold value; If the point meets the judgment condition, the point is determined to be an apple region highlight point, otherwise the point is not an apple region highlight point; When the detected highlight region is compensated for highlight, a preset color is used for compensation, and the preset color compensation value is [129, 56, 76]. If it is not an apple region highlight point, no compensation is performed.

3. The apple recognition and positioning method suitable for complex open environment of orchard according to claim 1, characterized in that, In S2, the image after highlight compensation is separated into R, G and B channels, and the ratio of B component to R component is compared with the set threshold to detect the region that needs shadow compensation: the R, G and B channels of the image after highlight compensation are separated, the r, g and b components of each pixel point are extracted, the light intensity coefficient threshold thre_2 is set, and the pixel point judgment condition for needing shadow enhancement is as follows: If the pixel point meets the pixel point judgment condition for needing shadow enhancement, the shadow is enhanced by using standard logarithmic transformation, and the formula of the standard logarithmic transformation is as follows: Wherein, r, g and b are the numerical values of r, g and b components respectively; If the pixel point does not meet the pixel point judgment condition for needing shadow enhancement, no shadow enhancement is performed.

4. The apple recognition and positioning method for complex open environment of orchard according to claim 1, characterized in that, In S4, after the RGB three-channel image containing only all visible apple regions is binarized, three morphological operations are performed, and the three morphological operations are morphological opening operation, morphological closing operation and morphological opening operation in sequence. In the image with a size of 1920*1080, the shape and size of the convolution kernel used are 10*10 rectangular convolution kernel, circular convolution kernel with a radius of 10 and 5*5 rectangular convolution kernel in sequence, and then 3*3 rectangular convolution kernel is used for median filtering of the image, and finally the apple region binary image with the eliminated small regions, broken small connections and smoothed edges is obtained.

5. The apple recognition and localization method for complex open orchard environment according to claim 1, characterized in that, The specific process of S5 includes the following steps: S501, the apple region binary image is subjected to contour extraction, and all the contours extracted are sequentially traversed and subjected to convex hull detection; S502, when there is a convex hull and the number of convex hulls is greater than 5, the number of pixel points of the contour where the convex hull is located is greater than 130, the convex defect detection is performed on the convex hull, and when the number of convex defects is greater than 2, it is proved that the contour has overlapping apples, which need to be separated; S503, the detected convex defects are sorted according to the defect depth, and the two pixel points with the largest defect depth are taken as the starting point and the ending point of the segmentation straight line respectively; S504, repeating S501 to S503 for iterative detection and separation of overlapping apples, and taking the condition that no convex hull satisfying S502 is found as the iteration end condition.

6. The apple recognition and localization method for complex open orchard environment according to claim 1, characterized in that, In S6, when the image after the separation of the adhered apples is subjected to apple recognition and positioning, the binary image after the separation of the adhered apples is subjected to Hough circle detection, and the detection results are corrected by using the minimum diameter constraint and the color satisfaction constraint to eliminate the misselected area; Wherein, the canny operator with upper threshold value of 45 pixel points and lower threshold value of 25 pixel points is used for edge extraction, and the extracted contour is subjected to Hough circle detection, and in the 1920*1080 image, the minimum center distance threshold value of the detected circle is set to 60 pixel points, the minimum circle radius is 10 pixel points, the maximum radius is 60 pixel points, and the center determination threshold value is 60 pixel points; When the detection result is corrected, the minimum diameter 40 pixel points of the fitted circle is set as the constraint condition.

7. An apple recognition and localization system suitable for complex open environment of orchard, characterized in that, It includes: High light compensation module: for detecting the high light area on the apple in the original RGB image containing apple tree, performing high light compensation on the detected high light area, and obtaining the high light compensated image; Shadow enhancement module: for separating the R channel, G channel and B channel of the high light compensated image, comparing the ratio of B component to R component with the set threshold value, detecting the area needing shadow compensation, and performing shadow enhancement; Image fusion module: for extracting A component in LAB color space and improved light adaptive I2 component in OHTA color space from the image after shadow enhancement, respectively performing global threshold segmentation, fusing the segmentation results of I2 and A component, and removing all non-apple areas of the original RGB image to obtain an RGB three-channel image containing all visible apple areas; specifically including: converting the image after shadow enhancement to LAB space, extracting the A channel image, and performing global threshold segmentation; then creating an image Image_a with the same size and type as the A channel image, when the value of the A channel pixel point is greater than the value of the image Image_a pixel point, the value of the single-channel image Image_a at the point is set to 255, otherwise it is set to 0, and finally a binary image retaining the red area on the apple is extracted; The image after shadow enhancement is subjected to channel separation, B channel and R channel images are extracted, B image and R image are processed according to the extraction algorithm of I2 channel image in the improved OHTA color space, and I2 channel image adaptive to light is obtained, and the I2 image extraction formula is as follows: In the formula, Image_i2 is an image of the I2 channel to be extracted, (x, y) is the position coordinate of the Image_i2 image, r is the pixel value of the R channel image at the (x, y) position, and b is the pixel value of the B channel image at the (x, y) position; OTSU adaptive segmentation is performed on the I2 channel image to obtain a binary image retaining all apple regions and part of the leaf regions; The binary image obtained by the I2 channel image segmentation is used to perform background elimination on the image subjected to shadow enhancement, to obtain an RGB image retaining only all apple regions and part of the leaf regions, and the RGB image retaining only all apple regions and part of the leaf regions is subjected to channel separation, the deviation of the pixel points corresponding to the three channels on the R channel from the G channel and the B channel is calculated according to the characteristic that the R channel value is the smallest in the RGB image of the green apple, and the pixel points not belonging to the apple are eliminated, and the deviation calculation formula of the R channel from the G channel and the B channel is as follows: In the formula, err1 is the difference between the pixel point values on the G channel and the R channel, err2 is the difference between the pixel point values on the B channel and the R channel, r is the pixel value of the pixel point on the R channel, b is the pixel value of the pixel point on the B channel, and g is the pixel value of the pixel point on the G channel; When err1>0 or err2>0, it is determined that the pixel point is a pixel point not on the apple, and the three channels of the pixel point are all set to 0, and finally a three-channel image retaining most of the apple regions is obtained; The binary image segmented by the A channel image and the three-channel image segmented by the I2 channel are used to process the original RGB image: when the pixel value of the binary image segmented by the A channel image is 0 and the pixel value of the three-channel image segmented by the I2 channel is [0, 0, 0], the original RGB image is [0, 0, 0], and an RGB three-channel image containing only all visible apple regions is obtained; An apple region binary image processing module is configured to binarize the RGB three-channel image containing only all visible apple regions, eliminate regions smaller than 0.2 times the size of a single apple in the current image by using continuous morphological operations and median filtering, fill holes in the apple region, smooth the apple region boundary, and obtain an apple region binary image; A connected apple separation module is configured to separate connected apples by using the apple region binary image; An apple recognition and positioning module is configured to perform apple recognition and positioning on the image after the connected apples are separated.

8. An electronic device, comprising: The method comprises: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the apple recognition and positioning method suitable for a complex open environment of an orchard according to any one of claims 1 to 6.

9. A storage medium, characterized by a computer program stored thereon, wherein the computer program is executed by a processor to implement the apple recognition and positioning method suitable for a complex open environment of an orchard according to any one of claims 1 to 6.

Images