Methods, apparatus, media, and equipment for picking point localization based on image fusion

Abstract

This application provides a method, apparatus, medium, and device for picking point localization based on image fusion, relating to the field of image processing. The method includes: converting a crop image to be processed into a grayscale image of the crop; calculating the gradients of the grayscale image of the crop in the directions of detecting vertical and horizontal edges to obtain vertical and horizontal grayscale gradient images; performing weighted fusion to obtain an intermediate edge image; performing a weighted summation of the crop image to be processed and the intermediate edge image to obtain a weighted fused edge-enhanced image; and inputting the crop image to be processed and the weighted fused edge-enhanced image into a picking point localization model to obtain the target crops in the crop image to be processed and the corresponding picking point coordinates for each target crop. This application meets the accuracy requirements for maturity recognition while ensuring the reliability of picking point localization, making it more suitable for the application of automated picking equipment in large-scale planting scenarios.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, apparatus, medium, and electronic device for picking point localization based on image fusion. Background Technology

[0002] Strawberry maturity identification and picking point location technology has significant application value in the field of agricultural automation. It helps automated picking equipment accurately determine the ripeness of strawberries and pinpoint the picking location, significantly improving picking efficiency, reducing labor costs, minimizing fruit damage, and ensuring product quality. This provides key technical support for the intelligent upgrading of large-scale strawberry cultivation. Traditional strawberry maturity identification and location rely on manual visual inspection, which suffers from low efficiency, high subjectivity, and easy fruit damage, making it difficult to meet the production needs of large-scale cultivation. Common technologies also include physical sensing identification and spectral analysis positioning, but these face many challenges in actual strawberry plantation environments. Physical sensing requires contact detection, which can easily damage strawberries and is inefficient in densely planted, overlapping fruit environments. While spectral analysis technology can obtain maturity information non-contactly, the equipment is expensive, sensitive to light conditions, and cannot directly achieve real-time picking point location, making it difficult to adapt to automated picking equipment. In contrast, computer vision-based identification and location technology, with its rich image information, fast detection speed, controllable cost, and ease of integration with automated equipment, has become the mainstream research direction.

[0003] However, existing computer vision-based technologies for strawberry maturity recognition and picking point localization still have significant technical shortcomings in complex real-world planting scenarios, making it difficult to meet the high reliability requirements of precision agriculture operations. Firstly, some solutions use the original RGB image as the sole input data, relying solely on color features in the image to determine strawberry maturity. In scenarios with fluctuating field lighting, fruit surface reflection, and leaves obscuring the fruit, color feature extraction is prone to inadequacy or distortion, leading to a higher rate of false positives and false negatives, especially in distinguishing between semi-ripe and immature strawberries. Secondly, while traditional edge enhancement methods can highlight the outlines of the fruit and stem to aid localization, they easily introduce environmental noise and destroy the original image's color information, failing to simultaneously ensure accurate stem localization. Third, some multi-branch feature fusion schemes lack adaptive adjustment mechanisms for strawberry planting scenarios. They only splice or weighted fuse features with fixed weights, and cannot dynamically adjust the contribution weights of different features according to changes in the scene such as strawberry maturity level and degree of occlusion. As a result, the synergistic effect of color features and edge structure features cannot be fully utilized. In scenarios such as overlapping fruits and complex background interference, the positioning deviation of picking points increases significantly. Fourth, although schemes that rely on multimodal data such as RGB-D can improve the ability to distinguish overlapping fruits by using depth information, they require high-precision depth cameras, which are costly. Moreover, in outdoor scenarios with dew and rain, depth data is easily affected by noise, resulting in decreased model detection accuracy and poor environmental adaptability.

[0004] In summary, while existing computer vision methods have their own optimizations, they all generally have shortcomings. Traditional machine vision methods rely on single color features, making it difficult to handle complex backgrounds and scenes with leaf occlusion; multimodal fusion methods are costly and have poor environmental adaptability; edge enhancement single-branch methods lose color feature information, leading to a decrease in maturity recognition accuracy. The core pain point in actual plantations is precisely the need to simultaneously rely on edge features to distinguish targets and locate fruit stems in complex backgrounds, and on color features to accurately determine maturity. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method, apparatus, medium, and electronic device for picking point localization based on image fusion, which at least partially solves the problems existing in the prior art.

[0006] In a first aspect of this application, a method for locating picking points based on image fusion is provided, the method comprising the following steps:

[0007] Acquire images of crops to be processed; wherein, the images of crops to be processed contain several crops to be processed;

[0008] Convert the crop image to a grayscale image.

[0009] Calculate the gradients of the grayscale image of the crop to be processed in the vertical edge detection direction and the horizontal edge detection direction respectively to obtain the grayscale gradient map in the vertical direction and the grayscale gradient map in the horizontal direction;

[0010] The vertical and horizontal grayscale gradient images are weighted and fused to obtain the middle edge image;

[0011] The crop image to be processed and the intermediate edge image are weighted and summed to obtain a weighted fused edge-enhanced image; wherein the weight corresponding to the crop image to be processed is greater than the weight corresponding to the intermediate edge image.

[0012] The crop image to be processed and the weighted fused edge enhancement image are input into the picking point localization model to obtain the target crop in the crop image to be processed and the corresponding picking point coordinates of each target crop; wherein, the target crop is the crop to be processed that is fully mature; the picking point localization model includes a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the crop image to be processed; the input of the second feature extraction network is the weighted fused edge enhancement image.

[0013] In a second aspect of this application, a picking point positioning device based on image fusion is provided, the device comprising:

[0014] The acquisition unit is used to acquire images of crops to be processed; wherein, the images of crops to be processed contain several crops to be processed.

[0015] The conversion unit is used to convert the crop image to be processed into a grayscale image of the crop.

[0016] The calculation unit is used to calculate the gradient of the grayscale image of the crop to be processed in the direction of detecting the vertical edge and the direction of detecting the horizontal edge, respectively, so as to obtain the grayscale gradient map in the vertical direction and the grayscale gradient map in the horizontal direction.

[0017] The intermediate image acquisition unit is used to perform weighted fusion of the vertical grayscale gradient map and the horizontal grayscale gradient map to obtain the intermediate edge image;

[0018] The fusion unit performs a weighted summation of the crop image to be processed and the intermediate edge image to obtain a weighted fused edge-enhanced image; wherein the weight corresponding to the crop image to be processed is greater than the weight corresponding to the intermediate edge image.

[0019] The result acquisition unit is used to input the crop image to be processed and the weighted fused edge enhancement image into the picking point localization model to obtain the target crop in the crop image to be processed and the picking point coordinates corresponding to each target crop; wherein, the target crop is the crop to be processed with the corresponding maturity level of full maturity; the picking point localization model includes a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the crop image to be processed; the input of the second feature extraction network is the weighted fused edge enhancement image.

[0020] In a third aspect of this application, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the aforementioned image fusion-based picking point localization method.

[0021] In a fourth aspect of this application, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0022] This application has at least the following beneficial effects:

[0023] The image fusion-based picking point localization method provided in this application generates a weighted fused edge-enhanced image by weightedly fusing the crop image to be processed with an initial edge image based on gradient calculation. This method retains the color and texture features used for maturity judgment in the original image while highlighting the contour structure of the fruit and stem through edge enhancement. It avoids the defects of traditional edge enhancement methods, such as destroying color information or introducing too much noise, thus laying a data foundation for achieving accurate maturity recognition and picking point localization simultaneously. Next, the picking point localization model employs a symmetrical dual-feature extraction network, taking the original image and the weighted fused edge-enhanced image as inputs respectively. The symmetrical structure ensures that the two feature paths are spatially aligned at the same scale, allowing color features and edge structure features to fully collaborate. This solves the problem of insufficient collaboration caused by feature scale mismatch or fixed weights in some multi-branch schemes. Especially for the non-square and small-volume morphological characteristics of the crop to be processed, it can more accurately capture the key features of thin stems and small fruits, reducing false positives and false negatives against complex backgrounds. This application does not rely on expensive equipment such as high-precision depth cameras. While controlling costs, it effectively addresses complex scenarios such as light fluctuations, leaf shading, and fruit overlap. It not only meets the accuracy requirements for maturity identification but also ensures the reliability of picking point positioning, making it more suitable for the application of automated picking equipment in large-scale planting scenarios. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of the image fusion-based picking point localization method provided in this application embodiment;

[0026] Figure 2 This is a Scharr weighted fusion edge enhancement effect diagram provided in an embodiment of this application; wherein, Figure 2 Image (a) is the original RGB image; Figure 2 (b) shows the edge enhancement effect of Scharr weighted blending;

[0027] Figure 3 A flowchart illustrating the construction process of the picking point positioning model provided in this application embodiment;

[0028] Figure 4 A diagram of the DEPS-YOLOv11 network architecture provided for embodiments of this application;

[0029] Figure 5 This is a schematic diagram of the EAFM feature fusion module structure provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the windmill-shaped convolutional PConv structure provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the Shape-IoU loss function provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of strawberry image annotation provided in an embodiment of this application;

[0033] Figure 9 This is a comparison chart of the detection performance of the YOLOv11 network and the DEPS-YOLOv11 network provided in the embodiments of this application; wherein, Figure 9 (a) shows the detection results of the original YOLOv11 network; Figure 9 (b) shows the detection results of the DEPS-YOLOv11 network;

[0034] Figure 10 This is a structural block diagram of the image fusion-based picking point positioning device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0037] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0038] Please refer to Figure 1 As shown, embodiments of this application provide a method for locating picking points based on image fusion, the method comprising the following steps:

[0039] S100, acquire images of crops to be processed; wherein, the images of crops to be processed contain several crops to be processed.

[0040] Specifically, each crop to be processed has a corresponding maturity level; maturity levels include fully mature, semi-mature, and immature; the crops to be processed are not square, and their volume is smaller than a preset volume. In one embodiment, the crop to be processed can be strawberries, and the image acquisition method is as follows: using an Azure Kinect DK depth camera as the acquisition device, the camera distance from the strawberry plant is controlled at 20-60 cm during the acquisition process. The acquisition targets are elevated strawberries at different maturity levels, including fully mature ripe strawberries, semi-mature white-ripe strawberries, and immature green strawberries.

[0041] S200 converts the crop image to be processed into a grayscale image of the crop.

[0042] S300, calculate the gradient of the grayscale image of the crop to be processed in the vertical edge detection direction and the horizontal edge detection direction respectively, so as to obtain the grayscale gradient image in the vertical direction and the grayscale gradient image in the horizontal direction.

[0043] Specifically, the Scharr operator is used to calculate the gradients of the grayscale image of the crop being processed in the directions of detecting vertical and horizontal edges, resulting in vertical and horizontal grayscale gradient maps. The Scharr operator uses two 3×3 convolution kernels (corresponding to the horizontal and vertical directions respectively) to slide across each pixel of the image. The convolution kernels are weighted and summed with the pixel region to obtain the gradient value of that point in the corresponding direction. The larger the gradient value, the more drastic the grayscale change at that point, and the more likely it is to be an edge.

[0044] The Scharr operator is a gradient calculation operator used for image edge detection. Its core function is to accurately extract the edge contours of targets (such as the fruit boundary and stem lines of a strawberry) by enhancing the regions of abrupt changes in gray values ​​in the image.

[0045] S400 performs weighted fusion of the vertical and horizontal grayscale gradient images to obtain the middle edge image.

[0046] Specifically, step S400 also includes:

[0047] S410, weighted fusion of the vertical and horizontal gray-level gradient maps is performed to obtain the initial edge image; wherein, the weights corresponding to the vertical and horizontal gray-level gradient maps are determined according to the growth direction of the crop to be processed.

[0048] In one embodiment, since the fruit stems of elevated strawberries mainly grow vertically, to enhance the stem edges to assist in picking point positioning, while also considering fruit edge representation, the weights for the X-direction (detecting vertical edges) and Y-direction (detecting horizontal edges) are set to 0.6, meaning the weight for detecting vertical edges is greater than the weight for detecting horizontal edges. Then, the two gradient images are weighted and fused to obtain the initial edge image, as shown below. Figure 2 As shown, where, Figure 2 Image (a) is the original RGB image; Figure 2 (b) shows the edge enhancement effect of Scharr weighted blending.

[0049] S420 performs weighted fusion of the vertical and horizontal grayscale gradient images to obtain the middle edge image; the middle edge image is a three-channel image.

[0050] Here, the CLAHE operator is used to optimize the sharpness and contrast of the initial edge map, which is then normalized (pixel values ​​are mapped to 0-255) and converted into a three-channel image.

[0051] Among them, the CLAHE operator is used to optimize the initial edge map. The core is to improve the contrast by dividing the image into blocks and adaptively enhancing the contrast. This solves the problems of blurred edges in local dark areas and loss of details in local bright areas in the initial edge map, making the edge lines of the fruit stem and fruit clearer and the contrast more balanced, thus providing higher quality edge features for subsequent fusion.

[0052] S500 performs a weighted summation of the crop image to be processed and the intermediate edge image to obtain a weighted fused edge-enhanced image; wherein, the weight corresponding to the crop image to be processed is greater than the weight corresponding to the intermediate edge image.

[0053] Specifically, the weight corresponding to the crop image to be processed is greater than the weight corresponding to the middle edge image. For example, the crop image to be processed and the middle edge image are weighted and fused at a ratio of 1:0.5 to generate a Scharr weighted fused edge enhancement map. This preserves the main features of the original RGB image, such as color and texture, while highlighting the outline features of the fruit and stem. Furthermore, it maximizes the preservation of the strawberry's color features while strengthening the edges, preventing the edge image from being too large and damaging the color characteristics.

[0054] S600, the crop image to be processed and the weighted fused edge enhancement image are input into the picking point localization model to obtain the target crop in the crop image to be processed and the picking point coordinates corresponding to each target crop; wherein, the target crop is the crop to be processed that is fully mature; the picking point localization model includes a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the crop image to be processed; the input of the second feature extraction network is the weighted fused edge enhancement image.

[0055] Specifically, the picking point location model can be a DEPS-YOLOv11 network, and the construction process of the picking point location model is as follows: Figure 3 As shown, the constructed DEPS-YOLOv11 network architecture is as follows: Figure 4 As shown, a symmetrical dual-backbone structure is adopted, including a first feature extraction network and a second feature extraction network. The input of the first feature extraction network is the image of the crop to be processed; the input of the second feature extraction network is a weighted and fused edge enhancement image. The structures of the first and second feature extraction networks are completely symmetrical.

[0056] This embodiment specifically targets the enhanced extraction of prominent fruit and stem contour features in edge enhancement maps, addressing the issue of low differentiation between fruits, stems, and leaves in ordinary RGB images. The two branches have completely identical structures and are both based on the YOLOv11-Pose framework. The dual-trunk symmetrical design ensures the spatial alignment of output features at the same scale, laying the foundation for subsequent cross-branch feature fusion.

[0057] In one exemplary embodiment of this application, the picking point positioning model includes a feature fusion module.

[0058] Here, the feature fusion module is the EAFM module, and the structure of the EAFM module is as follows: Figure 5 As shown, an EAFM module is added to the front end of the Head layer of the DEPS-YOLOv11 network. One EAFM module is added for each of the three sets of homogeneous scale features P3 (8x downsampling), P4 (16x downsampling), and P5 (32x downsampling) output by the dual branches. Specifically, the first EAFM module receives the P3 features from the first feature extraction network and the P3 features from the second feature extraction network; the second EAFM module receives the P4 features from the first feature extraction network and the P4 features from the second feature extraction network; and the third EAFM module receives the P5 features from the first feature extraction network and the P5 features from the second feature extraction network. The three sets of features fused by the EAFM module are directly connected to the subsequent upsampling, feature splicing, and detection units in the Head layer, ensuring that the dual-branch features achieve accurate complementarity across the entire scale.

[0059] The feature fusion module is used to perform the following steps:

[0060] S610, perform channel alignment on the image features extracted by the first feature extraction network and the edge features extracted by the second feature extraction network based on the average number of channels to obtain aligned image features and aligned edge features; wherein, the average number of channels is the average of the number of channels corresponding to the image features extracted by the first feature extraction network and the number of channels corresponding to the edge features extracted by the second feature extraction network.

[0061] Here, the channel dimension difference of the dual-branch input features is eliminated by the channel alignment unit. Let the number of input feature channels of the first feature extraction network be C1 and the number of input feature channels of the second feature extraction network be C2. As a unified number of channels, the features of the first feature extraction network and the second feature extraction network are mapped to C through two 1×1 convolutional layers, respectively. max Channel, output aligned features:

[0062] ;

[0063] Where B is the batch size, and H and W are the height and width of the feature map.

[0064] S620 generates weights corresponding to the aligned image features and the aligned edge features based on the aligned image features, aligned edge features, and the multilayer perceptron network.

[0065] Here, first... and Summation yields Features are compressed into 1×1 scale using global average pooling (GAP). Figure X gap To capture global information, a weight matrix W is generated via an MLP network, expressed as:

[0066] ;

[0067] Where W = [W1, W2], W1 is the weight of the first feature extraction network feature, and W2 is the weight of the second feature extraction network feature. The Sigmoid function ensures that the weight values ​​are in the range of [0, 1], enabling dynamic adjustment according to the scene.

[0068] S630, a weighted sum is performed based on the aligned image features, aligned edge features, the weights corresponding to the aligned image features, and the weights corresponding to the aligned edge features to obtain preliminary fusion features.

[0069] Here, a broadcast mechanism is used to match W1 and W2 with the feature map spatial dimension (H×W). , After multiplying each element and summing the results, we obtain the preliminary fusion characteristics:

[0070] ;

[0071] In the formula, × represents element-wise multiplication. These are intermediate features after fusion.

[0072] S640, if the average number of channels is not equal to the target number of output channels, the residual benchmark is mapped to the target output channels to obtain the residual characteristics.

[0073] Here, to preserve the core information of the original features of the two branches and alleviate the gradient vanishing effect in deep networks, the residual enhancement unit uses the mean of the aligned features as the residual benchmark. If C max With the target number of output channels C out Inconsistent, through a 1×1 convolutional layer Mapping to C out Channel, otherwise directly used as residual feature X res ,Right now:

[0074] ;

[0075] in Conv res This is a 1×1 convolutional layer used for channel adjustment.

[0076] S650 reduces the dimensionality of the initial fusion features to the target output channel in order to obtain the key fusion features.

[0077] S660 defines the sum of key fusion features and residual features as the target fusion feature.

[0078] Here, the initial fusion features will be... Reduced to C using a 1×1 convolutional layer out The channel, obtained , and residual characteristic X res The final fused feature X is output after addition. fuse This enables adaptive fusion of features from two branches.

[0079] In the strawberry detection task under complex environments described above, the Feature Fusion Module (EAFM module) demonstrates significant adaptability. For scenarios with leaf occlusion, the module automatically increases the weight of edge features in the second feature extraction network, strengthening the contour features of the fruit and stem to address the difficulty in distinguishing the fruit, stem, and leaves. For scenarios with uniform lighting and clear fruit color, the module automatically increases the weight of RGB features in the first feature extraction network, ensuring the accuracy of strawberry maturity classification. Simultaneously, the channel alignment design adapts to feature inputs from different channel dimensions in the dual-branch architecture, residual connections prevent the loss of key information during fusion, and it is highly compatible with the dual-backbone structure, effectively improving the robustness of the picking point localization model in complex planting scenarios.

[0080] In one exemplary embodiment of this application, if the average number of channels is equal to the target number of output channels, the residual benchmark is determined as the residual feature.

[0081] In one exemplary embodiment of this application, the first and second feature extraction networks each replace the first two standard 3×3 convolutions closest to the input image with windmill-shaped convolutions (PConv) in their C3k2 modules.

[0082] Specifically, the standard convolution in the traditional C3k2 module has limitations: on the one hand, strawberry stems are slender and small-scale, and the fruit is elliptical; the pixel grayscale of both decreases in a Gaussian-like manner from the center to the edge, while standard convolution uses symmetrical uniform weights, making it difficult to capture key features at the center of the target and susceptible to interference from branch and leaf clutter; on the other hand, when expanding the receptive field to improve the stem feature extraction capability, traditional methods lead to a surge in parameters or increased complexity, which is not conducive to lightweight deployment. Therefore, this application introduces the windmill-shaped convolution PConv into the C3k2 module. The structure of PConv is as follows... Figure 6 As shown, the first two standard 3×3 convolutions responsible for low-level feature capture within the module are replaced, specifically the first two standard 3×3 convolutions closest to the input image. This location is crucial for extracting detailed strawberry features. Without modifying the overall C3k2 structure, this results in the improved C3k2_PConv module. Introducing PConv allows the module to adapt to the Gaussian-like feature distribution of strawberries, enhancing key features of the stem and fruit. Simultaneously, it achieves efficient expansion of the receptive field with low parameter increments, maintaining the model's lightweight attributes and real-time inference capabilities, thus meeting the detection requirements of elevated strawberry picking scenarios.

[0083] PConv extracts strawberry target features through asymmetric padding, multi-directional parallel convolution, feature concatenation, and channel normalization. Let the input feature map of the C3k2 module be... Where h1 is the feature map height, w1 is the width, and c1 is the number of input channels. PConv first extracts multi-directional features through 4-way parallel convolution, and its calculation formula is:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] Taking one of the convolutional paths as an example, in the formula... This represents the input features after asymmetric padding (1 pixel left, 0 pixels right, 0 pixels top, and 3 pixels bottom). Represents the convolution operation. The number of output channels is 1×3 convolution kernel ( (c2 is the final output channel number of PConv), BN is the batch normalization layer, SiLU is the activation function, and X1 is the feature map output by this convolution path; the other 3 convolution paths only differ in padding parameters and kernel size, and the size of the output feature map is consistent with X1.

[0089] The relationship between the size of the feature map after parallel convolution and the size of the input feature map and the convolution stride s can be expressed as:

[0090] ;

[0091] In the PConv module of C3k2, the convolution stride s is set to 1 to preserve the detailed features of the strawberry target. The final output feature map size is [size missing]. , The output size is exactly the same as that of standard convolution, ensuring module compatibility.

[0092] After parallel convolution is completed, the feature maps from the four outputs are merged through a concatenation operation, as shown in the formula:

[0093] ;

[0094] Where Cat represents the feature concatenation operation, and the number of channels in the concatenated feature map is... Then, a 2×2 convolution kernel without padding is applied. After channel normalization, the final output feature map is calculated using the following formula:

[0095] ;

[0096] The output feature map is directly fed into the subsequent processing stage of the C3k2 module to complete the extraction of low-level features of the strawberry target.

[0097] In terms of parameter scale, when the number of input channels equals the number of output channels, the parameter calculation formula for a standard 3×3 convolution is:

[0098] ;

[0099] Where k=3 is the kernel size. The formula for calculating the parameters of PConv is:

[0100] ;

[0101] As can be seen from the comparison, PConv has a smaller parameter scale, while achieving a significant expansion of the receptive field, perfectly meeting the feature extraction needs of strawberry-shaped targets.

[0102] In one exemplary embodiment of this application, the loss function of the picking point positioning model includes a shape-aware distance loss function and a shape-aware shape loss function; the shape-aware distance loss function calculates the weighted center deviation by dynamically allocating directional weights; wherein, the penalty weight for the center offset of the short side direction of the bounding box of the crop to be processed is greater than the penalty weight for the center offset of the long side direction of the bounding box of the crop to be processed; the shape-aware shape loss function calculates the weighted length and width deviations.

[0103] Specifically, strawberry fruits are mostly oval-shaped, with slender stems and relatively small overall size. Traditional bounding box regression loss in the YOLOv11-Pose network suffers from insufficient regression accuracy because it fails to consider the key characteristics of strawberry targets—non-squareness and small scale. To address this issue, this application introduces the Shape-IoU loss function. Figure 7 This is a schematic diagram of the Shape-IoU loss function. The Shape-IoU loss function replaces the Complete IoU (CIoU) loss function in the original bounding box regression loss part. Shape-IoU incorporates the shape and scale factor of the bounding box itself into the loss calculation. By dynamically allocating directional weights, it prioritizes constraints on deviations that have a more significant impact on the strawberry, such as short side directional offset and aspect ratio errors of small-scale targets. This improves the fit of the fruit bounding box and the accuracy of the fruit stem picking point location, enhancing the model's robustness to complex occlusion scenes. Shape-IoU first determines the strawberry's ground truth bounding box width w. gt Height h gt And the scale factor, scale, generates the horizontal weight ww and the vertical weight hh, as follows:

[0104] ;

[0105] If the strawberry stem is perpendicular to the long side, hh will be greater than ww, thus providing a higher constraint on the vertical deviation.

[0106] Secondly, two types of weighted losses are constructed: one is the shape-aware distance loss, where weighted center bias is used to preferentially penalize short-side offset.

[0107] ;

[0108] Where, x c y c The x-coordinate and y-coordinate of the center of the prediction box are given. , denoted as x-coordinate and y-coordinate of the center of the ground truth (GT) box, and c is the diagonal length of the minimum bounding box between the predicted box and the GT box.

[0109] Second, shape perception loss is used, with weighted length and width biases to strengthen constraints on small-scale targets:

[0110] ;

[0111] in, , represents the width and height deviations between the weighted predicted bounding box and the ground truth bounding box, where w and h are the width and height of the predicted bounding box, respectively.

[0112] The final result is the total loss:

[0113] ;

[0114] Where B is the predicted bounding box, B gt The result is a ground truth bounding box. This loss function takes into account overlap, position, and aspect ratio constraints, precisely adapting to the needs of strawberry detection.

[0115] In one exemplary embodiment of this application, the picking point localization model performs data augmentation processing on the labeled training set images during training; wherein, the augmentation processing includes adjusting brightness, horizontal flipping, adjusting contrast, and adding Gaussian noise.

[0116] Specifically, the Azure Kinect DK depth camera was used as the data acquisition device, and the camera distance from the strawberry plant was controlled at 20-60 cm during the acquisition process. The subjects collected included elevated strawberries at different stages of maturity: ripe, white-ripe, and green. The data covered various lighting conditions, shooting distances, shooting angles, and complex scenarios that might be encountered during actual harvesting, such as unobstructed views, leaf obstructions, and fruit occlusion. All acquired images were stored in 1920×1080 resolution PNG format.

[0117] The Labelme annotation tool was used, with the same researcher annotating the images. Strawberry fruit categories were labeled using rectangles, categorized according to the proportion of red area: fully ripe (ripe strawberry, red area exceeding 70%), semi-ripe (white ripening, red area less than 70%, pink or red-white transition), and unripe (green strawberry, entirely green). Only fully ripe strawberries were annotated with two key points: a joint point at the junction of the fruit and stem, and a pick point 0.5-1 cm from the stem, serving as the picking point. The rectangles for ripe strawberries included both key points and part of the stem. The annotation results were initially stored in JSON format and then converted to TXT format for model training. Image annotations are shown below. Figure 8 As shown.

[0118] To improve the model's generalization ability and robustness to complex scenes, data augmentation processing was performed on the labeled training set images. Specifically, four techniques were used: adjusting brightness to simulate different light intensity changes, performing horizontal flipping (90°, 180°, 270°) to expand the diversity of sample spatial layout, adjusting contrast to enhance the grayscale difference between the fruit and the background, and adding Gaussian noise to simulate noise interference in actual imaging or transmission.

[0119] The augmented image is then subjected to Scharr weighted fusion edge enhancement processing, as described above.

[0120] The processed RGB image data and the Scharr-weighted fused edge-enhanced image data were randomly divided into training, validation, and test sets in a 7:2:1 ratio. The RGB images were used as input to the first feature extraction network, and the Scharr-weighted fused edge-enhanced images were used as input to the second feature extraction network. The constructed DEPS-YOLOv11 network was trained using the labeled dataset.

[0121] The trained DEPS-YOLOv11 model was applied to the validation set to evaluate its detection performance under different lighting conditions, complex backgrounds, and occlusions. The robustness and accuracy of the model were verified by calculating precision (P), recall (R), and average precision (AP).

[0122] The model was further applied to the test set to compare the detection effects of different models or improved schemes. Through visualization results and quantitative analysis, the advantages of the DEPS-YOLOv11 model in identifying strawberry maturity and detecting key picking points in complex agricultural scenarios were verified to ensure that it meets the detection needs in actual picking operations. Figure 9 This is a comparison chart of the detection performance of the original network (YOLOv11 network) and the DEPS-YOLOv11 network; among them, Figure 9 (a) shows the detection results of the original YOLOv11 network; Figure 9 (b) shows the detection effect of the DEPS-YOLOv11 network; the part in the white circle is the key point of positioning deviation.

[0123] This application employs the Scharr weighted fusion edge enhancement image processing method. Combining the growth morphology characteristics of elevated strawberry stems, the Scharr operator detection direction weights are set differently to directionally enhance the stem edge features and assist in the location of picking points. At the same time, the edge image clarity is optimized through the CLAHE operator and fused with the original RGB image at a specific ratio. This not only preserves the color and texture features required for strawberry ripeness recognition but also highlights the outlines of the fruit and stem, effectively alleviating the problem of confusion between fruit, stem, and green leaves in complex scenes and providing more comprehensive feature input for subsequent networks. A dual-branch network for edge enhancement feature fusion is constructed. The first feature extraction network extracts color and texture features from the original RGB image as input to support strawberry maturity classification. The second feature extraction network uses the Scharr weighted fusion edge enhancement image as input to enhance the contour structure features of the fruit and stem. The dual-branch symmetrical design ensures alignment of the output feature space. At the same time, an edge enhancement adaptive fusion module EAFM is designed. Through mechanisms such as channel alignment and dynamic weight generation, it can automatically adjust the contribution of the dual-branch features according to the scene. Combined with residual enhancement and feature dimensionality reduction, it avoids the loss of key information and achieves accurate complementarity of color and structural features at all scales, significantly improving the model's adaptability to complex planting scenarios. The windmill-shaped convolution PConv is introduced into the C3k2 module, replacing the standard convolution for low-level feature capture. This allows for adaptation to the Gaussian decay feature distribution of strawberry pixels without altering the overall module structure, accurately extracting small-scale details such as slender stems and oval fruits. Simultaneously, PConv employs a 4-way multi-directional parallel convolution design, achieving efficient receptive field expansion with low parameter increments. Its parameter scale is significantly lower than traditional standard convolutions, ensuring a lightweight model to meet the real-time inference needs of automated harvesting equipment while enhancing small-scale target detection capabilities and improving stem localization and fruit recognition accuracy. The Shape-IoU loss function replaces the traditional CIoU loss function. Addressing the non-square and small-scale morphological characteristics of strawberries, dynamic directional weights are generated based on the strawberry ground truth bounding box parameters, prioritizing the constraint of bias types that have a greater impact on strawberry detection. Furthermore, multiple loss dimensions are integrated, considering bounding box overlap, position, and aspect ratio optimization, significantly improving fruit bounding box fit and stem picking point localization accuracy. The model's robustness is particularly enhanced in scenarios with overlapping fruits and leaf occlusion.

[0124] The overall solution presented in this application forms a complete technology chain, with each module working synergistically. Scharr weighted fusion edge enhancement images provide high-quality input to the dual-branch network, while the dual-branch and EAFM modules ensure comprehensive feature fusion. PConv enhances detailed feature extraction, and Shape-IoU optimizes bounding box regression. Furthermore, the model does not rely on expensive equipment such as high-precision depth cameras, making costs controllable. It also balances lightweight design and real-time performance, making it easy to integrate with automated harvesting equipment. It can effectively handle complex agricultural scenarios and address the pain points of existing technologies, such as poor environmental adaptability, insufficient accuracy, and high costs. This provides practical and efficient technical support for intelligent harvesting in large-scale strawberry cultivation.

[0125] Please refer to Figure 10 As shown, an embodiment of this application provides a picking point positioning device 100 based on image fusion, the device comprising:

[0126] The acquisition unit 110 is used to acquire images of crops to be processed; wherein, the images of crops to be processed contain several crops to be processed.

[0127] The conversion unit 120 is used to convert the crop image to be processed into a grayscale image of the crop.

[0128] The calculation unit 130 is used to calculate the gradient of the grayscale image of the crop to be processed in the direction of detecting the vertical edge and the direction of detecting the horizontal edge, respectively, so as to obtain the grayscale gradient image in the vertical direction and the grayscale gradient image in the horizontal direction.

[0129] The intermediate image acquisition unit 140 is used to perform weighted fusion of the vertical grayscale gradient map and the horizontal grayscale gradient map to obtain the intermediate edge image.

[0130] The fusion unit 150 performs a weighted summation of the crop image to be processed and the intermediate edge image to obtain a weighted fused edge-enhanced image; wherein the weight corresponding to the crop image to be processed is greater than the weight corresponding to the intermediate edge image.

[0131] The result acquisition unit 160 is used to input the crop image to be processed and the weighted fused edge enhancement image into the picking point localization model to obtain the target crop in the crop image to be processed and the picking point coordinates corresponding to each target crop; wherein, the target crop is the crop to be processed with the corresponding maturity level of full maturity; the picking point localization model includes a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the crop image to be processed; the input of the second feature extraction network is the weighted fused edge enhancement image.

[0132] Embodiments of this application also provide a computer program product including program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above according to various exemplary embodiments of this application.

[0133] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0134] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0135] In an exemplary embodiment of this application, an electronic device capable of implementing the above-described method is also provided.

[0136] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0137] An electronic device according to this embodiment of the present application. The electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0138] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0139] The memory stores program code that can be executed by a processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this application.

[0140] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0141] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0142] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0143] The electronic device can also communicate with one or more external devices (such as keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (such as routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0145] In exemplary embodiments of this application, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this application may also be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0146] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0147] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0149] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0150] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0151] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0152] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A picking point positioning method based on image fusion, characterized in that, The method comprises: Collecting a to-be-processed crop image; wherein the to-be-processed crop image contains a plurality of to-be-processed crops; Converting the to-be-processed crop image into a to-be-processed crop grayscale image; Respectively calculating the gradients of the to-be-processed crop grayscale image in the detection vertical edge direction and the detection horizontal edge direction to obtain a vertical direction grayscale gradient image and a horizontal direction grayscale gradient image; Performing weighted fusion on the vertical direction grayscale gradient image and the horizontal direction grayscale gradient image to obtain an intermediate edge image; Performing weighted summation on the to-be-processed crop image and the intermediate edge image to obtain a weighted fusion edge enhancement image; wherein the weight corresponding to the to-be-processed crop image is greater than the weight corresponding to the intermediate edge image; Inputting the to-be-processed crop image and the weighted fusion edge enhancement image into a picking point positioning model to obtain target crops in the to-be-processed crop image and picking point coordinates corresponding to each target crop; wherein the target crops are to-be-processed crops with full maturity; the picking point positioning model comprises a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the to-be-processed crop image; the input of the second feature extraction network is the weighted fusion edge enhancement image.

2. The image fusion-based picking point positioning method according to claim 1, characterized in that, The weighted fusion on the vertical direction grayscale gradient image and the horizontal direction grayscale gradient image to obtain the intermediate edge image comprises: Performing weighted fusion on the vertical direction grayscale gradient image and the horizontal direction grayscale gradient image to obtain an initial edge image; wherein the weight corresponding to the vertical direction grayscale gradient image and the weight corresponding to the horizontal direction grayscale gradient image are both determined according to the growth direction of the to-be-processed crop; Optimizing the definition and contrast of the initial edge image according to a CLAHE operator, performing a normalization operation, and converting into a three-channel image to obtain the intermediate edge image.

3. The image fusion-based picking point positioning method according to claim 1, characterized in that, The picking point positioning model comprises a feature fusion module; wherein the feature fusion module is used to perform the following steps: Aligning the image features extracted by the first feature extraction network and the edge features extracted by the second feature extraction network according to an average channel number to obtain aligned image features and aligned edge features; wherein the average channel number is the average of the channel number corresponding to the image features extracted by the first feature extraction network and the channel number corresponding to the edge features extracted by the second feature extraction network; Generating the weight corresponding to the aligned image features and the weight corresponding to the aligned edge features according to the aligned image features, the aligned edge features, and a multi-layer perception network; Performing weighted summation on the aligned image features, the aligned edge features, the weight corresponding to the aligned image features, and the weight corresponding to the aligned edge features to obtain preliminary fusion features; If the average channel number is not equal to a target output channel number, mapping a residual reference to the target output channel to obtain residual features; Reducing the dimensionality of the preliminary fusion features to the target output channel to obtain key fusion features; Determining the sum of the key fusion features and the residual features as target fusion features.

4. The picking point positioning method based on image fusion according to claim 3, characterized in that, After the weighted summation on the aligned image features, the aligned edge features, the weight corresponding to the aligned image features, and the weight corresponding to the aligned edge features to obtain the preliminary fusion features, the feature fusion module is further used to perform the following steps: If the average channel number is equal to the target output channel number, the residual reference is determined as the residual feature.

5. The image fusion-based picking point positioning method according to claim 1, characterized in that, In the C3k2 modules of the first feature extraction network and the second feature extraction network, windmill-shaped convolution PConv is used to replace the first two layers of standard 3*3 convolution close to the input image.

6. The image fusion-based picking point positioning method according to claim 1, characterized in that, The loss function of the picking point positioning model comprises a shape-aware distance loss function and a shape-aware shape loss function; the shape-aware distance loss function calculates a weighted center deviation by dynamically assigning a direction weight; wherein the penalty weight of the center deviation in the short side direction of the to-be-processed crop bounding box is greater than the penalty weight of the center deviation in the long side direction of the to-be-processed crop bounding box; the shape-aware shape loss function calculates a weighted length-width deviation.

7. The image fusion-based picking point positioning method according to claim 1, characterized in that, During training, the picking point positioning model performs data enhancement processing on the labeled training set images; wherein the enhancement processing includes adjusting brightness, horizontal flipping, adjusting contrast and adding Gaussian noise.

8. An image fusion-based picking point positioning device, characterized by, The apparatus comprises: The acquisition unit is configured to acquire a to-be-processed crop image; wherein the to-be-processed crop image contains a plurality of to-be-processed crops. The conversion unit is configured to convert the to-be-processed crop image into a to-be-processed crop grayscale image. The calculation unit is configured to calculate the gradient of the to-be-processed crop grayscale image in the detection vertical edge direction and the detection horizontal edge direction respectively, to obtain a vertical direction grayscale gradient image and a horizontal direction grayscale gradient image. The intermediate image acquisition unit is configured to perform weighted fusion on the vertical direction grayscale gradient image and the horizontal direction grayscale gradient image, to obtain an intermediate edge image. The fusion unit is configured to perform weighted summation on the to-be-processed crop image and the intermediate edge image, to obtain a weighted fusion edge enhanced image; wherein the weight corresponding to the to-be-processed crop image is greater than the weight corresponding to the intermediate edge image. The result acquisition unit is configured to input the to-be-processed crop image and the weighted fusion edge enhanced image into a picking point positioning model, to obtain target crops in the to-be-processed crop image and picking point coordinates corresponding to each target crop; wherein the target crop is a to-be-processed crop with full maturity; the picking point positioning model comprises a first feature extraction network and a second feature extraction network; the input of the first feature extraction network is the to-be-processed crop image; the input of the second feature extraction network is the weighted fusion edge enhanced image.

9. A non-transitory computer-readable storage medium, comprising: The storage medium stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the method of any one of claims 1-7.

10. An electronic device, comprising: The apparatus comprises a processor and a non-transitory computer-readable storage medium as claimed in claim 9. The apparatus comprises a processor and a non-transitory computer-readable storage medium as claimed in claim 9.

Images