Apple identification method, system and device and storage medium

Abstract

The invention discloses an apple recognition method, system and device and a storage medium, and relates to the technical field of apple recognition and detection, and the method comprises the following steps: calculating feature points on each edge contour in an original infrared thermal image and a visible light image based on the curvature of the edge contour; determining the main direction of the feature point based on the position information of each feature point on the edge contour; performing parametric coding on the position and the main direction of each feature point to generate a feature vector of each feature point; matching based on the Euclidean distance of the feature vector of each feature point on the two images to obtain a plurality of matching pairs; determining a transformation matrix through the coordinates of the plurality of matching pairs; performing image registration on the original infrared thermal image and the visible light image based on the transformation matrix to obtain a registered image; and carrying out image fusion and target identification on the registered image and the visible light image to obtain the position, the category and the confidence of the apple. According to the invention, the accuracy of transformation matrix estimation and the precision of registration are improved.

Description

Technical Field

[0001] This invention relates to the field of apple recognition and detection technology, and in particular to an apple recognition method, system, device and storage medium. Background Technology

[0002] Apples require intensive harvesting during their ripening period, and harvesting is one of the most time-consuming and labor-intensive aspects of apple cultivation. With an aging population and the loss of agricultural labor, the labor costs for apple harvesting are increasing. Therefore, researching automated and intelligent apple-harvesting robots to replace manual labor in apple picking is urgently needed.

[0003] Apple recognition and localization based on computer vision is a current research focus for apple-picking robots. The mainstream methods for apple recognition are visible light image recognition and thermal imaging image recognition. Visible light image recognition is greatly affected by ambient lighting conditions, with accuracy decreasing significantly in strong light, backlight, and cloudy weather. Thermal imaging, while providing stable imaging results under these conditions, is significantly affected by temperature changes, has lower resolution, and lacks texture and other detailed information, leading to decreased recognition accuracy. Therefore, it is necessary to combine visible light and thermal imaging for apple recognition. Since their imaging principles differ, image registration is required beforehand during the recognition process.

[0004] Existing registration techniques, such as traditional feature point extraction methods (e.g., SIFT, SURF, and ORB), rely on stable local features. In complex scenarios such as foliage occlusion, they are prone to failure due to incomplete information, requiring manual intervention for optimization. Furthermore, they have poor robustness to noise. Deep learning-based feature point extraction methods rely on large-scale labeled data. When the sample size is insufficient, they are prone to overfitting, limiting their generalization ability. Summary of the Invention

[0005] Based on the shortcomings of the existing technologies, the present invention provides an apple recognition method, system, device and storage medium, which solves the problems that existing apple target detection based on visible light images has poor detection performance under poor lighting conditions, while infrared thermal image recognition is significantly affected by temperature changes and has low resolution.

[0006] The present invention adopts the following technical solution: In a first aspect, the present invention provides an apple identification method, comprising the following steps: Acquire raw infrared thermal and visible light images containing apples; Multiple edge contours are extracted from the original infrared thermal image and visible light image respectively, and feature points on each edge contour are calculated based on the curvature of the edge contours; the principal direction of each feature point is determined based on the position information of each feature point on the edge contour; the position and principal direction of each feature point are parameterized and encoded to generate the feature vector of each feature point. Matching is performed based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image to obtain multiple matching pairs; the transformation matrix is ​​determined by the coordinates of the multiple matching pairs; image registration is performed on the original infrared thermal image and the visible light image based on the transformation matrix to obtain the registered image; The registered image and the visible light image are fused to obtain a fused image. The fused image is then used to identify the apple target, and the location, category, and confidence level of the apple are obtained.

[0007] Preferably, the step of extracting multiple edge contours from the original infrared thermal image and visible light image respectively, and calculating feature points on each edge contour based on the curvature of the edge contours, includes the following steps: Image enhancement is performed on the original infrared thermal image to obtain an enhanced infrared thermal image; Multi-scale second-order gradient smoothing denoising is performed sequentially on the enhanced infrared thermal image and the visible light image. The convolution template is applied to each pixel, and its gradient magnitude is calculated. S With direction Based on gradient magnitude S With direction The edge points in the image are selected and connected to obtain the edge contour. Calculate the curvature of each pixel on each edge contour and iterate through them. If the curvature of the current pixel is the largest in a square neighborhood with a width of 9 pixels centered on it, then set it as a candidate corner point. Iterate through multiple candidate corner points. If the curvature value of the current candidate corner point is greater than the curvature threshold, then set it as a feature point.

[0008] Preferably, determining the main direction of each feature point based on its position information on the edge contour includes the following steps: For each edge contour If the current feature point If the feature point is not located at the starting or ending position of the edge contour, then the feature point corresponding to the minimum curvature on its left is obtained. The feature point corresponding to the minimum curvature on its right is ; Using second-order gradient convolution kernels respectively and All feature points between and and Second-order gradient smoothing sampling is performed on all feature points in between to obtain the first auxiliary point. Second auxiliary point The first auxiliary point and the second auxiliary point are specifically shown below: ; ; ; In the formula, For discrete convolution kernels, The size parameter of the convolution kernel. x For discrete index variables, and They are respectively and The number of pixels between two points and the current feature point; The direction of the angle bisector vector of the angle formed by the lines connecting the current feature point to the first auxiliary point and the second auxiliary point is taken as the main direction of the current feature point.

[0009] Preferably, the matching based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image includes the following steps: Obtain the Euclidean distance between the feature vectors of each feature point in the enhanced infrared thermal image and the visible light image; When the Euclidean distance between a feature point in an enhanced infrared thermal image and a feature point in a visible light image is less than a set distance threshold, the two feature points are determined to be a matching pair. Determining the transformation matrix using the coordinates of multiple matching pairs includes the following steps: Connect the two feature points of each matching pair with a straight line to obtain the matching line; calculate the tilt angle and length of each matching line; Calculate the histogram of the principal direction differences for each pair of matching points and take the maximum value. Use the reference tilt angle and set the tilt threshold. ; Filtering tilt angle at Find multiple first matching lines within the interval, and then calculate the average length of the multiple first matching lines. Set a length error threshold The filter length is within The second matching line of the interval is the final matching line; The matching pairs corresponding to the final matching line are valid matching pairs; The transformation matrix is ​​determined by the coordinates in the effective matching pairs, and the specific transformation matrix is ​​as follows: ; In the formula, PTo enhance the coordinates of matching points in infrared thermal images, P 'These are the coordinates of the matching point in the visible light image.' H Let be the transformation matrix.

[0010] Preferably, the step of enhancing the original infrared thermal image to obtain an enhanced infrared thermal image includes the following steps: The mean and standard deviation of the thermal temperature matrix are obtained, and the difference between the mean and standard deviation is used to obtain the lowest temperature; wherein, the temperature value corresponding to each pixel in the infrared thermal image of the apple is represented in matrix form to obtain the thermal temperature matrix; Based on the lowest temperature, each temperature value in the thermal temperature matrix is ​​normalized and mapped to obtain the grayscale intensity of each pixel. The grayscale intensities of multiple pixels are combined to obtain a thermal imaging grayscale image. Histogram equalization is performed on the thermal imaging grayscale image to obtain an enhanced infrared thermal image.

[0011] Preferably, the image fusion of the registered image and the visible light image specifically includes the following steps: The registered image and the visible light image are input into the image fusion module for pixel-level fusion to obtain the fused image; The fused image is input into the semantic segmentation module for pixel-by-pixel semantic recognition, resulting in a fused image containing semantic information.

[0012] Preferably, the fused image is used for apple target recognition based on the improved YOLOv8; wherein, the distance attention weight is introduced to improve the original loss function CIoU of the original YOLOv8 model; full-dimensional dynamic convolution is introduced into the C2f module of the Neck module in the original YOLOv8 model, and an efficient channel attention mechanism SA module is added to the Backbone part.

[0013] Secondly, the present invention provides an Apple identification system, comprising: The acquisition module is used to acquire raw infrared thermal images and visible light images containing the apple; The generation module is used to extract multiple edge contours from the original infrared thermal image and visible light image respectively, and calculate feature points on each edge contour based on the curvature of the edge contour; determine the main direction of each feature point based on the position information of each feature point on the edge contour; and parameterize the position and main direction of each feature point to generate the feature vector of each feature point. The registration module is used to match multiple matching pairs based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image; determine the transformation matrix through the coordinates of the multiple matching pairs; and perform image registration on the original infrared thermal image and the visible light image based on the transformation matrix to obtain the registered image. The recognition module is used to fuse the registered image and the visible light image to obtain a fused image, and to perform apple target recognition on the fused image to obtain the apple's location, category, and confidence level.

[0014] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described apple recognition method.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described apple identification method.

[0016] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention first extracts multiple edge contours from the original infrared thermal image and visible light image, respectively. Feature points are calculated based on the contour curvature, ensuring that the distribution of feature points is closely related to the overall shape of the object. Even with local occlusion, the complete contour edges still provide sufficient feature information, eliminating the need for manual optimization. This is superior to traditional corner detection methods (such as SIFT) and significantly improves the integrity of information and registration stability in complex scenes. Next, the principal direction of each feature point is determined based on its position on the edge contour. The position and principal direction of each feature point are parameterized and encoded to generate a feature vector for each feature point. This feature vector contains spatial orientation and morphological features, reducing mismatches caused by changes in viewpoint or slight deformation. Then, matching is performed based on the Euclidean distance of the feature vectors of each feature point in the original infrared thermal image and visible light image, resulting in multiple matching pairs. The transformation matrix is ​​determined using the coordinates of these multiple matching pairs. The transformation matrix calculated based on accurate matching pairs better reflects the true mapping relationship between the images, improving registration accuracy. Finally, image registration is performed on the original infrared thermal image and visible light image based on the transformation matrix, resulting in a precisely aligned registered image. The entire feature extraction and matching process of this invention does not rely on a large amount of data, nor does it require sample annotation, and it has strong generalization ability. Finally, the registered image and the visible light image are fused to obtain a fused image that combines the advantages of both images. Based on the fused image, apple target recognition is performed to obtain the apple's location, category, and confidence level. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an apple recognition method according to the present invention; Figure 2 The present invention includes a visible light image actually taken in an orchard under good lighting conditions and a corresponding enhanced infrared thermal image based on the enhanced thermal properties of outdoor apples. in, Figure 2 (a): Visible light image, Figure 2 (b): Enhanced infrared thermography; Figure 3 The present invention includes visible light images actually captured in a high-light environment in an orchard and corresponding enhanced infrared thermal images based on the enhanced thermal characteristics of outdoor apples. in, Figure 3 (a): Visible light image, Figure 3 (b): Enhanced infrared thermography; Figure 4 This is a feature point extraction diagram of visible light images under different lighting conditions according to the present invention; in, Figure 4 of (a): Figure 2 Feature point extraction map of (a), Figure 4 (b): Figure 2 Feature point extraction map of (b); Figure 5 This is a schematic diagram illustrating the determination of the main direction of feature points in different feature point extraction maps according to the present invention; in, Figure 5 of (a): Figure 4 (a) is a schematic diagram of the determination of the main direction of feature points. Figure 5 (b): Figure 4 (b) Schematic diagram of the main direction determination of feature points; Figure 6 This is a schematic diagram of the matching process of the present invention; in, Figure 6 of (a): Figure 5 (a) and Figure 5 (b) Initial matching diagram, Figure 6 (b): A schematic diagram of feature point (partial) matching after consistency screening by matching lines; Figure 7 This is a framework diagram of an image fusion network according to the present invention; Figure 8 For the present invention Figure 3 (a) and Figure 3 (b) The image obtained by image registration and fusion using an image fusion network; Figure 9 This is a diagram of the original YOLOv8 network structure. Figure 10This is a diagram of the improved YOLOv8 network structure of the present invention; Figure 11 The diagram shows the C2f module structure of the original YOLOv8 network structure and the C2f_ODConv module structure of the improved YOLOv8 network structure of this invention. in, Figure 11 (a): C2f module structure diagram, Figure 11 (b): C2f_ODConv module structure diagram; Figure 12 The diagram shows the SPPF module structure of the original YOLOv8 network structure and the C2f_ODConv module structure of the improved YOLOv8 network structure of this invention. in, Figure 11 (a): SPPF block structure diagram, Figure 10 (b): C2f_ODConv module structure diagram; Figure 13 This is a structural diagram of the SA module added to the YOLOv8 network structure according to the present invention; Figure 14 For the present invention to Figure 3 (a) Results of apple target recognition using the improved YOLOv8 model directly; Figure 15 To Figure 3 (a) and Figure 3 (b) Schematic diagram of the result of apple target recognition using the improved YOLOv8 model on the fused image obtained after image registration and image fusion. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the specific field of apple recognition, existing methods do not fully utilize the temperature difference between apples and their surrounding environment during the image preprocessing stage, and cannot be optimized specifically for the characteristics of apple recognition.

[0021] To address the aforementioned problems, this invention provides an apple identification method that fully combines the advantages of both methods to improve the accuracy and reliability of apple identification in open environments. (Refer to...) Figure 1 Specifically, it includes the following steps:

[0022] Step 1: Acquire the original infrared thermal image and visible light image containing the apple. Perform image enhancement on the original infrared thermal image based on the outdoor thermal characteristics of the apple. Linearly normalize the temperature values ​​of the thermal temperature matrix based on these characteristics, then perform grayscale conversion and histogram equalization to obtain the enhanced infrared thermal image.

[0023] Reference Figure 2 (a) Due to the physiological activities of apples and their absorption and reflection of environmental heat, their temperature is generally higher than that of the tree canopy environment. This is reflected in the thermal temperature matrix as the temperature value of the corresponding area of ​​the apple being higher than the average temperature of the matrix. In an infrared thermal image, each pixel represents the temperature value of the corresponding area. Representing the temperature value of each pixel in matrix form constitutes the thermal temperature matrix. Image enhancement based on the thermal characteristics of outdoor apples is used to enhance the acquired infrared thermal image. The specific process includes the following steps:

[0024] Step 101: Perform adaptive parameter calculation based on the statistical characteristics of the thermal temperature matrix, specifically calculating the temperature mean of the thermal temperature matrix. with standard deviation and in formula The minimum temperature is dynamically set and adaptively adjusted based on the temperature distribution characteristics of the image itself to eliminate noise points with excessively low temperatures and focus on the apple temperature and similar temperature areas.

[0025] Step 102: Using apple temperature range local linear normalization, the temperature values ​​of the thermal temperature matrix are mapped to the range of 0-255 to obtain a grayscale image. The formula is as follows: (1); In the formula, Thermal temperature matrix gray intensity at that location; Thermal temperature matrix Temperature value at; This represents the highest temperature in the thermal temperature matrix. The adaptive minimum temperature calculated above; Step 103: The grayscale image is processed by histogram equalization to obtain an enhanced infrared thermal image, such as... Figure 2 As shown in (b).

[0026] Step 2: Perform image registration on the enhanced infrared thermal image and the visible light image.

[0027] Multi-scale second-order gradient smoothing denoising is performed sequentially on the enhanced infrared thermal image and the visible light image. Edge contours are extracted by calculating pixel gradient magnitude and direction. Feature points and principal directions of feature points are calculated based on the curvature features of the contour edges. Feature vectors of each feature point are then generated. Matching is performed based on the similarity of feature vectors between feature points of the two images. Mismatched points are then eliminated using the consistency of the matching lines. Finally, the transformation matrix is ​​calculated to obtain the registered infrared thermal image and visible light image of the apple.

[0028] The specific steps for image registration between the visible light image and the enhanced infrared thermal image of the apple are as follows: Reference Figure 4 , Figure 4 (a) and Figure 4 (b) shows the feature point extraction maps of the visible light image and the infrared thermal image, respectively.

[0029] Step 201: Perform multi-scale second-order gradient smoothing denoising on the enhanced infrared thermal image and the visible light image sequentially, applying the convolution template to each pixel and calculating the pixel gradient magnitude. S With direction The gradient direction is simplified to four directions: horizontal (0°), 45° diagonal, vertical (90°), and 135° diagonal. The gradient magnitude of the current pixel is compared with its preceding and following adjacent pixels along these simplified gradient directions. If the current pixel has the largest gradient magnitude, it is retained as a candidate edge point; otherwise, it is discarded. Two thresholds are set: a high threshold and a low threshold. Candidate edge points with gradient magnitudes greater than the high threshold are categorized as strong edge points; those less than the low threshold are discarded; and those between the two thresholds and connected to strong edge points (through adjacent pixels) are retained as weak edge points and included in the edge classification. Finally, strong and weak edge points are connected to form the image edge contour. The calculation formula is as follows:

[0030] (2); (3); (4); In the formula, d x for x Convolution kernel in the direction, d y for y Convolution kernel in the direction.

[0031] Feature points are calculated based on the curvature of the edge contour, where curvature The calculation formula is as follows: (5); In the formula, u These are the positional parameters of the contour edge curve. For smoothing scale parameters, ; ; ; ; For convolution calculation; and The coordinates of the contour edge curve; Width The convolution kernel is 5 in this case; and These are the first and second derivatives of the second-order gradient convolution kernel.

[0032] If a pixel on an edge has the maximum curvature within a 9-pixel-wide square neighborhood centered on that pixel, then that pixel is designated as a candidate corner point. An adaptive local curvature threshold is used to eliminate rounded corner points. If the curvature value of a candidate corner point is less than the set curvature threshold, that point is considered a rounded corner point and removed from the candidate corner point list. The remaining candidate corner points are used as actual feature points in subsequent principal direction calculations. The formula for calculating the curvature threshold is as follows:

[0033] (6); In the formula, G This refers to the corner threshold control coefficient. C This represents the number of pixels at that edge within the adjacent region. K ( i ) is the first i The curvature of each pixel.

[0034] Figure 5 (a) and Figure 5 (b) is a schematic diagram of feature point principal orientation determination in visible light images and enhanced infrared thermal images. Step 202: Determine the principal orientation of the feature point by finding the relative positions of its nearest left and right feature points. Here, a principal orientation determination method based on the pose relationship of nearby minimum curvature points is used:

[0035] set up The set of feature points along a certain contour line. For the first on the outline i One feature point, n Let be the number of feature points on the contour line, and let a feature point on the contour line be... .when When not located at the beginning of the contour, set Left side The minimum curvature feature point is , right side The minimum curvature feature point is . , The two points are separated from the current feature point. , Each pixel is used. To ensure the robustness of the main direction selection, a second-order gradient convolution kernel is used to perform second-order gradient smoothing sampling on all feature points between the left and right minimum curvature feature points and the current feature point, thereby obtaining the left and right auxiliary points of the current feature point. , The calculation formula is as follows:

[0036] (7); (8); (9); In the formula, It is a discrete convolution kernel; Here, represents the size parameter of the convolution kernel, which is the number of pixels between the current feature point and the feature point with the minimum curvature. x It is a discrete index variable.

[0037] When a feature point is located at the start or end of the contour, the auxiliary points on the effective side (right / left side) are copied symmetrically with the current feature point as the center, and a mirror virtual point is generated as the auxiliary point on the missing side.

[0038] Finally, the direction of the angle bisector vector of the angle formed by the lines connecting the current feature point to the auxiliary points on the left and right sides is taken as the main direction.

[0039] Figure 6 (a) is a schematic diagram of the initial matching of a visible light image and an enhanced infrared thermal image. In step 203, the position and main direction of the feature points are parameterized and encoded to generate a feature vector for each feature point. Then, matching is performed based on the similarity of the feature vectors between the feature points in the two images. Here, the Euclidean distance method is used to determine their similarity. When the Euclidean distance between two points is less than the distance set threshold, the two points are determined to be matching points, and the matching points are connected by a straight line.

[0040] Figure 6 (b) is a schematic diagram of feature point (partial) matching after matching line consistency screening. In step 204, since the thermal imager and the visible light camera are fixed together and both are directly facing the fruit tree during acquisition, the overall image rotation characteristics make the direction of the matching point lines approximately consistent and close to horizontal. The lengths of each matching line are similar. The tilt angle of the matching line is calculated. and length d The formula is as follows:

[0041] (10); (11); In the formula, ( x 1, y 1) and (x 2, y 2) These are two corresponding matching points on the enhanced infrared thermal image and the visible light image, respectively. The width of the thermal image.

[0042] Then, mismatched points are removed using the consistency of the matching lines. First, the histogram of the principal direction difference of each pair of matching points is calculated, and the maximum value is taken. Set the tilt threshold as the reference tilt angle. The screening tilt angle is Matching lines for each interval, ensuring they are in the same direction; then calculate the average length of the matching lines after filtering. Set a length error threshold The filter length is within The matching line of the interval is the correct matching line.

[0043] Step 205: Based on the final selected valid matching pairs, determine the pixel position transformation relationship between the enhanced infrared thermal image and the visible light image. This relationship can be represented in matrix form as follows: (12); In the formula, P To enhance the coordinates of matching points in infrared thermal images, P 'These are the coordinates of the matching point in the visible light image.' H This is the transformation matrix that describes the positional correspondence between the two.

[0044] Calculate the transformation matrix H Then, pixel by pixel, the pixels of the enhanced infrared thermal image are mapped to the corresponding positions in the visible light image according to this relationship, completing the pixel-level registration of the two images and obtaining the registered enhanced infrared thermal image.

[0045] Step 3: Perform image fusion on the registered enhanced infrared thermal image and visible light image. Use an image fusion network model based on convolutional neural network to extract high-level semantic information from the two registered images, and use gradient residual dense structure to enhance detail capture, capture texture, edge information, etc. Finally, combine semantics and information to output the fused image.

[0046] Figure 7 This is a framework diagram of an image fusion network. It employs... Figure 7The image fusion network shown performs image fusion by cascading an image fusion module and a semantic segmentation module. The image fusion module, relying on a gradient residual dense structure, captures fine-grained spatial details (such as texture and edges) from multiple sources, including thermal imaging and visible light, to achieve pixel-level fusion. It can also receive semantic feedback for dynamic optimization. The semantic segmentation module classifies the fused image into semantic categories (pedestrians, buildings, etc.) pixel by pixel, using semantic loss to supervise itself and backfeed semantics, allowing the fusion module to understand key semantic regions. By introducing a semantic loss mechanism, high-level semantic information is fed back to the image fusion module, enabling effective utilization and optimization of semantic information. Simultaneously, a specially designed gradient residual dense structure significantly enhances the fusion network's ability to capture and describe fine-grained spatial details, thereby improving the quality and effect of image fusion.

[0047] The registered infrared thermal image and visible light image of the apple are input into the image fusion module. The module captures fine-grained spatial details such as texture and edge by relying on the gradient residual dense structure to complete pixel-level fusion. At the same time, it receives semantic feedback from the semantic segmentation module for dynamic optimization. The fused image enters the semantic segmentation module to classify semantic categories pixel by pixel. The fusion result is optimized in reverse through the semantic loss mechanism, and finally an enhanced fused image containing key semantic information is obtained.

[0048] Figure 3 (a) is a visible light image actually taken in a high-light environment in an orchard according to the present invention. Figure 3 (b) is the enhanced infrared thermal image based on the enhanced thermal properties of outdoor apples; Figure 8 for Figure 3 (a) and Figure 3 (b) The image fused using an image fusion network model after image registration. In step 3 above, the registered infrared thermal image and visible light image of the apple are fused. A high-level semantic information of the two registered images is extracted using an image fusion network model based on a convolutional neural network. Gradient residual dense structure is used to enhance detail capture, capturing texture, edge information, etc. Finally, the semantics and information are combined to output the fused image.

[0049] Step 4: Perform apple target recognition on the fused image and improve the YOLOv8 algorithm: improve the bounding box loss function to enhance the robustness of small target and occluded object detection; improve the network structure to enhance the model's feature extraction ability and optimize small target detection; introduce the attention mechanism SA module to suppress background interference, thereby improving the model's detection performance under different lighting conditions; finally, recognize the fused image and output the apple's location, category, and confidence score.

[0050] Figure 9This is the original YOLOv8 network structure diagram. The YOLOv8 network structure mainly consists of three parts: Backbone, Neck, and Head. The YOLOv8 Backbone primarily uses standard convolutions, C2f modules, and residual connections for feature extraction. C2f, as the main Backbone module, employs a more efficient feature channel allocation method compared to the traditional Bottleneck structure, improving the model's feature extraction capability. The YOLOv8 Neck uses a PAN-FPN (Path Aggregation Network-Feature Pyramid Network) structure. Through the top-down semantic propagation of FPN and the bottom-up local feature enhancement of PAN, it progressively fuses multi-scale features from the Backbone, allowing information from different scales to fully complement each other, enhancing the detection capability of multi-scale objects, optimizing the feature input of the detection head, and improving the model's generalization ability. The Head is responsible for the final object detection task, including object category prediction, bounding box regression, and confidence estimation, using an anchor-free mechanism for prediction.

[0051] Figure 10 This is a diagram of the improved YOLOv8 network structure used in this embodiment of the invention. In step 4 above, the YOLOv8 model was improved by: improving the bounding box loss function to enhance the robustness of small target and occluded object detection; improving the network structure to enhance the model's feature extraction capability and optimize small target detection; and introducing an attention mechanism (SA) module to suppress background interference, thereby improving the model's detection performance under different lighting conditions.

[0052] The first improvement in step 4 above modifies the original YOLOv8 loss function CIoU by introducing distance attention weights to optimize bounding box regression for small targets and occluded objects: The improved loss function WIoU is formulated as follows: (13); (14); (15); (16); (17); (18); (19); In the formula, r This indicates the added gradient gain; This indicates the outlier degree; a lower outlier degree indicates higher anchor frame quality. and It is an adjustable hyperparameter; Indicates the monotonic focus coefficient; is the normalization factor, representing the moving average of momentum; The penalty term for CIoU is used to amplify the loss of ordinary mass anchor frames; Indicates IoU loss; IoU This represents the intersection-union ratio (IUU) of the predicted and ground truth bounding box areas. and These represent the ground truth bounding box and the predicted bounding box, respectively; the coordinates of the center point of the bounding box are determined by... x , y express; and The height and width of the smallest bounding box that covers both the ground truth box and the predicted box; and The superscript * in the diagram indicates separation from the computation graph.

[0053] The WIoU function introduces distance attention weights, taking into account the relative position of the predicted box and the ground truth box during calculation. Even if the IoU of small objects is low, effective gradient optimization can still be obtained. At the same time, by dynamically adjusting the loss weights, small objects are not ignored during the optimization process, which can also improve the detection accuracy of small objects. For the detection of occluded objects, the distance weight mechanism reduces the dependence on IoU calculation, and even if the object is partially occluded, a relatively complete detection box can still be optimized.

[0054] Figure 11 (a) is the C2f module structure diagram of the YOLOv8 network structure. Figure 11 (b) is the structure diagram of the C2f_ODConv module for the improved YOLOv8 network structure. Figure 12 (a) is a diagram of the SPPF module structure of the YOLOv8 network. Figure 12 (b) Structural diagram of the SPPELAN module for improving the YOLOv8 network structure. The second improvement in step 4 above improves the network structure of the YOLOv8 model: on the one hand, a full-dimensional dynamic convolution (structure) is introduced in the C2f module, which enables the model to automatically adjust the convolution kernel according to the feature changes of different regions, and the full-dimensional dynamic convolution dynamically adjusts the convolution kernel in multiple dimensions; on the other hand, the structure of the SPPF module in the Backbone part is improved. Combining the attention mechanism of Spatial Pyramid Pooling (SPP) and Efficient Layer Aggregation Network (ELAN), a brand-new structure SPPELAN (structure) is introduced to replace SPPF. Its multi-scale pooling combines the advantages of the attention mechanism and can adaptively emphasize important local regions in the image, especially small target regions.

[0055] Figure 13This is a diagram of the SA module structure added to the YOLOv8 network. The third improvement in step 4 above involves adding the efficient channel attention mechanism SA module to the backbone. As an efficient attention mechanism, the SA module uses permutation units and channel shuffling operations to work together on the spatial and channel dimensions of the input feature map, allowing the model to focus more on local information. Even if the target is partially occluded, it can still effectively detect it. Simultaneously, it allows information from different channels to adaptively enhance local features, thereby improving the model's generalization ability. In different scenarios, such as occlusion and varying lighting conditions, it performs more stably and reliably.

[0056] See Figure 14 This is the result of apple target recognition directly from a visible light image. Figure 15 The results of apple target recognition using the apple recognition method based on the fusion of infrared thermal images and visible light images of this invention were compared with those of the model trained on visible light images, which sometimes failed to recognize some apples. It was found that the target detection method based on fused images proposed in this invention has higher accuracy and adaptability under adverse lighting conditions.

[0057] Based on the same concept, the present invention also provides an apple recognition system, including a data acquisition module, a generation module, a registration module and a recognition module.

[0058] The acquisition module is used to acquire raw infrared thermal images and visible light images containing apples.

[0059] The generation module is used to extract multiple edge contours from the original infrared thermal image and visible light image respectively, and calculate feature points on each edge contour based on the curvature of the edge contour; determine the main direction of each feature point based on the position information of the edge contour on which each feature point is located; and parametrically encode the position and main direction of each feature point to generate the feature vector of each feature point.

[0060] The registration module is used to match multiple matching pairs based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image; the transformation matrix is ​​determined by the coordinates of the multiple matching pairs; and the original infrared thermal image and the visible light image are registered based on the transformation matrix to obtain the registered image.

[0061] The recognition module is used to fuse the registered image and the visible light image to obtain a fused image. The fused image is then used to identify the apple target to obtain the apple's location, category, and confidence level.

[0062] The present invention provides an apple recognition method based on the fusion of infrared thermal images and visible light images, which can enhance the infrared thermal image of an apple based on the thermal characteristics of an outdoor apple. The enhanced thermal image is more conducive to apple target recognition.

[0063] The present invention provides an apple recognition method based on the fusion of infrared thermal image and visible light image. It utilizes the imaging characteristics of apples in infrared thermal image and visible light image, uses edge curvature to calculate feature points and main direction, and uses matching line consistency to remove mismatches, thereby achieving accurate matching of two images.

[0064] The present invention provides an apple recognition method based on the fusion of infrared thermal images and visible light images. The method uses an image fusion network model to perform image fusion, which can highlight the apple area and has a fast fusion speed.

[0065] This invention provides an apple recognition method based on the fusion of infrared thermal images and visible light images. It improves the original YOLOv8 model by improving the bounding box loss function to enhance the robustness of small target and occluded object detection; improving the network structure to enhance the model's feature extraction capability and optimize small target detection; and introducing an attention mechanism (SA) module to suppress background interference, thereby improving the model's detection performance under different lighting conditions. These improvements greatly enhance the detection performance of the apple target recognition algorithm.

[0066] This invention improves traditional image enhancement methods based on the thermal characteristics of outdoor apples, increasing the accuracy of apple target detection in infrared thermal images. Based on the imaging characteristics of apples in both infrared and visible light images, this invention utilizes edge curvature to calculate feature points and principal directions, and employs matching line consistency to remove mismatches, achieving accurate matching between two images. This invention uses an image fusion network for image fusion, highlighting the apple region while maintaining a fast fusion speed. This invention also improves upon the existing YOLOv8 model, enhancing the stability and reliability of apple target detection.

[0067] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described Apple recognition method.

[0068] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described apple identification method.

[0069] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0070] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An apple identification method, characterized in that, Includes the following steps: Acquire raw infrared thermal and visible light images containing apples; Multiple edge contours are extracted from the original infrared thermal image and visible light image respectively, and feature points on each edge contour are calculated based on the curvature of the edge contours; the principal direction of each feature point is determined based on the position information of each feature point on the edge contour; the position and principal direction of each feature point are parameterized and encoded to generate the feature vector of each feature point. Matching is performed based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image to obtain multiple matching pairs; the transformation matrix is ​​determined by the coordinates of the multiple matching pairs; image registration is performed on the original infrared thermal image and the visible light image based on the transformation matrix to obtain the registered image; The registered image and the visible light image are fused to obtain a fused image. The fused image is then used to identify the apple target, and the location, category, and confidence level of the apple are obtained.

2. The apple identification method as described in claim 1, characterized in that, The process of extracting multiple edge contours from the original infrared thermal image and visible light image, and calculating feature points on each edge contour based on the curvature of the edge contours, includes the following steps: Image enhancement is performed on the original infrared thermal image to obtain an enhanced infrared thermal image; Multi-scale second-order gradient smoothing denoising is performed sequentially on the enhanced infrared thermal image and the visible light image. The convolutional template is applied to each pixel, and its gradient magnitude is calculated. S With direction Based on gradient magnitude S With direction The edge points in the image are selected and connected to obtain the edge contour. Calculate the curvature of each pixel on each edge contour and iterate through them. If the curvature of the current pixel is the largest in a square neighborhood with a width of 9 pixels centered on it, then set it as a candidate corner point. Iterate through multiple candidate corner points. If the curvature value of the current candidate corner point is greater than the curvature threshold, then set it as a feature point.

3. The apple identification method as described in claim 2, characterized in that, Determining the main direction of each feature point based on its position information on the edge contour includes the following steps: For each edge contour If the current feature point If the feature point is not located at the starting or ending position of the edge contour, then the feature point corresponding to the minimum curvature on its left is obtained. The feature point corresponding to the minimum curvature on its right is ; Using second-order gradient convolution kernels respectively and All feature points between and and Second-order gradient smoothing sampling is performed on all feature points in between to obtain the first auxiliary point. Second auxiliary point The first auxiliary point and the second auxiliary point are specifically shown below: ； ； ； In the formula, For discrete convolution kernels, The size parameter of the convolution kernel. x For discrete index variables, and They are respectively and The number of pixels between two points and the current feature point; The direction of the angle bisector vector of the angle formed by the lines connecting the current feature point to the first auxiliary point and the second auxiliary point is taken as the main direction of the current feature point.

4. The apple identification method as described in claim 2, characterized in that, The matching based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image includes the following steps: Obtain the Euclidean distance between the feature vectors of each feature point in the enhanced infrared thermal image and the visible light image; When the Euclidean distance between a feature point in an enhanced infrared thermal image and a feature point in a visible light image is less than a set distance threshold, the two feature points are determined to be a matching pair. Determining the transformation matrix using the coordinates of multiple matching pairs includes the following steps: Connect the two feature points of each matching pair with a straight line to obtain the matching line; calculate the tilt angle and length of each matching line; Calculate the histogram of the principal direction differences for each pair of matching points and take the maximum value. Use the reference tilt angle and set the tilt threshold. ; Filtering tilt angle at Find multiple first matching lines within the interval, and then calculate the average length of the multiple first matching lines. Set a length error threshold The filter length is within The second matching line of the interval is the final matching line; The matching pairs corresponding to the final matching line are valid matching pairs; The transformation matrix is ​​determined by the coordinates in the effective matching pairs, and the specific transformation matrix is ​​as follows: ； In the formula, P To enhance the coordinates of matching points in infrared thermal images, P 'These are the coordinates of the matching point in the visible light image.' H Let be the transformation matrix.

5. The apple identification method as described in claim 2, characterized in that, The process of enhancing the original infrared thermal image to obtain an enhanced infrared thermal image includes the following steps: The mean and standard deviation of the thermal temperature matrix are obtained, and the difference between the mean and standard deviation is used to obtain the lowest temperature; wherein, the temperature value corresponding to each pixel in the infrared thermal image of the apple is represented in matrix form to obtain the thermal temperature matrix; Based on the lowest temperature, each temperature value in the thermal temperature matrix is ​​normalized and mapped to obtain the grayscale intensity of each pixel. The grayscale intensities of multiple pixels are combined to obtain a thermal imaging grayscale image. Histogram equalization is performed on the thermal imaging grayscale image to obtain an enhanced infrared thermal image.

6. The apple identification method as described in claim 1, characterized in that, The image fusion of the registered image and the visible light image specifically includes the following steps: The registered image and the visible light image are input into the image fusion module for pixel-level fusion to obtain the fused image; The fused image is input into the semantic segmentation module for pixel-by-pixel semantic recognition, resulting in a fused image containing semantic information.

7. The apple identification method as described in claim 1, characterized in that, Apple target recognition is performed on the fused image based on the improved YOLOv8. Specifically, distance attention weights are introduced to improve the original loss function CIoU of the original YOLOv8 model. Full-dimensional dynamic convolution is introduced into the C2f module of the Neck module in the original YOLOv8 model, and an efficient channel attention mechanism SA module is added to the Backbone part.

8. An Apple recognition system, characterized in that, include: The acquisition module is used to acquire raw infrared thermal images and visible light images containing the apple; The generation module is used to extract multiple edge contours from the original infrared thermal image and visible light image respectively, and calculate feature points on each edge contour based on the curvature of the edge contour; determine the main direction of each feature point based on the position information of each feature point on the edge contour; and parameterize the position and main direction of each feature point to generate the feature vector of each feature point. The registration module is used to match multiple matching pairs based on the Euclidean distance of the feature vectors of each feature point on the original infrared thermal image and the visible light image; determine the transformation matrix through the coordinates of the multiple matching pairs; and perform image registration on the original infrared thermal image and the visible light image based on the transformation matrix to obtain the registered image. The recognition module is used to fuse the registered image and the visible light image to obtain a fused image, and to perform apple target recognition on the fused image to obtain the apple's location, category, and confidence level.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the Apple recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the apple recognition method according to any one of claims 1-7.

Images