Irregular green pepper fruit identification method based on multi-scale feature fusion

Abstract

The invention discloses an irregular green pepper fruit identification method based on multi-scale feature fusion, and the method comprises the following steps: S1, collecting an original image data set of irregular green pepper fruits, marking an anchor frame, carrying out the data enhancement of the marked data set through a multi-scale Retinex algorithm, and obtaining an original image data set of the irregular green pepper fruits; randomly dividing the enhanced data set into a training set, a verification set and a test set according to a proportion of 8: 1: 1; s2, improving a YOLOv5 algorithm, and constructing an irregular green pepper fruit detection model FASK; s3, inputting to-be-detected image data into the FASK to obtain a feature layer with multi-scale change features; s4, inputting the feature layer with the multi-scale change feature into SK-Neck, and extracting an image feature map; and S5, obtaining a final detection image according to the image feature map. According to the method, the network can adaptively receive a field and extract irregular features, multi-scale feature fusion is carried out through a feature map alignment method and pixel-level weighted fusion operation, and the robustness of the model on irregular green pepper fruit detection is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to an irregular green pepper fruit recognition method based on multi-scale feature fusion, which is suitable for the green pepper fruit recognition scene with irregular shapes and branches and leaves blocking the target. Technical Background

[0002] Green pepper is a common cash crop and one of the main crops for people in most areas to increase their income and become rich. At present, the picking of green pepper is still mainly done by manpower, and the realization of automatic mechanical picking of green pepper fruits has become an urgent problem to be solved. In the real field environment, the shapes and scales of green pepper fruits vary, and it is common for branches and leaves and fruits to occlude each other. Rapid and accurate positioning and detection of picking objects is the technical basis of automatic mechanical picking. Therefore, studying the detection method of green pepper fruits with variable shapes in densely occluded environments and improving the detection and positioning performance of the model for dense targets can reduce the cost of manual picking, which has great application value and practical significance for the development of green pepper picking robots.

[0003] For green pepper fruit target detection, people have done various research, such as Chinese patent CN115761728A, published on March 7, 2023, which discloses a YOLOv51 green pepper intelligent detection method based on NSGA-II pruning. This patent adopts the YOLOv51 green pepper intelligent detection method based on NSGA-II pruning technology, and its parameter quantity, model size and GFlops are greatly reduced. The detection speed is also improved, and it can be applied to the picking environment of green pepper robots. However, green pepper fruits in real environments often have irregular shapes and grow in lush branches and leaves. Models trained only on common green pepper data do not have high generalization.

[0004] From the above, it can be seen that although the existing technology can effectively detect green pepper fruits under normal conditions, the detection accuracy for irregular green pepper fruits is still low, and there are still problems such as false detection and missed detection for fruits that are blocked by branches and leaves and overlapped by fruits. Summary of the invention

[0005] In view of the limitations of various technical solutions given in the background technology, and the problems of low detection accuracy of irregular-shaped green pepper fruits in the prior art, and poor detection effect in complex situations such as branch and leaf occlusion, fruit overlap, and background similarity, the present invention provides an irregular-shaped green pepper fruit recognition algorithm based on multi-scale feature fusion, which can detect irregular-shaped green pepper fruits in densely occluded situations with high accuracy and low missed detection rate.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for identifying irregular green pepper fruits by multi-scale feature fusion, characterized in that it comprises the following steps:

[0008] S1. Collect the original image dataset of irregular green pepper fruits; annotate the collected original data with anchor frames based on the fruit shape, use the multi-scale Retinex algorithm to enhance the sample data of the annotated dataset, and randomly divide the enhanced dataset into training set, validation set and test set in a ratio of 8:1:1;

[0009] S2. Based on the YOLOv5 network architecture, the adaptive feature extraction RCNS module is constructed based on the feature adaptive convolution FAConv, and the FARS module is built based on the RCNS module; the SK module with selective kernel attention is added to build the SK-Neck with multi-scale feature extraction capability; and then the irregular green pepper fruit detection model FASK is constructed;

[0010] S3, inputting the image data of the irregular-shaped green pepper fruit to be detected into the detection model FASK, adaptively changing the receptive field through the RCNS module and extracting the feature layer of the irregular shape; performing multi-scale feature fusion on the feature layer extracted by the RCNS module through the FARS module to obtain a feature layer with multi-scale change characteristics;

[0011] S4. Input the feature layer with multi-scale variation characteristics into SK-Neck to extract the image feature map.

[0012] S5. Obtain a final detection image of the irregular-shaped green pepper fruit to be detected according to the image feature map.

[0013] Preferably, the irregular green pepper fruit original image data set is collected through three shooting angles: vertical overhead shooting, 45° oblique shooting and horizontal straight shooting, and includes three different weather shooting environments: sunny day, cloudy day and rainy day.

[0014] Preferably, in S1, the specific process of the multi-scale Retinex algorithm includes the following steps:

[0015] (1) First, the original image is decomposed into multi-layer scale images. The decomposition process satisfies the following calculation formula:

[0016] I0=G(I0,σ0) (1)

[0017] I1=G(I0,σ1) (2) ......

[0019] I n =G(I0,σ n ) (3)

[0020] Among them, G is the Gaussian filter, σ1 represents the Gaussian kernel scale of the i-th layer, and I i Represents the image at the i-th level.

[0021] (2) Retinex processing is performed on each layer of scale image, and the processing process satisfies the following calculation formula:

[0022] L i =log(I i )-log G(I i , σ g ) (4)

[0023] S i =log G(I i , σ s )-log G(I i , σ g ) (5)

[0024] R i =exp(L i +S i ) (6)

[0025] Among them, L i represents the reflection component of the i-th scale, S i represents the illumination component of the i-th level, R i Represents the enhanced image at the i-th scale.

[0026] (3) The enhanced images of each layer are weighted and fused to obtain the final enhanced image. The weighted process satisfies the following calculation formula:

[0027]

[0028] Among them, ω i Represents the weight of the i-th level scale.

[0029] (4) Repeat the above steps until the number of data augmented samples is balanced with the number of original samples.

[0030] Preferably, in S3, the calculation process of adaptively adjusting the receptive field change and extracting the feature layer of irregular shape by the RCNS module specifically includes the following steps:

[0031] (1) Generate initial sampling coordinates for convolution kernels of different shapes and assign each initial sampling coordinate to a convolution kernel of different sizes. The sampling grid is as follows:

[0032]

[0033] (2) Generate feature-adaptive offsets through a convolutional layer with dimensions (B, 2N, W, H), where B is the batch size, N is the convolution kernel size, and W and H are the height and width of the feature map, respectively;

[0034] (3) The offset is used to identify the irregular shape of the sample to adaptively adjust the convolution kernel size and generate new sampling coordinates. The calculation formula of the new sampling coordinate ρ is expressed as:

[0035] ρ=P0+P n +θ offsets (9)

[0036] Among them, P0 is the center point of the current convolution window, P n is the initial sampling coordinate, θ offsets The offsets generated for the offset convolutional layer.

[0037] (4) The feature values ​​of non-integer positions are extracted from the input feature map by interpolation, and the sample shape of each position is changed by resampling to make the fused feature map smoother and more consistent. The specific calculation process is as follows:

[0038] x(q)≈∑ i ∑ j bilinear(q,(i,j))·x(i,j) (10)

[0039] Wherein, bilinear(q, (i, j)) represents the bilinear interpolation coefficient between the coordinate q and the integer coordinate point (i, j).

[0040] (5) Extract features from the feature map using the adjusted sampling positions.

[0041] Preferably, in S3, the FARS module performs multi-scale feature fusion on the feature layer extracted by the RCNS module by aligning the feature maps in spatial dimensions through a bilinear interpolation method, thereby performing pixel-level weighted fusion. The specific calculation process of the feature map alignment method and the pixel-level weighted fusion operation includes the following steps:

[0042] (1) Map the target pixel coordinates (x, y) of the high-resolution H2×W2 feature map F2 to the relative coordinates (x′, y′) in the low-resolution H1×W1 feature map F1, so that F1 is interpolated to the same resolution as F2. The specific calculation process is as follows:

[0043]

[0044]

[0045] (2) Determine the coordinates of the four pixel points x1, x2, y1, and y2 around the relative coordinate (x′, y′). The specific calculation process is as follows:

[0046]

[0047]

[0048]

[0049]

[0050] in, Indicates rounding down. Indicates rounding up.

[0051] (3) Calculate the interpolation weight according to the relative position (x′, y′). The specific formula is as follows:

[0052]

[0053]

[0054]

[0055]

[0056] (4) The values ​​of the four surrounding pixels are weighted averaged according to the interpolation weight to obtain the value of the target coordinates. The specific formula is as follows:

[0057]

[0058] Among them, F1(x1, y1), F1(x1, y2), F1(x2, y1), and F1(x2, y2) are the coordinates of the four corresponding pixels in the low-resolution feature map respectively.

[0059] (5) The aligned feature map F′1 is fused with the high-resolution feature map F2 pixel by pixel. The fused feature map is expressed as follows:

[0060] F fusion (x, y)=αF′1(x, y)+βF2(x, y) (22)

[0061] Among them, α and β are the learned weight matrices.

[0062] Preferably, in S4, the feature layer with multi-scale variation characteristics is input into SK-Neck to extract the image feature map, and the specific operation is: adding a selective kernel attention SK module at the end of the neck network of YOLOv5s to build SK-Neck, and finally obtaining a multi-scale spatial channel feature map. The SK-Neck network includes:

[0063] The eleventh layer is the RCNS module, the twelfth layer is the upsample module, the thirteenth layer is the residual connection Concat module, the fourteenth layer is the FARS module, the fifteenth layer is the RCNS module, the sixteenth layer is the Upsample module, the seventeenth layer is the residual connection Concat module, the eighteenth layer is the FARS module, the nineteenth layer is the RCNS module, the twentieth layer is the residual connection Concat module, the twenty-first layer is the FARS module, the twenty-second layer is the RCNS module, the twenty-third layer is the residual connection Concat module, the twenty-fourth layer is the FARS module, and the twenty-fifth layer is the selective kernel attention SK module.

[0064] Preferably, the specific calculation process of the selective kernel attention SK module and the Split Attention operation includes the following steps:

[0065] (1) The selective kernel attention SK module includes: a separation module, a fusion module and a selection module, wherein the separation module takes the input feature Figure X ∈R W×H×C It is divided into two branches, and the specific formula of the separation process is as follows:

[0066] U1=Slipt 3×3 (X) (23)

[0067] U2=Slipt 5×5 (X) (24)

[0068] Among them, U1 and U2 represent the output feature maps of branch 1 and branch 2 respectively; Slipt 3×3 and Slipt 5×5 They represent convolution modules with convolution kernels of 3×3 and 5×5 respectively;

[0069] (2) The fusion module adds the two sets of feature maps U1 and U2 according to the elements to obtain the feature map U, and embeds the global information through the global average pool to generate a feature vector containing different receptive field information. The specific formula is as follows:

[0070] U=Fuse(U1,U2) (25)

[0071] Z=f fc (f gp (U)) (26)

[0072] Among them, Fuse represents the fusion operation; U represents the feature map obtained after fusion; Z represents the output feature vector; f fc represents global average pooling; f gp represents a fully connected layer;

[0073] (3) The selection module uses the SoftMax function to obtain the final feature map F c , the specific formula is as follows:

[0074] F c =a c U1+b c U2 (27)

[0075] a c +b c =1 (28)

[0076] Among them, F c represents the final feature map, a c and b c Represents the attention weights on different convolution kernels.

[0077] Preferably, a system for identifying irregular-shaped green pepper fruits comprises:

[0078] An acquisition module, used for acquiring image data of irregular-shaped green pepper fruits to be detected;

[0079] The model building module is used to build the adaptive feature extraction RCNS module based on the feature adaptive convolution FAConv based on the YOLOv5 network architecture, and build the FARS module based on the RCNS module; add the selective kernel attention SK module to build the SK-Neck with multi-scale feature extraction capabilities; and then build the irregular green pepper fruit detection model FASK;

[0080] The first feature extraction module is used to input the image data of the irregular-shaped green pepper fruit to be detected into the detection model FASK, and adaptively change the receptive field through the RCNS module and extract the feature layer of the irregular shape;

[0081] The feature fusion module is used to perform multi-scale feature fusion on the feature layer extracted by the RCNS module through the FARS module to obtain a feature layer with multi-scale variation characteristics;

[0082] The second feature extraction module is used to input the feature layer with multi-scale variation characteristics into SK-Neck to extract the multi-scale spatial channel feature map;

[0083] The detection module is used to obtain a detection image of the irregular-shaped green pepper fruit to be detected according to the image feature map.

[0084] By means of the above technical solution, the present invention provides a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion, which has the following beneficial effects:

[0085] The invention is based on the YOLOv5s network architecture, adopts a multi-scale Retinex algorithm to generate enhanced data samples through weighted fusion of Gaussian processing and grayscale processing, and enhances the irregular green pepper fruit data set; FAConv is designed with the idea of ​​adjusting the convolution kernel by offset, and adaptive target features are realized through free deformation of the convolution kernel to obtain the receptive field of irregular features; the RCNS module is constructed based on FAConv, and the receptive field is adaptively adjusted and the feature layer of the irregular green pepper target is extracted through bilinear interpolation and resampling methods; the FARS module is constructed based on the RCNS module, and multi-scale feature fusion is performed through feature map alignment method and pixel-level weighted fusion operation to adapt to inputs of different resolutions and enhance the robustness of the model in complex environments; at the same time, a selective kernel attention SK module is added to construct SK-Neck with multi-scale feature extraction capability, and more extensive context information is captured by separation, fusion and selection operations on feature maps, thereby improving the model feature expression capability; and then an irregular green pepper fruit detection model FASK is constructed. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0087] Figure 1 It is a flow chart of an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to the present invention;

[0088] Figure 2 A schematic diagram of a multi-scale Retinex image enhancement process of an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to the present invention;

[0089] Figure 3 A calculation flow chart of the RCNS module for an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to the present invention;

[0090] Figure 4 It is a structural schematic diagram of the RCNS module of an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion of the present invention;

[0091] Figure 5A calculation flow chart of a feature map alignment method and a pixel-level weighted fusion operation in an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion of the present invention;

[0092] Figure 6 It is a calculation flow chart of the selective kernel attention SK module of an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion of the present invention;

[0093] Figure 7 A selective kernel attention SK module diagram for an embodiment of an irregular green pepper fruit recognition method based on multi-scale feature fusion;

[0094] Figure 8 This is a structural diagram of a FASK model for an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to the present invention;

[0095] Fig. 9 It is a curve diagram of the changes of various loss values ​​of FASK model training in an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion of the present invention;

[0096] Fig.10 It is a PR curve diagram of FASK model training in an embodiment of a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion of the present invention;

[0097] Fig.11 This is a schematic diagram of the detection results of the FASK model and the benchmark detection model YOLOv5s in the irregular green pepper fruit dataset in an embodiment of the present invention's irregular green pepper fruit recognition method based on multi-scale feature fusion. DETAILED DESCRIPTION

[0098] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0099] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0100] See also Figure 1The present invention proposes a method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion, comprising the following steps:

[0101] S1. Collect the original image dataset of irregular green pepper fruits; annotate the collected original data with anchor frames based on the fruit shape, use the multi-scale Retinex algorithm to enhance the sample data of the annotated dataset, and randomly divide the enhanced dataset into training set, validation set and test set in a ratio of 8:1:1;

[0102] As a preferred implementation of step S1, S1 specifically includes: collecting images through three shooting angles: vertical overhead shooting, 45° oblique shooting and horizontal straight shooting, including three different weather shooting environments: sunny day, cloudy day and rainy day.

[0103] S2. Based on the YOLOv5 network architecture, the adaptive feature extraction RCNS module is constructed based on the feature adaptive convolution FAConv, and the FARS module is built based on the RCNS module; the SK module with selective kernel attention is added to build the SK-Neck with multi-scale feature extraction capability; and then the irregular green pepper fruit detection model FASK is constructed;

[0104] As a preferred implementation of step S2, Figure 2 As shown, the specific process of the multi-scale Retinex algorithm includes the following steps:

[0105] S31. First, the original image is decomposed into multi-layer scale images. The decomposition process satisfies the following calculation formula:

[0106] I0=G(I0,σ0) (1)

[0107] I1=G(I0,σ1) (2) ......

[0109] I n =G(I0,σ n ) (3)

[0110] Among them, G is the Gaussian filter, σ1 represents the Gaussian kernel scale of the i-th layer, and I i Represents the image at the i-th level.

[0111] S32, perform Retinex processing on each layer of scale image, and the processing process satisfies the following calculation formula:

[0112] L i =log(I i )-log G(I i , σ g ) (4)

[0113] Si =log G(I i , σ s )-log G(I i , σ g ) (5)

[0114] R i =exp(L i +S i ) (6)

[0115] Among them, L i represents the reflection component of the i-th scale, S i represents the illumination component of the i-th level, R i Represents the enhanced image at the i-th scale.

[0116] S33, weighted fusion is performed on the enhanced images of each layer to obtain a final enhanced image, and the weighted process satisfies the following calculation formula:

[0117]

[0118] Among them, ω i Represents the weight of the i-th level scale.

[0119] S34. Repeat the above steps until the number of data-enhanced samples is balanced with the number of original samples.

[0120] S3, the image data of the irregular-shaped green pepper fruit to be detected is input into the detection model FASK, the RCNS module adaptively changes the receptive field and extracts the feature layer of the irregular shape; the FARS module performs multi-scale feature fusion on the feature layer extracted by the RCNS module to obtain a feature layer with multi-scale change characteristics;

[0121] As a preferred implementation of step S3, Figure 3 As shown in the figure, FAConv is used to obtain the receptive field of irregular shape features and the feature layer of information interaction. The calculation process specifically includes the following steps:

[0122] (1) Generate initial sampling coordinates for convolution kernels of different shapes and assign each initial sampling coordinate to a convolution kernel of different sizes. The sampling grid is as follows:

[0123]

[0124] (1) Generate feature-adaptive offsets through a convolutional layer with dimensions (B, 2N, W, H), where B is the batch size, N is the convolution kernel size, and W and H are the height and width of the feature map, respectively;

[0125] (2) The offset is used to identify the irregular shape of the sample to adaptively adjust the convolution kernel size and generate new sampling coordinates. The calculation formula of the new sampling coordinate ρ is expressed as:

[0126] ρ=P0+P n +θ offsets (9)

[0127] Among them, P0 is the center point of the current convolution window, P n is the initial sampling coordinate, θ offsets The offsets generated for the offset convolutional layer.

[0128] (3) Extract the feature values ​​of non-integer positions from the input feature map through interpolation, and change the sample shape of each position through resampling to make the fused feature map smoother and more consistent.

[0129] x(q)≈∑ i ∑ j bilinear(q,(i,j))·x(i,j) (10)

[0130] (4) Extract features from the feature map using the adjusted sampling positions.

[0131] As a preferred implementation of step S3, Figure 4 As shown, the FARS module performs multi-scale feature fusion on the feature layer extracted by the RCNS module. The specific operation is to align the feature map in the spatial dimension through the bilinear interpolation method, thereby performing pixel-level weighted fusion. The specific calculation process of the feature map alignment method and the pixel-level weighted fusion operation includes the following steps:

[0132] (1) Map the target pixel coordinates (x, y) of the high-resolution H2×W2 feature map F2 to the relative coordinates (x′, y′) in the low-resolution H1×W1 feature map F1, so that F1 is interpolated to the same resolution as F2. The specific calculation process is as follows:

[0133]

[0134]

[0135] (2) Determine the coordinates of the four pixel points x1, x2, y1, y2 around the relative coordinate (x′, y′). The specific calculation process is as follows:

[0136]

[0137]

[0138]

[0139]

[0140] in, Indicates rounding down. Indicates rounding up.

[0141] (3) Calculate the interpolation weight according to the relative position (x′, y′). The specific formula is as follows:

[0142]

[0143]

[0144]

[0145]

[0146] (4) The values ​​of the four surrounding pixels are weighted averaged according to the interpolation weight to obtain the value of the target coordinates. The specific formula is as follows:

[0147]

[0148] Among them, F1(x1, y1), F1(x1, y2), F1(x2, y1), and F1(x2, y2) are the coordinates of the corresponding four pixels in the low-resolution feature map.

[0149] (5) The aligned feature map F′1 is fused with the high-resolution feature map F2 pixel by pixel. The fused feature map is expressed as follows:

[0150] F fusion (x, y)=αF′1(x, y)+βF2(x, y) (22)

[0151] Among them, α and β are the learned weight matrices.

[0152] S4. Input the feature layer with multi-scale variation characteristics into SK-Neck to extract the image feature map.

[0153] As a preferred implementation of step S4, Figure 5 As shown, the feature layer with multi-scale variation characteristics is input into SK-Neck to extract the image feature map. The specific process is to build SK-Neck by adding a selective kernel attention SK module at the end of the neck network of YOLOv5s, and finally obtain a multi-scale spatial channel feature map. The selective kernel attention SK module-Neck network includes:

[0154] The eleventh layer is the RCNS module, the twelfth layer is the upsample module, the thirteenth layer is the residual connection Concat module, the fourteenth layer is the FARS module, the fifteenth layer is the RCNS module, the sixteenth layer is the Upsample module, the seventeenth layer is the residual connection Concat module, the eighteenth layer is the FARS module, the nineteenth layer is the RCNS module, the twentieth layer is the residual connection Concat module, the twenty-first layer is the FARS module, the twenty-second layer is the RCNS module, the twenty-third layer is the residual connection Concat module, the twenty-fourth layer is the FARS module, and the twenty-fifth layer is the selective kernel attention SK module.

[0155] The specific calculation process of the selective kernel attention SK module includes the following steps:

[0156] (1) The selective kernel attention SK module includes: a separation module, a fusion module and a selection module, wherein the separation module takes the input feature Figure X ∈R W×H×C It is divided into two branches, and the specific formula of the separation process is as follows:

[0157] U1=Slipt 3×3 (X) (23)

[0158] U2=Slipt 5×5 (X) (24)

[0159] Among them, U1 and U2 represent the output feature maps of branch 1 and branch 2 respectively; Slipt 3×3 and Slipt 5×5 They represent convolution modules with convolution kernels of 3×3 and 5×5 respectively;

[0160] (2) The fusion module adds the two sets of feature maps U1 and U2 according to the elements to obtain the feature map U, and embeds the global information through the global average pool to generate a feature vector containing different receptive field information. The specific formula is as follows:

[0161] U=Fuse(U1,U2) (25)

[0162] Z=f fc (f gp (U)) (26)

[0163] Among them, Fuse represents the fusion operation; U represents the feature map obtained after fusion; Z represents the output feature vector; f fc represents global average pooling; f gp represents a fully connected layer;

[0164] (3) The selection module uses the SoftMax function to obtain the final feature map F c, the specific formula is as follows:

[0165] F c =a c U1+b c U2 (27)

[0166] a c +b c =1 (28)

[0167] Among them, F c represents the final feature map, a c and b c Represents the attention weights on different convolution kernels.

[0168] S5. Obtain a final detection image of the irregular-shaped green pepper fruit to be detected according to the image feature map.

[0169] Example

[0170] In order to verify the recognition effect and superiority of the irregular green pepper fruit recognition method based on multi-scale feature fusion proposed in the present invention, the method adopted by the present invention is compared with the prior art as follows:

[0171] This paper evaluates the speed, computational performance and average detection accuracy of the model respectively. The evaluation indicators of model size and speed are parameters and frame rate per second (FPS), and the average detection accuracy (mAP) is 99. @.5 ) is used as an evaluation index that comprehensively considers accuracy and recall, and is defined as follows:

[0172]

[0173] In the formula, P is the accuracy, R is the recall rate, mAP @.5 The average value of the area of ​​the graph formed by calculating the precision and recall curves of each category reflects the comprehensive detection ability of the network for all recognition targets.

[0174] In order to verify the effectiveness and superiority of the FASK detection model in the irregular green pepper fruit dataset, two groups of experiments were conducted to evaluate the effect.

[0175] Experiment 1: Model comparison experiment

[0176] The improved model FASK in this paper is experimentally compared with the current mainstream target recognition models SSD, YOLOv5s, YOLOX, YOLOv7 and YOLOv8. The results are shown in Table 1.

[0177] Table 1 Model comparison experimental results

[0178]

[0179] As can be seen from Table 1, the precision, recall rate, and mAP of the improved model FASK @.5 , and the number of parameters are the best among the comparison models, which are 92.52%, 89.74%, 92.30% and 12.43M respectively. The precision, recall rate and average precision of the improved model FASK are 6.44%, 6.59% and 7.63% higher than those of the baseline model, respectively. The number of model parameters is reduced by 1.29M, and the model inference speed is increased by 2 frames·s -1 This shows that the improved model FASK pays more attention to the characteristics of irregular-shaped green pepper fruits, and has higher accuracy and stronger robustness. At the same time, the model parameter volume and inference speed are also suitable for deployment on embedded devices and adapt to complex situations in natural environments.

[0180] Experiment 2: Ablation experiment

[0181] In order to evaluate the optimization effect of each improved module in this paper, three groups of ablation experiments were carried out with YOLOv5s as the benchmark model. The experimental results are shown in Table 2.

[0182] By comparing experiments a and b, it is found that after the RCNS module is added to the original benchmark network, the accuracy, recall and average accuracy are increased by 2.38%, 0.54% and 0.56% respectively, and the model size is reduced by 1.29M, which proves that the backbone network with the RCNS module has significantly improved the recognition effect of irregular green pepper targets, while effectively reducing the number of parameters; by comparing experiments b and c, it can be seen that after applying the proposed multi-scale feature fusion strategy on the basis of the RCNS module, the accuracy, recall and average accuracy of the model are increased by 0.46%, 4.10% and 3.89% respectively. Except for the calculation speed, all indicators are evenly improved, indicating that the improved model is more sensitive to green pepper fruit targets in occluded and overlapping environments, and improves the robustness and adaptability of recognition in complex environments. By comparing experiments c and d, we found that after the introduction of the SK attention mechanism, the size and inference speed of the model remained unchanged, the accuracy, recall and average accuracy increased by 3.60%, 1.95% and 3.18% respectively, the model size remained at 12.43M, and the inference speed decreased by 6 frames·s-1. After adding the SK attention mechanism, the multi-scale features and global information were captured by adjusting the attention weights, and the connection between the irregular green pepper features in channels and space was improved. The overall model has been significantly improved in terms of computing speed, while the recall and recognition accuracy of the algorithm have been greatly improved, irrelevant factors such as branches and leaves in the cluttered background have been suppressed, and the model's ability to handle complex spatial relationships has been improved. Table 2

[0183] experiment Accuracy Recall mAP@.5 Parameter quantity Detection speed Baseline Model (YOLOv5s) 83.31 83.15 84.67 13.72 52 Benchmark model + RCNS module 85.69 83.69 85.23 12.43 48 Benchmark model + FARS module 86.15 87.79 89.12 12.43 48 FASK 92.52 89.74 92.30 12.43 54

[0184] Comprehensive analysis shows that the improved model has high robustness and generalization for the recognition of irregular-shaped green pepper fruits.

[0185] In summary, the present invention is based on the YOLOv5s network architecture, and adopts a multi-scale Retinex algorithm to generate enhanced data samples through weighted fusion of Gaussian processing and grayscale processing, thereby enhancing the irregular green pepper fruit data set; FAConv is designed with the idea of ​​adjusting the convolution kernel by offset, and adaptive target features are achieved through free deformation of the convolution kernel to obtain the receptive field of irregular features; the RCNS module is constructed based on FAConv, and the receptive field is adaptively adjusted and the feature layer of the irregular green pepper target is extracted through bilinear interpolation and resampling methods; the FARS module is built based on the RCNS module, and multi-scale feature fusion is performed through feature map alignment method and pixel-level weighted fusion operation to adapt to inputs of different resolutions and enhance the robustness of the model in complex environments; at the same time, a selective kernel attention SK module is added to build SK-Neck with multi-scale feature extraction capability, and a wider range of contextual information is captured by separation, fusion and selection operations on feature maps, thereby improving the model feature expression capability; and then an irregular green pepper fruit detection model FASK is constructed.

[0186] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for identifying irregular green pepper fruits based on multi-scale feature fusion, characterized in that: The following steps are involved: S1. Collect the original image dataset of irregular green pepper fruits; annotate the collected original data with anchor frames based on the fruit shape, use the multi-scale Retinex algorithm to enhance the sample data of the annotated dataset, and randomly divide the enhanced dataset into training set, validation set and test set in a ratio of 8:1:1; S2. Based on the YOLOv5 network architecture, the adaptive feature extraction RCNS module is constructed based on the feature adaptive convolution FAConv, and the FARS module is built based on the RCNS module; the SK module with selective kernel attention is added to build the SK-Neck with multi-scale feature extraction capability; and then the irregular green pepper fruit detection model FASK is constructed; S3, inputting the image data of the irregular-shaped green pepper fruit to be detected into the detection model FASK, adaptively changing the receptive field through the RCNS module and extracting the feature layer of the irregular shape; performing multi-scale feature fusion on the feature layer extracted by the RCNS module through the FARS module to obtain a feature layer with multi-scale change characteristics; S4. Input the feature layer with multi-scale variation characteristics into SK-Neck to extract the image feature map. S5. Obtain a final detection image of the irregular-shaped green pepper fruit to be detected according to the image feature map.

2. The method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to claim 1, characterized in that: The irregular green pepper fruit original image dataset is collected through three shooting angles: vertical overhead shooting, 45° oblique shooting, and horizontal straight shooting, and includes three different weather shooting environments: sunny day, cloudy day, and rainy day.

3. The irregular green pepper fruit recognition method based on multi-scale feature fusion according to claim 1 is characterized in that: In S1, the specific process of the multi-scale Retinex algorithm includes the following steps: S31. First, the original image is decomposed into multi-layer scale images. The decomposition process satisfies the following calculation formula: I0=G(I0,σ0) (1) I1=G(I0,σ1) (2) …… I n =G(I0,σ n ) (3) Among them, G is the Gaussian filter, σ1 represents the Gaussian kernel scale of the i-th layer, and I i Represents the image at the i-th level. S32, perform Retinex processing on each layer of scale image, and the processing process satisfies the following calculation formula: L i =log(I i )-logG(I i ,σ g ) (4) S i =log G(I i ,s s )-log G(I i ,s g ) (5) R i =exp(L i +S i ) (6) Among them, L i represents the reflection component of the i-th scale, S i represents the illumination component of the i-th level, R i Represents the enhanced image at the i-th scale. S33, weighted fusion is performed on the enhanced images of each layer to obtain a final enhanced image, and the weighted process satisfies the following calculation formula: Among them, ω i Represents the weight of the i-th level scale. S34. Repeat the above steps until the number of data-enhanced samples is balanced with the number of original samples.

4. The irregular-shaped green pepper fruit recognition method based on multi-scale feature fusion according to claim 1 is characterized in that: In S3, the calculation process of adaptively changing the receptive field and extracting the feature layer of irregular shape through the RCNS module specifically includes the following steps: S41. Generate initial sampling coordinates of convolution kernels of different shapes, and assign each initial sampling coordinate to convolution kernels of different sizes. The sampling grid is as follows: S42, generating a feature adaptive offset through a convolution layer, the dimension is (B, 2N, W, H), where B is the batch size, N is the convolution kernel size, W and H are the height and width of the feature map respectively; S43, using the offset to identify the irregular shape of the sample to adaptively adjust the convolution kernel size and generate new sampling coordinates. The calculation formula of the new sampling coordinate ρ is expressed as: p=P0+P n +θ offsets (9) Among them, P0 is the center point of the current convolution window, p n is the initial sampling coordinate, θ offsets The offsets generated for the offset convolutional layer. S44, extract the feature values ​​of non-integer positions from the input feature map by bilinear interpolation, and change the sample shape of each position by resampling to make the fused feature map smoother and more consistent. The specific calculation process is as follows: x(q)≈∑ i ∑ j bilinear(q,(i,j))·x(i,j) (10) Wherein, bilinear(q, (i, j)) represents the bilinear interpolation coefficient between the coordinate q and the integer coordinate point (i, j). S45. Extract features from the feature map using the adjusted sampling position.

5. The method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to claim 1, characterized in that: In S3, the FARS module performs multi-scale feature fusion on the feature layer extracted by the RCNS module by aligning the feature map in spatial dimensions through a bilinear interpolation method, thereby performing pixel-level weighted fusion. The specific calculation process of the feature map alignment method and the pixel-level weighted fusion operation includes the following steps: S51, map the target pixel coordinates (x, y) of the high-resolution H2×W2 feature map F2 to the relative coordinates (x′, y′) in the low-resolution H1×W1 feature map F1, so that F1 is interpolated to the same resolution as F2. The specific calculation process is as follows: S52, determine the coordinates of the four pixel points x1, x2, y1, y2 around the relative coordinate (x', y'), the specific calculation process is as follows: in, Indicates rounding down. Indicates rounding up. S53. Calculate the interpolation weight according to the relative position (x′, y′). The specific formula is as follows: S54, taking weighted average of the values ​​of the four surrounding pixels according to the interpolation weights to obtain the value of the target coordinates. The specific formula is as follows: Among them, F1(x1, y1), F1(x1, y2), F1(x2, y1), and F1(x2, y2) are the coordinates of the four corresponding pixels in the low-resolution feature map respectively. S55, the aligned feature map F'1 is fused with the high-resolution feature map F2 pixel by pixel. The fused feature map is expressed as follows: F fusion (x,y)=αF′1(x,y)+βF2(x,y) (22) Among them, α and β are the learned weight matrices.

6. The method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to claim 1, characterized in that: In S4, the feature layer with multi-scale variation characteristics is input into SK-Neck to extract the image feature map. The specific operation is: add a selective kernel attention SK module at the end of the neck network of YOLOv5s to build SK-Neck, and finally obtain a multi-scale spatial channel feature map. The SK-Neck network includes: The eleventh layer is the RCNS module, the twelfth layer is the upsample module, the thirteenth layer is the residual connection Concat module, the fourteenth layer is the FARS module, the fifteenth layer is the RCNS module, the sixteenth layer is the Upsample module, the seventeenth layer is the residual connection Concat module, the eighteenth layer is the FARS module, the nineteenth layer is the RCNS module, the twentieth layer is the residual connection Concat module, the twenty-first layer is the FARS module, the twenty-second layer is the RCNS module, the twenty-third layer is the residual connection Concat module, the twenty-fourth layer is the FARS module, and the twenty-fifth layer is the selective kernel attention SK module.

7. The method for identifying irregular-shaped green pepper fruits based on multi-scale feature fusion according to claim 6, characterized in that: The specific calculation process of the selective kernel attention SK module includes the following steps: S71, the selective kernel attention SK module includes: a separation module, a fusion module and a selection module, wherein the separation module inputs the feature map X∈R W×H×C It is divided into two branches, and the specific formula of the separation process is as follows: U1=Slips 3×3 (X) (23) U2=Skip 5×5 (X) (24) Among them, U1 and U2 represent the output feature maps of branch 1 and branch 2 respectively; Slipt 3×3 and Slipt 5×5 They represent convolution modules with convolution kernels of 3×3 and 5×5 respectively; S72, the fusion module adds the two sets of feature maps U1 and U2 according to the elements to obtain the feature map U, and embeds the global information through the global average pool to generate a feature vector containing different receptive field information. The specific formula is as follows: U=Fuse(U1,U2) (25) Z=f fc (f gp (U)) (26) Among them, Fuse represents the fusion operation; U represents the feature map obtained after fusion; Z represents the output feature vector; f fc represents global average pooling; f gp represents a fully connected layer; S73, the selection module uses the SoftMax function to obtain the final feature map F c , the specific formula is as follows: F c =a c U1+b c U2 (27) a c +b c =1 (28) Among them, F c represents the final feature map, a c and b c Represents the attention weights on different convolution kernels.

8. The irregular green pepper fruit recognition system based on multi-scale feature fusion according to claim 1, characterized in that: include: An acquisition module, used for acquiring image data of irregular-shaped green pepper fruits to be detected; The model building module is used to build the adaptive feature extraction RCNS module based on the feature adaptive convolution FAConv based on the YOLOv5 network architecture, and build the FARS module based on the RCNS module; add the selective kernel attention SK module to build the SK-Neck with multi-scale feature extraction capabilities; and then build the irregular green pepper fruit detection model FASK; The first feature extraction module is used to input the image data of the irregular-shaped green pepper fruit to be detected into the detection model FASK, and adaptively change the receptive field through the RCNS module and extract the feature layer of the irregular shape; The feature fusion module is used to perform multi-scale feature fusion on the feature layer extracted by the RCNS module through the FARS module to obtain a feature layer with multi-scale variation characteristics; The second feature extraction module is used to input the feature layer with multi-scale variation characteristics into SK-Neck to extract the multi-scale spatial channel feature map; The detection module is used to obtain a detection image of the irregular-shaped green pepper fruit to be detected according to the image feature map.

Images