A multi-frequency information fusion double-main nerve network fruit recognition method

Abstract

This invention discloses a fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion, relating to the field of fruit recognition technology. The method includes: inputting an RGB image into a predetermined first backbone network to extract color feature maps, generating a fused image based on the RGB image, inputting the fused image into a preset second backbone network to extract edge feature maps; integrating the color feature maps and edge feature maps to generate a fused sequence; generating a position encoding sequence from the fused sequence and inputting the position encoding sequence into a Transformer layer for processing to generate a fused feature map; performing a two-dimensional discrete wavelet transform on the fused feature map to obtain a low-frequency feature map and high-frequency feature maps in multiple directions; generating an enhanced fused feature map based on the low-frequency feature map and the high-frequency feature maps in multiple directions; and training based on the enhanced fused feature map to generate a fruit recognition model. This invention improves the precision accuracy and robustness of fruit recognition in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of fruit recognition technology, specifically to a dual-backbone neural network fruit recognition method that integrates multi-frequency information. Background Technology

[0002] With the development of computer vision and deep learning technologies, the method of fruit recognition using neural networks has gradually been widely used. Typical fruit recognition methods usually use a single image modality for feature extraction and recognition, and rely on color, texture or edge information for classification and detection.

[0003] In existing technologies, typical fruit recognition methods usually involve directly inputting the collected RGB images of the fruit into a single backbone network to extract features of a single modality, such as color, texture, or edge. Based on the extracted features, the target fruit is identified and located. These methods are simple in structure and quick to train, and have high detection accuracy, especially in fruit recognition scenarios with obvious color contrast and little background interference. They can effectively realize the automated identification and sorting of fruits, and improve the automation level and production efficiency of agricultural production.

[0004] However, real-world orchard scenarios are often complex and varied, with diverse fruit appearances, uneven lighting conditions, and potentially highly similar fruit and background colors. Traditional single-modal feature extraction methods struggle to effectively distinguish between background and fruit areas, especially for fruits with blurred edges, those obscured by branches and leaves, or those at different stages of maturity. These methods often suffer from low recognition accuracy and inaccurate localization. Furthermore, existing methods typically neglect the complementarity and correlation between multi-scale and multi-frequency features contained in fruit images, failing to effectively integrate multi-frequency features to improve generalization ability for fruits in different scenarios and at different scales. Consequently, existing models exhibit poor stability and robustness in complex environments. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion, the method comprising:

[0006] S11: Input the acquired RGB image into a predetermined first backbone network to extract color feature maps, generate a fused image based on the RGB image, and input the fused image into a preset second backbone network to extract edge feature maps;

[0007] S12: Integrate the color feature map and the edge feature map to generate a fusion sequence, generate a position encoding sequence from the fusion sequence, and input the position encoding sequence into the Transformer layer for processing to generate a fusion feature map;

[0008] S13: Perform a two-dimensional discrete wavelet transform on the fused feature map to obtain a low-frequency feature map and high-frequency feature maps in multiple directions. Based on the low-frequency feature map and the high-frequency feature maps in multiple directions, generate an enhanced fused feature map.

[0009] S14: Train the fruit recognition model based on the enhanced fusion feature map to generate a fruit recognition model, input the fruit image to be tested into the fruit recognition model, and output the fruit recognition result.

[0010] Furthermore, the steps for generating the fused image are as follows:

[0011] S111, extract the three color channels in the RGB image respectively to obtain a color channel image, wherein the color channel image includes an R channel image, a G channel image and a B channel image;

[0012] S112, perform two-dimensional high-pass filtering on the R channel image, G channel image and B channel image respectively to obtain high-frequency color channel images, the high-frequency color channel images including high-frequency R channel images, high-frequency G channel images and high-frequency B channel images;

[0013] S113, each color channel image is fused with the corresponding high-frequency color channel image pixel by pixel to obtain a fused color channel image, the fused color channel image including a fused R channel image, a fused G channel image and a fused B channel image;

[0014] S114: The fused R-channel image, fused G-channel image, and fused B-channel image are stitched together on the same channel dimension to generate a fused image.

[0015] Furthermore, the convolution kernels for the two-dimensional high-pass filtering process include horizontal high-pass convolution kernels, vertical high-pass convolution kernels, and diagonal high-pass convolution kernels.

[0016] Furthermore, the steps for performing two-dimensional high-pass filtering on the R-channel image, G-channel image, and B-channel image are as follows:

[0017] In obtaining color channel images Channel image at pixel position pixel values , ;

[0018] Convolution kernels based on two-dimensional high-pass filtering for pixel values Perform a two-dimensional convolution operation to generate a three-way convolution result for the β-channel image, the three-way convolution result including horizontal convolution data, vertical convolution data and diagonal convolution data;

[0019] Represented as:

[0020] ;

[0021] ;

[0022] ;

[0023] in, for Horizontal convolutional data of channel images, For vertical convolutional data, For diagonal convolution data, This is a two-dimensional convolution operation;

[0024] based on High-frequency data is obtained by calculating the three-way convolution results of the channel image. Channel image;

[0025] Represented as:

[0026]

[0027] in, High frequency Channel image, .

[0028] Furthermore, the step of integrating the color feature map and the edge feature map to generate a fused sequence is as follows:

[0029] S121: Obtain the color feature map and edge feature map respectively. The spatial dimensions of the color feature map and edge feature map are both... ,in, The number of RGB images input to the first backbone network simultaneously. This represents the number of channels in the color feature map and the edge feature map. , These are the height and width of the color feature map and the edge feature map, respectively.

[0030] S122: Flatten the color feature map and edge feature map respectively, and divide each color feature map and edge feature map into spatial dimensions. Rearranged as To generate a flattening sequence, the flattening sequence including a color flattening sequence and an edge flattening sequence;

[0031] S123: The color flattening sequence and the edge flattening sequence are spliced ​​together to obtain a fused sequence, the spatial size of which is... .

[0032] Furthermore, the steps for generating the fused feature map are as follows:

[0033] a1: Based on sine and cosine functions and a fusion sequence, a position code is generated. A position code sequence is constructed based on multiple position codes, wherein the spatial size of the position code is... The position-coded sequence is obtained based on the position coding.

[0034] a2: The position-encoded sequence is input into the Transformer layer for processing to generate an output sequence, the spatial size of which is... , wherein Number of channels The length of the embedding vector after mapping in the Transformer layer, and used as the number of channels in the fused feature map;

[0035] a3: Reshape the output sequence to rearrange it into a fused feature map, the spatial size of which is... .

[0036] Furthermore, the steps to obtain the low-frequency feature map and the high-frequency feature maps in multiple directions are as follows:

[0037] For each channel of the fused feature map, a two-dimensional convolution is performed using four sets of two-dimensional wavelet convolution kernels to obtain a low-frequency feature map and high-frequency feature maps in three directions. The low-frequency feature map is denoted as... High-frequency feature maps include horizontal high-frequency feature maps. Vertical high-frequency feature map and diagonal high-frequency feature maps ;

[0038] The spatial size of the fused feature map is The stride of each convolutional kernel is 2, and the spatial dimensions of the four feature maps are... ;

[0039] Furthermore, the steps for generating the enhanced fusion feature map are as follows:

[0040] A 3×3 2D convolution kernel is used to process low-frequency feature maps and multi-directional high-frequency feature maps. The convolution stride is 1. The 2D convolution pads each feature map with zeros around its perimeter. After convolution, a SiLU activation function is applied. The number of output channels remains consistent with the number of input channels. The spatial dimensions are [missing information]. ;

[0041] The processed low-frequency feature map, horizontal high-frequency feature map, vertical high-frequency feature map, and diagonal high-frequency feature map are respectively denoted as... The four types of feature maps are subjected to a two-dimensional inverse wavelet transform to be spliced ​​together to generate an enhanced fused feature map, the spatial dimension of which is [missing information]. Among them, the upper left region of the enhanced fusion feature map corresponds to The upper right area corresponds to The lower left area corresponds to The lower right area corresponds to .

[0042] Furthermore, the steps for generating the fruit recognition model are as follows:

[0043] S141: The enhanced fusion feature map is used as the input to the detection head of the object detection network. The detection head is located at each spatial position of the enhanced fusion feature map. The above process outputs one or more original prediction boxes corresponding to the given location through a convolution operation. The parameters of the original prediction boxes include: the center point of each original prediction box relative to the spatial location. coordinate offset The width w and height h of the original prediction box;

[0044] S142: Match each original predicted bounding box output by the detection head with the ground truth bounding boxes provided in the training samples, and calculate the standard IoU value for each successfully matched ground truth bounding box and original predicted bounding box.

[0045] The formula for calculating the standard IoU value is:

[0046]

[0047] in, Standard IoU value, This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. The area of ​​the union of the two regions. This represents the rectangular region corresponding to the original prediction box. This refers to the rectangular region corresponding to the actual bounding box. This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. Let be the area of ​​the union region of the original predicted bounding box and the ground truth bounding box, where ;

[0048] S143: Perform scaling transformations on the original prediction boxes to obtain smaller and larger prediction boxes;

[0049] S144: Based on the calculation formula of standard IoU value, obtain the reduced IoU value between the reduced prediction box and the ground truth box, and the magnified IoU value between the magnified prediction box and the ground truth box. Based on the reduced IoU value, magnified IoU value and standard IoU value, construct a scale response set.

[0050] S145: Calculate the regression loss weights of the original predicted boxes based on the scale response set, and calculate the UIoU bounding box regression loss term of the original predicted boxes based on the regression loss weights;

[0051] S146: Weighted summation of the UIoU bounding box regression loss terms of all original predicted boxes in the training image to obtain the total bounding box regression loss;

[0052] S147: Based on the total bounding box regression loss, train the target detection network and output the target detection network with an error less than or equal to the preset test error threshold as the fruit recognition model.

[0053] Furthermore, the logic of the scaling transformation is as follows:

[0054] A reduced prediction box is constructed based on half the width and half the height of the original prediction box. An enlarged prediction box is constructed based on twice the width and twice the height of the original prediction box, with its width and height being twice that of the original prediction box. The center point coordinates of the reduced and enlarged prediction boxes are the same as those of the original prediction box.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] This invention employs a dual-backbone network to extract color and edge features separately, and effectively integrates feature information from different frequency domains based on a multi-scale fusion strategy. This achieves deep fusion of color and edge information, thereby improving the integrity and robustness of feature representation and enhancing the accuracy of refined identification of fruit targets.

[0057] Meanwhile, based on the sequential stacking structure of positional encoding and Transformer layers, this invention effectively strengthens the spatial positional association between color and edge features to achieve long-range interaction and fusion of different modal features. This significantly improves the network's ability to associate the spatial position and semantic information of fruit targets in complex environments, thereby enhancing its resistance to feature noise and the accuracy of spatial positioning during fruit detection.

[0058] Furthermore, based on the dynamic scale adaptation mechanism of the Unified-IoU loss function, this invention can dynamically capture the scale changes of the bounding box and optimize the regression weights of the detection box in a targeted manner, so as to achieve a more accurate regression optimization effect for target boxes of different scales, further improving the adaptability of the fruit target recognition model to scale changes in real-world scenarios, and enhancing the reliability and stability of the fruit recognition results overall.

[0059] In summary, this invention, based on a dual-backbone neural network with multi-frequency information fusion, effectively integrates multi-scale features of color and edges, thereby improving the precision accuracy and robustness of fruit recognition in complex scenarios. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0061] Figure 1 A flowchart illustrating a fruit recognition method using a dual-backbone neural network with multi-frequency information fusion, provided in an embodiment of the present invention;

[0062] Figure 2 This is a diagram showing the fruit recognition effect before optimization in an embodiment of the present invention;

[0063] Figure 3 This is an optimized fruit recognition effect diagram according to an embodiment of the present invention;

[0064] Figure 4 This is a diagram illustrating the fruit recognition effect under various lighting conditions according to an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Please see Figure 1 As shown in the figure, this embodiment discloses a fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion. The method includes:

[0067] S11: Input the acquired RGB image into a predetermined first backbone network to extract color feature maps, generate a fused image based on the RGB image, and input the fused image into a preset second backbone network to extract edge feature maps;

[0068] It should be noted that: the color feature map is used to characterize the spatial distribution features of the color dimensions such as brightness, saturation and color boundary in the RGB image, and the edge feature map is used to characterize the fruit boundary and fruit texture changes in the fused image;

[0069] RGB images are acquired using image acquisition devices (including but not limited to industrial cameras and high-resolution imaging sensors) installed on the fruit cultivation platform;

[0070] The construction steps of the first backbone network are as follows:

[0071] Multiple processing structures are configured and connected in a hierarchical order. Each processing structure includes a two-dimensional convolution layer, a non-linear activation method, and a spatial dimension reduction. The input of each two-dimensional convolution layer is defined as an input feature map, and the input feature map of the two-dimensional convolution layer is the output of the previous processing structure. For example:

[0072] If the first layer processing structure is (Conv1→SiLU1→Downsample1)→output ;

[0073] The input to Conv2 in the two-layer processing structure is then... ;

[0074] Here, Conv1 is the first 2D convolutional layer, Conv2 is the second 2D convolutional layer, SiLU1 is the first SiLU activation function layer, and Downsample1 is the first spatial dimension reduction operation. Input feature map;

[0075] In this processing structure, all two-dimensional convolution operations employ a 3×3 kernel and a stride of 1. The padding method for all two-dimensional convolution operations in the first backbone network is based on the acquired input features. Figure 4 The zero-padding method is performed with one zero-padding cycle; the activation operation uses the SiLU function; the spatial dimension reduction is achieved by setting a convolution with a stride of 2;

[0076] It should be noted that the first backbone network preferably has 3 to 5 layers. The zero-padding method of "padding one circle" is to add a row or column of all 0 pixel values ​​to each of the four edges (top, bottom, left, and right) of the input feature map, which is the operation of "padding one circle of 0".

[0077] Multiple processing structures are connected in a hierarchical order to construct a continuous feature extraction path, which serves as the main structure of the first backbone network. The RGB image is set as the input of the first backbone network, and the color feature map is used as the output of the first backbone network to generate the first backbone network.

[0078] Specifically, the steps for generating the fused image are as follows:

[0079] S111, extract the three color channels in the RGB image respectively to obtain a color channel image, wherein the color channel image includes an R channel image, a G channel image and a B channel image;

[0080] S112, perform two-dimensional high-pass filtering on the R channel image, G channel image and B channel image respectively to obtain high-frequency color channel images, the high-frequency color channel images including high-frequency R channel images, high-frequency G channel images and high-frequency B channel images;

[0081] Specifically, the convolution kernels for the two-dimensional high-pass filtering process include horizontal high-pass convolution kernels, vertical high-pass convolution kernels, and diagonal high-pass convolution kernels;

[0082] Represented as:

[0083] , ,

[0084] in, For horizontal high-pass convolution kernels, It is a vertical high-pass convolution kernel. A diagonal high-pass convolution kernel;

[0085] Specifically, the steps for performing two-dimensional high-pass filtering on the R-channel image, G-channel image, and B-channel image are as follows:

[0086] In obtaining color channel images Channel image at pixel position pixel values , ;

[0087] Convolution kernels based on two-dimensional high-pass filtering for pixel values Perform a two-dimensional convolution operation to generate a three-way convolution result for the β-channel image, the three-way convolution result including horizontal convolution data, vertical convolution data and diagonal convolution data;

[0088] Represented as:

[0089] ;

[0090] ;

[0091] ;

[0092] in, for Horizontal convolutional data of channel images, For vertical convolutional data, For diagonal convolution data, This is a two-dimensional convolution operation;

[0093] based on High-frequency data is obtained by calculating the three-way convolution results of the channel image. Channel image;

[0094] Represented as:

[0095]

[0096] in, High frequency Channel image, ;

[0097] S113, each color channel image is fused with the corresponding high-frequency color channel image pixel by pixel to obtain a fused color channel image, the fused color channel image including a fused R channel image, a fused G channel image and a fused B channel image;

[0098] The formula for pixel-by-pixel fusion is:

[0099]

[0100] In the formula, To blend color channel images at pixel locations pixel values, For color channel images at pixel positions pixel values, For high-frequency color channel images at pixel locations pixel values, As a weighting factor, the preferred one is... The value range is [0.7, 0.9];

[0101] S114, the fused R channel image, fused G channel image, and fused B channel image are stitched together on the same channel dimension to generate a fused image;

[0102] It should be noted that the fused image has the same size and channel structure as the input RGB image. The fused R channel image, G channel image and B channel image are stitched together in the channel dimension, which is a prior art technique and will not be described in detail.

[0103] Based on the generation logic of the main structure of the first backbone network, the main structure of the second backbone network is obtained, the fused image is used as the input of the second backbone network, and the edge feature map is used as the output of the second backbone network.

[0104] It should be noted that the first backbone network and the second backbone network have the same network structure but different inputs and outputs; the edge features are the edge intensity variation regions represented on multiple channels after the second backbone network extracts the high-frequency information contained in the fused image, including but not limited to object contour regions, texture variation regions, and boundary discontinuity regions.

[0105] S12: Integrate the color feature map and the edge feature map to generate a fusion sequence, generate a position encoding sequence from the fusion sequence, and input the position encoding sequence into the Transformer layer for processing to generate a fusion feature map;

[0106] Specifically, the steps for integrating the color feature map and the edge feature map to generate a fused sequence are as follows:

[0107] S121: Obtain the color feature map and edge feature map respectively. The spatial dimensions of the color feature map and edge feature map are both... ,in, The number of RGB images input to the first backbone network simultaneously. This represents the number of channels in the color feature map and the edge feature map. , These are the height and width of the color feature map and the edge feature map, respectively.

[0108] It should be noted that the number of channels in the color feature map and edge feature map is greater than the number of channels in the RGB image;

[0109] S122: Flatten the color feature map and edge feature map respectively, and divide each color feature map and edge feature map into spatial dimensions. Rearranged as To generate a flattening sequence, the flattening sequence including a color flattening sequence and an edge flattening sequence;

[0110] It should be noted that: This is the input format required by technologies such as Transformer, MLP, and positional encoding.

[0111] S123: Concatenate the color flattening sequence and the edge flattening sequence to obtain the blended sequence. The fusion sequence The space dimensions are ;

[0112] Specifically, the steps for generating the fused feature map are as follows:

[0113] a1: Based on sine and cosine functions and a fusion sequence, a position code is generated. A position code sequence is constructed based on multiple position codes, wherein the spatial size of the position code is... ;

[0114]

[0115] in, For the location index is In the fusion sequence, the first The position code of each channel. For the location index is In the fusion sequence, the first The position code of each channel, For the position index in the fused sequence, , For channel indexing, ;

[0116] The position-coded sequence is obtained based on the position coding and is represented as follows:

[0117]

[0118] a2: The position-encoded sequence is input into the Transformer layer for processing to generate an output sequence, the spatial size of which is... The Number of channels The length of the embedding vector after mapping in the Transformer layer, and used as the number of channels in the fused feature map;

[0119] It should be noted that, under the existing Transformer linear mapping, the input channel vector at each position is changed from the original number of channels. Mapped to a length of The vectors that form the output of the Transformer encoder;

[0120] The Transformer layers are constructed in a stacked manner, containing a total of 8 encoder layers connected in sequence. The output of each layer serves as the input of the next layer, forming a layer-by-layer feature propagation structure, as shown below:

[0121]

[0122] in, For the output sequence, For the first One Transformer encoder layer;

[0123] It should be noted that the connection in sequence refers to the connection according to the layout order of the encoder layer;

[0124] a3: Reshape the output sequence to rearrange it into a fused feature map, the spatial size of which is... ;

[0125] Represented as:

[0126]

[0127] The Reshape process is used to transform the output sequence. Transform into a fused feature map ;

[0128] It should be noted that the above method combines low-frequency feature maps, horizontal high-frequency feature maps, vertical high-frequency feature maps, and diagonal high-frequency feature maps according to spatial region stitching, and restores them into an enhanced fused feature map. The process belongs to the existing technology of two-dimensional inverse wavelet reconstruction based on wavelet filter banks in image processing. It has been widely used in the fields of frequency domain feature reconstruction and image multi-resolution processing, so the specific calculation process of the inverse transform will not be described in detail.

[0129] S13: Perform a two-dimensional discrete wavelet transform on the fused feature map to obtain a low-frequency feature map and high-frequency feature maps in multiple directions. Based on the low-frequency feature map and the high-frequency feature maps in multiple directions, generate an enhanced fused feature map.

[0130] Specifically, the steps to obtain low-frequency feature maps and high-frequency feature maps in multiple directions are as follows:

[0131] For each channel of the fused feature map, a two-dimensional convolution is performed using four sets of two-dimensional wavelet convolution kernels to obtain a low-frequency feature map and high-frequency feature maps in three directions. The low-frequency feature map is denoted as... High-frequency feature maps include horizontal high-frequency feature maps. Vertical high-frequency feature map and diagonal high-frequency feature maps ;

[0132] The spatial size of the fused feature map is Each convolutional kernel has a stride of 2. Two-dimensional convolution halves the spatial size of the fused feature map while maintaining the same number of channels. The spatial dimensions of the four types of feature maps are updated to ;

[0133] Among them, the low-frequency feature map is extracted. wavelet convolution kernel is Extract the horizontal high-frequency feature map wavelet convolution kernel is Extract the vertical high-frequency feature map wavelet convolution kernel is Extract the diagonal high-frequency feature map wavelet convolution kernel is ;

[0134] Specifically, the steps for generating the enhanced fusion feature map are as follows:

[0135] A 3×3 two-dimensional convolution is used to process low-frequency feature maps and multi-directional high-frequency feature maps. The convolution stride is 1. The two-dimensional convolution pads each feature map with zeros around its perimeter. After convolution, a SiLU activation function is applied. The number of output channels remains consistent with the number of input channels, and the spatial resolution remains the same. ;

[0136] The processed low-frequency feature map, horizontal high-frequency feature map, vertical high-frequency feature map, and diagonal high-frequency feature map are respectively denoted as... The four types of feature maps are subjected to a two-dimensional inverse wavelet transform to be spliced ​​together to generate an enhanced fused feature map, the spatial dimension of which is [missing information]. Among them, the upper left region of the enhanced fusion feature map corresponds to The upper right area corresponds to The lower left area corresponds to The lower right area corresponds to ;

[0137] It should be noted that the two-dimensional high-pass filtering operation used in step S112 is used for edge feature extraction at the image channel level; while the wavelet filter bank-based operation in S13 is used for multi-scale frequency domain decomposition of the fused feature map output by Transformer. Although some filter kernels are the same, the processing objects, semantic levels and processing goals are completely different and do not constitute technical content duplication.

[0138] S14: Train based on enhanced fusion feature map to generate fruit recognition model, input the fruit image to be tested into the fruit recognition model, and output fruit recognition result;

[0139] Specifically, the steps for generating the fruit recognition model are as follows:

[0140] S141: The enhanced fusion feature map is used as the input to the detection head of the object detection network. The detection head is located at each spatial position of the enhanced fusion feature map. The above process outputs one or more original prediction boxes corresponding to the given location through a convolution operation. The parameters of the original prediction boxes include: the center point of each original prediction box relative to the spatial location. coordinate offset The width w and height h of the original prediction box;

[0141] S142: Match each original predicted bounding box output by the detection head with the ground truth bounding boxes provided in the training samples, and calculate the standard IoU value for each successfully matched ground truth bounding box and original predicted bounding box.

[0142] The formula for calculating the standard IoU value is:

[0143]

[0144] in, Standard IoU value, This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. The area of ​​the union of the two regions. This represents the rectangular region corresponding to the original prediction box. This refers to the rectangular region corresponding to the actual bounding box. This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. Let be the area of ​​the union region of the original predicted bounding box and the ground truth bounding box, where ;

[0145] It should be noted that: and The position and dimensions are determined by the center point coordinates and width and height parameters output by the detection head;

[0146] S143: Perform a scaling transformation on the original prediction box, the scaling transformation including:

[0147] Construct a reduced prediction box with a width and height half that of the original prediction box; construct an enlarged prediction box with a width and height twice that of the original prediction box; the center point coordinates of the reduced and enlarged prediction boxes are consistent with those of the original prediction box.

[0148] S144: Based on the calculation formula of standard IoU value, obtain the reduced IoU value between the reduced prediction box and the ground truth box, and the magnified IoU value between the magnified prediction box and the ground truth box. Based on the reduced IoU value, magnified IoU value and standard IoU value, construct a scale response set.

[0149] Represented as:

[0150]

[0151] in, To reduce the IoU value, To amplify the IoU value;

[0152] S145: Calculate the regression loss weights of the original predicted boxes based on the scale response set, and calculate the UIoU bounding box regression loss term of the original predicted boxes based on the regression loss weights;

[0153] The formula for calculating the regression loss weight is as follows:

[0154]

[0155] In the formula, To regress the loss weights, As a balance factor, the preferred one is... It is 0.5;

[0156] The formula for calculating the UIoU bounding box regression loss term of the original predicted bounding box is:

[0157]

[0158] The regression loss is the original predicted bounding box. For standard IoU loss terms;

[0159] S146: Weighted summation of the UIoU bounding box regression loss terms of all original predicted boxes in the training image to obtain the total bounding box regression loss;

[0160] Represented as:

[0161]

[0162] in, The total bounding box regression loss is used to train the images. This represents the total number of original predicted bounding boxes used in the calculation from the training image. For the first The UIoU bounding box regression loss term for each original predicted bounding box;

[0163] S147: Based on the total bounding box regression loss, train the target detection network and output the target detection network with an error less than or equal to the preset test error threshold as the fruit recognition model.

[0164] It should be noted that the test error threshold is set by the experimenters.

[0165] refer to Figure 2 , Figure 3 and Figure 4 , Figure 2 The images displayed when existing recognition methods identify fruits have problems such as low fruit recognition accuracy and inaccurate positioning for fruits with blurred edges, those obscured by branches and leaves, or those at different stages of maturity.

[0166] Figure 3 The optimized recognition method displays the image when identifying the fruit, enabling the recognition of... Figure 2 To optimize the issues of low fruit recognition accuracy and inaccurate positioning, Figure 4 The results show that, under the optimized recognition method, accurate fruit identification can be achieved even when the fruit is under different lighting conditions;

[0167] It should be noted that: in the figure, noripe is the identification mark for immature fruit, halfripe is the identification mark for semi-ripe fruit, and ripe is the identification mark for ripe fruit;

[0168] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A fruit recognition method using a dual-backbone neural network with multi-frequency information fusion, characterized in that, The method includes: S11: Input the acquired RGB image into a predetermined first backbone network to extract color feature maps, generate a fused image based on the RGB image, and input the fused image into a preset second backbone network to extract edge feature maps; S12: Integrate the color feature map and the edge feature map to generate a fusion sequence, generate a position encoding sequence from the fusion sequence, and input the position encoding sequence into the Transformer layer for processing to generate a fusion feature map; S13: Perform a two-dimensional discrete wavelet transform on the fused feature map to obtain a low-frequency feature map and high-frequency feature maps in multiple directions. Based on the low-frequency feature map and the high-frequency feature maps in multiple directions, generate an enhanced fused feature map. S14: Train based on enhanced fusion feature map to generate fruit recognition model, input the fruit image to be tested into the fruit recognition model, and output fruit recognition result; The steps to generate a fruit recognition model are as follows: S141: The enhanced fusion feature map is used as the input to the detection head of the object detection network. The detection head is located at each spatial position of the enhanced fusion feature map. The above process outputs one or more original prediction boxes corresponding to the given location through a convolution operation. The parameters of the original prediction boxes include: the center point of each original prediction box relative to the spatial location. coordinate offset The width w and height h of the original prediction box; S142: Match each original predicted bounding box output by the detection head with the ground truth bounding boxes provided in the training samples, and calculate the standard IoU value for each successfully matched ground truth bounding box and original predicted bounding box. The formula for calculating the standard IoU value is: in, Standard IoU value, This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. The area of ​​the union of the two regions. This represents the rectangular region corresponding to the original prediction box. This refers to the rectangular region corresponding to the actual bounding box. This represents the area of ​​the intersection region between the original predicted bounding box and the ground truth bounding box. Let be the area of ​​the union region of the original predicted bounding box and the ground truth bounding box, where ; S143: Perform scaling transformations on the original prediction boxes to obtain smaller and larger prediction boxes; S144: Based on the calculation formula of standard IoU value, obtain the reduced IoU value between the reduced prediction box and the ground truth box, and the magnified IoU value between the magnified prediction box and the ground truth box. Based on the reduced IoU value, magnified IoU value and standard IoU value, construct a scale response set. S145: Calculate the regression loss weights of the original predicted boxes based on the scale response set, and calculate the UIoU bounding box regression loss term of the original predicted boxes based on the regression loss weights; S146: Weighted summation of the UIoU bounding box regression loss terms of all original predicted boxes in the training image to obtain the total bounding box regression loss; S147: Based on the total bounding box regression loss, train the target detection network and output the target detection network with an error less than or equal to the preset test error threshold as the fruit recognition model.

2. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 1, characterized in that, The steps to generate the fused image are as follows: S111, extract the three color channels in the RGB image respectively to obtain a color channel image, wherein the color channel image includes an R channel image, a G channel image and a B channel image; S112, perform two-dimensional high-pass filtering on the R channel image, G channel image and B channel image respectively to obtain high-frequency color channel images, the high-frequency color channel images including high-frequency R channel images, high-frequency G channel images and high-frequency B channel images; S113, each color channel image is fused pixel by pixel with the corresponding high-frequency color channel image to obtain a fused color channel image, the fused color channel image including a fused R channel image, a fused G channel image and a fused B channel image; S114: The fused R-channel image, fused G-channel image, and fused B-channel image are stitched together on the same channel dimension to generate a fused image.

3. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 2, characterized in that, The convolutional kernels for the two-dimensional high-pass filtering process include horizontal high-pass kernels, vertical high-pass kernels, and diagonal high-pass kernels.

4. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 3, characterized in that, The steps for performing two-dimensional high-pass filtering on the R-channel image, G-channel image, and B-channel image are as follows: In obtaining color channel images Channel image at pixel position pixel values , ; Convolution kernels based on two-dimensional high-pass filtering for pixel values Perform a two-dimensional convolution operation to generate a three-way convolution result for the β-channel image, the three-way convolution result including horizontal convolution data, vertical convolution data and diagonal convolution data; Represented as: ； ； ； in, for Horizontal convolutional data of channel images, For vertical convolutional data, For diagonal convolution data, This is a two-dimensional convolution operation; based on High-frequency data is obtained by calculating the three-way convolution results of the channel image. Channel image; Represented as: in, High frequency Channel image, .

5. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 4, characterized in that, The steps for integrating the color feature map and the edge feature map to generate the fused sequence are as follows: S121: Obtain the color feature map and edge feature map respectively. The spatial dimensions of the color feature map and edge feature map are both... ,in, The number of RGB images input to the first backbone network simultaneously. This represents the number of channels in the color feature map and the edge feature map. , These are the height and width of the color feature map and the edge feature map, respectively. S122: Flatten the color feature map and edge feature map respectively, and divide each color feature map and edge feature map into spatial dimensions. Rearranged as To generate a flattening sequence, the flattening sequence including a color flattening sequence and an edge flattening sequence; S123: The color flattening sequence and the edge flattening sequence are spliced ​​together to obtain a fused sequence, the spatial size of which is... .

6. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 5, characterized in that, The steps for generating the fused feature map are as follows: a1: Based on sine and cosine functions and a fusion sequence, a position code is generated. A position code sequence is constructed based on multiple position codes, wherein the spatial size of the position code is... The position-coded sequence is obtained based on the position coding. a2: The position-encoded sequence is input into the Transformer layer for processing to generate an output sequence, the spatial size of which is... , wherein Number of channels The length of the embedding vector after mapping in the Transformer layer, and used as the number of channels in the fused feature map; a3: Reshape the output sequence to rearrange it into a fused feature map, the spatial size of which is... .

7. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 6, characterized in that, The steps to obtain low-frequency feature maps and high-frequency feature maps in multiple directions are as follows: For each channel of the fused feature map, a two-dimensional convolution is performed using four sets of two-dimensional wavelet convolution kernels to obtain a low-frequency feature map and high-frequency feature maps in three directions. The low-frequency feature map is denoted as... High-frequency feature maps include horizontal high-frequency feature maps. Vertical high-frequency feature map and diagonal high-frequency feature maps ; The spatial size of the fused feature map is The stride of each convolutional kernel is 2, and the spatial dimensions of the four feature maps are... .

8. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 7, characterized in that, The steps for generating the enhanced fusion feature map are as follows: A 3×3 2D convolution kernel is used to process low-frequency feature maps and multi-directional high-frequency feature maps. The convolution stride is 1. The 2D convolution pads each feature map with zeros around its perimeter. After convolution, a SiLU activation function is applied. The number of output channels remains consistent with the number of input channels. The spatial dimensions are [missing information]. ; The processed low-frequency feature map, horizontal high-frequency feature map, vertical high-frequency feature map, and diagonal high-frequency feature map are respectively denoted as... The four types of feature maps are subjected to a two-dimensional inverse wavelet transform to be spliced ​​together to generate an enhanced fused feature map, the spatial dimension of which is [missing information]. Among them, the upper left region of the enhanced fusion feature map corresponds to The upper right area corresponds to The lower left area corresponds to The lower right area corresponds to .

9. The fruit recognition method based on a dual-backbone neural network using multi-frequency information fusion according to claim 8, characterized in that, The logic of the scaling transformation is as follows: A reduced prediction box is constructed based on half the width and half the height of the original prediction box. An enlarged prediction box is constructed based on twice the width and twice the height of the original prediction box, with its width and height being twice that of the original prediction box. The center point coordinates of the reduced and enlarged prediction boxes are the same as those of the original prediction box.

Images