A tea bud detection method based on RGB and spectral image bimodal fusion

Abstract

The application discloses a kind of based on RGB and spectral image double mode fusion's tea bud detection method, comprising:1, obtain tea garden original hyperspectral image and carry out dimension reduction and wave band screening, obtain optimal hyperspectral image and corresponding RGB image and label information;2, construct DMF-YOLO network, respectively extract multi-scale feature to double mode image, by double dimension cross-modal fusion, multi-scale pyramid aggregation and target detection head processing, output the prediction result of tea bud;3, for the tiny dense characteristics of tea bud, construct the composite loss function consisting of scale perception positioning loss, classification loss and confidence loss;4, the network is iteratively trained to loss function convergence by optimization algorithm, and the detection model is obtained.The application can effectively solve the problems of wave band redundancy, mode alignment difficulty and complex background interference in double mode data fusion, thereby the accuracy and robustness of tea bud target detection in complex agricultural environment can be significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically a tea bud detection method based on dual-modal fusion of RGB and spectral images. Background Technology

[0002] Accurate target detection is crucial for automated tea harvesting. Existing technologies widely employ target detection methods based on RGB images captured by a single visible light camera. However, tea buds and mature leaves are highly similar in color and shape, resulting in low visual differentiation. Models struggle to construct differential features solely based on RGB information, leading to frequent false negatives and missed detections. Furthermore, environmental factors in tea gardens, such as fluctuating natural light, complex background interference, and branch occlusion, further exacerbate the limitations of detection based on single image information.

[0003] To compensate for the limitations of visible light images, hyperspectral images have been introduced into target detection because they can simultaneously acquire spatial and spectral information of a target, reflecting the absorption or reflection characteristics of objects at different wavelengths of photons due to differences in composition. Tea buds and older leaves exhibit significant differences in the content of key biochemical substances such as chlorophyll and water, and hyperspectral imaging can effectively capture these biochemical differences. However, hyperspectral data suffers from high redundancy, and when fusion with RGB images for multimodal processing, existing mid- or late-stage fusion strategies often employ simple feature stitching or addition operations, failing to address the feature alignment challenges between heterogeneous modalities and thus limiting the detection accuracy in complex backgrounds. Summary of the Invention

[0004] This invention aims to address the shortcomings of existing technologies by proposing a tea bud detection method based on dual-modal fusion. This method addresses issues such as band redundancy, modal alignment difficulties, and interference from complex backgrounds in hyperspectral and visible light fusion tasks. Through an adaptive alignment and weighted feature fusion module, it comprehensively and accurately captures the visual and biochemical characteristics of tea bud targets, significantly improving the accuracy and robustness of target detection in complex agricultural environments.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The tea bud detection method based on dual-modal fusion of RGB and spectral images of this invention is characterized by the following steps: Step 1: Obtain the original hyperspectral image set in the tea garden environment and perform dimensionality reduction and band selection processing to obtain the hyperspectral image set corresponding to the optimal band. ,in, Indicates the first A hyperspectral image, Indicates the height of the hyperspectral image. Indicates the width of the hyperspectral image. Indicates the optimal number of bands; Will The corresponding RGB image is denoted as ,right All tea bud locations were labeled to obtain a tag set. ,make The Middle The labeling information for each tea bud is recorded as follows: ,in, Indicates the first The coordinates of the center point of the annotation box containing each tea bud and The first The width and height of the label box containing each tea bud. For the first The actual category label of the label box containing each tea bud; Step 2: Construct a DMF-YOLO tea bud detection network based on dual-modal feature fusion, including: a multimodal dual-branch feature extraction network, a dual-dimensional cross-modal feature fusion module, a multi-scale feature aggregation neck network, and a target detection head, and then respectively... and Processing is performed to obtain the prediction results. ; Step 2.1: The multimodal dual-branch feature extraction network consists of an RGB modal backbone feature extraction network and a hyperspectral modal feature extraction network, which are respectively used for... and Downsampling and multi-scale feature extraction are performed to obtain the corresponding results. Visible light image feature sequences at various scales and Hyperspectral biochemical signature sequences at various scales ,in, express The Visible light image features at various scales, express The Hyperspectral biochemical characteristics at various scales; Step 2.2, and The input is processed together in the dual-dimensional cross-modal feature fusion module using dynamic weighting and symmetrical bidirectional cross-attention interaction, and the output is the first... Cross-modal fusion features at various scales ; Step 2.3: The multi-scale feature aggregation neck network fuses multi-scale cross-modal feature sequences. Multi-scale aggregation is performed, resulting in a multi-scale aggregated feature sequence. ,in, Indicates the first Aggregational features at various scales; Step 2.4, Target detection head pair Process and output. and Prediction results ,include: The Middle Prediction information of the prediction box containing each tea bud ,in, Representing the first The coordinates of the center point of the prediction box containing each tea bud and They represent the first The width and height of the prediction frame containing each tea bud; Representing the Confidence of the prediction frame containing each tea bud; Representing the The predicted category probability of the prediction box containing each tea bud; Step 3, based on and Construct a scale-aware localization loss Classification loss With confidence loss Composite loss function composed of dynamic weights ; Step 4: Iteratively train the DMF-YOLO tea bud detection network based on dual-modal feature fusion using an optimization algorithm, and calculate... Adjust network parameters using backpropagation until... The process continues until convergence, resulting in a trained tea bud detection model used for detecting tea bud targets.

[0006] The tea bud detection method based on dual-modal fusion described in this invention is also characterized in that step 2.1 includes: Step 2.1.1: The visible light modal backbone feature extraction network includes: The system consists of several cascaded processing stages, each containing a convolutional layer for performing downsampling and a C2f feature extraction module. when At that time, Enter the first Downsampling is performed in the convolutional layer of the first stage, and the resulting downsampling features are then input into the second stage. Spatial texture feature extraction is performed in the C2f module to obtain the first... Visible light spatial characteristics at various scales ; when At that time, the first Visible light spatial characteristics at various scales Enter the first Processing is carried out in the stage; the result is obtained. Visible light spatial characteristics at various scales ; and thus by Visible light spatial features at different scales constitute a visible light spatial feature sequence. ; Step 2.1.2: The hyperspectral modal feature extraction network includes: The process consists of cascaded processing stages, each containing a convolutional layer and a C2f_SE feature extraction module; any cascaded processing stage... Each C2f_SE feature extraction module consists of a C2f feature extraction unit and an SE channel attention branch connected in series: when ,Will Enter the first Downsampling is performed in the convolutional layer of the first stage, and the resulting downsampling features are then input into the second stage. Spatial spectral features are extracted from the C2f unit of the C2f_SE module, and the output is the first... Intermediate features Then enter the number The process is performed in the SE channel attention branch of the C2f_SE module to obtain the first... Hyperspectral biochemical characteristics at various scales ; when At that time, the first Hyperspectral biochemical characteristics at scale Enter the first In the stage, the first equation is obtained using equation (1). Hyperspectral biochemical characteristics at various scales ; and obtained by A hyperspectral feature sequence composed of hyperspectral features at different scales ; (1) In equation (1), This indicates a global average pooling operation. and These represent the nonlinear mapping operations of the dimension-reduction fully connected layer and the dimension-upgrading fully connected layer, respectively. This represents a channel-by-channel multiplication operation.

[0007] Furthermore, step 2.2 includes: Step 2.2.1, After being processed by the linear mapping layer, the corresponding result is obtained. Query vector for visible light modes at each scale Key vector Value vector and the Query vector of hyperspectral modes at each scale Key vector Value vector Therefore, by using equations (2) and (3), we can obtain the first... Visible light enhancement features of hyperspectral modal injection at various scales and the Hyperspectral enhancement features of visible light mode injection at various scales : (2) (3) In equations (2) and (3), For the first Reliability weighting coefficients for visible light modes at each scale. For the reliability weighting coefficients of the hyperspectral modes; For the first The channel dimension of the key vector at each scale; T represents the transpose; Step 2.2.2, for and Perform aggregation to obtain the first... Each characteristic residual term Thus, by using equation (4), the first... Cross-modal fusion features at various scales : (4) In equation (4), Concat represents the feature concatenation operation along the channel dimension, Conv represents the convolution operation for integrating feature information, Gate represents the channel gating operation for filtering redundant information, and + represents the element-wise addition operation.

[0008] Furthermore, step 2.3 includes: Step 2.3.1: Utilize the top-down path in the neck network of the multi-scale feature pyramid to fuse feature sequences across modalities. Upsampling and feature concatenation are performed to obtain the corresponding results. Top-down augmentation feature sets at various scales ;in, Indicates the first The first scale A top-down enhancement feature; when season ; when At that time, for the first Top-down enhancement features at each scale After upsampling, and By splicing them together, we get ; Step 2.3.2: Utilize the bottom-up path in the multi-scale feature pyramid neck network to enhance the top-down feature sequence. Cross-scale fusion is performed to obtain bottom-up localization feature sequences. ; when season ; when At that time, using convolutional layers to... Bottom-up localization features at various scales After downsampling, and By splicing and merging, the first... Bottom-up localization features at various scales ; Step 2.3.3: Using equations (5) and (6) to... By performing local refinement and global logic optimization, the first... Aggregation features at each scale : (5) (6) In equations (5) and (6), This represents a local feature extraction unit in the C2f_MSA module. This indicates a flattening operation in spatial dimensions. Representative level normalization, Represents a multi-head self-attention mechanism. This represents the operation of restoring the three-dimensional topology. Indicates the first The flattened one-dimensional patch sequence features at each scale, where + indicates an element-wise additive residual join operation.

[0009] Furthermore, step 3 includes: Step 3.1: Construct scale-aware localization loss using equation (7). : (7) In equation (7), The total number of samples, For the first The total number of tea bud targets matched in each sample. express and Complete intersection and union ratio; and They respectively represent the following: and The Gaussian distribution parameters derived from the center point coordinates and the width and height geometric attributes; express and The second-order Wasserstein distance between them; This is a preset distance normalization constant; These are the preset balance hyperparameters; Step 3.2, Calculation as well as The class probability deviation between them is used to construct the classification loss. ; Step 3.3, Calculation as well as Calculate the confidence loss based on the confidence deviation between them. ; Step 3.4: Construct a composite loss function for network parameter optimization using equation (8). : (8) In equation (8), These are the preset weighting coefficients for localization loss, classification loss, and confidence loss, respectively.

[0010] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in performing the method described therein, and the processor is configured to execute the program stored in the memory.

[0011] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to perform the steps of the method described thereon.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention takes into account the significant differences in biochemical components between tea buds and mature leaves, and therefore introduces the C2f_SE module to achieve adaptive adjustment of hyperspectral channel weights. This design can automatically learn and enhance key spectral responses reflecting the characteristics of tea buds, effectively suppress irrelevant background noise, and provide the model with highly discriminative biochemical feature inputs.

[0013] 2. This invention addresses the feature mismatch issue in the data structure of visible light and hyperspectral images by designing a TCSA module to achieve symmetrical bidirectional cross-attention interaction. This mechanism realizes deep mapping and complementary advantages of heterogeneous modalities in spatial and channel dimensions, resolves visual ambiguity under complex lighting conditions, and significantly enhances the model's ability to identify similar background interference.

[0014] 3. This invention addresses the issues of dense target overlap and foliage occlusion in the complex background of tea gardens. To this end, a C2f_MSA module is introduced into the neck network to achieve simultaneous modeling of local saliency and global logic. This architecture effectively eliminates redundant interference from environmental noise by capturing global dependencies between pixels, significantly improving the model's detection recall and robustness under occlusion conditions.

[0015] 4. This invention addresses the problem of traditional localization metrics failing to converge due to the extremely small size of tea buds. To resolve this, a scale-aware loss function based on normalized Wasserstein distance (NWD) is introduced. This method models the target as a two-dimensional Gaussian distribution for distance measurement, overcoming the gradient oscillation problem caused by extremely small targets and achieving high-precision localization and stable training of tiny tea buds. Attached Figure Description

[0016] Figure 1 This is a flowchart of the band selection method based on OIF and mRMR of the present invention; Figure 2 This is a structural diagram of the DMF-YOLO dual-modal detection network of the present invention; Figure 3 This is a structural diagram of the fusion SE attention module C2f_SE in the network of this invention; Figure 4 This is a structural diagram of the two-dimensional cross-modal attention module (TCSA) in the network of this invention; Figure 5 This is a structural diagram of the multi-head self-attention module C2f_MSA in the network of this invention. Detailed Implementation

[0017] In this embodiment, a tea bud detection method based on dual-modal fusion of RGB and spectral images is performed according to the following steps: Step 1: Obtain the original hyperspectral image set in the tea garden environment and perform dimensionality reduction and band selection processing to obtain the hyperspectral image set corresponding to the optimal band. ,in, Indicates the first A hyperspectral image, Indicates the height of the hyperspectral image. Indicates the width of the hyperspectral image. This indicates the optimal number of bands.

[0018] like Figure 1 The flowchart shown is a representation of the band selection method of this invention. This step first calculates the standard deviation and correlation coefficient of the bands for unsupervised pre-screening; then, guided by the label set, a forward greedy search is used to calculate the complementary information. Through this mechanism, highly redundant bands in the hyperspectral data are eliminated, while core bands sensitive to tea bud moisture and metabolites are retained.

[0019] Will The corresponding RGB image is denoted as ,right All tea bud locations were labeled to obtain a tag set. ,make The Middle The labeling information for each tea bud is recorded as follows: ,in, Indicates the first The coordinates of the center point of the annotation box containing each tea bud and The first The width and height of the label box containing each tea bud. For the first The actual category label of the label box containing each tea bud.

[0020] Step 2, as follows Figure 2 As shown, a DMF-YOLO tea bud detection network based on dual-modal feature fusion is constructed. Its overall architecture includes: a multimodal dual-branch feature extraction network, a dual-dimensional cross-modal feature fusion module (TCSA), a multi-scale feature pyramid neck network, and a target detection head; the features obtained in the aforementioned steps are then... and The data are input into the network and processed to obtain the final prediction results. .

[0021] Step 2.1: The multimodal dual-branch feature extraction network consists of an RGB modal backbone feature extraction network and a hyperspectral modal feature extraction network, which are used to extract features from the RGB modal backbone network and the hyperspectral modal feature extraction network, respectively. and Downsampling and multi-scale feature extraction are performed to obtain the corresponding results. Visible light image feature sequences at various scales and Hyperspectral biochemical signature sequences at various scales ,in, express The Visible light image features at various scales, express The Hyperspectral biochemical characteristics at various scales.

[0022] Step 2.1.1: The visible light modal backbone feature extraction network includes: The system consists of several cascaded processing stages, each containing a convolutional layer for performing downsampling and a C2f feature extraction module. when At that time, Enter the first Downsampling is performed in the convolutional layer of the first stage, and the resulting downsampling features are then input into the second stage. Spatial texture feature extraction is performed in the C2f module to obtain the first... Visible light spatial characteristics at various scales .

[0023] when At that time, the first Visible light spatial characteristics at various scales Enter the first Processing is carried out in the stage; the result is obtained. Visible light spatial characteristics at various scales ; and thus by Visible light spatial features at different scales constitute a visible light spatial feature sequence. .

[0024] Step 2.1.2, the hyperspectral modal feature extraction network includes: Each convolutional layer and Each C2f_SE feature extraction module, such as Figure 3 The diagram shows the structure of the fused SE attention module C2f_SE; and the progressive downsampling process in step 2.1.1 is followed. Processing is performed to obtain equation (1). Hyperspectral biochemical characteristics at various scales .

[0025] (1) Its internal data processing logic is as follows: First, the input features are used to extract intermediate features through C2f units. Subsequently, the intermediate features are processed along the spatial dimension using global average pooling (GAP). The vector is compressed and mapped to a one-dimensional global channel feature vector to converge the global receptive field. This feature vector is then passed sequentially through dimensionality-reducing fully connected layers. ReLU activation function, upgraded fully connected layer A nonlinear transformation is performed to capture the dependencies between different spectral channels. Finally, a normalized channel weight vector is generated using the Sigmoid activation function and then... Perform channel-by-channel multiplication.

[0026] During the feature extraction stage, the module automatically learns and amplifies the spectral feature responses that are sensitive to tea buds, suppresses background noise channels, and improves the quality of underlying biochemical features.

[0027] Step 2.2, as follows Figure 4 The diagram shows the structure of the Two-Dimensional Cross-Modal Attention Module (TCSA), designed to address the differences in representation emphasis between hyperspectral and visible light images. This module integrates visible light image features... With hyperspectral biochemical characteristics Perform symmetrical bidirectional cross-attention processing to obtain the first... Cross-modal fusion features at various scales : Step 2.2.1: Input bimodal features Mapping these to the query, key, and value spaces respectively, we obtain the first... Query vector for visible light modes at each scale Key vector Value vector and query vectors for hyperspectral modes Key vector Value vector .

[0028] Using Equation (2), the query vector of the visible light mode and the key vector and value vector of the hyperspectral mode are subjected to cross-attention interaction processing to obtain the visible light enhancement features injected by the hyperspectral mode. Simultaneously, by using Equation (3) to perform cross-attention interaction processing on the query vector of the hyperspectral mode and the key vector and value vector of the visible light mode, the hyperspectral enhancement features injected into the visible light mode are obtained. : (2) (3) In equations (2) and (3), and Representing the first The reliability weighting coefficients for visible light modes and hyperspectral modes at each scale, the and These are all learnable parameters within the network, which are continuously and dynamically updated based on backpropagation of the error gradient during the network training process. This represents the channel dimension of the key vector corresponding to the current k-th scale. This is the scaling factor; This represents the matrix transpose operation.

[0029] Step 2.2.2: For the initial input and Perform aggregation to obtain the first... The characteristic residuals of each sample Subsequently, equation (4) is used to evaluate the two enhanced features and feature residuals obtained in the preceding steps. The features are then concatenated, and the concatenated features are integrated by a convolutional layer (Conv). A channel gate unit dynamically filters out useless information, and finally, the residuals of the original bi-branch features are combined. Adding each element one by one, we get the first... Cross-modal fusion features at various scales : (4) This mechanism achieves deep alignment between spatial details and biochemical features while preventing the loss of original modal information.

[0030] Step 2.3: Multi-scale feature pyramid neck network fuses feature sequences across modalities. Multi-scale aggregation and global logic optimization are performed to obtain the corresponding multi-scale aggregated feature sequence. ;in, Indicates the first The sample at the th Aggregated features at various scales.

[0031] Step 2.3.1: Utilize the top-down path in the neck network of the multi-scale feature pyramid to fuse feature sequences across modalities. Upsampling and feature concatenation are performed to obtain the corresponding results. Top-down augmentation feature sets at various scales ;in, Indicates the first The first scale A top-down enhancement feature.

[0032] when season ; when At that time, for the first Top-down enhancement features at each scale After upsampling, and By splicing them together, we get .

[0033] Step 2.3.2: Utilize the bottom-up path in the multi-scale feature pyramid neck network to enhance the top-down feature sequence. Cross-scale fusion is performed to obtain bottom-up localization feature sequences. ; when season ; when At that time, using convolutional layers to... Bottom-up localization features at various scales After downsampling, and By splicing and merging, the first... Bottom-up localization features at various scales .

[0034] Step 2.3.3, as follows Figure 5 The diagram shows the structure of the multi-head self-attention module C2f_MSA. To overcome the problem of limited receptive field in convolution operations, Equation (5) is used to locate features from the bottom up. Local feature extraction and spatial dimension flattening are performed to obtain the first... One-dimensional patch sequence features at various scales : (5) In equations (5) and (6), This represents the local feature extraction unit in the C2f_MSA module. This indicates a spatial dimension flattening operation.

[0035] The sequence After performing layer normalization (LN), the multi-head self-attention mechanism (MSA) is used to calculate the association weights between each patch, thereby utilizing Equation (6) to analyze the one-dimensional patch sequence features. Global correlation modeling, residual connections, and topology restoration are performed to obtain the first... Final enhanced features at each scale : (6) In Equation (6), Reshape represents the operation of restoring the 3D topological structure; + represents the residual connection operation of element-wise addition. This operation establishes global pixel associations and can use global contextual clues to infer the structure when local features are occluded, thus improving the detection performance in complex tea garden environments.

[0036] Step 2.4, Target Detection Head Pair Process and output. and Prediction results ,include: The Middle Prediction information of the prediction box containing each tea bud ,in, Representing the first The coordinates of the center point of the prediction box containing each tea bud and They represent the first The width and height of the prediction frame containing each tea bud; Representing the Confidence of the prediction frame containing each tea bud; Representing the The predicted category probability of each tea bud within its prediction box.

[0037] Step 2.4.1: Enhance the features at each scale The classification and regression branches of the target detection head are input separately and decoupled convolution is performed to obtain the category response features and bounding box regression features at the corresponding scales.

[0038] Step 2.4.2: Perform mapping and decoding processing on the bounding box regression features and the category response features to obtain the first... In the nth sample Prediction box for each tea bud Confidence level and predicted category probability .

[0039] Step 3: Since tea buds occupy very few pixels in the image, if the traditional Intersection over Union (IoU) is used, even if the detection box undergoes a tiny pixel-level positional shift, the IoU value will drop sharply to 0, causing a cliff-like oscillation of the loss gradient and hindering network convergence. This invention proposes to model the bounding box as a two-dimensional Gaussian distribution and calculate the optimal transmission distance.

[0040] Step 3.1: First, place the actual annotation box. and prediction boxes They are modeled as two-dimensional Gaussian distributions with sample and target labels, respectively. and Subsequently, the second-order Wasserstein distance between the Gaussian distributions corresponding to the ground truth bounding boxes and the predicted bounding boxes is calculated using equation (7). : (7) In equation (7), This represents the mean vector of the Gaussian distribution corresponding to the actual bounding boxes. This represents the mean vector of the Gaussian distribution corresponding to the predicted bounding box. This represents the covariance matrix of the Gaussian distribution corresponding to the true bounding boxes. This represents the covariance matrix of the Gaussian distribution corresponding to the predicted bounding box. This represents the square root of the covariance matrix of the true bounding boxes. The trace of a matrix is ​​represented by the sum of the elements on the main diagonal of the matrix.

[0041] The actual annotation box and prediction boxes The geometric properties are transformed into a Gaussian distribution parameter set. and Subsequently, Equation (8) is used to perform distance measurement and nonlinear mapping on the Gaussian distribution parameter sets corresponding to the ground truth bounding boxes and the predicted bounding boxes, and then combined with the perfect intersection-union ratio for aggregation and summation to obtain the overall scale-aware localization loss of the network. : (8) In equation (8), The total number of samples, For the first The total number of tea bud targets matched in each sample. express and Complete intersection and union ratio; and They respectively represent the following: and The Gaussian distribution parameters derived from the center point coordinates and the width and height geometric attributes; express and The second-order Wasserstein distance between them; This is a preset distance normalization constant; These are the preset balancing hyperparameters.

[0042] Step 3.2, Calculation as well as The class probability deviation between them is used to construct the classification loss. ; Step 3.3, Calculation as well as Calculate the confidence loss based on the confidence deviation between them. ; Step 3.4: Using equation (9), dynamically weight and sum the localization loss, classification loss, and confidence loss obtained in the previous steps to obtain the composite loss function for network parameter optimization. : In equation (9), These are the preset weighting coefficients for localization loss, classification loss, and confidence loss, respectively.

[0043] Step 4: Iteratively train the DMF-YOLO network using optimization algorithms such as gradient descent, and calculate the composite loss function to adjust the network parameters through backpropagation until the composite loss function is reached. The process continues until convergence, thus obtaining a well-trained tea bud detection model.

[0044] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the above-described tea bud detection method, and the processor is configured to execute the program stored in the memory.

[0045] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the above-described tea bud detection method.

Claims

1. A method for detecting tea buds based on dual-modal fusion of RGB and spectral images, characterized in that, Includes the following steps: Step 1: Obtain the original hyperspectral image set in the tea garden environment and perform dimensionality reduction and band selection processing to obtain the hyperspectral image set corresponding to the optimal band. ,in, Indicates the first A hyperspectral image, Indicates the height of the hyperspectral image. Indicates the width of the hyperspectral image. Indicates the optimal number of bands; Will The corresponding RGB image is denoted as ,right All tea bud locations were labeled to obtain a tag set. ,make The Middle The labeling information for each tea bud is recorded as follows: ,in, Indicates the first The coordinates of the center point of the annotation box containing each tea bud and The first The width and height of the label box containing each tea bud. For the first The actual category label of the label box containing each tea bud; Step 2: Construct a DMF-YOLO tea bud detection network based on dual-modal feature fusion, including: a multimodal dual-branch feature extraction network, a dual-dimensional cross-modal feature fusion module, a multi-scale feature aggregation neck network, and a target detection head, and then respectively... and Processing is performed to obtain the prediction results. ; Step 2.1: The multimodal dual-branch feature extraction network consists of an RGB modal backbone feature extraction network and a hyperspectral modal feature extraction network, which are respectively used for... and Downsampling and multi-scale feature extraction are performed to obtain the corresponding results. Visible light image feature sequences at various scales and Hyperspectral biochemical signature sequences at various scales ,in, express The Visible light image features at various scales, express The Hyperspectral biochemical characteristics at various scales; Step 2.2, and The input is processed together in the dual-dimensional cross-modal feature fusion module using dynamic weighting and symmetrical bidirectional cross-attention interaction, and the output is the first... Cross-modal fusion features at various scales ; Step 2.3: The multi-scale feature aggregation neck network fuses multi-scale cross-modal feature sequences. Multi-scale aggregation is performed, resulting in a multi-scale aggregated feature sequence. ,in, Indicates the first Aggregational features at various scales; Step 2.4, Target detection head pair Process and output. and Prediction results ,include: The Middle Prediction information of the prediction box containing each tea bud ,in, Representing the first The coordinates of the center point of the prediction box containing each tea bud and They represent the first The width and height of the prediction frame containing each tea bud; Representing the Confidence of the prediction frame containing each tea bud; Representing the The predicted category probability of the prediction box containing each tea bud; Step 3, based on and Construct a scale-aware localization loss Classification loss With confidence loss Composite loss function composed of dynamic weights ; Step 4: Iteratively train the DMF-YOLO tea bud detection network based on dual-modal feature fusion using an optimization algorithm, and calculate... Adjust network parameters using backpropagation until... The process continues until convergence, resulting in a trained tea bud detection model used for detecting tea bud targets.

2. The tea bud detection method based on dual-modal fusion according to claim 1, characterized in that, Step 2.1 includes: Step 2.1.1: The visible light modal backbone feature extraction network includes: The system consists of several cascaded processing stages, each containing a convolutional layer for performing downsampling and a C2f feature extraction module. when At that time, Enter the first Downsampling is performed in the convolutional layer of the first stage, and the resulting downsampling features are then input into the second stage. Spatial texture feature extraction is performed in the C2f module to obtain the first... Visible light spatial characteristics at various scales ; when At that time, the first Visible light spatial characteristics at various scales Enter the first Processing is carried out in the stage; the result is obtained. Visible light spatial characteristics at various scales ; and thus by Visible light spatial features at different scales constitute a visible light spatial feature sequence. ; Step 2.1.2: The hyperspectral modal feature extraction network includes: The process consists of cascaded processing stages, each containing a convolutional layer and a C2f_SE feature extraction module; any cascaded processing stage... Each C2f_SE feature extraction module consists of a C2f feature extraction unit and an SE channel attention branch connected in series: when ,Will Enter the first Downsampling is performed in the convolutional layer of the first stage, and the resulting downsampling features are then input into the second stage. Spatial spectral features are extracted from the C2f unit of the C2f_SE module, and the output is the first... Intermediate features Then enter the number The process is performed in the SE channel attention branch of the C2f_SE module to obtain the first... Hyperspectral biochemical characteristics at various scales ; when At that time, the first Hyperspectral biochemical characteristics at scale Enter the first In the stage, the first equation is obtained using equation (1). Hyperspectral biochemical characteristics at various scales ; and obtained by A hyperspectral feature sequence composed of hyperspectral features at different scales ; (1) In equation (1), This indicates a global average pooling operation. and These represent the nonlinear mapping operations of the dimensionality-reducing fully connected layer and the dimensionality-upgrading fully connected layer, respectively. This represents a channel-by-channel multiplication operation.

3. The tea bud detection method based on dual-modal fusion according to claim 1, characterized in that, Step 2.2 includes: Step 2.2.1, After being processed by the linear mapping layer, the corresponding result is obtained. Query vector for visible light modes at each scale Key vector Value vector and the Query vector of hyperspectral modes at each scale Key vector Value vector Therefore, by using equations (2) and (3), we can obtain the first... Visible light enhancement features of hyperspectral modal injection at various scales and the Hyperspectral enhancement features of visible light mode injection at various scales : (2) (3) In equations (2) and (3), For the first Reliability weighting coefficients for visible light modes at each scale. For the reliability weighting coefficients of the hyperspectral modes; For the first The channel dimension of the key vector at each scale; T represents the transpose; Step 2.2.2, for and Perform aggregation to obtain the first... Each characteristic residual term Thus, by using equation (4), the first... Cross-modal fusion features at various scales : (4) In equation (4), Concat represents the feature concatenation operation along the channel dimension, Conv represents the convolution operation for feature information integration, Gate represents the channel gating operation for filtering redundant information, and + represents the element-wise addition operation.

4. The tea bud detection method based on dual-modal fusion according to claim 1, characterized in that, Step 2.3 includes: Step 2.3.1: Utilize the top-down path in the neck network of the multi-scale feature pyramid to fuse feature sequences across modalities. Upsampling and feature concatenation are performed to obtain the corresponding results. Top-down augmentation feature sets at various scales ;in, Indicates the first The first scale A top-down enhancement feature; when season ; when At that time, for the first Top-down enhancement features at each scale After upsampling, and By splicing them together, we get ; Step 2.3.2: Utilize the bottom-up path in the multi-scale feature pyramid neck network to enhance the top-down feature sequence. Cross-scale fusion is performed to obtain bottom-up localization feature sequences. ; when season ; when At that time, using convolutional layers to... Bottom-up localization features at various scales After downsampling, and By splicing and merging, the first... Bottom-up localization features at various scales ; Step 2.3.3: Using equations (5) and (6) to... By performing local refinement and global logic optimization, the first... Aggregation features at each scale : (5) (6) In equations (5) and (6), This represents a local feature extraction unit in the C2f_MSA module. This indicates a flattening operation in spatial dimensions. Representative level normalization, Represents a multi-head self-attention mechanism. This represents the operation of restoring the three-dimensional topology. Indicates the first The flattened one-dimensional patch sequence features at each scale, where + indicates an element-wise additive residual join operation.

5. The tea bud detection method based on dual-modal fusion according to claim 1, characterized in that, Step 3 includes: Step 3.1: Construct scale-aware localization loss using equation (7). : (7) In equation (7), The total number of samples, For the first The total number of tea bud targets matched in each sample. express and Complete intersection and union ratio; and They respectively represent the following: and The Gaussian distribution parameters derived from the center point coordinates and the width and height geometric attributes; express and The second-order Wasserstein distance between them; This is a preset distance normalization constant; These are the preset balance hyperparameters; Step 3.2, Calculation as well as The class probability deviation between them is used to construct the classification loss. ; Step 3.3, Calculation as well as Calculate the confidence loss based on the confidence deviation between them. ; Step 3.4: Construct a composite loss function for network parameter optimization using equation (8). : (8) In equation (8), These are the preset weighting coefficients for localization loss, classification loss, and confidence loss, respectively.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the method of any one of claims 1-5, the processor being configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method according to any one of claims 1-5.

Images