Real-time fruit detection method based on RGB-D image fusion

Abstract

The invention provides a real-time fruit detection method based on RGB-D image fusion, and the method achieves the precise and real-time detection of fruits in an orchard through the fusion of the color information of an RGB image and the space information of a depth image. According to the method, a lightweight feature extraction network is designed, the network refers to a deep learning model in the field of face detection, and a Network In Network (NIN) structure is combined, so that the number of parameters is effectively reduced, and the calculation efficiency is improved. Besides, the invention also introduces a feature fusion module based on an attention mechanism, and the module can adaptively focus on a key region in the RGB-D data and efficiently fuse the features of the RGB image and the depth image, thereby effectively reducing the noise interference in the depth image and improving the accuracy of feature fusion. And finally, a prediction module with a pyramid structure is adopted, and the module can receive the fused features and perform multi-scale prediction so as to output the position information and confidence of the fruits, and accurate detection of the fruits is realized. According to the method, the accuracy and robustness of fruit detection are improved, the calculation complexity is reduced, real-time detection is realized, and powerful technical support is provided for automatic management and intelligent picking of an orchard.

Description

Technical Field

[0001] The present invention belongs to the field of software engineering and agricultural automation technology. The method mainly focuses on agricultural intelligent technology and is specifically applied to fruit detection and positioning, aiming to improve the accuracy and real-time performance of automated fruit picking and yield estimation. Background Art

[0002] The continuous increase in fruit planting area and output has exacerbated the problems of high labor costs, labor shortages and low efficiency in agricultural production. The rapid development of artificial intelligence has promoted the development of agricultural robot picking technology, among which fruit image recognition plays an important role in fruit robot picking technology. The recognition accuracy represents whether the fruit can be picked accurately. Therefore, agricultural mechanization and intelligence are the future development trends of agriculture, and more accurate automatic fruit detection and positioning will have a positive impact on mechanized harvesting and yield estimation.

[0003] In object detection tasks, deep learning methods based on a single input, especially those based on RGB images, have become common algorithms for fruit detection. However, such algorithms can produce false detections when faced with complex tasks that require multiple types of information, such as overlapping apple recognition, which requires depth information but only has RGB information. Multimodal deep learning methods can make up for the shortcomings of single-input algorithms and bring better detection results by combining various modal information with complementarity and consistency.

[0004] In the field of computer vision, using multimodal data to improve model performance has become an effective way. For example, a weed detection method based on RGB-D multimodal information fusion is used to accurately detect weeds in wheat fields in natural environments. A classification model for litchi fruit recognition was trained by using redundant image background information and then inputting RGB features into the network. An improved method based on multimodal multi-scale Faster R-CNN can stably detect smaller fruits.

[0005] Great progress has been made in multimodal fusion fruit detection, but the goal of fruit detection is to be deployed on agricultural machinery for real-time automatic yield estimation or picking, which requires the fruit detection network deployed on the hardware to have as few parameters as possible. Therefore, we propose a real-time fruit detection method based on RGB-D image fusion.

[0006] In summary, the real-time fruit detection method based on RGB-D image fusion reduces the requirements on hardware devices to achieve real-time rate and provides smooth accuracy and inference speed at the same time. Summary of the invention

[0007] This paper proposes an innovative real-time fruit detection method based on RGB-D image fusion, aiming to solve the problems of low accuracy, poor robustness and high computing resource consumption faced by traditional fruit detection in complex orchard environments. This method integrates the color information of RGB images and the spatial information of depth images, and realizes accurate and efficient detection of fruits through deep learning and multimodal data fusion technology.

[0008] First, the present invention designs a multimodal fusion system framework including a feature extraction module, a fusion module and a prediction module. The feature extraction module is one of the core components of the system, which can extract rich multi-scale features from RGB images and depth images respectively. These features not only contain visual information such as the color and texture of the fruit, but also geometric information such as the position and shape of the fruit in space, providing a solid foundation for subsequent feature fusion and prediction.

[0009] In the feature fusion module, the present invention introduces an innovative feature fusion strategy based on the attention mechanism. Due to the differences in information representation between RGB images and depth images, direct feature fusion may lead to information redundancy and interference. Therefore, the present invention adopts the attention mechanism to dynamically adjust the weights of different modal features, so that the network can adaptively select the feature information that contributes most to the detection task. This fusion strategy not only improves the accuracy of detection, but also enhances the model's adaptability to complex environments.

[0010] The prediction module adopts a pyramid structure design to better detect fruits of different sizes. The module constructs multiple feature maps of different scales and fuses features of different scales using strategies such as upsampling and channel adjustment, thus forming a multi-scale feature map with rich contextual information. During the prediction process, the module can make full use of the information of these feature maps to achieve accurate positioning and classification of fruits.

[0011] In addition, in order to reduce computing resource consumption and improve detection speed, the present invention adopts a lightweight network structure in the feature extraction module. This network structure not only reduces the number of network parameters, but also improves computing efficiency, so that the system can achieve real-time detection under limited computing resources.

[0012] In summary, the present invention proposes a real-time fruit detection method based on RGB-D image fusion, which realizes accurate and efficient detection of fruits by designing a multimodal fusion system framework, adopting an innovative feature fusion strategy, constructing a pyramid-structured prediction module, and using a lightweight network structure. Compared with existing fruit detection methods, the present invention has higher accuracy and robustness in complex orchard environments, while reducing computing resource consumption, providing strong technical support for agricultural automation tasks such as fruit picking and yield estimation.

[0013] The present invention is characterized in that:

[0014] 1. This paper designs a lightweight feature extraction network, which refers to the deep learning model in the field of face detection and combines the Network In Network (NIN) structure to reduce the number of parameters. This module can extract multi-scale features from RGB images and depth images respectively.

[0015] 2. To address the noise problem in depth images, the present invention adopts a feature fusion module based on the attention mechanism. This module can fuse the features of RGB images and depth images according to the key areas in the adaptively focused RGB-D data to reduce the interference of noise and improve the accuracy of feature fusion.

[0016] 3. The present invention adopts a prediction module with a pyramid structure, which can receive the fused features and the features of the RGB image to perform target prediction. The module can output the location information and confidence of the fruit to achieve accurate detection of the fruit. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework of the present invention.

[0018] Figure 2 Network implementation for feature extraction.

[0019] Figure 3 The network framework for the attention fusion module.

[0020] Figure 4 It is the network framework of the prediction module. DETAILED DESCRIPTION

[0021] The algorithm proposed in this paper is a real-time fruit detection algorithm based on RGB-D image fusion. The algorithm achieves accurate detection of fruits by fusing the color information of RGB images and the spatial information of depth images. The core of the algorithm is to design a multimodal fusion system including feature extraction, feature fusion and prediction modules, and adopt a lightweight network structure and a pyramid structure prediction module to reduce the computational complexity and achieve real-time detection.

[0022] The algorithm proposed in this paper is a real-time fruit detection algorithm based on RGB-D image fusion. The algorithm achieves accurate detection of fruits by fusing the color information of RGB images and the spatial information of depth images. The core of the algorithm is to design a multimodal fusion system including feature extraction, feature fusion and prediction modules, and adopt a lightweight network structure and a pyramid structure prediction module to reduce the computational complexity and achieve real-time detection.

[0023] The feature extraction module is one of the key parts of the algorithm, which is responsible for extracting multi-scale features from RGB images and depth images. These features contain key information such as the color, texture, and shape of the fruit, providing a basis for subsequent feature fusion and prediction.

[0024] In the feature extraction process, the algorithm uses a lightweight network structure Network In Network (NIN) to reduce the number of network parameters and improve computational efficiency. At the same time, this method also uses a multi-scale feature extraction strategy to capture feature information of different scales through convolution kernels of different scales to improve the expression ability of features.

[0025] The feature fusion module is another key part of the algorithm, which is responsible for fusing the features of RGB images and depth images. Due to the differences in information representation between RGB images and depth images, direct feature fusion may lead to information redundancy and interference. Therefore, this method introduces an attention mechanism to dynamically adjust the weights of different modal features.

[0026] Specifically, the algorithm first calculates the attention weight of each feature channel, which reflects the importance of the feature channel to the detection task. Then, the features are weighted and summed according to the weight to obtain the fused features. In this way, the algorithm can adaptively select the feature information that contributes most to the detection task and improve the accuracy of detection.

[0027] The prediction module is the output part of the algorithm, which is responsible for detecting fruits based on the fused features. The prediction module adopts a pyramid structure design, which captures the information of fruits of different sizes by constructing multiple feature maps of different scales.

[0028] During the prediction process, the algorithm uses convolutional layers to predict multi-scale feature maps to obtain the location information and confidence of the fruit. In order to improve the accuracy and stability of detection, the algorithm also uses post-processing techniques such as non-maximum suppression (NMS) to remove redundant detection frames.

[0029] In the model training phase, the algorithm uses the collected data set to jointly train the feature extraction network, fusion module, and prediction module. During the training process, the algorithm uses the cross entropy loss function to measure the difference between the predicted results and the true labels, and updates the network parameters through the back propagation algorithm. In order to improve the generalization ability of the model, this method also uses data enhancement technology to enrich the training samples. At the same time, the regularization method is introduced to prevent the model from overfitting. Through continuous training and optimization, the algorithm can gradually converge to the optimal solution and achieve accurate detection of fruits.

[0030] This method provides powerful technical support for automated management and intelligent picking in orchards, and improves the production efficiency and economic benefits of orchards.

Claims

1. A real-time fruit detection method based on RGB-D image fusion, which achieves accurate and real-time detection of fruits in an orchard by fusing the color information of RGB images with the spatial information of depth images. The method comprises: 1) A lightweight feature extraction network is designed. This network refers to the deep learning model in the field of face detection and combines the Network In Network (NIN) structure to extract multi-scale features from RGB images and depth images respectively; 2) The extracted multi-scale features are fused using a feature fusion module based on the attention mechanism to reduce noise interference in the depth image and improve the accuracy of feature fusion; 3) The pyramid-structured prediction module receives the fused features, performs multi-scale prediction, outputs the location information and confidence of the fruit, and realizes real-time fruit detection.

2. The design of the lightweight feature extraction network according to claim 1 is characterized in that: 1) Use the NIN structure to reduce the number of network parameters and improve computational efficiency; 2) Extract color and texture features from RGB images, and extract spatial shape features from depth images.

3. The feature fusion module based on the attention mechanism according to claim 1, characterized in that: 1) Identify and focus on key areas in RGB-D data, such as the edges and textures of fruits; 2) Adaptively fuse the features of RGB images and depth images according to key areas to enhance feature representation capabilities.

4. The feature fusion module based on the attention mechanism according to claim 1, characterized in that: 1) Construct multiple feature maps of different scales to capture information about fruits of different sizes; 2) Use the convolutional layer to predict the multi-scale feature map, output the location information and confidence of the fruit, and use post-processing techniques such as non-maximum suppression (NMS) to remove redundant detection frames and improve detection accuracy.

Images