A multi-modal fruit perception system, apparatus and storage medium

Abstract

The application discloses a multi-modal fruit perception system, device and storage medium, and belongs to the technical field of computer vision. In view of the problem that the fruit perception model in the prior art has poor perception effect in a weak light environment, the application collects video stream data of fruit plants at multiple angles, constructs a multi-modal fruit image data set, and inputs a multi-modal visual data fusion encoder into a feature extraction pyramid structure of a target detection model to obtain a multi-modal visual data fusion backbone network. Then, the multi-modal visual data fusion backbone network is connected to the head of the target detection model for training, and a trained fruit detection module is obtained. The multi-modal fruit image data is predicted through the fruit detection module, and finally, the prediction result is processed through a fruit perception module to obtain fruit position and category information. The application can realize high-precision detection in a real complex environment, and meets the requirements of edge lightweight deployment and high-precision fruit perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, more particularly, to a multi-modal fruit perception system, device and storage medium. BACKGROUND

[0002] Fruit detection is an important research direction in the field of computer vision, aiming to automatically identify and detect different types of fruits and vegetables through computer algorithms and technologies. In modern agricultural production, tomatoes are a widely planted and consumed vegetable crop. How to improve the picking efficiency is a problem that needs to be solved for large-scale cultivation farms. The existing tomato picking and maturity classification process is carried out by manual operation, which consumes a large amount of labor. This process is not only time-consuming and labor-intensive, but also subjective, lacking a unified selection basis, which affects the quality of agricultural products that rely on tomatoes of different maturity.

[0003] With the rapid development of artificial intelligence technology, it has become possible to use computer vision and machine learning methods to classify fruits of different maturity. For example, Taiheng Zeng et al. published a lightweight tomato real-time detection method based on improved YOLO and mobile deployment in Computers and Electronics in Agriculture, which proposed a lightweight tomato target detection algorithm based on YOLOv5. By replacing the focal layer and backbone network of YOLOv5, combining channel pruning and optimizing the original hyperparameters, the parameter amount is reduced, but the model accuracy is lower than the original model. Guoxu Liu et al. published an anchor-free detector for tomato detection in Frontiers in Plant Science, which introduced a convolution block attention module into the backbone network of Attentive-DLA34, but when the overlap or occlusion area is high, the detection rate will decrease. In addition, picking in rainy weather, picking in weak light environment and other problems are often encountered in actual picking environment. RGB images often have a lot of noise, and existing tomato detection algorithms cannot avoid performance degradation by relying solely on RGB images.

[0004] To solve this problem, current researches have adopted multi-modal data to improve algorithm performance. For example, H. Gan et al. published in Computers and Electronics in Agriculture to detect immature green citrus fruits using color and thermal images, designed a new color-thermal joint probability algorithm, which effectively fused information from color and thermal images, but only ideal thermal images could be collected in the early morning. Jiacheng Rong et al. published in Computers and Electronics in Agriculture to detect and count tomato clusters based on RGB-D fusion improved YOLOv5, proposed an improved YOLOv5 that fused RGB images and depth images to reduce false recognition of background tomatoes, but only tomato clusters with different maturity could be detected, which had low practical value. Shih-Yu Chen et al. published in Computers and Electronics in Agriculture to detect defects in real-time using near-infrared snapshot hyperspectral imaging of green coffee beans, developed a multi-modal real-time coffee bean defect detection algorithm based on deep learning, but hyperspectral imaging had high requirements for the environment.

[0005] In summary, the difficulties of fruit detection in practical application environment in the prior art include: (1) In the dense planting environment, the size of the fruit is not uniform, the overlap of the shelter is serious, and the performance of the fruit detection is reduced due to the interference of background factors. Visible light vision data is easily disturbed by external factors, and it is difficult to ensure the stability of the fruit detection model. (2) Selection of multi-modal data, part of the modal can effectively improve the detection performance but the requirement for collection is also relatively higher, another part of the modal has relatively low collection requirement but the performance improvement is limited. (3) Method to realize high-performance multi-modal fusion, current fruit detection work for multi-modal visual data is divided into input fusion, feature fusion and decision fusion three categories; for different fruit perception tasks, the appropriate fusion method needs to be selected to improve the robustness of the model. (4) On the basis of realizing high-precision fruit perception, meet the demand of edge lightweight deployment; the existing fruit perception model either realizes high-precision detection but needs high-performance equipment, or realizes lightweight model but the precision is not enough. Therefore, how to design an efficient and reasonable feature extraction architecture to extract and fuse input data, and a high-performance lightweight fruit perception model that performs well in real production environment, is a problem that needs to be solved at present. SUMMARY

[0006] 1. Technical problems to be solved

[0007] In view of the problem that the fruit perception model in the prior art has poor perception effect in a weak light environment, the application provides a multi-modal fruit perception system, a device and a storage medium, introduces visible light, depth and near-infrared multi-modal image data, designs a lightweight fruit detection module based on a multi-modal visual data fusion encoder, constructs a multi-modal fruit perception system, realizes real-time perception and classification of fruits, and has the characteristics of edge lightweight deployment and high-precision fruit perception.

[0008] 2. Technical solution

[0009] The object of the application is achieved by the following technical solutions.

[0010] A multi-modal fruit perception system, the system comprising a data acquisition module, a fruit detection module and a fruit perception module;

[0011] The data acquisition module acquires multi-angle fruit plant video stream data and constructs a multi-modal fruit image data set.

[0012] The fruit detection module comprises a multi-modal visual data fusion encoder, a multi-modal visual data fusion backbone network and a target detection model; the multi-modal visual data fusion encoder is connected to the feature extraction pyramid structure of the target detection model to obtain a multi-modal visual data fusion backbone network, and then the multi-modal visual data fusion backbone network is connected to the head of the target detection model for training to obtain a trained fruit detection module; the multi-modal fruit image data is predicted by the fruit detection module.

[0013] The fruit perception module processes the prediction result by the fruit perception module to obtain fruit position and category information.

[0014] Further, a multi-modal perception device is used to take pictures of fruit plants under different light conditions to obtain visible light image data, depth image data and near-infrared image data, which constitute a multi-modal fruit image data set.

[0015] Further, the multi-modal visual data fusion encoder comprises a visible light visual data feature extraction path, a depth and near-infrared visual data feature extraction path, a residual aggregation network module and a cross-domain attention module.

[0016] Further, the calculation function of the cross-domain attention module is:

[0017]

[0018] f2=(C·SiLU(A·SiLU(f1)+B) T +D) T

[0019]

[0020] wherein, f1 represents the first feature map outputted by the cross-domain attention module, f2 represents the second feature map outputted by the cross-domain attention module, f2 h represents the split of the second feature map along the height direction, f2 w represents the split of the second feature map along the width direction, f3 represents the final feature map outputted by the cross-domain attention module, a and b both represent independent learnable vectors, d1, d2, d3, d4 and d5 all represent independent convolution calculation layers, W represents the width of the feature map inputted into the cross-domain attention module, H represents the height of the feature map inputted into the cross-domain attention module, i represents a certain width of the feature map inputted into the cross-domain attention module, j represents a certain height of the feature map inputted into the cross-domain attention module, x represents a certain channel of the feature map inputted into the cross-domain attention module, c represents the number of channels of the feature map inputted into the cross-domain attention module, x c represents the feature map inputted into the cross-domain attention module, h represents a certain column of the feature map inputted into the cross-domain attention module, w represents a certain row of the feature map inputted into the cross-domain attention module, A and B both represent the learnable parameters of the full connection mapping with the same dimension as x c represents the feature map inputted into the cross-domain attention module, C and D both represent the learnable parameters of the full connection mapping with the same dimension as x c T represents the feature map inputted into the cross-domain attention module, T represents the transpose operation on the feature map of the current position.

[0021] Further, the visible light image data is inputted into the visible light visual data feature extraction channel to extract features and obtain a first feature map;

[0022] The depth image data and the near-infrared image data are inputted into the depth and near-infrared visual data feature extraction channel to extract features and obtain a second feature map;

[0023] The first feature map and the second feature map are merged and inputted into the residual aggregation network module to calculate a third feature map;

[0024] The third feature map is inputted into the cross-domain attention module to calculate a final feature map.

[0025] Further, the multi-modal visual data fusion backbone network comprises a multi-modal visual data fusion encoder, a feature extraction pyramid structure of a target detection model and a cross-domain attention module; the feature extraction pyramid structure of the target detection model comprises a first original feature extraction layer, a second original feature extraction layer, a first maximum pooling layer and a second maximum pooling layer in the backbone network of the target detection model.

[0026] Further, after processing the first original feature extraction layer and the second original feature extraction layer of the target detection model, a cross-domain attention module is added after the first original feature extraction layer and the second original feature extraction layer respectively to obtain a first feature extraction layer and a second feature extraction layer; the multi-modal visual data fusion encoder, the second maximum pooling layer, the second feature extraction layer, the first maximum pooling layer and the first feature extraction layer constitute a multi-modal visual data fusion backbone network.

[0027] The final feature map is taken as a large-scale feature map output by the multi-modal visual data fusion backbone network, and the large-scale feature map is sequentially passed through the second maximum pooling layer and the second feature extraction layer, the first maximum pooling layer and the first feature extraction layer to obtain a medium-scale feature map and a small-scale feature map.

[0028] Further, the multi-modal fruit image dataset is divided into training set data and validation set data; the multi-modal visual data fusion backbone network is connected to the head of the target detection model, and the training set data is input for training, and the validation set data is input for evaluation to obtain a fruit perception module.

[0029] A multi-modal fruit perception device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multi-modal fruit perception system.

[0030] A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the multi-modal fruit perception system applied to a data acquisition module, a fruit detection module or a fruit perception module.

[0031] 3. Benefits

[0032] Compared with the prior art, the advantages of the present application are:

[0033] (1) The multi-modal fruit perception system, device and storage medium of the present application can realize excellent detection performance of the multi-modal fruit perception system in various real complex scenes by combining depth image data and near-infrared image data to assist visible light image data for fruit detection.

[0034] (2) The multi-modal fruit perception system, device and storage medium of the present application, in the multi-modal visual data fusion encoder, combine the visible light visual data feature extraction path, the depth and near-infrared visual data feature extraction path and the residual aggregation network module to realize efficient fusion of multi-modal information, and have good anti-multi-modal feature conflict ability.

[0035] (3) The multi-modal fruit perception system, device and storage medium of the present application can capture information of cross-channel, direction perception and position perception, realize more rich feature information expression, and help the system to locate and identify the fruit target more accurately.

[0036] (4) The multi-modal fruit perception system, device and storage medium of the present application design residual aggregation network and cross-domain attention module, effectively reduce the parameter quantity of the fruit detection module, and improve the lightweight demand of the system on the edge device. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The flowchart of the multi-modal fruit perception system of the embodiment of the present application is constructed;

[0038] Figure 2 The first schematic diagram of the fruit plant video stream data collected by the embodiment of the present application is shown in the figure;

[0039] Figure 3 The second schematic diagram of the fruit plant video stream data collected by the embodiment of the present application is shown in the figure;

[0040] Figure 4 The training schematic diagram of the fruit plant video stream data of the embodiment of the present application is shown in the figure;

[0041] Figure 5 The structure schematic diagram of the multi-modal fruit perception system of the embodiment of the present application is shown in the figure;

[0042] Figure 6 The schematic diagram of the multi-modal visual data fusion encoder of the embodiment of the present application is shown in the figure;

[0043] Figure 7 The schematic diagram of the multi-layer perception module of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0044] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0045] EMBODIMENT

[0046] As Figure 1As shown, a multi-modal fruit perception system provided by the embodiment. The multi-modal fruit perception system comprises a data acquisition module, a fruit detection module and a fruit perception module. The data acquisition module is used to acquire multi-angle fruit plant video stream data and construct a multi-modal fruit image dataset; the fruit detection module comprises a multi-modal visual data fusion encoder, a multi-modal visual data fusion backbone network and a target detection model, the multi-modal visual data fusion encoder is connected to the feature extraction pyramid structure of the target detection model to obtain the multi-modal visual data fusion backbone network, and then the multi-modal visual data fusion backbone network is connected to the head of the target detection model for training to obtain the trained fruit detection module, and the multi-modal fruit image data is predicted by the fruit detection module; the fruit perception module processes the prediction result by the fruit perception module to obtain the fruit position and category information.

[0047] Specifically, in the embodiment, first, in the data acquisition module, multi-angle fruit plant video stream data is acquired to construct a multi-modal fruit image dataset. The fruit plants are photographed under different light conditions, including sunny, cloudy, back light, weak light, artificial light source, sodium light source and the like, in the embodiment, the images in the fruit plant video stream data are extracted at an interval of 10 frames to obtain visible light image data, depth image data and near-infrared image data, the visible light image data, the depth image data and the near-infrared image data of the same frame constitute a multi-modal fruit image data sample, and all samples constitute a multi-modal fruit image dataset.

[0048] In the embodiment, the multi-modal perception device is manually held to acquire multi-angle fruit plant video stream data under different distances and different heights. The multi-modal perception device is fixed by a machine to acquire the simulation picture of the fruit picking device, and the multi-modal data space is aligned by using the interface of the multi-modal perception device, and artificial data labeling is performed to construct a multi-view multi-modal fruit image dataset.

[0049] Specifically, the camera with visible light, depth and near-infrared multi-modal perception function is used to acquire multi-angle fruit plant video stream data of different plants under different time periods and different light conditions, and visible light, depth and near-infrared visual data are extracted from the multi-angle fruit plant video stream data. As shown in Figure 2 As shown, a schematic diagram of the shooting position of the perception device on a single plant, an Azure Kinect device is manually held at a distance of about 1 meter from the fruit plant, and different angles and different distances are adopted to shoot along a row of fruit plants. The manual shooting is kept stable to ensure clear picture and to contain a small part of motion blur samples. In addition, as shown in Figure 3As shown, an Azure Kinect device was mounted on a mechanical vehicle using a bracket. It was moved at a constant speed and height, panning along a row of fruit-bearing plants to capture simulated images of actual harvesting. The camera moved repeatedly, acquiring video stream data of the fruit plants from multiple perspectives across multiple rows, constructing a multimodal fruit image dataset. Further, the acquired multi-angle video stream data was traversed, and the visible light image data, depth image data, and near-infrared image data of each frame were separated using ffmpeg. The modal data were then sequentially merged and saved into three folders: RGB, Depth, and IR. It should be noted that using ffmpeg to separate the video stream data is a current technology.

[0050] In this embodiment, the Azure Kinect device is an existing device. The Azure Kinect device has a frame rate of 30 FPS, a resolution of 720P, a depth mode of NFOV_2X2BINNED, an inertial measurement unit of LSM6DSMUS / 1.6kHz, and a field of interest of 90°×59° / 75°×65°.

[0051] It is worth noting that, in order to achieve high-precision fruit detection and ensure the diversity of training data, data with varying characteristics should be collected as much as possible during the data acquisition process. Therefore, the different time periods, lighting conditions, and distances in this embodiment are specifically manifested as follows: based on different sunlight intensities, the collection periods include 7-8 AM, 10-11 AM, 2-3 PM, and 6-7 PM; based on different lighting conditions, the collection lighting conditions include sunny days, cloudy days with direct sunlight, cloudy days with backlight, and nighttime conditions with weak light, artificial light sources, and sodium light sources. Furthermore, in this embodiment, by collecting rich and diverse data, the generalization ability and robustness of the multimodal fruit sensing system can be improved.

[0052] like Figure 4As shown, data cleaning operations are performed on the obtained multi-modal fruit plant image dataset, and repeated and invalid multi-modal fruit image datasets are manually cleaned up. The remaining multi-modal fruit image dataset is relabeled, and spatial alignment is performed using a camera built-in interface. Using the LabelMe tool, the multi-modal fruit image dataset is traversed, and the fruit in each visible light image data is manually labeled with segmentation data and maturity data. Finally, the fruit segmentation information is traversed, the circumscribed rectangle of the fruit segmentation polygon is calculated, and the rectangle is extended upward by 15 pixels and downward, leftward and rightward by 10 pixels respectively to generate fruit target detection labeling information, thereby constructing a multi-modal fruit image dataset. In this embodiment, the target fruit segmentation polygon is uniformly expanded outward to generate the target detection labeling box, which improves the accuracy and uniformity of the labeling data. It should be noted that, in this embodiment, the LabelMe tool is used to traverse the image and label the fruit segmentation data, which is prior art.

[0053] It is worth noting that, in this embodiment, if only visible light image data is used to construct a fruit perception system, it is easy to be affected by environmental light and generate noise, thereby causing the performance of the fruit perception system to decline. Therefore, in this embodiment, the depth and near-infrared image data are combined with the visible light image data to assist in fruit detection, which can achieve excellent detection performance of the fruit perception system in various real complex scenes.

[0054] Further, in the fruit detection module, a multi-modal visual data fusion encoder, a multi-modal visual data fusion backbone network and a target detection model are included. As shown in Figure 5 The multi-modal visual data fusion encoder is connected to the feature extraction pyramid structure of the target detection model to obtain a multi-modal visual data fusion backbone network, and then the multi-modal visual data fusion backbone network is connected to the head of the target detection model for training to obtain a trained fruit detection module.

[0055] Specifically, as shown in Figure 6 The multi-modal visual data fusion encoder includes a visible light visual data feature extraction path, a depth and near-infrared visual data feature extraction path, a residual aggregation network module and a cross-domain attention module.

[0056] In this embodiment, the calculation function of the cross-domain attention module is:

[0057]

[0058] f2=(C·SiLU(A·SiLU(f1)+B) T +D) T

[0059]

[0060] wherein, f1 represents the first feature map output by the cross-domain attention module, f2 represents the second feature map output by the cross-domain attention module, f2 h represents the splitting of the second feature map along the height direction, f2 w represents the splitting of the second feature map along the width direction, f3 represents the final feature map output by the cross-domain attention module, and both α and β represent independent learnable vectors, δ1, δ2, δ3, δ4 and δ5 each represent an independent convolution calculation layer, W represents the width of the feature map input into the cross-domain attention module, H represents the height of the feature map input into the cross-domain attention module, i represents a certain width of the feature map input into the cross-domain attention module, j represents a certain height of the feature map input into the cross-domain attention module, x represents a certain channel of the feature map input into the cross-domain attention module, and c represents the number of channels of the feature map input into the cross-domain attention module. c represents the feature map input into the cross-domain attention module, h represents a certain column of the feature map input into the cross-domain attention module, w represents a certain row of the feature map input into the cross-domain attention module, and both A and B represent learnable parameters of a fully connected mapping with the same dimension as x c . c T .

[0061] Further, for the visible light vision data feature extraction path, comprising:

[0062] S1: using a common convolution, a batch normalization (BN) layer and a SiLU (Sigmoid Linear Unit, SiLU) activation function to construct a Conv module;

[0063] S2: replacing the Conv module of the visible light vision data feature extraction path in the backbone network of the target detection model (YOLO model) with the constructed Conv module;

[0064] S3: inputting visible light image data into the visible light vision data feature extraction path to obtain a first feature map C0.

[0065] It should be noted that in the embodiment, the target detection model (YOLO model) used is a YOLOv7-tiny model. In addition, in the prior art, the Conv module includes ordinary convolution, batch normalization layer and LeakyReLU (LeakyRectified Linear Unit, LeakyReLU) activation function, and in the constructed Conv module in the embodiment, the SiLU activation function is a combination of Sigmoid and LeakyReLU, which has the characteristics of no upper bound, lower bound, smoothness and non-monotonicity, and thus the effect of the SiLU activation function on the deep model is better than that of the LeakyReLU activation function. Therefore, compared with the original LeakyReLU activation function, the use of the SiLU activation function can improve the comprehensive performance of the model.

[0066] As shown in Figure 6 and Figure 7 For the depth and near-infrared vision data feature extraction path, it includes:

[0067] S1: split the depth image data and the near-infrared image data into 4x4 size matrices, arrange and flatten them in the order of left to right and top to bottom to obtain a preprocessed feature map M1;

[0068] S2: use a cross-block residual layer and a cross-channel residual layer to construct a fully connected feature extraction residual layer;

[0069] S3: use three fully connected feature extraction residual layers to construct a multi-layer perceptron module, input the preprocessed feature map M1, and calculate to obtain a second feature map M0 output by the multi-layer perceptron module.

[0070] In the embodiment, the depth and near-infrared vision data feature extraction path is composed of a Conv module and a multi-layer perceptron module. The Conv module is used to split the input multi-modal fruit image to reduce the calculation complexity of the multi-layer perceptron module. For the multi-layer perceptron module, a fully connected feature extraction residual layer is constructed through a cross-block residual layer and a cross-channel residual layer, and then a multi-layer perceptron module is constructed through three fully connected feature extraction residual layers. Therefore, the multi-layer perceptron module constructed can enable the depth and near-infrared vision data feature extraction path to quickly and effectively extract the global features of the depth and near-infrared data.

[0071] For the residual aggregation network module, it includes:

[0072] S1: input the first feature map C0 and the second feature map M0 output by the multi-layer perceptron module and the visible light vision data feature extraction path into the constructed Conv module, and calculate to obtain a feature map E1;

[0073] S2: input the sum of the obtained feature map E1 and the first feature map C0 using the constructed Conv module, and calculate to obtain a feature map E2;

[0074] S3: repeat the above steps to calculate to obtain feature maps E3, E4, …, E7, and E8, wherein E4 is obtained by adding E2 and E3 and then processing the result using the Conv module, E6 is obtained by adding E4 and E5 and then processing the result using the Conv module, and E8 is obtained by adding E6 and E7 and then processing the result using the Conv module.

[0075] S4: input the obtained feature maps E1, E2, …, E7, and E8 using the constructed Conv module, and construct a residual aggregation network module, wherein the residual aggregation network module outputs a third feature map E0.

[0076] In this embodiment, the residual aggregation network module is composed of eight Conv modules with the same number of feature processing channels in cascade and one Conv module with eight times the number of feature processing channels. The feature maps (E1, E2, …, E7, and E8) output by the eight Conv modules in cascade are merged and input into the Conv module with eight times the number of feature processing channels to obtain the third feature map E0 output by the residual aggregation network module. It should be noted that the Conv modules in the eight Conv modules with the same number of feature processing channels in cascade and the one Conv module with eight times the number of feature processing channels in this embodiment are all the constructed Conv modules. Thus, in this embodiment, the use of the eight Conv modules with the same number of feature processing channels in cascade and the one Conv module with eight times the number of feature processing channels makes the aggregation effect of the residual aggregation network module more superior.

[0077] Thus, the visible light image data is input into the visible light visual data feature extraction path for feature extraction to obtain the first feature map C0; the depth image data and the near-infrared image data are input into the depth and near-infrared visual data feature extraction path for feature extraction to obtain the second feature map M0; the first feature map C0 and the second feature map M0 are merged and input into the residual aggregation network module to calculate to obtain the third feature map E0; and the third feature map E0 is input into the cross-domain attention module to obtain the final feature map X0.

[0078] Further, for the multi-modal visual data fusion backbone network, it comprises a multi-modal visual data fusion encoder, a cross-domain attention module, and a feature extraction pyramid structure of a target detection model. The feature extraction pyramid structure of the target detection model comprises a first original feature extraction layer F0, a second original feature extraction layer F1, a first maximum pooling layer MP1, and a second maximum pooling layer MP2 in the backbone network of the target detection model.

[0079] Therefore, after processing the first original feature extraction layer F0 and the second original feature extraction layer F1 of the target detection model, a cross-domain attention module is added after the first original feature extraction layer F0 and the second original feature extraction layer F1, respectively, to obtain the first feature extraction layer P0 and the second feature extraction layer P1. It should be noted that the processing of the first original feature extraction layer F0 and the second original feature extraction layer F1 means replacing the Conv module in the first original feature extraction layer F0 and the second original feature extraction layer F1 with the constructed Conv module. For the obtained first feature extraction layer P0 and second feature extraction layer P1, the information of cross-channel, direction perception and position perception can be captured, the problem of overfitting of the multi-layer perception module in the multi-modal visual data fusion encoder can be inhibited, and more rich feature information expression can be realized. Therefore, the multi-modal visual data fusion encoder, the second maximum pooling layer MP2, the second feature extraction layer P1, the first maximum pooling layer MP1 and the first feature extraction layer P0 constitute a multi-modal visual data fusion backbone network. In this embodiment, the multi-modal visual data fusion backbone network can capture cross-channel, direction perception and position perception information, realize more rich feature information expression, and help the multi-modal fruit perception system to locate and identify fruit targets more accurately.

[0080] In this embodiment, the cross-domain attention module in the multi-modal visual data fusion encoder is taken as the bottom layer P2 of the feature extraction pyramid structure of the target detection model, the final feature map X0 is taken as the large-scale feature map N2 output by the multi-modal visual data fusion backbone network, and the large-scale feature map N2 is sequentially passed through the second maximum pooling layer MP2 and the second feature extraction layer P1, the first maximum pooling layer MP1 and the first feature extraction layer P0 to obtain the medium-scale feature map N1 and the small-scale feature map N0. Therefore, the large-scale feature map N2, the medium-scale feature map N1 and the small-scale feature map N0 are the outputs of the multi-modal visual data fusion backbone network.

[0081] Further, in the fruit detection module, the fruit perception module is trained using the multi-modal fruit image dataset. Specifically, the multi-modal visual data fusion backbone network is connected to the head of the target detection model (YOLO model) and input training set data for training. It should be noted that the Conv module in the head of the target detection model (YOLO model) is replaced with the constructed Conv module, thereby obtaining the fruit detection module. In this embodiment, the multi-modal fruit image dataset is divided into training set data and validation set data, and the fruit perception module is trained and verified using the training set data and the validation set data.

[0082] In this embodiment, three loss functions are used to calculate the training loss, and the calculation formula is:

[0083]

[0084] wherein, Loss Total denotes the total loss function, Loss box denotes the bounding box loss, i.e. the CIoU loss function, h input denotes the height of the input fruit plant image, w input denotes the width of the input fruit plant image, Loss conf denotes the confidence loss, i.e. the BCEWithLogitsLoss loss function, classes denotes the number of classes, Loss cls denotes the classification loss, i.e. another BCEWithLogitsLoss loss function.

[0085] It should be noted that during the training process, the threshold value a is set to 0.09, and during the training process, the loss value change of the fruit perception module after each iteration of training is recorded. If the loss value continues to be lower than the threshold value a, it proves that the fruit perception module has reached an ideal state, and if the loss value drops to a lower value and then starts to rise, it proves that the fruit perception module has overfitting, and after fine-tuning the parameters, it is retrained.

[0086] In this embodiment, the average precision (Average Precision) is calculated by the interpolation method to obtain the fruit perception module with the highest average precision. The average precision is the area under the P-R curve, which is used to measure the judgment standard between the accuracy (Precision) and the recall (Recall). In this embodiment, the calculation formula of the average precision is:

[0087]

[0088] wherein, AP denotes the average precision, p denotes the accuracy, and r denotes the recall. In this embodiment, the average precision is used as an evaluation index, the multi-modal fruit image dataset is input, the performance of the fruit perception module is evaluated, and the threshold value b is set. If the average precision of the multi-modal fruit image dataset is less than b, the key hyperparameters are optimized, the data is re-input for training, and when the average precision of the multi-modal fruit image dataset is greater than b, the final fruit perception module is obtained. The key hyperparameters include the selection of the optimizer, the learning rate adjustment function, the number of model training iterations, the initial learning rate, the data enhancement ratio, the momentum factor, and the input image size. In this embodiment, the optimizer is set to the Adam optimizer, the learning rate adjustment function is set to the cosine annealing function, the number of model training iterations is set to 200 times, the initial learning rate is set to 0.01, the data enhancement ratio is set to 0.9, the momentum factor is set to 0.937, and the input image size is set to 640 pixels x 640 pixels.

[0089] During the test, the average accuracy AP of the fruit perception module on the multi-modal fruit image dataset is recorded under the condition that the intersection over union IoU is set to 0.5. The value of the threshold β is determined by the detection performance of the existing target detection model. In this embodiment, it is known by calculation that the average accuracy of the YOLOv5s model is 0.9627, the average accuracy of the YOLOv7-tiny model is 0.9573, the average accuracy of the YOLOX-tiny model is 0.9688, and the average accuracy of the YOLOv8s model is 0.9716 using the same data. Therefore, the threshold β is set to 0.95, and if the average accuracy exceeds the threshold β under the condition that the intersection over union IoU is set to 0.5, the accuracy of the fruit detection module meets the requirements, that is, an ideal fruit perception module is obtained. If the average accuracy does not reach the threshold β under the condition that the intersection over union IoU is set to 0.5, the accuracy of the fruit perception module does not meet the requirements, and the key hyperparameters need to be reconfigured and retrained until the average accuracy exceeds the threshold β under the condition that the intersection over union IoU is set to 0.5.

[0090] Therefore, in this embodiment, the multi-modal fruit image data is input into the fruit detection module to obtain a prediction result, and the prediction result is processed in the fruit perception module to finally obtain the spatial position and category information of the target fruit. In this embodiment, the fruit perception module decodes the prediction result into the spatial position and category information of the fruit. It should be noted that in this embodiment, the processing of the prediction result by the fruit perception module can be calculated by the prior art.

[0091] Therefore, the multi-modal fruit perception system provided in this embodiment overcomes the problem of significant performance degradation of the traditional detection method in real environments such as fruit overlap, branch and leaf shielding, and weak light environment, solves the problem of edge deployment relying on high-precision lightweight algorithms, helps to improve the accuracy and stability of machine picking, has the characteristics of strong generalization ability and high detection accuracy, and provides a feasible scheme for the detection and classification of other crops.

[0092] In addition, the embodiment further provides a multi-modal fruit perception device. The multi-modal fruit perception device is a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the steps of the multi-modal fruit perception system according to the embodiment when executing the program. The computer device can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including a single server or a server cluster composed of multiple servers), etc. that can execute the program. The computer device according to the embodiment at least includes but is not limited to the memory and the processor that can be connected to each other through a system bus. The memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. The memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device, or an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the memory can include both the internal storage unit and the external storage device of the computer device. In the embodiment, the memory is usually used to store an operating system and various application software installed on the computer device, etc. In addition, the memory can also be used to temporarily store various data that have been output or will be output. The processor in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is usually used to control the overall operation of the computer device, and in the embodiment, the processor is used to execute program codes or process data stored in the memory.

[0093] The embodiment further provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the multi-modal fruit perception system applied to the data acquisition module, the fruit detection module, or the fruit perception module.

[0094] The above description of the application and its embodiments is illustrative and not restrictive, and the application can be practiced in other specific forms without departing from the spirit or essential character thereof. The drawings described herein are only one of the many embodiments of the application and are not limiting, and any reference signs in the claims should not be construed as limiting the claims to the figure in which the reference signs are used. Therefore, if a person skilled in the art is inspired by the disclosure, and without departing from the spirit of the application, he can design similar structural forms and embodiments without creativity, which should also be within the scope of protection of the patent. In addition, the word "comprising" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" such elements. The multiple elements stated in the product claims can also be implemented by one element through software or hardware. The words "first", "second" and the like are used to indicate names, and not to indicate any particular order.

Claims

1. A multimodal fruit sensing system, characterized in that, The system includes a data acquisition module, a fruit detection module, and a fruit sensing module; The data acquisition module acquires video stream data of fruit plants from multiple angles to construct a multimodal fruit image dataset; The fruit detection module includes a multimodal visual data fusion encoder, a multimodal visual data fusion backbone network, and a target detection model. The multimodal visual data fusion encoder is connected to the feature extraction pyramid structure of the target detection model to obtain the multimodal visual data fusion backbone network. The multimodal visual data fusion backbone network is then connected to the head of the target detection model and trained to obtain a trained fruit detection module. Multimodal fruit image data is then used to predict the results through the fruit detection module. The fruit sensing module processes the prediction results to obtain fruit location and category information; The fruit plants were photographed under different lighting conditions using a multimodal sensing device to obtain visible light image data, depth image data and near-infrared image data. The visible light image data, depth image data and near-infrared image data constituted a multimodal fruit image dataset. The multimodal visual data fusion encoder includes a visible light visual data feature extraction path, a depth and near-infrared visual data feature extraction path, a residual aggregation network module, and a cross-domain attention module; The calculation function for the cross-domain attention module is: in, f 1 represents the first feature map output by the cross-domain attention module. f 2 represents the second feature map output by the cross-domain attention module. f 2 h This represents the splitting of the second feature map along the height direction. f 2 w This represents the splitting of the second feature map along the width direction. f 3 represents the final feature map output by the cross-domain attention module. α and β Each represents an independent learnable vector. δ 1. δ 2. δ 3. δ 4 and δ 5 represents an independent convolutional computation layer. W This represents the width of the feature map input to the cross-domain attention module. H This represents the height of the feature map input to the cross-domain attention module. i This represents a certain width of the feature map input to the cross-domain attention module. j This represents a certain height of the feature map input to the cross-domain attention module. x This represents a channel of the feature map input to the cross-domain attention module. c This represents the number of channels in the feature map input to the cross-domain attention module. x c This represents the feature map of the input cross-domain attention module. h This represents a column of the feature map input to the cross-domain attention module. w This represents a row of the feature map input to the cross-domain attention module. A , B All indicate that they have the same x c Learning parameters for fully connected mappings of the same dimension C , D All indicate that they have the same x c T Learning parameters for fully connected mappings of the same dimension T This indicates that the feature map at the current position is transposed.

2. The multimodal fruit sensing system according to claim 1, characterized in that, The visible light image data is input into the visible light visual data feature extraction path to extract features and obtain the first feature map; The depth image data and near-infrared image data are input into the depth and near-infrared visual data feature extraction pathway to extract features and obtain the second feature map. The first and second feature maps are merged and then input into the residual aggregation network module to calculate the third feature map; The third feature map is input into the cross-domain attention module to calculate the final feature map.

3. The multimodal fruit sensing system according to claim 2, characterized in that, The multimodal visual data fusion backbone network includes a multimodal visual data fusion encoder, a feature extraction pyramid structure of the object detection model, and a cross-domain attention module; the feature extraction pyramid structure of the object detection model includes a first original feature extraction layer, a second original feature extraction layer, a first max pooling layer, and a second max pooling layer in the backbone network of the object detection model.

4. The multimodal fruit sensing system according to claim 3, characterized in that, After processing the first and second original feature extraction layers of the object detection model, a cross-domain attention module is added after the first and second original feature extraction layers respectively to obtain the first and second feature extraction layers; the multimodal visual data fusion encoder, the second max pooling layer, the second feature extraction layer, the first max pooling layer, and the first feature extraction layer constitute the multimodal visual data fusion backbone network; The final feature map is used as the large-scale feature map output by the multimodal visual data fusion backbone network. The large-scale feature map is then passed sequentially through the second max pooling layer, the second feature extraction layer, the first max pooling layer, and the first feature extraction layer to obtain the medium-scale feature map and the small-scale feature map.

5. A multimodal fruit sensing system according to claim 4, characterized in that, The multimodal fruit image dataset is divided into training set data and validation set data. The multimodal visual data is fused into a backbone network and then fed into the head of the target detection model. The training set data is then used for training, and the validation set data is used for evaluation to obtain the fruit perception module.

6. A multimodal fruit sensing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal fruit sensing system as described in any one of claims 1-5.

7. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal fruit sensing system according to any one of claims 1-5, applied to a data acquisition module, a fruit detection module, or a fruit sensing module.

Images