Apple fruit detection method in complex scene based on multi-modal fusion technology

Abstract

The invention discloses an apple fruit detection method in a complex scene based on a multi-modal fusion technology. The method comprises the following steps: 1, constructing a data set; 2, carrying out feature fusion on the visible light image and the infrared light image; 3, the visible light image features obtained through fusion are masked and fused with visual features; 4, fusing the fused visual features with the text features; 5, filtering the visible light image features; 6, inputting the visible light image features and the text features into word-region similarity calculation; and 7, selecting an optimal target frame in combination with the Soft-NMS and the Focal-EIOU. According to the method, feature fusion is carried out on three modes of an infrared light image, a visible light image and a text, and complementary advantages of different modes are comprehensively considered. And soft non-maximum suppression (Soft-NMS) and Focal-EIOU are utilized to select an optimal target frame, so that the detection precision of the apple fruits under complex conditions is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and mainly relates to a method for detecting apple fruits in complex scenes based on multimodal fusion technology. Background Art

[0002] In recent years, artificial intelligence technologies, particularly deep learning, have made significant progress in the field of computer vision. Object detection, a key research area, has been widely applied in agriculture, including automated fruit picking, yield assessment, and quality inspection. However, one of the main challenges facing object detection in orchards is the occlusion of fruit by other fruits, or by fruit from background objects such as leaves and branches. This occlusion not only affects the precise positioning of the target but also disrupts the extraction of the fruit's appearance features, resulting in reduced detection performance.

[0003] Multimodal fusion technology can significantly improve the performance of target detection models by combining and analyzing information from different sources (such as visual modalities and textual modalities). Specifically, in agricultural target detection, the combination of textual modalities (such as natural language queries based on fruit descriptions) and visual modalities (such as fruit image features) can provide richer contextual information, giving the model a more comprehensive understanding of target features. By introducing multimodal information, the model can better adapt to complex relationships in occluded scenes, identify partially occluded fruits, and improve the robustness and accuracy of detection.

[0004] Traditional object detection methods, such as detectors based on classic machine learning (e.g., Haar features + Adaboost, HOG + SVM) and early deep learning models (e.g., the R-CNN series, the YOLO series, and SSD), typically rely solely on a single visual modality for object detection. In orchard scenarios, these methods struggle to extract complete target features from a single visual input when fruit is obscured by leaves or branches, resulting in a significant drop in detection performance. Furthermore, these methods lack effective fusion of multimodal data and are unable to fully leverage the semantic prior knowledge provided by text modalities. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes an apple fruit detection method in complex scenes based on multimodal fusion technology, in order to fully utilize the complementary advantages of different modalities and realize accurate recognition of apple fruits in occluded scenes, thereby effectively improving the robustness and accuracy of detection.

[0006] In order to solve the technical problem, the present invention adopts the following technical solution:

[0007] The present invention is characterized in that the method for detecting apple fruit in complex scenes based on multimodal fusion technology is performed according to the following steps:

[0008] Step 1: Build the dataset:

[0009] Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset in, represents the i-th visible light image of the c-th apple, represents the i-th infrared image of apples in the c-th category, C represents the number of categories, and I represents the total number of images of each category of apples;

[0010] Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. in, express , and Among them, a c,i express The ath apple fruit, A c,i for The total number of apple fruits, express The ath apple fruit a c,i The true label, express The ath apple fruit a c,i The label box, and express The center point coordinates, express width and height;

[0011] Step 2: Set the visible light image of apples of type c Input into the GLIP model for extraction to obtain the visible light visual information of apple type c Thus we get the visible light visual information set of apple in, represents the jth visual in the set of visible light images of apples of category c, and J represents the total number of visible light images in each category of apples;

[0012] Step 3: Construct a text prompt and use the BERT pre-trained language model to embed the text prompt to generate a text vector T = {t1,…,t c ,…,t C}, where t c The text representing apples of type c;

[0013] Step 4: Mask the text of any type of apple in T to obtain the masked text information T Mask ={t1,…,t c ,t mask ,…,t C}, where t mask Text representing a certain type of apple that is masked;

[0014] Step 5: Build an apple fruit detection network and perform RGB 、Img Inf and T Mask Process and obtain the candidate bounding box set and its matching score set in, represents the i-th visible light image of the c-th apple The bth candidate box of the ath apple fruit; express Matching score;

[0015] Step 6: Construct the total loss function of the apple fruit detection network

[0016] Step 7: Use the gradient descent method to train the apple fruit detection network and calculate To update the network parameters, when the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained;

[0017] Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score in, represents the bth apple of the cth apple in the target visible light image and the i-th apple in the a-th apple in the visible light image. * prediction boxes, express The matching score, B * is the total number of predicted boxes in the target visible light image;

[0018] Use soft non-maximum suppression algorithm to improve Score * The candidate boxes are processed to obtain the score ranking, and the candidate box with the highest score is selected as the fruit detection result of the target image.

[0019] The method for detecting apple fruit in complex scenarios based on multimodal fusion technology described in the present invention is also characterized by being performed in the following steps:

[0020] Step 5.1, Img RGB and Img Inf Input to infrared image encoder Φ RGB and visible light image encoder Φ Inf Feature extraction is performed in the corresponding visible light image feature set of the apple And infrared image feature set in, Indicates that the i-th light image in the c-th apple Features, represents the i-th infrared image of the c-th apple characteristics;

[0021] Infrared image encoder Φ RGB Using formula (1) to obtain the visible light fusion feature set

[0022]

[0023] In formula (1), represents the fusion feature of the i-th visible light image in the c-th apple, softmax represents the activation function, They are The query vector, The weight of the key vector, The weight of the value vector, d is Dimensions;

[0024] Step 5.2: Input V into the image encoder for feature extraction to obtain the visible light visual feature set in, represents the jth visible light visual feature of the cth type apple;

[0025] Step 5.3, create a A mask matrix of all 1s of the same size And through formula (2) and formula (3) and Perform L2 norm normalization to obtain the normalized visible light image feature set

[0026] And the normalized visible light visual feature set

[0027]

[0028] In formula (2) and formula (3), represents the normalized visible light image vector of the i-th apple in the c-th category, represents the jth visible light visual vector in the cth apple after normalization;

[0029] Step 5.4: Calculate IF and The similarity distance matrix between in, Represents the i-th visible light image vector of the c-th apple and the jth visual vector the distance between them;

[0030]

[0031] Step 5.5: Use equations (5) and (6) to calculate The mean and standard deviation

[0032]

[0033] Step 5.6: Calculate using formula (7) Distance threshold

[0034]

[0035] In formula (7), I×J is the total number of elements in Dist, α, β, γ are 4 hyperparameters. For distance The upper quartile of ; min(Dist) represents the minimum value of all elements in the distance matrix Dist, and max(Dist) represents the maximum value of all elements in the distance matrix Dist;

[0036] Step 5.7, and Compare and record in Dist Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained

[0037] The processed mask matrix and Perform dot product to obtain the i-th masked image feature in the c-th apple Thus, the masked visible light image features are obtained

[0038] Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping to obtain The query vector The key vector and The value vector of

[0039]

[0040] In formula (8) and formula (9), d k for Dimension; LayerNormq() is the query Mapping function, LayerNormkv is the key value The mapping function, Linear() is the linear transformation function;

[0041] Step 5.9: Get the fused visual features through formula (10) in, represents the jth visible light visual fusion feature of the cth apple;

[0042]

[0043] In formula (10), softmax() is a fully connected layer, for Dimensions;

[0044] Step 5.10: Use formula (11) to obtain the jth secondary fusion visible light visual feature of the cth apple

[0045]

[0046] In formula (11), X-MHA() is the multi-head attention mechanism function;

[0047] Step 5.11: Use formula (12) to get the text features of apple

[0048]

[0049] In formula (12), tanh() is the normalization function;

[0050] Step 5.12: Obtain the filtered visible light image features through formula (13) in, is the i-th filtered visible light image feature of the c-th apple;

[0051]

[0052] Step 5.13: Generate using formula (14) and Similarity score So as to obtain the matching score

[0053]

[0054] Step 5.14: Based on S, use RPN to generate a set of candidate bounding boxes in, represents the bth candidate box of the ath apple fruit in the i-th visible light image of the c-th apple;

[0055] Step 5.15: Calculate using formula (15) Raw matching score

[0056]

[0057] In formula (15), cosine is the cosine similarity function;

[0058] Step 5.16: Calculate using formula (16) Match score Thus, the matching score set of all candidate boxes is obtained

[0059] Furthermore, step 6 is performed as follows:

[0060] Step 6.1: Use formula (17) to construct the IOU loss L IOU :

[0061]

[0062] Step 6.2: Use formula (18) to construct EIOU loss L EIOU :

[0063] (18)

[0065] In formula (18), Represent candidate boxes and real frame The center point, Represent candidate boxes and real frame width, Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between; represent and The Euclidean distance between

[0066] Step 6.3, construct formula (19) to construct Focal-EIOU loss L Focal-EIOU :

[0067]

[0068] In formula (19), τ is the hyperparameter that controls the curvature of the curve;

[0069] Step 6.4: Use formula (20) to construct the classification loss

[0070]

[0071] Step 6.5: Use formula (21) to construct the total loss function

[0072]

[0073] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the apple fruit detection method in complex scenarios, and the processor is configured to execute the program stored in the memory.

[0074] The present invention provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, which is characterized in that when the computer program is run by a processor, the steps of the apple fruit detection method in complex scenarios are executed.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] 1. The present invention innovatively deeply fuses the three modalities of infrared light images, visible light images, and text, giving full play to the advantages of each for the detection of apple fruits. Infrared light images can effectively capture the temperature and radiation characteristics of apple images, which can provide additional structural information for the target; visible light images provide rich color and texture information to help identify details; text information enriches the semantic expression and provides additional background or context understanding. By comprehensively utilizing the characteristics of these three modalities and deeply fusing the three modalities, the present invention can effectively improve the accuracy of apple fruit detection in challenging environments such as changing lighting, complex backgrounds, or target occlusion.

[0077] 2. In order to enable the target detection model to learn more visual features, the present invention proposes a dual masking mechanism, that is, masking the key information in the visible light image and text before feature fusion. Compared with a single masking of the image or text, the dual masking mechanism proposed in the present invention allows the model to focus more on feature learning in key areas, allowing the model to learn more visual features, thereby improving the recognition ability of the target area.

[0078] 3. This invention uses the Focal-EIOU loss function to optimize the object detection model during the training phase, effectively improving the model's ability to detect objects in complex scenarios. During the inference phase, a soft non-maximum suppression algorithm is used to accurately screen candidate boxes, significantly improving detection accuracy in occluded objects and effectively reducing false and missed detections.

[0079] 4. This paper innovatively applies a large-scale visual-linguistic open vocabulary object detection model to the field of apple fruit detection. Because it has a parallel architecture of image encoder and visual encoder, it can be well used for multimodal feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0081] In this example, a method for detecting apples in complex scenes based on multimodal fusion technology is to deeply fuse infrared images, visible light images, and text features by leveraging the complementary advantages of different modalities. A multi-head attention mechanism is used to filter irrelevant and negative visible light image features before calculating word-region similarity to avoid introducing more noise. Soft non-maximum suppression and Focal-EIOU are also used to accurately select the target box. This significantly improves the detection accuracy of apples under complex conditions. The model structure diagram of this method is shown in the figure below. Figure 1 As shown, the steps are as follows:

[0082] Step 1: Build the dataset:

[0083] Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset in, represents the i-th visible light image of the c-th apple, represents the i-th infrared image of apples in the c-th category, C represents the number of categories, and I represents the total number of images of each category of apples;

[0084] Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. in, express , and Among them, a c,i express The ath apple fruit, A c,i for The total number of apple fruits, express The ath apple fruit a c,i The true label, express The ath apple fruit a c,i The label box, and express The center point coordinates, express width and height;

[0085] Step 2: Set the visible light image of apples of type c Input into the GLIP model for extraction to obtain the visible light visual information of apple type c Thus we get the visible light visual information set of apple in, represents the jth visual in the set of visible light images of apples of category c, and J represents the total number of visible light images in each category of apples;

[0086] Step 3: Construct a text prompt and use the BERT pre-trained language model to embed the text prompt to generate a text vector T = {t1,…,t c ,…,t C}, where t c The text representing apples of type c;

[0087] Step 4: In order to solve the problem that the model cannot introduce sufficient visual information in the subsequent visual text fusion, the present invention proposes a text masking strategy. Mask the text of any type of apple in T to obtain the masked text information T Mask ={t1,…,t c ,t mask ,…,t C}, where t mask Text representing a certain type of apple that is masked;

[0088] Step 5: Build an apple fruit detection network and perform RGB 、Img Inf and T MaskProcessing is performed to obtain the candidate bounding box O and its matching score set Score;

[0089] Step 5.1, Img RGB and Img Inf Input to infrared image encoder Φ RGB and visible light image encoder Φ Inf Feature extraction is performed in the corresponding visible light image feature set of the apple And infrared image feature set in, Indicates that the i-th light image in the c-th apple Features, represents the i-th infrared image of the c-th apple characteristics;

[0090] Φ RGB Using formula (1) to obtain the visible light fusion feature set

[0091]

[0092] In formula (1), represents the fusion feature of the i-th visible light image in the c-th apple, softmax represents the activation function, They are The query vector, The weight of the key vector, The weight of the value vector, d is Dimensions;

[0093] Step 5.2: Input V into the image encoder for feature extraction to obtain the visible light visual feature set in, represents the jth visible light visual feature of the cth type apple;

[0094] Step 5.3, create a A mask matrix of all 1s of the same size And through formula (2) and formula (3) and Perform L2 norm normalization to obtain the normalized visible light image feature set

[0095] And the normalized visible light visual feature set

[0096]

[0097] In formula (2) and formula (3), represents the normalized visible light image vector of the i-th apple in the c-th category, represents the jth visible light visual vector in the cth apple after normalization;

[0098] Step 5.4: Calculate IF and The similarity distance matrix between in, Represents the i-th visible light image vector of the c-th apple and the jth visual vector the distance between them;

[0099]

[0100] Step 5.5: Use equations (5) and (6) to calculate The mean and standard deviation

[0101]

[0102] Step 5.6: Calculate using formula (7) Distance threshold

[0103]

[0104] In formula (7), I×J is the total number of elements in Dist, α, β, γ are 4 hyperparameters. is the upper quartile of the distance; min(Dist) represents the minimum value of all elements in the distance matrix Dist, and max(Dist) represents the maximum value of all elements in the distance matrix Dist. In this example, The initial values ​​of α, β, and γ are set to 1.0, 0.3, 0.4, and 0.2;

[0105] Step 5.7, and Compare and record in Dist Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained

[0106] The processed mask matrix and Perform dot product to obtain the i-th masked image feature in the c-th apple Thus, the masked visible light image features are obtained

[0107] Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping to obtain The query vector The key vector and The value vector of

[0108]

[0109] In formula (8) and formula (9), d k for Dimension; LayerNormq() is the query Mapping function, LayerNormkv is the key value The mapping function, Linear() is the linear transformation function;

[0110] Step 5.9: Get the fused visual features through formula (10) in, represents the jth visible light visual fusion feature of the cth apple;

[0111]

[0112] In formula (10), softmax() is a fully connected layer, for Dimensions;

[0113] Step 5.10: Use formula (11) to obtain the jth secondary fusion visible light visual feature of the cth apple

[0114]

[0115] In formula (11), X-MHA() is the multi-head attention mechanism function. In this example, the number of heads of the multi-head attention mechanism is 4;

[0116] Step 5.9: Use formula (12) to get the text features of apple

[0117]

[0118] In formula (12), tanh() is the normalization function;

[0119] Step 5.11: Obtain the filtered visible light image features through formula (13) in, is the i-th filtered visible light image feature of the c-th apple;

[0120]

[0121] Step 5.12: Generate text features using formula (14) Visible light image features Match score ,in, express and Similarity score of

[0122]

[0123] Step 5.13: Generate candidate bounding boxes using RPN based on S in, represents the bth candidate box of the ath apple fruit in the i-th visible light image of the c-th apple;

[0124] Step 6.14: Calculate the candidate box using formula (15) Raw matching score

[0125]

[0126] In formula (15), cosine is the cosine similarity function;

[0127] Step 6.15: Calculate using formula (16) Match score Then we get the matching score set Score of all candidate boxes:

[0128]

[0129] Step 6. Construct the loss function of the apple fruit detection network:

[0130] Step 6.1: Use formula (17) to construct the IOU loss L IOU :

[0131]

[0132] Step 6.2: Use formula (18) to construct EIOU loss L EIOU :

[0133]

[0134] In formula (18), Represent candidate boxes and real frame The center point, Represent candidate boxes and real frame width, Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between; represent and The Euclidean distance between

[0135] Step 6.3, construct formula (19) to construct Focal-EIOU loss L Focal-EIOU :

[0136]

[0137] In formula (19), τ is the hyperparameter that controls the curvature of the curve;

[0138] Step 6.4: Use formula (20) to construct the classification loss

[0139]

[0140] Step 6.5: Use the total loss function of formula (21)

[0141]

[0142] Step 7, based on The apple fruit detection network is trained. When the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained;

[0143] Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score in, represents the bth apple in the i-th visible light image of the c-th apple. * The matching score of the predicted boxes, B * is the total number of prediction boxes;

[0144] Use soft non-maximum suppression algorithm to improve Score *The candidate boxes are processed and ranked by scores, and the candidate boxes with the highest scores are selected as the fruit detection results of the target image. This combination of EIOU and focal loss mechanism and soft non-maximum suppression target bounding box optimization and selection strategy effectively improves the model's positioning accuracy and detection accuracy for target areas that match text descriptions, and reduces missed detections and false detections.

[0145] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0146] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0147] In summary, the present invention first extracts features from infrared and visible light images separately using a dual-image encoder. A cross-attention mechanism is then introduced at a specific layer to achieve deep fusion of features from both modalities. The fused visible light image features are further processed using a masking mechanism and then fed into a multi-head attention mechanism along with the visible light visual features. This enhances the visible light visual features, allowing them to fully learn the fused visible light image features. Next, the enhanced visible light visual features are fed into the multi-head attention mechanism along with text features, further enriching the text features and integrating them with features from both infrared and visible light images. Ultimately, the text features fuse multimodal information. This is then evaluated using a multi-layer perceptron. The final output of the visible light image and text features is then fed into the multi-head attention mechanism to filter out negative visible light image features, thus reducing the impact of irrelevant features on subsequent operations. Finally, soft non-maximum suppression and Focal-EIOU are used to optimize candidate boxes and select the optimal target box. This improves apple fruit detection accuracy under complex model conditions.

Claims

1. A method for detecting apple fruit in complex scenes based on multimodal fusion technology, characterized in that: The steps are as follows: Step 1: Build a dataset: Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset in, represents the i-th visible light image of the c-th apple, represents the i-th infrared image of apples in the c-th category, C represents the number of categories, and I represents the total number of images of each category of apples; Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. in, express , and Among them, a c,i express The ath apple fruit, A c,i for The total number of apple fruits, express The ath apple fruit a c,i The real label, express The ath apple fruit a c,i The label box, and express The center point coordinates, express width and height; Step 2: Set the visible light image set of apples of type c Input into the GLIP model for extraction to obtain the visible light visual information of the c-th apple Thus, we can obtain the visible light visual information set of apple in, represents the jth vision in the visible light image set of the cth type of apple, and J represents the total number of visible light visions in each type of apple; Step 3: Construct a text prompt and use the BERT pre-trained language model to embed the text prompt to generate a text vector T = {t1,…,t c ,…,t C }, where t c Text representing apples of type c; Step 4: Mask the text of any type of apple in T to obtain the masked text information T Mask ={t1,…,t c ,t mask ,…,t C }, where t mask Text representing a certain type of apple that is masked; Step 5: Build an apple fruit detection network and perform RGB , Img Inf and T Mask Processing to obtain a candidate bounding box set and its matching score set in, represents the i-th visible light image of the c-th apple The bth candidate box of the ath apple fruit; express The matching score of Step 6: Construct the total loss function of the apple fruit detection network Step 7: Use the gradient descent method to train the apple fruit detection network and calculate To update the network parameters, when the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained; Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score in, represents the bth apple of the ath apple in the cth apple in the target visible light image. * prediction boxes, express The matching score, B * is the total number of predicted boxes in the target visible light image; Use the soft non-maximum suppression algorithm to improve the score * The candidate boxes are processed to obtain the score ranking, and the candidate box with the highest score is selected as the fruit detection result of the target image.

2. According to claim 1, the method for detecting apple fruit in complex scenes based on multimodal fusion technology is characterized in that: The steps are as follows: Step 5.1, Img RGB and Img Inf Input to the infrared image encoder Φ RGB and the visible light image encoder Φ Inf Feature extraction is performed in the image, and the corresponding visible light image feature set of the apple is obtained. And infrared image feature set in, Indicates that the i-th apple in the c-th category can be imaged Features, represents the i-th infrared image of the c-th apple Features; Infrared image encoderΦ RGB Using formula (1), we can get the visible light fusion feature set: In formula (1), represents the fusion feature of the i-th visible light image in the c-th apple, softmax represents the activation function, They are The query vector The weight of the key vector, The weight of the value vector, d is Dimensions; Step 5.2: Input V into the image encoder for feature extraction to obtain the visible light visual feature set in, represents the jth visible light visual feature of the cth type of apple; Step 5.3, create a The mask matrix of all 1s of the same size And through formula (2) and formula (3) and Perform L2 norm normalization to obtain the normalized visible light image feature set And the normalized visible light visual feature set In formula (2) and formula (3), represents the i-th visible light image vector of the c-th apple after normalization, represents the jth visible light visual vector in the cth apple after normalization; Step 5.4: Calculate IF and The similarity distance matrix between in, represents the i-th visible light image vector in the c-th apple and the jth visual vector The distance between Step 5.5: Use equation (5) and equation (6) to calculate The mean and standard deviation Step 5.6: Calculate using formula (7) The distance threshold In formula (7), I×J is the total number of elements in Dist, α, β, γ are 4 hyperparameters. For distance The upper quartile of ; min(Dist) represents the minimum value of all elements in the distance matrix Dist, and max(Dist) represents the maximum value of all elements in the distance matrix Dist; Step 5.7: and Compare and record in Dist Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained The processed mask matrix and Perform dot product to get the i-th masked image feature in the c-th apple Thus, the masked visible light image features are obtained Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping and get The query vector The key vector and The value vector of In formula (8) and formula (9), d k for Dimension; LayerNormq() is the query Mapping function, LayerNormkv is the key value The mapping function, Linear() is the linear transformation function; Step 5.9: Get the fused visual features through formula (10) in, represents the jth visible light visual fusion feature of the cth type of apple; In formula (10), softmax() is a fully connected layer. for Dimensions; Step 5.10: Use formula (11) to obtain the jth secondary fusion visible light visual feature of the cth apple In formula (11), X-MHA() is the multi-head attention mechanism function; Step 5.11: Use formula (12) to get the text features of apple In formula (12), tanh() is the normalization function; Step 5.12: Obtain the filtered visible light image features through formula (13): in, is the i-th filtered visible light image feature of the c-th apple; Step 5.13: Generate using formula (14) and Similarity score So as to get the matching score Step 5.14: Based on S, use RPN to generate a candidate bounding box set in, represents the bth candidate box of the ath apple fruit in the i-th visible light image of the c-th apple; Step 5.15: Calculate using formula (15) Raw Match Score In formula (15), cosine is the cosine similarity function; Step 5.16: Calculate using formula (16) Match score Thus, the matching score set of all candidate boxes is obtained 3. The method for detecting apple fruit in complex scenes based on multimodal fusion technology according to claim 2, characterized in that: The step 6 is carried out as follows: Step 6.1: Use formula (17) to construct the IOU loss L IOU : Step 6.2: Use formula (18) to construct the EIOU loss L EIOU : In formula (18), Represent candidate boxes and real frame The center point of Represent candidate boxes and real frame The width, Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between them; represent and The Euclidean distance between Step 6.3: Construct Focal-EIOU loss L using formula (19) Focal-EIOU : In formula (19), τ is the hyperparameter that controls the curvature of the curve; Step 6.4: Use formula (20) to construct the classification loss Step 6.5: Use formula (21) to construct the total loss function 4. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the apple fruit detection method in a complex scene as described in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting apple fruit in a complex scene in any one of claims 1 to 3 are executed.

Images