A method for detecting apple fruit in complex scenes based on multimodal fusion technology

Abstract

The present invention discloses a method for detecting apple fruits in complex scenarios based on multimodal fusion technology, comprising the following steps: 1. constructing a data set; 2. fusing visible light images with infrared image features; 3. masking the fused visible light image features and fusing them with visual features; 4. fusing the fused visual features with text features; 5. filtering the visible light image features; 6. inputting the visible light image features and text features into word-region similarity calculation; 7. selecting the optimal target frame by combining Soft-NMS and Focal-EIOU. The present invention fuses features of three modalities: infrared images, visible light images, and text, comprehensively considering the complementary advantages of different modalities. Soft non-maximum suppression (Soft-NMS) and Focal-EIOU are also used to select the optimal target frame, effectively improving the detection accuracy of apple fruits under complex conditions.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and mainly relates to a method for detecting apple fruits in complex scenes based on multimodal fusion technology. Background Art

[0002] In recent years, artificial intelligence technologies, particularly deep learning, have made significant progress in the field of computer vision. Object detection, a key research area, has been widely applied in agriculture, including automated fruit picking, yield assessment, and quality inspection. However, one of the main challenges facing object detection in orchards is the occlusion of fruit by other fruits, or by fruit from background objects such as leaves and branches. This occlusion not only affects the precise positioning of the target but also disrupts the extraction of the fruit's appearance features, resulting in reduced detection performance.

[0003] Multimodal fusion technology can significantly improve the performance of target detection models by combining and analyzing information from different sources (such as visual modalities and textual modalities). Specifically, in agricultural target detection, the combination of textual modalities (such as natural language queries based on fruit descriptions) and visual modalities (such as fruit image features) can provide richer contextual information, giving the model a more comprehensive understanding of target features. By introducing multimodal information, the model can better adapt to complex relationships in occluded scenes, identify partially occluded fruits, and improve the robustness and accuracy of detection.

[0004] Traditional object detection methods, such as detectors based on classic machine learning (e.g., Haar features + Adaboost, HOG + SVM) and early deep learning models (e.g., the R-CNN series, the YOLO series, and SSD), typically rely solely on a single visual modality for object detection. In orchard scenarios, these methods struggle to extract complete target features from a single visual input when fruit is obscured by leaves or branches, resulting in a significant drop in detection performance. Furthermore, these methods lack effective fusion of multimodal data and are unable to fully leverage the semantic prior knowledge provided by text modalities. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes an apple fruit detection method in complex scenes based on multimodal fusion technology, in order to fully utilize the complementary advantages of different modalities and realize accurate recognition of apple fruits in occluded scenes, thereby effectively improving the robustness and accuracy of detection.

[0006] In order to solve the technical problem, the present invention adopts the following technical solution:

[0007] The present invention is characterized in that the method for detecting apple fruit in complex scenes based on multimodal fusion technology is performed according to the following steps:

[0008] Step 1: Build the dataset:

[0009] Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset in, represents the i-th visible light image of the c-th apple, represents the i-th infrared image of apples in the c-th category, C represents the number of categories, and I represents the total number of images of each category of apples;

[0010] Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. in, express , and Among them, a c,i express The ath apple fruit, A c,i for The total number of apple fruits, express The ath apple fruit a c,i The true label, express The ath apple fruit a c,i The label box, and express The center point coordinates, express width and height;

[0011] Step 2: Set the visible light image of apples of type c Input into the GLIP model for extraction to obtain the visible light visual information of apple type c Thus we get the visible light visual information set of apple in, represents the jth visual in the set of visible light images of apples of category c, and J represents the total number of visible light images in each category of apples;

[0012] Step 3: Construct a text prompt and use the BERT pre-trained language model to embed the text prompt to generate a text vector T = {t1,…,t c ,…,t C}, where t c The text representing apples of type c;

[0013] Step 4: Mask the text of any type of apple in T to obtain the masked text information T Mask ={t1,…,t c ,t mask ,…,t C}, where t mask Text representing a certain type of apple that is masked;

[0014] Step 5: Build an apple fruit detection network and perform RGB 、Img Inf and T Mask Process and obtain the candidate bounding box set and its matching score set in, represents the i-th visible light image of the c-th apple The bth candidate box of the ath apple fruit; express Matching score;

[0015] Step 6: Construct the total loss function of the apple fruit detection network

[0016] Step 7: Use the gradient descent method to train the apple fruit detection network and calculate To update the network parameters, when the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained;

[0017] Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score in, represents the bth apple of the cth apple in the target visible light image and the i-th apple in the a-th apple in the visible light image. * prediction boxes, express The matching score, B * is the total number of predicted boxes in the target visible light image;

[0018] Use the soft non-maximum suppression algorithm to improve the score * The candidate boxes are processed to obtain the score ranking, and the candidate box with the highest score is selected as the fruit detection result of the target image.

[0019] The method for detecting apple fruit in complex scenarios based on multimodal fusion technology described in the present invention is also characterized by being performed in the following steps:

[0020] Step 5.1, Img RGB and Img Inf Input to infrared image encoder Φ RGB and visible light image encoder Φ Inf Feature extraction is performed in the corresponding visible light image feature set of the apple And infrared image feature set in, Indicates that the i-th light image in the c-th apple Features, represents the i-th infrared image of the c-th apple characteristics;

[0021] Infrared image encoder Φ RGB Using formula (1) to obtain the visible light fusion feature set

[0022]

[0023] In formula (1), represents the fusion feature of the i-th visible light image in the c-th apple, softmax represents the activation function, They are The query vector, The weight of the key vector, The weight of the value vector, d is Dimensions;

[0024] Step 5.2: Input V into the image encoder for feature extraction to obtain the visible light visual feature set in, represents the jth visible light visual feature of the cth type apple;

[0025] Step 5.3, create a A mask matrix of all 1s of the same size And through formula (2) and formula (3) and Perform L2 norm normalization to obtain the normalized visible light image feature set

[0026] And the normalized visible light visual feature set

[0027]

[0028] In formula (2) and formula (3), represents the normalized visible light image vector of the i-th apple in the c-th category, represents the jth visible light visual vector in the cth apple after normalization;

[0029] Step 5.4: Calculate IF and The similarity distance matrix between in, Represents the i-th visible light image vector of the c-th apple and the jth visual vector the distance between them;

[0030]

[0031] Step 5.5: Use equations (5) and (6) to calculate The mean and standard deviation

[0032]

[0033] Step 5.6: Calculate using formula (7) Distance threshold

[0034]

[0035] In formula (7), I×J is the total number of elements in Dist, α, β, γ are 4 hyperparameters. For distance The upper quartile of ; min(Dist) represents the minimum value of all elements in the distance matrix Dist, and max(Dist) represents the maximum value of all elements in the distance matrix Dist;

[0036] Step 5.7, and Compare and record in Dist Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained

[0037] The processed mask matrix and Perform dot product to obtain the i-th masked image feature in the c-th apple Thus, the masked visible light image features are obtained

[0038] Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping to obtain The query vector The key vector and The value vector of

[0039]

[0040] In formula (8) and formula (9), d k for Dimension; LayerNormq() is the query Mapping function, LayerNormkv is the key value The mapping function, Linear() is the linear transformation function;

[0041] Step 5.9: Get the fused visual features through formula (10) in, represents the jth visible light visual fusion feature of the cth apple;

[0042]

[0043] In formula (10), softmax() is a fully connected layer, for Dimensions;

[0044] Step 5.10: Use formula (11) to obtain the jth secondary fusion visible light visual feature of the cth apple

[0045]

[0046] In formula (11), X-MHA() is the multi-head attention mechanism function;

[0047] Step 5.11: Use formula (12) to get the text features of apple

[0048]

[0049] In formula (12), tanh() is the normalization function;

[0050] Step 5.12: Obtain the filtered visible light image features through formula (13) in, is the i-th filtered visible light image feature of the c-th apple;

[0051]

[0052] Step 5.13: Generate using formula (14) and Similarity score So as to obtain the matching score

[0053]

[0054] Step 5.14: Based on S, use RPN to generate a set of candidate bounding boxes in, represents the bth candidate box of the ath apple fruit in the i-th visible light image of the c-th apple;

[0055] Step 5.15: Calculate using formula (15) Raw matching score

[0056]

[0057] In formula (15), cosine is the cosine similarity function;

[0058] Step 5.16: Calculate using formula (16) Match score Thus, the matching score set of all candidate boxes is obtained

[0059] Furthermore, step 6 is performed as follows:

[0060] Step 6.1: Use formula (17) to construct the IOU loss L IOU :

[0061]

[0062] Step 6.2: Use formula (18) to construct EIOU loss L EIOU :

[0063] (18)

[0065] In formula (18), Represent candidate boxes and real frame The center point, Represent candidate boxes and real frame width, Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between; represent and The Euclidean distance between

[0066] Step 6.3, construct formula (19) to construct Focal-EIOU loss L Focal-EIOU :

[0067]

[0068] In formula (19), τ is the hyperparameter that controls the curvature of the curve;

[0069] Step 6.4: Use formula (20) to construct the classification loss

[0070]

[0071] Step 6.5: Use formula (21) to construct the total loss function

[0072]

[0073] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the apple fruit detection method in the complex scene, and the processor is configured to execute the program stored in the memory.

[0074] The present invention provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, which is characterized in that when the computer program is run by a processor, the steps of the apple fruit detection method in complex scenarios are executed.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] 1. The present invention innovatively deeply fuses the three modalities of infrared light images, visible light images, and text, giving full play to the advantages of each for the detection of apple fruits. Infrared light images can effectively capture the temperature and radiation characteristics of apple images, which can provide additional structural information for the target; visible light images provide rich color and texture information to help identify details; text information enriches the semantic expression and provides additional background or context understanding. By comprehensively utilizing the characteristics of these three modalities and deeply fusing the three modalities, the present invention can effectively improve the accuracy of apple fruit detection in challenging environments such as changing lighting, complex backgrounds, or target occlusion.

[0077] 2. In order to enable the target detection model to learn more visual features, the present invention proposes a dual masking mechanism, that is, masking the key information in the visible light image and text before feature fusion. Compared with a single masking of the image or text, the dual masking mechanism proposed in the present invention allows the model to focus more on feature learning in key areas, allowing the model to learn more visual features, thereby improving the recognition ability of the target area.

[0078] 3. This invention uses the Focal-EIOU loss function to optimize the object detection model during the training phase, effectively improving the model's ability to detect objects in complex scenarios. During the inference phase, a soft non-maximum suppression algorithm is used to accurately screen candidate boxes, significantly improving detection accuracy in occluded objects and effectively reducing false and missed detections.

[0079] 4. This paper innovatively applies a large-scale visual-linguistic open vocabulary object detection model to the field of apple fruit detection. Because it has a parallel architecture of image encoder and visual encoder, it can be well used for multimodal feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0081] In this example, a method for detecting apples in complex scenes based on multimodal fusion technology is to deeply fuse infrared images, visible light images, and text features by leveraging the complementary advantages of different modalities. A multi-head attention mechanism is used to filter irrelevant and negative visible light image features before calculating word-region similarity to avoid introducing more noise. Soft non-maximum suppression and Focal-EIOU are also used to accurately select the target box. This significantly improves the detection accuracy of apples under complex conditions. The model structure diagram of this method is shown in the figure below. Figure 1 As shown, the steps are as follows:

[0082] Step 1: Build the dataset:

[0083] Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset in, represents the i-th visible light image of the c-th apple, represents the i-th infrared image of apples in the c-th category, C represents the number of categories, and I represents the total number of images of each category of apples;

[0084] Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. in, express , and Among them, a c,i express The ath apple fruit, A c,i for The total number of apple fruits, express The ath apple fruit a c,i The true label, express The ath apple fruit a c,i The label box, and express The center point coordinates, express width and height;

[0085] Step 2: Set the visible light image of apples of type c Input into the GLIP model for extraction to obtain the visible light visual information of apple type c Thus we get the visible light visual information set of apple in, represents the jth visual in the set of visible light images of apples of category c, and J represents the total number of visible light images in each category of apples;

[0086] Step 3: Construct a text prompt and use the BERT pre-trained language model to embed the text prompt to generate a text vector T = {t1,…,t c ,…,t C}, where t c The text representing apples of type c;

[0087] Step 4: In order to solve the problem that the model cannot introduce sufficient visual information in the subsequent visual text fusion, the present invention proposes a text masking strategy. Mask the text of any type of apple in T to obtain the masked text information T Mask ={t1,…,t c ,t mask ,…,t C}, where t mask Text representing a certain type of apple that is masked;

[0088] Step 5: Build an apple fruit detection network and perform RGB 、Img Inf and T MaskProcessing is performed to obtain the candidate bounding box O and its matching score set Score;

[0089] Step 5.1, Img RGB and Img Inf Input to infrared image encoder Φ RGB and visible light image encoder Φ Inf Feature extraction is performed in the corresponding visible light image feature set of the apple And infrared image feature set in, Indicates that the i-th light image in the c-th apple Features, represents the i-th infrared image of the c-th apple characteristics;

[0090] Φ RGB Using formula (1) to obtain the visible light fusion feature set

[0091]

[0092] In formula (1), represents the fusion feature of the i-th visible light image in the c-th apple, softmax represents the activation function, They are The query vector, The weight of the key vector, The weight of the value vector, d is Dimensions;

[0093] Step 5.2: Input V into the image encoder for feature extraction to obtain the visible light visual feature set in, represents the jth visible light visual feature of the cth type apple;

[0094] Step 5.3, create a A mask matrix of all 1s of the same size And through formula (2) and formula (3) and Perform L2 norm normalization to obtain the normalized visible light image feature set

[0095] And the normalized visible light visual feature set

[0096]

[0097] In formula (2) and formula (3), represents the normalized visible light image vector of the i-th apple in the c-th category, represents the jth visible light visual vector in the cth apple after normalization;

[0098] Step 5.4: Calculate IF and The similarity distance matrix between in, Represents the i-th visible light image vector of the c-th apple and the jth visual vector the distance between them;

[0099]

[0100] Step 5.5: Use equations (5) and (6) to calculate The mean and standard deviation

[0101]

[0102] Step 5.6: Calculate using formula (7) Distance threshold

[0103]

[0104] In formula (7), I×J is the total number of elements in Dist, α, β, γ are 4 hyperparameters. is the upper quartile of the distance; min(Dist) represents the minimum value of all elements in the distance matrix Dist, and max(Dist) represents the maximum value of all elements in the distance matrix Dist. In this example, The initial values ​​of α, β, and γ are set to 1.0, 0.3, 0.4, and 0.2;

[0105] Step 5.7, and Compare and record in Dist Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained

[0106] The processed mask matrix and Perform dot product to obtain the i-th masked image feature in the c-th apple Thus, the masked visible light image features are obtained

[0107] Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping to obtain The query vector The key vector and The value vector of

[0108]

[0109] In formula (8) and formula (9), d k for Dimension; LayerNormq() is the query Mapping function, LayerNormkv is the key value The mapping function, Linear() is the linear transformation function;

[0110] Step 5.9: Get the fused visual features through formula (10) in, represents the jth visible light visual fusion feature of the cth apple;

[0111]

[0112] In formula (10), softmax() is a fully connected layer, for Dimensions;

[0113] Step 5.10: Use formula (11) to obtain the jth secondary fusion visible light visual feature of the cth apple

[0114]

[0115] In formula (11), X-MHA() is the multi-head attention mechanism function. In this example, the number of heads of the multi-head attention mechanism is 4;

[0116] Step 5.9: Use formula (12) to get the text features of apple

[0117]

[0118] In formula (12), tanh() is the normalization function;

[0119] Step 5.11: Obtain the filtered visible light image features through formula (13) in, is the i-th filtered visible light image feature of the c-th apple;

[0120]

[0121] Step 5.12: Generate text features using formula (14) Visible light image features Match score ,in, express and Similarity score of

[0122]

[0123] Step 5.13: Generate candidate bounding boxes using RPN based on S in, represents the bth candidate box of the ath apple fruit in the i-th visible light image of the c-th apple;

[0124] Step 6.14: Calculate the candidate box using formula (15) Raw matching score

[0125]

[0126] In formula (15), cosine is the cosine similarity function;

[0127] Step 6.15: Calculate using formula (16) Match score Then we get the matching score set Score of all candidate boxes:

[0128]

[0129] Step 6. Construct the loss function of the apple fruit detection network:

[0130] Step 6.1: Use formula (17) to construct the IOU loss L IOU :

[0131]

[0132] Step 6.2: Use formula (18) to construct EIOU loss L EIOU :

[0133]

[0134] In formula (18), Represent candidate boxes and real frame The center point, Represent candidate boxes and real frame width, Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between; represent and The Euclidean distance between

[0135] Step 6.3, construct formula (19) to construct Focal-EIOU loss L Focal-EIOU :

[0136]

[0137] In formula (19), τ is the hyperparameter that controls the curvature of the curve;

[0138] Step 6.4: Use formula (20) to construct the classification loss

[0139]

[0140] Step 6.5: Use the total loss function of formula (21)

[0141]

[0142] Step 7, based on The apple fruit detection network is trained. When the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained;

[0143] Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score in, represents the bth apple in the i-th visible light image of the c-th apple. * The matching score of the predicted boxes, B * is the total number of prediction boxes;

[0144] Use the soft non-maximum suppression algorithm to improve the score *The candidate boxes are processed and ranked by scores, and the candidate boxes with the highest scores are selected as the fruit detection results of the target image. This combination of EIOU and focal loss mechanism and soft non-maximum suppression target bounding box optimization and selection strategy effectively improves the model's positioning accuracy and detection accuracy for target areas that match text descriptions, and reduces missed detections and false detections.

[0145] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0146] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0147] In summary, the present invention first uses a dual-image encoder to extract features from both infrared and visible light images. A cross-attention mechanism is then introduced at a specific layer to achieve deep fusion of the two modal features. The fused visible light image features are further processed using a masking mechanism and then fed into a multi-head attention mechanism along with the visible light visual features. This enhances the visible light visual features and fully learns the fused visible light image features. Next, the enhanced visible light visual features are fed into the multi-head attention mechanism along with the text features, further enriching the text features and integrating them with the features of both the infrared and visible light images. Ultimately, the text features fuse multimodal information. This is then evaluated using a multi-layer perceptron. The final output of the visible light image and text features is then fed into the multi-head attention mechanism to filter out negative visible light image features, thus reducing the impact of irrelevant features on subsequent operations. Finally, a soft non-maximum suppression algorithm and Focal-EIOU are used to optimize candidate boxes and select the optimal target box. This improves apple fruit detection accuracy under complex model conditions.

Claims

1. A method for detecting apple fruit in complex scenes based on multimodal fusion technology, characterized in that: The steps are as follows: Step 1: Build the dataset: Step 1.1: Use a depth camera to collect visible light images and infrared images of apples in the orchard, and obtain a visible light image dataset of apples. and Apple's infrared image dataset ,in, Indicates the Apple A visible light image, Indicates the Apple Infrared image, represents the number of categories, Represents the total number of images of each type of apple; Step 1.2: Use the annotation tool to annotate the visible light images of apples and obtain the annotation set of the visible light image dataset of apples. ,in, express , and ,in, express No. Apple fruit, for The total number of apple fruits, express Middle Apple fruit The true label, express Middle Apple fruit The label box, and , express The center point coordinates, express width and height; Step 2: The visible light image dataset of apples of category c Input to Extract the visible light visual information of apple of type c in the model , thus obtaining the visible light visual information set of the apple ;in, Indicates the The first in the collection of visible light images of apples A visual, Indicates the total number of visible light vision in each type of apple; Step 3: Build a text prompt and use The pre-trained language model performs text embedding on the text prompt to generate a text vector ,in, The text representing apples of type c; Step 4: Mask the text of any type of apple in and get the masked text information ,in, Text representing a certain type of apple that is masked; Step 5: Build an apple fruit detection network and 、 and Process and obtain the candidate bounding box set 、 and its matching score set ,in, Indicates the Apple Visible light images Middle The first apple fruit candidate boxes; express Matching score; Step 5.1, and Input to the visible light image encoder and infrared image encoder Feature extraction is performed in the corresponding visible light image feature set of the apple And infrared image feature set ,in, Indicates the Apple Visible light image Features, Indicates the Apple Infrared image characteristics; Visible light image encoder Using formula (1) to obtain the visible light fusion feature set ; (1) In formula (1), Indicates the Apple The fusion features of visible light images, represents the activation function, 、 、 They are The query vector, The weight of the key vector, The weight of the value vector, for Dimensions; Step 5.2, Input into the image encoder for feature extraction to obtain the visible light visual feature set ,in, Indicates the Apple Visible light vision features; Step 5.3, create a A mask matrix of all 1s of the same size , and through formula (2) and formula (3) and conduct Norm normalization is performed to obtain the normalized visible light image feature set And the normalized visible light visual feature set ; (2) (3) In formula (2) and formula (3), After normalization, Apple visible light image vectors, After normalization, Apple Visible light visual vector; Step 5.4: Calculate using formula (4) and The similarity distance matrix between ,in, Indicates the Apple visible light image vector Hedi Vision vector the distance between them; (4) Step 5.5: Use equations (5) and (6) to calculate The mean and standard deviation ; (5) (6) Step 5.6: Calculate using formula (7) Distance threshold : (7) In formula (7), are 4 hyperparameters, For distance The upper quartile of Represents the distance matrix The minimum value of all elements in , Represents the distance matrix The maximum value of all elements in ; Step 5.7, and Compare and record middle Less than The index position of the mask matrix The corresponding index position in is set to 0, and the processed mask matrix is ​​obtained ; The processed mask matrix and Perform dot product and get Apple Image features with masks , thereby obtaining the masked visible light image features ; Step 5.8: Use equations (8) and (9) to calculate and Perform linear mapping to obtain The query vector , The key vector and The value vector of ; (8) (9) In formula (8) and formula (9), For query The mapping function, Key ,value The mapping function, is a linear transformation function; Step 5.9: Get the fused visual features through formula (10) = ,in, Indicates the Apple Visible light vision fusion features; (10) In formula (10), is the fully connected layer, for Dimensions; Step 5.10: Use formula (11) to get Apple Visible light visual features after secondary fusion ; (11) In formula (11), is the multi-head attention mechanism function; Step 5.11: Use formula (12) to get the text features of apple : (12) In formula (12), is the normalization function; Step 5.12: Obtain the filtered visible light image features through formula (13) ,in, For the Apple filtered visible light image features; (13) Step 5.13: Generate using formula (14) and Similarity score , thus obtaining the matching score ; (14) Step 5.14, according to ,use Generate candidate bounding box sets ,in, Indicates the Apple In the visible light image The first apple fruit candidate boxes; Step 5.15: Calculate using formula (15) Raw matching score : (15) In formula (15), is the cosine similarity function; Step 5.16: Calculate using formula (16) Match score , thereby obtaining the matching score set of all candidate boxes : (16) Step 6: Construct the total loss function of the apple fruit detection network ; Step 7: Use the gradient descent method to train the apple fruit detection network and calculate To update the network parameters, when the number of training iterations reaches the set number or When convergence occurs, the training stops, and the optimal apple fruit detection model is obtained; Step 8: After obtaining the target visible light image and text prompt, input them into the optimal apple fruit detection model for processing to obtain all predicted boxes. Match score ,in, Indicates the first Apple In the visible light image The first apple fruit prediction boxes, express The matching score, is the total number of predicted boxes in the target visible light image; Using soft non-maximum suppression algorithm The candidate boxes are processed to obtain the score ranking, and the candidate box with the highest score is selected as the fruit detection result of the target image.

2. The method for detecting apple fruit in complex scenes based on multimodal fusion technology according to claim 1, characterized in that: Described step 6 is carried out as follows: Step 6.1: Use formula (17) to construct loss : (17) Step 6.2: Use formula (18) to construct loss : (18) In formula (18), Represent candidate boxes and real frame The center point, Represent candidate boxes and real frame width, , Represent candidate boxes and real frame of high, and Represents a candidate box and real frame The width and height of the minimum bounding box between; represent and The Euclidean distance between Step 6.3: Construct formula (19) loss : (19) In formula (19), is the hyperparameter that controls the curvature of the curve; Step 6.4: Use formula (20) to construct the classification loss : (20) Step 6.5: Use formula (21) to construct the total loss function : (21)。 3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the apple fruit detection method in a complex scenario as described in any one of claims 1-2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting apple fruit in a complex scene according to any one of claims 1 to 2 are executed.

Images